license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags:

• qwen3_5
• quantized
• w4a16
• auto-round
• compressed-tensors
• vllm
• uncensored base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound

W4A16 pack-quantized (compressed-tensors, vLLM-ready) version of DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — a Qwen3.8-27B (qwen3_5) finetune, Heretic/uncensored.

Built for single-GPU (RTX 3090 24 GB) vLLM serving with long context: the main transformer body is int4, lm_head / embeddings / MTP draft module are int8, and a vocab-truncated draft head (40960 tokens) is included for MTP speculative decoding.

Note on use: the base model is uncensored (refusal behavior removed by its authors). Responsibility for how you use this quantization lies with you.

What this is

How much context fits (measured on RTX 3090)

Pool size varies a few percent between boots; 140k @ 0.94 is the safe everyday setting. FP8 KV cache (--kv-cache-dtype fp8), prefix caching on, MTP speculative decoding with 3 draft tokens.

Run it (example)

vllm serve ababaka/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound \
  --served-model-name qwen3.8-27b \
  --host 0.0.0.0 --port 18020 \
  --gpu-memory-utilization 0.94 \
  --max-model-len 140000 \
  --max-num-seqs 8 \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --mamba-ssm-cache-dtype float16 \
  --async-scheduling \
  --max-num-batched-tokens 2048 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' \
  --compilation-config '{"max_cudagraph_capture_size":32,"custom_ops":["+rms_norm","+silu_and_mul"]}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --enable-prefix-caching --mamba-cache-mode align

For 160k context use --gpu-memory-utilization 0.96 --max-model-len 160000. --language-model-only skips the vision tower (text serving). The chat template of the base model is used by default; a Sharp custom template (peculiar-ragdoll/Qwen-Sharp-Chat-Templates, reasoning_effort via --default-chat-template-kwargs) was used in testing.

Files

• model-00001..07-of-00007.safetensors — quantized transformer + int8 lm_head/embed_tokens
• model_extra_tensors.safetensors — int8 MTP module + mtp.draft_lm_head (40960-token draft head)
• mtp_draft_vocab_ids.pt — token-id map for the truncated draft vocabulary
• model.safetensors.index.json, config.json (+ quantization_config.json), tokenizer/processor configs

Base

Downloaded DavidAU/...-Heretic-Uncensored-NM-DAU in FP16 (13 shards incl. model-mtp-restored.safetensors, ~55 GB). Verified same skeleton as the reference Qwen3.8-27B quant: qwen3_5, 64 layers (48 linear + 16 full), vocab 248320, tie_word_embeddings=false, mtp.fc + 7 MTP linears, model.visual.* tower.

Quantization (auto-round 0.14.2)

layer_config kept in BF16: lm_head, linear_attn.in_proj_a/b (all 48 linear-attention layers), visual.* (whole vision tower), mtp (whole MTP module). Full run on RTX 3090: 64/64 layers in ~2h41m (~141–159 s/layer), peak VRAM 15.5 GB / RAM 27.3 GB. Result: 400/607 submodules quantized; ignore list 303 entries (110 visual + 192 linear_attn + lm_head), no mtp/embed entries — MTP was copied unquantized into model_extra_tensors.safetensors by the exporter.

Post-quantization pipeline (in this order)

1. quant_lm_head — lm_head → int8 g128 (round-trip rel. error 0.64%, ~1.3 GB VRAM saved)
2. quant_embed — embed_tokens → int8 g128, scales in bf16 (untied embeddings, error 0.60%, ~1.3 GB saved)
3. quant_mtp — mtp.fc + 7 MTP linears → int8 g128 (errors 0.6–1.5%)
4. build_draft_vocab — draft head sliced from the int8 lm_head to the 40960 most frequent token ids → mtp.draft_lm_head.* (213 MB) in model_extra_tensors.safetensors
   • mtp_draft_vocab_ids.pt

Final config_groups (order matters for compressed-tensors matching in vLLM — first match wins, so the lm_head regex must precede the mtp one):

group_0  targets=["Linear"]             int4 g128 sym   — main body
group_1  targets=["re:.*lm_head$"]      int8 g128       — lm_head + mtp.draft_lm_head
group_2  targets=["re:.*embed_tokens$"] int8 g128       — embed_tokens
group_3  targets=["re:^mtp\\..*"]       int8 g128       — MTP module

ignore: 302 entries (visual + linear_attn norms/projections); lm_head/mtp/embed are NOT ignored. Verified by loading in vLLM: MTP drafter detected, 40960-token draft head recognized, coherent generation (no random-token garbage).

License

Base model is Apache 2.0; this quantization follows the base model's license.