license: apache-2.0 base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text tags:

• qwen3_5
• autoround
• w4a16
• int4
• 4-bit
• compressed-tensors
• vllm
• marlin
• mtp
• vision
• uncensored

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — W4A16 AutoRound (attention + GatedDeltaNet excluded)

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU, produced with AutoRound (llm-compressor 0.14 + auto-round 0.15.1, 400 iterations, 256 calibration samples), with the full-attention and GatedDeltaNet projections kept in BF16 because measurement showed they carry most of the quantization damage at 4 bits.

• Quantized → compressed-tensors pack-quantized (uint4b8), served by vLLM's Marlin kernels: the MLP projections only (192 modules: mlp.{gate,up,down}_proj × 64 layers).
• Kept in BF16 (bit-identical to the base model):
  • All full-attention projections — self_attn.{q,k,v,o}_proj, 64 modules
  • All GatedDeltaNet projections — linear_attn.{in_proj_qkv,in_proj_z,out_proj}, 144 modules
  • Vision tower (model.visual.*, 333 tensors) and MTP head (mtp.*, 15 tensors)
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b
• Calibration: 256 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template.
• Size: ~30.2 GB (base BF16 ~52 GB). 70% of the quantized parameters stay in 4 bits.

Quality

Perplexity on 50 held-out texts from wikitext-103, 16,007 evaluated tokens — deliberately not the calibration set. Same texts, same tokenization, same method, both models served by vLLM. The BF16 baseline is 8.0880 (reproduced identically across every run of this series).

Where the 4-bit damage comes from

Each group of modules was restored to BF16 in turn — surgically, on top of the fully-quantized 4-bit checkpoint, without re-quantizing — and the perplexity re-measured on the identical corpus:

Attention is still the single largest contributor at 4 bits (3.01 pp), and no single group was enough to reach the 5% band — attention alone leaves +5.73%. The attention + GatedDeltaNet pair is what crosses it.

The MLP is not excluded on purpose: it holds 17.1 B of the 24.3 B quantized parameters (70%), so restoring it to BF16 would produce a 42 GB checkpoint against 52 GB for plain BF16 — the quantization would stop being a compression. Keeping it in 4 bits is what makes this build 30.2 GB.

Honest note on the numbering

This build is dominated by its 6-bit sibling in the same series (…-W6A16-AutoRound, 27.6 GB, +2.40%): that one is smaller and more faithful, and being smaller it also reads fewer bytes per token. So on size, fidelity and decode bandwidth together, the 6-bit build wins.

What this checkpoint establishes is that +3.72% is reachable at 4 bits for this model — the project's stated target of ≤5% is met — and it documents exactly which module groups the 4-bit error lives in. If you are choosing one artifact to serve, choose the W6A16 build; if you specifically want a 4-bit MLP for its kernel/throughput characteristics, this is the most faithful 4-bit option we found.

Serving notes

The quantized weights are 4-bit compressed-tensors uint4b8, served by Marlin:

Using MarlinLinearKernel for CompressedTensorsWNA16

Attention and GatedDeltaNet projections run as ordinary BF16 GEMMs, which is what costs the bytes — those 208 modules are 20.4 B parameters that stay at 2 bytes each.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28: text, MTP speculative decoding and vision confirmed; 192 packed modules, 208 weights left in BF16.
• This is a thinking model: with a small max_tokens the whole budget can be consumed by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• The exclusion is surgical: the MLP was AutoRound-tuned with the attention and GatedDeltaNet quantized, so a re-quantization with those excluded from the start would likely do slightly better.
• No training, fine-tuning or abliteration was performed here — this is a weight-only quantization. The behavioural characteristics, abliteration and usage warnings of the upstream model are inherited unchanged.
• Perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation.

License

Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.