license: apache-2.0 base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text tags:

• qwen3_5
• autoround
• w8a16
• int8
• 8-bit
• compressed-tensors
• vllm
• marlin
• mtp
• vision
• uncensored

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — W8A16 AutoRound

8-bit weight-only quantization (W8A16, int8 symmetric, group_size 128) of DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU using AutoRound via llm-compressor 0.14

• auto-round 0.15.1, with 400 AutoRound iterations and 256 calibration samples.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized, served by vLLM's Marlin kernels (uint8b128).
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
• Calibration: 256 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template.
• Size: ~31.6 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens. Deliberately not the calibration set. Same texts, same tokenization, same method (raw text, no chat template), both models served by vLLM.

This checkpoint is the third and highest-fidelity arm of a series run on the same protocol. The BF16 baseline reproduced identically (8.0880) across the W6A16 and W8A16 runs, so the deltas below are directly comparable:

Reading this result

+5.91% is still above the ~5% degradation usually accepted for weight-only quantization, even though this is 8-bit with double the tuning budget. Two honest caveats:

1. Three variables changed at once versus the W4A16 baseline — bit width, iterations and calibration samples. So this is a best-effort fidelity run, not a controlled isolation of the bit-width effect. Do not read the delta as "what 8 bits buys you".

2. The returns are clearly diminishing: +8.99% → +7.36% → +5.91%. Doubling bits and the tuning budget bought roughly one percentage point. That pattern suggests the residual error is not dominated by weight precision — if it were, 8-bit (int8, group 128) would typically land well under 1–2%.

A plausible next lever, and the one we would try first, is excluding the GatedDeltaNet (linear-attention) projections from quantization — keeping in_proj_qkv, in_proj_z and the linear-attention out_proj in BF16 — since hybrid linear-attention layers are a known sensitivity in this architecture family. That trades some compression for fidelity instead of pushing the bit width further on the same modules. Note this is a hypothesis, not a measured result.

Perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation. Short side-by-side chat samples against the BF16 original read as equivalent, which is exactly why the perplexity number — not eyeballing — is what is published here.

Serving notes

8-bit compressed-tensors weights are served by Marlin, so unlike the 6-bit build of this model there is no dependency on the JIT-compiled Humming kernels:

Using MarlinLinearKernel for CompressedTensorsWNA16

No throughput advantage was measured. On the final checkpoints, same script and TP2, the W8A16 model was slower on plain decode (24.6 vs 26.5 tok/s) and faster with MTP speculative decoding (42.6 vs 34.5) and on vision (19.4 vs 17.3). These are 2-prompt cold-JIT runs, not benchmarks; the sign flips between phases, so no reliable throughput claim can be made in either direction. The reason to prefer this build is fidelity, and it costs ~6 GB more than the 6-bit build.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W8A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28: text, MTP speculative decoding and vision all confirmed, 400 packed modules, in_proj_a/b correctly left in BF16.
• This is a thinking model: with a small max_tokens the whole budget can be consumed by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• No training, fine-tuning or abliteration was performed here — this is a weight-only quantization. The behavioural characteristics, abliteration and usage warnings of the upstream model are inherited unchanged.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.

License

Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.