license: apache-2.0 base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text tags:

• qwen3_5
• autoround
• w6a16
• int6
• 6-bit
• compressed-tensors
• vllm
• humming
• mtp
• vision
• uncensored

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — W6A16 AutoRound

6-bit weight-only quantization (W6A16, int6 symmetric, group_size 128) of DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU using AutoRound via llm-compressor 0.14

• auto-round 0.15.1.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized.
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
• Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.
• Size: ~25.5 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens. Deliberately not the calibration set, which would flatter the result. Same texts, same tokenization (identical token count for both), same method (raw text, no chat template), both models served by vLLM:

This is above the ~5% degradation usually accepted for weight-only quantization, so treat the LM fidelity as degraded more than a typical quant. Read this number with the caveats below rather than as a verdict on 6-bit quantization in general.

How this compares to W4A16

A sibling run on a different but architecturally identical checkpoint (IstroSec/ThinkingCap-Qwen3.8-27B-abliterated) used the same protocol and recipe at 4-bit and measured BF16 8.1133 → W4A16 8.8426 = +8.99%.

The two BF16 baselines are close (8.09 vs 8.11), so the deltas are roughly comparable and suggest 6-bit recovers about 1.6 percentage points over 4-bit. This is suggestive, not a controlled comparison: the checkpoints differ, and perplexity sensitivity to quantization is a property of the individual model. To isolate the bit-width effect you would need to quantize this checkpoint at both widths with everything else held fixed.

If you need tighter fidelity

The first knob to turn is more AutoRound tuning, which is what the upstream guidance in this project recommends: iters=400 and nsamples=256 (roughly double the wall-clock). The recipe is also bit-width-agnostic, so producing a W8A16 variant of this same checkpoint is a one-flag change if 6-bit turns out not to be worth the tradeoff.

Note that perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation. Short side-by-side chat samples between this checkpoint and the BF16 original read as equivalent — perplexity averages small probability shifts over every token and is the more sensitive instrument here.

Serving notes — Humming, not Marlin

This is the most important operational difference from the W4A16 builds. For 6-bit weights vLLM uses the HummingLinearKernel (CompressedTensorsWNA16 with scalar_types.uint6b32), not Marlin: Marlin only implements uint4b8 and uint8b128. Confirmed on load:

Using HummingLinearKernel for CompressedTensorsWNA16

The Humming kernels are JIT-compiled, so expect a slower first load and different (generally lower) throughput than the Marlin-served W4A16 builds. Measured on 8×RTX 3090 (cold JIT, TP2, 2-prompt smoke test, enforce_eager): ~28 tok/s plain decode, ~45 tok/s with MTP speculative decoding. These are cold-start floor numbers from a 2-prompt run, not a throughput benchmark.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W6A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28: text, MTP speculative decoding and vision all confirmed, 400 packed modules, in_proj_a/b correctly left in BF16.
• This is a thinking model: with a small max_tokens the whole budget can be consumed by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• No training, fine-tuning or abliteration was performed here — this is a weight-only quantization. The behavioural characteristics, abliteration and usage warnings of the upstream model are inherited unchanged.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.

License

Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.