Qwen3.6-27B-abliterated-AutoRound-INT4-MTP

license: apache-2.0 base_model: Qwen/Qwen3.6-27B base_model_relation: quantized library_name: vllm pipeline_tag: text-generation language:

• en
• zh
• ru tags:
• qwen3.6
• int4
• autoround
• abliterated
• uncensored
• mtp
• speculative-decoding
• vllm
• long-context
• coding
• multimodal
• vision model-index:
• name: Qwen3.6-27B-abliterated-AutoRound-INT4-MTP results:
  • task: type: text-generation name: Code generation dataset: name: LiveCodeBench-hard (subset, reproducible harness) type: livecodebench-hard-subset metrics:
    • type: pass@1 value: 70.0 name: hard pass@1 (%) verified: false
  • task: type: text-generation name: Tool use dataset: name: Tool-use tasks (own harness) type: tool-use-own metrics:
    • type: pass@1 value: 87.0 name: tool-use pass@1 (%) verified: false

A gentle Heretic abliteration of the pristine Qwen/Qwen3.6-27B → AutoRound INT4 → the 15-tensor MTP head grafted back in BF16 so speculative decoding works out of the box in vLLM. Removing refusals cost zero hard-coding points (70 = 70 vs the clean base) and tool-use went up (87 vs 83). Pick a preset, run one script, done.

Highlights

Benchmarks — decensoring cost nothing, and where it sits

Same base Qwen/Qwen3.6-27B, one harness. Decensoring left hard unchanged (70 = 70 vs the reference narrow quant) and tool-use went up (83 → 87):

[!NOTE] Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in benchmark-results/). Conditions: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=4. Solutions run in a restricted sandbox (no network) — safe, identical scoring. The Lorbus row is our own measurement of their public quant on the same tasks, not their claim. The 80B Coder-Next row is likewise our own measurement of the UD-IQ4_XS GGUF (via LM Studio) on the same task set — a bigger, different model shown for scale. Reproduce it, don't trust it.

Surgically gentle abliteration

Abliteration strips the refusal direction — done carelessly it also scrambles the model's brains. Heretic co-minimizes refusals and KL divergence from the original. Lower KL = closer to the untouched model = intelligence kept. We tuned for minimum drift:

~18× below Heretic's own showcase, 50–100× below manual. The model barely moved from the base, so <think> reasoning and coding stayed intact. (KL is model/harness-dependent, so cross-model figures aren't a strict race — the order-of-magnitude gap is the point.) We kept 11/100 refusals as the price of rock-bottom drift — brains first.

Kept in BF16 (not quantized) — surgical precision

Everything heavy is 4-bit; only the fragile bits that carry reasoning, speed and vision stay full-precision.

Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

Use num_speculative_tokens: 4. Speculative decoding is lossless — n changes speed only, never output. All presets use n=4.

Multimodal — sees images & video

The full vision-language model, not a text-only trim. The entire vision tower (333 tensors) is kept in full BF16 — quantization never touched it — so image/video understanding is identical to the pristine base. Serve with --limit-mm-per-prompt '{"image":4,"video":1}'. Published benchmarks cover text/coding/tool-use; vision quality is inherited, not separately re-scored.

Quickstart — pick your GPU

vllm serve antonyMox/Qwen3.6-27B-abliterated-AutoRound-INT4-MTP \
  --quantization auto-round \
  --max-model-len 40960 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --trust-remote-code

Long context is cheap here: Qwen3.6 is a hybrid (Mamba/GDN + attention), so KV costs only ~37 MB/1k tokens at fp8. The model's native maximum is 256k (262,144) tokens — how much of it fits depends on how much VRAM you give vLLM:

Context vs mem-util (48 GB card, measured)

Our default preset is a deliberate 0.60 / 200k — it keeps ~19 GB free so you can load a second model on the same card. Want the full 256k? Just raise --gpu-memory-utilization to ~0.65–0.68.

⚠️ Pitfall — never set --max-model-len at exact KV capacity

[!WARNING] On vLLM 0.19 hybrid models: if max-model-len equals KV capacity (startup log shows Maximum concurrency ... : 1.00x), requests in the last ~3 % of the window pass validation then hang forever — no response, no error, nothing logged, GPU idle. The scheduler can't allocate the final KV block (draft tokens need headroom the validator ignores). Above the window → instant HTTP 400 (fine); below the danger zone → fine. Only the top sliver dead-locks.

[!TIP] Keep --max-model-len ≥ 2 attention blocks (2 × 1632) below capacity — our presets already do. Check the log: Maximum concurrency for N tokens: 1.13x must be ≥ ~1.10x, not 1.00x. Also: the log line GPU KV cache size: X tokens understates real capacity ~3× on hybrids (it divides by KV-cache groups) — trust only the Maximum concurrency line.

Long-context recall (measured)

Needle test on a 150k-token prompt (48 GB preset), exact-quote recall by depth:

7/8 exact. The miss (*) at 50% returned the neighbouring line — mild "lost-in-the-middle", not a hallucination. Prefix caching works: a repeat 150k query answered in 5.5 s vs 131 s cold.

🧬 The family

Clean quants of the same base — pick by hardware:

• wide · 26 GB · hard 78 🏆 — maximum reasoning, for 48 GB+ cards.
• abliterated · 19 GB · hard 70 (this one) — uncensored, fits 24 GB cards.
• 35B-A3B · 22 GB · hard 48 (coming soon) — fast MoE (~188 tok/s).

All ours: AutoRound INT4, MTP preserved, multimodal, published benchmarks. Measured against the community reference Lorbus narrow INT4 (hard 70) — see the comparison table above.

Recipe

1. Base — pristine Qwen/Qwen3.6-27B BF16.
2. Abliteration — Heretic, gentle: KL 0.0089, refusals 11/100, <think> intact.
3. Quant — AutoRound W4A16, group 128, 200 iters, dense. BF16-kept: mtp.*, visual.*, Mamba control.
4. MTP — 15 mtp.* tensors grafted from the pristine BF16.
5. Sampler — ships temp 1.0 / top_p 0.95 / top_k 20 (generation_config.json).

Why HuggingFace shows "~6.7B params"

Not a small model — it's how the widget reads a packed INT4 quant. The 800 quantized tensors pack 8× int4 into each int32, so HF counts 3.06 B int32 slots, not the ~24 B real int4 weights. With the BF16 parts, the true model is Qwen3.6-27B, 19 GB. Every AutoRound/GPTQ INT4 looks like this. Verified: language_model 4.55 B · other 1.27 B · vision 0.46 B · mtp 0.42 B.

⚠️ Usage & responsibility

[!CAUTION] This is an uncensored research artifact — read before use.

• Reduced refusals by design — can produce content a safety-aligned model would decline.
• No safety guarantees — outputs are not filtered; review before use, especially in production.
• You are responsible — for lawful, ethical use in your jurisdiction; all consequences rest with you.
• No liability — provided "as is" as a community artifact, no warranty of any kind.
• Not for harm — for research, evaluation, and lawful applications only.

License

Apache-2.0, same as the base. Quantized & abliterated from Qwen/Qwen3.6-27B. Community build; benchmark numbers from our own harness — JSON included, methodology open.