license: apache-2.0 base_model: Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 base_model_relation: quantized library_name: vllm tags:

• qwen3_5
• qwen3.8
• compressed-tensors
• w8a8
• int8
• abliterated
• gfx90a
• mi210
• rocm

Qwen3.8-27B abliterated — W8A8 + INT8 GDN projections + INT8 lm_head

INT8 W8A8 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16, itself an abliterated derivative of Qwen/Qwen3.8-27B. Abliterated models have had refusal behaviour deliberately removed; that property is inherited here unchanged.

Produced and measured on a single AMD MI210 (gfx90a / CDNA2).

What is different from a normal W8A8 of this model

Two things are quantized beyond a normal W8A8 build of this model:

1. The three large gated-delta-net (GDN) projections — in_proj_qkv, in_proj_z, out_proj — across all 48 linear_attention layers. That is a deliberate departure from the usual recipe for this architecture, which protects the whole linear-attention path as quantization-sensitive. Read the evaluation section; a plain W8A8 build is the conservative choice.

2. lm_head (248,320 x 5,120). This one is close to free — see below.

Measured on one MI210, decode:

The GDN projections were 5.54 B parameters sitting in BF16 — 11.07 GB read every decode step against 19.3 GB for everything already quantized, i.e. ~30% of kernel time in one unquantized tensor family. lm_head was another 2.54 GB in a single call.

What is quantized

Everything the parent W8A8 recipe quantizes (MLPs on all 64 layers, self_attn projections on the 16 full_attention layers), plus:

Still BF16, deliberately:

Scheme

Symmetric per-output-channel INT8 weights, matching the parent checkpoint exactly:

scale = max|W| / 127        # computed in fp32, then STORED AS BF16
q     = round(W / scale.to(bf16)).clamp(-128, 127)

Casting the scale to BF16 before dividing is load-bearing — it is what reproduces the parent's statistics (row maxima usually 127 but often 120–126, and -128 present). Dividing by an fp32 scale and casting afterwards yields subtly different weights.

Activations are unchanged: 8-bit per-token dynamic, computed at inference.

Getting vLLM to honour a quantized lm_head

Worth writing down, because it is not obvious and cost several attempts. vLLM's compressed-tensors path does support a quantized lm_head — ParallelLMHead gets a CompressedTensorsLinearMethod whenever get_scheme matches, and the embedding() requirement does not apply to it. The obstacle is purely target matching:

• targets: ["Linear"] never matches, because the module's class is ParallelLMHead.
• Adding the literal "lm_head" is still not enough on this model: it is a ForConditionalGeneration wrapper, so the prefix is nested and only an exact layer-name match resolves.

So this config uses targets: ["Linear", "re:.*lm_head$"] and drops lm_head from ignore. No vLLM code change is required. (If you are debugging this yourself: testing find_matched_target with a torch.nn.Linear is misleading — nested names appear to match via the Linear class-name target. Pass a real ParallelLMHead.)

No calibration data is used or needed — this step is a pure weight transform. quantize_gdn_int8.py in this repo reproduces it from the parent checkpoint exactly. Max relative weight error introduced: 0.39%.

Evaluation — and its limits

Measured against the same checkpoint with GDN projections left in BF16, same harness, same hardware.

The GDN step costs ~2.3% perplexity; the lm_head step costs ~0.01%.

GDN projections, token-weighted over 4,437 tokens: +2.3%

lm_head on top of that, token-weighted over 4,463 tokens: +0.012% (two of three corpora improved — this is noise)

That asymmetry is expected: lm_head gets a separate scale for each of 248,320 vocab rows, and it sits outside the recurrent state so its error cannot compound along the sequence.

Long-context recall: 9/9, identical to the unquantized-GDN control

The needle test was run against both checkpoints on purpose: this hardware declines the fast paged-attention kernel, so a long-context failure could have been the attention path rather than the weights. Running both makes the result attributable.

The needle test above covers the GDN change. It was not re-run for lm_head, because the mechanism that motivated it — error compounding through a recurrent state — does not apply to a final projection.

What was NOT evaluated. No standard benchmark suite (MMLU, GSM8K, HumanEval, etc). No multi-turn or long-generation testing. No coding or math evaluation. The perplexity probe is three passages, two of them Python source from one project. The needle test is a single fact. Given that this deliberately quantizes a path the usual recipe protects, and that GDN is recurrent so errors can compound in ways short probes do not reveal, treat these numbers as a smoke test, not a quality claim. If you evaluate it properly, please open a discussion with what you find.

Running it

vLLM on ROCm, single MI210:

vllm serve <this-model> \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.90 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --speculative-config '{"method": "dflash", "model": <dflash-draft>, "num_speculative_tokens": 12}'

with VLLM_ROCM_USE_AITER=1. Notes for gfx90a specifically:

• VLLM_ROCM_USE_AITER=1 needs an image whose AITER carries gfx90a code objects, or the INT8 GEMM silently falls back to a generic Triton kernel at roughly a third of the rate. Confirm with grep 'Selected.*ScaledMMLinearKernel' in the server log — you want AiterInt8ScaledMMLinearKernel, not TritonInt8ScaledMMLinearKernel.
• On a host with GPUs of more than one architecture, pass only the gfx90a render nodes (--device=/dev/dri/renderD<N>, not all of /dev/dri). vLLM resolves the architecture once at import from amdsmi's physical device 0 and ignores HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES, so a foreign card can switch every gfx9 path off silently. That alone was worth 2.4x here.
• cudagraph_mode: FULL_DECODE_ONLY matters — vLLM otherwise drops to PIECEWISE under speculative decoding.
• Speculation depth is worth tuning and is not a fixed property of the model. On this card, measured: MTP N=1 57.2 tok/s, MTP N=2 68.2, dflash N=8 78.0, dflash N=12 80.6, dflash N=16 74.9. Re-measure after any change that moves step time — the optimum shifts.
• Keep the dflash draft model in BF16. Quantizing it to int8 makes each step 7.7% cheaper but costs 13.7% of accepted tokens, for a net -6.5%.

Tooling for gfx90a: davetha/mi210-vllm.

Provenance

Qwen/Qwen3.8-27B → Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (abliteration) → W8A8 INT8 (llm-compressor, weights RTN + dynamic per-token activations) → this (GDN projections and lm_head additionally INT8 — quantize_gdn_int8.py, quantize_lmhead_int8.py).