license: apache-2.0 base_model: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated tags:

• qwen3_5
• coder
• gptq
• int4
• w4a16
• vllm
• xpu
• intel-arc
• mtp
• gdn
• text-generation
• conversational
• en
• zh library_name: vllm

Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16

GPTQ Int4 W4A16 (symmetric, group-128, desc_act=False, lm_head=False) quantization of huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated (dense Qwen3.8-27B coder, 64 layers of hybrid linear/full attention), built for the vLLM XPU backend on Intel Arc — validated on an Arc Pro B60.

The recipe mirrors SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 field-for-field, so this is a drop-in swap of that checkpoint onto the Coder390 base rather than a new quantizer configuration. As of 2026-10-18 it is the only GPTQ quantization of Coder390 on the Hub — the family has GGUF, MLX, EXL3, NVFP4, compressed-tensors W4A16 and MXFP4 builds, but no GPTQ until this one.

Contents

• Quantized (400 modules → qweight int32 + scales/qzeros): all 64 dense mlp.{gate,up,down}_proj, all 16 full-attention self_attn.{q,k,v,o}_proj, and all 48 linear_attn.{in_proj_qkv,in_proj_z,out_proj}. in_proj_a/in_proj_b are skipped, as in the reference recipe.
• BF16: embeddings, lm_head, every norm, A_log/conv1d/dt_bias, the full 27-block visual tower, and the MTP draft block (mtp.*, 15 tensors) — held out of quantization with dynamic={"-:.*mtp.*": {}} and stored unquantized inside shard 5, so speculative decoding needs no separate draft file.
• text_config.mtp_num_hidden_layers is 1 (as shipped upstream).

2,399 tensors / 5 shards / 18.24 GiB · SHA256SUMS included.

Structurally identical to the reference checkpoint (same 2,399 tensors, same per-dtype element counts: F16 3,219,819,760 · I32 3,067,527,168 · BF16 424,699,392), which is what makes the two directly comparable below.

Recipe

The calibration is the one deliberate deviation from the classic GPTQ budget (128 × 2048 = 262,144): on the 45 GB-RAM host used here the larger pile terminated the job every time (GPU engine reset, or a host OOM-kill), and 64 × 1024 completed cleanly. The reference checkpoint does not record how it was calibrated, so token-set parity was never possible anyway. If you re-quantize, note that a different calibration changes the Hessians and therefore the weights — it is a different artifact, not a re-run.

Measured serving

Measured on Intel Arc Pro B60 (24 GB, Battlemage), i5-12400F host, GPU power capped at 180 W, vLLM 0.30.0+xpu, --dtype float16 --kv-cache-dtype fp8 --language-model-only, MTP speculative decoding k=4, prompt 512 / generation 128 tokens:

MTP is worth having: with the draft disabled the same build measured 7.987 tok/s against 16.620 tok/s at k=4 — 2.08× — on an MXFP4 leg of the same experiment. Note the acceptance ratio is an upper bound on the gain, never the gain itself: the speculative step costs 1.361× more per step here (125.2 → 170.5 ms).

Long context (single stream, 12,288-token ceiling): prefill stays linear at ~1,372 tok/s through 4,096 tokens (marginal 1,268 tok/s against the fitted line, R² = 0.9986), while decode degrades gently — 40.200 tok/s @512 → 37.315 tok/s @12,000 (−7.2 %), TTFT 0.463 → 9.539 s. With --kv-cache-dtype fp8 the KV cache is 1.82 GiB, i.e. ~17,630 tokens of capacity at these settings.

Measured quality

15/16 (93.8 %) on a 16-prompt task suite spanning math, code (executed), instruction constraints and fact/trap questions. Treat that as ±1 item: one item (format_no_letter_e) converges on some server instances and loops to the token cap on others at temperature=0, deterministically within an instance but not across them, so a single-item delta between builds is not signal.

Two harness traps this checkpoint sets, both worth knowing before you benchmark it:

• Always pass --reasoning-parser qwen3. The engine emits </think> (added token 248069) but the detokenizer suppresses it from content and leaves reasoning null, so without the parser the whole reasoning trace arrives concatenated in front of the answer with no delimiter. vLLM's own 16-prompt quality harness scores 8/16 that way versus 15/16 with the flag — a plausible-looking 50 % scored on the model's private musings.
• Quality items can flip between server instances even at greedy decoding (see above). Re-run a lone failure before believing it.

Serving

vllm serve Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16 \
  --served-model-name coder390-gptq \
  --quantization gptq --dtype float16 --kv-cache-dtype fp8 \
  --max-model-len 12288 --max-num-seqs 1 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.93 --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

Notes:

• --language-model-only skips the visual tower, which is what the benchmarks above used. The tower is intact and BF16 (333 tensors) if you need vision.
• MTP resolves automatically: the draft is the BF16 mtp.* block in shard 5, activated by B70_MTP_BF16_DRAFT=1 on this stack and by the compressed-tensors ignore list.
• On a cold container the first engine init can fail with ValueError: ... KV cache is needed, which is larger than the available KV cache memory (0.18 GiB) because torch.compile's transient pool eats the KV budget; the retry succeeds against a warm compile cache. Run the server under --restart unless-stopped (or restart the same container once) and expect ~220 s to healthy rather than treating the first failure as fatal.
• Decode throughput on this host is bimodal (~40.2 vs ~48.5 tok/s, ~20 %), with the mechanism unidentified and step time — not acceptance — responsible. Compare rates only within a single session, and report a median and a spread rather than one number.

Relation to the MXFP4 sibling

alpha2beta/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-MXFP4 is the same base model quantized to MXFP4 (compressed-tensors mxfp4-pack-quantized, W4A4). Same tensors held in BF16, same folder layout, same 5-shard/16-shard family naming. On the same host MXFP4 is markedly slower than GPTQ at batch 1 — W4A4 dynamic activation quantization and a non-fused FP4 GEMM, multiplied by eager execution — which is why this GPTQ build exists as the throughput option.

Provenance

Built with gptqmodel 7.6.0 on vLLM 0.30.0+xpu / torch 2.13.0+xpu / transformers 5.16.1, driving the Arc Pro B60 as the quantization accelerator, with offload_to_disk=True (meta-device shell + module-by- module streaming) because the 52 GB BF16 source does not fit in the 45 GB host RAM.

Base weights: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated (BF16, Apache-2.0). Reference schema: SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16.

Full build record, attempt log, benchmarks and the operating guide: alpha2beta/vllm.xpu @ b60 — see REPRODUCIBILITY.md (§6, §7 deviation 8, §8 known issues 1/11/12/13) and GPTQ_quantization.md.