license: apache-2.0 base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags:

vllm
fp8
compressed-tensors
abliterated
qwen3_5

Huihui-Qwen3.8-27B-abliterated-FP8 — v2

⚠️ This is v2. It is quantized from the 2026-08-24 upstream re-release (739e3c5b), in which huihui-ai narrowed the ablation to layers 18–51. The earlier upstream build ablated layers 15–63; the narrower range retains more of the original model's performance. v1 of this repo — the quantization of the older, more heavily ablated weights — has been deleted and is no longer downloadable. If you pulled this repo before 2026-08-24, re-download it.

FP8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with llm-compressor (FP8_DYNAMIC).

huihui-ai publish only BF16 weights and a GGUF build for this model — no FP8 — so this repo fills that gap for vLLM users.


Version	v2
Base revision	`739e3c5b89849f6c238ce1e5b70008612ae42cdd` (2026-08-24)
Ablated layers upstream	18–51

Scheme


Weights	FP8 `e4m3`, per-channel (static)
Activations	FP8 `e4m3`, dynamic per-token
Format	`compressed-tensors` (`float-quantized`)
Calibration	none required (data-free pipeline)

256 dense Linear modules are quantized. Everything the upstream Qwen/Qwen3.8-27B-FP8 release leaves alone is left in BF16 here too:

linear_attn.* — the hybrid Mamba projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj), 48 layers
the whole vision tower (visual.blocks.*, visual.merger.*)
embed_tokens, lm_head, and all norms
the MTP drafter (mtp.*) — kept in BF16 rather than FP8, so speculative decoding still works

Note the upstream FP8 release uses per-tensor weight scales; this one uses per-channel, a finer-grained (and therefore more accurate) scheme at the same size.

Serving with vLLM

vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8-v2 \
  --max-model-len 262144 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'

Sampling defaults follow the upstream model card's thinking-mode recommendation (temp 1.0 / top_p 0.95 / top_k 20).

Caveats

This is an abliterated model: refusal behaviour has been removed upstream. Safety properties are not those of the original Qwen release.
Quantization was verified structurally and by generation, not by a benchmark suite; no perplexity or eval numbers are claimed.
With reasoning_effort: xhigh this model family will spend a very large output budget inside the reasoning block — measured 16k tokens / 26 min for one hard question on a GB10. It converges and returns a full answer, but if you cap max_tokens below what it needs you get finish_reason: length and an empty content. This is not specific to this quantization or to abliteration — stock Qwen/Qwen3.8-27B-FP8 behaves identically.

Huihui-Qwen3.8-27B-abliterated-FP8 — v2

⚠️ This is v2. It is quantized from the 2026-08-24 upstream re-release (739e3c5b), in which huihui-ai narrowed the ablation to layers 18–51. The earlier upstream build ablated layers 15–63; the narrower range retains more of the original model's performance. v1 of this repo — the quantization of the older, more heavily ablated weights — has been deleted and is no longer downloadable. If you pulled this repo before 2026-08-24, re-download it.

huihui-ai publish only BF16 weights and a GGUF build for this model — no FP8 — so this repo fills that gap for vLLM users.

Version

Base revision

739e3c5b89849f6c238ce1e5b70008612ae42cdd (2026-08-24)

Ablated layers upstream

18–51

Scheme

Weights

FP8 e4m3, per-channel (static)

Activations

FP8 e4m3, dynamic per-token

Format

compressed-tensors (float-quantized)

Calibration

none required (data-free pipeline)

256 dense Linear modules are quantized. Everything the upstream Qwen/Qwen3.8-27B-FP8 release leaves alone is left in BF16 here too:

linear_attn.* — the hybrid Mamba projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj), 48 layers

the whole vision tower (visual.blocks.*, visual.merger.*)

embed_tokens, lm_head, and all norms

the MTP drafter (mtp.*) — kept in BF16 rather than FP8, so speculative decoding still works

Note the upstream FP8 release uses per-tensor weight scales; this one uses per-channel, a finer-grained (and therefore more accurate) scheme at the same size.

Serving with vLLM

vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8-v2 \ --max-model-len 262144 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'

Sampling defaults follow the upstream model card's thinking-mode recommendation (temp 1.0 / top_p 0.95 / top_k 20).

Caveats

This is an abliterated model: refusal behaviour has been removed upstream. Safety properties are not those of the original Qwen release.

Quantization was verified structurally and by generation, not by a benchmark suite; no perplexity or eval numbers are claimed.

With reasoning_effort: xhigh this model family will spend a very large output budget inside the reasoning block — measured 16k tokens / 26 min for one hard question on a GB10. It converges and returns a full answer, but if you cap max_tokens below what it needs you get finish_reason: length and an empty content. This is not specific to this quantization or to abliteration — stock Qwen/Qwen3.8-27B-FP8 behaves identically.