license: apache-2.0 base_model: JonathanColetti/Qwen3.8-27B-Uncensored base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags:

• qwen3_5
• qwen3.8
• fp8
• w8a8
• mtp
• speculative-decoding
• vision
• multimodal
• abliterated
• uncensored
• vllm language:
• en
• zh
• th

Qwen3.8-27B-Uncensored-FP8-MTP

FP8 (block 128×128, E4M3) quantization of an abliterated Qwen3.8-27B, with the vision tower and the working MTP speculative-decoding head both preserved.

What "abliterated" means here. The base model has had its refusal direction removed — it will not decline requests the way the official Qwen3.8-27B does. That modification comes from the base model, not from this repository; this repo contributes the FP8 quantization only. The base is Apache-2.0 and publicly available. Use accordingly.

Why this build

Surveying 148 abliterated/uncensored Qwen3.8-27B repositories in August 2026 turned up exactly one FP8 build, and it was access-gated. Everything else was GGUF, NVFP4 (Blackwell-only), AWQ, or unquantized BF16. On Hopper hardware — where FP8 has native tensor-core support — that left no ungated option.

Only a handful of builds in any format preserve the MTP head, and getting it to actually work under vLLM required the checkpoint to be in the native fp8 format rather than compressed-tensors (see below).

Measured against the official build

Same hardware (H100 47GB vGPU), same vLLM 0.27.1, same flags.

The one coding task both models fail is the same one, for the same reason: the model spends its whole token budget reasoning and never emits an answer.

Hardware requirements

FP8 needs tensor-core support to be worth anything. On architectures without it, vLLM either refuses the checkpoint or falls back to a dequantize-then-multiply path that is slower than a 4-bit AWQ build.

On Ampere (A40, A100, A6000). These have the VRAM — an A40 has 48 GB, the same as the H100 47C this was tested on — but no FP8 tensor cores. vLLM can fall back to Fp8MarlinLinearMethod, which keeps weights in FP8 and dequantizes inside the kernel, so memory still works out. Whether that path accepts block-wise 128×128 scales (as opposed to per-tensor or per-channel) is untested here — we had no Ampere card to try it on. If you are on Ampere, an INT8 W8A16 or AWQ build of the same base is the safer choice.

VRAM

Weights occupy ~30 GB. The rest goes to KV cache, which decides how much context you can actually serve.

At 47 GB the whole 262,144-token context fits for a single request with room for roughly one more — Maximum concurrency for 262,144 tokens per request: 1.3x. Shorter contexts scale proportionally: ~20 concurrent requests at 16k each.

Without --kv-cache-dtype fp8, KV for 262k context needs 16.17 GiB instead of ~8.5 GiB and a 47 GB card will refuse to start.

Serving

vllm serve pramoths/Qwen3.8-27B-Uncensored-FP8-MTP \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.94 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 16 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

--kv-cache-dtype fp8 is required for 262k context on a 47GB card — fp16 KV needs 16.17 GiB and will refuse to start. --max-num-seqs must stay below the available Mamba state block count; the default of 256 fails to boot.

Verify MTP is actually working — it can fail silently:

curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total

A Mean acceptance length of 1.00 means every draft is being rejected. You are paying the drafting cost for nothing.

What was and was not quantized

Follows Qwen/Qwen3.8-27B-FP8 exactly — the distinction is matmul versus control parameter, not which component of the model:

• FP8: q/k/v/o_proj and gate/up/down_proj, in the language layers and in the MTP head.
• BF16: LayerNorms, gates, A_log, dt_bias, conv1d, and the gated-DeltaNet in_proj_a / in_proj_b — these govern recurrent state, where quantization error accumulates rather than cancelling.
• BF16: the entire vision tower, lm_head, embed_tokens, mtp.fc.

Reproducing

Scripts, benchmarks, and a writeup of the five silent failure modes encountered along the way: https://github.com/pramoth/qwen38-fp8-forge

Quantization needs no calibration data and no GPU — roughly two minutes of CPU.

Attribution

• Qwen/Qwen3.8-27B — original model, Apache-2.0
• JonathanColetti/Qwen3.8-27B-Uncensored — abliterated base
• Qwen/Qwen3.8-27B-FP8 — the quantization recipe this build copies