license: apache-2.0 base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers language:

• en tags:
• nvfp4
• fp4
• w4a4
• gptq
• quantized
• compressed-tensors
• llm-compressor
• vllm
• vision-language
• thinking
• mtp
• speculative-decoding
• math-calibration
• gsm8k
• kv-cache-calibrated

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882 — NVFP4 (W4A4, KV-calibrated)

Quantization-only derivative of DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU (the 735-882 "Stage 1b" Heretic checkpoint). No training, no additional abliteration. All credit for the model itself belongs upstream.

A note on the additional math calibration data

A first attempt calibrated only on a (reasoning + chat + code) blend, and its grade-school math collapsed sharply. Measured on identical hardware (RTX PRO 6000 Blackwell, lm-eval 0.4.12 vLLM backend, greedy, 0-shot raw prompts):

Adding math to the calibration blend lifts gsm8k by roughly +0.09 over the no-math build, with no other task regressing (see Quality Verification). The residual gap to the BF16 source (0.600 vs 0.899) is, we think, the expected W4A4 cost on a reasoning-heavy task. Doubling the math calibration in the shipped build was also tried; it did not measure better than the shipped build, so the calibration above is the one shipped.

Quality Verification

All values below were benchmarked under identical conditions: lm-eval 0.4.12 (vLLM backend) in the vllm/vllm-openai:qwen38-x86_64-cu130 image (torch 2.13.0+cu130), single RTX PRO 6000 Blackwell (96 GB), --gpu-memory-utilization 0.85, max_model_len 4096, greedy decoding. Prompts are 0-shot raw task prompts — no chat template, no thinking (the lm-eval runs recorded chat_template: null; equivalent to the "Instruct mode" the upstream card benches in).

* gsm8k is the largest single W4A4 cost on this base; the math-calibration section above covers it.

Reading: on every other task the W4A4 build is within ~3 pt of the BF16 source. Absolute values differ from third-party harness numbers (e.g. the upstream card's llama.cpp benches); compare against the BF16 column, which was measured under the same conditions.

Reproducibility

Built with llm-compressor oneshot + GPTQModifier (recipe.yaml included: block size 128, dampening 0.01, actorder static, MSE observer, group size 16) with a domain-matched calibration blend (long reasoning + general chat + code + GSM8K train math), on a GB10 (sm_121, CUDA 13.0, torch 2.11.0+cu130, transformers 5.14.1, llm-compressor 0.13.0), with the KV-cache calibration pass enabled (kv_cache_scheme: fp8, symmetric, per-tensor, memoryless_minmax) riding on the same one-shot pass.

What the KV calibration adds. 32 scalar FP32 tensors — k_scale/v_scale (one per full-attention layer's k_proj/v_proj; this model has 16 full-attention layers, the rest being Gated DeltaNet linear attention, which keeps a recurrent state and has no paged KV cache) — plus quantization_config.kv_cache_scheme in config.json. vLLM's compressed-tensors path reads these into the attention layers and multiplies the loaded K/V entries by the per-layer scale when quantizing them into the fp8 paged cache.

Post-quantization structural fixes (MTP restoration, index.json rebuild, ignore guarantee for mtp.* in config.json, tower split into model-towers.safetensors) are idempotent and re-derivable from the BF16 source. Structural check: PASS.

Serve with vLLM

vllm serve <this-repo> \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --kv-cache-dtype fp8 --enable-prefix-caching \
  --max-num-batched-tokens 2048 --max-num-seqs 2 \
  --gpu-memory-utilization 0.44 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

(Flag values are a tested GB10 / 128 GB unified-memory configuration — 256K context works at this setting. On a dedicated GB10 you can raise --gpu-memory-utilization toward ~0.8 for a larger KV pool; avoid ~0.95, which starves the OS on a unified pool.)

Engineering notes

• This quantization run (calibration, QC, card, upload) was performed with an LLM agent operating the operator's own toolchain.

Lineage

Qwen/Qwen3.8-27B → DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU → this. Apache 2.0, unbroken from the base model.