license: other license_name: polyform-small-business-1.0.0 license_link: LICENSE base_model: IstroSec/ThinkingCap-Qwen3.8-27B-abliterated base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text language:

• en
• multilingual tags:
• qwen3_5
• qwen3_8
• thinkingcap
• abliterated
• uncensored
• fp8
• fp8_block
• compressed-tensors
• llmcompressor
• vllm
• mtp
• vision

ThinkingCap-Qwen3.8-27B-abliterated-FP8-BLOCK

FP8 (W8A8) quantization with block-scaled weights and dynamic activations of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B.

31.3 GB (from 55.6 GB bf16). Near-lossless: KL 0.0046 against the bf16 model, lower than the per-channel FP8-DYNAMIC build of the same size (0.0068), and as fast on a DGX Spark. vLLM runs it as W8A8 on Blackwell GPUs (tested on GB10 only) and on Hopper, where vLLM 0.30 by default re-rounds its block scales (see Hardware). On Ada GPUs (RTX 40xx, L40S, L4) vLLM 0.30 has no block-scaled W8A8 kernel and falls back to weight-only FP8: there, use FP8-DYNAMIC (see Hardware).

What is quantized

Produced with llm-compressor 0.14.0, scheme FP8_BLOCK: FP8 (E4M3) weights with one scale per 128×128 block, FP8 input activations quantized at run time per token in groups of 128. Data-free: no calibration set. Round-to-nearest per block with no data is deterministic by construction, so the same command with the same library versions should give the same build; this was not checked by a rebuild.

Quantized: every Linear in the text decoder: the 64 MLPs, the 16 full-attention layers' q/k/v/o_proj, and the three large Gated DeltaNet projections (in_proj_qkv, in_proj_z, out_proj) in the other 48 layers. Each of them has both dimensions divisible by 128 (checked before quantizing), so no block is padded.

Kept in bf16:

Each block scale belongs to one 128×128 tile of one layer, so the scales stay correct when vLLM runs in_proj_qkv and in_proj_z as one fused GEMM: this build needs no special scale handling. (The NVFP4 build does; see its card.)

Evaluation

Heretic --evaluate-model (its default evaluation) against the bf16 abliterated checkpoint, 100 harmful + 100 harmless prompts:

KL under 0.01 is imperceptible; under 0.05 is good. Block scales cut the per-channel build's KL by about a third at the same size, and match the earlier DeltaNet-bf16 build with 5.5 GB less. Heretic decodes greedily (no sampling noise); a re-run with a different batch size can still move a count by about one prompt, so differences of one or two are not significant (this build 8/100, FP8-DYNAMIC 7/100, the bf16 source 6/100).

GPTQ-calibrated variants of both FP8 layouts were measured too (same size, not published): they lowered the chat-text KLD below but not Heretic's KL (0.0069 per-channel, 0.0070 block), so this data-free build is the one published.

Chat and agent text (a second measurement, not Heretic). Exact KL divergence from the bf16 model on held-out chat, math, code, tool-use and coding-agent transcripts (context 2048, 60 chunks), scored only at the 16,356 token positions of the model's own responses:

This text comes from the same seven datasets as the calibration text of the NVFP4A16 and W4A16 builds (other questions and repositories; the same tool definitions and agent system prompts), so it suits calibrated builds; both builds compared here are data-free, and Heretic's prompts, independent of it, rank them the same way.

Provenance: original ThinkingCap refuses 97/100 → bf16 abliteration 6/100 at KL 0.065 vs. the original → this quantization adds a further KL of 0.0046 relative to the abliterated model.

Usage

vLLM (the block-scaled FP8 kernel is chosen automatically)

Tested 2026-10-07 with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) on a DGX Spark (GB10; vLLM chose CutlassFp8BlockScaledMMKernel, and CutlassFP8ScaledMMLinearKernel for FP8-DYNAMIC), text only (--language-model-only), --max-model-len 32768, MTP on: a test prompt at temperature 0 was answered correctly (on both builds), and 8 varied prompts at temperature 0 ran at 16.88 tokens/s, the same speed as FP8-DYNAMIC on the same prompts (their answers were not inspected). A forced 512-token request without MTP, run twice, gave 7.96-7.97 tokens/s (FP8-DYNAMIC 7.91-7.93). Image input was not tested.

vllm serve IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-FP8-BLOCK \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-model-len 65536 --gpu-memory-utilization 0.85

Sampling (Qwen3.8 recommendations, which ThinkingCap uses unchanged): thinking mode temperature 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking mode temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5. Thinking budget via chat_template_kwargs: {"reasoning_effort": "xhigh"}, with xhigh (default, recommended), medium or low.

Notes:

• Image input: for text-only use keep --language-model-only (it also skips the vision tower's memory). For images, drop it and keep MTP. If startup then fails in profile_run with 'NoneType' object has no attribute 'size', vLLM reused a compiled graph cached by an earlier run with different multimodal limits (vllm#58203, a duplicate of vllm#50891): start with a fresh VLLM_CACHE_ROOT or set VLLM_USE_AOT_COMPILE=0. Image input was not tested with this build.
• KV cache: only the 16 full-attention layers have one, so the FP8 KV cache (--kv-cache-dtype fp8) buys less than on a dense model. Leave it at auto unless you're memory-bound; if you do enable it, fp8_e5m2 needs no scales.
• MTP: the head is bf16 and listed in quantization_config.ignore. Measured 69.6% draft acceptance (3,471 of 4,989 drafted tokens, mean acceptance length 3.09) with vLLM 0.30.0 on GB10, in the test above (3 speculative tokens, temperature 0, 8 prompts); FP8-DYNAMIC 69.5% on the same prompts. Figures from sampling-based runs are not directly comparable.

Transformers loads it too (dequantized to bf16 on load, so it needs the full ~56 GB):

from transformers import AutoModelForImageTextToText
m = AutoModelForImageTextToText.from_pretrained("IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-FP8-BLOCK", device_map="cuda")

Not for llama.cpp: this is compressed-tensors format. GGUF builds are made separately from the bf16 source.

Hardware

How vLLM 0.30 runs this build and FP8-DYNAMIC, per vLLM 0.30's source; tested only on GB10:

Turing and older are not covered here (Turing has no bf16; vLLM refuses FP8 on Volta). Tensor parallelism: the block shapes pass vLLM's check at TP 1, 2, 4 and 8. Other vLLM versions and other engines were not checked; the per-channel FP8-DYNAMIC build is the more widely supported format.

Reproduce

Build command, for the record: the build scripts are in a private repository, not published.

python3 23_quant_nvfp4_gdn.py --fp8-block
python3 02_regraft_mtp.py <bf16-source-dir> <that-output-dir>

The same in plain llm-compressor 0.14:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
    targets="Linear", scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn\\.in_proj_(a|b)$"],
)
oneshot(model=model, recipe=recipe)
# llm-compressor 0.14 keeps mtp.* in bf16 when it saves; 02_regraft_mtp.py then only adds the MTP Linears
# to quantization_config.ignore (older versions drop mtp.*: copy it from the bf16 checkpoint first)

Limitations

Everything from the bf16 card applies: no safety filter, residual refusals around 6/100 (this build 8/100 in the test above); thinking and non-thinking modes were not evaluated separately. You are the safety layer. Speed was measured only on GB10; on GPUs without a block-scaled FP8 kernel (Ada, Ampere) vLLM runs the weights only in FP8 (see Hardware).

License

PolyForm Small Business License 1.0.0 + BottleCap personal-use grant, inherited from ThinkingCap (see LICENSE). Upstream Qwen materials and the abliteration adapter are Apache-2.0 (see NOTICE). Commercial use beyond the PolyForm terms: contact BottleCap AI.

Credits

bottlecapai (ThinkingCap, and the DeltaNet quantization recipe) · MuXodious (abliteration adapter) · p-e-w/heretic · vllm-project/llm-compressor · Qwen team