license: other license_name: polyform-small-business-1.0.0 license_link: LICENSE base_model: IstroSec/ThinkingCap-Qwen3.8-27B-abliterated base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text tags:

qwen3_5
autoround
w4a16
int4
compressed-tensors
vllm
marlin
mtp
vision
abliterated
uncensored

ThinkingCap-Qwen3.8-27B-abliterated — W4A16 AutoRound

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of IstroSec/ThinkingCap-Qwen3.8-27B-abliterated using AutoRound via llm-compressor 0.14 + auto-round 0.15.

Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).
Preserved in BF16 (untouched, bit-identical to the base model):
- Vision tower (model.visual.*, 333 tensors)
- MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
- lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.
Size: ~19.5 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens. Deliberately not the calibration set, which would flatter the result. Same texts, same tokenization (identical token count for both), same method (raw text, no chat template), both models served by vLLM:

Model	Perplexity
base BF16	8.1133	—
W4A16 AutoRound	8.8426	+8.99%

This is above the ~5% degradation usually accepted for W4A16 (int4, group_size 128), so treat the LM fidelity as degraded more than a typical W4A16 quant of this size. The degradation is not visible in short side-by-side chat samples, which read as equivalent to the BF16 original — perplexity averages small probability shifts over every token and is the more sensitive instrument here. If you need tighter fidelity, re-quantizing with more AutoRound iterations (iters=400) and more calibration samples (nsamples=256) is the first knob to turn.

Note that perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/ThinkingCap-Qwen3.8-27B-abliterated-W4A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Vision inputs work normally (image/video); the vision tower runs in BF16.
The MTP draft head loads from the same checkpoint for speculative decoding.
Validated on vLLM 0.28 (text, MTP speculative decoding, vision), Marlin kernels (CompressedTensorsWNA16).
This is a thinking model: with a small max_tokens the whole budget can be consumed by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat endpoint over raw completion.

Notes

Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
Abliteration is inherited from the base model — quantizing does not remove it, and this derivative carries the same behavioural characteristics and the same usage warnings.
Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.

License

ThinkingCap-Qwen3.8-27B-abliterated — W4A16 AutoRound

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of IstroSec/ThinkingCap-Qwen3.8-27B-abliterated using AutoRound via llm-compressor 0.14 + auto-round 0.15.

Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).

Preserved in BF16 (untouched, bit-identical to the base model):

Vision tower (model.visual.*, 333 tensors)
MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)

Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.

Size: ~19.5 GB (base BF16: ~52 GB).

Quality

Model

Perplexity

base BF16

8.1133

—

W4A16 AutoRound

8.8426

+8.99%

Note that perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/ThinkingCap-Qwen3.8-27B-abliterated-W4A16-AutoRound \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --reasoning-parser qwen3 \ --enable-prefix-caching \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Vision inputs work normally (image/video); the vision tower runs in BF16.

The MTP draft head loads from the same checkpoint for speculative decoding.

Validated on vLLM 0.28 (text, MTP speculative decoding, vision), Marlin kernels (CompressedTensorsWNA16).

This is a thinking model: with a small max_tokens the whole budget can be consumed by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat endpoint over raw completion.

Notes

Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).

Abliteration is inherited from the base model — quantizing does not remove it, and this derivative carries the same behavioural characteristics and the same usage warnings.

Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.