license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text base_model: DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP language:

en
zh tags:
qwen
qwen3.5
awq
int4
w4a16
compressed-tensors
lmdeploy
heretic
uncensored
abliterated
davidau
finetune

Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-W4A16-AWQ

W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization of DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP.

Quantization method

Scheme: W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP heads and vision tower, which stay BF16).
Tooling: llmcompressor one-shot offline quantization with CPU offloading (compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.
AWQ activation smoothing: AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
Unquantized (kept BF16): embeddings, lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and MTP heads.

Benchmarks

See the original model's README for full benchmark tables.

Usage

Tested with LMDeploy turbomind:

from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    "TheUnderscore/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-W4A16-AWQ",
    backend_config=TurbomindEngineConfig(
        tp=1,
        model_format="compressed-tensors",
        language_model_only=True,
    ),
)
print(pipe("Hello, who are you?").text)

Files

quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).
model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.

Quantization method

Scheme: W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP heads and vision tower, which stay BF16).

Tooling: llmcompressor one-shot offline quantization with CPU offloading (compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.

AWQ activation smoothing: AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.

Unquantized (kept BF16): embeddings, lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and MTP heads.

Usage

Tested with LMDeploy turbomind:

from lmdeploy import pipeline, TurbomindEngineConfig pipe = pipeline( "TheUnderscore/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-W4A16-AWQ", backend_config=TurbomindEngineConfig( tp=1, model_format="compressed-tensors", language_model_only=True, ), ) print(pipe("Hello, who are you?").text)

Files

quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).

model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.