license: other base_model: DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP tags:

qwen3_5
autoround
w4a16
gptq
compressed-tensors
vllm
mtp
vision

Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP — W4A16 AutoRound

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP using AutoRound via llm-compressor 0.12 + auto-round 0.13.

Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).
Preserved in BF16 (untouched, bit-identical to the base model):
- Vision tower (model.visual.*, 333 tensors)
- MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
- lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.
Size: ~19.5 GB (base BF16: ~52 GB).

Usage (vLLM ≥ 0.26)

Production example (4× RTX 3090, tool calling + reasoning parser + MTP speculative decoding). Replace the local path with the repo id DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound to load directly from the Hub:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export OMP_NUM_THREADS=4
export CUDA_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_ALLOC_CONF=expandable_segments:True
export NCCL_P2P_DISABLE=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

exec vllm serve \
    /root/models/DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \
    --served-model-name "qwen3.6-27b" \
    --tensor-parallel-size 4 \
    --max-model-len 163840 \
    --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --enable-prefix-caching \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.88 \
    --disable-custom-all-reduce \
    --max-num-seqs 16 \
    --host 0.0.0.0 \
    --port 3434 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Vision inputs work normally (image/video); the vision tower runs in BF16.
The MTP draft head loads from the same checkpoint for speculative decoding.
Validated on vLLM 0.26 (text, MTP speculative decoding, vision).

Notes

Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
chat_template-instruct.jinja is the author's alternate template shipped with the base model.
Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.

Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP — W4A16 AutoRound

Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).

Preserved in BF16 (untouched, bit-identical to the base model):

Vision tower (model.visual.*, 333 tensors)
MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)

Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.

Size: ~19.5 GB (base BF16: ~52 GB).

Usage (vLLM ≥ 0.26)

export CUDA_DEVICE_ORDER=PCI_BUS_ID export OMP_NUM_THREADS=4 export CUDA_VISIBLE_DEVICES=0,1,2,3 export PYTORCH_ALLOC_CONF=expandable_segments:True export NCCL_P2P_DISABLE=1 export FLASHINFER_DISABLE_VERSION_CHECK=1 exec vllm serve \ /root/models/DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \ --served-model-name "qwen3.6-27b" \ --tensor-parallel-size 4 \ --max-model-len 163840 \ --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --trust-remote-code \ --enable-prefix-caching \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.88 \ --disable-custom-all-reduce \ --max-num-seqs 16 \ --host 0.0.0.0 \ --port 3434 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Vision inputs work normally (image/video); the vision tower runs in BF16.

The MTP draft head loads from the same checkpoint for speculative decoding.

Validated on vLLM 0.26 (text, MTP speculative decoding, vision).

Notes

Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).

chat_template-instruct.jinja is the author's alternate template shipped with the base model.

Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.