language:
- en license: apache-2.0 tags:
- qwen3.6
- gptq
- int4
- quantized
- dynamic-quantization
- mtp
- multi-token-prediction
- uncensored
- reasoning
- consumer-hardware pipeline_tag: text-generation base_model:
- DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP library_name: gptqmodel inference_library_version: "7.3.2" quantization_config: quant_method: gptq bits: 4 group_size: 128 desc_act: true
GPTQ INT4 quantization of the original DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP, optimized for dual-RTX-2080-Ti deployment.
- Method: GPTQ (via modelcloud/gptqmodel v7.3.2)
- Base precision: INT4 (4-bit weights, group_size=128, symmetric, desc_act=true)
- Dynamic per-layer override: Layers 68-79 projection layers quantized at INT8 instead of INT4 to preserve reasoning quality in the upper transformer layers
- LM head: Not quantized (preserved at BF16)
- Total size: ~19 GB (vs ~56 GB BF16 original)
This quantization was specifically optimized and tested on a dual-RTX-2080-Ti (22GB each) setup with NVLink interconnect, running a custom vLLM fork designed for consumer GPUs.
- Inference engine: weicj/vLLM-2080Ti-Definitive v0.1.14 (fork of vLLM with SM75-specific patches and INT4 GPTQ kernel optimizations)
- GPU config: 2× NVIDIA RTX 2080 Ti (22GB VRAM each, compute capability 7.5), connected via NVLink (2 links per GPU, 25.781 GB/s each)
- Tensor parallelism: TP=2
- Context length: 262K tokens
- MAX_NUM_SEQS: 2
- GPU memory utilization: 0.93
- MTP depth: 3
Measured on dual RTX 2080 Ti (NVLink) with the configuration above. All tests generate 1024 output tokens.
| Prompt Length (tokens) | TTFT (ms) | ITL Avg (ms) | ITL Std (ms) | Prefill Time (ms) | Prefill Speed (tok/s) | Output Time (ms) | Decode Speed (tok/s) |
|---|---|---|---|---|---|---|---|
| 4,096 | 3,265 | 39.99 | 0.50 | 3,260 | 1,256 | 15,358 | 66.68 |
| 8,192 | 6,730 | 40.62 | 0.79 | 6,725 | 1,218 | 11,903 | 86.03 |
| 16,384 | 14,034 | 41.46 | 0.64 | 14,029 | 1,168 | 17,537 | 58.39 |
| 32,768 | 29,943 | 42.84 | 1.10 | 29,937 | 1,095 | 16,194 | 63.23 |
| 65,536 | 66,835 | 45.29 | 1.25 | 66,829 | 981 | 16,531 | 61.94 |
| 131,072 | 160,941 | 50.18 | 2.40 | 160,936 | 814 | 19,620 | 52.19 |
- TTFT: Time To First Token
- ITL: Inter-Token Latency (average between consecutive output tokens)
- Prefill Speed: prompt tokens processed per second
- Decode Speed: output tokens generated per second
llm = LLM( model="cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4", quantize="gptq", tensor_parallel_size=2, max_model_len=262144, gpu_memory_utilization=0.93, max_num_seqs=2, )
model = GPTQModel.from_quantized( "cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4", device_map="auto", )
- The dynamic INT8 override on layers 68-79 projection layers is intentional and required for maintaining reasoning quality on this model family. This is a per-layer selective quantization strategy, not a uniform INT8 quantization.
- The forked vLLM (weicj/vLLM-2080Ti-Definitive) includes critical patches for SM75 architecture and INT4 GPTQ kernel optimizations that are not present in upstream vLLM.
- For benchmark results on the base model, refer to the original repository.