base_model: windowsxp811203/Qwen3.8-27B-Abliterated base_model_relation: quantized library_name: gguf pipeline_tag: text-generation license: apache-2.0 language:

• en
• zh tags:
• qwen
• qwen3.5
• qwen3.8
• gguf
• llama.cpp
• abliterated
• uncensored
• mtp
• speculative-decoding
• vulkan
• amd

Qwen3.8-27B-Abliterated — IQ4-MIX-MTP GGUF

Custom mixed-precision GGUF quantization of windowsxp811203/Qwen3.8-27B-Abliterated, derived from Qwen/Qwen3.8-27B.

This quant was built specifically for high-throughput local inference on a 16 GB GPU while preserving the model's built-in MTP (Multi-Token Prediction) head for speculative decoding in llama.cpp.

The main GGUF contains the MTP tensors. No separate draft model is required when using:

--spec-type draft-mtp

File

Quantizer output:

model size = 52115.19 MiB (16.00 BPW)
quant size = 13573.45 MiB (4.17 BPW)

Quantization layout

This is not a uniform IQ4_XS quant. Different tensor groups use different precisions.

Small F32 tensors, norms, biases and state tensors that were not suitable for quantization remain in their original precision.

Why the MTP block is higher precision

The MTP block was intentionally protected with Q6_K, and nextn.eh_proj with Q8_0, instead of compressing it to the body quantization level.

The goal is to preserve speculative-drafting quality and acceptance rate. The additional VRAM/storage cost is small relative to the full 27B model.

Importance matrix

An importance matrix was generated from:

• Dataset: Salesforce/wikitext
• Config: wikitext-103-raw-v1
• Context per chunk: 2048
• Chunks: 10
• Importance-matrix entries: 496

Command used:

.\llama-imatrix.exe `
  -m .\Qwen3.8-27B-Abliterated-BF16.gguf `
  -f .\calibration.txt `
  -o .\Qwen3.8-27B-Abliterated-imatrix.gguf `
  -c 2048 `
  --chunks 10 `
  --no-ppl `
  -ngl 18

The MTP block itself is not exercised by the standard imatrix forward pass, so it was explicitly assigned conservative Q6_K/Q8_0 types during quantization.

Quantization command

Built with llama.cpp build 10441 / commit 0177dcc73.

.\llama-quantize.exe `
  --imatrix .\Qwen3.8-27B-Abliterated-imatrix.gguf `
  --tensor-type '^blk\.64\.nextn\.eh_proj\.weight$=q8_0' `
  --tensor-type '^blk\.64\.attn_q\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_k\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_v\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_output\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_gate\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_up\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_down\.weight$=q6_k' `
  --tensor-type 'ffn_gate\.weight$=iq4_xs' `
  --tensor-type 'ffn_up\.weight$=iq4_xs' `
  --tensor-type 'ffn_down\.weight$=iq4_xs' `
  --tensor-type 'attn_qkv\.weight$=iq3_s' `
  --tensor-type 'attn_gate\.weight$=iq3_s' `
  --tensor-type 'attn_q\.weight$=q4_k' `
  --tensor-type 'attn_k\.weight$=q4_k' `
  --tensor-type 'attn_v\.weight$=q4_k' `
  --tensor-type 'attn_output\.weight$=q4_k' `
  --token-embedding-type iq2_s `
  --output-tensor-type q5_k `
  .\Qwen3.8-27B-Abliterated-BF16.gguf `
  .\Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  iq4_xs `
  14

Tested hardware

Tested on:

• GPU: AMD Radeon RX 9070 XT 16 GB
• Backend: Vulkan
• OS: Windows
• llama.cpp: build 10441, commit 0177dcc73
• GPU offload: all model layers
• Parallel slots: 1

Measured performance

These numbers are from local tests and should be treated as hardware/workload-specific, not universal benchmarks.

32K allocated context, MTP-3

eval time = 81200.04 ms / 4070 tokens
generation = 50.11 tokens/s
draft acceptance = 0.74563
2814 accepted / 3774 generated
mean len = 3.24

64K allocated context, MTP-3

eval time = 182513.24 ms / 9018 tokens
generation = 49.40 tokens/s
draft acceptance = 0.77141
6297 accepted / 8163 generated
mean len = 3.31

These tests started with relatively short prompts. Decode speed falls as the live context grows. In agentic coding workloads with roughly 25K–30K tokens of active history, observed generation was closer to the mid-20 tokens/s range depending on MTP acceptance.

Recommended llama.cpp server command — 64K

.\llama-server.exe `
  -m .\Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  --alias qwen38-local `
  -ngl all `
  --fit off `
  --jinja `
  -fa on `
  -np 1 `
  -c 65536 `
  -b 2048 `
  -ub 256 `
  -ctk q8_0 `
  -ctv q4_0 `
  --kv-unified `
  --reasoning on `
  --reasoning-budget 2048 `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --spec-draft-type-k q8_0 `
  --spec-draft-type-v q4_0 `
  --host 127.0.0.1 `
  --port 8080

For maximum raw throughput, keep the active conversation/context reasonably short. A 64K allocation does not mean generation speed will remain constant when all 64K tokens are occupied.

MTP notes

The most reliable tested configuration for this quant was:

--spec-type draft-mtp
--spec-draft-n-max 3

Higher speculative draft lengths are not automatically faster. In local testing, n-max 8 with a high p-min performed substantially worse on the tested Windows/Vulkan setup.

Download with Hugging Face CLI

Replace YOUR_USERNAME with the repository owner:

hf download YOUR_USERNAME/Qwen3.8-27B-Abliterated-IQ4-MIX-MTP-GGUF `
  Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  --local-dir .

Notes on quality

• This is a lossy quantization of the BF16 source model.
• The embedding tensor is aggressively compressed to IQ2_S.
• The output head is retained at Q5_K.
• The MTP block is deliberately higher precision than the main body.
• No comprehensive downstream benchmark suite has been run on this exact GGUF quantization.
• Results can differ between Vulkan, HIP/ROCm, CUDA, CPU backends, context lengths and sampling settings.

Abliterated model notice

The upstream checkpoint is an abliterated variant. This repository does not perform additional fine-tuning or alignment changes; it provides a GGUF quantization of that source checkpoint.

Users are responsible for evaluating the model for their intended use and complying with applicable laws, licenses and platform policies.

Credits

• Base model: Qwen/Qwen3.8-27B
• Abliterated checkpoint: windowsxp811203/Qwen3.8-27B-Abliterated
• Inference / quantization: llama.cpp
• Importance-matrix calibration text: Salesforce/wikitext

License

The source checkpoint metadata reports Apache-2.0. Please also review the upstream model repositories and their terms before redistribution or use.