玩模型开放模型 · 清晰来源 · 自主选择
返回模型目录
JonathanColetti /

Qwen3.8-27B-Uncensored-GGUF

HF 源站

JonathanColetti/Qwen3.8-27B-Uncensored-GGUF is an uncensored open-source model hosted on Hugging Face.

Text Generationllama.cppGGUFChineseEnglishMultilingualLicense: apache-2.027Buncensoredqwen3.8mtpspeculative-decodingimatrixquantizedconversational
README.md说明 / 使用方法

license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized pipeline_tag: text-generation library_name: llama.cpp language:

  • en
  • zh tags:
  • gguf
  • uncensored
  • qwen3.8
  • mtp
  • speculative-decoding
  • imatrix
  • quantized

Qwen3.8-27B-Uncensored-GGUF

未审查版(Uncensored)Qwen3.8-27B,以 GGUF 量化格式发布,保留并验证了多 Token 预测(Multi-Token Prediction, MTP)头部。

模型的拒绝行为已大幅减少,但并未完全消除。具体数值参见“实测行为”章节。模型的能力、训练数据和架构与原版保持一致。

目录

  • 处理方法
  • 文件说明
  • 概览
  • 文件列表
  • 困惑度(Perplexity)
    • 与未修改基座模型的基线对比
    • 基线的测试方法
  • 重要性矩阵(Importance Matrix)
  • 自行构建其他低比特量化版本
    • 第三方量化版本
  • 使用方法
    • ComfyUI
    • llama cpp
  • 验证
  • 实测行为
    • 指标解读
    • 重要注意事项
  • 环境要求
  • 局限性
  • 许可证
  • 本模型投机解码实测
    • IQ2_M

处理方法

  • 使用 Heretic 移除拒绝方向(Refusal Directions),在最小化拒绝次数的同时约束与基座模型的 KL 散度。没有手写移除拒绝的代码,无需微调,也没有引入额外的训练数据。
  • 消融(Abliteration)过程在 bf16 精度下运行(未使用 4-bit 量化)。生成的 LoRA 直接合并入 bf16 基座中,因此所发布的权重并非经过量化后往返转换的产物。
  • 合并后,mtp.* 张量直接从基座检查点逐字复制。消融过程从未修改过它们,而是修改主网络层中的 attn.o_proj 与 mlp.down_proj。
  • Draft 头是在未修改的模型上训练的,因此接受率可能会略有下降。投机解码(Speculative Decoding)会对每个 Token 进行针对目标模型的验证,因此输出质量不受任何影响。
  • imatrix 直接根据 f16 权重计算,而非来自中间量化版本,从而使校准看到的是真实的权重。

文件说明

类别文件适用场景
融合版(Fused)Qwen3.8-27B-Uncensored-<QUANT>.gguf单文件。MTP 作为内置 Draft 头直接集成。
Target + Draft 分离版Qwen3.8-27B-Uncensored-noMTP-<QUANT>.gguf + Qwen3.8-27B-Uncensored-draft-<QUANT>.gguf你的运行时环境需要显式指定 --model-draft 参数。
视觉投影(Vision)mmproj-Qwen3.8-27B-Uncensored-F16.gguf配合兼容的视觉运行时进行图像输入。
llama-server 专用的 MTP 头Qwen3.8-27B-Uncensored-draft-<QUANT>.gguf专为 llama-server 的 --model-draft 参数提供的 MTP 头文件。

Draft 头提供 Q8_0 和 Q4_0 两种精度的权重文件。Q8_0 为默认版本,下方的投机解码测试数据即基于此测量。Q4_0 可为显存紧张的场景节省 1.5 GB 显存,其接受率未在此处专门测量。

F16 视觉投影器使用标准的 mmproj 前缀,以便兼容的运行时自动发现。

概览

属性详情
基座模型Qwen/Qwen3.8-27B
架构Qwen3_5ForConditionalGeneration
层数64
词表大小248320
MTP 层数1
视觉支持是
上下文长度262144
量化规格IQ2_M, IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0
imatrixwikitext-2 raw, 200 chunks, 已发布
转换工具llama.cpp a94d563ed

文件列表

文件名大小MTPPPL (wikitext-2)
Qwen3.8-27B-Uncensored-IQ2_M.gguf10.6 GB是PPL = 7.8581 +/- 0.27481
Qwen3.8-27B-Uncensored-IQ4_XS.gguf15.3 GB是PPL = 7.1583 +/- 0.25019
Qwen3.8-27B-Uncensored-Q4_K_M.gguf16.8 GB是PPL = 7.1814 +/- 0.25227
Qwen3.8-27B-Uncensored-Q5_K_M.gguf19.5 GB是PPL = 7.1573 +/- 0.25055
Qwen3.8-27B-Uncensored-Q6_K.gguf22.4 GB是PPL = 7.1689 +/- 0.25149
Qwen3.8-27B-Uncensored-Q8_0.gguf29.0 GB是PPL = 7.1764 +/- 0.25195
Qwen3.8-27B-Uncensored-draft-Q4_0.gguf1.7 GB--
Qwen3.8-27B-Uncensored-draft-Q8_0.gguf3.2 GB--
Qwen3.8-27B-Uncensored-noMTP-IQ2_M.gguf10.2 GB否PPL = 7.8581 +/- 0.27481
Qwen3.8-27B-Uncensored-noMTP-IQ4_XS.gguf15.1 GB否-
Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.gguf16.5 GB否-
Qwen3.8-27B-Uncensored-noMTP-Q5_K_M.gguf19.2 GB否-
Qwen3.8-27B-Uncensored-noMTP-Q6_K.gguf22.1 GB否-
Qwen3.8-27B-Uncensored-noMTP-Q8_0.gguf28.6 GB否-
mmproj-Qwen3.8-27B-Uncensored-F16.gguf0.9 GB--
Qwen3.8-27B-Uncensored-imatrix.dat13.6 MB--

困惑度(Perplexity)

在本构建上测量,所有量化版本均在同一次会话中基于相同的 f16 基线进行测试,因此各行数据具备横向可比性。

文件名PPL (wikitext-2)对比 f16
Qwen3.8-27B-Uncensored-f16.gguf (基准线,未发布)7.1557 +/- 0.25104
Qwen3.8-27B-Uncensored-Q5_K_M.gguf7.1573 +/- 0.25055+0.0016
Qwen3.8-27B-Uncensored-IQ4_XS.gguf7.1583 +/- 0.25019+0.0026
Qwen3.8-27B-Uncensored-Q6_K.gguf7.1689 +/- 0.25149+0.0132
Qwen3.8-27B-Uncensored-Q8_0.gguf7.1764 +/- 0.25195+0.0207
Qwen3.8-27B-Uncensored-Q4_K_M.gguf7.1814 +/- 0.25227+0.0257
Qwen3.8-27B-Uncensored-IQ2_M.gguf7.8581 +/- 0.27481+0.7024

在看排名之前,请务必先注意误差范围。 除 IQ2_M 之外的每一行都处于 0.026 的差距区间内,而标准误差约为 0.25,因此这些量化版本与 f16 之间以及它们彼此之间在统计上无法明确区分,这里的顺序仅是随机噪声。切勿得出 Q8_0 差于 Q5_K_M 的结论。该测量真正能够区分出的只有 IQ2_M,其比基线高出约 2.8 个标准误差。

noMTP-* 对应的文件未列出,因为它们的测量结果与融合版完全一致。在常规前向传递期间,MTP 模块是不激活的,本测试也证实了这一点:融合版和 noMTP 版 IQ2_M 测得的 PPL 均为 7.8581,融合版和 noMTP 版 f16 均为 7.1557。

测试语料库为 Salesforce/wikitext,wikitext-2-raw-v1,test-00000-of-00001.parquet,提取 text 列使用 \n 拼接。此文件与用于重要性矩阵计算的文件完全相同,详见下文。

llama-perplexity -m Qwen3.8-27B-Uncensored-IQ2_M.gguf \
  -f calibration.txt -ngl 99 --chunks 20

与未修改基座模型的基线对比

上表中的每一行都是本模型,因此该表展示的仅是量化带来的损失。它并未反映消融过程本身的代价。要评估消融代价,需要在相同测试环境下测试未经修改的原版基座模型,结果如下:

模型文件PPL (wikitext-2)对比基座 bf16
Qwen/Qwen3.8-27BQwen3.8-27B-BF16.gguf6.5129 +/- 0.04209基线
Qwen/Qwen3.8-27BQwen3.8-27B-Q8_0.gguf6.5122 +/- 0.04208-0.0007
本模型Qwen3.8-27B-Uncensored-BF16.gguf (未发布)6.5563 +/- 0.04248+0.0434

基座文件来自 ggml-org/Qwen3.8-27B-GGUF。本模型的 bf16 GGUF 是在本地从发布的 bf16 权重转换而来的,出于与 f16 相同的原因未在此发布。

权重修改(消融)带来了 0.0434 的困惑度上升,相对增幅约 0.7%。在相同 Token 上进行配对差值测试(而非两次独立运行)时,差值为 0.0437 +/- 0.0163,因此这是真实存在的代价,而非测试噪声。基座 Q8_0 数据作为对照:它与基座 bf16 的差异仅为 0.0007,远低于此处报告的差距。

分布重叠度是直接测量的,而非从困惑度差距推断:

评估指标测量结果
基座 bf16 与本模型 bf16 的平均 KL 散度0.0141 +/- 0.0025
最大 KL 散度19.956
首选 Token 一致率(Same top token)96.72%
比对 Token 数20480

此项测量与“实测行为”下的首 Token KL 散度不同(后者是 Heretic 优化的目标)。该项测试运行在普通的语料库文本上,涵盖每个位置,而不仅限于第一个 Token。

基线的测试方法

在单台机器、单次会话、相同可执行程序中,使用相同的语料库文件和参数测试每一行,并将每个模型全部卸载到 GPU:

llama-perplexity -m Qwen3.8-27B-BF16.gguf \
  -f calibration.txt -c 2048 -b 2048 -ngl 99
参数项说明
语料库calibration.txt,字节级完全等同于重要性矩阵下的文件,md5 为 d998c24b049cf7c009dbf2672da70b5a
覆盖范围全文件,145 个 chunk,每个 chunk 2048 tokens
构建版本llama.cpp a94d563ed
硬件环境NVIDIA H200 NVL,使用 -ngl 99 完全卸载至 GPU

这些数据无法与上一张表格直接比较。 上表只测了 20 个 chunk,而本表读取了整个文件,两者涵盖的文本量不同,不能直接跨表比较。各个表格内部的各行之间是具备可比性的。全文件运行的标准误差约为 0.042(相比之下 20 chunk 运行的标准误差为 0.25),这正是如此微小的差距依然值得报告的原因。

困惑度只能检测出明显的量化损坏,仅此而已。它无法评估推理、代码、多语言能力或拒绝行为。请参阅“重要注意事项”章节。

重要性矩阵(Importance Matrix)

Qwen3.8-27B-Uncensored-imatrix.dat 是构建此仓库中所有量化版本所使用的重要性矩阵。无论是融合版还是 noMTP 版,全部 12 个 IQ2_M、IQ4_XS、Q4_K_M、Q5_K_M、Q6_K 以及 Q8_0 文件均在其元数据中记录了该矩阵。独立的 draft-Q8_0 头以及 mmproj-Qwen3.8-27B-Uncensored-F16.gguf 投影器则未包含,因为两者在构建时均未使用 imatrix。

发布该矩阵是为了让此处的文件具备可复现性,并方便你自行构建未包含在本仓库中的其他量化规格。

配置项详细信息
语料库Salesforce/wikitext,wikitext-2-raw-v1,test-00000-of-00001.parquet
组合方式提取 text 列使用 \n 拼接,共 1,292,013 字节,md5 为 d998c24b049cf7c009dbf2672da70b5a
Chunks200 x 512 tokens
计算基准基于 f16 GGUF 计算,而非中间量化版本
构建版本llama.cpp a94d563ed

使用前需了解的两点:

  • 尽管扩展名为 .dat,但它是 GGUF 格式(general.type = imatrix)。在支持 GGUF imatrix 之前的旧版 llama.cpp 将无法加载它。
  • 它不包含 blk.64(即 MTP 块)的条目。llama-imatrix 在常规前向计算中从不激活 Draft 头,因此没有为其收集激活信息。这在后续步骤中非常关键。

文件来源可检验而非仅凭口头声明:使用此文件将 f16 重新量化为 Q4_K_M,能够完全复现发布的 Qwen3.8-27B-Uncensored-Q4_K_M.gguf,全部 866 个张量在字节级完全一致。

自行构建其他低比特量化版本

本仓库提供的最小量化版本是 10.6 GB 的 IQ2_M。如果你需要体积更小的版本,或者未发布的量化类型,可以使用上述 imatrix 自行构建。

你将需要 f16 GGUF 文件,由于其大小达 54 GB,本仓库未直接提供。你可以从已公开的 bf16 权重构建:

hf download JonathanColetti/Qwen3.8-27B-Uncensored --local-dir Qwen3.8-27B-Uncensored
python convert_hf_to_gguf.py Qwen3.8-27B-Uncensored \
  --outfile Qwen3.8-27B-Uncensored-f16.gguf --outtype f16

如需构建 noMTP 变体,只需在上述命令中添加 --no-mtp。MTP 分片已直接合并进 bf16 仓库中,无需预先进行任何恢复操作。

必须固定(Pin)MTP 块的精度。 由于 imatrix 中缺少 blk.64 的条目,而 IQ3_XXS、IQ2_XXS、IQ2_S 和 IQ2_M 强制要求逐张量的重要性数据,因此如果未对 MTP 块进行类型固定,融合版的低比特构建不仅会破坏 Draft 头,llama-quantize 甚至会直接报错拒绝运行。将 blk.64 固定为 q8_0 可以绕过这一要求并保持 Draft 头的完整:

llama-quantize \
  --imatrix Qwen3.8-27B-Uncensored-imatrix.dat \
  --tensor-type 'blk\.64\.=q8_0' \
  --token-embedding-type q4_K \
  Qwen3.8-27B-Uncensored-f16.gguf Qwen3.8-27B-Uncensored-IQ2_XXS.gguf IQ2_XXS

--token-embedding-type q4_K 是该模型最关键的体积压缩杠杆。llama.cpp 在所有 IQ2/IQ1 类型中都会强制将 token_embd 提升至 Q5_K,在 248320 的大词表下,该部分约占参数总量的 8% 到 10%。切勿低于 q4_K。对于 noMTP-* 构建,由于没有 MTP 块需要固定,因此可以省略 --tensor-type 参数。

构建完成后,请验证该模块是否完好保留,而不是假设它存在:

python quantize.py inspect Qwen3.8-27B-Uncensored-IQ2_XXS.gguf   # 预期输出 65/65, has_mtp: true

2-bit 警告: IQ2_M 是本仓库中性能下降最明显的文件,任何低于它的自建版本表现都会更差。这种性能损失对该模型的特有用途(无审查)打击最为严重。临近原有拒绝边界的行为本就是该模型最脆弱的属性(参见“重要注意事项”),而 2-bit 会进一步加剧这一问题。困惑度测试会显示模型并未彻底损坏,但它无法证明拒绝边界在 2-bit 下是否还能保持 Q6_K 时的表现,本仓库也未在 2-bit 下对拒绝边界进行评测。

第三方量化版本

有第三方发布了基于此模型制作的多种混合精度变体:

  • zerodigest/Qwen3.8-27B-Uncensored-YMQ-MTP-GGUF

此版本非本仓库制作,亦无任何附属关联。 该文件并非由本人构建,本人未对其文件及发布的指标进行验证。其引用的任何测试指标都不是在本卡所用的测试框架下测得的,因此无法与本文档中的困惑度数值直接比较。在此提供链接仅因部分用户咨询了未发布的规格尺寸,并不代表任何形式的背书。

使用方法

ComfyUI

测试环境版本:

  • ComfyUI commit 0a33ed6c28f926d14536235771c222f9e6d1026b
  • ComfyUI QwenVL commit e79582111e5835787574bcc17adc4bf4cf3a07f4
  • llama-cpp-python 版本 0.3.48,commit c57b174711166b889f06a35a30af95a75db24705
# 安装 ComfyUI 以及 QwenVL 节点
git clone https://github.com/comfyanonymous/ComfyUI.git
git -C ComfyUI checkout 0a33ed6c28f926d14536235771c222f9e6d1026b
git clone https://github.com/1038lab/ComfyUI-QwenVL.git ComfyUI/custom_nodes/ComfyUI-QwenVL
git -C ComfyUI/custom_nodes/ComfyUI-QwenVL checkout e79582111e5835787574bcc17adc4bf4cf3a07f4

# 安装依赖项
uv pip install -r ComfyUI/requirements.txt
uv pip install -r ComfyUI/custom_nodes/ComfyUI-QwenVL/requirements.txt
CMAKE_ARGS="-DGGML_CUDA=on -DLLAMA_CPP_PYTHON_VISION=on" \
  uv pip install \
  "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git@c57b174711166b889f06a35a30af95a75db24705"

# 下载语言模型和 F16 视觉投影器
hf download JonathanColetti/Qwen3.8-27B-Uncensored-GGUF \
  Qwen3.8-27B-Uncensored-Q4_K_M.gguf \
  mmproj-Qwen3.8-27B-Uncensored-F16.gguf \
  --local-dir \
  ComfyUI/models/LLM/GGUF/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF

# 启动 ComfyUI
cd ComfyUI
python main.py --listen 127.0.0.1 --port 8188

在 ComfyUI/custom_nodes/ComfyUI-QwenVL/custom_models.json 中添加该模型:

{
  "hf_models": {},
  "gguf_models": {
    "Qwen3.8 27B Uncensored Q4 K M": {
      "author": "JonathanColetti",
      "repo_name": "Qwen3.8-27B-Uncensored-GGUF",
      "repo_id": "JonathanColetti/Qwen3.8-27B-Uncensored-GGUF",
      "model_files": [
        "Qwen3.8-27B-Uncensored-Q4_K_M.gguf"
      ]
    }
  }
}

视觉投影器文件名以 mmproj 开头,因此会被自动识别。

llama cpp

llama-server -m Qwen3.8-27B-Uncensored-Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -ngl 99 -c 8192

目标模型 + 显式指定 Draft 模型:

llama-server -m Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --model-draft Qwen3.8-27B-Uncensored-draft-Q8_0.gguf \
  -ngl 99 -c 8192

--spec-draft-n-max 默认值为 3。吞吐表现取决于你的硬件配置,建议自行扫描测试最佳值——针对不同 Draft 长度的测试数据可参见 qwen3.8-spec-decode-bench。

验证

在量化完成后,每个构建产物均经过了 MTP 张量存留情况的检查,而不是仅根据转换参数进行推断:

python quantize.py inspect Qwen3.8-27B-Uncensored-Q4_K_M.gguf

此工具会输出元数据键、声明的 block_count 以及实际存在的层块数。如果一个融合版文件中实际存在的块数未超过其声明的层数,则说明未能保留 MTP 块。

文件名MTP实际层数 / 声明层数
Qwen3.8-27B-Uncensored-f16.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-f16.ggufFalse64/64
Qwen3.8-27B-Uncensored-IQ2_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-IQ2_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-IQ4_XS.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-IQ4_XS.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q4_K_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q5_K_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q5_K_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q6_K.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q6_K.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q8_0.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q8_0.ggufFalse64/64

实测行为

在相同参数设置下与未经修改的基座模型进行基准对比。差值(Δ)是最关键的指标:它单独分离出了权重编辑所带来的代价。

任务基座模型未审查版(本模型)Δ
MMLU83.483.3-0.2
ARC-Challenge58.957.7-1.2
HellaSwag82.882.9+0.1
Winogrande76.175.3-0.8
平均值-0.5

测试通过 lm-evaluation-harness 以 Zero-shot 方式运行,精度为 bf16,两个模型在同一次会话中评分。每个差值均处于或接近报告的标准误差范围内(MMLU +/- 0.30、ARC +/- 1.44、HellaSwag +/- 0.38、Winogrande +/- 1.21),因此没有明显的迹象表明其脱离了运行间的随机噪声。

这些均为 0-shot 成绩,不可与千问官方公布的分数直接对比(官方评测采用 Few-shot 提示词)。两个模型之间的成绩是直接可比的,这才是评测的目的。另外需要说明的是,对于达到此 MMLU 水平的模型而言,ARC-Challenge 的得分偏低;由于基座模型在相同设置下得分也是 58.9,因此这是该推理调优模型对评测格式敏感所致,并非消融造成的损伤。

基准测试未涵盖的方面:未包含生成式评测(如 GSM8K、HumanEval),未包含数学或编程代码评测,未包含多语言评测,且测试框架仅加载了文本网络栈。此处未对视觉塔或 MTP 投机解码进行评测。

测量项基座模型本模型
拒绝次数(100 个留出的有害提示词)98/10012/100
相对基座的 KL 散度(首 Token)00.1191

搜索过程:进行了 200 次 Heretic 试验,得到 23 个非劣解(帕累托前沿点)。最终发布的模型为标有标记的行:

拒绝次数KL 散度状态
12/1000.1191<= 发布版本
13/1000.1052
19/1000.0722
23/1000.0635
26/1000.0507
27/1000.0410
35/1000.0406
36/1000.0387
41/1000.0366
44/1000.0352
46/1000.0334
48/1000.0331
51/1000.0321
52/1000.0294
60/1000.0290
76/1000.0280
77/1000.0247
83/1000.0204
86/1000.0193
91/1000.0170
96/1000.0146
97/1000.0044
98/1000.0004

指标解读

拒绝率(Refusal rate) 是指在来自 mlabonne/harmful_behaviors(测试集切分)的 100 个留出提示词上的拒绝计数——这些是明确具有危害性的请求,而非正常良性请求。因此该数值并非“过度拒绝率”:它不能告诉你模型拒绝正常合法任务的频率。它反映的是在面对有害请求时,模型保留了原版多少的安全防护行为。

KL 散度(KL divergence) 是针对未修改基座模型在首个 Token 分布上测得的,代表优化器用来衡量“我们对模型破坏了多少”的代理指标。数值越低越接近基座。这只是一个代理参考,并非实际能力的直接测量。较低的 KL 散度并不能证明推理或编程能力完整保留,本文档在此不做任何此类保证。

这两项指标相互权衡。Heretic 会在其间搜索帕累托前沿;最终发布的版本只是该前沿上的一个选择点,而非全局最优解。

重要注意事项

  • 拒绝测试在非思考模式(Non-thinking mode)下进行。 该模型的对话模板会开启一个 <think> 块,因此评测中显式闭合了该标签,以便直接对回答进行打分而非评判思考过程。在开启思考模式时,拒绝率可能会发生变动(增减皆有可能)。
  • 评测样本仅为来自单一数据集的 100 个提示词。 其结论只能推广到该类有害请求的分布范围内。模型在其他主题上的拒绝表现尚未进行表征。
  • 困惑度仅基于 wikitext-2 测试(参见文件列表)。它仅能检测严重的量化损伤,无法检测在推理、代码或多语言任务上的能力损失。
  • 量化会加剧上述所有问题。 上述各项评测均基于 bf16 合并模型;而你下载的文件是经过量化的。

环境要求

llama.cpp 在 PR #22673 中引入了 MTP 投机解码支持。早于该合并请求的版本在加载这些文件时会直接静默忽略 MTP 张量。

局限性

  • 拒绝行为只是被减少,而非彻底消除,也未进行重新引导。该模型能够尝试解答原模型会拒绝的许多请求,但仍有相当一部分请求会被拒绝——详情参见“实测行为”。
  • 模型在原有拒绝边界附近的行为比基座模型更不稳定。
  • 较低精度的量化会加剧这一问题,其中 IQ2_M 影响最为显著。建议在 Q6_K 或 Q8_0 上评估其行为,而非 IQ2_M 或 IQ4_XS。本仓库未在 2-bit 下对拒绝边界进行测试。
  • 能力基准测试显示,在 MMLU、ARC-Challenge、HellaSwag 和 Winogrande 上平均相较基座下降了 0.5 分。详见“实测行为”。未执行生成式、数学、代码或多语言评测。

许可证

遵循 Apache 2.0 协议,继承自 Qwen/Qwen3.8-27B。基座模型的许可证及合理使用政策同样适用于你对本衍生模型的使用。

本模型投机解码实测

提示类型投机模式 (spec_type)n_maxtok/s相对基线提升
散文(prose)无 (none)-74.81.00x
散文(prose)draft-mtp189.01.19x
散文(prose)draft-mtp285.71.15x
散文(prose)draft-mtp372.00.96x
散文(prose)draft-mtp471.10.95x
散文(prose)draft-mtp562.90.84x
散文(prose)draft-mtp653.80.72x
散文(prose)draft-mtp749.90.67x
散文(prose)draft-mtp859.90.80x
代码(code)无 (none)-74.71.00x
代码(code)draft-mtp195.41.28x
代码(code)draft-mtp292.91.24x
代码(code)draft-mtp382.61.11x
代码(code)draft-mtp474.91.00x
代码(code)draft-mtp567.40.90x
代码(code)draft-mtp659.40.80x
代码(code)draft-mtp755.60.74x
代码(code)draft-mtp870.90.95x
对话(chat)无 (none)-74.71.00x
对话(chat)draft-mtp190.61.21x
对话(chat)draft-mtp284.21.13x
对话(chat)draft-mtp376.11.02x
对话(chat)draft-mtp470.40.94x
对话(chat)draft-mtp564.30.86x
对话(chat)draft-mtp655.20.74x
对话(chat)draft-mtp750.20.67x
对话(chat)draft-mtp854.10.72x

IQ2_M

在本构建上测量:NVIDIA H200 NVL,生成 256 个 Token,3 次重复取中位数,n_max 扫描范围 1 到 3。上表数据源自初始发布版本,未记录具体硬件配置,因此请比较相对加速比而非绝对速率。

提示类型投机模式 (spec_type)n_maxtok/s相对基线提升
散文(prose)无 (none)-75.41.00x
散文(prose)draft-mtp185.21.13x
散文(prose)draft-mtp283.21.10x
散文(prose)draft-mtp377.41.03x
代码(code)无 (none)-75.51.00x
代码(code)draft-mtp195.81.27x
代码(code)draft-mtp299.81.32x
代码(code)draft-mtp396.01.27x
对话(chat)无 (none)-75.31.00x
对话(chat)draft-mtp187.41.16x
对话(chat)draft-mtp281.31.08x
对话(chat)draft-mtp378.41.04x

MTP 头在 2-bit 量化下仍能保留,因为它是固定为 q8_0 精度的,并未与主网络层一起被量化,因此投机解码在此配置下依然有效。

将 noMTP-IQ2_M 与发布的 draft-Q8_0 搭配使用时,在散文提示词下且 n_max 为 2 时速率可达 97.7 tok/s(即 1.30x),在该任务上超越了单文件融合版。原因在于:融合版的 MTP 头与主模型共享了低比特的 token_embd(q4_K)和 output(Q5_K)张量,而独立的 Draft 头则拥有专属的 Q8_0 副本。这种分离架构需要 13.3 GB 权重显存(相比之下融合版为 10.6 GB),因此仅在显存充裕时才建议选用。

源站原文

license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized pipeline_tag: text-generation library_name: llama.cpp language:

  • en
  • zh tags:
  • gguf
  • uncensored
  • qwen3.8
  • mtp
  • speculative-decoding
  • imatrix
  • quantized

Qwen3.8-27B-Uncensored-GGUF

Uncensored Qwen3.8-27B, published as GGUF quantizations with the multi token prediction (MTP) head retained and verified.

Refusal behaviour has been substantially reduced, not eliminated. See Measured behaviour for the numbers. Capabilities, training data, and architecture are otherwise unchanged.

Table of Contents

  • Method
  • Whats here
  • Overview
  • Files
  • Perplexity
    • Baseline against the unmodified base model
    • How the baseline was measured
  • Importance matrix
  • Building other low bit quants yourself
    • Third party quants
  • Usage
    • ComfyUI
    • llama cpp
  • Verification
  • Measured behaviour
    • How to read these
    • Caveats that matter
  • Requirements
  • Limitations
  • License
  • Speculative decoding, measured on this model
    • IQ2_M

Method

  • Refusal directions removed with Heretic, which co minimizes refusal count against KL divergence from the base model. No handwritten refusal removal code, no finetuning, no additional training data.
  • Abliteration runs at bf16 (no 4 bit quantization). the resulting LoRA is merged into the bf16 base, so the published weights are not a quantized round trip.
  • mtp.* tensors are copied verbatim from the base checkpoint after merging. Abliteration never touches them. Instead abliteration modifies attn.o_proj and mlp.down_proj in the main stack.
  • The draft head was trained against the unmodified model, so acceptance rate may fall slightly. Speculative decoding verifies every token against the target, so output quality is unaffected.
  • imatrix is computed directly from the f16, not from an intermediate quantization, so calibration sees the real weights.

Whats here

FamilyFilesUse when
FusedQwen3.8-27B-Uncensored-<QUANT>.ggufOne file. MTP rides inline as a built-in draft.
Target + draftQwen3.8-27B-Uncensored-noMTP-<QUANT>.gguf + Qwen3.8-27B-Uncensored-draft-<QUANT>.ggufYour runtime wants an explicit --model-draft.
Visionmmproj-Qwen3.8-27B-Uncensored-F16.ggufImage input with a compatible vision runtime.
MTP Head for llama serverQwen3.8-27B-Uncensored-draft-<QUANT>.ggufMTP head for llama-server, specifically for the flag --model-draft

The draft head ships at Q8_0 and Q4_0, the same weights at two precisions. Q8_0 is the default and is what the speculative decoding tables below were measured with. Q4_0 saves 1.5 GB for tight VRAM budgets, and its acceptance rate has not been measured here.

The F16 vision projector uses the standard mmproj prefix for automatic discovery by compatible runtimes.

Overview

BaseQwen/Qwen3.8-27B
ArchitectureQwen3_5ForConditionalGeneration
Layers64
Vocab248320
MTP layers1
Visionyes
Context262144
QuantsIQ2_M, IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0
imatrixwikitext-2 raw, 200 chunks, published
Converted withllama.cpp a94d563ed

Files

FileSizeMTPPPL (wikitext-2)
Qwen3.8-27B-Uncensored-IQ2_M.gguf10.6 GByesPPL = 7.8581 +/- 0.27481
Qwen3.8-27B-Uncensored-IQ4_XS.gguf15.3 GByesPPL = 7.1583 +/- 0.25019
Qwen3.8-27B-Uncensored-Q4_K_M.gguf16.8 GByesPPL = 7.1814 +/- 0.25227
Qwen3.8-27B-Uncensored-Q5_K_M.gguf19.5 GByesPPL = 7.1573 +/- 0.25055
Qwen3.8-27B-Uncensored-Q6_K.gguf22.4 GByesPPL = 7.1689 +/- 0.25149
Qwen3.8-27B-Uncensored-Q8_0.gguf29.0 GByesPPL = 7.1764 +/- 0.25195
Qwen3.8-27B-Uncensored-draft-Q4_0.gguf1.7 GB--
Qwen3.8-27B-Uncensored-draft-Q8_0.gguf3.2 GB--
Qwen3.8-27B-Uncensored-noMTP-IQ2_M.gguf10.2 GBnoPPL = 7.8581 +/- 0.27481
Qwen3.8-27B-Uncensored-noMTP-IQ4_XS.gguf15.1 GBno-
Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.gguf16.5 GBno-
Qwen3.8-27B-Uncensored-noMTP-Q5_K_M.gguf19.2 GBno-
Qwen3.8-27B-Uncensored-noMTP-Q6_K.gguf22.1 GBno-
Qwen3.8-27B-Uncensored-noMTP-Q8_0.gguf28.6 GBno-
mmproj-Qwen3.8-27B-Uncensored-F16.gguf0.9 GB--
Qwen3.8-27B-Uncensored-imatrix.dat13.6 MB--

Perplexity

Measured on this build, every quant in one session against the same f16 baseline, so the rows are comparable to each other.

FilePPL (wikitext-2)vs f16
Qwen3.8-27B-Uncensored-f16.gguf (baseline, not shipped)7.1557 +/- 0.25104
Qwen3.8-27B-Uncensored-Q5_K_M.gguf7.1573 +/- 0.25055+0.0016
Qwen3.8-27B-Uncensored-IQ4_XS.gguf7.1583 +/- 0.25019+0.0026
Qwen3.8-27B-Uncensored-Q6_K.gguf7.1689 +/- 0.25149+0.0132
Qwen3.8-27B-Uncensored-Q8_0.gguf7.1764 +/- 0.25195+0.0207
Qwen3.8-27B-Uncensored-Q4_K_M.gguf7.1814 +/- 0.25227+0.0257
Qwen3.8-27B-Uncensored-IQ2_M.gguf7.8581 +/- 0.27481+0.7024

Read the error bars before reading the ordering. Every row except IQ2_M sits inside a span of 0.026 against a standard error of roughly 0.25, so those quants are not separable from the f16 or from each other, and their ordering here is noise. Do not conclude that Q8_0 is worse than Q5_K_M. The only difference this measurement actually resolves is IQ2_M, which is about 2.8 standard errors above the baseline.

The noMTP-* twins are not listed because they measure identically to their fused counterparts. The MTP block is inert during a normal forward pass, which was confirmed here: fused and noMTP IQ2_M both return 7.8581, and fused and noMTP f16 both return 7.1557.

Corpus is Salesforce/wikitext, wikitext-2-raw-v1, test-00000-of-00001.parquet, text column joined with \n. This is the same file used for the importance matrix, described in full below.

llama-perplexity -m Qwen3.8-27B-Uncensored-IQ2_M.gguf \
  -f calibration.txt -ngl 99 --chunks 20

Baseline against the unmodified base model

Every row above is this model, so that table shows what quantization costs and nothing more. It does not show what the abliteration costs. That needs the unmodified base model run on the same harness, which is what follows.

ModelFilePPL (wikitext-2)vs base bf16
Qwen/Qwen3.8-27BQwen3.8-27B-BF16.gguf6.5129 +/- 0.04209baseline
Qwen/Qwen3.8-27BQwen3.8-27B-Q8_0.gguf6.5122 +/- 0.04208-0.0007
This modelQwen3.8-27B-Uncensored-BF16.gguf (not shipped)6.5563 +/- 0.04248+0.0434

Base files come from ggml-org/Qwen3.8-27B-GGUF. The bf16 GGUF of this model was converted locally from the published bf16 weights and is not shipped here, for the same reason the f16 is not.

The weight edit costs 0.0434 perplexity, about 0.7% relative. Measured as a paired difference over the same tokens rather than as two independent runs, the gap is 0.0437 +/- 0.0163, so it is a real cost rather than noise. The base Q8_0 row is the control: it lands 0.0007 from the base bf16, well below the gap being reported.

Distribution overlap was measured directly rather than inferred from the perplexity gap:

Mean KL divergence, base bf16 against this model at bf160.0141 +/- 0.0025
Maximum KL divergence19.956
Same top token96.72%
Tokens compared20480

This is a different measurement from the first token KL under Measured behaviour, which is the figure Heretic optimizes against. This one runs over ordinary corpus text and covers every position, not just the first.

How the baseline was measured

One session, one machine, one binary, the same corpus file and the same settings for every row, with each model fully offloaded:

llama-perplexity -m Qwen3.8-27B-BF16.gguf \
  -f calibration.txt -c 2048 -b 2048 -ngl 99
Corpuscalibration.txt, byte identical to the file under Importance matrix, md5 d998c24b049cf7c009dbf2672da70b5a
Coveragewhole file, 145 chunks of 2048 tokens
Built withllama.cpp a94d563ed
HardwareNVIDIA H200 NVL, full offload at -ngl 99

These numbers do not line up with the table above. That table stops at 20 chunks and this one reads the whole file, so the two cover different amounts of text and cannot be compared directly. Rows within each table are comparable to each other. The whole file run carries a standard error of roughly 0.042 against 0.25 for the 20 chunk run, which is why a gap this small is worth reporting at all.

Perplexity detects gross quantization damage and nothing else. It does not measure reasoning, code, multilingual ability, or refusal behaviour. See Caveats that matter.

Importance matrix

Qwen3.8-27B-Uncensored-imatrix.dat is the importance matrix every quantization in this repo was built with. All twelve of the IQ2_M, IQ4_XS, Q4_K_M, Q5_K_M, Q6_K and Q8_0 files, fused and noMTP alike, record it in their own metadata. The standalone draft-Q8_0 head and the mmproj-Qwen3.8-27B-Uncensored-F16.gguf projector do not, because neither was built with one.

It is published so the files here can be reproduced, and so you can build quants that are not in this repo.

CorpusSalesforce/wikitext, wikitext-2-raw-v1, test-00000-of-00001.parquet
Assemblytext column joined with \n, giving 1,292,013 bytes, md5 d998c24b049cf7c009dbf2672da70b5a
Chunks200 x 512 tokens
Computed fromthe f16 GGUF, not an intermediate quantization
Built withllama.cpp a94d563ed

Two things to know before you use it:

  • It is GGUF format despite the .dat extension (general.type = imatrix). llama.cpp builds predating GGUF imatrix support will not load it.
  • It contains no entries for blk.64, the MTP block. llama-imatrix never activates the draft head during a normal forward pass, so no activations are collected for it. This matters below.

Provenance is checkable rather than asserted: requantizing the f16 to Q4_K_M with this file reproduces the published Qwen3.8-27B-Uncensored-Q4_K_M.gguf to byte-identical tensors across all 866 tensors.

Building other low bit quants yourself

The floor here is IQ2_M at 10.6 GB. If you want something smaller, or a type that is not published, the imatrix above lets you build it.

You will need the f16 GGUF, which is not published here because it is 54 GB. Build it from the bf16 weights, which are public:

hf download JonathanColetti/Qwen3.8-27B-Uncensored --local-dir Qwen3.8-27B-Uncensored
python convert_hf_to_gguf.py Qwen3.8-27B-Uncensored \
  --outfile Qwen3.8-27B-Uncensored-f16.gguf --outtype f16

Add --no-mtp to that command for the noMTP variant. The MTP shard is already grafted into the bf16 repo, so nothing needs restoring first.

The MTP block must be pinned. Because the imatrix has no blk.64 entries, and because IQ3_XXS, IQ2_XXS, IQ2_S and IQ2_M require per-tensor importance data, a fused low-bit build without a pin does not merely degrade the draft head. llama-quantize refuses to run at all. Pinning blk.64 to q8_0 sidesteps the requirement and keeps the draft head intact:

llama-quantize \
  --imatrix Qwen3.8-27B-Uncensored-imatrix.dat \
  --tensor-type 'blk\.64\.=q8_0' \
  --token-embedding-type q4_K \
  Qwen3.8-27B-Uncensored-f16.gguf Qwen3.8-27B-Uncensored-IQ2_XXS.gguf IQ2_XXS

--token-embedding-type q4_K is the largest size lever on this model. llama.cpp force-bumps token_embd to Q5_K on every IQ2/IQ1 ftype, and at 248320 vocab that is roughly 8 to 10% of parameters. Do not go below q4_K. Omit the --tensor-type pin for noMTP-* builds, since there is no block to pin.

Verify afterwards that the block survived, rather than assuming it did:

python quantize.py inspect Qwen3.8-27B-Uncensored-IQ2_XXS.gguf   # expect 65/65, has_mtp: true

2 bit warning. IQ2_M is the most degraded file in this repo, and anything you build below it will be worse. Expect the loss to land hardest on the thing this model is used for. Behaviour near the old refusal boundary is already its least stable property (see Caveats that matter), and 2 bit compounds exactly that. Perplexity will tell you the model is not broken. It will not tell you the refusal boundary still behaves the way it does at Q6_K, and nothing in this repo measures that at 2 bit.

Third party quants

A third party publishes a range of mixed-precision variants derived from this model:

  • zerodigest/Qwen3.8-27B-Uncensored-YMQ-MTP-GGUF

Not produced by or affiliated with this repo. It is not built by me, I have not verified its files or its published numbers, and any metrics quoted there were not measured on the harness used here, so they are not comparable to the perplexity figures in this card. Linked because people ask for sizes I do not ship, not as an endorsement.

Usage

ComfyUI

Tested versions:

  • ComfyUI commit 0a33ed6c28f926d14536235771c222f9e6d1026b
  • ComfyUI QwenVL commit e79582111e5835787574bcc17adc4bf4cf3a07f4
  • llama cpp python version 0.3.48 at commit c57b174711166b889f06a35a30af95a75db24705
# Install ComfyUI and the QwenVL nodes
git clone https://github.com/comfyanonymous/ComfyUI.git
git -C ComfyUI checkout 0a33ed6c28f926d14536235771c222f9e6d1026b
git clone https://github.com/1038lab/ComfyUI-QwenVL.git ComfyUI/custom_nodes/ComfyUI-QwenVL
git -C ComfyUI/custom_nodes/ComfyUI-QwenVL checkout e79582111e5835787574bcc17adc4bf4cf3a07f4

# Install dependencies
uv pip install -r ComfyUI/requirements.txt
uv pip install -r ComfyUI/custom_nodes/ComfyUI-QwenVL/requirements.txt
CMAKE_ARGS="-DGGML_CUDA=on -DLLAMA_CPP_PYTHON_VISION=on" \
  uv pip install \
  "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git@c57b174711166b889f06a35a30af95a75db24705"

# Download the language model and F16 vision projector
hf download JonathanColetti/Qwen3.8-27B-Uncensored-GGUF \
  Qwen3.8-27B-Uncensored-Q4_K_M.gguf \
  mmproj-Qwen3.8-27B-Uncensored-F16.gguf \
  --local-dir \
  ComfyUI/models/LLM/GGUF/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF

# Start ComfyUI
cd ComfyUI
python main.py --listen 127.0.0.1 --port 8188

Add the model to ComfyUI/custom_nodes/ComfyUI-QwenVL/custom_models.json:

{
  "hf_models": {},
  "gguf_models": {
    "Qwen3.8 27B Uncensored Q4 K M": {
      "author": "JonathanColetti",
      "repo_name": "Qwen3.8-27B-Uncensored-GGUF",
      "repo_id": "JonathanColetti/Qwen3.8-27B-Uncensored-GGUF",
      "model_files": [
        "Qwen3.8-27B-Uncensored-Q4_K_M.gguf"
      ]
    }
  }
}

The projector is discovered automatically because its filename starts with mmproj.

llama cpp

llama-server -m Qwen3.8-27B-Uncensored-Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -ngl 99 -c 8192

Target plus explicit draft:

llama-server -m Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --model-draft Qwen3.8-27B-Uncensored-draft-Q8_0.gguf \
  -ngl 99 -c 8192

--spec-draft-n-max defaults to 3. Throughput depends on your hardware, so sweep it — measurements across draft lengths are in qwen3.8-spec-decode-bench.

Verification

Each artifact was checked post-quantization for MTP tensor survival rather than inferred from the conversion flag:

python quantize.py inspect Qwen3.8-27B-Uncensored-Q4_K_M.gguf

This reports metadata keys, declared block_count, and blocks actually present. A fused file whose present-block count does not exceed its declared count did not retain the MTP block.

FileMTPblocks
Qwen3.8-27B-Uncensored-f16.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-f16.ggufFalse64/64
Qwen3.8-27B-Uncensored-IQ2_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-IQ2_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-IQ4_XS.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-IQ4_XS.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q4_K_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q4_K_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q5_K_M.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q5_K_M.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q6_K.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q6_K.ggufFalse64/64
Qwen3.8-27B-Uncensored-Q8_0.ggufTrue65/65
Qwen3.8-27B-Uncensored-noMTP-Q8_0.ggufFalse64/64

Measured behaviour

Benchmarked against the unmodified base model on identical settings. The delta is the figure that matters: it isolates what the weight edit cost.

TaskBaseUncensoredΔ
MMLU83.483.3-0.2
ARC-Challenge58.957.7-1.2
HellaSwag82.882.9+0.1
Winogrande76.175.3-0.8
Mean-0.5

zero shot via lm-evaluation-harness, bf16, both models scored in the same session. Every delta is within or close to the reported standard error (MMLU +/- 0.30, ARC +/- 1.44, HellaSwag +/- 0.38, Winogrande +/- 1.21), so none is clearly separable from run-to-run noise.

These are 0-shot and are not comparable to Qwen's published scores, which use few-shot prompting. They are directly comparable to each other, which is the point. Note also that ARC-Challenge is low for a model at this MMLU. The base scores 58.9 under the same settings, so that is format sensitivity in a reasoning-tuned model, not abliteration damage.

What the benchmarks do not cover: no generative evaluation (GSM8K, HumanEval), no math or code, no multilingual, and the harness loads the text stack only. Nothing here measures the vision tower or MTP speculative decoding.

MeasurementBase modelThis model
Refusals (100 held-out harmful prompts)98/10012/100
KL divergence vs base (first-token)00.1191

Search: 200 Heretic trials, 23 non-dominated points. The published model is the marked row.

refusalsKL divergence
12/1000.1191<= published
13/1000.1052
19/1000.0722
23/1000.0635
26/1000.0507
27/1000.0410
35/1000.0406
36/1000.0387
41/1000.0366
44/1000.0352
46/1000.0334
48/1000.0331
51/1000.0321
52/1000.0294
60/1000.0290
76/1000.0280
77/1000.0247
83/1000.0204
86/1000.0193
91/1000.0170
96/1000.0146
97/1000.0044
98/1000.0004

How to read these

Refusal rate is the count of refusals over 100 heldout prompts from mlabonne/harmful_behaviors (test split) - explicitly harmful requests, not benign ones. So this number is not an over-refusal rate: it does not tell you how often the model declines legitimate work. It tells you how much of the original safety behaviour on harmful requests remains.

KL divergence is measured against the unmodified base model over first-token distributions, and is the optimizers proxy for "how much did we damage the model". Lower is closer to base. It is a proxy, not a capability measurement. A low KL does not certify that reasoning or coding ability survived, and nothing here does certify that.

The two trade off against each other. Heretic searches a Pareto front between them; the published point is one choice on that front, not a global optimum.

Caveats that matter

  • Refusals were measured in non-thinking mode. This models chat template opens a <think> block, so the evaluation closes it explicitly to score answers rather than reasoning traces. With thinking enabled the refusal rate may differ, in either direction.
  • The measurement is 100 prompts from one dataset. It generalizes to that distribution of harmful requests and no further. Refusal behaviour on other topics is uncharacterized.
  • Perplexity is wikitext-2 only (see the Files table). It detects gross quantization damage. It does not detect capability loss on reasoning, code, or multilingual work.
  • Quantization compounds everything above. The measurements were taken on the bf16 merge; the files you download are quantized.

Requirements

MTP speculative decoding landed in llama.cpp PR #22673. Builds older than that will load these files and silently ignore the MTP tensors.

Limitations

  • Refusals are reduced, not eliminated, and not redirected. This model attempts many requests the original declines, but a meaningful fraction still get refused — see Measured behaviour.
  • Behaviour near the old refusal boundary is less stable than the base model.
  • Lower quants compound that, and IQ2_M compounds it most. Evaluate behaviour on Q6_K or Q8_0, not on IQ2_M or IQ4_XS. Nothing here measures the refusal boundary at 2-bit.
  • Capability benchmarks show a 0.5-point mean drop vs base across MMLU, ARC-Challenge, HellaSwag and Winogrande. See Measured behaviour. No generative, math, code, or multilingual evaluation was run.

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B. The base model's license and acceptable use policy still apply to your use of this derivative.

Speculative decoding, measured on this model

promptspec_typen_maxtok/svs baseline
prosenone-74.81.00x
prosedraft-mtp189.01.19x
prosedraft-mtp285.71.15x
prosedraft-mtp372.00.96x
prosedraft-mtp471.10.95x
prosedraft-mtp562.90.84x
prosedraft-mtp653.80.72x
prosedraft-mtp749.90.67x
prosedraft-mtp859.90.80x
codenone-74.71.00x
codedraft-mtp195.41.28x
codedraft-mtp292.91.24x
codedraft-mtp382.61.11x
codedraft-mtp474.91.00x
codedraft-mtp567.40.90x
codedraft-mtp659.40.80x
codedraft-mtp755.60.74x
codedraft-mtp870.90.95x
chatnone-74.71.00x
chatdraft-mtp190.61.21x
chatdraft-mtp284.21.13x
chatdraft-mtp376.11.02x
chatdraft-mtp470.40.94x
chatdraft-mtp564.30.86x
chatdraft-mtp655.20.74x
chatdraft-mtp750.20.67x
chatdraft-mtp854.10.72x

IQ2_M

Measured on this build: NVIDIA H200 NVL, 256 generated tokens, median of 3 repetitions, n_max swept 1 to 3. The table above dates from the original release and its hardware is not recorded, so compare the ratios rather than the absolute rates.

promptspec_typen_maxtok/svs baseline
prosenone-75.41.00x
prosedraft-mtp185.21.13x
prosedraft-mtp283.21.10x
prosedraft-mtp377.41.03x
codenone-75.51.00x
codedraft-mtp195.81.27x
codedraft-mtp299.81.32x
codedraft-mtp396.01.27x
chatnone-75.31.00x
chatdraft-mtp187.41.16x
chatdraft-mtp281.31.08x
chatdraft-mtp378.41.04x

The MTP head survives 2-bit quantization because it is pinned to q8_0 rather than quantized with the rest of the stack, so speculative decoding still pays here.

Pairing noMTP-IQ2_M with the published draft-Q8_0 reaches 97.7 tok/s on the prose prompt at n_max 2, or 1.30x, which beats the fused file on that prompt. The reason is that the fused MTP head shares the low-bit token_embd (q4_K) and output (Q5_K) tensors with the main model, while the standalone draft carries its own Q8_0 copies. The split setup needs 13.3 GB of weights against 10.6 GB for the fused file, so it is the better option only if you have the VRAM to spare.

HUGGING FACE

上游数据

HF 下载量268.9万
HF 点赞量1125

HF 下载量为源站最近 30 天统计,与本站点击分开。

SITE STATS

本站统计

下载点击—
请求数—
匿名去重—最近 7 天—

本站下载量记录按钮点击,不代表下载完成;短时连点会去重。

来源与版本

作者
JonathanColetti
最近更新
2026/08/29
最新创建
2026/08/14
HF 趋势
9
数据快照
2026/09/16
版本
未知
许可证
apache-2.0