zaakirio /

gemma-4-12b-it-uncensored-GGUF

zaakirio/gemma-4-12b-it-uncensored-GGUF is an uncensored open-source model hosted on Hugging Face.

README.md说明 / 使用方法

base_model: zaakirio/gemma-4-12b-it-uncensored license: gemma pipeline_tag: image-text-to-text tags:

gguf
llama.cpp
gemma
gemma4
heretic
abliterated
uncensored
decensored
conversational
multimodal
audio

gemma-4-12b-it-uncensored — GGUF

zaakirio/gemma-4-12b-it-uncensored 的 GGUF 量化版本，该模型是基于 google/gemma-4-12B-it 去除审查（通过 Heretic 消融定向特征）的去审查版本。

这些文件适用于 llama.cpp 运行。

Gemma 4 12B（2026年6月的“Unified”统一发布版）是一款**免编码器的统一多模态（encoder-free unified multimodal）**模型：文本、图像、音频和视频均直接投影到单个纯解码器 Transformer 中，上下文窗口最高达 256K tokens。消融（Abliteration）仅作用于语言权重，因此这些多模态能力得以完整保留。

文件列表 (Files)

文件名遵循 gemma-4-12b-it-uncensored-<QUANT>.gguf 规范。

量化版本	大小	说明
Q2_K	4.50 GB	体积最小；质量最低。仅在内存极其受限时使用。
Q3_K_M	5.67 GB	体积小；适合低内存环境。
Q4_K_S	6.54 GB	紧凑型 4-bit。
Q4_K_M	6.87 GB	推荐 — 体积与质量的最佳平衡。
Q5_K_M	7.96 GB	质量更高，体积略大。
Q6_K	9.11 GB	接近无损质量。
Q8_0	11.80 GB	相较于 BF16 源模型基本无损。
f16	22.20 GB	全精度；供参考或重新量化使用。

不知道如何选择？建议优先选用 Q4_K_M。 如果内存充足且追求最高保真度，可升级到 Q5/Q6/Q8；仅在显存或内存严重受限时才降级到 Q3/Q2。

多模态投影器（用于图像/音频输入 — 参见多模态）：

文件	大小	说明
`mmproj-gemma-4-12B-it-bf16.gguf`	0.16 GB	多模态投影器 — 可与上述任意量化版本搭配使用。
`mtp-gemma-4-12b-it-uncensored.gguf`	0.47 GB	可选的 MTP 投机解码草稿模型（drafter） — 在 GPU 上提供更快的生成速度。详见借助 MTP 加速生成。

使用方法 (Usage)

llama.cpp（自动从本仓库下载所选量化文件）：

# 交互式对话
llama-cli -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

# 带有 Web 界面的兼容 OpenAI 接口服务
llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja -c 4096

-c 设置上下文窗口长度。Gemma 4 12B 最高支持 256K tokens（-c 262144）；在内存允许的情况下，处理长对话/长文档时可以调高该值 — 4096 仅为一个轻量级默认值。

或者使用已下载到本地的文件：

llama-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf --jinja -p "Hello, who are you?"

下载单个文件：

pip install -U "huggingface_hub[cli]"
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" --local-dir ./

提示词格式与配置 (Prompt format & settings)

对话模板已内嵌在 GGUF 文件中 — 支持聊天模板的工具会自动应用它（在 llama.cpp 中请始终加上 --jinja）。作为参考，Gemma 4 引入了新的回合标记符 — <|turn> / <turn|>，替代了 Gemma 3 的 <start_of_turn> / <end_of_turn> — 并且现已支持专用的 system 角色：

<|turn>system
{系统指令（可选）}<turn|>
<|turn>user
{提示词}<turn|>
<|turn>model

图像和音频输入使用 <|image|> 和 <|audio|> 占位符标记，多模态流水线会自动进行填充（参见多模态）。

推荐采样参数（Google 默认值）：--temp 1.0 --top-p 0.95 --top-k 64。

思考模式（Thinking mode）： Gemma 4 具备推理通道。如需禁用，请向 llama-server 传递参数 --chat-template-kwargs '{"enable_thinking":false}'。

多模态 (Multimodal)

在 llama.cpp 中，多模态输入通过单独的投影器文件提供；语言模型 .gguf 本身仅支持文本，直接输入多媒体数据会被拒绝。本仓库包含 mmproj-gemma-4-12B-it-bf16.gguf，它同时包含视觉和音频编码器（clip.has_vision_encoder 与 clip.has_audio_encoder）。

实际体验： 图像和音频输入均能正常工作。 已使用 llama-mtmd-cli 进行验证：它可直接接受 .wav/.mp3，并对真实语音生成了准确的多语言转录文本。llama.cpp 将音频标记为实验性功能（"audio input is in experimental stage and may have reduced quality"），因此请将 ASR 质量视为尽力而为（best-effort）。视频则被处理为采样视频帧（加音频），取决于客户端是否完成视频抽帧。

当通过 -hf 加载时，llama.cpp 会自动从本仓库下载投影器 — 图像即可直接工作：

llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

对于本地文件，请使用 --mmproj 显式指定：

llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja

# 下载这两个文件
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mmproj-gemma-4-12B-it-bf16.gguf" --local-dir ./

在命令行中使用 llama-mtmd-cli 进行单次图像或音频推理：

# 图像
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --image photo.png -p "What's in this image?"

# 音频（在 llama.cpp 中为实验性功能；推荐 16 kHz WAV）
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --audio clip.wav -p "Transcribe this audio."

该投影器为原版未修改的 Gemma 4 投影器（消融去审查并未对其进行改动）。推荐使用 bf16 — 其体积足够小，进行量化收益微乎其微。

借助 MTP（投机解码）加速生成

Gemma 4 提供了一个轻量级（约 0.4B）多标记预测（MTP, multi-token-prediction）草稿模型（google/gemma-4-12B-it-assistant），通过自投机解码加速生成：草稿模型一次性预测多个 token，主模型在单次前向传递中验证它们。输出结果与常规解码完全一致 — 投机解码在数学上是严格等价的，因此它仅改变速度，绝不会影响生成质量，模型也不会因为加速而降低无审查程度：主模型会验证并决定每个输出的 token；草稿模型仅负责提议。（Google 官方也强调了这一点 — MTP“保证与标准自回归生成完全相同的质量” — llama.cpp 的实现 PR 也通过完全复现 Gemma 官方公布的 AIME-26 成绩证实了这一点。）

出于同样的原因，草稿模型不需要进行去审查消融：本仓库提供了 Google 未修改的官方上游 Gemma 4 草稿模型（文件重命名为 mtp-gemma-4-12b-it-uncensored.gguf 仅为便于与主干模型并列存放 — 权重完全来自官方上游），你的去审查主干模型会验证每个 token。与上述多模态投影器一样，它独立于对齐调整。

Gemma 4 的 MTP 支持已合并入 llama.cpp PR #23398（合并于 2026-06-07；亦可参阅支持讨论 #22735），因此需要使用该日期或之后的 llama.cpp 版本。

# 获取主干模型与 MTP 草稿模型
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mtp-gemma-4-12b-it-uncensored.gguf" --local-dir ./

# 启用 MTP 运行服务
llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --model-draft mtp-gemma-4-12b-it-uncensored.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 99 --flash-attn auto --jinja

⚠️ MTP 在 GPU 上有显著提升，但在 Apple Silicon 上反而更慢

性能收益完全取决于所使用的计算后端。

在 GPU 上有明显提升。 实现 PR 在稠密版 Gemma 4 (31B) 模型上测得超过 2 倍的吞吐量（NVIDIA DGX Spark 环境，在 --spec-draft-n-max 4 下平均草稿接受率约为 0.59），Unsloth 也报告使用 MTP 的 Gemma 4 吞吐量提升了 1.5–2.2 倍。（PR #23398，Unsloth MTP 指南。）

在 Apple Silicon (Metal) 上性能出现倒退。 在当前模型上的实际测试结果如下 — Q4_K_M，Apple M4 Pro，采用 Gemma 默认采样参数（--temp 1.0 --top-p 0.95 --top-k 64），通过 llama-server 测得：

M4 Pro / Metal · Q4_K_M	生成速度	草稿接受率
MTP 关闭（基准）	~27 tok/s	—
MTP 开启 (`--spec-draft-n-max 4`)	~12 tok/s（约慢 2 倍）	~0.45

草稿 token 确实被接受了（~45%），但在 Metal 后端上，评估草稿模型的额外开销超过了其减少的解码步数所带来的收益。这并非该模型特有的问题：一份独立评测报告（#23752 — M1 Max 上的 Qwen3.5-9B）也发现在所有设置下 MTP 都更慢，即使接受率达到 100% 也是如此，因此在 Metal 上调整采样参数无法挽救该开销。

使用建议： 在 NVIDIA (CUDA) GPU 上部署时建议开启 MTP；在 Mac 上请保持关闭。在 GPU 上，采用贪心解码或低温采样会进一步提高接受率并扩大加速比。

关于基础模型 (About the base model)

使用 Heretic（自动方向消融技术）制作的去审查衍生版本。与原版对比：

指标	去审查版本	原版
拒绝回答次数 (/100 个有害提示词)	23	99
KL 散度 (无害提示词)	0.043	0 （按定义）

这里的拒绝次数是基于 Heretic 的关键词启发式统计得出的，已知该统计方法存在过量统计倾向（会将带有免责声明但实际执行的合规回答标记为拒绝回答；根据 arXiv:2512.13655，其精准度约为 11%）。我们仅报告实际测得的标记数值，并未在本模型上运行基于分类器的完整评测，因此实际合规回答比例很可能更高。关于具体参数与详细说明，请参阅源模型卡片。

预期用途与免责声明 (Intended use & disclaimer)

该模型大幅去除了原有的拒绝行为，将遵循原版会拒绝的指令。模型仅供学术研究和不受限的本地使用。用户对其使用方式以及遵守相关法律法规负全部责任，同时须遵守适用于本衍生模型的基础模型 Gemma 许可协议。本模型不适合所有受众。

来源信息 (Provenance)

量化来源：使用 llama.cpp convert_hf_to_gguf.py + llama-quantize 对 zaakirio/gemma-4-12b-it-uncensored (BF16) 进行量化。
基础模型：google/gemma-4-12B-it
去审查工具：p-e-w 开发的 Heretic · 技术原理：Arditi et al. (2024)
MTP 草稿模型（mtp-gemma-4-12b-it-uncensored.gguf）：未经修改的官方上游 google/gemma-4-12B-it-assistant (~0.4 B)，通过 unsloth/gemma-4-12b-it-GGUF 提供的 Q8_0 GGUF 版本。未经消融 — 其仅负责提议 token；去审查主干模型会对每个 token 进行验证，因此生成结果与单独运行该模型完全一致。

源站原文

base_model: zaakirio/gemma-4-12b-it-uncensored license: gemma pipeline_tag: image-text-to-text tags:

gguf
llama.cpp
gemma
gemma4
heretic
abliterated
uncensored
decensored
conversational
multimodal
audio

gemma-4-12b-it-uncensored — GGUF

GGUF quantizations of zaakirio/gemma-4-12b-it-uncensored, a decensored (Heretic-abliterated) version of google/gemma-4-12B-it.

These files run with llama.cpp.

Gemma 4 12B (the "Unified" release, June 2026) is an encoder-free unified multimodal model: text, image, audio, and video all project straight into a single decoder-only transformer, with a context window of up to 256K tokens. Abliteration touches only the language weights, so those capabilities carry over unchanged.

Files

Filenames follow gemma-4-12b-it-uncensored-<QUANT>.gguf.

Quant	Size	Notes
Q2_K	4.50 GB	Smallest; lowest quality. Very tight memory only.
Q3_K_M	5.67 GB	Small; usable on low RAM.
Q4_K_S	6.54 GB	Compact 4-bit.
Q4_K_M	6.87 GB	Recommended — best size/quality balance.
Q5_K_M	7.96 GB	Higher quality, slightly larger.
Q6_K	9.11 GB	Near-lossless.
Q8_0	11.80 GB	Effectively lossless vs the BF16 source.
f16	22.20 GB	Full precision; reference / re-quantizing.

Not sure which to pick? Start with Q4_K_M. Go up to Q5/Q6/Q8 if you have the memory and want maximum fidelity; drop to Q3/Q2 only if you're memory-constrained.

Multimodal projector (for image/audio input — see Multimodal):

File	Size	Notes
`mmproj-gemma-4-12B-it-bf16.gguf`	0.16 GB	Multimodal projector — pair with any quant above.
`mtp-gemma-4-12b-it-uncensored.gguf`	0.47 GB	Optional MTP speculative-decoding drafter — faster generation on GPUs. See Faster generation with MTP.

Usage

llama.cpp (auto-downloads the chosen quant from this repo):

# Interactive chat
llama-cli -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

# OpenAI-compatible server with web UI
llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja -c 4096

-c sets the context window. Gemma 4 12B supports up to 256K tokens (-c 262144); raise it for long chats/documents as memory allows — 4096 is just a light default.

Or with a file you've already downloaded:

llama-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf --jinja -p "Hello, who are you?"

Download a single file:

pip install -U "huggingface_hub[cli]"
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" --local-dir ./

Prompt format & settings

The chat template is embedded in the GGUF — chat-aware tools apply it automatically (always pass --jinja with llama.cpp). For reference, Gemma 4 introduced new turn tokens — <|turn> / <turn|>, replacing Gemma 3's <start_of_turn> / <end_of_turn> — and now supports a dedicated system role:

<|turn>system
{system instructions — optional}<turn|>
<|turn>user
{prompt}<turn|>
<|turn>model

Image and audio inputs use the <|image|> and <|audio|> placeholder tokens, which the multimodal pipeline fills in for you (see Multimodal).

Recommended sampling (Google defaults): --temp 1.0 --top-p 0.95 --top-k 64.

Thinking mode: Gemma 4 has a reasoning channel. To disable it, pass --chat-template-kwargs '{"enable_thinking":false}' to llama-server.

Multimodal

In llama.cpp, multimodal input ships via a separate projector file; the language .gguf alone is text-only and will reject media. This repo includes mmproj-gemma-4-12B-it-bf16.gguf, which carries both a vision and an audio encoder (clip.has_vision_encoder and clip.has_audio_encoder).

In practice: image and audio input both work. Verified with llama-mtmd-cli: it accepts .wav/.mp3 directly and produced an accurate multilingual transcription of real speech. llama.cpp flags audio as experimental ("audio input is in experimental stage and may have reduced quality"), so treat ASR quality as best-effort. Video is handled as sampled frames (plus audio) and depends on your client doing the frame extraction.

When you load via -hf, llama.cpp auto-downloads the projector from this repo — images just work:

llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

With local files, pass it explicitly with --mmproj:

llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja

# download both files
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mmproj-gemma-4-12B-it-bf16.gguf" --local-dir ./

One-shot image or audio from the CLI with llama-mtmd-cli:

# Image
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --image photo.png -p "What's in this image?"

# Audio (experimental in llama.cpp; 16 kHz WAV)
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --audio clip.wav -p "Transcribe this audio."

The projector is the unmodified Gemma 4 one (abliteration leaves it untouched). bf16 is preferred — it's small enough that quantizing it buys nothing.

Faster generation with MTP (speculative decoding)

Gemma 4 ships a small (~0.4 B) multi-token-prediction (MTP) drafter (google/gemma-4-12B-it-assistant) that speeds up generation through self-speculative decoding: the drafter proposes several tokens at once and the full model verifies them in a single forward pass. Output is identical to normal decoding — speculative decoding is mathematically exact, so it changes speed only, never quality, and the model stays exactly as uncensored: the main model verifies and decides every emitted token; the drafter merely proposes. (Google state the same — MTP "guarantee[s] the exact same quality as standard autoregressive generation" — and the implementation PR confirmed it by reproducing Gemma's published AIME-26 score.)

For that same reason the drafter does not need to be abliterated: this repo ships Google's unmodified upstream Gemma 4 drafter (the file is renamed mtp-gemma-4-12b-it-uncensored.gguf only so it sits beside the trunk — the weights are upstream), and your decensored trunk verifies every token. It's alignment-independent, exactly like the multimodal projector above.

MTP for Gemma 4 landed in llama.cpp PR #23398 (merged 2026-06-07; see also support discussion #22735), so it needs a llama.cpp build from that date or later.

# Grab the trunk + the MTP drafter
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mtp-gemma-4-12b-it-uncensored.gguf" --local-dir ./

# Serve with MTP enabled
llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --model-draft mtp-gemma-4-12b-it-uncensored.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 99 --flash-attn auto --jinja

⚠️ MTP helps on GPUs but is slower on Apple Silicon

The benefit depends entirely on the backend.

On a GPU it's a win. The implementation PR measured >2× throughput on the dense Gemma 4 (31B) model — NVIDIA DGX Spark, ~0.59 mean draft acceptance at --spec-draft-n-max 4 — and Unsloth report 1.5–2.2× for Gemma 4 with MTP. (PR #23398, Unsloth MTP guide.)

On Apple Silicon (Metal) it's a regression. Measured here on this model — Q4_K_M, Apple M4 Pro, Gemma's default sampling (--temp 1.0 --top-p 0.95 --top-k 64), through llama-server:

M4 Pro / Metal · Q4_K_M	Generation	Draft acceptance
MTP off (baseline)	~27 tok/s	—
MTP on (`--spec-draft-n-max 4`)	~12 tok/s (≈2× slower)	~0.45

The drafts are accepted (~45%), but on Metal the draft-evaluation overhead outweighs the decode steps it saves. This isn't specific to this model: an independent report (#23752 — Qwen3.5-9B on an M1 Max) found MTP slower at every setting, even at 100% acceptance, so no sampling tweak rescues it on Metal.

Recommendation: enable MTP when serving on an NVIDIA (CUDA) GPU; leave it off on a Mac. On a GPU, greedy / low-temperature sampling raises acceptance and widens the speedup.

About the base model

A decensored derivative produced with Heretic (automatic directional ablation). Compared with the original:

Metric	Decensored	Original
Refusals (/100 harmful prompts)	23	99
KL divergence (harmless prompts)	0.043	0 (by definition)

The refusal count is Heretic's keyword heuristic, which is known to over-count (it flags disclaimer-wrapped compliance as a refusal; ~11% precision per arXiv:2512.13655). We report only the measured marker figure and did not run a classifier-based eval on this model, so real compliance is likely higher. See the source model card for parameters and details.

Intended use & disclaimer

This model has had its refusal behaviour substantially removed and will comply with requests the original would have declined. Provided for research and unrestricted local use. You are responsible for how you use it and for complying with applicable law and the base model's Gemma license, which carries over to this derivative. Not for all audiences.

Provenance

Quantized from zaakirio/gemma-4-12b-it-uncensored (BF16) using llama.cpp convert_hf_to_gguf.py + llama-quantize.
Base model: google/gemma-4-12B-it
Decensoring tool: Heretic by p-e-w · technique: Arditi et al. (2024)
MTP drafter (mtp-gemma-4-12b-it-uncensored.gguf): the unmodified upstream google/gemma-4-12B-it-assistant (~0.4 B), Q8_0 GGUF via unsloth/gemma-4-12b-it-GGUF. Not abliterated — it only proposes tokens; the decensored trunk verifies every one, so output is identical to running this model alone.

返回模型目录

zaakirio /

gemma-4-12b-it-uncensored-GGUF

HF 源站

zaakirio/gemma-4-12b-it-uncensored-GGUF is an uncensored open-source model hosted on Hugging Face.

Image-Text-to-Text llama.cpp GGUF License: gemma 12B gemma4 heretic abliterated uncensored decensored conversational multimodal audio

README.md说明 / 使用方法

base_model: zaakirio/gemma-4-12b-it-uncensored license: gemma pipeline_tag: image-text-to-text tags:

gguf
llama.cpp
gemma
gemma4
heretic
abliterated
uncensored
decensored
conversational
multimodal
audio

gemma-4-12b-it-uncensored — GGUF

zaakirio/gemma-4-12b-it-uncensored 的 GGUF 量化版本，该模型是基于 google/gemma-4-12B-it 去除审查（通过 Heretic 消融定向特征）的去审查版本。

这些文件适用于 llama.cpp 运行。

文件列表 (Files)

文件名遵循 gemma-4-12b-it-uncensored-<QUANT>.gguf 规范。

量化版本	大小	说明
Q2_K	4.50 GB	体积最小；质量最低。仅在内存极其受限时使用。
Q3_K_M	5.67 GB	体积小；适合低内存环境。
Q4_K_S	6.54 GB	紧凑型 4-bit。
Q4_K_M	6.87 GB	推荐 — 体积与质量的最佳平衡。
Q5_K_M	7.96 GB	质量更高，体积略大。
Q6_K	9.11 GB	接近无损质量。
Q8_0	11.80 GB	相较于 BF16 源模型基本无损。
f16	22.20 GB	全精度；供参考或重新量化使用。

不知道如何选择？建议优先选用 Q4_K_M。 如果内存充足且追求最高保真度，可升级到 Q5/Q6/Q8；仅在显存或内存严重受限时才降级到 Q3/Q2。

多模态投影器（用于图像/音频输入 — 参见多模态）：

文件	大小	说明
`mmproj-gemma-4-12B-it-bf16.gguf`	0.16 GB	多模态投影器 — 可与上述任意量化版本搭配使用。
`mtp-gemma-4-12b-it-uncensored.gguf`	0.47 GB	可选的 MTP 投机解码草稿模型（drafter） — 在 GPU 上提供更快的生成速度。详见借助 MTP 加速生成。

使用方法 (Usage)

llama.cpp（自动从本仓库下载所选量化文件）：

# 交互式对话
llama-cli -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

# 带有 Web 界面的兼容 OpenAI 接口服务
llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja -c 4096

或者使用已下载到本地的文件：

llama-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf --jinja -p "Hello, who are you?"

下载单个文件：

pip install -U "huggingface_hub[cli]"
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" --local-dir ./

提示词格式与配置 (Prompt format & settings)

<|turn>system
{系统指令（可选）}<turn|>
<|turn>user
{提示词}<turn|>
<|turn>model

图像和音频输入使用 <|image|> 和 <|audio|> 占位符标记，多模态流水线会自动进行填充（参见多模态）。

推荐采样参数（Google 默认值）：--temp 1.0 --top-p 0.95 --top-k 64。

思考模式（Thinking mode）： Gemma 4 具备推理通道。如需禁用，请向 llama-server 传递参数 --chat-template-kwargs '{"enable_thinking":false}'。

多模态 (Multimodal)

实际体验： 图像和音频输入均能正常工作。 已使用 llama-mtmd-cli 进行验证：它可直接接受 .wav/.mp3，并对真实语音生成了准确的多语言转录文本。llama.cpp 将音频标记为实验性功能（"audio input is in experimental stage and may have reduced quality"），因此请将 ASR 质量视为尽力而为（best-effort）。视频则被处理为采样视频帧（加音频），取决于客户端是否完成视频抽帧。

当通过 -hf 加载时，llama.cpp 会自动从本仓库下载投影器 — 图像即可直接工作：

llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

对于本地文件，请使用 --mmproj 显式指定：

llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja

# 下载这两个文件
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mmproj-gemma-4-12B-it-bf16.gguf" --local-dir ./

在命令行中使用 llama-mtmd-cli 进行单次图像或音频推理：

# 图像
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --image photo.png -p "What's in this image?"

# 音频（在 llama.cpp 中为实验性功能；推荐 16 kHz WAV）
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --audio clip.wav -p "Transcribe this audio."

该投影器为原版未修改的 Gemma 4 投影器（消融去审查并未对其进行改动）。推荐使用 bf16 — 其体积足够小，进行量化收益微乎其微。

借助 MTP（投机解码）加速生成

Gemma 4 的 MTP 支持已合并入 llama.cpp PR #23398（合并于 2026-06-07；亦可参阅支持讨论 #22735），因此需要使用该日期或之后的 llama.cpp 版本。

# 获取主干模型与 MTP 草稿模型
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mtp-gemma-4-12b-it-uncensored.gguf" --local-dir ./

# 启用 MTP 运行服务
llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --model-draft mtp-gemma-4-12b-it-uncensored.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 99 --flash-attn auto --jinja

⚠️ MTP 在 GPU 上有显著提升，但在 Apple Silicon 上反而更慢

性能收益完全取决于所使用的计算后端。

M4 Pro / Metal · Q4_K_M	生成速度	草稿接受率
MTP 关闭（基准）	~27 tok/s	—
MTP 开启 (`--spec-draft-n-max 4`)	~12 tok/s（约慢 2 倍）	~0.45

使用建议： 在 NVIDIA (CUDA) GPU 上部署时建议开启 MTP；在 Mac 上请保持关闭。在 GPU 上，采用贪心解码或低温采样会进一步提高接受率并扩大加速比。

关于基础模型 (About the base model)

使用 Heretic（自动方向消融技术）制作的去审查衍生版本。与原版对比：

指标	去审查版本	原版
拒绝回答次数 (/100 个有害提示词)	23	99
KL 散度 (无害提示词)	0.043	0 （按定义）

这里的拒绝次数是基于 Heretic 的关键词启发式统计得出的，已知该统计方法存在过量统计倾向（会将带有免责声明但实际执行的合规回答标记为拒绝回答；根据 arXiv:2512.13655，其精准度约为 11%）。我们仅报告实际测得的标记数值，并未在本模型上运行基于分类器的完整评测，因此实际合规回答比例很可能更高。关于具体参数与详细说明，请参阅源模型卡片。

预期用途与免责声明 (Intended use & disclaimer)

来源信息 (Provenance)

量化来源：使用 llama.cpp convert_hf_to_gguf.py + llama-quantize 对 zaakirio/gemma-4-12b-it-uncensored (BF16) 进行量化。
基础模型：google/gemma-4-12B-it
去审查工具：p-e-w 开发的 Heretic · 技术原理：Arditi et al. (2024)
MTP 草稿模型（mtp-gemma-4-12b-it-uncensored.gguf）：未经修改的官方上游 google/gemma-4-12B-it-assistant (~0.4 B)，通过 unsloth/gemma-4-12b-it-GGUF 提供的 Q8_0 GGUF 版本。未经消融 — 其仅负责提议 token；去审查主干模型会对每个 token 进行验证，因此生成结果与单独运行该模型完全一致。

源站原文

base_model: zaakirio/gemma-4-12b-it-uncensored license: gemma pipeline_tag: image-text-to-text tags:

gguf
llama.cpp
gemma
gemma4
heretic
abliterated
uncensored
decensored
conversational
multimodal
audio

gemma-4-12b-it-uncensored — GGUF

GGUF quantizations of zaakirio/gemma-4-12b-it-uncensored, a decensored (Heretic-abliterated) version of google/gemma-4-12B-it.

These files run with llama.cpp.

Files

Filenames follow gemma-4-12b-it-uncensored-<QUANT>.gguf.

Quant	Size	Notes
Q2_K	4.50 GB	Smallest; lowest quality. Very tight memory only.
Q3_K_M	5.67 GB	Small; usable on low RAM.
Q4_K_S	6.54 GB	Compact 4-bit.
Q4_K_M	6.87 GB	Recommended — best size/quality balance.
Q5_K_M	7.96 GB	Higher quality, slightly larger.
Q6_K	9.11 GB	Near-lossless.
Q8_0	11.80 GB	Effectively lossless vs the BF16 source.
f16	22.20 GB	Full precision; reference / re-quantizing.

Not sure which to pick? Start with Q4_K_M. Go up to Q5/Q6/Q8 if you have the memory and want maximum fidelity; drop to Q3/Q2 only if you're memory-constrained.

Multimodal projector (for image/audio input — see Multimodal):

File	Size	Notes
`mmproj-gemma-4-12B-it-bf16.gguf`	0.16 GB	Multimodal projector — pair with any quant above.
`mtp-gemma-4-12b-it-uncensored.gguf`	0.47 GB	Optional MTP speculative-decoding drafter — faster generation on GPUs. See Faster generation with MTP.

Usage

llama.cpp (auto-downloads the chosen quant from this repo):

# Interactive chat
llama-cli -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

# OpenAI-compatible server with web UI
llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja -c 4096

-c sets the context window. Gemma 4 12B supports up to 256K tokens (-c 262144); raise it for long chats/documents as memory allows — 4096 is just a light default.

Or with a file you've already downloaded:

llama-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf --jinja -p "Hello, who are you?"

Download a single file:

pip install -U "huggingface_hub[cli]"
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" --local-dir ./

Prompt format & settings

<|turn>system
{system instructions — optional}<turn|>
<|turn>user
{prompt}<turn|>
<|turn>model

Image and audio inputs use the <|image|> and <|audio|> placeholder tokens, which the multimodal pipeline fills in for you (see Multimodal).

Recommended sampling (Google defaults): --temp 1.0 --top-p 0.95 --top-k 64.

Thinking mode: Gemma 4 has a reasoning channel. To disable it, pass --chat-template-kwargs '{"enable_thinking":false}' to llama-server.

Multimodal

In practice: image and audio input both work. Verified with llama-mtmd-cli: it accepts .wav/.mp3 directly and produced an accurate multilingual transcription of real speech. llama.cpp flags audio as experimental ("audio input is in experimental stage and may have reduced quality"), so treat ASR quality as best-effort. Video is handled as sampled frames (plus audio) and depends on your client doing the frame extraction.

When you load via -hf, llama.cpp auto-downloads the projector from this repo — images just work:

llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

With local files, pass it explicitly with --mmproj:

llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja

# download both files
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mmproj-gemma-4-12B-it-bf16.gguf" --local-dir ./

One-shot image or audio from the CLI with llama-mtmd-cli:

# Image
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --image photo.png -p "What's in this image?"

# Audio (experimental in llama.cpp; 16 kHz WAV)
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --audio clip.wav -p "Transcribe this audio."

The projector is the unmodified Gemma 4 one (abliteration leaves it untouched). bf16 is preferred — it's small enough that quantizing it buys nothing.

Faster generation with MTP (speculative decoding)

MTP for Gemma 4 landed in llama.cpp PR #23398 (merged 2026-06-07; see also support discussion #22735), so it needs a llama.cpp build from that date or later.

# Grab the trunk + the MTP drafter
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mtp-gemma-4-12b-it-uncensored.gguf" --local-dir ./

# Serve with MTP enabled
llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --model-draft mtp-gemma-4-12b-it-uncensored.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 99 --flash-attn auto --jinja

⚠️ MTP helps on GPUs but is slower on Apple Silicon

The benefit depends entirely on the backend.

On Apple Silicon (Metal) it's a regression. Measured here on this model — Q4_K_M, Apple M4 Pro, Gemma's default sampling (--temp 1.0 --top-p 0.95 --top-k 64), through llama-server:

M4 Pro / Metal · Q4_K_M	Generation	Draft acceptance
MTP off (baseline)	~27 tok/s	—
MTP on (`--spec-draft-n-max 4`)	~12 tok/s (≈2× slower)	~0.45

Recommendation: enable MTP when serving on an NVIDIA (CUDA) GPU; leave it off on a Mac. On a GPU, greedy / low-temperature sampling raises acceptance and widens the speedup.

About the base model

A decensored derivative produced with Heretic (automatic directional ablation). Compared with the original:

Metric	Decensored	Original
Refusals (/100 harmful prompts)	23	99
KL divergence (harmless prompts)	0.043	0 (by definition)

The refusal count is Heretic's keyword heuristic, which is known to over-count (it flags disclaimer-wrapped compliance as a refusal; ~11% precision per arXiv:2512.13655). We report only the measured marker figure and did not run a classifier-based eval on this model, so real compliance is likely higher. See the source model card for parameters and details.

Intended use & disclaimer

Provenance

Quantized from zaakirio/gemma-4-12b-it-uncensored (BF16) using llama.cpp convert_hf_to_gguf.py + llama-quantize.
Base model: google/gemma-4-12B-it
Decensoring tool: Heretic by p-e-w · technique: Arditi et al. (2024)
MTP drafter (mtp-gemma-4-12b-it-uncensored.gguf): the unmodified upstream google/gemma-4-12B-it-assistant (~0.4 B), Q8_0 GGUF via unsloth/gemma-4-12b-it-GGUF. Not abliterated — it only proposes tokens; the decensored trunk verifies every one, so output is identical to running this model alone.