base_model: google/gemma-4-12B-it library_name: transformers license: gemma license_name: gemma-terms-of-use license_link: https://ai.google.dev/gemma/terms pipeline_tag: image-text-to-text language:

• en tags:
• abliterated
• uncensored
• gemma4
• refusal-removal
• weight-editing
• research

gemma4-12b-it-abliterated

A weight-edited derivative of google/gemma-4-12B-it with the refusal direction orthogonalised out of the residual writers.

This model has no refusal behaviour. Read Intended use and Security before loading it. If you need a model that declines requests, use the base model instead.

Model details

Method

1. Capture the last-token residual for 150 harmful (AdvBench-style) and 150 harmless (Alpaca) prompts at every layer.
2. d_l = normalize(mean(harmful) - mean(harmless)).
3. Orthogonalise the residual writers with a Gaussian profile centred on layers 24 and 28, sigma 6, alpha 1.0, embedding included.
4. W' = W - alpha * g(l) * d (d^T W), with g(l) = exp(-(l-24)^2 / 72) + exp(-(l-28)^2 / 72); for the embedding E' = E - alpha (E d) d^T.
5. Save bf16. No training, no fine-tuning.

The embedding edit is not optional: without it the direction is re-injected at every token and the rest of the edit barely registers.

Results

Scored by deepseek-flash on a three-way rubric (comply / soft-redirect / refuse) plus actionability 0-3. Identical prompts for both models.

Large benchmark, 769 harmful / 275 benign, built from JailbreakBench, XSTest and HarmfulQA, de-duplicated against the capture set:

Actionability on harmful: 0.75 → 1.82 of 3. On benign: 2.00 → 1.84.

AdvBench holdout (92 prompts, the set used for the ablation work):

Degenerate-output rate is identical to the base model on the same 92 harmless prompts (9/92), so the edit adds no coherence damage.

The base model complies with 2 of 92 AdvBench prompts but 246 of 769 on the large benchmark, so the large set is the harder test and the one to read.

Why alpha is small and the band is wide

A narrow band needs a very high alpha to remove the direction, and high alpha mangles the model. The wide band is what reaches both low refusals and intact coherence.

Generation

Gemma 4 ends a turn with <turn|> (token 106). It must be a stop token, or the model starts a second turn and visibly repeats itself.

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

mid = "pasangimhana/gemma4-12b-it-abliterated"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForImageTextToText.from_pretrained(
    mid, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Explain how a bill becomes law."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt").to(model.device)

stop = [tok.eos_token_id, tok.convert_tokens_to_ids("<turn|>")]
with torch.inference_mode():
    out = model.generate(**ids, max_new_tokens=512, do_sample=True,
                         temperature=0.7, eos_token_id=stop,
                         pad_token_id=tok.pad_token_id)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

The chat template opens an empty thought channel before generation. That is normal for this checkpoint and does not need to be disabled.

Intended use

Research on representation-level refusal: measuring how refusal is distributed across layers, how weight edits propagate through normalisation, and how evaluation instruments change the reported result.

Appropriate uses: interpretability work, red-team tooling that needs a model without refusals, synthetic-data generation where the base model's refusals are the obstacle.

Inappropriate uses: anything the Gemma Prohibited Use Policy forbids, any deployment where the absence of refusal is a liability, any setting where users have not been told the model is unfiltered.

Security

This model will comply with requests the base model refuses, including requests that are illegal or harmful. That is the point of the edit and it cannot be switched off at runtime.

• Do not expose it as a public endpoint without an external policy layer.
• Do not put it behind an agent with tool access and no human approval step.
• Output may contain instructions that are dangerous to follow. Treat generations as untrusted input, the same as any model output, but with a lower prior on self-restraint.
• Prompt injection is more consequential here: a model that does not refuse is a model that will follow an injected instruction it would otherwise decline.

No safety filtering is included, and none is planned.

Limitations

• The direction comes from 150 + 150 prompts from two corpora. A different distribution will move it. The benchmark is disjoint at the string level but not distributionally independent.
• 6 of 769 held-out harmful prompts still refuse. The ones inspected are requests a text model cannot literally satisfy, or ones where it answers while arguing the opposite premise.
• Actionability on benign prompts falls from 2.00 to 1.84. Cause not diagnosed.
• No capability benchmark (MMLU, GSM8K) was run. The coherence guard and the benign suites are the only evidence that general ability is intact.
• Judge calibration: 24 hand-labelled rows, Cohen's kappa 0.56 overall and 0.71 on the harmful slice. Substantial agreement where the headline is decided, poor agreement on benign prompts that cannot be answered literally.

Files

License and terms

Distributed under the Gemma Terms of Use, the same terms as the base model, including the Prohibited Use Policy.

This is an unaffiliated derivative. Not produced, endorsed or supported by Google. Removing refusal behaviour from the weights does not remove the obligations attached to the weights. If you redistribute this model, or a quantised or fine-tuned version of it, keep the notice and the terms with it.

Disclaimer

Provided as-is, without warranty of any kind, for research use. No fitness for any purpose is claimed. The author is not responsible for outputs, downstream fine-tunes, quantisations or deployments. If you need refusal behaviour, use the base model.

Citation

@misc{gimhana2026gemma4abl,
  title  = {Removing the refusal direction from Gemma 4 12B},
  author = {Pasan Gimhana},
  year   = {2026},
  note   = {Weight-space abliteration, calibrated three-way judge}
}