license: apache-2.0 base_model: orcarouter/Gemma-4-26B-A4B-it-Uncensored-FP8 base_model_relation: quantized library_name: mlx pipeline_tag: text-generation language:

• en tags:
• mlx
• apple-silicon
• gemma
• gemma-4
• mixture-of-experts
• abliterated
• uncensored
• 4-bit
• function-calling
• reasoning

Gemma-4-26B-A4B-it-Uncensored-MLX-4bit

MLX 4-bit build of orcarouter/Gemma-4-26B-A4B-it-Uncensored-FP8 for Apple Silicon.

orcarouter ships MLX builds for the rest of its lineup but released this model only as FP8, which has no practical serving path on Metal. This is that missing build. It is a format conversion, not a retrain: no re-abliteration, no merging, no fine-tuning. The abliteration is orcarouter's, unchanged.

The FP8 checkpoint cannot be read by mlx_lm.convert, and two further problems were hiding behind that one. All three are documented under Conversion, with the scripts in this repo.

Model details

Evaluation

Measured on this build with mlx-lm 0.31.3 on an Apple M1 Max (64 GB), and on the vanilla gemma-4-26B-A4B-it 4-bit MLX build under identical settings and seed, so the deltas isolate the abliteration plus the conversion path. Refusal is judged the way orcarouter's card does it — a rule-based opening-phrase classifier, three classes (refusal / caveat = answered behind a disclaimer / answer) — indicative, not LLM-judge grade. Generation was capped at 48 tokens for classification; harmful completions were not stored.

Safety — harmful-prompt refusal (lower = more uncensored)

Cells are refusal % / caveat %. The vanilla column is the sanity check: it must refuse the harmful sets or the classifier is meaningless (orcarouter reports ~97% on JailbreakBench for the base). For reference, orcarouter reports the FP8 source at 8.0 / 32.0 (AdvBench) and 4.0 / 24.0 (JailbreakBench) under vLLM.

Over-refusal — benign prompts wrongly refused (lower = better)

Capability retention — vs the vanilla 4-bit base (same runtime, same settings, same seed)

In progress. MMLU / ARC-Challenge / GSM8K-CoT / IFEval are running now on all three builds under the settings below; this table and the performance row will be filled in and the repo made public when they finish. Until then treat this build as refusal-validated, capability-unverified.

orcarouter's own card reports their FP8 build at −4.0 (MMLU) / −0.7 (GSM8K) / −4.0 (IFEval prompt-level) against the bf16 base under vLLM — a different baseline and runtime, so not directly comparable, but the same order of cost. Not run: MMLU-Pro and CMMLU. lm-eval 0.4.13 via mlx_lm.evaluate; subsets are the first n examples under seed 1234, identical for both builds.

Performance on Apple Silicon

Apple M1 Max, 64 GB, iogpu.wired_limit_mb=57344, mlx_lm.server, ~12k-token prompt, cold prefill (--prompt-cache-size 0), 256 output tokens, median of 3.

Prefill on Apple Silicon is compute-bound; what matters is active parameters (~3.8B), not total. Enable the prompt cache in production — these figures deliberately measure cold prefill.

Usage

pip install -U mlx-lm

python -m mlx_lm.chat --model 00cyre/Gemma-4-26B-A4B-it-Uncensored-MLX-4bit

# OpenAI-compatible server
python -m mlx_lm.server --model 00cyre/Gemma-4-26B-A4B-it-Uncensored-MLX-4bit \
  --max-tokens 8192 --prompt-cache-size 8

Per-request thinking toggle through the server — the model returns reasoning and content separately:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "00cyre/Gemma-4-26B-A4B-it-Uncensored-MLX-4bit",
  "messages": [{"role": "user", "content": "In one sentence, what are you?"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

Tool calling uses the standard OpenAI tools array; mlx_lm.server parses the model's tool tokens into tool_calls. Works as a backend for OpenAI-compatible agent harnesses.

Conversion

mlx_lm.convert fails on the FP8 source. Three separate problems, each found by a check that refused to default:

1. mlx_lm treats every compressed-tensors checkpoint as 4-bit affine. The container carries int4, int8 and fp8; the loader lets float-quantized (FP8) fall through to the 4-bit branch, reinterpreting F8_E4M3 bytes as packed ints. Reported as ml-explore/mlx-lm#1865. Fix: conversion/dequant_fp8.py — weight.float() * weight_scale (static per-output-channel [out, 1] bf16 scales; activations are dynamic with nothing stored) to a bf16 checkpoint. It raises on any fp8 tensor with no scale, which surfaced that shard boundaries split weight/scale pairs; scales are collected in a first pass over all shards.
2. llm-compressor unstacks the experts. To quantize per expert it writes experts.<e>.{gate,up,down}_proj — 11,520 tensors (30 × 128 × 3) that mlx_lm's Gemma-4 sanitize does not recognise; it expects Google's fused experts.gate_up_proj / experts.down_proj. Fix: conversion/restack_experts.py — restack to [128, 1408, 2816] / [128, 2816, 704], shapes pinned from the vanilla MLX build's quantization scales, with a round-trip assertion on the split the loader performs.
3. router.per_expert_scale was dropped on export. One [128] bf16 tensor per layer, missing from the FP8 checkpoint. It is not ones — 0.9805 to 1.0234 in every layer of Google's base — while router.scale is bit-identical between the base and the export, so abliteration never touched the router. Fix: conversion/restore_per_expert_scale.py — restore Google's values from the vanilla MLX build (stored unquantized, lossless). A transformers/vLLM load of the FP8 export silently initialises this tensor to ones; this build restores the trained values on the reasoning that it recovers the model that was actually abliterated. To use ones instead, delete the restored tensors before quantizing.

Then: python -m mlx_lm convert --hf-path <bf16-stacked> --mlx-path <out> --quantize --q-bits 4 --q-group-size 64. Full log in conversion/mlx_lm_convert.log.

Validation beyond exit code: every layer-0 expert and router tensor matches the vanilla MLX build in dtype and shape (14/14, including the restored [128]); the keys absent vs vanilla are exactly the vision tower the loader drops; loads in 3.2 s; coherent generation; tool calling and thinking termination verified through the server.

Intended use and limitations

Refusal behaviour has been substantially removed. It will comply with requests a safety-aligned model declines. You are responsible for what you generate with it. Intended for research, and for deployments that supply their own safety layer.

• Text-only. Use the FP8 source under vLLM for images.
• Abliteration is not free — see the capability table.
• One garbled token was observed in an 80-token sample during validation and has not been characterised.
• The refusal classifier is rule-based; treat those percentages as indicative.

License

apache-2.0, matching Google's gemma-4-26B-A4B-it card. orcarouter's FP8 release declares the Gemma Terms of Use and Prohibited Use Policy; if you rely on that release, comply with those.

Credits

• orcarouter — the abliteration this build carries unchanged
• Google — Gemma 4
• ml-explore/mlx, mlx-lm, lm-eval