license: gemma library_name: transformers pipeline_tag: text-generation base_model: AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16 model_type: gemma4_unified language:

• en tags:
• 12b
• 4-bit
• aarch64
• abliterated
• aeon
• aeon-7
• agentic
• arm64
• biprojection
• blackwell
• chat
• chunked-prefill
• coding
• conversational
• dgx-spark
• english
• flashinfer-cutlass
• fp4
• fp8
• fp8-kv-cache
• function-calling
• gb10
• gemma
• gemma-4
• gemma-4-12B
• gemma4
• gemma4_unified
• google
• gpu
• grace-blackwell
• instruct
• k4-biprojection
• long-context
• mixed-precision
• modelopt
• multimodal
• native-fp4
• nvfp4
• nvidia
• openai-api
• openai-compatible
• prefix-caching
• production-ready
• quantized
• reasoning
• refusal-removed
• safetensors
• sm_121a
• text-generation
• thinking
• tool-calling
• uncensored
• unfiltered
• vision
• vision-language
• vllm
• weight-quantization

Gemma-4-12B-it AEON Abliterated — K=4 Biprojection (Mixed NVFP4 + FP8)

The smallest and fastest variant — 9.3 GB, 21 tok/s single-stream / 318 tok/s concurrent on a DGX Spark. A mixed-precision quantization of our K=4 biprojection abliteration of google/gemma-4-12B-it: 4-bit NVFP4 on the MLP weights, 8-bit FP8 on the attention. Delivers the NVFP4-MLP-only sibling's reasoning quality (MMLU 76.8) at 20% smaller size and 34% faster single-stream, with stronger coding. Loads in vLLM with --quantization modelopt.

This is the maximum-density / maximum-throughput pick. When peak reasoning quality matters, use the near-lossless FP8 sibling (13 GB).

Refusal behavior has been removed; the model responds to a wide range of prompts the base would decline. Operator-side safety is your responsibility — see the arbitration clause at the bottom.

🚀 QuickStart

Docker (recommended, DGX Spark / Blackwell)

Complete copy-paste recipe — pull the container, pull the model, serve. Plain decode: no speculative drafter (MTP is net-neutral on the GB10 for this 12B), so there is no drafter to download and no --speculative-config.

# 1. Pull the unified AEON vLLM container (v0.24.0, sm_121a)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Download the model (fresh)
huggingface-cli download AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4-FP8 \
  --local-dir ./Gemma-4-12B-AEON-K4-NVFP4-FP8

# 3. (no drafter — plain decode)

# 4. Serve
docker run -d --name aeon-gemma12b --gpus all --ipc=host --shm-size=16g --net=host \
  -v $(pwd)/Gemma-4-12B-AEON-K4-NVFP4-FP8:/model:ro \
  --entrypoint vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate:latest \
  serve /model \
    --served-model-name gemma12b \
    --quantization modelopt \
    --attention-backend triton_attn \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --enable-auto-tool-choice \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 8192 \
    --max-num-seqs 16 \
    --gpu-memory-utilization 0.70 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code

# 5. Call (OpenAI-compatible)
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma12b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":128}' \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

Plain vLLM

pip install "vllm>=0.23.0" "nvidia-modelopt>=0.43" "transformers>=5.10"
vllm serve AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4-FP8 \
  --quantization modelopt --attention-backend triton_attn \
  --reasoning-parser gemma4 --tool-call-parser gemma4 --enable-auto-tool-choice \
  --enable-chunked-prefill --enable-prefix-caching \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 8192 --max-num-seqs 16 \
  --gpu-memory-utilization 0.70 --trust-remote-code

DGX Spark unified-memory note. On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6–0.7; above ~0.8 the shared CPU+GPU pool page-thrashes. Discrete-VRAM GPUs can run higher.

⚠️ Needs vLLM ≥ 0.23.0 (for the Gemma4UnifiedForConditionalGeneration loader and the MIXED_PRECISION modelopt path) and a Blackwell GPU (DGX Spark GB10 sm_121a, B100/B200 sm_100, RTX 50-series sm_120) for the NVFP4 GEMM. The AEON vLLM Ultimate container ships the loader pre-built for sm_121a. On Hopper/Ampere the FP4 weights dequantize to BF16 (no speed benefit) — use the FP8 sibling instead.

That's it. Everything below is detail.

Performance — DGX Spark (v0.24.0, aeon-vllm-ultimate:latest)

Fastest of the 12B K4 family. The mixed NVFP4(MLP) + FP8(attn) recipe gives the smallest footprint (9.3 GB) and the highest throughput of any usable quant of this model: ~21 tok/s single-stream and ~917 tok/s aggregate at c=64 on ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.24.0, GB10 / sm_121a).

This is a dense 12B run with plain decode — no speculative drafter. MTP/EAGLE speculation is net-neutral on the GB10 (the drafter forward + the 262k-vocab lm_head + multi-position verify cost roughly cancel the acceptance gain on this hardware), so the production config runs vanilla decode and leans on concurrency for aggregate throughput. The strength here is twofold: the quant-vs-speed tradeoff (mixed wins the family on both size and speed) and clean concurrency scaling to c=64.

Single-stream (c=1) by category

Decode is essentially flat across categories (~21 tok/s) — expected for a dense model with no drafter, where single-stream is memory-bandwidth-bound. The 9.3 GB footprint is what drives the single-stream lead over the other quants. DFlash acceptance is N/A: this config runs plain decode (no speculative drafter), per the MTP-net-neutral measurement on GB10.

Aggregate throughput by concurrency

Throughput scales cleanly all the way to c=64, where the mixed variant peaks at ~917 tok/s — the best aggregate of any 12B K4 quant (see the family chart: FP8 747, NVFP4 794, BF16 458 at c=64). The DFlash high-concurrency fix in this image is what lets the engine reach c=64 without crashing.

Long-context draft-acceptance is not reported for this card — it runs plain decode (no drafter), so there is no acceptance curve to track across context length.

What we fixed for the DGX Spark

All AEON models now run on a single unified container — ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-07-01-v0.24.0; rollback :2026-06-18-v0.23.0-dflashfix). It is vLLM v0.24.0 built from source for sm_121a merged with the AEON speculative-decoding stack, tuned end-to-end for the GB10's unified-memory Blackwell architecture. The two changes that matter most for this card:

• DFlash high-concurrency fix. The speculative drafter previously crashed at ≥32 concurrent requests (padded vs. unpadded KV block-table shape mismatch in FlashAttention). The fix slices the drafter's KV block-table to the unpadded batch (block_table[:num_reqs]) — a port of upstream PR #43982, which fixed this for MTP but never for DFlash. The engine now scales cleanly to c=64 for every model in the fleet, including the plain-decode runs like this one.
• Unified container + native sm_121a build. One image (vLLM 0.24.0, TORCH_CUDA_ARCH_LIST=12.1a) ships the Gemma4UnifiedForConditionalGeneration loader and the SM120-family CUTLASS NVFP4/FP8 kernels GB10 dispatches to, with sm_121a boot + CUDA-graph patches and conservative unified-memory KV headroom.

Stock baseline pending fresh vanilla re-bench. No same-harness stock/vanilla baseline exists for the Gemma-4-12B cards yet; the numbers above are all measured on the optimized aeon-vllm-ultimate:latest (vLLM 0.24.0) build. A fully-vanilla comparison run will be added when complete.

Why mixed precision — measured capability

The full-FP4 (W4A4) NVFP4 quant costs ~21pp on hard reasoning because the FP4 activations perturb precise multi-step logit propagation. This model sidesteps that: the bulk MLP weights go 4-bit NVFP4 (where most of the size lives), while the reasoning-sensitive attention stays at 8-bit FP8. The result keeps the 4-bit size/speed advantage without the W4A4 reasoning collapse.

All axes evaluated through the vLLM serving path, identical prompts/settings for every model. MMLU is the balanced 285-question set (5 × all 57 subjects) — a diverse measure, not the worst-case single-subject slice.

vs the NVFP4-MLP-only sibling, this model matches MMLU and IFEval, and is better on coding (HumanEval functional +5.5pp) — at 20% smaller size. It does not reach FP8's reasoning (the shared NVFP4 MLP is the bottleneck); the FP8 sibling remains the quality pick.

Throughput (DGX Spark GB10, FP8 KV cache, greedy)

Fastest variant by a clear margin — +34% single-stream and +25–41% concurrent vs the other usable quants, at the smallest footprint. On a memory-bandwidth-bound box like the Spark, the 9.3 GB footprint is what drives the single-stream win.

Quantization methodology

vLLM dispatches each layer to its own kernel: FlashInferCutlassNvFp4LinearKernel for the MLP, FlashInferFP8ScaledMMLinearKernel for the attention.

Kept at full BF16

lm_head, model.language_model.embed_tokens, model.embed_vision*, model.embed_audio*, model.vision_embedder*. (A small number of attention projections that ModelOpt's calibration left unquantized also remain BF16 — higher precision, no downside.)

vLLM loader notes (for reproducers)

1. Google's Gemma-4-12B is the encoder-free Gemma4UnifiedForConditionalGeneration. ModelOpt's HF export needs two touch-ups to load in vLLM: rename the vision keys to vLLM's vision_embedder.* layout, and add model.vision_embedder* to the quant ignore list. Both are scripted in make_vllm_ready.py (gemma4-nvfp4/).
2. The attention layers must use per-tensor FP8 (FP8_DEFAULT_CFG), not FP8_PER_CHANNEL_PER_TOKEN. vLLM's ModelOptMixedPrecisionConfig only routes per-layer quant_algo ∈ {FP8, NVFP4, W4A16_NVFP4}; the per-channel/per-token variant exports as FP8_PER_CHANNEL_PER_TOKEN, which falls through to the unquantized path and fails to load. The full mixed recipe is in quantize_k4_nvfp4.py --recipe mixed_mlp_nvfp4_attn_fp8.

Abliteration methodology (inherited from the BF16 base)

K=4 multi-direction norm-preserving biprojection (extends TrevorJS's recipe). Basis layers L24/L37/L39/L26 (top-K by SNR), o_proj + mlp.down_proj edited on 24/48 layers, scale=1.0. See the BF16 card for the full biprojection math + capability comparison vs base.

Behavior

• Benign prompts: matches the NVFP4-MLP-only sibling (the capability table confirms it numerically).
• Previously-refused prompts: full responses, usually after a brief disclaimer paragraph.
• Tool calling via --enable-auto-tool-choice --tool-call-parser gemma4.
• Multimodal vision path preserved (BF16).
• KV cache: use --kv-cache-dtype fp8_e4m3 (the published default). Do not combine NVFP4 KV cache with speculative decoding — 4-bit KV collapses draft acceptance.

Available formats

Acknowledgements

TrevorJS (biprojection), p-e-w/heretic (abliteration framework), NVIDIA ModelOpt (NVFP4/FP8 toolkit + Gemma-4 reference recipes), AEON-7 (K-direction extension, mixed-precision recipe + vLLM loader fixes, capability eval).

License

Inherits the Gemma license.

Arbitration Clause

By accessing, downloading, using, running inference on, fine-tuning, merging, quantizing, distributing, integrating, or otherwise interacting with this model, you acknowledge and agree to the following:

1. Sole Responsibility. You, the user, are solely and exclusively responsible for (a) every prompt you or your downstream system issue to this model, (b) every response this model produces in reply, (c) every downstream action taken by you, your systems, your agents, or your users in reliance on those responses, and (d) any harm — direct, indirect, consequential, foreseeable, or otherwise — that results from any of the above.

2. No Warranty. This model is provided strictly "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, non-infringement, safety, alignment, factual accuracy, or legal compliance in any jurisdiction. No contributor, author, publisher, or hosting platform assumes liability of any kind for outputs or downstream use.

3. Legal Compliance. You are responsible for ensuring that your use of this model complies with all applicable laws, regulations, terms of service, industry codes of conduct, professional ethical standards, and organizational policies in every jurisdiction in which you operate or in which your outputs may be received. The unaligned nature of this model does not grant you any legal authorization you did not already have.

4. Operational Safety Layer. An uncensored model is not a toy. You are expected to implement appropriate downstream safety layers proportionate to your deployment context, including but not limited to: input validation, output filtering, content moderation, audit logging, rate limiting, access controls, and human-in-the-loop review for high-risk workflows. A production deployment of this model without such layers is unsafe by construction and is not a supported use case.

5. Heightened Duty of Care. The absence of internal refusal behavior means the duty of care that would ordinarily rest partly with the model rests entirely with you. You are expected to exercise greater — not lesser — caution, forethought, and ethical discipline when operating this model than you would operate a base aligned model. If you are uncertain whether your contemplated use is ethical, legal, or wise, the correct action is to not make the request.

6. No Endorsement of Outputs. The authors, contributors, and publishers of this model do not endorse, adopt, or take responsibility for any specific output this model produces. Outputs are a stochastic function of the prompt, the weights, and the sampler state — not a statement of position by any human.

7. Arbitration. Any dispute, claim, or controversy arising out of or relating to the use of this model, its outputs, or this clause shall be resolved through binding individual arbitration under the rules of a mutually agreed arbitration body (or, absent agreement, the American Arbitration Association's Consumer Arbitration Rules), waiving any right to a jury trial, class action, representative action, or consolidated proceeding. Venue shall be the jurisdiction of the disputing party bringing the claim. Costs and attorneys' fees shall be allocated per the applicable arbitration rules. This clause does not expand, and where legally prohibited does not establish, any liability in the other direction; it limits how the user may proceed when alleging harm tied to their own use of this model.

8. Indemnification. You agree to indemnify, defend, and hold harmless the authors, contributors, and publishers of this model from and against any claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from or related to your use of the model or your breach of this clause.

9. Severability. If any provision of this clause is held unenforceable in a given jurisdiction, the remaining provisions remain in full force in that jurisdiction, and the unenforceable provision is replaced by the closest enforceable equivalent consistent with the original intent.

10. Acceptance. Your use of this model constitutes your acceptance of this clause in full. If you do not accept, do not use the model.

This model is a tool with no opinions of its own. You supply the opinions. You supply the judgement. You supply the ethics. The outputs carry your fingerprints, not the model's.

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.