license: apache-2.0 base_model: llmfan46/gemma-4-31B-it-uncensored-heretic base_model_relation: quantized tags:

• exl3
• exllamav3
• gemma4
• heretic
• uncensored
• vision inference: false

gemma-4-31B-it-uncensored-heretic EXL3 3.00bpw h4

EXL3 export of llmfan46/gemma-4-31B-it-uncensored-heretic for ExLlamaV3 / TabbyAPI. Vision tower included, quantized to 6 bpw.

Load with ExLlamaV3 or TabbyAPI's exllamav3 loader. This will not load in Transformers, vLLM or llama.cpp.

This is the 4-bit-head companion to -h6. Same 3.0 bpw weights; the head and the vision tower are smaller. See Differences from h6.

Quantization

From quantization_config.json:

830 language-model modules at 3 bpw, lm_head at 4 bits, 354 vision modules at 6 bpw. embed_tokens is carried unchanged in BF16 (2.625 GiB) — ExLlamaV3 keeps it in system RAM rather than VRAM.

config.json carries an audio_config, but the checkpoint has no audio tensors — neither does the source or google/gemma-4-31B-it. This is an image+text model.

Converted on an RTX 4090 on 2026-09-19 from the BF16 source (62.5 GB).

Differences from h6

The language-model layers are the same size in both, 10.245 GiB — same 3.0 bpw over the same shapes. Only the head and the vision tower changed size.

The vision difference is not a choice: ExLlamaV3 1.4.2 could not quantize Gemma 4's vision tower at all, so h6 stores it at BF16. 1.5.0 can.

Size

Component breakdown, summed from the safetensors headers:

Measured VRAM

Vision disabled, cache_mode: 3,3, real prefills at ~95% of the window — not just successful loads.

RTX 4090, 24 GB (chunk_size: 128, max_batch_size: 2):

262,144 was the top of the sweep, not a limit — 3.3 GB was still free.

RTX 5080, 16 GB (chunk_size: 256, max_batch_size: 1):

These are single-prefill verifications. A window that survives one clean prefill can still OOM under sustained traffic, so run below the number in the table.

Only 10 of Gemma 4's 60 layers hold a per-token K/V cache — the other 50 are sliding-window with fixed-size state — so at 3,3 context costs about 17.9 KiB/token, far less than the layer count suggests. On a card near its limit the prefill working set, not the cache, is what binds.

Vision

The tower costs 454 MiB of VRAM. Setting EXL3_VISION_PINNED=1 holds it in pinned host memory instead, cutting that to 54 MiB (measured at 61,440 on a 5080). Nothing else offloads — that flag applies to the vision component only.

Serving

TabbyAPI, as run here on a 4090:

max_seq_len: 98304
cache_size: 98304
cache_mode: 3,3
chunk_size: 128
gpu_split_auto: true
prompt_template: gemma4

chat_template.jinja is a separate file — Gemma 4 does not put the template in tokenizer_config.json.

Quality

Not evaluated. The numbers above are memory and context measurements; no benchmark or side-by-side against h6 has been run. 3.0 bpw with a 4-bit head is an aggressive quant chosen to fit long context on one card, and a 4-bit head is a further reduction on top of an already lossy export. If you have the VRAM, prefer a higher-bpw build.

Credit and license

Apache 2.0, matching google/gemma-4-31B-it and the source model.

Decensoring credit belongs to llmfan46 — this repo is only the EXL3 export. Quantization by ExLlamaV3.