license: apache-2.0 base_model: huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated pipeline_tag: text-generation library_name: litert-lm tags:
- gemma-4
- gemma
- litert
- litert-lm
- abliterated
- edge-ai
- on-device
- conversion
Huihui Gemma 4 E4B QAT Q4_0 Unquantized Abliterated — LiteRT-LM
A LiteRT-LM conversion of
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.
This repository contains a ready-to-run .litertlm bundle converted from the
original Hugging Face BF16 Safetensors checkpoint for use with
Google LiteRT-LM.
This repository is not a new fine-tune and does not claim authorship of the original model weights. It is a runtime-format conversion of the Huihui model.
Model lineage
google/gemma-4-E4B
↓
google/gemma-4-E4B-it
↓
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
↓
BF16 Safetensors
↓
litert-torch export_hf
↓
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm
| Property | Value |
|---|---|
| Base model | huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated |
| Format | LiteRT-LM (.litertlm) |
| Architecture | Gemma 4 E4B |
| Quantization | dynamic_wi4_afp32 |
| Source model | QAT Q4 |
| Context length | 4096 tokens |
| Prefill lengths | 128, 256, 512 |
| Vision | ✅ Included |
| Audio | ❌ Audio encoder is not present in the resulting bundle |
| Tool calling metadata | ✅ |
| Thinking channel metadata | ✅ |
| Target | CPU / GPU edge inference |
About the source model
The source model was published by
huihui-ai as an abliterated variant of
google/gemma-4-E4B-it.
According to the original model card, abliteration significantly reduces the model's refusal behavior and safety filtering. This conversion does not restore or add safety alignment.
Please read the source model card before use:
Conversion
The model was converted from the original Hugging Face Safetensors checkpoint using the LiteRT Torch Hugging Face exporter.
The conversion command was:
litert-torch export_hf \
--model="$HOME/Models/huggingface/huihui-gemma-4-e4b-qat" \
--output_dir="$HOME/Models/litert/huihui-gemma-4-e4b-qat-full" \
--task=image_text_to_text \
--cache_length=4096 \
--prefill_lengths=128,256,512 \
--externalize_embedder=true \
--moe_exports_implementation=eager \
--export_vision_encoder=true \
--export_audio_encoder=true \
--quantization_recipe=dynamic_wi4_afp32 \
--jinja_chat_template_override=litert-community/gemma-4-E4B-it-litert-lm \
--bundle_litert_lm=true
The LiteRT-compatible Gemma 4 E4B chat template was taken from:
litert-community/gemma-4-E4B-it-litert-lm
The resulting bundle is:
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm
No GGUF or Ollama model was used as an intermediate format.
Download
Install the Hugging Face CLI if needed:
pip install -U huggingface_hub
Download only the LiteRT-LM bundle:
hf download \
vokash3/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated-LiteRT-LM \
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
--local-dir .
This produces:
./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm
Run with LiteRT-LM
Install the current LiteRT-LM CLI:
uv tool install litert-lm
Run the model:
litert-lm run \
./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
--prompt="Hello! Tell me briefly who you are."
For an interactive run:
litert-lm run ./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm
Available backends and CLI options can change as LiteRT-LM evolves, so check the installed version before using backend-specific flags:
litert-lm run --help
Benchmarks
The benchmark figures from the earlier non-QAT LiteRT-LM conversion are not
reused here because this artifact is produced from a different upstream
checkpoint and uses the dynamic_wi4_afp32 LiteRT export recipe.
This bundle should be benchmarked independently. For example:
litert-lm benchmark \
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
--backend=gpu
litert-lm benchmark \
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
--backend=cpu
The intended reference workload is 256 prefill tokens, 256 decode tokens and a 4096-token maximum context.
QAT and LiteRT quantization
The direct source checkpoint is the Huihui QAT/Q4_0 unquantized model. QAT is
part of the upstream model lineage; the LiteRT conversion then applies its own
runtime-oriented dynamic_wi4_afp32 recipe.
These are separate stages and should not be interpreted as the same quantization format.
Conversion environment
ai-edge-litert 2.2.0
ai-edge-litert-nightly 2.3.0.dev20260822
aiosqlite 0.22.1
litert-cli-nightly 0.2.0.dev20260824
litert-converter 0.5.0.dev20260822
litert-lm 0.16.1
litert-lm-api 0.16.1
litert-lm-builder 0.16.1
litert-torch-nightly 0.10.0.dev20260824
Several components are nightly builds, so conversion behavior and compatibility may change between releases.
Platforms
LiteRT-LM is designed for on-device inference across supported desktop and edge platforms. Consult the upstream project for the current platform/backend support matrix:
The .litertlm file in this repository was created as a portable LiteRT-LM
bundle rather than a GGUF model for llama.cpp/Ollama.
Google AI Edge Gallery / Edge Studio compatibility
The refreshed bundle is exported as an image_text_to_text model with the vision
encoder and vision adapter included.
The model has been successfully imported and initialized in Google AI Edge Gallery / Edge Studio, including AI Chat mode.
The working export uses:
--task=image_text_to_text
--export_vision_encoder=true
--export_audio_encoder=true
--prefill_lengths=128,256,512
Inspection of the generated bundle confirms the presence of the text inference components plus the Gemma 4 vision encoder and vision adapter.
Audio status
The export was requested with --export_audio_encoder=true, but inspection of
the resulting .litertlm bundle does not show a separate audio encoder.
Therefore this release should currently be treated as a text + vision
bundle; audio support is not claimed.
Benchmarks
Benchmark configuration:
Prefill tokens : 256
Decode tokens : 256
Context : 4096
| Device | Backend | Prefill, tok/s | Decode, tok/s | TTFT |
|---|---|---|---|---|
| Radeon RX 5700 XT | GPU | 926.31 | 47.82 | 0.297 s |
| Intel Core i5-11400 | CPU | 177.40 | 7.18 | 1.582 s |
| Mac mini M4 | GPU | 531.24 | 25.16 | 0.522 s |
| Mac mini M4 | CPU | 405.78 | 13.71 | 0.704 s |
| MacBook Air M1 | GPU | 273.03 | 13.80 | 1.010 s |
| MacBook Air M1 | CPU | 129.90 | 7.93 | 2.097 s |
| MacBook Neo A18 Pro | GPU | 239.40 | 14.51 | 1.138 s |
| MacBook Neo A18 Pro | CPU | 285.49 | 9.41 | 1.003 s |
| iPhone 16 Pro Max A18 Pro | GPU | 212.51 | 18.35 | 1.34 s |
| iPhone 16 Pro Max A18 Pro | CPU | 221.49 | 6.21 | 1.43 s |
| Tecno Camon 40 Pro 5G | GPU | 100.78 | 2.96 | 2.90 s |
| Tecno Camon 40 Pro 5G | CPU | 34.54 | 2.27 | 7.87 s |
Linux desktop benchmark
Test system:
OS: Linux Mint 22.1
CPU: Intel Core i5-11400
RAM: 32 GB
GPU: AMD Radeon RX 5700 XT
VRAM: 8 GB
Driver: radeonsi / Navi 10
Kernel: Linux 6.8
GPU
Prefill speed: 926.31 tokens/s
Decode speed: 47.82 tokens/s
Init time: 4.4219 s
Time to first token: 0.2973 s
CPU
Prefill speed: 177.40 tokens/s
Decode speed: 7.18 tokens/s
Init time: 1.5860 s
Time to first token: 1.5824 s
Apple Silicon
Mac mini M4 16 GB
| Backend | Prefill | Decode | TTFT |
|---|---|---|---|
| GPU | 531.24 tok/s | 25.16 tok/s | 0.522 s |
| CPU | 405.78 tok/s | 13.71 tok/s | 0.704 s |
MacBook Air M1 16 GB
| Backend | Prefill | Decode | TTFT |
|---|---|---|---|
| GPU | 273.03 tok/s | 13.80 tok/s | 1.010 s |
| CPU | 129.90 tok/s | 7.93 tok/s | 2.097 s |
MacBook Neo A18 Pro 8 GB
| Backend | Prefill | Decode | TTFT |
|---|---|---|---|
| GPU | 239.40 tok/s | 14.51 tok/s | 1.138 s |
| CPU | 285.49 tok/s | 9.41 tok/s | 1.003 s |
Mobile devices
Mobile benchmarks were performed using Google AI Edge Gallery.
iPhone 16 Pro Max — A18 Pro
| Backend | Prefill | Decode | TTFT |
|---|---|---|---|
| GPU | 212.51 tok/s | 18.35 tok/s | 1.34 s |
| CPU | 221.49 tok/s | 6.21 tok/s | 1.43 s |
Tecno Camon 40 Pro 5G — Dimensity 7300 Ultimate
| Backend | Prefill | Decode | TTFT |
|---|---|---|---|
| GPU | 100.78 tok/s | 2.96 tok/s | 2.90 s |
| CPU | 34.54 tok/s | 2.27 tok/s | 7.87 s |
Notes
The Radeon RX 5700 XT provides the highest measured performance among the tested devices:
Prefill: 926.31 tokens/s
Decode: 47.82 tokens/s
The model also runs successfully on mobile devices through Google AI Edge Gallery.
Performance depends heavily on the LiteRT backend, GPU drivers, available memory, operating system, and LiteRT-LM version. The benchmark numbers above should therefore be treated as practical reference measurements rather than universally reproducible performance figures.
Important notes and limitations
- This is a conversion, not a separately trained model.
- Behavioral characteristics of the source Huihui model remain relevant.
- The source model has substantially reduced refusal/safety behavior.
- Output can include sensitive, controversial, unsafe, inaccurate or otherwise undesirable content.
- Users should review outputs before using them in production or public-facing applications.
- Runtime behavior may differ from Transformers or GGUF builds because of differences in quantization, execution backend, sampling, prompt formatting and runtime implementation.
- Exact token-for-token parity with the original Safetensors checkpoint is not expected.
- This repository does not provide independent benchmark or quality claims.
- The conversion is intended for LiteRT-LM multimodal chat / text-and-vision use.
License and upstream terms
The source Huihui repository is published on Hugging Face with the Apache-2.0 license metadata.
Users should also review the terms, notices and usage requirements associated with the upstream Gemma model family and the original Huihui model before redistribution or deployment.
Upstream resources:
- Huihui source model: https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
- Google Gemma 4 E4B IT: https://huggingface.co/google/gemma-4-E4B-it
- LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM
- LiteRT Torch: https://github.com/google-ai-edge/litert-torch
- LiteRT Community Gemma 4 E4B: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm
Credits
This repository is a runtime-format conversion and packaging project built on top of several upstream models, tools and community resources.
Google / Google DeepMind
Google / Google DeepMind created the Gemma model family and the Gemma 4
architecture underlying this model. The instruction-tuned upstream family is
represented by google/gemma-4-E4B-it.
Google also develops the LiteRT ecosystem used for on-device execution.
huihui-ai
huihui-ai created and published the direct source checkpoint:
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
The QAT/Q4_0 and abliterated characteristics originate from the upstream Huihui model. This repository does not claim authorship of those weights, training changes or behavioral modifications.
LiteRT-LM
Google AI Edge LiteRT-LM provides the runtime and .litertlm bundle format
used by this repository. It provides the infrastructure for packaging model
components, tokenizer data, metadata, KV-cache configuration and executable
LiteRT/TFLite components for local inference.
LiteRT Torch
LiteRT Torch provides the Hugging Face export pipeline used to create this
artifact. The conversion uses litert-torch export_hf with the
dynamic_wi4_afp32 recipe, externalized embeddings, Gemma 4 MoE export
settings, multiple prefill lengths and multimodal export options.
LiteRT Community
LiteRT Community published the Gemma 4 E4B LiteRT-LM reference repository used for the compatible Jinja chat template:
litert-community/gemma-4-E4B-it-litert-lm
vokash3
vokash3 performed and published the LiteRT-LM conversion in this repository, including source-checkpoint preparation, export configuration experiments, selection of the LiteRT quantization recipe, multimodal packaging, bundle inspection, LiteRT-LM compatibility testing, Google AI Edge Gallery testing, documentation of the current audio limitation, and publication of the resulting artifact.
No authorship of the original Gemma or Huihui weights is claimed.
Upstream resources
- Google Gemma 4 E4B IT: https://huggingface.co/google/gemma-4-E4B-it
- Huihui source model: https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
- LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM
- LiteRT Torch: https://github.com/google-ai-edge/litert-torch
- LiteRT Community Gemma 4 E4B: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm
Repository contents
.
├── README.md
└── Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm
Reproducibility summary
Source:
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
Source format:
QAT Q4_0 unquantized Hugging Face checkpoint
Exporter:
litert-torch export_hf
Options:
--task=image_text_to_text
--cache_length=4096
--prefill_lengths=128,256,512
--externalize_embedder=true
--moe_exports_implementation=eager
--export_vision_encoder=true
--export_audio_encoder=true
--quantization_recipe=dynamic_wi4_afp32
--jinja_chat_template_override=litert-community/gemma-4-E4B-it-litert-lm
--bundle_litert_lm=true
Output:
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm