license: apache-2.0 base_model:

• jialinyyzz/humanizer
• google/gemma-4-12B pipeline_tag: text-generation library_name: transformers tags:
• quantization
• fp8
• int8
• compressed-tensors
• w8a8
• abliteration
• uncensored
• gemma

humanizer-abliterated — FP8 (E4M3, weight-only + dynamic activations)

8-bit FP8-DYNAMIC of the 11.96B abliterated rewrite model — 54% of the BF16 original, language-model linears only.

Status: structurally verified and runnable, not benchmarked. See Verification for exactly what was measured and what was not.

What this is

An 8-bit compressed-tensors quantization of distributedcognition/humanizer-abliterated, produced with llmcompressor.model_free_ptq directly from the BF16 safetensors. The result is an ordinary Hugging Face model directory whose linear weights are stored in 8-bit with compressed-tensors metadata in config.json, so it loads in vLLM, SGLang, or Transformers without a conversion step.

Quantization scope

Only linear layers inside the language-model decoder were quantized. Everything else stays BF16:

• lm_head and model.embed_tokens — the input embedding and the output projection.
• The vision and audio towers (re:.*vision.*, re:.*audio.*) — this is a text-only pipeline.

Of 677 source tensors, 328 linear weights were quantized.

Keeping the embedding at BF16 is not cosmetic. tie_word_embeddings=True, so the embedding table is the output projection — quantizing it quantizes the logits. The failure mode, measured directly, is silent: the weights load without error, the file is a plausible size, and the model then returns the same handful of low-ID tokens no matter what you feed it.

Each quantized layer stores a per-output-channel scale in weight_scale.

Usage

vLLM

vllm serve moeshawky/humanizer-abliterated-FP8-DYNAMIC

SGLang

python -m sglang.launch_server --model-path moeshawky/humanizer-abliterated-FP8-DYNAMIC

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "moeshawky/humanizer-abliterated-FP8-DYNAMIC"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="auto")

Prompting. This is not a chat model. It ships prompt_format.json holding the instruction and separator it was tuned on: render instr + draft + sep, where sep is "\n\n### Rewritten:\n\n". Passing a chat-formatted prompt degrades output quality.

Verification

Measured:

• Weight fidelity. FP8 ≈ 31.6 dB and INT8 ≈ 39.9 dB SNR against the BF16 source on dequantized attention and MLP projections.
• Output distribution and generation. Last-position logits over the 262,144-token vocabulary have σ = 4.8125 for FP8 and 4.6875 for INT8, against 4.8125 for the BF16 source, with an identical top-5 continuation and coherent greedy output.
• Tensor census. 328 quantized tensors and 328 scales, with all embeddings, lm_head, and the vision/audio towers confirmed BF16 by header inspection.
• Schemas. config.json declares quant_method=compressed-tensors, format=float-quantized, weights 8-bit / num_bits: 8 / channel strategy / symmetric, input activations 8-bit dynamic per-token.

Not measured — do not quote this repo for any of these:

• No perplexity, KL divergence, or task accuracy. The source ships MMLU artifacts; these quantized checkpoints were not re-scored against them. A quantization that preserves fluency can still be worse on tasks, and nothing here rules that out.
• No throughput or latency. INT8 W8A8 speed depends on the serving kernel, not the artifact.
• No ablation-robustness test. These models are refusal-removed; quantization may partially restore refusal. Untested here.

Provenance

model_free_ptq uses no calibration dataset: weight scales come from the weights themselves and activations are quantized at runtime. The artifact is therefore dataset-independent.

License and obligations

Upstream is apache-2.0. That is this artifact's license, but the base models (jialinyyzz/humanizer and google/gemma-4-12B) carry their own terms, including regional access limits on the Qwen line. If you redistribute or serve this checkpoint, you inherit those obligations; read them before deploying.