license: apache-2.0 base_model:

• Altworld/Hemmingway-1
• Qwen/Qwen3.8-27B pipeline_tag: text-generation library_name: transformers tags:
• quantization
• fp8
• int8
• compressed-tensors
• w8a8
• abliteration
• uncensored
• qwen

Hemmingway-1-27B-abliterated — INT8 W8A8 (dynamic)

8-bit INT8-W8A8 of the 27.32B abliterated rewrite model — 54% of the BF16 original, language-model linears only.

Status: structurally verified and runnable, not benchmarked. See Verification for exactly what was measured and what was not.

What this is

An 8-bit compressed-tensors quantization of distributedcognition/Hemmingway-1-27B-abliterated, produced with llmcompressor.model_free_ptq directly from the BF16 safetensors. The result is an ordinary Hugging Face model directory whose linear weights are stored in 8-bit with compressed-tensors metadata in config.json, so it loads in vLLM, SGLang, or Transformers without a conversion step.

Quantization scope

Only linear layers inside the language-model decoder were quantized. Everything else stays BF16:

• lm_head and model.embed_tokens.
• The MTP (multi-token-prediction) head — all 15 mtp.* tensors including its 8 linear weights, so speculative decoding still runs, at BF16.
• The Gated DeltaNet state tensors: conv1d (4D depthwise), A_log, dt_bias.

Why the state tensors are excluded: conv1d is a depthwise kernel that 8-bit channel-wise quantization mangles badly — measured at 1.0 max relative error against its own magnitude. A_log and dt_bias are 48-element decay vectors where absolute precision is the whole value.

Of 866 source tensors, 496 linear weights were quantized: attention projections across 48 Gated DeltaNet and 16 full-attention layers, plus MLP gate/up/down in all 64 layers.

Keeping the embedding at BF16 is not cosmetic. tie_word_embeddings=False, so lm_head and the embedding are separate tensors and both stay at full precision. The failure mode, measured directly, is silent: the weights load without error, the file is a plausible size, and the model then returns the same handful of low-ID tokens no matter what you feed it.

Each quantized layer stores a per-output-channel scale in weight_scale.

Usage

vLLM

vllm serve moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8

SGLang

python -m sglang.launch_server --model-path moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="auto")

Hybrid attention. 48 of 64 layers use Gated DeltaNet linear attention and 16 use full attention. the recurrent state stays BF16, so long-context behaviour should match the source; the quantized part is the projections around it.

Verification

Measured:

• Weight fidelity. FP8 ≈ 31.6 dB and INT8 ≈ 39.6–40.8 dB SNR against the BF16 source across sampled self-attention, linear-attention, and MLP projections.
• Output distribution and generation. Last-position logits σ = 2.0469 (FP8) and 2.0781 (INT8) against 2.0625 for the BF16 source; near-identical top-5 and a word-for-word identical opening sentence (the wind carried the smell of salt and rust) before natural argmax divergence.
• Tensor census. 496 quantized tensors and 496 scales across 13 shards, with lm_head, embed_tokens, all 15 mtp.* tensors, and all SSM state tensors confirmed BF16.
• Schemas. config.json declares quant_method=compressed-tensors, format=int-quantized, weights 8-bit / num_bits: 8 / channel strategy / symmetric, input activations 8-bit dynamic per-token.

Not measured — do not quote this repo for any of these:

• No perplexity, KL divergence, or task accuracy. The source ships MMLU artifacts; these quantized checkpoints were not re-scored against them. A quantization that preserves fluency can still be worse on tasks, and nothing here rules that out.
• No throughput or latency. INT8 W8A8 speed depends on the serving kernel, not the artifact.
• No ablation-robustness test. These models are refusal-removed; quantization may partially restore refusal. Untested here.

Provenance

model_free_ptq uses no calibration dataset: weight scales come from the weights themselves and activations are quantized at runtime. The artifact is therefore dataset-independent.

License and obligations

Upstream is apache-2.0. That is this artifact's license, but the base models (Altworld/Hemmingway-1 and Qwen/Qwen3.8-27B) carry their own terms, including regional access limits on the Qwen line. If you redistribute or serve this checkpoint, you inherit those obligations; read them before deploying.