license: apache-2.0 base_model: Hikari07jp/Ternary-Bonsai-2-27B-Abliterated-GGUF library_name: llama-cpp pipeline_tag: text-generation tags:

• qwen
• llama.cpp
• ternary
• abliterated
• uncensored
• bonsai
• prism

Ternary-Bonsai-2-27B — Abliterated, with the MTP head requantised in Q4_0

PrismML's Ternary-Bonsai-2-27B is a ternary 27B: every weight is −1, 0 or +1, which is what lets a 27B run on a 16 GB card. The abliteration — the refusals removed by flipping a small fraction of those digits — is Hikari07jp's work.

What is mine here is the MTP head requantised in Q4_0, and the measurements: the head is a draft, so its precision costs nothing in output quality, and at 228 MiB instead of 430 MiB it is what makes the combination fit on an 8 GB card at all. The trunk is bit-identical to the abliterated build.

The trunk is bit-identical to Hikari07jp/Ternary-Bonsai-2-27B-Abliterated-GGUF; only blk.64.* differs from that file.

This is not the stock PrismML build, and not BoldingBuilds' files either (different bytes). The chain is prism-ml → Hikari07jp (ablation) → this repository (MTP head requantised in Q4_0, plus the measurements).

How it was made

Nothing was re-quantised in the trunk: the file starts from Hikari07jp's abliterated build, byte for byte.

1. The MTP head is taken as published. PrismML ships a next-token-prediction head with the Bonsai 2 27B; the reference build BoldingBuilds/Ternary-Bonsai-2-27B-Abliterated-v2-PQ2_0-MTP-GGUF carries it as the block blk.64.* (15 tensors, 430,41 MiB in Q8_0).
2. The head is requantised in Q4_0 (tools/mtp_requant.py): dequantise to fp32, requantises in Q4_0 (18 octets per 128 weights), and writes the tensors back with their new types and offsets. The head goes from 430,41 MiB to 227,91 MiB.
3. The head is merged into the trunk (tools/mtp_merge.py): byte surgery, no re-encoding — the 851 trunk tensors are copied verbatim, the 15 head tensors are appended, qwen35.block_count goes from 64 to 65 and qwen35.nextn_predict_layers is set to 1. (The count matters: the fork computes n_layer() = block_count − n_layer_nextn, so without it the loader would take blk.63 — a trunk block — as the head and load only 63 layers: a wrong model.)
4. The bytes are checked (tools/validate_quant.py): the encoder used to produce the Q4_0 head is byte-identical to the fork's C reference quantize_row_q4_0_ref.

The result is a file whose trunk is bit-identical to the abliterated build and whose only difference is the head.

Measured

Every number below was measured on an RTX 3070 8 GB with the desktop running, on the files as published, with PrismML's llama.cpp fork (the build the Bonsai Hadamard packing needs).

Perplexity: wikitext-2-raw, 18 944 tokens, 37 chunks of 512, llama-perplexity -m M -f ppl_corpus_wikitext2_78k.txt -c 512 -fit on, the same protocol and the same corpus for every file. Decode rate: 200 generated tokens per run, two runs per file, counted with the server's own tokenizer and cross-checked against the server's internal counter.

The PQ2_0 packing is worth 5,72 % of perplexity over PTQ1_0 (8,4557 against 8,9392) — the same weights, re-quantised. It is also the slower and the larger of the two: 6 540 MiB of weights against 5 395 MiB, and about 5 tok/s less. That trade-off is the interesting part of this model.

Running it

The Bonsai Hadamard packing needs a llama.cpp that supports it; upstream llama.cpp does not read these files (invalid ggml type 141/143). PrismML's fork does.

llama-server \
  --model Ternary-Bonsai-2-27B-Abliterated-PQ2_0.gguf --alias bonsai27-abliterated \
  --ctx-size 2048 --cache-type-k q4_0 --cache-type-v q4_0 \
  --n-gpu-layers 65 --fit off --flash-attn on --jinja -b 512 -ub 128 --parallel 1

• --n-gpu-layers 65, not 64. The count includes the output layer: 64 leaves the first block on the CPU and switches the fused GDN off, which costs 27 % of the decode rate.
• The KV cache is q4_0; the recurrent-state buffer is 149,6 MiB per slot, so the server's --parallel is what decides how many streams fit.
• Alias it. The file's internal name is Ternary-Bonsai-2-27B-PQ2_0, so without an alias it is indistinguishable from the stock build in an API's model list. The registry's convention is the same: an uncensored variant keeps the same model entry but is served under its own name.
• The vision projector (Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf, from the stock repository) does not fit with the PQ2_0 builds on 8 GB — the server aborts allocating its 600 MiB. It fits with the PTQ1_0 packing.

The MTP head, and why requantising it changes nothing

The MTP head is a draft: it proposes tokens that the main model then verifies, so whatever precision the head carries, the tokens that come out are the main model's. Measured on the file in this repository:

• same perplexity as the headless build: 8,4593 against 8,4557 (within the ±0,24 of the measurement);
• zero divergent tokens over 200 greedy tokens against the headless build, same prompt, same compute path.

Requantising the head in Q4_0 is what makes the combination fit on 8 GB: the published …-v2-PQ2_0-MTP.gguf (7,13 GiB) needs 6 970 MiB for the main model alone, and the recurrent-state buffer of the MTP mode (598 MiB) does not fit. With the head in Q4_0 (227,91 MiB instead of 430,41 MiB) the whole thing runs in 7 724 MiB.

One honest caveat: at --spec-draft-n-max 2 fifteen tokens differ from the headless build, at the same index, with the stock Q8_0 head as well — it is the effect of verifying two or three tokens in one forward pass, not of the requantisation.

The merge is byte surgery, no re-encoding: the trunk is bit-identical to the abliterated build, and the head comes from BoldingBuilds/Ternary-Bonsai-2-27B-Abliterated-v2-PQ2_0-MTP-GGUF (blk.64.*). The tools are in tools/.

License and attribution

Apache-2.0, like the base model. This is a derivative of prism-ml/Ternary-Bonsai-2-27B-gguf at revision 6ed5e12bf84b7a63069882c91dd9e9218647d17b (Apache-2.0, itself derived from Qwen); the base repository's LICENSE and NOTICE.txt apply unchanged and are reproduced in LICENSE and NOTICE.txt here. The MTP head comes from BoldingBuilds' build of the same base, under the same license.