license: apache-2.0 base_model: abliterant/Qwen3.8-27B-RANA-abliterated base_model_relation: quantized library_name: gguf pipeline_tag: image-text-to-text tags:

• abliterant
• gguf
• llama.cpp
• imatrix
• abliteration
• refusal-direction
• qwen3
• vision-language

Abliterant

Qwen3.8-27B-RANA-abliterated-GGUF

Abliterant GGUF quantizations of abliterant/Qwen3.8-27B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.8-27B, for llama.cpp and compatible apps. Vision (mmproj-*) and the MTP speculative-decoding head (mtp-*) are included as separate files, in the same layout as ggml-org/Qwen3.8-27B-GGUF.

Quick start · Evaluation · Available files · Abliterant models

This is a safety-alignment-removed research model. Read Intended use and Limitations before using it. Method, full evaluation and release gates are on the BF16 card.

At a glance

Release family

Quick start

The release tests used llama.cpp at commit 4b1a27f (CUDA, RTX PRO 6000 Blackwell). Use a build of that revision with llama-server on your PATH.

Download all three files before running the local-file command:

hf download abliterant/Qwen3.8-27B-RANA-abliterated-GGUF \
  Qwen3.8-27B-RANA-abliterated-Q4_K_M.gguf \
  mmproj-Qwen3.8-27B-RANA-abliterated-BF16.gguf \
  mtp-Qwen3.8-27B-RANA-abliterated-Q8_0.gguf --local-dir .

Run from the directory containing those downloads:

llama-server -m Qwen3.8-27B-RANA-abliterated-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-RANA-abliterated-BF16.gguf \
  -md mtp-Qwen3.8-27B-RANA-abliterated-Q8_0.gguf --spec-type draft-mtp \
  --jinja -fa on -ngl 99 -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0

Alternatively, let llama.cpp fetch the files remotely. The release also tested automatic fetching: the mmproj-* file is picked up automatically; the mtp-* file only when --spec-type draft-mtp is given.

llama-server -hf abliterant/Qwen3.8-27B-RANA-abliterated-GGUF:Q4_K_M --spec-type draft-mtp \
  --jinja -fa on -ngl 99 -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0

• Thinking is on by default. The sampling flags above are Qwen's recommended thinking-mode settings; llama.cpp's own default min_p is 0.05, so set --min-p 0 explicitly.
• Long technical requests can need 20–50k tokens of reasoning (see the FP8 card); raise -c accordingly (native context 262k).

What changed

• Source: the published BF16 repo, converted with llama.cpp's convert_hf_to_gguf.py (commit 4b1a27f): main model with --no-mtp, MTP head with --mtp, vision tower with --mmproj. Tensor counts and sizes match ggml-org's conversion of the base model (851 / 53.8 GB, 18 / 5.9 GB, 334 / 0.93 GB).
• Importance matrix: llama-imatrix on the BF16 GGUF with bartowski's public calibration text (Qwen3.8-27B-calibration-v6.txt from bartowski/Qwen3.8-27B-GGUF). General text, no refusal-related prompts.
• Quantization: llama-quantize --imatrix with llama.cpp's default tensor layouts for each type (no per-tensor overrides).

The refusal-abliteration method, full BF16 evaluation and release gates are documented on the BF16 release card. This GGUF conversion and quantization are not additional training.

Evaluation

Functional and refusal checks cover Q4_K_M only; all 15 quantizations have distribution-closeness measurements, not a full downstream evaluation.

Quantization quality

BF16 GGUF perplexity on the same text: 6.776 ± 0.104 (base Qwen3.8-27B in bartowski's table: 6.744 ± 0.103). The table puts each quant next to bartowski's quant of the base model, measured with the same protocol (his perplexity.md, llama.cpp b10896). It is a different model, so the comparison is indicative only.

Most of his files are larger at the same name (by up to 2.3 GB) because he overrides the type of some tensors; these quants use llama.cpp's default layouts. At equal size the two are close: this repo's Q4_K_M (16.55 GB) has the same KLD as his Q4_K_S (16.36 GB), 0.0156, and this IQ4_XS has a slightly lower KLD than his at 0.4 GB smaller.

Functional checks (Q4_K_M, llama-server)

Served with llama-server, Q4_K_M + mmproj-…-BF16 + mtp-…-Q8_0:

• Vision: reads the code word and shape from a synthetic image: pass.
• Tool calling: 3-turn call → result → second call with a new argument: pass.
• MTP speculative decoding: 348 of 486 drafted tokens accepted (71.6%) on one 510-token generation at temperature 0: pass.
• The same three checks pass when the files are fetched with -hf … --spec-type draft-mtp.

Refusals (Q4_K_M, seed 1)

Same prompts, seed and sampling as the BF16 and FP8 builds (refusal suite v2, seed 1, thinking on, 16k-token budget), judged by openai/gpt-oss-safeguard-20b. The GGUF was served with llama.cpp, the other two with vLLM, so part of any difference can come from the engine.

HarmBench labels follow the rule used for every build: each gpt-oss HARD_REFUSAL is re-judged (BF16 card, G1 re-adjudication disclosure). Q4_K_M and BF16 had none in seed 1; FP8's 3 raw ones re-judged as 2 answers and 1 degenerate, shown above. Paired with BF16 on the same prompts (exact McNemar): answers on HarmBench 13 Q4_K_M-only vs 11 BF16-only (p = 0.84), held-out 10 vs 13 (p = 0.68); budget hits on HarmBench 5 vs 11 (p = 0.21), held-out 4 vs 7 (p = 0.55). None of the differences is significant. The one held-out hard refusal is a raw label on a finished answer that contains no refusal phrase; it was not re-judged.

Available files

KLD = mean KL divergence of each quant's next-token distribution from the BF16 GGUF; "same top token" = how often both pick the same most likely token. Measured with llama-perplexity on wiki.test.raw, 100 chunks of 512 tokens. Lower KLD is closer to BF16.

Also in this repo:

• mmproj-Qwen3.8-27B-RANA-abliterated-{BF16,F16}.gguf (0.93 GB): the vision tower. Same weights as the base model's (abliteration does not touch it).
• mtp-Qwen3.8-27B-RANA-abliterated-{Q8_0,BF16}.gguf (3.2 / 5.9 GB): the MTP head as a speculative draft for --spec-type draft-mtp. It is the abliterated MTP head, consistent with the main model.
• Qwen3.8-27B-RANA-abliterated-BF16/ (2 parts, 53.8 GB): unquantized GGUF, the KLD reference.
• Qwen3.8-27B-RANA-abliterated-imatrix.gguf: the importance matrix used for every quant.
• SHA256SUMS, results/: checksums and the measurements behind this card.

Which one? Q8_0 and Q6_K are near-lossless. Q5_K_M and Q4_K_M are the usual choices when memory is tight. Below 4 bits the KLD rises quickly; IQ3/Q3 and IQ2/Q2 are for fitting into 12–16 GB, with a visible quality cost.

Limitations and intended use

Limitations

• Functional checks and refusal behavior were evaluated on Q4_K_M only, one seed and one judge. Other quants were checked for closeness to BF16 (KLD) but not for refusal behaviour; the lowest-bit quants drift the most from BF16 and may behave differently.
• KLD is measured on English Wikipedia text at 512-token context. It says how closely a quant tracks BF16, not how it scores on downstream tasks.
• Everything listed under Limitations on the BF16 card applies here too: judge-measured refusal rates, long reasoning on technical requests, and the capability changes measured there.

Reduced refusal does not establish greater safety, accuracy, factual reliability, or universal compliance. The llama.cpp/vLLM engine difference prevents attributing every comparison difference to quantization alone.

Intended use

• Research only: interpretability, red-teaming, and robustness evaluation of refusal behaviour.
• Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.
• You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

Provenance and license

• Qwen team: base model Qwen/Qwen3.8-27B.
• Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
• Jim Lai (grimjim): prior work on norm-preserving abliteration.
• llama.cpp / ggml-org: conversion, quantization and inference; bartowski: the calibration text and the public KLD table used for comparison.
• Benchmarks/datasets: HarmBench, StrongREJECT, JailbreakBench, CategoricalHarmfulQA; wikitext-2.

The artifacts retain the Apache-2.0 license inherited from Qwen; see the Apache-2.0 terms and the upstream model. Original release preparation and measurements are credited to preemware, alongside the contributors above.