license: apache-2.0 base_model: orcarouter/Qwen3.8-27B-Uncensored base_model_relation: quantized pipeline_tag: text-generation library_name: gguf tags:

• gguf
• llama.cpp
• ninfer
• mtp
• speculative-decoding
• imatrix
• uncensored
• abliterated
• qwen3.8

Qwen3.8-27B OrcaRouter Uncensored — MTP, sized for 16 GB GPUs

GGUF and NInfer builds of orcarouter/Qwen3.8-27B-Uncensored, made so the model and its native MTP head fit in 16 GB of VRAM with a useful context. OrcaRouter's single-direction ablation came first of 12 uncensored Qwen3.8-27B variants in abliterlitics' comparison: it complies with 82% of the harmful prompts in their judge set while every capability benchmark stays within ±1.3 points of the original.

Every number below was measured on one RTX 4080 16 GB; the method is at the end.

Files

• Every file uses a per-tensor mixed-precision allocation with an importance matrix, not a single quant type.
• The MTP block (blk.64) is kept at q6_K / q8_0 in every file.
• The .ninfer files are the GGUFs above repacked for NInfer, block for block, with no requantization.

Quality against the other published OrcaRouter quants

Mean KL divergence against a Q8_0 of the same OrcaRouter weights (lower is better), and the share of tokens whose top-1 prediction matches the reference.

¹ Better than UD-IQ4_XS, but 1.23 GiB larger. Qwen3.8 keeps KV only in 16 of its 64 layers, so a full 32K q4_0 cache is ~0.6 GiB: on a 16 GB card that also drives a desktop there is no context to give up for it.

• UD-IQ3_XXS, 0.5 GiB larger than RentedNoodle's file, sits 40% closer to the reference on both texts.
• The same quantization gives the same KLD on OrcaRouter as on llmfan46's Heretic weights (0.027 / 0.090 there), so the quantization error does not depend on the ablation.

Why OrcaRouter and not Heretic

The same quantization was first published on llmfan46's Heretic weights (codavidgarcia/Qwen3.8-27B-Uncensored-Heretic-MTP-UD-GGUF). The quantization error is the same on both (UD-IQ4_XS: 0.028 here, 0.027 there); the difference is the base. Changes against the original Qwen3.8-27B, from abliterlitics:

Speed (RTX 4080 16 GB)

NInfer, RCO-IQ3_S, OrcaRouter weights. A harness-like conversation that grows from 10K to 23K tokens (system prompt + tool results), 400-token answers, thinking off:

Prefill held 1.4–1.75K tok/s and the prompt cache reused 80–89% of each turn. Decode follows acceptance: it is highest when the answer repeats code already in context.

Same quantization on the Heretic weights (not re-measured on OrcaRouter; the files are the same size and type mix):

Running it

llama.cpp (16 GB, UD-IQ4_XS)

llama-server -m Qwen3.8-27B-OrcaRouter-Uncensored-MTP-UD-IQ4_XS.gguf -ngl 99 --flash-attn on \
  -c 32768 -ub 256 -ctk q4_0 -ctv q4_0 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2

This sits at the edge of 16 GB. With a desktop on the same card, watch "shared GPU memory" in Task Manager: once part of the model spills to system RAM, decode drops sharply.

NInfer

Built with iamwavecut/ninfer-all at e1debb4 (2026-10-09), CUDA, sm_89. Other NInfer forks may reject the files.

ninfer-serve qwen38-27b-orca-rco-iq3s.ninfer --model-id qwen3.8-27b-uncensored \
  --max-context 65536 --kv-capacity auto --kv-headroom-mib 256 --prefill-chunk 512 \
  --kv-dtype rk8v4 --gdn-state-fp16 --spec mtp --draft-tokens 3 \
  --max-concurrency 1 --device-state-slots 0 --host-cache-mib 1024 \
  --max-private-continuations 1 --max-shared-prefixes 1 \
  --max-long-anchors-per-continuation 3 --auto-long-anchors --branch-anchors \
  --context-cache-policy rolling --host 127.0.0.1 --port 8080

• RCO-IQ3_S at 64K leaves 256 MiB of headroom: it needs the desktop at ~1.5 GB of VRAM or less.
• UD-IQ3_XXS runs 80K with --kv-headroom-mib 768 --prefill-chunk 1024.
• With UD-IQ3_XXS, 96K does not start on 16 GB, and 3 MTP drafts beat 2 and 4.
• The artifacts were packed without the proposal head, so --lm-head-draft is not available.

Caveats

• Thinking. The chat template is Qwen's own, unchanged by OrcaRouter: reasoning_effort is low, medium or xhigh (default), and enable_thinking: false turns thinking off. NInfer also accepts none, minimal (= low) and max (= xhigh) on the OpenAI reasoning_effort field.
• Vision. These files are text-only. OrcaRouter's mmproj was built for the same weights but was not tested with them.
• Uncensored. This model has had its refusal behaviour removed. It will comply with harmful, unethical or illegal requests the original Qwen3.8-27B would refuse. OrcaRouter released it for research, red-teaming and robustness evaluation; you are responsible for how you use it and for adding your own safety layers before any deployment. Quantization can make behaviour near the old refusal boundary less stable.

Evaluation

Source: OrcaRouter's F16 GGUF. Reference: Q8_0 from the same F16. Texts: wikitext-2 test and two llama.cpp source files. llama-perplexity --kl-divergence, context 1024, 40 chunks per text (~20K scored tokens each). Speed: NInfer's own server timings over the OpenAI API, prompts built from llama.cpp server sources.

Credits

• Qwen for Qwen3.8-27B.
• OrcaRouter for the ablation and the F16 GGUF.
• llama.cpp and NInfer.

License: Apache-2.0, as the base model.