license: apache-2.0 base_model: orcarouter/Qwen3.8-27B-Uncensored base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags:

gptq
int4
qwen3.8
intel-arc
xpu
vllm
abliterated
uncensored

Qwen3.8-27B-Uncensored GPTQ int4 (asymmetric)

An int4 quant of orcarouter/Qwen3.8-27B-Uncensored (an abliterated finetune of Qwen/Qwen3.8-27B by OrcaRouter). I made it to run on a single Intel Arc Pro B70. It's asymmetric int4, group 128, GPTQ with an MSE clip search, stored in standard GPTQ v1 format (sym: false). The vision tower, the MTP head and the lm_head stay BF16.

Safety: this inherits the base model's removed refusal behaviour. It has no meaningful built-in guardrails. Read the base model card for its intended-use terms. Use it responsibly and within the Apache-2.0 license.

Quality vs the BF16 base

KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):

	private chat	code	technical prose	all	top-1 agree
symmetric GPTQ int4 g128 (usual recipe)	0.038	0.078	0.096	0.069	91.9%
this quant	0.018	0.040	0.060	0.038	94.1%

The served version (fp8 lm_head, see the cookbook) measures KL 0.039 end to end. On the arithmetic canary from the B70 cookbook it scores 5/6 vs 4/6 for symmetric GPTQ. The one miss is sum(i*i for i in range(4)) -> 30.

Running it

Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 with the launcher and patches in riceharvest/b70-qwen38-cookbook. That setup gives 262K context with vision, MTP k=4, and fp8 KV on one card.

hf download darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym --local-dir qwen38-uncensored-b70
MODEL_DIR=$PWD/qwen38-uncensored-b70 ./serve.sh     # from the cookbook repo

On a B70 at 275 W, one request at a time:


Decode, short prompts (greedy)	64-128 tok/s depending on content (35 ms per MTP verify step)
Decode at 65K / 100K context	~79 / 65-70 tok/s greedy, ~64 sampled (43 / 46 ms per step)
Prefill	~2050 tok/s at 6K, ~1450 at 74K
Context	262,144 tokens, ~320K-token KV pool

Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should load it. I only tested on Intel XPU. Stock vLLM XPU v0.31 does not load asym GPTQ correctly without the cookbook's patch_asym_gptq.py.

How it was made

Quantizer: b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.
Calibration: 128 x 2048 tokens. A mix of private chat samples (not released), reasoning/agentic text, code and technical prose, disjoint from the eval windows.
Not quantized: lm_head, MTP layers, vision tower, the GDN in_proj_a/in_proj_b.
Export: GPTQ v1 (qzeros store z-1). The single group with z=0 was set to z=1 (one weight changed). It was checked against the raw quant using GPTQModel's dequant semantics.

Credits: Qwen team (base model), OrcaRouter (finetune), SergiioB (B70 vLLM cookbook this builds on).

Qwen3.8-27B-Uncensored GPTQ int4 (asymmetric)

Safety: this inherits the base model's removed refusal behaviour. It has no meaningful built-in guardrails. Read the base model card for its intended-use terms. Use it responsibly and within the Apache-2.0 license.

Quality vs the BF16 base

KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):

private chat

code

technical prose

all

top-1 agree

symmetric GPTQ int4 g128 (usual recipe)

0.038

0.078

0.096

0.069

91.9%

this quant

0.018

0.040

0.060

0.038

94.1%

Running it

hf download darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym --local-dir qwen38-uncensored-b70 MODEL_DIR=$PWD/qwen38-uncensored-b70 ./serve.sh # from the cookbook repo

On a B70 at 275 W, one request at a time:

Decode, short prompts (greedy)

64-128 tok/s depending on content (35 ms per MTP verify step)

Decode at 65K / 100K context

~79 / 65-70 tok/s greedy, ~64 sampled (43 / 46 ms per step)

Prefill

~2050 tok/s at 6K, ~1450 at 74K

Context

262,144 tokens, ~320K-token KV pool

How it was made

Quantizer: b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.

Calibration: 128 x 2048 tokens. A mix of private chat samples (not released), reasoning/agentic text, code and technical prose, disjoint from the eval windows.

Not quantized: lm_head, MTP layers, vision tower, the GDN in_proj_a/in_proj_b.

Export: GPTQ v1 (qzeros store z-1). The single group with z=0 was set to z=1 (one weight changed). It was checked against the raw quant using GPTQModel's dequant semantics.

Credits: Qwen team (base model), OrcaRouter (finetune), SergiioB (B70 vLLM cookbook this builds on).