Nex-N2.5-mini-Uncensored-W4A16-AutoRound

W4A16 (INT4 weights, group size 128, symmetric, BF16 activations) quantization of orcarouter/Nex-N2.5-mini-Uncensored (the abliterated Nex-N2.5-mini, a Qwen3.5-MoE 35B-A3B vision-language model) in auto_gptq format, loadable by vLLM (Marlin / XPU) and SGLang.

Why this exists

orcarouter publishes the Uncensored weights as BF16 (65 GiB), FP8 (34 GiB, Hopper+), NVFP4 (22 GiB, Blackwell only), MLX (Apple) and GGUF (llama.cpp). None of those loads in vLLM on a 32 GiB non-Blackwell card. [quant-mind/Nex-N2.5-mini-W4A16-AutoRound] (https://huggingface.co/quant-mind/Nex-N2.5-mini-W4A16-AutoRound) filled that gap for the STOCK model; this repository applies the same published recipe to the Uncensored weights, so the two differ in base weights only. Built for the murakumo fleet's Intel Arc Pro B70 (32 GiB) head, where vLLM XPU + XPU graph measured ~1,128 tok/s aggregate on the stock W4A16 checkpoint vs ~137 tok/s for llama.cpp IQ4_XS on the same card (root ADR-2609181615, com-junkawasaki/root).

Recipe (identical to quant-mind's config.json)

Intel AutoRound 0.15.1, --scheme W4A16 --group_size 128 (sym), --iters 200, --nsamples 128 --seqlen 2048, calibration NeelNanda/pile-10k, --low_gpu_mem_usage
Kept in BF16: mlp.gate (router), mlp.shared_expert_gate, mtp, lm_head, embeddings, the visual tower
Quantized: linear_attn.*_proj, self_attn.{q,k,v,o}_proj, all 256 mlp.experts.*, mlp.shared_expert.*
Export format: auto_gptq
Built on Modal (H100) by the job tools/modal-quant/nex_n25_mini_uncensored_w4a16.py in kotoba-lang/murakumo.

Use

vllm serve com-kotobalabs/Nex-N2.5-mini-Uncensored-W4A16-AutoRound     --reasoning-parser qwen3 --max-model-len 65536 --gpu-memory-utilization 0.90

The model thinks before answering; without --reasoning-parser qwen3 the reasoning leaks into content.

Caveats

Abliterated / uncensored derivative: it will not refuse. Deploy behind your own policy layer; the murakumo gateway runs it only behind governed organisms.
Quality vs the BF16 source is not evaluated here beyond a coherence probe; the quantization error profile is the one AutoRound publishes for this recipe.
License follows the base: Apache-2.0 (nex-agi/Nex-N2.5-mini) as relicensed by orcarouter.

Why this exists

Recipe (identical to quant-mind's config.json)

Intel AutoRound 0.15.1, --scheme W4A16 --group_size 128 (sym), --iters 200, --nsamples 128 --seqlen 2048, calibration NeelNanda/pile-10k, --low_gpu_mem_usage

Kept in BF16: mlp.gate (router), mlp.shared_expert_gate, mtp, lm_head, embeddings, the visual tower

Quantized: linear_attn.*_proj, self_attn.{q,k,v,o}_proj, all 256 mlp.experts.*, mlp.shared_expert.*

Export format: auto_gptq

Built on Modal (H100) by the job tools/modal-quant/nex_n25_mini_uncensored_w4a16.py in kotoba-lang/murakumo.

Caveats

Abliterated / uncensored derivative: it will not refuse. Deploy behind your own policy layer; the murakumo gateway runs it only behind governed organisms.

Quality vs the BF16 source is not evaluated here beyond a coherence probe; the quantization error profile is the one AutoRound publishes for this recipe.

License follows the base: Apache-2.0 (nex-agi/Nex-N2.5-mini) as relicensed by orcarouter.