license: mit base_model:
- deepreinforce-ai/Ornith-1.0-397B base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags:
- abliterated
- uncensored
- mixture-of-experts
- qwen3_5_moe
- w4a16
- auto-round
- gptq
- vllm
- multimodal
- dgx-spark
Ornith-1.0-397B — Abliterated · W4A16
A decensored (abliterated) rebuild of deepreinforce-ai/Ornith-1.0-397B — a 397B-parameter Qwen3.5-MoE multimodal reasoning / agentic-coding model — re-quantized to W4A16 (AutoRound, auto_round:auto_gptq) so it runs on 2× NVIDIA DGX Spark (GB10, 128 GB each) with vLLM.
Refusal behavior was removed from the language model only; the vision tower, embeddings, routers and norms are untouched. Coding, reasoning and vision (image + video) remain intact.
⚠️ Uncensored model. Safety refusals have been substantially removed. You are responsible for how you use it. Intended for local/research use on hardware you control. It will attempt almost any request.
Highlights
| Base | deepreinforce-ai/Ornith-1.0-397B (Qwen3.5-MoE, 397B total / ~17B active) |
| Architecture | 60 layers, hidden 4096, 512 experts (top-10), hybrid Gated-DeltaNet linear + full attention (45 + 15), MTP head, SigLIP-style vision tower (image + video) |
| Context | 262,144 tokens (256K) |
| Quantization | W4A16 — AutoRound RTN, int4, group-size 128, symmetric, auto_round:auto_gptq packing |
| Kept at BF16 | embeddings, lm_head, all routers (mlp.gate, shared_expert_gate), norms, and the entire vision tower |
| Size | ~196 GB, 47 safetensors shards |
| Runtime | vLLM ≥ 0.17 (tested 0.20.1), served across 2× DGX Spark via Ray TP=2 |
Validation (before → after abliteration)
Measured on the 2-node vLLM cluster; refusal on mlabonne/harmful_behaviors (N=40, identical prompts), KL over top-k first-token logprobs on mlabonne/harmless_alpaca (N=40).
| Metric | Reference W4A16 | This model (abliterated) |
|---|---|---|
| Refusal rate | 30.0 % | 7.5 % (−75 %) |
| KL divergence (harmless first-token) | — | 0.116 — capability preserved |
| Coding smoke-test | — | ✅ works |
| Image understanding | — | ✅ works (OCR + scene) |
| Video understanding | — | ✅ works (frames + motion) |
| 128K context (needle retrieval) | — | ✅ retrieved both needles @ 126,709 tok |
| 256K context (needle retrieval) | — | ✅ retrieved both needles @ 252,229 tok |
Serving performance (2× DGX Spark, TP=2 over ConnectX-7, single-stream):
| input ctx | decode tok/s | prefill tok/s | TTFT |
|---|---|---|---|
| ~1K | 22 | 824 | 1.1 s |
| ~8K | 23 | 1,102 | 6.6 s |
| ~128K | 18 | 886 | 134 s |
| ~256K | 17 | 704 | 329 s |
Decode is ~flat across context length because the hybrid linear-attention layers keep KV small; the single-stream ceiling is the per-token cross-node all-reduce over the 200 GbE ConnectX-7 link (no NVLink).
How it was built
- Refusal directions. A memory-frugal layer-streaming forward over the local quant (per-layer gptq→bf16 dequant, no full model in memory) computed Arditi difference-of-means refusal directions
r_ℓ ∈ ℝ⁴⁰⁹⁶on 128 harmful vs. 128 harmless prompts (per-layer directions, adjacent-layer cosine ≈ 0.91). - Abliterate + re-quantize (streaming). The 122 BF16 shards of the base were streamed from the Hub one at a time and the residual-write matrices of the LM were orthogonalized against the refusal direction —
W ← W − r rᵀWforself_attn.o_proj,linear_attn.out_proj,mlp.shared_expert.down_proj, and per-expertmlp.experts.down_proj(hidden = output axis) — then re-quantized with AutoRound'squantize_weight_rtnin the exactauto_round:auto_gptqlayout, un-fusing experts toexperts.{e}.{gate,up,down}_proj. Output is byte-identical to the reference AutoRound quant on every non-abliterated tensor.
Never staged the full 794 GB BF16 model — the whole pipeline is streaming + data-free (AutoRound model_free/RTN, no calibration).
Serving (vLLM, 2× DGX Spark)
The model (~196 GB) exceeds a single 128 GB Spark, so serve it across both nodes with tensor-parallel = 2 (Ray). Grade/serve config that works on GB10:
vllm serve /path/to/this-model \
--served-model-name ornith --trust-remote-code \
--tensor-parallel-size 2 --distributed-executor-backend ray \
--max-model-len 262144 --kv-cache-dtype fp8 \
--gpu-memory-utilization 0.90 --max-num-seqs 1 \
--enable-chunked-prefill --max-num-batched-tokens 8192 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--limit-mm-per-prompt '{"image":48,"video":4}'
GB10 notes: pin NCCL to the ConnectX-7 interface; disable Ray's OOM monitor (RAY_memory_monitor_refresh_ms=0) since on unified memory the weights legitimately use most of each node's 121 GiB; --enable-chunked-prefill is required for long context (uncapped prefill OOMs); nvidia-smi reports memory as N/A on GB10 — use free -h.
Notes for agent frameworks
- Reasoning model: replies begin with an inline
Thinking Process:chain-of-thought inmessage.content(no separate reasoning field). Budgetmax_tokensaccordingly and treatcontentas reasoning. - Tool calls: with
--enable-auto-tool-choice --tool-call-parser qwen3_coder, OpenAI-styletools/tool_choice:"auto"return structuredtool_calls(verified). The CoT still appears incontentalongside the tool call.
Files
config.json, quantization_config.json, model-*-of-*.safetensors (+ index), tokenizer.json/tokenizer_config.json/vocab.json, chat_template.jinja, generation_config.json, preprocessor_config.json, processor_config.json, video_preprocessor_config.json. See docs/ for the full validation report and an agent-integration brief, and scripts/ for the abliteration/quant/serve/eval pipeline used to build this.
License & credits
MIT (inherited from the base model). Base model: deepreinforce-ai/Ornith-1.0-397B. Abliteration follows the residual-direction method (Arditi et al.; tooling inspired by p-e-w/heretic and elder-plinius/OBLITERATUS); quantization via Intel AutoRound. Not affiliated with or endorsed by the base-model authors.