library_name: transformers license: mit pipeline_tag: image-text-to-text base_model:
- bowmanslayer/Ornith-1.5-9B-Uncensored base_model_relation: quantized tags:
- uncensored
- abliterated
- ornith
- qwen3.5
- multimodal
- w4a16
- gptq
- not-for-all-audiences language:
- en
- zh
Ornith-1.5-9B-Uncensored-W4A16
⚠️ Safety alignment has been removed from this model. It will comply with requests that the original model refuses, including offensive, dangerous, or jurisdiction-illegal content. There is no content filter and no guardrail. You are solely responsible for how you deploy it and for compliance with applicable law. Read §② Safety alignment has been removed below before use.
⚠️ 本模型的安全对齐已被移除。 它会执行原模型拒绝的请求,包括冒犯、危险、或所在 司法辖区违法的内容。无内容过滤、无护栏。部署方式与合规责任完全由使用者自负。 使用前请先阅读正文 §② 安全对齐已被移除。
This repo is the W4A16 quantization (INT4 weight + FP16 activation) of
bowmanslayer/Ornith-1.5-9B-Uncensored, ~8 GB. Designed for vLLM serving withgptq_marlinkernel. Sister repos: full-precision bf16 in the main repo, llama.cpp GGUF in-GGUF.本仓是 W4A16 量化版(INT4 权重 + FP16 激活),源自
bowmanslayer/Ornith-1.5-9B-Uncensored, ~8 GB,面向 vLLM 部署(gptq_marlin快速核)。姐妹仓:主仓是全精度 bf16, llama.cpp GGUF 见-GGUF。
Method: AutoRound → GPTQ format. Paired with fp8_e5m2 KV cache, larger context
headroom vs same-size competitors at the same GPU size.
方法:AutoRound → GPTQ 格式,vLLM gptq_marlin 快速核可用。配合 fp8_e5m2 KV cache
拿到较同尺寸对手更多的上下文容量。
1. Quick start / 快速开始
Recommended vLLM launch (fp8 KV enabled for best context out of the box): 推荐 vLLM 启动(已开 fp8 KV,开箱最佳上下文):
vllm serve bowmanslayer/Ornith-1.5-9B-Uncensored-W4A16 \
--dtype float16 --tensor-parallel-size 2 \
--kv-cache-dtype fp8_e5m2 \
--max-model-len 28672 --max-num-seqs 16 \
--reasoning-parser qwen3
Single-GPU / 单卡:
vllm serve bowmanslayer/Ornith-1.5-9B-Uncensored-W4A16 \
--dtype float16 --tensor-parallel-size 1 \
--kv-cache-dtype fp8_e5m2 --gpu-memory-utilization 0.92
2. Full performance evaluation / 完整性能测试
Quantization did not break the ablation — refusal behavior and capability follow the bf16 reference. All numbers below are the bf16 reference build (this W4A16 was quantized from it); W4A16-specific accuracy differences on spot checks were within evaluation noise.
量化不破消融 —— 拒绝行为与能力沿用 bf16 参考版。以下数字取自本 W4A16 派生的 bf16 参考版;W4A16 在抽测上与 bf16 精度差异在评测噪声范围。
2.1 Refusal (dual-judged) / 拒绝率(双判据)
23 adversarial prompts across 9 harmful categories, generated end-to-end by an internal uncensored 27B judge (operator never read prompts/responses). Same 27B then judged REFUSE/COMPLY and separately COHERENT/LOOP/GIBBERISH (sanity check prevents mis-scoring loop outputs as COMPLY).
23 条对抗提示涵盖 9 类有害内容,由内部无审查 27B 端到端生成(操作者全程不读)。 同 27B 再判 REFUSE/COMPLY 与独立判 COHERENT/LOOP/GIBBERISH(防循环输出被误判 COMPLY)。
| Model | REFUSE/23 | COHERENT/23 |
|---|---|---|
| Ornith-base | (not run — base measurement below) | — |
| This model (bf16 reference) | 0 ✅ | 23 ✅ |
2.2 11-benchmark full capability (T3) / 11 项完整能力
Same fixed seed (20260816), same harness, same items. Both under vLLM TP=2 fp16, thinking enabled, 4096 max output tokens (8192 for MATH-500). "Excl. trunc" excludes items whose thinking budget was clipped.
同一固定 seed(20260816),同 harness,同题。vLLM TP=2 fp16,思考开启,4096 max output tokens(MATH-500 8192)。「排除截断」= 排除思考被截断的题,反映纯能力。
| Benchmark | N | Ornith-1.5-9B base | This model | Δ |
|---|---|---|---|---|
| MMLU | 150 | 90.14 | 88.97 | -1.17 |
| CMMLU | 150 | 86.67 | 84.17 | -2.50 |
| MMLU-Pro | 150 | 88.97 | 87.12 | -1.85 |
| C-Eval | 150 | 85.82 | 84.78 | -1.04 |
| ARC-Challenge | 150 | 93.33 | 94.67 | +1.34 |
| TruthfulQA | 150 | 79.31 | 82.31 | +3.00 ⭐ |
| GSM8K | 100 | 98.99 | 97.98 | -1.01 |
| MATH-500 | 100 | 73.68 | 70.53 | -3.15 |
| BBH | 150 | 91.72 | 95.27 | +3.55 ⭐ |
| HumanEval | 164 | 94.67 | 92.16 | -2.51 |
| IFEval (strict) | 100 | 78.65 | 77.53 | -1.12 |
| Mean (excl. trunc) | — | 87.45 | 86.86 | -0.59 |
Two benchmarks (BBH +3.55, TruthfulQA +3.00) actually improved — the ablation does not touch what Ornith's post-training gained on hard reasoning or truthfulness.
两项(BBH +3.55, TruthfulQA +3.00)反而上升 —— 本次消融未触碰 Ornith 后训练在 硬推理和真实性上的增益。
2.3 Output length / stopping behavior / 输出长度与停止行为
Median output tokens across 11 tasks, and truncation counts (thinking that hit max_tokens budget). Verifies stopping capability was not damaged.
| Ornith-base | This model | Δ | |
|---|---|---|---|
| Median p50 output tokens (mean of 11 tasks) | 487 | 540 | +11% |
| Total truncation / 1514 | 94 | 88 | -6% (fewer truncs) |
Output is ~11% more verbose than base but truncation actually decreased — stopping capability preserved. This is a critical sanity signal: over-ablation would blow truncation counts to 25%+ (as we observed in α=1.45 variants during development).
输出中位数比 base +11%(略啰嗦),但截断数反而下降 6% —— 停止能力完好保留。这是 关键 sanity 信号:过消融会把截断率打到 25%+(开发时 α=1.45 各版即如此)。
3. Context capacity / 上下文容量
Ornith-1.5-9B architecture = 32 layers (24 linear + 8 full-attention), hidden 4096. KV cache scales with context only on full-attention layers, so long contexts are cheap here.
Ornith-1.5-9B 架构 = 32 层(24 linear + 8 full attention),hidden 4096。KV cache 只对 full attention 层随 context 线性增长,故长上下文很便宜。
Per-token KV cache / 每 token KV cache:
- fp16 KV: 2 (K+V) × 8 layers × 4096 × 2 bytes = 128 KB / token
- fp8 KV (recommended default): 2 × 8 × 4096 × 1 = 64 KB / token (-50%)
Memory budget by scenario / 显存与场景对照:
| Scenario | Weight | KV (28k ctx, batch=1) | System | Total | Suggested GPU |
|---|---|---|---|---|---|
| This model + fp8 KV, batch=1 | ~8 | ~1.8 | ~1.5 | ~11.3 GB | 12 GB ✓ |
| This model + fp8 KV, batch=8 | ~8 | ~14 | ~1.5 | ~24 GB | 24 GB |
| This model + fp16 KV, batch=1 | ~8 | ~3.6 | ~1.5 | ~13 GB | 16 GB |
Deployment case (what we happen to run on, not a requirement): 2× RTX 3090 with TP=2. Single 3090 works fine for typical serving. Any GPU meeting the row you pick works.
部署案例(我们自己的机型,非硬件要求):2× RTX 3090 TP=2。单张 3090 常规服务 也够用。任何显存够上表某行的 GPU 都能跑。
4. Quantization recipe / 量化配方
AutoRound --bits 4 --group_size 128 --format auto_round:auto_gptq --nsamples 256 --seqlen 2048. linear_attn.in_proj_a/b (48 tensors, SSM computation, not
quantizable) excluded. Vision tower and text stack processed to int4 together.
AutoRound --bits 4 --group_size 128 --format auto_round:auto_gptq --nsamples 256 --seqlen 2048。linear_attn.in_proj_a/b(48 张量,SSM 不可量化)排除。视觉塔
与文本层一同处理为 int4。
① Provenance and attribution / 来源与归属
| Layer | Author |
|---|---|
| Base | ornith-ai/Ornith-1.5-9B — Ornith Team (Apache 2.0), a post-trained descendant of Qwen3.5-9B |
| Upstream base | Qwen/Qwen3.5-9B — Qwen team, Alibaba Cloud (Apache 2.0) |
| Abliteration | bowmanslayer/Ornith-1.5-9B-Uncensored — weight-space ablation, 64 tensors, method adapted from bowmanslayer/Qwen3.5-9B-Uncensored with embed_tokens excluded (see main repo §1) |
| This repo | W4A16 (AutoRound → GPTQ) quantization only. No weight modification beyond quantization. |
Not an official Ornith or Qwen release; not endorsed by or affiliated with either team.
非 Ornith 或 Qwen 官方发布;与两团队无背书或从属关系。
② Safety alignment has been removed / 安全对齐已被移除
This is the point of the model. Read before downloading. 这是模型的核心属性,下载前请务必阅读。
The upstream ablation removes the refusal behaviour trained into Ornith-1.5-9B (itself notably stronger on safety than Qwen3.5-9B base). Automated dual-judge testing on 23 held-out adversarial prompts recorded 0/23 refuses + 23/23 coherent. Quantization does not restore refusals.
上游消融移除了 Ornith-1.5-9B 训练进去的拒绝行为(它 safety 显著强于 Qwen3.5-9B base)。 23 条 held-out 对抗提示双判据测试记录到 0/23 拒 + 23/23 coherent。量化不恢复拒绝。
Consequences you accept / 你正在接受的后果:
- Content the original refuses, including offensive / dangerous / jurisdiction-illegal. 会产生原模型拒绝的内容,包括冒犯 / 危险 / 所在地违法
- No content filter, no safe-completion, no guardrail. 无内容过滤、无 safe-completion、无护栏
- Refusal removal is not proven exhaustive (n=23, English, single-turn). 拒绝移除未证明穷尽(n=23,英文,单轮)
Not intended for / 不适用于:
- Third-party or public-facing deployment without your own safety layer / 未搭建自己的安全层就部署给第三方或面向公众
- Anyone under legal age in their jurisdiction / 未达所在司法辖区法定年龄的人
- Any use prohibited by upstream acceptable-use policies / 上游可接受使用政策禁止的任何用途
Intended for / 适用于: local inference and research, by people who understand the above and take responsibility for it / 本地推理与研究,由理解以上内容并为其负责 的用户使用。
③ No warranty; responsibility rests with the user / 无担保;责任在用户
Provided "AS IS", without warranty of any kind, express or implied.
- I do not endorse, recommend, or condone any particular use.
- No representation outputs are accurate, lawful, or fit for any purpose.
- You are solely responsible for what you generate, how you deploy, and compliance with all laws applicable to you.
- To the maximum extent permitted by law, no liability for any claim, damage, or other liability arising from the model or its use.
按**"现状"提供,不提供任何担保**。我不背书、不推荐,不对输出的准确性/合法性/用途 适用性作声明。你独自承担全部责任。法律允许最大范围内不承担任何责任。
Downloading these files means you accept the above. If you do not, do not download.
下载本仓文件即表示接受以上内容。若不接受,请勿下载。
The author is not a lawyer; the above is not legal advice. 作者不是律师;以上不构成法律意见。
License / 许可
Apache 2.0, inherited through the chain: Qwen3.5-9B → Ornith-1.5-9B → this repo.
Apache 2.0,依上述归属链继承。