base_model: huihui-ai/Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated base_model_relation: quantized license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text language:
- en
- zh tags:
- nvfp4
- fp4
- w4a4
- gptq
- quantized
- compressed-tensors
- llm-compressor
- vllm
- qwen3_5
- vision-language
- thinking
- uncensored
- abliterated
- creative
- 1m-context
TL;DR: Huihui-Qwythos-9B-Claude-Mythos, quantized to NVFP4 (W4A4) for vLLM on NVIDIA Blackwell. 7.5 GB, wikitext-2 PPL 8.25, 1M-context creative writer, abliterated.
Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated NVFP4
NVFP4 (W4A4) quantization of
huihui-ai/Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated,
packed in the compressed-tensors nvfp4-pack-quantized format with
llm-compressor. Weights are
quantized with GPTQ (error-compensated rounding) and an MSE observer, on a
domain-matched calibration blend.
Near-lossless. Fused layers (q/k/v, gate/up) share one NVFP4 global scale, so vLLM loads it cleanly with no per-layer-scale warning or fallback. wikitext-2 perplexity for this build: 8.25.
- About 7.5 GB on disk versus about 19.3 GB for the bf16 source (about 39%).
- Built for vLLM on NVIDIA Blackwell, where both the 4-bit weight and 4-bit activation paths are accelerated. On pre-Blackwell GPUs vLLM runs it weight-only.
- Loading and generation verified in vLLM on an NVIDIA GB10 (Blackwell, sm_121).
Uncensored / abliterated model. It follows instructions without content guardrails, including NSFW. Behaviour and alignment are inherited entirely from the base model.
Fidelity
Near-lossless versus the bf16 source, 7.5 GB vs 19.3 GB bf16 (~39%), at wikitext-2 perplexity 8.25. GPTQ error compensation and an MSE observer keep the drop from bf16 minimal; the header lists the full characteristics and Quantization covers the recipe.
Quickstart
NVFP4 is auto-detected from config.json (compressed-tensors); no quantization flag
needed. --reasoning-parser qwen3 splits the <think> block into reasoning_content.
vllm serve maci0/Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NVFP4 \
--served-model-name qwythos-9b-nvfp4 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--kv-cache-dtype fp8 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder
- Supports up to 1048576 tokens; keep at least 128K to preserve thinking quality.
--max-model-len 131072is a safe default; raise it if memory allows. - Add
--language-model-onlyto skip the vision tower and free KV cache for text use. - The parser flags are not auto-detected; pass them explicitly. Drop the tool-call line if you do not need tool calling.
Python (OpenAI client)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
r = client.chat.completions.create(
model="qwythos-9b-nvfp4",
messages=[{"role": "user", "content": "Write a short mythic tale about a lantern that remembers the sea."}],
)
print(r.choices[0].message.content)
curl
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwythos-9b-nvfp4",
"messages": [{"role": "user", "content": "Write a short mythic tale about a lantern that remembers the sea."}]
}'
About the base model
An abliterated (uncensored) build of empero-ai/Qwythos-9B-Claude-Mythos-5-1M, a 9B
Qwen3.5-family vision-language model tuned for creative / Mythos writing with a Claude
distillation, and a 1M-token context window.
- 32 decoder layers: hybrid gated delta-net linear attention plus full attention, dense MLP, plus a vision tower for image and video input.
- 1M context (
max_position_embeddings1048576). - Thinking mode by default, with an instruct toggle. In non-thinking mode the base model adheres strictly to the first prompt's instructions (output only, no fluff).
Quantization
| Scheme | NVFP4, W4A4 |
| Weight rounding | GPTQ (Hessian-based error compensation), MSE observer |
| Weights | FP4 (E2M1), group_size=16, tensor_group, FP8 (E4M3) group scales, shared across fused layers |
| Activations | FP4, dynamic per-group, FP8 (E4M3) scales |
| Quantized | all language-model Linear layers |
| Kept in bf16 | vision tower (model.visual.*), lm_head, MTP head |
| Untouched | gated delta-net Conv1d and SSM params (A_log, dt_bias), never Linear |
GPTQ is a quantization-time cost only; inference speed and format are identical to plain round-to-nearest NVFP4, but it chooses better 4-bit values.
Calibration: 512 domain-matched samples (long reasoning + general chat + code),
max_seq_len=2048, text-only path through the VL model.
Recommended sampling
Thinking mode is the default.
- Thinking, general / creative:
temperature=1.0,top_p=0.95,top_k=20 - Instruct / non-thinking:
temperature=0.7,top_p=0.80,top_k=20 - To run non-thinking, set
{%- set enable_thinking = false %}in the chat template, or passextra_body={"chat_template_kwargs": {"enable_thinking": false}}.
Reproduction
Toolchain: llmcompressor==0.12.0, compressed-tensors==0.17.1, transformers==5.12.1,
torch==2.11.0+cu130, on an NVIDIA GB10 (Blackwell, sm_121). llm-compressor 0.12 shares
the NVFP4 global scale across fused layers automatically (q/k/v, gate/up).
Related
- Base model: huihui-ai/Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated
- Space: Rogue Quants
- Collection: NVFP4 Quants
- Sibling NVFP4 quants:
Notes
- Needs NVIDIA Blackwell (sm_121, e.g. GB10) for accelerated W4A4; pre-Blackwell GPUs run it weight-only.
--reasoning-parserand--tool-call-parserare not auto-detected; pass them explicitly.- Thinking mode is on by default; toggle it via the chat template or
chat_template_kwargs. - No refusal guardrails; you are responsible for how you use it.
License
Apache-2.0, following the base model. Intended use and all responsibility for use follow the base model.
Credits
- Base model: huihui-ai (abliteration) / empero-ai (Qwythos)
- Quantization tooling: llm-compressor / compressed-tensors