base_model:

• huihui-ai/Huihui-Qwen3.5-9B-abliterated base_model_relation: quantized license: apache-2.0 license_link: https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE library_name: transformers pipeline_tag: image-text-to-text tags:
• quark
• quantized
• mxfp4
• awq
• qwen3_5
• abliterated
• uncensored
• rocm
• rdna4
• vllm

Huihui-Qwen3.5-9B-abliterated-Quark-AWQ-MXFP4

AMD Quark AWQ + MXFP4 (OCP, group 32, E8M0) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated, produced to run on AMD RDNA4 (gfx1201) under vllm-radiance.

Vision tower, lm_head and the MTP head are left BF16, so this is still a working vision-language model — not a text-only quant.

Why this exists

MXFP4 checkpoints that actually land on RDNA4's qualified W4A8/MXFP4 kernels are scarce. AMD publishes amd/Qwen3.8-27B-Quark-AWQ-MXFP4 for the stock model; this repo is the same recipe applied to the abliterated variant, so you can have the uncensored weights and the fast kernels. Run on 2x Radeon AI PRO R9700.

Recipe

Replays AMD's exact configuration, read back from their checkpoint's config.json.orig_with_algo_config, so the result lands on the same kernels:

Layers: 32 decoder layers, hidden 4096. Conversion took 34 min (2041 s) on one R9700 (whole model in host RAM, one decoder layer on GPU at a time; peak ~20 GiB VRAM).

The architecture is hybrid: full_attention_interval=4, so only 1 layer in 4 holds a KV cache and the rest keep a fixed-size recurrent (gated delta net) state. KV is therefore much cheaper than a dense model of the same size, and --max-num-seqs barely moves VRAM.

Serving

--attention-backend=R4D does NOT work on this model. Radiance's R4D attention kernel requires a GQA ratio of exactly 6; this checkpoint is 16 q / 4 kv = 4, and it fails at engine init with:

NotImplementedError: R4D attention does not support gqa=4, needs == 6.

The GQA ratio is invariant under tensor parallelism, so no --tensor-parallel-size will fix it. Use ROCM_AITER_UNIFIED_ATTN (still AITER-accelerated) or TRITON_ATTN. The 27B sibling linked above is GQA 6 and does run on R4D.

Known-good vLLM arguments:

vllm serve <this-repo> \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --reasoning-parser qwen3 \
  --trust-remote-code

--mamba-cache-mode align is needed for prefix caching to work with the hybrid layers.

Gotchas

• quantization_config.algo_config is stripped to null in config.json, matching AMD's published checkpoint. The pre-strip version is kept as config.json.orig_with_algo_config for anyone reproducing the recipe.
• Quark 0.12's preprocess_for_quantization raises on qwen3_5, and the image's Quark has no dense qwen3_5 LLMTemplate, so the QConfig is built directly and preprocessing is skipped. There is nothing to unfuse in a dense model, so this is safe.
• Cold start under Radiance is ~3 minutes (imports, weight load, torch.compile, HIP graph capture). Size your health-check timeouts accordingly; the compile result is cached per config, so subsequent starts are much faster.
• vLLM 0.30 returns reasoning in a field named reasoning, not reasoning_content.

Provenance and intent

This is a format conversion only. The abliteration is huihui-ai's work, not mine, and no additional guardrail modification was performed — dequantized weights land within a few percent of the BF16 source. The upstream weights are Apache-2.0 and already public; this repo adds a quantization that runs on AMD consumer/prosumer hardware.

Because these are abliterated weights they will not refuse much. That is the point of the upstream model, and it is your responsibility to use them lawfully and to add your own safeguards if you expose them to other people.

Credits

• huihui-ai — the abliterated source model
• Qwen — the base model
• AMD Quark and amd/Qwen3.8-27B-Quark-AWQ-MXFP4 — the recipe
• magiccodingman/vllm-radiance — the RDNA4 runtime