base_model:
- Ar4ikov/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM
- orcarouter/Qwen3.8-27B-Uncensored base_model_relation: quantized license: apache-2.0 library_name: vllm pipeline_tag: image-text-to-text tags:
- qwen3.8
- qwen3_5
- awq
- w4a16
- asymmetric
- compressed-tensors
- hyperqwen
- speculative-decoding
- vllm
- vision
- uncensored
Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast
Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM prepared for
HyperQwen — Qwen3.8-27B served fast on one 24 GB card — the way the
vllm-hyprfastQwen image expects it. This is the uncensored (abliterated) finetune; its behaviour is inherited from orcarouter/Qwen3.8-27B-Uncensored. The body is untouched: int4 asymmetric
AWQ, group 128, zero points, quantized with llm-compressor from bf16 weights; the vision
tower, the SSM gate projections and the MTP head's norms stay bf16. What changed is what
HyperQwen's prepare/ pipeline changes, so that a 24 GB card has room for a KV cache and
speculative decoding has something small to score:
| tensor | in the source export | here |
|---|---|---|
lm_head | bf16, 2.5 GB | int4 GPTQ (group 128, symmetric), calibrated on 400k of the model's own final hidden states (drafter/gptq_lm_head.py; RTN int4 KL 0.0070 -> GPTQ 0.0024 against the bf16 head) |
embed_tokens | bf16, 2.5 GB | int8 (group 128, symmetric), round-trip error 0.65% |
MTP module (mtp.*, 8 linears incl. mtp.fc) | bf16, 850 MB | int8 (group 128, symmetric) |
mtp.draft_lm_head + mtp_draft_vocab_ids.pt | — | 40,960-row draft head (int4, sliced from the int4 lm_head), HyperQwen's shipped id list |
| body (64 layers), vision tower | int4 asym g128 / bf16 | unchanged |
Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen keeps lm_head in int8 instead: +0.6% perplexity less, a few tok/s less.
Serving
The container does everything (download, verify, serve), with the vision tower on:
git clone https://github.com/Ar4ikov/vllm-hyprfastQwen && cd vllm-hyprfastQwen
cp .env.example .env # CHECKPOINT=uncensored and MODEL=/app/models/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast
docker compose --profile single up -d
Bare metal, on the fork branch this was measured with (Ar4ikov/HyperQwen@awq-asym,
vLLM 0.29.0 + the series + marlin-int8-asym-zp):
hf download Ar4ikov/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast --local-dir models/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast
VISION=1 MODEL=$PWD/models/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast SPEC=mtp CTX=fast bash single-user/start_qwen.sh
Plain vLLM 0.29 also loads it (vllm serve Ar4ikov/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast --max-model-len 65536), without the
speculative decoding, the draft head or the int8 Marlin path that need the patch series.
Measured
RTX 3090 (350 W), vLLM 0.29.0, HyperQwen bench/run_benchmarks.sh single, second run
after boot, VISION=1 (the tower streamed from host RAM per image). C1 = one stream of
real prompts with 1,024-token answers; tok/step = tokens accepted per forward pass.
| profile | C1 T=default | C1 T=0 | tok/step | C8 T=default | KV pool |
|---|---|---|---|---|---|
SPEC=mtp CTX=fast (64k, bf16 KV) | 112.7 tok/s | 127.6 | 2.74 / 2.95 | 434 tok/s | 80,185 |
SPEC=dflash2 CTX=fast KV_MEM=4600000000 DFLASH_MAX_LEN=49152 | 135.9 | 139.3 | 3.29 / 3.29 | 401 | 53,233 |
production line: DFlash2 k=15 + INT8_ACT=int8 PREFILL_ATTN=int8 (40k; TTFT 96 ms) | 116.7 | 135.5 | 2.97 / 3.44 | n/a (4 slots) | 42,113 |
Images are described correctly in every profile (a drawn red square, blue circle and a
line of text: boost/image_smoke.py). The whole table, the int8 (W4A8) profiles and the
kernel measurements: github.com/Ar4ikov/vllm-hyprfastQwen.
Why a separate repo
HyperQwen's pipeline rewrites the checkpoint in place (int8 heads, the draft head) and its launchers, verify script and Docker entrypoint expect that layout. Doing it once and publishing the result turns a ~10-minute CPU step per machine into a download, and keeps the original export as it is for transformers and plain vLLM users.
Quantization recipe of the body, calibration set and the AWQ mappings for the hybrid
Gated-DeltaNet / attention layers: see the source export's card and its recipe.yaml
(kept here). Root model: Qwen/Qwen3.8-27B, Apache-2.0.