license: apache-2.0 base_model:
- orcarouter/Qwen3.8-Flash-Next-Uncensored base_model_relation: quantized pipeline_tag: image-text-to-text library_name: vllm language:
- en
- zh tags:
- qwen3.8
- flash-next
- qwen4_exp
- autoround
- gptq
- int4
- int8
- fp8
- w4a16
- w4a8
- mixed-precision
- moe
- mtp
- vllm
- gfx1100
- uncensored
- abliterated
Qwen3.8-Flash-Next-Uncensored-W4A16-Attn8-FP8PLE
A mixed-precision AutoRound quantization of orcarouter/Qwen3.8-Flash-Next-Uncensored
(revision 8336e613ea508b13c2159bd0f68965d97a606b95), the BF16 abliterated build of Qwen3.8-Flash-Next.
The routed experts were tuned against INT8 activations, so the checkpoint is a plain
W4A16 GPTQ file that also serves as the weight side of a W4A8 kernel. It targets a vLLM
plugin for AMD RX 7900 XTX (gfx1100) cards that runs the experts as INT4 weights times INT8
activations on RDNA3 integer WMMA (vllm_gfx1100, not part of upstream vLLM); any vLLM build
that loads AutoRound GPTQ MoE checkpoints can serve it as ordinary W4A16.
The layout mirrors todiadiyatmo/Qwen3.8-Flash-Next-W4A16-Attn8-FP8PLE, so it drops into the
same serving configuration (TP2 x PP2 with expert parallelism, FP8 PLE table in host RAM, optional
MTP draft head). Vision tower and MTP head are preserved.
Warning. The base model has had its safety alignment removed by abliteration. It will comply with harmful requests the original Qwen3.8-Flash-Next refuses. Use it only for legitimate research, red-teaming and evaluation, behind your own moderation. See the source model card for the full disclaimer; you assume responsibility for how you use it and for what it generates.
Recipe
| Component | Precision | How |
|---|---|---|
Routed experts, 512 per layer x 48 layers (gate_proj, up_proj, down_proj) | INT4, group 128, symmetric (zero point 8), BF16-exact scales | AutoRound SignRound, 200 iterations per block, activation-aware: expert inputs fake-quantized to INT8 per 128-wide group (absmax/127, round half away, clamp +-127), exactly the plugin kernel's scheme |
| Expert down projections | smoothed | AWQ-style channel scaling folded from down_proj input channels into up_proj rows (exact, no runtime cost), grid-searched per expert before tuning |
QSA attention q_proj, k_proj, v_proj, o_proj (12 layers) | INT8 per-channel symmetric (W8A16) | tuned in the same block passes |
GDN in_proj_qkv, in_proj_z, out_proj (36 layers) | INT8 per-channel symmetric (W8A16) | tuned in the same block passes |
| PLE n-gram lookup table (128 shards, 2560-dim) | FP8 E4M3FN + one global BF16 scale | storage format only; decoded to BF16 in software when rows are gathered |
| MTP draft head: routed experts | INT4, group 128, symmetric (RTN) | round-to-nearest with the same quantizer |
MTP draft head: attention, indexer, fc_embedding, fc_hidden | INT8, group 128, symmetric (RTN) | round-to-nearest |
Router, shared expert, hyper-connections, GDN control projections, indexer, PLE projections, norms, embeddings, lm_head, vision tower, rest of MTP | BF16 | unchanged |
Calibration: 128 windows of 2048 tokens packed from chat-templated
coding, reasoning (with <think> blocks), general chat, SFT, function-calling and web-text samples
(shares: {"code": 28, "reasoning": 37, "chat": 21, "sft": 13, "tools": 14, "web": 15}).
Scales were tuned in BF16 and stored as FP16, which is lossless for BF16 values, so the served scales equal the tuned ones
(944590848/944590848 scale entries are exact BF16 values).
Tooling: auto-round 0.16.0, transformers 5.17.0, torch 2.11.0+cu128, tuned on NVIDIA H100 80GB HBM3 in 5.8 h. The checkpoint was streamed block by block from disk; the PLE table was served from memory-mapped shards during calibration.
Files
model-*-of-*.safetensors: language model (AutoRound GPTQ packing:qweightint32,qzerosint32,scalesfp16), vision tower and BF16 layersmodel-plefp8-*.safetensors: PLE table shards (float8_e4m3fn) andweight_scalemtp-model-*.safetensors: quantized MTP draft headmodel.safetensors.index.json: merged index over all of the above (227679 tensors, 123.5 GB)config.json:quantization_config(auto-round /auto_round:auto_gptq) with per-layerextra_config,text_config.ple_embedding_dtype = float8_e4m3fn
Verification
- Every index entry resolves to a tensor in its file; every source tensor is present, quantized, or replaced by design (1658/1658).
- PLE FP8 dequantization error on the first shard: max abs 0.00299, relative RMS 0.0266.
- MTP GPTQ packing reproduces the quantized weights exactly and stays within one scale step of the source on 13 sampled tensors.
- Quality screens (task accuracy against the BF16 source) have not been run yet; treat this as a serving candidate until they are.
Serving
With the gfx1100 plugin: point its launcher (vllm-gfx1100, serving-config.json) at this repo; VLLM_GFX1100_MOE_W4A8=1 (its default) selects the INT8-activation expert path.
Elsewhere: vllm serve valoomba/Qwen3.8-Flash-Next-Uncensored-W4A16-Attn8-FP8PLE --quantization auto-round --dtype bfloat16 ... with a build that supports qwen4_exp.
MTP is optional: --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
Provenance
- Source:
orcarouter/Qwen3.8-Flash-Next-Uncensored@8336e613ea508b13c2159bd0f68965d97a606b95(Apache-2.0) - Quantization scripts:
plugins/gfx1100/quant/in the plugin repository (calibration builder, auto-round run, post-processing) - Run record:
quantization_run.json; verification:verification.json