license: apache-2.0 base_model: Qwen/Qwen3.8-Flash-Next tags:

• gguf
• qwen
• qwen3.8
• qwen4exp
• mixture-of-experts
• quantized
• mixed-precision
• uncensored
• strata
• xolotl
• consumer-gpu language:
• en
• zh
• code pipeline_tag: text-generation

Qwen3.8-Flash-Next · custom mixed-tier, abliterated

A 125B-parameter MoE (6B active) running on one gaming PC: RTX 4090 (24 GB) + 90 GB RAM + NVMe, at 152 tok/s decode and 4,285 tok/s prefill.

Runs on Xolotl, an engine built for exactly this model and this class of machine, on top of Strata.

This is a refusal-abliterated build: a rank-2 refusal-direction edit was applied to the model before quantisation. The edit touches the dense backbone; the routed experts are untouched.

What it is

Qwen3.8-Flash-Next is a sparse MoE with 48 layers and 512 routed experts per layer, plus a 51.2B n-gram (PLE) table held off-accelerator. QwenLM describes the architecture as "an early preview of the architecture used in Qwen4", and the GGUF's arch string is literally qwen4exp.

This is not a llama.cpp / Ollama / LM Studio GGUF. It uses per-expert quantisation types those runtimes do not load — they expect one type per tensor, and here each layer's experts can carry different formats. Use Xolotl.

How the bits are spent

Most of a MoE's parameters live in experts that any one token touches rarely, so a single global bit-width is the wrong shape for this model. Instead:

• Every routed expert gets its own format. The gate/up projections and the down projection are chosen separately, per layer, from a tier ladder measured on real routed tokens.
• The down projection stays at a higher precision than gate/up. A down projection writes directly into the residual stream, so its error is not averaged away the way gate/up error is. This is also what preserves the abliteration edit, which lives in the down path.
• The dense backbone is Q8_0. On this file the Q8_0 emit is exactly identity (the source was already Q8_0, and amax/127 recovers the block scale bit-exactly), so the backbone is bit-identical to the parent at +2.4 GiB over the nearest lossy tier. That is a deliberate quality choice, not an oversight.
• The PLE table is a separate tier and is read from disk with O_DIRECT rather than held in RAM or VRAM — each row is read whole and its errors do not average out, so it gets a precision of its own.

The tier assignment is produced by a calibration-driven allocation. The allocation method, the screening procedure and the tier ladder's derivation are not published.

Format mix in this file

Speed

Measured on this machine: Linux, RTX 4090 24 GB, Ryzen 7 5700X, 90 GB DDR4-3200, model on a 3.5 GB/s NVMe. 16,383-token prompt, 1,024 generated tokens, greedy:

Decode is bound by RAM bandwidth once the expert arena is warm, so numbers scale with memory speed more than with the GPU. Longer generations read higher than short ones; a 64-token reply measures lower than this table.

Tags: xol-qfn.

Quality

Being measured — placeholders below will be filled in before this card is announced.

Gauntlet (the core 7 tasks, 10 turns each, hidden tests graded after every turn, thinking at the model's default xhigh effort, greedy):

Task 03 sits at 0.50 for every model tested, the API included. Task 14 is the only separation: the API did not solve it (0.00), this quant did at turn 9. On task 01 this quant solved at turn 5 where the API needed 9.

Treat this as one run per model, not a ranking: run-to-run variance on these tasks is large, and the API's serving precision is not something we can verify. What it does establish is that the quantisation has not damaged the model — which is the question a quant card owes an answer to.

Refusal probe (6 boundary prompts across violence / adult / drugs / security, greedy):

Top-1 agreement and KL against the full-precision edited model are still being measured; they are deliberately not estimated.

What is already established: the quantisation was validated by reading the output file (arch, tensor count, per-tensor types) rather than by trusting the quantiser's exit code, and the abliteration was verified to survive quantisation by measuring the component of the activation change along the edit direction — 0.96–1.01 retained across early, middle and late layers. A whole-tensor norm ratio is the wrong metric for that test and was not used.

Roadmap

Being explicit about V1's rounding, because it matters: the tier allocation, per-expert screening and importance measurement are ours, but the rounding in V1 is llama.cpp's round-to-nearest. Our GPTQ plus GSQ-style refinement (the path the V4 release used) was not wired into the QFN chain. That is a real gap, not a rounding of words — RTN and GPTQ+refine are different products at the same byte size. V2 closes it.

Swift's speed does not come from a serving flag: the token reduction lives in fine-tuned weights, so V2 requires the Swift 1.5 BF16 master rather than a re-quantisation of this file.

Quick start

git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DSTRATA_ENABLE_CUDA=ON -DSTRATA_GGML_CUDA=ON -DSTRATA_MMQ_KQUANTS=ON \
      -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp
ninja -C build-xolotl strata

Then serve it with the config below (strata-xol-qfn.json, port 8099). hf download the model and its pack directory first; the pack is what teaches the engine this model's per-expert layout.

The engine treats the experts as a cache: the hottest ones in VRAM, the rest pinned in RAM, with STRATA_ROUTE_RESIDENT preferring whichever tier is already resident. That bias is why the decode number above is possible on a 78.83 GiB model in 90 GB of RAM.

Credits

• Qwen for Qwen3.8-Flash-Next (Apache-2.0).
• mrfakename for the rank-2 refusal-direction edit style.
• Niko1221 for Strata, the tiered MoE engine this builds on.
• ggml / llama.cpp for the compute backend.
• Abliteration, quantisation, engine port and measurements: Indra's Mirror (Xolotl).

Support

If this has been useful and you want to help keep it going, you can buy me a coffee.