base_model: Qwen/Qwen3.8-Flash-Next base_model_relation: quantized library_name: mlx-serve license: other license_name: qwen-community-1.0 license_link: LICENSE pipeline_tag: text-generation tags:

• mlx
• mlx-serve
• qwen4_exp
• moe
• sparse-attention
• ngram-embedding

Qwen3.8-Flash-Next for mlx-serve (4-bit experts, 8-bit rest)

mlx-serve pack of Qwen/Qwen3.8-Flash-Next, the Qwen4 preview architecture (model_type: qwen4_exp). About 63 GB resident (128 GB Macs). Includes the MTP head and the vision tower (image and video input).

mlx-serve --model ddalcu/SamQuant--Qwen3.8-Flash-Next-Uncensored-MLX-Serve-Mix-C58 --serve

Measured on an M4 Max 128 GB (mlx-serve, first port, no tuning yet): ~67-69 GB resident, decode 29-34 tok/s serial, prefill ~400 tok/s on a 25k-token prompt, a needle at 24.8k tokens recovered with sparse attention engaged. The MTP head loads and drafts (~1 accepted token per round) but its round cost is not yet competitive with serial decode, so leave it off for now.

What is different about this model

This is not a Qwen3.5-style pack. Three things around the usual GDN + MoE trunk:

• Gated residual streams. The residual is 4 streams wide (4 x 2560). Every block reads a sigmoid-mixed average of the normalized streams and writes back through per-stream scalar gates. The final mixer replaces the usual final norm.
• N-gram embedding (51B parameters). A second embedding table indexed by hashed bigrams and trigrams of the token ids: 16 heads, each a prime-sized bucket space of ~20M rows, 160 dims per row, injected once before layer 1. It is a lookup, no compute, which is why Qwen quotes the model as 125B: the full checkpoint is 125B trunk + 51B n-gram + 4B MTP = 180B (360 GB bf16).
• Qwen Sparse Attention. Past 2048 tokens each attention layer only reads the 512 most relevant 4-token blocks per query (picked by a small indexer), plus the query's own partial block. Attention cost stays flat with context. Native 262k context.

How this pack stores the n-gram table

The 51B table is NOT in the safetensors shards. It is one merged 4-bit table in ngram_table.bin (32.0 GB, safetensors format, .bin so nothing mlx-loads it). mlx-serve mmaps the file and, per token, dequantizes the 16 rows it needs on the CPU (16 x 80 bytes) and uploads only the resulting 2560-vector. The table never becomes resident: its cost is page cache, which the OS evicts as needed. That is the difference between this pack and mlx-lm style packs that ship the table as 128 quantized tensors and load it onto the GPU (+32 GB resident, ~107 GB total for a 4-bit pack).

Expected effect: decode speed unchanged (16 tiny reads against a ~20 ms step), cold-cache prefill of very long prompts may pay up to ~1 s per 8k tokens of random reads on the SSD, warm cache is free. No user-space cache is needed, the page cache already is an LRU over exactly this access pattern.

Widths

Every (1 + w) RMSNorm has the +1 folded into the stored weight; depthwise convs are transposed to MLX's [C, K, 1]; experts.gate_up_proj is split into switch_mlp.gate_proj / up_proj. The vision tower ships dense bf16 in model-vision.safetensors (~0.9 GB).

Serving notes

• Memory. ~63 GB resident plus KV cache. mlx-serve sizes the context to what fits; --kv-quant 8 halves the cache.
• MTP. The checkpoint's own 1-layer speculative head is loaded from the pack and works (--mtp or per-request "enable_mtp": true), but as of this build it decodes slower than serial. Default-off; a later mlx-serve release will flip it once the round cost is fixed.
• v1 limits in mlx-serve. One request at a time (no batched decode), no prefix-cache reuse between turns yet, PLD/DFlash speculation off (MTP is the speculative path). Very long prompts (past ~64k) want a smaller --prefill-chunk because the sparse-attention selection is built per chunk.
• Thinking is on by default ("enable_thinking": false turns it off). Tools use Qwen3.8's XML call format; mlx-serve parses and schema-coerces it.
• Images and video go through the Qwen3-VL-style tower (model.visual.*, dense bf16). MTP is declined on image turns (serial decode).

Conversion

tests/convert_qwen38_flash_next.py in the mlx-serve repo. It streams the 360 GB bf16 checkpoint shard by shard from the Hub (download, quantize, delete), so it converts on a machine with ~150 GB free. The engine was validated against HF transformers (trunk) and the vLLM/SGLang MTP math on a tiny random model before the full conversion.