license: apache-2.0 base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags: [qwen3.8, moe, nvfp4, fp8, compressed-tensors, vllm, dgx-spark, gb10, 1m-context, yarn, mtp, tensor-parallel]
Qwen3.8-Flash-Next-Uncensored-NVFP4 — 2× DGX Spark (GB10) build
A byte-identical mirror of orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4
@ 56eb374a747fe0c359983553d869c27b86d1c8ce (Apache-2.0, original card in README.orcarouter.md),
plus everything needed to reproduce a production deployment on two NVIDIA DGX Spark (GB10) nodes: pre-exported FP8 PLE
tables, the vLLM patches, the launcher and the measured throughput below.
Headline: 1M context, 3.76M-token KV pool, 3.7k tok/s prefill, 63–66 tok/s per stream, 236 tok/s at 8 streams
At a glance
| metric | value |
|---|---|
| max context | 1,048,576 tokens (YaRN ×4 over native 262,144) |
| KV cache pool | 3.76 M tokens (FP8 KV + FP8 indexer, 28 GB budget) |
| prefill | 3,723 tok/s at 8k prompt · 2,737 tok/s at 1 M |
| decode, one stream | 63–66 tok/s (MTP speculative decoding, 2 draft tokens) |
| decode, 8 streams | 236 tok/s aggregate |
| time to first token, 1 M-token prompt | 6.2 min |
| setup | 2 × DGX Spark (GB10, 128 GB each), tensor parallel 2, nodes linked by one direct 200 GbE cable (RoCE v2), vLLM nightly 0.30.1rc1.dev616 |
How the nodes are wired and what each technique does: docs/.
Documentation
| page | what it covers |
|---|---|
| docs/README.md | index of all documentation |
| docs/network-architecture.md | how the two Spark nodes are connected: ports, the one direct 200 GbE cable and its two PCIe halves, LAN/storage planes, NCCL settings, fabric capacity vs use |
| docs/techniques.md | every technique in the stack: FP8/NVFP4, TP=2, YaRN 1 M, QSA + FP8 KV/indexer, MTP, CUDA graphs, GPU-direct PLE, memory map |
| docs/fabric.md | InfiniBand / RoCE fabric traffic from the ib_port_* metrics: traffic per PCIe half, peaks vs link capacity, error counters |
| docs/memory.md | how memory is used on each node: unified GPU/CPU memory (weights, KV cache pool, CUDA graphs), page cache and host processes, NVMe disk |
| docs/metrics/README.md | production metrics from VictoriaMetrics since 2026-08-31: traffic, tokens, cache hits, latency percentiles, per-era comparison, raw data |
| docs/timeline.md | project timeline: ideas, changes, improvements, rejected experiments and incidents, with a visual timeline |
| docs/lessons-learned.md | experience: what went wrong, why, and the rule we follow now |
| deploy/README.md | how to use every script and config in deploy/ to reproduce the deployment |
1. What is in this repo
| path | size | content |
|---|---|---|
model.safetensors.index.json + shards 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 | 183 GB (170 GiB) | checkpoint @56eb374a |
model-mtp.safetensors | MTP head for speculative decoding | |
ple-fp8/: rank 0 .bin · .json · rank 1 .bin · .json | 2 × 25.6 GB | per-TP-rank FP8 export of the layer-1 PLE (n-gram embedding) table, with scale + checksums |
deploy/ | launcher, configuration, vLLM patches, PLE tools, benchmark and quality scripts to reproduce the deployment — every file is explained in deploy/README.md | |
docs/ | network architecture, techniques, production metrics, timeline, lessons learned — see Documentation | |
assets/ | README charts (SVG) |
2. The model
| architecture | Qwen3.8-Flash-Next (Qwen4-exp), MoE, vision-language, QSA sparse attention with indexer, PLE n-gram embedding, MTP head |
| quantization | weight-only, compressed-tensors: dense layers + lm_head FP8 per-channel (W8A16), experts NVFP4 (W4A16) — no activation scales, so W4A4 kernels do not apply; MoE runs on Marlin |
| native context | 262,144 → 1,048,576 with YaRN ×4 |
| PLE table | 320 M-row vocabulary × 160, BF16 in the checkpoint (~95 GiB); served as FP8 per-rank shards from NVMe |
3. Hardware and topology
| nodes | 2 × NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified memory each, sm_121a) |
| parallelism | tensor-parallel 2 (rank 0 = head + OpenAI compatible API, rank 1 = headless worker) |
| interconnect | one direct 200 GbE QSFP cable between ConnectX-7 port 1 of each node (200 Gb/s per direction), RoCE v2; NCCL drives both PCIe x4 halves of that port (NCCL_IB_HCA==rocep1s0f1,roceP2p1s0f1), MTU 1500 (MTU 9000 measured ~0 % gain). Port 0 goes to the storage switch (NFS) |
| storage | checkpoint + PLE tables on local NVMe (~8 min cold boot) |
4. Software
| component | version |
|---|---|
| vLLM | nightly 0.30.1rc1.dev616+g1a001d584 |
| torch | 2.13.0+cu130 |
| flashinfer | 0.7.0.post1 |
| transformers | 5.16.1 (pinned, see lessons learned L16) |
5. Serving configuration (key flags)
vllm serve orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 --revision 56eb374a \
--tensor-parallel-size 2 --nnodes 2 --node-rank <0|1> --master-addr <head-ip> --master-port 25000 [--headless on rank 1] \
--max-model-len 1048576 \
--hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn",
"rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144},
"max_position_embeddings":1048576}}' \
--kv-cache-dtype fp8 --attention-config '{"indexer_kv_dtype": "fp8"}' \
--kv-cache-memory-bytes 28000000000 \
--engram-config '{"cpu_offload": false}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,3,4,6,8,10,12],"max_cudagraph_capture_size":12}' \
--max-num-seqs 16 --max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image": 64, "video": 2}' --chat-template chat_template.qwen38.jinja
The authoritative, complete version is deploy/qwen38.env + deploy/build/run-node.sh.
Lessons learned while getting here (YaRN override shape, memory traps, NCCL NCCL_IB_HCA setup, never compiling on a serving node, benchmark noise) are collected in docs/lessons-learned.md.
6. PLE table: FP8 on NVMe, read GPU-direct
The PLE n-gram table is the largest single tensor (~95 GiB BF16). Instead of pinning it in RAM, each rank mmaps its FP8 shard
from local NVMe and the GPU reads the mapped pages directly (GB10 has pageableMemoryAccess=1, i.e. the GPU walks host page tables).
A Triton gather kernel (qwen38_ple_uva.py) does the lookup inside the FULL decode CUDA graph through vLLM's own
VocabParallelEmbedding path (mask → quant_method.embedding → int8-sum all-reduce).
| CPU gather (before) | GPU-direct (now) | |
|---|---|---|
| lookup latency per step | 37–45 µs | 15–16 µs |
| host RAM held per rank | 47.7 GiB pinned BF16 (upstream default) | 0 pinned; 23.8 GiB FP8 in page cache (any residency works) |
Setup: put ple-fp8/*rank0of2.* on node 0 and *rank1of2.* on node 1, in the directory QWEN38_PLE_MMAP_DIR points to
(QWEN38_PLE_MMAP=1). The json carries the revision, row range, scale and head/tail sha256 — the loader refuses a mismatched shard.
Apply deploy/build/patches-next/ with apply-patches-next.sh (patch 10 is sha-guarded against the vLLM file it modifies).
To regenerate the tables from the checkpoint: deploy/build/ple_export.py (~100 s per rank).
7. Throughput
Method: deploy/build/upg_bench.py against the live OpenAI compatible endpoint, 2026-10-03. Median of 3 repeats (524 k and 1 M: single run).
Prefill uses unique prompts (no prefix-cache hits); decode uses one fixed prompt with ignore_eos and MTP n=2.
Prefill
Prefill throughput by prompt length
Prefill tok/s (time to first token) — 8k: 3,723 (2.2 s) · 32k: 3,682 (8.7 s) · 131k: 3,592 (35.3 s) · 524k: 3,154 (2.7 min) · 1 M: 2,737 (6.2 min). Full numbers in the table below.
Time to first token vs prompt length
Time to first token grows about linearly with prompt length — 2.2 s at 8k, 8.7 s at 32k, 35.3 s at 131k, 164.8 s at 524k, 370.1 s at 1 M (≈ 0.27 ms per prompt token up to 131k, 0.37 ms at 1 M).
Prefill speed kept as prompts grow
Prefill speed as a share of the 8k speed (3,723 tok/s) — 32k: 99 %, 131k: 96 %, 524k: 85 %, 1 M: 74 %.
Prefill under concurrency
With 8 simultaneous unique prompts the total prefill rate stays ~3.7k tok/s, so each request waits longer — 8k prompt: 2.2 s alone vs 11.7 s with 8 at once; 32k prompt: 8.7 s alone vs 44.7 s with 8 at once.
| prompt tokens | concurrency | prefill tok/s | TTFT p50 |
|---|---|---|---|
| 8,192 | 1 | 3,723 | 2.2 s |
| 8,192 | 8 | 3,690 | 11.7 s |
| 32,768 | 1 | 3,682 | 8.7 s |
| 32,768 | 8 | 3,669 | 44.7 s |
| 131,072 | 1 | 3,592 | 35.3 s |
| 524,288 | 1 | 3,154 | 164.8 s |
| 1,000,000 | 1 | 2,737 | 370.1 s |
Decode
Decode throughput
1,024 generated tokens — 1 stream 62.4 tok/s; 8 streams 235.6 tok/s total, 30.5 per stream. 8,192 generated tokens — 1 stream 65.8 tok/s; 8 streams 223.2 tok/s total, 29.6 per stream.
Decode scaling 1 to 8 streams
Going from 1 to 8 concurrent streams multiplies total output by 3.8× (1,024-token generations: 62.4 → 235.6 tok/s) and 3.4× (8,192-token generations: 65.8 → 223.2 tok/s).
| generated tokens | concurrency | aggregate tok/s | per-stream tok/s | MTP accept |
|---|---|---|---|---|
| 1,024 | 1 | 62.4 | 63.0 | 0.70 |
| 1,024 | 8 | 235.6 | 30.5 | 0.69 |
| 8,192 | 1 | 65.8 | 65.9 | 0.75 |
| 8,192 | 8 | 223.2 | 29.6 | 0.72 |
Interconnect
Interconnect traffic between the two nodes
RoCE traffic on the head node (send + receive, both PCIe halves of the 200 GbE link) — prefill 8k ×1: 29.4 Gb/s · prefill 32k ×8: 32.8 Gb/s · prefill 131k: 32.0 Gb/s · prefill 1 M: 24.5 Gb/s · decode 1 stream: 2.3 Gb/s · decode 8 streams: 8.1 Gb/s. The link carries ~185 Gb/s in one direction when both PCIe halves are driven (RDMA benchmark; 200 Gb/s cable rate); see docs/network-architecture.md.
Capacity
| KV cache pool (28 GB budget, FP8 KV + FP8 indexer) | 3,756,138 tokens |
| max context per request | 1,048,576 |
| max concurrent sequences | 16 |
| cold boot (NVMe, both ranks) | ~8 min |
8. Long context and quality
| check | result |
|---|---|
| needle at 962,348 tokens (depth 0.25) | answer 094KBVOY (expected 094KBVOY), TTFT 347 s |
| needle at 942,275 tokens (depth 0.75) | answer OMTJFSCH (expected OMTJFSCH), TTFT 338 s |
| teacher-forced top-1 agreement vs previous build, 63 fixed documents | 97.75 % (same-build noise floor 98.05 %) |
| mean NLL drift vs previous build | +0.019 % (noise floor −0.09 %) |
FP8 KV + FP8 indexer cost no measurable quality on this probe while doubling KV capacity.
9. Quick start
Short version below; every step, file and variable is explained in deploy/README.md.
# on BOTH nodes: checkpoint in Hugging Face cache layout (run-node.sh loads it by repo id, offline)
HF_HOME=~/hf-cache hf download orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 --revision 56eb374a747fe0c359983553d869c27b86d1c8ce
# this repo's deploy/ and ple-fp8/ (node 0 needs *rank0of2.*, node 1 needs *rank1of2.*)
hf download quocbao747/Qwen3.8-Flash-Next-Uncensored-NVFP4-2xGB10 --include 'deploy/*' 'ple-fp8/*' --local-dir ~/qwen38
cd ~/qwen38/deploy
bash build/make-venv-next.sh && bash build/apply-patches-next.sh # vLLM nightly dev616 + patches 02, 03, 10
bash build/prebuild-jit-next.sh # with the model STOPPED, never on a serving node
# edit qwen38.env (IPs, NCCL_IB_HCA, socket interfaces, HF_CACHE_DIR_LOCAL, QWEN38_PLE_MMAP_DIR);
# put VLLM_API_KEY=... in qwen38.local.env (chmod 600)
ROLE=worker bash build/run-node.sh # node 1
ROLE=head bash build/run-node.sh # node 0 -> OpenAI compatible API on :8081
10. Licence and attribution
Weights: Apache-2.0, from orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4
(itself derived from Qwen/Qwen3.8-Flash-Next) — see README.orcarouter.md for the original card, intended use and caveats of the uncensored variant.
Deployment scripts and patches in deploy/: Apache-2.0.