Repository navigation
Flash Next FP4-Local on one GB10: measured gains over 1.1.5 we can offer as PRs #1367
Description
We serve Qwen3.8 Flash Next FP4-Local on one DGX Spark (GB10) with TensorFold 1.1.5 plus four patches of our own. The patches and every raw result are public in our recipe, https://github.com/Arminova/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold. We would like to offer them upstream as small pull requests, one change each, with the receipt your template asks for. The first, the expert loader, follows as a pull request that links here.
Both servers below were measured on the same GB10 on the same day with the same tools. Settings: nvidia/Qwen3.8-Flash-Next-FP4-Local at b9ce18a6, --context 262144 --parallel 3, prompt cache off except in the cache row, and TF_FLASHNEXT_DRAFT_HEAD=affine4 TF_FLASHNEXT_CHUNK=4096 on both.
| 1.1.5 | 1.1.5 + our patches | |
|---|---|---|
| Ready after start | 170 s | 75 s |
| Prompt, ~3,900 / ~15,300 tokens (tok/s) | 1,731 / 1,832 | 2,269 / 2,266 |
| Prompt, ~146,000 / ~246,000 tokens, one stream (tok/s) | 1,767 / 1,700 | 2,069 / 1,858 |
Mixed load, 1 / 2 / 3 streams (bench_concurrent --mixed, tok/s) |
69.3 / 80.2 / 96.0 | 75.3 / 91.4 / 116.4 |
By kind of text, code / explanation / prose / German (accept_probe, tok/s) |
65.3 / 48.8 / 36.2 / 39.5 | 74.9 / 53.6 / 41.3 / 46.1 |
| Follow-up on a ~141,000-token document, time to first token (prompt cache at the engine's size) | 78.3 s | 4.3 s |
| Task suite (130 items) | 122 | 123 |
With our optional FP8 KV cache, five streams of 262,144 tokens fit on the same box:
- prompts run at 2,117 / 1,967 tok/s at ~146,000 / ~246,000 tokens
- the mixed load reaches 129.1 tok/s at 5 streams
- the task suite scores 124
On our build, every reply checked was equal across these pairs:
- concurrent and solo: 28 of 28 (
bench_concurrent --alone --serial) - drafted and
"draft": false: on the bench prompts, and on 300- to 15,000-token prompts by reply hash - a prompt resumed from the prompt cache and the same prompt fresh: by reply hash, greedy and seeded sampled
What we can offer, grouped by what it does to the bits. Each number was measured with that change's switch off against on, on our build (1.1.4 plus the three FP4-Local PRs that are now in 1.1.5), on the same box on 10-25:
-
Same bits as 1.1.5:
- load time (the first pull request): the model loads in about 54 s instead of about 146 s with upstream defaults, with the same replies
- requests sent together starting in lockstep: four identical requests sent at once, 165 to 236 tok/s (code) and 119 to 166 (chat)
- loaded weight files leaving the page cache, so the n-gram table stays in memory: prompts +9.2 % at ~3,900 tokens and +3.9 % at ~15,300
- two prompt-path changes that keep every reply equal: +2.3 % / +1.8 % and +1.7 % / +2.0 % on the same two prompt sizes
-
Changes rounding:
- two prompt-path changes: +4.3 % and +4.6 % on prompts
- the language-model head: +5.5 % at one stream (code), +6.5 % / +5.5 % / +2 % on the mixed load at 1 / 2 / 4 streams
- the optional FP8 KV cache: five streams of 262,144 tokens, and prompts 2-6 % faster
Per CONTRIBUTING, these need a gate agreed first. With all of them on, our task suite scores 123 / 130 (bf16 cache) and 124 (FP8), against 1.1.5's 122.
Two questions:
- Which of these would you like, and in what order?
- For the ones in group 2, which gate do you want? For example, FP64 references per projection class, or teacher-forced top-token agreement.
Activity
ashhart commentedon Oct 28, 2026
Thanks @Arminova, and sorry for the wait. The expert loader shipped in 1.1.8 (#1368).
Group 1 (same bits): yes to all of them, one pull request each. We'd take the page-cache change next, as it has the largest prompt gain. Each pull request needs:
"draft": false, and resumed against fresh;TF_FLASHNEXT_*ones;SUMMARYline for any kernel change.Group 2 (rounding): we'll pass on an FP8 KV cache for Flash Next. We no longer add lossy formats, even opt-in; the older
--kv-cache fp8for the Qwen hybrids and GLM predates that rule. The language-model head and the two prompt changes can come if they keep the checkpoint's activation precision and fp32 sums and only change the order. The gate is the one settled on #1353:Anything that lowers a precision is out. Before you start on receipts, could you say in a line each what those three change? The task suite is good to see, but it isn't the gate.
Arminova commentedon Oct 28, 2026
Thanks, Ash, that's clear. To answer your question, a line each:
We'll close this issue for now. Our next Flash Next work is a larger change, and we'll open it as its own issue with its receipts when it's ready. We run every receipt on one GB10 next to a day job, so it may take a little while.
ashhart commentedon Oct 28, 2026
Thanks @Arminova, that's clear, and withdrawing the two that lowered a precision is the right call.
The prompt tile is a same-precision reorder, so when it comes it's judged as on #1353 and #1409:
Closing this is fine, and we'll look out for the larger change's issue.