Repository navigation
Flash Next CUDA: serve nvidia/Qwen3.8-Flash-Next-NVFP4 on one GB10 and at --tp 2 #1754
Description
The native CUDA build does not load nvidia/Qwen3.8-Flash-Next-NVFP4, NVIDIA's own ModelOpt export of Flash Next: it keeps its attention, DeltaNet and shared-expert layers in BF16 (the FP4-Local export keeps MXFP8), its n-gram rows in FP8 and its MTP head's routed experts as FP8 block experts. It is the checkpoint other engines publish numbers for and the one users download first, so serving the publisher's weights matters for benchmarking and adoption.
I would like to propose three stacked pull requests:
- Serve the checkpoint on one GB10 and at
--tp 2(two ranks, one a host): BF16 dense and shared layers in the ModelOpt loader, FP8 n-gram rows, FP8 MTP-head experts,--tpfor the CUDA families with ranks, and a checkpoint digest the ranks agree on. One GPU's arithmetic is the existing one. - Split the dense layers across the two ranks with exact rank sums: each rank runs its own attention and DeltaNet heads, the out projections and the shared expert split by input columns, and a round's drafted windows verify in one shared pass. Each layer's two rank sums are an FP32 reduce-scatter, one BF16 round of each half and a BF16 all-gather, the same bytes as an FP32 all-reduce.
- Faster prompt kernels with the same bytes (BF16 prompt tiles, the DeltaNet chain, hc writeback fused with the next norm, the shared expert's slot, K32 expert down), found by comparing a 4.2k-token prefill phase by phase with another engine's profile.
The arithmetic change and its gate (2). The split adds two FP32 halves where one GPU adds one sum, so TP=2 replies differ from those without the split; one GPU's do not change. Per CONTRIBUTING.md, the proposed gate is an independent FP64 reference on one GPU for each changed projection class (the out projections, the shared expert's down projection and the per-row MoE sum), whole and as two halves added once, prompt and decode rows apart, with the split allowed one FP32 rounding, plus teacher-forced fidelity over fixed histories (top-token differences with both margins, and mean NLL against one GPU). Drafted = plain, resumed = fresh and concurrent = solo hold at TP=1 and TP=2.
On two GB10s with the series, a 4.2k-token cold prefill at --tp 2 runs at about 2,500 tok/s and one stream decodes at about 67 tok/s (MTP drafts); one GB10 serves it at about 1,700 tok/s prefill. Each pull request will carry the receipts from the repository's benchmark clients and the kernel contract SUMMARY lines.
Activity
laudney commentedon Oct 31, 2026
/atc claim
ashhart commentedon Nov 1, 2026
Thanks @laudney, serving NVIDIA's own export matters, and the plan is clear. Here's what we'd take, in order:
--tp 2, with the dense and shared layers unsplit at TP=2 (eachrank runs them whole, as the MXFP8 path does now), so replies at
--tp 2are the same bytes as one GPU. Thecheckpoint digest the ranks agree on is good. Please keep
--tpfor the CUDA families with ranks in its owncommit, since it's shared. Receipts: tokens equal between one GPU and
--tp 2, drafted = plain, resumed = freshand concurrent = solo at both, and the CUDA contract
SUMMARYwith planted controls for the BF16 dense, FP8 n-gramand FP8 MTP-expert kernels.
planted control. They're welcome at any time after PR 1.
one sum is a no even with the FP64 gate, because it changes TP=2 replies. Splitting by output columns keeps them:
gather the attention and DeltaNet head outputs (and the shared expert's intermediate) in BF16, then let each rank
compute its own output columns of the out and down projections over the full input, in one GPU's order. The
gathers move about as many bytes as the reduce-scatter. If that doesn't pay, we'd rather keep the dense layers
unsplit.
Happy to look at PR 1 as soon as it's up.
laudney commentedon Nov 1, 2026
Thanks @ashhart, that order works for us.
zig/src/native/cuda.zigis 598 on current main), and--tpfor the CUDA families with ranks stays its own commit (feat(cuda): serve --tp to the CUDA families with ranks). The dense and shared layers run whole on each rank there. I'm adding the receipts you list: tokens equal between one GPU and--tp 2, drafted = plain, resumed = fresh and concurrent = solo at both, and the contractSUMMARYwith planted controls for the BF16 dense, FP8 n-gram and FP8 MTP-expert kernels.laudney commentedon Nov 1, 2026
@ashhart, the branches now follow your later landing order: PR1, standalone PR2, then the kernel PRs.
Each kernel PR has one operator commit with its replacement tests, controls and resource records. PR2 has two commits on PR1. The descriptions use the Why / What / Gate / Numbers format of our merged PRs; each Gate states the applicable requirements and evidence. Every PR links this issue. PR1's published head, title and body are unchanged.
PR2 gathers BF16 inputs and computes each output over the full K in the original slice order; disjoint FP32 outputs are copied as bits. Its fresh complete host gate passes 486 steps and 6,346 tests, with 29 skips, zero unexpected golden differences and 19/19 Cortheon fuzz checks. Its earlier complete CUDA contract passes 546 numerical kernels and detects all 349 controls, with six declared known findings; it exits 1 for 181 resource issues reproduced on PR1.
The later kernel series retains all qualified runtime and CUDA source. The two changed host-test files fix a reducer-count assumption, explicitly discover the composition fixture, and skip its four POSIX socket-pair tests on Windows. The final full FlashNext module cross-compiles for Windows and passes 228/228 host tests again. The focused fixture passes 36/36, the full FlashNext module passes 228/228 at two seeds, and the planted half-output guard defect remains red. Native-server and CUDA-runner compilation passes. The previously recorded affected 16-case/7-control and TP1/TP2 split-on/off native receipts therefore remain applicable to the unchanged runtime.
The standard complex-prompt benchmark of that later series passes 32/32 requests (about 4.2k input tokens, 512 output tokens), with MTP/copy drafts enabled and cache off: 45.9 tok/s single-request decode; 33.0/41.4/55.9/56.0 aggregate output tok/s at client concurrency 1/2/4/8, with four requests active at most. These are absolute later-series results, not an additive gain or a standalone PR2 benchmark. The combined source is available on the qualified-series branch.
The BF16/HC alternatives still need your canonical choice relative to #1764. GDN prefetch and NVFP4 K32-down remain local after no clear default-chunk benefit. The prompt default stays 1536 rows. The Windows fixture fix is published without runtime changes. Current CI: #1784, #1792, #1846, #1848 and #1847 each pass all 9 checks; #1849 passes 8 of 9. PR1 retains its unchanged head, title and body and all 9 passing checks. A separate HC-mix Windows failure is in the unchanged concurrent file-cache test; a maintainer rerun is requested because GitHub returns 403 to my rerun request. No PR has been merged.
laudney commentedon Nov 1, 2026
@ashhart, one maintainer CI rerun is needed for #1849. The Windows job failed only
qwen35_cost_cache_host.test.window cost cache: concurrent writers and readers see only complete tables, atqwen35_cost_files_test.zig:87(!writer.failed).The test, its host wrapper and the shared
lanes/cost_cache.zigimplementation are byte-identical to frozen PR16d2fc0361, whose Windows check passed. Its direct parent #1847 and PR2 #1784 now pass all nine checks, including Windows. HC mix's other eight checks passed. No HC or Flash Next test failed in this Windows run. This appears to be a concurrent file-cache test failure outside the changed code; the job log is retained.I tried to rerun just that failed job, but GitHub returned
403: Must have admin rights to Repository. The workflow has finished; please rerun that failed job. PR1 remains unchanged; I have not added a commit to trigger another CI run.fix(flashnext): word the --tp load refusals and credit the PR
fix(flashnext): word the --tp load refusals and credit the PR
fix(flashnext): word the --tp load refusals and credit the PR
fix(flashnext): word the --tp load refusals and credit the PR
ashhart commentedon Nov 3, 2026
#1771 is in (main d38bba3): nvidia/Qwen3.8-Flash-Next-NVFP4 serves on one GB10. Thanks @laudney. The
--tp 2splits of the dense layers and the prompt kernels follow in their own PRs, so this stays open for those.