Repository navigation
Conversation
|
@ashhart could you prioritise this one for review and merge? Serving Qwen3.8-Flash-Next from NVIDIA's own NVFP4 export gives the project a leg up on early Qwen4 support: the loader paths it adds (BF16 dense and shared layers in ModelOpt, FP8 n-gram rows, FP8 block experts in the MTP head), |
|
Two commits on top, no force-push:
Working on the receipts asked for on #1754 next: tokens equal between one GPU and |
|
The expert-parallel fix is in (
The mixed round ( Kernel coverage ( |
|
@ashhart the tokens-equal receipt found a difference I want to flag before fixing it. At this branch's tip ( The dense and shared layers run whole on each rank here, so the difference comes from main's routed experts: The fix I'd like to add here is expert parallel for the routed experts: each rank loads 256 whole experts (all 640 channels, so the same bytes a rank as now), a small kernel maps each rank's picks to its own experts and skips the rest, and the rows a rank skips stay exact zeros, so the existing all-reduce adds |
|
Thanks @laudney. Our plan for the three: #1771 lands on its own. The expert-parallel fix (whole experts per rank, so TP=2 gives one GPU's bytes) is exactly the shape we want, and the contract run is clean. The only gate findings are three lean ones from the contract commit (the |
|
Thanks @laudney. We ran this branch against its main base on one GB10, on our Qwen3.8 Flash Next FP4-Local checkpoint (we don't have NVIDIA's NVFP4 export, so this run covers no regression for the formats we already serve, not the new checkpoint itself). Base, new, new, base:
That reads GREEN. Main is frozen for the 1.2.1 release right now, so this lands right after it, on its own as planned. |
|
Thanks @laudney, and congratulations: 1.2.1 just shipped, so this lands next. We read it through: at one GPU the FP4-Local and MLX paths are the same calls as before. The routed
|
At load, --tp 4 or 8 failed with a bare BadRank, and --tp 2 on an MLX checkpoint or on captured kernels with a bare UnsupportedQuantization, an error other one-GPU load paths also raise. Each --tp refusal now has its own error, and Flash Next sets explains_open so cuda.zig asks explain() at load: with no engine it words only load refusals, so a request's words (out-of-memory, a long prompt) never describe a failed load. The CHANGELOG entry spells Qwen3.8-Flash-Next and credits ashhart#1771 for ashhart#1754.
|
@ashhart both are in, as new commits on top (no force-push):
Checks at |
|
@ashhart GitHub now shows #1771 as conflicting with main (
Do you want us to rebase the branch, or will you resolve these when you land it? If you want us to update it, should we rebase (force-push, so the SHAs in this thread change) or merge main in on top? Nothing else has changed since my comment this morning. |
|
Thanks @laudney. Please rebase onto current main, and a force-push is fine. A linear branch is easier to gate than a merge of main, and we track the PR, not its SHAs. CHANGELOG.md conflicts we resolve ourselves at landing, so keep your branch's line and don't fight it. The two code conflicts ( |
backend() built the ranked table with a runtime call, so at TP=2 it returned a pointer into its own stack frame and the first prompt chunk on two GB10s jumped to address 0x25. The table is now built at comptime like the scoring table, and a test reads it after a real call and an overwritten stack, so a table in that frame fails it.
Flash Next's two-rank path was built and consumed --tp, but the CUDA server refused the flag. It now serves --tp, Fish, Qwen4 and LFM2, which open outside family admission, refuse it with the same words family() uses, and --idle-unload is refused with --tp on CUDA, since closing rank 0 ends its followers and a reload would have none.
Two Flash Next ranks only started when the operator set TENSORFOLD_CHECKPOINT_SHA256, which nothing documented. Each rank now sends the cached full digest GLM's ranks use, and the variable still overrides it. A cold digest reads every byte, so the link and its hello wait 30 minutes for the peer when the digest is computed.
NVIDIA's NVFP4 Flash Next checkpoint keeps its attention, DeltaNet and shared-expert projections in BF16, so they are absent from quantized_layers, and the check refused every one of them. Each projection may now be MXFP8/32 or unlisted, and the parts the loader stitches into one projection must all be one or the other.
NVIDIA's NVFP4 Flash Next checkpoint stores its n-gram table as E4M3 rows with one BF16 scale for the whole table, which the loader refused. A third row format reads them as the reference does, each code times the scale rounded once to BF16, through the mapped gather, the positional reader, the distinct gather and warm-ahead.
The MTP head of NVIDIA's NVFP4 Flash Next checkpoint keeps its routed experts in E4M3 with BF16 128x128 block scales, which only an NVFP4 loader read. They now pack for the existing FP8 expert kernels, the MoE pass dispatches them from a file of their own, and the head's memory estimate counts them at a byte a weight with their scales; a device test runs that call at the head's 512 experts, 640 by 2560, against an f64 reference, as reply equality cannot catch a wrong draft that verification rejects.
The recipe gains the two-rank command for nvidia/Qwen3.8-Flash-Next-NVFP4 with what each rank needs: the same executable, NCCL 2.30 or later, --prompt-cache-gib 0 with drafts and the digest's first-start cost, and states that concurrent streams do not yet share a verify pass; the changelog records the branch.
cuda.zig went past the 600-line rule with main's latest lines, so Fish's and Qwen4's --tp refusal moves into refusals.backend(), which open() already calls before either family opens, and the test that the server serves --tp and admits Flash Next's two ranks moves beside the refusals' own rank tests; open() reads as it does on main.
The checkpoint's BF16 dense and shared layers run main's qmmf BF16 lane and its MTP head's routed experts run main's fp8_experts kernels, which the contract had no gates 1-4 fixture for at Flash Next's shapes. A new set runs the lane's BF16-weight product with bf16 and fp32 outputs at the dense and shared shapes (decode rows), and the FP8 expert down at decode and prompt pair counts against the float64 reference, each with a planted 1.5x scale control; the other experts hold NaN codes, so a misread expert goes red. It joins the full contract and runs alone as flashnext-modelopt-contract.
At --tp 2 two of six replies differed from one GPU because each routed NVFP4 expert was cut by intermediate channels and the ranks' FP32 halves added in the all-reduce. Each rank now loads 256 whole experts, fn_ep_picks maps the pass's picks to the rank's own ids with every other id (and the shared slot) skipped, and the rows a rank skips stay +0, so the all-reduce adds x + 0 and every pair keeps one GPU's sums. One GPU and the expert kernels are unchanged; the TP device test now checks the two ranks' sum equals one GPU's bytes, and fn_ep_picks has its contract fixture, control and nvcc 13.0 records.
The NVFP4 checkpoint's entry now says that at --tp 2 each rank owns half the routed experts whole, so replies are one GPU's bytes, which the expert-parallel fix on this branch made true.
PR1 adds three comment warnings against the current upstream base. Keep the contract descriptions on one line within 120 columns so Lean accepts the branch without changing test or runtime behavior.
At load, --tp 4 or 8 failed with a bare BadRank, and --tp 2 on an MLX checkpoint or on captured kernels with a bare UnsupportedQuantization, an error other one-GPU load paths also raise. Each --tp refusal now has its own error, and Flash Next sets explains_open so cuda.zig asks explain() at load: with no engine it words only load refusals, so a request's words (out-of-memory, a long prompt) never describe a failed load. The CHANGELOG entry spells Qwen3.8-Flash-Next and credits ashhart#1771 for ashhart#1754.
|
@ashhart Rebased onto current main (
The two code conflicts:
The side-stream port is the same code as in the old merge, now in the commit that first moves the native branch into Checks on |
|
Thanks @laudney for the rebase. It's gated and queued to land. The rebase had left three copies of your changelog line, so we kept one, the fullest, with the thanks on it. |
What this changes
The native CUDA build serves
nvidia/Qwen3.8-Flash-Next-NVFP4, NVIDIA's own ModelOpt export, as downloaded: its BF16attention, DeltaNet and shared-expert layers (where the FP4-Local export keeps MXFP8), FP8 n-gram rows, and the MTP
head's FP8 block experts. Serving the publisher's original weights matters for benchmarking and adoption: it is the
checkpoint other engines publish numbers for, and the one users download first.
--tp 2now reaches the CUDAfamilies with ranks, and two ranks agree on a computed checkpoint digest before they serve.
At
--tp 2the dense and shared layers run whole on each rank, and the routed experts are expert-parallel: each rankloads 256 whole experts, a small kernel (
fn_ep_picks) maps a pass's picks to the rank's own ids and skips the rest,and the rows a rank skips stay
+0, so the existing all-reduce addsx + 0. Replies at--tp 2are one GPU's bytes.(Main's intermediate-channel split of each expert added two FP32 halves, which changed 2 of 6 replies against one GPU.)
Commits:
fix(flashnext): keep the ranked vtable static— at TP=2backend()returned a pointer into its own stack frame.feat(cuda): serve --tp to the CUDA families with ranksfeat(flashnext): ranks agree on a computed checkpoint digestfeat(flashnext): accept BF16 dense and shared layers in ModelOptfeat(flashnext): FP8 rows in the n-gram tablefeat(flashnext): FP8 block experts in the MTP head(with an f64 device test at the head's shapes)docs(flashnext): serve NVIDIA's NVFP4 checkpoint at TP=2refactor(cuda): keep --tp's refusals and test out of cuda.zig(the 600-line rule)test(contract): gate Flash Next NVFP4's BF16 and FP8 expert kernelsfix(flashnext): keep one GPU's expert sums at --tp 2docs(changelog): note --tp 2 replies equal one GPU's for Flash Next, and merges of main forCHANGELOG.mdFor #1754. TP=2 splits of the dense layers and the prompt kernels follow in their own pull requests, as agreed there.
Receipt
553beebef(served receipts; later commits change onlyCHANGELOG.md), CUDA 13.4 (nvcc13.4.92), two DGX Spark GB10s (sm_121, 128 GB each) on a direct 200 GbE link for
--tp 2, one for TP=1;nvidia/Qwen3.8-Flash-Next-NVFP4revisionfc694b54;--context 65536 --prompt-cache-gib 0.--tp 2(fn-exactcases,token_sha): prose t=0bfdb2a6d0b2a, prose t=0.7a80787f5ca38, prose t=1.081767b5bb34a, code t=0c993e964aaab, code t=0.7c8cb5fcf645a, code t=1.0d146bc48932aat both; the mixed round (d98bfa8c2a22,2084a6db8b67,a4bb1abe239e) and a cancelled stream'sretry (
59b9985f506f) equal at both too.--tp 2(tools/bench_concurrent.py --alone --serial,token_shaagainst"draft": false): draftedequals
"draft": falsein 6 of 6, every concurrent cell equals its solo run, a mixed round of drafted and plainstreams equals its solo runs, and a cancelled stream's next request equals a fresh one.
tools/bench_openai.py, 256 tokens, greedy, median of 3):--tp 246.4 tok/s (fibonacci) and 34.4(chat); TP=1 49.5 and 36.3. Before: upstream refuses this checkpoint.
tools/bench_cache.py --require-cold, a 4,189-token prompt, 0 cached tokens):--tp 24.69 s and4.63 s, TP=1 2.81 s. Before: refused.
SUMMARY 531 kernels: 531 pass gates 1-4, 0 fail, 6 known findings; 346 controls: 346 red as planted, 0 missed; 181 limit problemsat99ade8e2b. The 181 limit problems are in launches this pullrequest does not change, with main's sources and records: the Qwen3.5 attention and compaction launches
(
qwen35,qwen35_kv8,qwen35_keep), the Qwen4 launches with no compiled usage recorded (qwen4_*), and the twoK32 expert tile launches main keeps pending (
nvfp4_experts_tile). The new fixtures alone at553beebef(
tf-cuda-test flashnext-modelopt-contract):SUMMARY 5 kernels: 5 pass gates 1-4, 0 fail; 3 controls: 3 red as planted, 0 missed— the qmmf BF16 lane (bf16 and fp32 out) at the dense and shared shapes,fp8_experts' decodeand prompt downs at the MTP head's 2,560 x 640, and
fn_ep_picks, with 1.5x-scale and off-by-one controls;fn_ep_picks' nvcc 13.0 records passzig build test-kernel-contract.zig build test-cuda-hostpasses;tf-cuda-test flashnext-nvfp4-tpchecks two expert-parallel ranks'summed outputs are byte-equal to one GPU's.
routed experts share this path.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.--tp 2keeps one GPU's sums.SUMMARYlines are above.Merge-readiness update (2026-11-01)
Rebased onto
9e8c368b0and retained both changelog sections. The only other cleanup is three contract-comment lint warnings. The FlashNext CUDA paths, CUDA runtime and CUDA kernel sources remain byte-identical to609d4e5e2; the existing device receipts keep that source identity. Lean is 424 inherited findings and zero new. The shared CUDA--tpchange remains its own commit, nowf89390b18. CI will run again on the updated head.