Repository navigation
Conversation
backend() built the ranked table with a runtime call, so at TP=2 it returned a pointer into its own stack frame and the first prompt chunk on two GB10s jumped to address 0x25. The table is now built at comptime like the scoring table, and a test reads it after a real call and an overwritten stack, so a table in that frame fails it.
Flash Next's two-rank path was built and consumed --tp, but the CUDA server refused the flag. It now serves --tp, Fish, Qwen4 and LFM2, which open outside family admission, refuse it with the same words family() uses, and --idle-unload is refused with --tp on CUDA, since closing rank 0 ends its followers and a reload would have none.
Two Flash Next ranks only started when the operator set TENSORFOLD_CHECKPOINT_SHA256, which nothing documented. Each rank now sends the cached full digest GLM's ranks use, and the variable still overrides it. A cold digest reads every byte, so the link and its hello wait 30 minutes for the peer when the digest is computed.
NVIDIA's NVFP4 Flash Next checkpoint keeps its attention, DeltaNet and shared-expert projections in BF16, so they are absent from quantized_layers, and the check refused every one of them. Each projection may now be MXFP8/32 or unlisted, and the parts the loader stitches into one projection must all be one or the other.
NVIDIA's NVFP4 Flash Next checkpoint stores its n-gram table as E4M3 rows with one BF16 scale for the whole table, which the loader refused. A third row format reads them as the reference does, each code times the scale rounded once to BF16, through the mapped gather, the positional reader, the distinct gather and warm-ahead.
The MTP head of NVIDIA's NVFP4 Flash Next checkpoint keeps its routed experts in E4M3 with BF16 128x128 block scales, which only an NVFP4 loader read. They now pack for the existing FP8 expert kernels, the MoE pass dispatches them from a file of their own, and the head's memory estimate counts them at a byte a weight with their scales; a device test runs that call at the head's 512 experts, 640 by 2560, against an f64 reference, as reply equality cannot catch a wrong draft that verification rejects.
The recipe gains the two-rank command for nvidia/Qwen3.8-Flash-Next-NVFP4 with what each rank needs: the same executable, NCCL 2.30 or later, --prompt-cache-gib 0 with drafts and the digest's first-start cost, and states that concurrent streams do not yet share a verify pass; the changelog records the branch.
cuda.zig went past the 600-line rule with main's latest lines, so Fish's and Qwen4's --tp refusal moves into refusals.backend(), which open() already calls before either family opens, and the test that the server serves --tp and admits Flash Next's two ranks moves beside the refusals' own rank tests; open() reads as it does on main.
The checkpoint's BF16 dense and shared layers run main's qmmf BF16 lane and its MTP head's routed experts run main's fp8_experts kernels, which the contract had no gates 1-4 fixture for at Flash Next's shapes. A new set runs the lane's BF16-weight product with bf16 and fp32 outputs at the dense and shared shapes (decode rows), and the FP8 expert down at decode and prompt pair counts against the float64 reference, each with a planted 1.5x scale control; the other experts hold NaN codes, so a misread expert goes red. It joins the full contract and runs alone as flashnext-modelopt-contract.
At --tp 2 two of six replies differed from one GPU because each routed NVFP4 expert was cut by intermediate channels and the ranks' FP32 halves added in the all-reduce. Each rank now loads 256 whole experts, fn_ep_picks maps the pass's picks to the rank's own ids with every other id (and the shared slot) skipped, and the rows a rank skips stay +0, so the all-reduce adds x + 0 and every pair keeps one GPU's sums. One GPU and the expert kernels are unchanged; the TP device test now checks the two ranks' sum equals one GPU's bytes, and fn_ep_picks has its contract fixture, control and nvcc 13.0 records.
The NVFP4 checkpoint's entry now says that at --tp 2 each rank owns half the routed experts whole, so replies are one GPU's bytes, which the expert-parallel fix on this branch made true.
PR1 adds three comment warnings against the current upstream base. Keep the contract descriptions on one line within 120 columns so Lean accepts the branch without changing test or runtime behavior.
This was referenced Nov 1, 2026
laudney
marked this pull request as ready for review
November 1, 2026 14:43
laudney
force-pushed
the
pr-fn-bf16-k32-ready
branch
from
November 1, 2026 15:35
9460418 to
24b0f89
Compare
laudney
changed the title
Gather full BF16 head and shared inputs so each rank computes its own output columns with the original full-K slice order. Copy the disjoint FP32 outputs as raw bits, preserving replies while reducing duplicated TP2 work and retaining whole routed experts.
Load an independently built full-head image as the GDN byte oracle. Compare full and half head layouts on finite and mixed non-finite inputs at both ranks, with repeated outputs and explicit witnesses so templated kernels cannot hide a shared arithmetic change.
laudney
force-pushed
the
pr-fn-bf16-k32-ready
branch
from
November 1, 2026 16:21
24b0f89 to
2f50a9a
Compare
Remove the whole shared expert's scatter by writing FP32 results straight into slot 10. Retain PR2's full-input column copies for split outputs, honor pinned K slices, and verify both routes with replacement bytes, controls, records and a host trace. Skip the POSIX socket fixture on Windows.
Use 128-row tiles for eligible BF16 prompt projections while keeping the pinned K slices, ordered MMA chain and output rounding. Let the shared-slot selector prefer this path and verify replacement bytes, row boundaries, full MTP scratch, controls and measured resources.
Read K64 weight packs in two halves for unfused BF16 and FP32 prompt outputs to reduce shared staging. Preserve the ordered K16 chain and all parent K64 entries, with replacement bytes, output controls, measured resource records and same-buffer timing fixtures.
laudney
force-pushed
the
pr-fn-bf16-k32-ready
branch
from
November 1, 2026 16:48
2f50a9a to
2da6299
Compare
Owner
|
Thanks @laudney. This sits on #1846/#1847, and we took #1764 for that tile and fusion instead (it's in the landing queue now), so this stack's base won't land as it stands. Once #1764 is on main, please restack this one operator commit onto main and re-run its receipts on that head. We'll review it standalone then. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The BF16 parent uses 98,432 shared bytes per 128x128 K64 prompt block. This PR stages the same packs in K32 halves for the two unfused outputs, using 49,152 bytes. Standalone-on-PR1 proxies measure 1.039–1.060x in two GB10 runs; the isolated 64k increment is below 1%, so no stable incremental served gain is claimed.
Refs #1754. Depends on BF16 #1846, after shared #1792, PR2 #1784 and frozen PR1 #1771. One operator commit above #1846; head
2f50a9a81. Incremental diff.What
Gate (GB10, sm_121, CUDA 13.0.88)
Ash's latest landing order requires PR2 before shared #1792; this child follows #1846 on that stack. The issue requirements require matching replacement bytes, records and a planted control. CI is checked separately on this head.
test-flashnext-compose-host(36/36), contract/scratch and shared tests. The fixture asserts the unique BF16 gather, its two immediate interleave copies, full-K640 down and raw FP32 output copy, permitting 64-row reducers and using initialized mock mixed input. The earlier filtered 30/200 counts did not execute that fixture.x86_64-windows-gnu; Windows execution remains a separate CI check. The corrected shared target passes 36/36 on Linux, and the full final series passes 228/228 again. Runtime and CUDA code are unchanged.tf-cuda-testfast native andtensorfold-nativesafe native. Inert images are compile-only; the resulting binaries were not executed.The full-series source proof identifies only two host-test-file changes from the qualified combined series; runtime and CUDA source remain identical. That preserves applicability of the combined native matrix linked from PR2 #1784, without claiming a new GPU run after reordering. Later compiler sets and other physical SMs remain unrun. These focused gates are not a clean global CUDA-suite claim.
Numbers
These are isolated builds on PR1, before the PR2-first composition. Parent/candidate calls alternate into one physical output buffer over the normal rank-0 projection profile; MTP and all-logit heads are separate.
The small served difference does not establish a stable increment. Unchanged fused/MTP paths also show timing noise; the proxy gains cannot be added to parent or shared-slot gains, or promoted to a PR2-first series measurement.
Overlap
Order: #1771 → #1784 → #1792 → #1846 → this PR. HC #1847/#1849 is independent on PR2. GDN prefetch and NVFP4 K32-down stay local and off.
The parent's canonical BF16 choice with #1764 applies: its family-level dispatcher must not bypass K32 selection or recompute pinned full-input slices. Resolve that integration choice before combining the alternatives.