Repository navigation
Conversation
78e3592 to
34704a3
Compare
laudney
changed the title
64d499e to
5970dab
Compare
laudney
changed the title
5970dab to
9ec8e7e
Compare
|
@ashhart #1784 also conflicts with main ( Do you want us to rebase it now, or wait until #1771 lands and rebase it once onto main then? Unless you prefer otherwise, we'll wait, so it gets one rebase onto the landed PR 1. |
backend() built the ranked table with a runtime call, so at TP=2 it returned a pointer into its own stack frame and the first prompt chunk on two GB10s jumped to address 0x25. The table is now built at comptime like the scoring table, and a test reads it after a real call and an overwritten stack, so a table in that frame fails it.
Flash Next's two-rank path was built and consumed --tp, but the CUDA server refused the flag. It now serves --tp, Fish, Qwen4 and LFM2, which open outside family admission, refuse it with the same words family() uses, and --idle-unload is refused with --tp on CUDA, since closing rank 0 ends its followers and a reload would have none.
Two Flash Next ranks only started when the operator set TENSORFOLD_CHECKPOINT_SHA256, which nothing documented. Each rank now sends the cached full digest GLM's ranks use, and the variable still overrides it. A cold digest reads every byte, so the link and its hello wait 30 minutes for the peer when the digest is computed.
NVIDIA's NVFP4 Flash Next checkpoint keeps its attention, DeltaNet and shared-expert projections in BF16, so they are absent from quantized_layers, and the check refused every one of them. Each projection may now be MXFP8/32 or unlisted, and the parts the loader stitches into one projection must all be one or the other.
NVIDIA's NVFP4 Flash Next checkpoint stores its n-gram table as E4M3 rows with one BF16 scale for the whole table, which the loader refused. A third row format reads them as the reference does, each code times the scale rounded once to BF16, through the mapped gather, the positional reader, the distinct gather and warm-ahead.
The MTP head of NVIDIA's NVFP4 Flash Next checkpoint keeps its routed experts in E4M3 with BF16 128x128 block scales, which only an NVFP4 loader read. They now pack for the existing FP8 expert kernels, the MoE pass dispatches them from a file of their own, and the head's memory estimate counts them at a byte a weight with their scales; a device test runs that call at the head's 512 experts, 640 by 2560, against an f64 reference, as reply equality cannot catch a wrong draft that verification rejects.
The recipe gains the two-rank command for nvidia/Qwen3.8-Flash-Next-NVFP4 with what each rank needs: the same executable, NCCL 2.30 or later, --prompt-cache-gib 0 with drafts and the digest's first-start cost, and states that concurrent streams do not yet share a verify pass; the changelog records the branch.
cuda.zig went past the 600-line rule with main's latest lines, so Fish's and Qwen4's --tp refusal moves into refusals.backend(), which open() already calls before either family opens, and the test that the server serves --tp and admits Flash Next's two ranks moves beside the refusals' own rank tests; open() reads as it does on main.
The checkpoint's BF16 dense and shared layers run main's qmmf BF16 lane and its MTP head's routed experts run main's fp8_experts kernels, which the contract had no gates 1-4 fixture for at Flash Next's shapes. A new set runs the lane's BF16-weight product with bf16 and fp32 outputs at the dense and shared shapes (decode rows), and the FP8 expert down at decode and prompt pair counts against the float64 reference, each with a planted 1.5x scale control; the other experts hold NaN codes, so a misread expert goes red. It joins the full contract and runs alone as flashnext-modelopt-contract.
At --tp 2 two of six replies differed from one GPU because each routed NVFP4 expert was cut by intermediate channels and the ranks' FP32 halves added in the all-reduce. Each rank now loads 256 whole experts, fn_ep_picks maps the pass's picks to the rank's own ids with every other id (and the shared slot) skipped, and the rows a rank skips stay +0, so the all-reduce adds x + 0 and every pair keeps one GPU's sums. One GPU and the expert kernels are unchanged; the TP device test now checks the two ranks' sum equals one GPU's bytes, and fn_ep_picks has its contract fixture, control and nvcc 13.0 records.
The NVFP4 checkpoint's entry now says that at --tp 2 each rank owns half the routed experts whole, so replies are one GPU's bytes, which the expert-parallel fix on this branch made true.
PR1 adds three comment warnings against the current upstream base. Keep the contract descriptions on one line within 120 columns so Lean accepts the branch without changing test or runtime behavior.
At load, --tp 4 or 8 failed with a bare BadRank, and --tp 2 on an MLX checkpoint or on captured kernels with a bare UnsupportedQuantization, an error other one-GPU load paths also raise. Each --tp refusal now has its own error, and Flash Next sets explains_open so cuda.zig asks explain() at load: with no engine it words only load refusals, so a request's words (out-of-memory, a long prompt) never describe a failed load. The CHANGELOG entry spells Qwen3.8-Flash-Next and credits ashhart#1771 for ashhart#1754.
Gather full BF16 head and shared inputs so each rank computes its own output columns with the original full-K slice order. Copy the disjoint FP32 outputs as raw bits, preserving replies while reducing duplicated TP2 work and retaining whole routed experts.
Load an independently built full-head image as the GDN byte oracle. Compare full and half head layouts on finite and mixed non-finite inputs at both ranks, with repeated outputs and explicit witnesses so templated kernels cannot hide a shared arithmetic change.
9ec8e7e to
1077d5f
Compare
|
@ashhart Rebased onto #1771 as it now stands (
Main changed a lot under this PR since its base. The port:
Checks on I did not rerun anything on a GPU after this rebase. The TP=2 byte and speed numbers in the description were measured on the pre-rebase commits. |
|
Thanks @laudney. We reviewed this independently, and it lands after our TP=2 run on two GB10s. No dot product is split across ranks: every production cut is by output rows over the full K, and cut weights pin |
Refs #1754. Depends only on #1771, frozen at
6d2fc0361. This standalone PR2 is two commits on that base: the output-column split and its frozen-PR1 oracle. Incremental diff. The five operator PRs follow PR2.Why
At
--tp 2, running every BF16 dense projection whole on both ranks leaves much of the checkpoint's work duplicated. Splitting output columns keeps one GPU's sums and makes the two-GB10 setup faster: the standalone PR2 comparison gives 21.02% higher plain decode and 7.98% lower cold 64k TTFT against corrected PR1, with the default 1,536-row chunk unchanged.What
Gate
Ash's landing order is PR1, PR2, then #1792. All five operator PRs follow this standalone PR2. The arithmetic requirement is to keep one GPU's sum order and reply bytes, then show that output splitting pays against unsplit dense layers.
Standalone qualification uses one or two DGX Spark GB10s, sm_121, CUDA 13.0.88, and
nvidia/Qwen3.8-Flash-Next-NVFP4at revisionfc694b54fb0174e0913e6adf86691ef85a4ead47.34704a314, Zig 0.17.0:zig build test test-kernel-contract test-golden fuzz-cortheon -j6 --summary all --test-timeout 10mexits 0. All 486 build steps succeed; 6,346 of 6,375 tests pass and 29 are skipped. Goldens give 322/335 equal with 13 declared known differences and zero unexpected differences, then 55/55 equal with zero unexpected differences. Cortheon fuzz passes 19/19.The later operator series has separate combined-runtime receipts: 16 affected contract cases pass, all seven affected planted controls are red, and the 326 checkpoint head/output comparisons pass again. Its 450 qmmf SM entries match the qualified standalone SASS union. That later runtime also passes TP1/TP2 draft/plain and canonical-token checks, mixed requests, cancellation, cold 2k/8k/64k prefixes, four concurrent 8k requests and non-drafting cache reuse; its TP2 split-disabled control passes. These receipts apply to the later kernel composition.
Numbers
Repeated standalone PR2 comparison against corrected PR1, TP2, safe native builds, same checkpoint and 1,536-row chunk:
Cold 2k/8k/32k TTFT falls by 7.13%/8.24%/8.23%. The separate same-branch off/on/off control gives a 21.12% plain-decode gain. The standalone 64k profile has three fresh prompts per setting. TP1 retains higher cold throughput in the recorded reference runs.
Later operator series: absolute standard
spark-dockerbenchmark of the combined64d499e8fruntime. This run includes the later operators, with MTP and copy drafts enabled, prompt cache zero, a four-active-request cap, 4,218–4,219 input tokens and 512 output tokens per request:All 32 requests complete without failures: 16,384 output tokens, zero cache hits, and 70.37% draft acceptance (11,548 accepted of 16,410 verified draft tokens). At client concurrency eight, requests queue behind the four-active cap. These are absolute later-series results, not a standalone PR2 measurement or an additive estimate of the gains above. The later composition's 64k exactness checks use one fresh prompt per setting.
Overlap
Shared down #1792, BF16 prompt #1846, K32 staging #1848, HC normalization #1847 and HC stream mix #1849 are separate follow-ups to this PR. Review PR2 relative to
pr-fn-nvfp4.BF16 #1846 and HC #1847 overlap #1764 and need a maintainer choice of canonical implementations before both alternatives are combined. PR2's output cuts require pinned full-weight K slices. #1764's family dispatcher bypasses qmmf and recomputes
splitK, so a later combination must preserve those pinned plans.