You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Kernel contract: the nvfp4_expert_waves<2, 2, 4> past-the-first-wave control stays green on an RTX 5090 (its 1,024-row shape fits one wave) #1778
On an RTX 5090 (sm_120, 170 SMs, nvcc 13.3, driver 615.71), the full tf-cuda-test kernel-contract misses one planted control:
PASS shapes control: nvfp4_expert_waves<2, 2, 4> with its items dropped past the first wave: gate 3 at the two-wave shape refuses it: 11 shapes bit-equal to the kernel it replaces (worst 0.000 of the bound), 0 edges refused by the kernel
CONTROL MISSED control: nvfp4_expert_waves<2, 2, 4> with its items dropped past the first wave: gate 3 at the two-wave shape refuses it: shapes did not go red
SUMMARY 520 kernels: 520 pass gates 1-4, 0 fail, 6 known findings; 347 controls: 346 red as planted, 1 missed; ...
The down control on the same card goes red (nvfp4_expert_waves<1, 0, 1>: "at 1024x2560x640 its bytes differ … from byte 20480"). On a GB10 (sm_121, 48 SMs) both go red as planted.
Likely cause. The control's two-wave shape is a fixed 1,024 rows (waves_rows in zig/tests/cuda/contract_nvfp4_fact.zig), and the comment beside it gives the arithmetic: "1,024 rows: 128+ items of 16 pairs, 2,560+ gate-up and 10,240+ down units: two waves below 2,560 resident warps". A GB10 holds fewer than 2,560 warps, so the gate-up launch takes two waves there. An RTX 5090 can hold up to 8,160 (170 SMs x 48), so all 2,560 gate-up units run in the first wave, there is no second wave for the control to drop, and its output is bit-equal. The down launch (10,240 units) still takes two waves on the 5090, which fits its control going red. I haven't measured the kernel's actual occupancy, so this is the code's arithmetic rather than a profile.
A possible fix. Size the waves shape from the device, as 1b9a987 does for the one-wave test (it reads the classic kernel's resident warps): pick rows so each control's launch needs at least two waves on the GPU under test. Or have the control say it was not exercised (one wave) instead of passing silently.
Where it was seen. On the Nemotron 3.5 Transcribe stack (#1632 at dd7cc1a06, rebased on ab4b5d8, with #1769's brace fix on top so the controls build). That stack doesn't touch the NVFP4 experts kernels, contract_nvfp4_fact.zig, the gates or the contract runner. kernel-contract has no per-set filter, so I didn't rerun the full contract on main alone; I can if that helps.
Activity
rootsbymenda commentedon Nov 1, 2026
/atc claim
rootsbymenda commentedon Nov 1, 2026
/atc presence ceac29284a9d56a6377a4bc89aeeb3d7df9a27d760e423b9e7edcf1bba4a3cf4