[ajensenwaud]

warp_tiles, the body of tf_exl3_grouped_0/1/2, loads the next k tile's lane words while it decodes the current one, as exllamav3's experts kernel does at pf = 1. The decode, the mma chain and the warp sums are unchanged. Split out of #728 as asked there.

The load-ahead builds on sm_120 and up only (__CUDA_ARCH__ >= 1200). sm_80/86/89 compile main's loop: their SASS for the three kernels is byte-identical to main's under nvcc 13.0 and 13.3, and their records are main's. Only GB10 was measured, so only sm_12x changes.

Two commits:

1. 0e4b754e exl3 cuda: the grouped experts kernel loads a tile ahead.
2. 7b5d949b exl3 cuda: the grouped load-ahead on sm_120 up only, bit-equal to main's loop under the contract. The grouped projection moves from exl3_experts.cu to exl3_grouped.cuh, with the tile loading as a template argument (main's loop, the load-ahead, or a planted fault).

Contract

• Main's loop stays as a contract-only baseline, tf_ctl_exl3_grouped_old_0/1/2 in contract_controls.cu, with its records. tf_exl3_grouped_0/1/2 are registered as its replacement (contract_exl3_grouped.zig, contract_exl3_grouped_ref.zig).
• Shapes: 1, 2 and 3 k tiles a warp (K 256 / 512 / 768); GLM-5.3-Flash's and MiMo-V2.6-Flash's experts (4096 x 2048, one and two ranks); the Qwen3.5 EXL3 pack's (2048 x 512); and the tile edges, for all three codebooks. The test experts mix widths so both lane-word counts are exercised.
• Two planted controls: the next load reads tile i + 2, and the last tile decodes the previous tile's words.
• Gate 4 uses the float64 gamma bound (EXL3 weights are not integers). At the runner's seeds fp16 sums overflow and bf16 misses by about 240x; a host test checks it.
• GB10 (sm_121, nvcc 13.3), runner filtered to these kernels: 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 2 controls: 2 red as planted, 0 missed; 19 shapes bit-equal to the kernel it replaces. Gate 5 on the device passes for all six launches. The filter was a local change to the runner, so this is not a full-contract SUMMARY.

Same bits, served

• GLM-5.3-Flash A/B on two GB10s, TP=2, tensorfold-native from main b30d5b9d (A) and A plus the load-ahead (B), the same flags (--context 32768 --parallel 4, DFlash2, temperature 0, thinking off): 30/30 identical by token SHA, drafted and "draft": false, solo and 4 streams, including every 512-token speed run across arms and reps. drafted == plain and 4 streams == solo hold within each arm. (Run before the sm_120 gate; the sm_121 code is the same.)
• MiMo-V2.6-Flash (MiMo-V2.6-Flash on CUDA: EXL3, DFlash, two GPUs #728): tf-mimo m1's teacher-forced logits over 1,905 steps are byte-equal with and without the load-ahead.
• SASS (sm_121, nvcc 13.3): every floating-point instruction count is unchanged in all three kernels (HMMA 240, HADD2 or HFMA2 480, FADD 3); loads (LDG 382 -> 694) and integer address work change.

Speed (GB10)

	A (main)	B (load-ahead)
GLM-5.3-Flash, 1 stream, drafted (chat / code / agent / report)	29.4 / 41.9 / 34.3 / 28.6	29.6 / 42.1 / 34.4 / 28.8	+0.5-0.7%
GLM-5.3-Flash, 1 stream, "draft": false	17.5 / 17.5 / 17.5 / 17.5	17.6 / 17.6 / 17.6 / 17.6	+0.4-0.7%
GLM-5.3-Flash, 4 streams aggregate, drafted / plain	58.9 / 42.4	58.9 / 42.7	-0.1% / +0.7%
MiMo-V2.6-Flash tf_exl3_grouped_2, µs a launch (nsys, CUDA graph nodes)	109	96.6	-11%

GLM figures are medians of 3 reps, 512 tokens; GLM's prefill runs the prompt kernels, which this does not touch.

Records

Registers a thread (sm_80 / 86 / 89 / 120 / 121; 32,832 shared bytes a block on all):

	nvcc 13.0	nvcc 13.3
grouped_0	128 / 128 / 128 / 149 / 149	128 / 128 / 128 / 158 / 158
grouped_1	128 / 127 / 127 / 157 / 157	128 / 128 / 128 / 158 / 158
grouped_2	128 / 128 / 128 / 145 / 145	128 / 128 / 128 / 148 / 148

sm_80/86/89 are main's values under 13.0, 13.2 and 13.3. sm_12x's 13.2 entries stay pending (no 13.2 compiler here). On sm_12x shared memory already bounds a block count at three an SM, so the extra registers cost no occupancy. A host test checks every SM under every compiler and goes red on a planted wrong value.

Host: test-kernel-contract, test-cuda-host, test-kernel-symbols-host, compile-cuda-tests and native pass; zig fmt --check and git diff --check are clean; lean finds 426 problems, as main.

[ajensenwaud]

@ashhart could you please rerun the desktop job when you get a moment? Its one failure is unrelated to this change (CUDA kernel and contract records only): electron/agent/engine/hub-endpoint.test.ts failed in cleanup with ENOTEMPTY: directory not empty, rmdir .../tf-engine-test-* on macOS. The harness removes its temp dir with fs.rm(root, { recursive: true, force: true }) (app/electron/agent/engine/test-fixtures/harness.ts:36), which can race the engine still writing there. The Zig, fmt and release-path checks pass. Rerunning needs admin rights, so I can't.

[ashhart]

Thanks @ajensenwaud for splitting the load-ahead out and for the GLM-5.3-Flash A/B. We merged 49f3d38 onto main a1e6d34 (clean). fmt and lean are clean, and test-kernel-contract, test-cuda-host, test-kernel-symbols-host and the Linux CUDA host compiles pass. In warp_tiles (exl3_experts.cu:32-45) only the load order changes. The same words reach the same decode, mma chain and warp sums, and no load passes the warp's last tile. So the bits should hold for every family on this kernel: GLM-5.3-Flash, the Qwen3.5-MoE EXL3 path, and MiMo and Muse once they land. DeepSeek-V4.1 runs its own copy. The records follow the contract doc: 13.0 marked later_pending, 13.3 measured, 13.2 left pending.

Two asks:

1. Please register the changed kernels with the contract. Today tf_exl3_grouped_0/1/2 only have gate-5 records. Keep main's loop as a contract-only baseline and register the new kernel as its replacement, bit-equal at one, two and three k tiles a warp and at GLM's, MiMo's and the Qwen3.5 EXL3 pack's K and N, for all three codebooks. Add a planted control that must go red, for example a load-ahead that reads tile i+2 or reuses the stale words on the last tile. contract_experts3.zig:39-45 is the pattern.
2. At 145-168 registers, sm_80 drops from four blocks an SM to three, and only GB10 was measured. Please either A/B it on an sm_80 or sm_86 card, or keep the load-ahead to __CUDA_ARCH__ >= 1200 so the other SMs keep main's code and records.

Then on our GB10 pair we'll record nvcc 13.2's limits for the three kernels and run the contract. We'll also repeat GLM-5.3-Flash at TP=2 against main (tokens drafted and plain, solo and four streams, three speed rounds) and run the Qwen3.5-MoE EXL3 pack on one GB10, comparing its tokens with main's.

[ajensenwaud]

Thanks @ashhart, both done in 2f6a6e5d (rebased on main 15a0b2a9); the description is updated.

1. Contract. Main's loop stays as a contract-only baseline in contract_controls.cu (tf_ctl_exl3_grouped_old_0/1/2, with records), and tf_exl3_grouped_0/1/2 are registered as its replacement (contract_exl3_grouped.zig, contract_exl3_grouped_ref.zig).
   • Bit-equal for all three codebooks at 1, 2 and 3 k tiles a warp, at GLM-5.3-Flash's and MiMo-V2.6-Flash's expert shapes (4096 x 2048, one and two ranks), the Qwen3.5 EXL3 pack's (2048 x 512), and the tile edges. The test experts mix widths so both lane-word counts are covered.
   • Two planted controls: the next load reads tile i + 2, and the last tile decodes the previous tile's words.
   • Gate 4 uses the float64 gamma bound, since EXL3 weights aren't integers; at the runner's seeds fp16 overflows and bf16 misses by about 240x, which a host test checks.
   • On GB10 (sm_121, nvcc 13.3): 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 2 controls: 2 red as planted, 0 missed, 19 shapes bit-equal. Gate 5 on the device passes for all six launches (158/158/148 registers, and 127/128/128 for the baselines). That run used a local filter on the runner, so it isn't a full-contract SUMMARY; the 214 limit problems on that line are other families' launches, none of these.
2. sm_80. We don't have an sm_80 or sm_86 card, so the load-ahead is now behind __CUDA_ARCH__ >= 1200. sm_80/86/89 compile main's loop; their SASS is byte-identical to main's under nvcc 13.0 and 13.3, and their records are main's again under 13.0, 13.2 and 13.3. sm_120/121 carry the measured load-ahead registers: 149/157/145 under 13.0 and 158/158/148 under 13.3. 13.2 stays pending on sm_12x because we have no 13.2 compiler.

[ashhart]

Thanks @ajensenwaud, both are done in 2f6a6e5:

• Main's loop is a contract-only baseline, and tf_exl3_grouped_0/1/2 are registered as its replacement. The checks
  run at one to three k tiles a warp, at GLM's, MiMo's and the Qwen3.5 EXL3 pack's shapes, and for all three
  codebooks, with the tile i + 2 and stale-last-tile controls.
• With the load-ahead behind __CUDA_ARCH__ >= 1200, sm_80/86/89 build main's loop and keep main's records, and the
  record test pins that.

Next on our side:

• the full contract on a GB10, since your run filtered the runner;
• nvcc 13.2's records for the three kernels on sm_120/121;
• GLM-5.3-Flash at TP=2 on two GB10s against main: token hashes drafted and plain, one and four streams, three speed
  rounds;
• a Qwen3.5-MoE EXL3 pack on one GB10 against main.

desktop passed on this head, so no rerun is needed.

[ashhart]

Thanks @ajensenwaud for rebasing. Our gate passed on it, and it was in tonight's landing batch. Then two of tonight's landings collided with it, both our doing:

• The kernel-contract files (docs/kernel-contract.md, contract_controls.cu, the registry and contract_tests.zig). Those are additions on both sides, and we've merged them.
• exl3_experts.cu, a real overlap. minimax_m3: a CUDA backend for MiniMax-M3.1-Flash at TP over GB10s, from the release's own formats #1388 (MiniMax-M3.1-Flash on CUDA, landing now) moves tf_exl3_plan_finish into exl3_plan.cuh and changes the grouped experts' finish for SwiGLU-OAI. Your first commit edits warp_tiles in the same file, and your second moves the grouped body into exl3_grouped.cuh.

You don't need to do anything yet. Once #1388 is on main, we'll rebase your two commits onto it ourselves. We'll keep both moves, and keep your load-ahead exactly as you wrote it. Then we'll re-run the kernel contract on a GB10, since the merge touches the same kernel. That puts it just after 1.2.0, which we're cutting this morning, rather than in it. If you'd rather do the rebase yourself, say so here and we'll wait for your push.

[ashhart]

Thank you @ajensenwaud. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ajensenwaud]

Thank you.

[ashhart]

Thanks @ajensenwaud! The EXL3 grouped experts kernel that loads the next tile while it decodes, on sm_120 and sm_121, is in 1.2.1: https://github.com/ashhart/TensorFold/releases/tag/v1.2.1