[optimisme]

Part of #1737. Opt-in: off by default, and this is not proposed as a default. Merged with main at 1ce45264, which has #1741 and #1763. Numbers and every cell are in the receipt.

A shared decode round launches each attention kernel once per stream. With TF_FLASHNEXT_ATTN_LANES=1 it launches each once for the round: every row finds its stream's caches and position in a table uploaded once a round.

It applies to two to 32 streams of up to 16 rows each, text on contiguous caches, on our own glue. Page-table KV, media, captured sets and prompt rows keep today's path. So do one stream and a round with no rows: the kernels handle both, but the runtime does not send them here.

Kernels (one GB10, nvcc 13.3)

SUMMARY 463 kernels: 463 pass gates 1-4, 0 fail, 6 known findings; 324 controls: 324 red as planted, 0 missed; 223 limit problems; gates 2447636 ms, limits 2416665 ms

• Each new kernel is registered as a replacement of the per-stream launch and compared byte for byte from 2 to 32 lanes, plus one stream, a stream with zero rows, and streams at different context lengths (0, 3, 127, 128, 2,040 and 9,000).
• The limit problems are stale records already on main: 251 there against 252 with this stack, and the extra one is tf_nemo_router_tiles on sm_89, which this does not touch. The new kernels' records are current on all five SM targets.
• lean_check: 0 new over main.

Tokens: with the switch on, 4,616 comparisons across main and all three branches were equal and none unequal: concurrent against alone and alone against "draft": false, at 1, 4, 8 and 16 streams, greedy and sampled. Resumed equals fresh on a 2,571-token prompt. bench_cache.py --require-reuse reports unqualified, on main too, because the first repeat is cold.

Speed, main against this branch, alternating fresh servers:

• Short contexts: +0.8 to +2.7% from 4 streams up, one stream ties. One cell reads -1.5%, from one run of the two (-3.0% and +0.0%).
• Long contexts (about 30k tokens): not settled. Replies are the same, but these cells are mostly prompt fill, and main and the read-out branch, which share this code, vary by up to 17% between runs. Attention falls outside that range in two cells, 7.46 against 7.66 tok/s and 6.76 against 6.79, so I am not claiming a tie or a gain there.

On current main: retested after #1763 brought in pending DeltaNet rows and the shared expert on a side stream, both on by default. Same build with the switch unset and on, a fresh server each, 1, 4 and 16 streams, greedy and sampled, short contexts: 540 comparisons equal, none unequal, replies equal between the two settings. 16 streams +1.6 to +2.7%, 4 streams +1.0 to +1.6%, 1 stream within 0.4%. flashnext-attn-lanes gives 29 passes and 0 failures. test-cuda-host passes 4,188 of 4,207 with 17 skipped and the same 2 drama_voice_tool cleanup failures. The full kernel contract SUMMARY above is from before these merges and was not rerun. The last main commit, one file in core, came in after the tests.

Not checked: a long-context run with the prompt cache off, which would remove most of that noise. HIP and Metal do not use this path.

[optimisme]

@ashhart this is updated on current main, with #1741 and #1763 in, and merges cleanly. It stays opt-in, and the workflows are waiting for your approval.

Same build with TF_FLASHNEXT_ATTN_LANES unset and at 1, short contexts, greedy and sampled:

Streams	Gain
16	+1.6 to +2.7%
4	+1.0 to +1.6%
1	within 0.4%

All 540 reply comparisons are equal, and replies match between the two settings. flashnext-attn-lanes passes 29 of 29. The full kernel contract was not rerun on this merge, and the long-context question is unchanged. Details are in the description and the receipt.

[ashhart]

Thanks @optimisme. Our GB10 run on Qwen3.8 Flash Next MLX 4-bit (TF_FLASHNEXT_ATTN_LANES=1 against main, warm-up then base, new, new, base): prompt tokens identical in all 20 rows, and served decode at 8 streams up 2.0 to 3.6% (greedy chat 300.0 -> 310.7, greedy code 353.8 -> 365.6, sampled chat 144.9 -> 147.9 tok/s; both new runs above both base runs). One prefill cell (32K after a 10K offset) read lower because a single new run came in at 1636 against about 1750 for every other run, so that's one sample, not a change. We squashed your merge-based branch onto current main under your authorship, with zig fmt for Zig 0.17's spellings and the changelog thanks, and it passes our gate. It lands in the next cycle.

[ashhart]

Thank you @optimisme. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @optimisme! This is in TensorFold 1.2.2, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.2