[Bizuayeu]

What this changes

A prompt step now takes two chunks while no other stream decodes, so a cold prompt gets faster; decode and the output
bytes do not change. On my hardware (GLM-5.3-Flash XSFP4, --parallel 4, the mean cold ttft_s of six requests):

	TP=3, three GB10: main 1b4f2f979 → this head	TP=2, two GB10: main → this head
8k-token prompt	5.002 → 4.630 s (+8.0%)	6.029 → 5.856 s (+3.0%)
32k-token prompt	19.738 → 18.308 s (+7.8%)	23.827 → 23.113 s (+3.1%)
128k-token prompt	83.717 → 77.813 s (+7.6%)	99.917 → 97.025 s (+3.0%)
Decode, fibonacci-raw	57.65, 57.64 → 57.74, 57.65 tok/s	44.47, 44.44 → 44.46, 44.48 tok/s

It holds about 250 MiB more a rank at TP=3 and 210 MiB at TP=2 (the second chunk's rows). Details in the receipt.

A GLM prompt chunk already exchanges its partials in row pieces on a second stream (TF_GLM_OVERLAP_PIECES), but at
each site the main stream still waits for the last pieces' exchange and glue before the next site starts. On three
GB10 at TP=3 that wait was about 10% of a 32k or a 128k prompt's time (the engine's prompt profile, its wait_*
spans), and 5% at TP=2.

With TF_GLM_PROMPT_PAIR=1 (the default) at TP=2 or more, rank 0 has a prompt step take the next chunk too when that
chunk holds 512 rows or more, no kept state starts at it, no other stream is decoding, and the index rings rebase
where a chunk at a time's would (below). The two chunks run in one pass, site
by site: the first chunk's mixer, its fills, then on the side stream its exchange, glue and sharing, while the main
stream goes on to the second chunk's mixer; each chunk waits for its own glued rows only before its next site. The
other ranks take the paired step from the ids rank 0 sends (they tell it by the part's length; nothing on the wire
changes).

• Why only while no other stream decodes: a paired step lasts about two chunks, and at the default
  TENSORFOLD_PREFILL_SHARE of 1 the rounds of the streams decoding beside a long prompt wait for each step. Pairing
  beside them nearly halved their rounds (three decoders beside a 32k prompt kept 0.50 tok/s against 0.86 unpaired at
  TP=3, 0.41 against 0.72 at TP=2). With a stream decoding, the lanes pass pair = false to Leader.chunk and the
  step is main's one chunk; a prompt alone, or several prompts filling with nothing decoding, pair.
• Why the bits cannot change: each chunk runs the pieced path's kernels on its own rows, and the second chunk's mixer
  at a layer follows the first's on the main stream, so the first chunk's caches, KDA state and conv rows of that
  layer are in. tf-cuda-test glm-pair-check compares paired prompts with twins a chunk a step: the first token, the
  next window's logits and tokens, and every byte of the state, which holds the caches.
• Where a pair may start: prompt_pair.prefill readies the index rings (4,096 rows) once for both chunks, so a step
  pairs only where a chunk at a time rebases them at the same point, at the first chunk or not at all (ringsAgree).
  From chunk 1 of rings based at 0 (after a resume from a state kept at an odd chunk, or once decoders finish
  mid-prompt) that step goes alone and the next one pairs. A resumed step is judged by its kept state's base, which
  rank 0 holds on the host, since the state comes back after the choice; prefill refuses any other pair on every rank.
• What it holds: the second chunk's own rows across sites (the stream, hc's rows, the exchange rooms, the final rows
  and logits) in Buffers.initTwin, sharing the first chunk's scratch: about 250 MiB a rank at TP=3 (6,193 KV pages
  against 6,232) and 210 MiB at TP=2 (1,807 against 1,841). Its pieces share the first chunk's side stream, so every
  rank posts both chunks' exchanges in one order.
• With pairing on at TP=2 or more, the side stream runs at the highest priority so a site's exchanges and glue go
  ahead of the other chunk's grids (unpaired steps such as a prompt's tail then use it too). At TP=1 a pair has no
  exchanges to hide: the engine makes no twin and the side stream keeps its normal priority (prompt_pair.wanted).
  TF_GLM_PROMPT_PAIR=0 is main's path, priority included, and every rank must agree on the setting
  (Fingerprint.pair, layout 15).
• Not paired: with a DFlash2 drafter (its taps of a second chunk are not in this change), with prompt logprobs, or
  under TF_GLM_PROF. With the prompt cache on, a step is not paired when a kept state starts at its next chunk; the
  cache keeps a few states a prompt on the 2,048-row grid (the history's end, a rewind point, shared blocks), so most
  steps pair. A paired step covers 4,096 rows: the heat bands and the cancel are checked once a step.
• TF_GLM_CHUNK_SOLO=4096 (glm cuda: TF_GLM_CHUNK_SOLO takes two grid chunks in one pass while no other stream decodes #1823) takes the pair's place, as its description says: a solo pass and a paired step both
  reach the other ranks as a part of two chunks, so one server cannot mix them. With it set, prompt_pair.wanted makes no
  twin. Leader.chunk asks prompt_solo.at or prompt_pair.pairedAt, and on any one server only one of them can say yes.
• The check of the other streams is glm cuda: TF_GLM_CHUNK_SOLO takes two grid chunks in one pass while no other stream decodes #1823's prompt_solo.othersDecode, so cuda_lanes.zig doesn't change. For the line
  limit beside glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #1580, the engine leaves the choice of a forward's pieces to prompt_pair.piecesOf, cuda_follow.zig
  calls prompt_pair.pairedAt itself, and Cancel sits in cuda_engine_types.zig. The ranks' tests moved in glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404.
• A paired step takes the step's indexer split (glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404's choice, made once at the step's start) for both chunks and
  counts it once. b's indexer gather runs on the main stream while a's exchanges may still be in flight on the side
  stream, on the same communicator. That is safe because NCCL runs a communicator's launches in the order the host
  issues them, whatever their streams (in 2.30, ncclLaunchPrepare has each launch's stream wait on the
  communicator's device stream, and ncclLaunchFinish moves that stream on to the launch), and every rank issues both
  in one order.

Related: #1327 caps a filling prompt's steps while replies decode; this change pairs only while none do, so the two
never act on the same step, and the lanes advance by the rows Leader.chunk ran. #1352's wait at the end of a step
it bills to decoders (settle_step, p < 1 with a stream decoding) never meets a paired step for the same reason.

Receipt

• Environment: TensorFold cf2d5d685 (tree as measured) against main 1b4f2f979; three GB10 (sm_121, 128 GB unified
  memory) at TP=3 over a switchless RoCE ring, and two at TP=2; driver 580.178.04; nvcc 13.3.1; GLM-5.3-Flash XSFP4
  (Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f9). Each served with --parallel 4 --prompt-cache-gib 0. TP=3 launched,
  in order, this head, its earlier form (pairing beside decoders too), this head with TF_GLM_PROMPT_PAIR=0, the
  share twice, main, then the second of each; TP=2 main, paired, unpaired twice, paired, main, then the share. The
  head (3a9caf188) rebases onto glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404 and adds the review's fixes, which change no paired step of these runs
  (prompts from position 0, no kept states, TP=2 and 3); the stacked run with glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404 and glm cuda: a decode step's select gives each row a 1,024-thread block #1424 is in
  glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423 (comment). This head (90170240d) is those commits
  rebased onto main 7651ce64f, where glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404, lane host: a long prompt's step owed to decoders ends on its device #1352, lanes: --decode-share sizes a filling prompt's chunks beside live replies (#840) #1327, glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524, glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732 and glm cuda: TF_GLM_CHUNK_SOLO takes two grid chunks in one pass while no other stream decodes #1823 have landed, a commit that moves lines
  and changes no behaviour, and a last one that gives way to TF_GLM_CHUNK_SOLO. Besides the base, cuda_lanes.zig keeps
  main's lines, prefillChunk takes the pair's path only where the engine has its twin, and Fingerprint holds pair
  beside solo and glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524's roce_multi as layout 15, so with TF_GLM_CHUNK_SOLO unset the GPU receipts below,
  from cf2d5d685 and 3a9caf188, were not taken again. They predate glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732; the run with TF_GLM_KV_SHARE is below.

• Exactness on the head at TP=3 and TP=2: glm-pair-check passes at both ("6 prompts, one with a mark at chunk 1, equal their twins a
  chunk a step in the first token, the next window's logits and tokens, the rings' bases and every state byte"); with
  the earlier choice, which pairs from chunk 1, the marked prompt fails at TP=3 ("the paired state differs (pos 8192/8192,
  rings' base 8192/6144, …)"). glm-split-check passes at both too ("prompts of { 6444, 4098, 4101, 4096, 6656 } rows (6
  paired steps) equal their whole twins …"). No GPU case resumes into a pair; a resumed step rests on the rule above
  and its kept state's base. Also at cf2d5d685: glm-pair-check passed at TP=3 and TP=2 on its five prompts;
  glm-lanes-concurrent on decode-check prompts joined two by two passed at both (prompts filling together, and steps
  beside decoding streams); glm-lanes-cover (its 4,388-token copy prompt pairs: copy drafts, cancels mid-prompt,
  kept states at marks, resumes) passed at TP=2. At TP=3 its "copy drafts, sampled: the proposer judged 0 copied rows"
  failed on main's runner too, with the same lines before and after it: the sampled reply there does not repeat W,
  which that line assumes (at TP=2 it judges 188 rows). bench_concurrent.py --alone --serial at levels 1
  and 4: every cell equal on every build. token_sha256 is the same over every launch of each window (main, unpaired,
  paired, and at TP=3 the earlier form), for both decode prompts and all three phases of the 8k, 32k and 128k prompts.

• Prompt speed before and after: the cold ttft_s from bench_cache.py --require-cold --streams 1 (first, repeated
  and concurrent are all cold at --prompt-cache-gib 0), the mean of six requests, in seconds:

  TP, prompt	main	unpaired (same binary)	paired	gain against unpaired / main
  3, 8k	5.002	5.000	4.630	+7.99% / +8.04%
  3, 32k	19.738	19.725	18.308	+7.74% / +7.81%
  3, 128k	83.717	83.621	77.813	+7.46% / +7.59%
  2, 8k	6.029	6.017	5.856	+2.75% / +2.96%
  2, 32k	23.827	23.780	23.113	+2.89% / +3.09%
  2, 128k	99.917	99.391	97.025	+2.44% / +2.98%

  Every paired request beat every unpaired one at every size on both TPs. At TP=3 the earlier form (pairing beside
  decoders too) took 4.628, 18.353 and 77.832 s in the same chain: waiting for decoders costs a prompt alone nothing.

• The share at p = 1 (three decoders beside a 32k prompt, v13_client.py mix): at TP=3 this head kept 0.88 tok/s
  paired and 0.82 unpaired during the prompt (19.3 and 19.6 s); at TP=2 0.72 and 0.72 (23.5 and 23.7 s).

• Decode speed before and after (bench_openai.py --tokens 256 --reps 3 --temperatures 0, medians, fibonacci-raw):
  at TP=3 main 57.65 and 57.64 tok/s, paired 57.74 and 57.65, unpaired 57.55 and 57.55; at TP=2 main 44.47 and
  44.44, paired 44.46 and 44.48, unpaired 44.34 and 44.33 (decode windows take no pieces).

• With glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732's share, on 3159c6492 merged with main 6d3fa0161 (the head's base predates fbbc20d0f's fix to
  contract_controls.cu, which the GPU runner needs): TP=3, PARALLEL=4, TF_KV_PAGES=on, TF_GLM_KV_SHARE=16,
  launched paired, unpaired twice, paired, then paired under TF_KV_POISON=1. The 175 replies at temperature 0 (45
  resumed) give one reply per prompt across the five launches and equal glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732's own receipts on the same seven prompts;
  bench_concurrent is 50 of 50 against alone and 20 of 20 against serial, and kept states land at the same cached in
  every launch (63,488 on a 64,030-token prompt, 126,976 on a 128,030-token one). Prompt time, unpaired then paired: 64K
  cold 35.12 → 32.11 s (+9.4%), 128K resumed at 63,488 36.28 → 32.99 s (+10.0%), 136K cold 83.56 → 77.33 s (+8.1%).
  glm-pair-check, glm-split-check, glm-paged-check and glm-lanes-concurrent pass on all three ranks;
  glm-lanes-cover fails only "copy drafts, sampled", as main's runner does at TP=3. The twin adds 0.25 GiB (63.85
  against 63.60), and MemAvailable after load on rank 0 was 8.9 to 9.4 GiB paired, 9.4 to 9.9 unpaired.

• The same at TP=2, on this head merged with main b17fb507a, two GB10 joined by one QSFP cable, same launches. The
  175 replies at temperature 0 (45 resumed) give one reply per prompt across the five launches and equal glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732's own
  TP=2 receipts on the two prompts they share; bench_concurrent is 50 of 50 against alone and 20 of 20 against
  serial, and kept states land at the same cached as at TP=3 in every launch. Prompt time, unpaired then paired: 64K
  cold 41.69 → 40.10 s, 128K from 63,488 43.32 → 41.69 s, 136K cold 102.41 → 99.42 s. The paired launch under
  TF_KV_POISON=1 served to its end at the paired times. glm-pair-check, glm-split-check, glm-paged-check,
  glm-lanes-concurrent and glm-lanes-cover pass on both ranks, and no launch waited on the heat guard.

• Tests run on this head (Linux x86_64): zig build test 456/456 steps (1,894 of 1,913 tests, 19 skipped) and zig build test-cuda-host 276/276 steps (4,240 of 4,250 tests, 10 skipped); zig build compile-config-server 7/7, zig build compile-cuda-tests 10/10 (also
  with -Dtarget=aarch64-linux-gnu), test-kv-pool 33/33, the GLM host steps (test-glm-prompt-settings-host 6/6,
  test-glm-tp3-host 8/8, test-glm-pages-host, test-glm-fp8-kv-host, test-glm-prompts-host 7/7), zig fmt --check, the lean check (424 problems against
  main's 424, none new), and it merges clean with main 1ddac9620. The host cases: ringsAgree pairs from chunk 2 of rings based
  at 0 and not from chunk 1; wanted makes no twin at TP=1 or beside solo passes.

• GPU checks on this head, on two and on three GB10: with TF_GLM_CHUNK_SOLO unset, glm-pair-check, glm-split-check and glm-lanes-concurrent pass
  at both, and glm-lanes-cover passes at TP=2 and fails only "copy drafts, sampled" at TP=3, as main's runner does. With =4096, glm-solo-check and
  glm-lanes-concurrent pass at both.

• Not run, and why: Metal and HIP, which this path does not reach; a DFlash2 drafter (not paired).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (A paired
  step is one lane step of two chunks.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  (Concurrent equals solo and alone in the receipt; glm-pair-check compares the state's bytes, and the cover's
  resumes passed at cf2d5d685.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No sum
  changes; glm-pair-check compares the bytes.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's
  SUMMARY line is in the receipt. (No kernel changes.)
• Prompt processing is no slower than the last release. (Against main 1b4f2f979: faster at every size on both
  TPs.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The host tests
  read prompt_pair's takesTwo, ringsAgree and wanted, and Settings.pair, which main does not have;
  glm-pair-check needs a GPU and a checkpoint, and its prompt with a mark at chunk 1 fails with the earlier choice.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[Bizuayeu]

Updated the description: it now opens with the measured effect on my hardware, prompt and decode before and after against main, taken from the receipt below. No code change.

[Bizuayeu]

Stacked with #1404, a paired step silently drops #1404's indexer split. Whichever of the two lands second needs the fix below; I'll add it to that one when it rebases.

Engine.prefillChunk reads and clears the split flag (index_split, set by Leader.chunk, or on a follower from the header's flag 2) before it hands a paired step to prompt_pair.prefill, and prompt_pair.prefill never sets it on its two forwards. With both changes in, every paired step runs the indexer whole on every rank, and nearly every step of a lone cold prompt pairs. The merge is clean there (cuda_engine.zig, cuda_prompt_pair.zig), and the bits are the same either way, so only the speed shows it.

The fix in my stacked build (main 1b4f2f979, then this PR, #1404 and #1424):

// cuda_engine.zig, Engine.prefillChunk
if (part.len > rows + 1) return prompt_pair.prefill(e, part, start, len, cancel, marks, split);
// cuda_prompt_pair.zig, prefill(..., split: bool)
fa.index_split = split; // the step's choice (TF_GLM_INDEX_SPLIT) for both chunks
fb.index_split = split;
// ... after pairBody
e.index_splits += @intFromBool(split);

The twin buffers already share the first chunk's select scratch, whose pools hold #1404's extra world rows, and the two chunks' selects run one after the other on the main stream. Leader.chunk conflicts anyway; there the paired step's header carries the flag rank 0 chooses at the step's start, so both chunks share one choice: the first step, paired at position 0, keeps its second chunk (rows 2,048 to 4,096) whole, where the indexer is cheap.

Separately, and loud rather than silent: glm-split-check calls Leader.chunk without this PR's pair argument, so in the stacked tree zig build compile-cuda-tests fails (no CI job builds it). Updating the call would not catch the hole above: with pairing on, the check's prompts pair only from position 0, where no step splits. In the stacked branch the check steps by Chunk.rows, pairs where it can, and counts the splits a step at a time. Pairing leaves the first four prompts fewer split steps (the two-chunk one becomes a single paired step that splits none), so a fifth prompt, 3 chunks and min_rows, pairs its second step past from. On the GPUs at TP=3 the stacked check passes: "prompts of { 6444, 4098, 4101, 4096, 6656 } rows (6 paired steps) equal their whole twins in the first token, the next window's logits and tokens, and every state byte". I haven't run it without the fix; by reading, its fifth prompt would then count 0 split steps of the 1 it wants.

Stacked against main on three GB10 at TP=3, launched main, stacked, stacked, main; token_sha256 is the same over all four launches, and on the prompt lines every stacked launch beat every main launch:

	main	stacked
cold 8k prompt, ttft_s	5.007	4.625	+8.3%
cold 32k prompt, ttft_s	19.783	18.189	+8.8%
cold 128k prompt, ttft_s (first and repeated phases)	83.906	74.254	+13.0%
decode after a 136k prompt, tok/s	57.80	59.18	+2.4%
decode (bench_openai's two prompts), tok/s	51.13	51.20	+0.1%

The 128k gain is about this PR's and #1404's together (+7.6 to 8.0% and +4.6% in their receipts). The cold 136k prompt's ttft_s is left out: on the stacked build each of its phases, and the 128k prompt's concurrent phase, took 2 s waits at the heat guard (92/88 °C here), which no main launch did.

[ashhart]

Thanks, Bizuayeu. The pairing is carefully built:

• The twin buffers hold apart exactly what the side stream touches.
• b's mixer follows a's on the main stream at every layer.
• Both chunks' exchanges go through one side stream in one order on every rank.
• TF_GLM_PROMPT_PAIR=0 is main's path.

On main b30d5b9d3 it passes fmt, lean (no new findings), zig build test, the GLM host steps and the Linux CUDA
cross-compiles. A few things before it can land:

1. Order with glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404. We'd like glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404 to land first. Please rebase onto it and carry the fix from your comment:

   • one flag-2 choice at the step's start;
   • split passed to prompt_pair.prefill for both chunks, with index_splits counted;
   • glm-split-check on the new Leader.chunk and Chunk, with your fifth prompt so a step pairs past from.

   On our stacked tree without it, zig build compile-cuda-tests -Dtarget=aarch64-linux-gnu fails at
   glm_split_tests.zig:57 (6 arguments expected, 5 given). With it, it builds. A sentence in the description on
   why b's indexer gather (main stream) is safe beside a's in-flight exchange on the same communicator would help
   too.

2. Pairs that start on an odd chunk. prompt_pair.prefill calls st.ready(ra + rb) once at the pair's start
   (cuda_prompt_pair.zig:69). With 4,096-row index rings, a pair that starts 2,048 rows past the ring's base
   rebases at its start, while one chunk at a time rebases at the second chunk (cuda_state.zig:467).

   • Rows below the base are never read, so tokens can't change. But the rings' bytes and base end up different
     from a chunk-a-step run.
   • This happens after a resume from a state kept at 2,048·odd, or once decoders finish mid-prompt. Every
     glm-pair-check prompt pairs from position 0, so it isn't covered.
   • Could you either pair only where a chunk at a time would rebase at the same point (one unpaired chunk realigns
     it), or give b its own rebased view? Please also add a case that pairs from chunk 1.

3. Default and cost.

   • With pairing on by default, the twin takes about 34 KV pages at TP=2.
   • The side stream runs at top priority for every prompt step.
   • Pairing also turns on at TP=1: pairedAt has no world check and the twin is allocated at every world. There are
     no exchanges to hide at TP=1, and there is no receipt.
   • Please limit it to world > 1 (or add a TP=1 receipt). We'll decide the default after our NVFP4 runs.

4. Small. twin_sizes ties fields to make()'s size indexes by hand. A comptime check would keep a future size
   entry from shifting them silently.

[Bizuayeu]

A few things before it can land

#1423 (comment)

Rebased onto #1404 and force-pushed; until #1404 lands its commits show here too. The four points:

1. Order with glm cuda: a prompt chunk's indexer rows split across the ranks, their picks all-gathered #1404. Carried as in my comment: one flag-2 choice at the step's start, which both chunks take; split passed to prompt_pair.prefill, with index_splits counted once a step; glm-split-check on the new Leader.chunk and Chunk, with the fifth prompt so a step pairs past from. zig build compile-cuda-tests -Dtarget=aarch64-linux-gnu builds. The description now says why b's indexer gather on the main stream is safe beside a's exchange in flight on the same communicator: NCCL runs a communicator's launches in the order the host issues them, whatever their streams, and every rank issues both in one order.
2. Pairs that start on an odd chunk. A step now pairs only where a chunk at a time rebases the index rings at the same point: both at the first chunk, or neither (ringsAgree). From chunk 1 of rings based at 0 the step goes alone and the next one pairs, so one unpaired chunk realigns the prompt. A resumed step is judged by its kept state's base, which rank 0 holds on the host, since the state itself comes back after the choice; prefill refuses any other pair on every rank. No GPU case resumes into a pair, so that branch rests on the rule and the kept state's base. glm-pair-check adds a prompt of four chunks with a mark at chunk 1 (chunks 0 and 1 alone, then a pair), counts the paired steps with the rings' base followed on the host, and compares both twins' base and mtp_base beside the state's bytes. With the old choice, which pairs from chunk 1, that case fails at TP=3 ("the paired state differs (pos 8192/8192, rings' base 8192/6144, MTP base 6136/6136, first byte 122681856 of 271069952)": the next window rebases the chunk-a-step twin's rings at 8,192, and the twin that paired from chunk 1 keeps 6,144); with this one all six prompts pass.
3. Default and cost. Limited to world > 1: at TP=1 there is no twin, and the side stream keeps its normal priority (prompt_pair.wanted). glm-pair-check refuses TP=1. The default stays on until your NVFP4 runs.
4. Small. twin_sizes and twin_fields are one table now, twin_at, from which make() reads the 13 fields and the twin takes its allocation and copy; make()'s sizes are a [twin_checked]usize, so adding or removing a size stops the build until the indexes are checked again.

On the head (3a9caf188), glm-pair-check and glm-split-check pass at TP=3 and TP=2, each of the three new commits builds compile-config-server and compile-cuda-tests, and zig build test fails only cuda_cost_identity, as main does on that host; the receipt in the description lists the rest.

[ashhart]

Thanks @Bizuayeu, all four points are in:

• The step's split is chosen once at its start (cuda_follow.zig:231). Both chunks take it, and it is counted once (cuda_prompt_pair.zig:83-84, :98).
• ringsAgree (:49) follows State.ready's rebase rule exactly, and prefill re-checks it on every rank (:71).
• The twin and the top-priority side stream need world > 1 (:37).
• twin_at with a [twin_checked]usize closes the size-index gap.

On main 5897e0bca the merge is clean, and zig build compile-cuda-tests -Dtarget=aarch64-linux-gnu now builds. fmt, test-cuda-host, the GLM host steps and the x86 cross-compiles pass.

One small thing: lean reports one new finding, a multi-line comment at zig/tests/cuda/glm_split_tests.zig:40-41. Please fold it into one line of at most 120 columns.

The order stays as agreed: #1404 lands first, then this one. On NVFP4 at TP=2 we'll run:

• glm-pair-check and glm-split-check;
• token_sha256 against main for the decode prompts and the cold 8k, 32k and 128k prompts, paired and with TF_GLM_PROMPT_PAIR=0;
• cold first-token times at the same sizes for main, unpaired and paired;
• decoders' tok/s beside a long prompt;
• a resume from a state kept at an odd chunk;
• the KV pages admitted with the twin.

We'll settle the default from those.

[Bizuayeu]

One small thing: lean reports one new finding, a multi-line comment at zig/tests/cuda/glm_split_tests.zig:40-41. Please fold it into one line of at most 120 columns.

#1423 (comment)

Folded into one line of 109 columns in 488074b. lean on the branch reports 433 problems, 434 before, none new.

[Bizuayeu]

Rebased onto main 7c0688951, which this branch conflicted with once #1404 and #1352 landed. #1404's three commits
drop out, and the eight here are unchanged (git range-diff) but for one line of context in the second, 2c20ed74e:
cuda_lanes.zig keeps #1352's if (s.settle_step) try self.x.e.stream.synchronize(); after this branch's
l.fill = at + step.rows;. A step settles only while a stream decodes and pairs only while none does, so the two never
meet. The receipts in the description are from before the rebase (cf2d5d685 and 3a9caf188); the description now
says so, and no longer says the branch sits on #1404. On the new head ba004d872: the GLM host steps,
compile-cuda-tests (x86_64 and aarch64), compile-config-server 7/7, zig fmt --check, lean 0 new against main's
425; test-cuda-host and zig build test fail only Qwen3.5's cuda_cost_identity, which fails the same way on main
here. Against open PRs it now overlaps in two lines: #1327's l.fill = at + engine.prompt_rows; in Lanes, where
this branch advances by step.rows (engine.prompt_rows on every unpaired step), and #1411's test import in
cuda.zig, beside this branch's two.

[ashhart]

Hi @Bizuayeu, a status update on #1423.

We read the full diff. The exactness design holds up: the twin keeps every field the side stream touches apart,
and the main stream runs its fills, mixers, FFN and MTP in a fixed order. We found nothing blocking.

What's still owed is the device run. It needs two GB10s at once (TP=2, and TP=3 if three are free). It compares
main, this head paired, and this head with TF_GLM_PROMPT_PAIR=0. Main doesn't know that setting, so the
unpaired arm is a separate launch of this binary. We'll also log the KV pages admitted at startup. Pairing costs
some, and that number decides whether it stays on by default or ships opt-in. If it ships opt-in, we'll adjust the
changelog line with you.

Nothing is needed from you right now.

[Bizuayeu]

Nothing is needed from you right now.

(#1423 (comment))

Thank you for reading it through! I'll wait for the device run, and I'm glad to adjust the changelog line with you if it ships opt-in.

[Bizuayeu]

Rebased onto main 24358c1e2, which this branch conflicted with once #1524 landed. The eight commits are unchanged (git range-diff) but for the second, 310817652: cuda_lanes.zig keeps #1327's comment above this branch's l.fill = at + step.rows;, cuda_ranks.zig reads both TF_GLM_ROCE_MULTI and TF_GLM_PROMPT_PAIR, and Fingerprint.layout goes to 14, as #1524 took 13 for roce_multi and this branch's pair is another field. The first has one line of context. The receipts in the description are from before the rebase. On the new head 92ada54ef: zig build test 442/445 steps and test-cuda-host 268/270 (4,003 of 4,013 tests, 9 skipped), the one failure in both cuda_cost_identity's timing-dependent test, as on main; test-kv-pool 28/28, compile-cuda-tests 10/10 (also for aarch64), compile-config-server 7/7, the GLM host steps, zig fmt --check, and tools/zig/lean_check.py with nothing new (425 against main's 425). It merges clean with main 94b90c9db.

[Bizuayeu]

A note for whichever of this and #1580 lands second: both take Fingerprint.layout from 13 to 14, on the same line, so a merge shows no conflict. The second needs 15, as pair and l2pf are different fields. I'll bump it when I rebase; this is in case you rebase it yourself.

[Bizuayeu]

Rebased onto main ab4b5d8e6, which this branch conflicted with once #1732 landed. The first commit dropped out, as #1732 made the same move of the lanes' host checks. In what is now the first, cuda_lanes.zig's chunk calls take #1732's marksOf(s, &keeps) beside this branch's argument and step. A new last commit moves othersDecode to cuda_prompt_pair.zig, which keeps cuda_lanes.zig at 600 lines. The other six are unchanged (git range-diff), and the layout stays 14. The receipts in the description are from before the rebase. #1732's marks reach pairedAt through marksOf, so under TF_GLM_KV_SHARE a step stays unpaired when a kept state starts at its next chunk, but the pair and the share have not run together on a GB10. On the new head 3159c6492: zig build test 446/449 steps and test-cuda-host 272/274 (4,102 of 4,112 tests, 9 skipped), the one failure in both cuda_cost_identity's timing-dependent test, which this branch does not touch (run alone, it fails 58 of 300 runs); test-kv-pool 30/30, compile-cuda-tests 10/10 (also for aarch64), compile-config-server 7/7, the GLM host steps, zig fmt --check, and tools/zig/lean_check.py with nothing new (424 against main's 424). It merges clean with main 87726b034.

[Bizuayeu]

zig-windows on 3159c6492 was cancelled at the job's 60-minute limit, with nothing printed by zig build test after its first five minutes and no test reported as failed. The other eight checks pass. It doesn't look specific to this branch: six of the 172 zig-windows jobs that ran in the last day were cut the same way, main f410c640e among them, and the same merge passes zig build test --test-timeout 10m on Windows 11 here (412/412 steps in 16 minutes). Could you re-run that job when you pick this up? I can't from a fork.

[Bizuayeu]

Ran the pair with #1732's share on three GB10, which my rebase note left open. Replies match across paired, unpaired and poisoned launches and match #1732's own receipts, kept states land in the same places, and paired prompt steps keep their gain on top of the share (+8.1% to +10.0%). The numbers are in the description.

[Bizuayeu]

Added the TP=2 run with #1732's share to the receipts.

[Bizuayeu]

Rebased onto main with #1823 in: this head (90170240d) takes layout 15, TF_GLM_CHUNK_SOLO=4096 takes the pair's place and makes no twin (prompt_pair.wanted), as #1823's description says, and othersDecode is the one in prompt_solo. GPU checks on this head, on two and on three GB10: with it unset, glm-pair-check, glm-split-check and glm-lanes-concurrent pass at both, and glm-lanes-cover passes at TP=2 and fails only "copy drafts, sampled" at TP=3, as main's runner does. With =4096, glm-solo-check and glm-lanes-concurrent pass at both. The host tests are in the description.