[BadAd84]

Makes the one-shot RoCE all-gather serve two to four ranks instead of exactly two, and derives each
peer's HCA pairing from the /24 its RoCE GIDs share instead of from the index.

Follows the question in #1288, from the other side: rather than lending small gathers to another path,
this makes cuda/roce.zig's own gather serve the wider world.

Why

Two things were two-rank-shaped:

1. The gather served one peer, one HCA pair (both[1 - group.rank]), refusing any world but 2.
2. Each peer's HCA was paired by index (links[p][i] = i). On a crossed mesh - port h on one box is
   a different cable from port h on its peer - that pairing is wrong and the queue pair refuses RTR.

What changes

• peerRank/myIndexAt define the mapping for any world: peer p of rank r is (r + 1 + p) % world,
  and a write to rank r lands in the slot r's peer list holds for the sender.
• The region carries a slot per peer (recv_peer_stride, flag_peer_stride); every rank RDMA-writes its
  shard to each peer's region and the kernel polls every peer's flag.
• selectHyperlinks pairs by shared subnet, falling back to index-h when no subnet matches (a switched
  fabric) and refusing when the peers disagree on the link count, since the kernel waits on one count for
  every peer.
• TF_ROCE_MULTI=1 is not needed; the world decides.

Measured

Three GB10s, TP=3, one stream, 400-token decode, 3 rounds, same checkpoint and image:

• the gather works at world 3 where it previously refused (RoCE all-gather: 3 rank(s), 1 link(s) over 2 HCA(s));
• decode 45.0 -> 45.9 tok/s, about +2%, against the +2.2% the prototype in GLM-5.3-Flash at TP=3: its small all-gathers stay on NCCL, and fabric/tp's three-to-eight-rank RoCE (#1209) refuses a switchless ring with BadRecord. Which route should GLM take, and would per-peer devices be in scope? #1288 reported.

The transport is a data mover: it changes which bytes arrive where, not any arithmetic. The existing
gather tests pass, the served replies are unaffected, and test-cuda-host runs at its baseline failure
set (26, all pre-existing and unrelated - see below).

Two things a reviewer should know

A test of ours was asserting something impossible. RoCE gather: peer and slot indices agree at every world marked seen[r] for the w-1 peers - all of them r != me by construction - and then required
every index in 0..w to have been seen, including me, which is never a peer of itself. It failed at
every world for any correct mapping, so it tested nothing and would have shipped red. It now asserts the
real invariant: the peers cover every rank except mine, and mine is never covered.

The file was already at the module limit. cuda/roce.zig was 597 lines against the repository's 600,
so the ABI declarations move to roce_abi.zig and the region layout, peer index maps and their tests to
roce_region.zig. roce.zig is 562 lines. lean_check.py reports 428 problems, the branch's baseline;
test-cuda-host reports 26 failures, the baseline.

Verified on hardware after the move: the image builds, serves and answers at TP=3 (up in 64 s).

[ashhart]

Thanks @BadAd84. Pairing each peer's rails by the /24 its GIDs share is the right fix for a crossed mesh, and the peer and slot maps check out (roce_region.zig:11-18). On main 5897e0bca the merge is clean, and fmt, lean (no new findings), test-runtime-host, test-tp and the Linux CUDA cross-compiles pass.

We've answered #1288: three and four ranks go through fabric/tp's roce_multi, with per-peer devices added there, so a switchless ring works for every family on the fabric. So please keep cuda/roce.zig at two ranks here, with the /24 pairing for a crossed pair, and take the wider pairing to roce_multi. Three things to carry with it, wherever it lands:

1. A rank whose selectLinks refuses returns before the agreement gather (zig/src/cuda/roce.zig:174-179), while ranks whose own peers agree wait in that gather (:189). Counts can differ by rank (A-B and A-C two links, B-C one), so please let the refusal join the gather, as a failed connect does.
2. A peer with no shared /24 is paired by index (:329-335). If both named devices face one peer, the other peer is paired over a cable that doesn't reach it, and the first gather times out. Please refuse there.
3. The new tests never run: test-runtime-host fell from 257 tests to 256, and it still passes with roce_region.zig:83 broken. Please reference that file from a test root, and add host tests for selectLinks: crossed, straight and switched pairs, and a mismatch refused on every rank.

Worlds above two also stay opt-in until a cold-prompt receipt is in: in #1288's runs, prompts were 1.5-2.2% slower while the wire shared NCCL's NIC functions.

Small: Layout.of puts the flags after a gap of their own size (roce_region.zig:42-44), so TP=2's region grows by two slots.

[BadAd84]

Thank you - the two layout questions are fixed and pushed on top.

5. The region gap. Layout.of put send a full slots * s past flags instead of past the flags region, so TP=2 grew by two slots and the flags overlapped send. send now follows the flags' own size.
6. The index fallback. With no shared /24 it paired every peer with HCAs 0..n-1, so two peers could be paired over one port while the other port was paired over a cable that does not reach. A port is now marked once it faces a peer, the fallback takes only untaken ports, and refuses when none is left rather than inventing a link.

Still open, and I would rather list them than imply otherwise:

1. The refusal ordering is a real deadlock as you describe - a rank whose selectLinks refuses returns before the agreement gather while its peers wait in it. The fix is to make every rank's refusal visible to the gather rather than a local return; I am working on it.
2. The new tests never run. You are right that test-runtime-host fell by one and still passes with roce_region.zig:83 broken. I will reference the file from the test root and prove it by breaking it first.
3. Understood on worlds above two staying opt-in until a cold-prompt receipt exists, given the 1.5-2.2% prompt cost in GLM-5.3-Flash at TP=3: its small all-gathers stay on NCCL, and fabric/tp's three-to-eight-rank RoCE (#1209) refuses a switchless ring with BadRecord. Which route should GLM take, and would per-peer devices be in scope? #1288's runs.

Also noted that #1288 has been answered through fabric/tp's roce_multi; if you would rather this change confined itself to cuda/roce.zig as the fallback for worlds it still serves, say so and I will scope it that way instead.

[BadAd84]

3. The tests are reachable now, and I proved it your way. zig/src/cuda/root.zig references roce_region.zig beside roce.zig. With roce_region.zig:83 changed to expect the opposite of what it should, test-runtime-host exits 1 and reports the failure; restored, it exits 0. So the suite can see the file now, and a broken rule test fails it rather than passing quietly.

[BadAd84]

1. The deadlock is fixed. You were exactly right: a rank whose selectLinks refused returned from setup before group.gather, while every other rank was already inside that gather waiting for its bit - one rank unable to use RoCE hung all of them instead of falling back. The selection is now a flag rather than an early return: the refusal is logged, connect is skipped, and the rank reaches the gather with a false bit, so the gather's existing check tears the group down everywhere and NCCL carries the gathers, which is what the log line already claimed. check-native-cuda-host is clean.

That leaves only item 4 open on this PR - worlds above two staying opt-in until there is a cold-prompt receipt, which I accept given the 1.5-2.2% prompt cost in #1288. If the change should be scoped to cuda/roce.zig as the fallback now that #1288 is answered through fabric/tp's roce_multi, tell me and I will cut it down.

[ashhart]

Thanks @BadAd84 for the quick fixes. Yes, please cut it down as you offered: keep cuda/roce.zig at two ranks with the /24 pairing for a crossed pair, and take three and four ranks to fabric/tp's roce_multi, carrying the refusal-joins-the-gather and no-shared-/24 rules with them.

[ashhart]

Thanks @BadAd84 for the quick fixes. We checked them on the head:

• a refusing rank now reaches the agreement gather with a false bit (zig/src/cuda/roce.zig:176-201);
• the index fallback takes only free ports, and refuses when none is left (:334-344);
• roce_region.zig's tests run from the runtime test root.

The merge with current main is clean, and zig fmt passes.

The main ask stands. Please cut this PR down to cuda/roce.zig at two ranks, with the /24 pairing for a crossed pair:

• openRoce keeps g.world != 2 (zig/src/cuda/tp_group.zig:54);
• roce_gather.cu and the region layout stay as on main;
• the description reports a two-rank run.

Three and four ranks go to fabric/tp's roce_multi. For the cut-down PR:

1. Please add host tests for the pairing: crossed, straight, and no shared /24 (index pairing, as on main). Each should check that both ranks pick mirror-image links; the crossed case fails on main. A function over the two sides' GIDs, outside Gather, would let them run without HCAs.
2. The lean check reports a new multi-line comment at roce.zig:174.
3. Small: the layout fix moved the gap instead of removing it. The flags still start one flags region past the receive slots (roce_region.zig:43), so TP=2's region is 512 bytes larger than main's. Main's layout is the one to keep.

When the no-shared-/24 rule moves to roce_multi, please settle every peer's /24 matches before any index fallback. Today an earlier peer's fallback can take both ports that a later peer then matches by /24 (roce.zig:313-344), and with equal link counts nothing refuses.

Once it's cut down, we'll run TP=2 on a crossed pair and on a straight pair. We'll check tokens against main and that the RoCE gather comes up on both ranks, then take decode speed rounds.

[Bizuayeu]

Three and four ranks go to fabric/tp's roce_multi. [...] When the no-shared-/24 rule moves to roce_multi, please settle every peer's /24 matches before any index fallback.

#1495 (comment)

A heads-up so the same work isn't done twice: the roce_multi side is in #1516, as the per-peer devices #1288 scoped. Each peer's stripes go over the HCAs that share its /24, settled for every peer before any queue pair moves and never paired by index across peers, so a peer with no shared /24 refuses; a rank whose pick or connect refuses still votes in the agreement instead of returning early. It has host tests for those and for #1288's three properties, and on our switchless ring of three GB10 the world-3 tf-tp-test suite passes where main refuses at connect (BadRecord). @BadAd84, that should let you cut this one down to two ranks without carrying the wider pairing over.

[BadAd84]

Ok yeah, thanks for flagging #1516 before I took this further.

For 2 ranks main already works: the 2-rank path swaps one peer's blob and needs no /24 rule, because with a single peer there is only one link choice per side anyway. Nothing is left to fix on the refusal front either: both stages are symmetric exchanges, the blob and then the connected bit, so neither rank can return past a wait its peer is still in. Everything here is the wider pairing, so cutting it to 2 ranks leaves a no-op. I would rather pull it than land that, which is what I am doing.

I also ran #1516 on our ring, since I offered. Built from 82fa993a6, world 3, TP=3:

• with TF_GLM_COMM=roce: 45.14 tok/s decode at one stream
• with TF_GLM_COMM=nccl: 44.84 tok/s, same prompt set
• reply digest f8ba801b8fab on both

So it boots and serves correctly at world 3 on a crossed mesh, which is the important part, but no transport gain is measurable here, and roce_multi logs nothing on the way up while our own path prints its gather line, so I cannot tell from the log whether it engaged or fell back. On the same ring our own two-rank-code-plus-pairing build measured 46.12/46.14.

For reference, the mesh this was measured on: spark4 P2p1s0f0 10.0.146.4 reaches spark6 P2p1s0f1, P2p1s0f1 10.0.145.4 reaches spark5 P2p1s0f0, p1s0f0 10.0.46.4 reaches spark6 p1s0f1, p1s0f1 10.0.45.4 reaches spark5 p1s0f0, spark5 to spark6 on 10.0.156 and 10.0.56. Pairing peers by port index joins the wrong cables there; pairing by shared /24 joins the right ones.

[Bizuayeu]

So it boots and serves correctly at world 3 on a crossed mesh, which is the important part, but no transport gain is measurable here, and roce_multi logs nothing on the way up while our own path prints its gather line, so I cannot tell from the log whether it engaged or fell back.

#1495 (comment)

Thanks for running it. Both runs gathered over NCCL, which fits the equal speeds, the one digest and the silent log: on main and in #1516, GLM keeps every world above two on NCCL whatever the comm setting (its own RoCE gather stops at two ranks), and #1516 changes only fabric/tp, which GLM does not reach on its own. Your own build's 46.12 against 44.84 on the same ring is a useful measure of what a RoCE gather is worth there at world 3.

GLM borrows roce_multi in #1524 (head a634d64d0, on top of #1516), behind TF_GLM_ROCE_MULTI=1 with TF_TP_MASTER and TF_TP_KERNEL_IMAGE. There each rank logs TF_GLM_ROCE_MULTI: gathers up to … KiB a rank on the wire, its probe equal to NCCL's, then the first gather is on the wire; a wire that does not open says so in a warning, or refuses at startup with TF_TP_ROCE_FALLBACK=error.

For #1516 on its own, the direct check is tf-tp-test suite at world 3: on a switchless ring main refuses at connect (BadRecord), and with #1516 each peer's stripes go over the HCAs on its /24. Your mesh, where pairing by port index joins the wrong cables, is the case that rule is for.