Repository navigation
GLM-5.3-Flash at TP=3: its small all-gathers stay on NCCL, and fabric/tp's three-to-eight-rank RoCE (#1209) refuses a switchless ring with BadRecord. Which route should GLM take, and would per-peer devices be in scope? #1288
Description
Update (10-27): route (b) prototyped and measured at TP=3
No route was chosen yet, so I built (b) on my side to have numbers: cuda/tp_group lends GLM's small all-gathers (up
to 256 KiB a rank) to roce_multi's wire beside NCCL, opened on every rank or none after a probe against NCCL
(TF_GLM_ROCE_MULTI=1), with the per-peer devices of question 2 so the ring connects. Three GB10 at TP=3,
GLM-5.3-Flash XSFP4, --parallel 4 --prompt-cache-gib 0, the same binary with the wire on and off, alternated:
| NCCL | the wire | change | |
|---|---|---|---|
| Decode, c=1, drafts on (mean of two prompts' medians) | 51.17, 51.22 tok/s | 52.10, 52.57 tok/s | +2.2% |
| Decode, c=1, drafts off | 30.47, 30.59 tok/s | 30.96, 31.02 tok/s | +1.5% |
| Decode after a 136k-token prompt, drafts on | 57.91 to 58.16 tok/s | 58.93 to 59.60 tok/s | +1.9 to 2.6% |
| Decode, c=4 (the code prompt), drafts on | 145.1, 145.1 tok/s | 144.1, 144.4 tok/s | −0.6% |
A cold 136k-token prompt, ttft_s, drafts on |
87.18 to 87.39 s | 88.52 to 88.72 s | 1.5 to 1.6% slower |
| The same, drafts off | 83.37 to 83.53 s | 84.76 to 85.73 s | 1.7 to 2.2% slower |
The output bytes are the same with and without the wire (token_sha256 in every launch, for both decode prompts and
all three phases of the 136k prompt). Decode gains, but every launch with the wire is slower at prompt processing, so
I am not proposing it in this form. The prompt's exchanges never use the wire (they are past its 256 KiB and go over
NCCL from the side stream), and they slow down only when NCCL shares the wire's NIC functions. The wire's queue
pairs sit on rocep1s0f0/f1, two of NCCL's four devices. With NCCL kept off those two, a wire launch's prompt was as
fast as NCCL's; with NCCL on those two alone, it was 1.1% slower again (follow-ups below). NCCL on two devices is itself
8.5% slower at prompt processing, so that is no fix. Next I'll test what on a shared function slows NCCL: the wire's
idle queue pairs, its registered memory, or NCCL merging a PCIe half's two functions into one device.
How it was measured
- Engine: TensorFold
1b4f2f979with the prototype (not pushed); the wire's kernel ismailbox.cubuilt as
tp_mailbox.fatbinwithzig/build/tp.zig's flags and read fromTF_TP_KERNEL_IMAGE;TF_TP_ROCE_FALLBACK=error,
so a wire that did not open would have refused to serve. - Launches, each a fresh start: wire, NCCL, NCCL, wire with the drafts, then the same four with
--no-drafts. Each:
bench_openai.py --tokens 256 --reps 3 --temperatures 0,bench_concurrent.py --alone --serialat levels 1 and 4,
andbench_cache.py --require-cold --streams 1 --tokens 256on a 136k-token prompt (its three phases are all cold). - Every rank logged its probe ("gathers up to 256 KiB a rank on the wire, its probe equal to NCCL's") and its first
gather on the wire (16 bytes, the greedy pick). One more launch of each kind read rank 0's threads' CPU from
/proc: with NCCL, two threads stay busy in decode and through the prompt; with the wire, its proxy holds a core in
decode, where a thread created with the NCCL communicator drops from 99% to 3%, and naps through the prompt (3%). - Follow-ups on 10-27: with the proxy napping 20 µs and doubling up to 1 ms, the same alternated launches left the
prompt 1.3 to 2.3% slower (decode +1.5 to 2.5%). Then one cold 136k prompt an arm withTF_GLM_PROF=1, diagnoses
rather than receipts: the compute spans matched, and the side stream's NCCL exchanges grew (the glued rows' relay
+15 to 16%). With NCCL onroceP2p1s0f0/f1only, the wire launch's prompt took −0.1% against NCCL's; with NCCL on
rocep1s0f0/f1only (the wire's), +1.1%.
Summary
GLM-5.3-Flash serves at TP=3 on three GB10s, and every small all-gather it makes there goes over NCCL: the layers' row shards (families/glm/cuda_forward.zig:204) and the greedy picks (families/glm/cuda_engine_pick.zig:16). GLM's group is cuda/tp_group, whose RoCE gather is two-rank only: Group.openRoce returns at any other world (zig/src/cuda/tp_group.zig:49), so allGather takes ncclAllGather (:61). Since #1209 the shared TP fabric gathers over RoCE at three to eight ranks (fabric/tp/roce_multi.zig), but GLM does not use the fabric, and fabric/tp/existing.zig:16 (the fabric over the released Group) takes two ranks only. Before writing code, I'd like to know which route you want, and whether a switchless ring is in scope.
Where a switchless ring stands
My three GB10s are cabled in a ring with no switch (the /32 ring of #1178). Each cable is its own /30, carried over both PCIe halves of one ConnectX-7 port, so each box has four RoCE devices and each device reaches one peer. NCCL carries the ring today with all four devices in NCCL_IB_HCA and NCCL_IB_SUBNET_AWARE_ROUTING=1, which sends each peer over the port whose /30 reaches it. roce_multi refuses the same setup:
roce_pick.selectFrompicks this rank's ports from the subnets of all peers together, at most two (fabric/tp/roce_pick.zig:127-130,TF_TP_ROCE_HCAS).VerbsWire.pairingthen wants every picked device to share a /24 with each peer, and returnsBadRecordfor a device that shares none (fabric/tp/roce_multi_verbs.zig:415).- On the ring, on
8086c205c,tf-tp-test suiteatTF_TP_WORLD=3withTF_COMM_BACKEND=roceandTF_TP_ROCE_FALLBACK=errorstops there on every rank. Rank 0 (ranks 1 and 2 print the same with their own device):
info: [tensorfold] tp roce: rocep1s0f0 GID 3 (10.41.13.1), MTU enum 3, 2 queue pairs
warning: [tensorfold] tp roce: rank 0 connect failed: BadRecord
error: RoceUnavailable
With the default fallback, every rank would take NCCL instead.
In #981 you wrote that more than two ranks "should come first, on the shared TP fabric (#636 with #852's transport) rather than a second loaded copy of NCCL". I'd like to follow that, and I see #981 is claimed.
Questions
- For GLM's small gathers at three ranks, which route do you want?
- (a) GLM moves its TP onto a
fabric/tpsession, as DeepSeek-V4.1 does. - (b)
cuda/tp_groupgains a multi-rank fast path that borrowsroce_multi's wire and kernel, set up over the group's NCCL communicator ascuda/roce.zigis now. - (c) The work on Nemotron 3.5 Super VL on two GB10s: TP=2 serial decode is no faster than TP=1 here, and what TP=3/4 would need #981 brings the released CUDA families onto the fabric, and GLM waits for it.
- (a) GLM moves its TP onto a
- Would
roce_multitake per-peer devices, striping each peer over the devices that share its /24 instead of refusing? Two ranks and switched fabrics would keep today's records and bits.
NCCL on the ring, for the baseline
The same suite with TF_COMM_BACKEND=nccl on the ring: SUMMARY rank 0: 201 passed, 0 failed, and
| per call | |
|---|---|
decode_window, 86 exchanges of 8192 B |
27.78 µs (window 2,389.2 µs) |
all_gather 8192 B, eager / graph |
21.50 / 25.46 µs |
all_gather 16384 B, eager / graph |
24.74 / 29.06 µs |
#1209's world-4 run on a switched fabric had 38.27 µs per exchange over NCCL and 14.71 µs over RoCE.
Why it matters here
On the same three boxes and on a pair, 8086c205c serving GLM-5.3-Flash (XSFP4, drafts off), with TF_GLM_PROF=1 and nsys on rank 0:
| TP=3 (NCCL) | TP=2 (the two-rank RoCE gather) | |
|---|---|---|
c=1: the gathers and their waits in a decode step (gather_a + gather_b) |
3.09 of 33.50 ms (9.2%) | 2.16 of 41.18 ms (5.3%) |
| exchange kernels a step, c=1 / c=4 / c=8 (distinct prompts) | 2.18 / 4.74 / 6.71 ms | 1.79 / 2.07 / 3.02 ms |
What I can do
If you're fine with a shape, I'd like to write it as two pull requests: the per-peer devices in roce_multi, and GLM's small gathers on the route you pick, each with the world-3 runs on the ring and a GLM-5.3-Flash TP=3 receipt. I can also run any branch you'd like tried on a three-box switchless ring.
Written with Weave (My Symbiotic AI)
Activity
Bizuayeu commentedon Oct 25, 2026
A follow-up with a measurement: per-peer devices in
roce_multinow run on the switchless triangle in a local prototype (a branch onfaccd11c8, not a PR).tf-tp-test suitedecode_window(86 × 8,192 B)TF_COMM_BACKEND=roce,TF_TP_ROCE_FALLBACK=errorTF_COMM_BACKEND=ncclThe shape, kept inside what the wire already promises:
n_hcastays one count for every source (at most 2), so each peer gets the same number of stripes k, carried by the HCAs that share a /24 with that peer. On the triangle k = 1 (one function of each cable's port), on a switch k = 2 as today.pairBoth.connectbrings only the paired queue pairs to RTR/RTS. Each post reaches a peer over that peer's HCAs. The record, the region layout,mailbox.cuand the two-rank wire are unchanged. On a switch the pairs, rkeys and stripes are the same as before (a fake-fabric test checks this).choosetoward each peer alone. It keeps the largest per-peer count whose union fits two HCAs. Two ranks keepchooseas is.test-roce-multi-host29/29 (24 on main) andtest-tp116/117 (104/105 on main).zig fmt --checkpasses, andlean_checkmatches main. The new tests cover a fake switchless ring of three (only each cable's queue pairs connect, and bytes and flags match over one stripe), a ring of four and mis-cabled fabrics (refused before any queue pair moves), and the port pick for three ranks over a real bootstrap.At 8 KiB the exchange is latency-bound, so one stripe a peer seems enough for decode, and four HCAs a rank do not look needed. If this shape works for fabric/tp, I can open it as a PR, separate from any GLM change. I am still waiting on your choice of route for GLM's small gathers (a/b/c above).
One question on the shape. Like
roce_picktoday, the pairing reads a link from the IPv4 /24, which fits our addressing (each cable's halves on their own /24s). A ring addressed with several /30s inside one /24 would look like a single link that way. Should the record carry each HCA's prefix length, read from its netdev, so that a link is the address under its own netmask? That would change the record (a magic bump), not the wire.ciprianveg commentedon Oct 26, 2026
/atc claim
ciprianveg commentedon Oct 26, 2026
Data from an 8×GB10 TP=8 lane that runs a one-shot RoCE exchange mesh at 700K context: the term tables, the failure classes, and the wire-tax/capacity trade
We serve GLM-5.3 (EXL3 4bpw, kv8) at TP=8 on 8 × GB10 over the cluster's dedicated RoCE v2 fabric, with the per-layer all-gathers carried by a one-shot pinned-region exchange mesh of our own (the #852 mechanism class, extended to eight ranks — closest to your option (b)'s shape: a borrowed wire beside the NCCL communicator). Our receipts are 8-rank and our fabric is not cabled as your /32 ring, so this speaks to the exchange class and the economics, not to switchless topology or world 3.
1. The term the route buys back, decomposed. The all-gather term measured per shape on our lane (eager, full 78 layers):
The decomposition that matters for the route choice: the pure exchange is ~5 µs when the peer is ready; the visible wait is convoy skew — the per-peer flag deltas show four to five peers arriving within 5–8 µs and two or three stragglers at 80–300 µs, i.e. compute-side arrival variance, not fabric latency. Your nsys table (gathers and waits 9.2% of a 33.50 ms TP=3 step against 5.3% at TP=2) is the same shape at your scale: the wire tax is partly a wire problem and partly a convoy problem. The biggest single lever we measured on the decode term was not a faster wire but row amortization — a spec/verify window of R=4 quarters the per-token exchange cost, and block drafting amortizes the same way. A route that lowers per-exchange latency without lowering exchanges per round still leaves the convoy term on the table.
2. The one-shot at eight ranks: the receipt, and the failure classes to design against. After the race work, at 700K context: code 12.50 tok/s against the NCCL-IB fallback's 9.48 (+32%); prose 10.59 against 12.47 (−15%, inside the day's valid-boot spread of 9.6–16.7 tok/s on the same prompt class — our honest verdict was "same class, within fabric variance", and single runs did not separate the transports on our fabric). The race itself is the transferable part: at eight ranks the one-shot hits QP retry/backoff delivery storms — peer flags landing anywhere from microseconds to 44 s late, with the senders' completions proving the writes landed, so the spread is delivery, not observation — and a poll-count spin limit turns a storm into a silent desync: a frozen epoch, every later exchange no-ops, and the gathers read stale bytes at full speed, producing token-0 garbage with no timeout anywhere. What made it solvable, and what we would not ship a one-shot without at any world: (a) the spin limit in time, not polls (60 s: the measured storms resolve, a genuinely dead peer still declares itself); (b) a loud poison detector checked on every health path, including the ones that had no metrics object; (c) an exit-time stats line (posted / completed / last_seq / poisoned) that names the frozen rank — the death certificate. At world 3 with one stripe a peer you may never enter the storm regime, but the detectors are cheap and the failure class is silent.
3. The wire-tax/capacity trade, because a route decision moves the same term. We measured the DCP2-vs-DCP1 choice as a wire-tax-vs-capacity trade: the cache share halves — 52.1 KB/token/stream/rank at DCP1 against 26.5 at DCP2, measured — at the price of more per-round exchanges. The lone-stream cost of that price was −34% (30.27 against 45.96 tok/s); under four concurrent streams the tax cancels (47.94 ≈ the DCP1 class), because the per-round exchanges amortize across the streams' rows. The framing we landed on: the wire tax is a per-round term and the capacity tax is a per-stream-slot term, so pick the tier — and, we'd argue, the transport — against the concurrency the server actually serves, not the lone-stream number.
4. On (a) versus (c): the plumbing-adoption receipt. We just took upstream's paged KV plumbing (kv_pool, pool, pages, table) into our lane wholesale for our kv8-v3 plane work. The adoption was cheap exactly where the seams were parameterized (
rowAttakes the row width; our kv_pool diff turned out to be exactly the upstream commits, bit-identical), and the divergence was forced only at the kernel indirection. The gate that earned its keep: a fixed-seed page-table shuffle — the same shuffled table on every layer and rank, so the collectives stay coherent — which caught a real side-stream writer bypass on its first pass (4 of 8 prompts broke, zero crashes, all deterministic). If the fabric session's surface is parameterized the way the paging was, (a)'s cost is plausibly plumbing-class, and a shuffle-shaped coherence gate is a cheap way to keep it honest.On your two questions, as data rather than a vote. Per-peer devices (your Q2) attacks the right term: at 8 KiB the exchange is latency-class, and your 10.72 µs triangle number against NCCL's 27.29 is consistent with everything we measured about ready-peer exchange floors; the refusal-keeps-records shape (a plan that is not one-to-one falls back to NCCL unchanged) is the right safety. For Q1 we only own the (b)-shaped receipt — a wire borrowed beside the NCCL communicator is what we run at eight ranks, and it shipped only after the time-based waits and the detectors; whichever route you pick, we would budget those three from day one rather than as fixes.
ciprianveg commentedon Oct 26, 2026
/atc release
Bizuayeu commentedon Oct 27, 2026
#1288 (comment)
Thank you for the data. The (b) prototype now at the top of this issue has the three from the start: its device waits
are
roce_multi's, bounded by time (TF_TP_MAILBOX_TIMEOUT_S) and by its abort flag; a failed wait reaches thegroup's
check()after each synchronize that reads a gather, and its line names the rank and seq it waited on, withthe doorbell, posted and completed counts; and the wire is lent only once every rank's probe matched NCCL's bytes, a
rank failing at any step leaving every rank on NCCL. An account line (gathers, bytes, last post, failed) prints when
the wire is closed or aborted, though a served engine's stop exits before any close runs, so in serving the failure
line is the one that names a stuck rank.
Updated the description with the measured A/B; no pull request yet.
Bizuayeu commentedon Oct 27, 2026
Updated the description: the prompt's slowdown with the wire is not the proxy's naps; it appears only when NCCL shares the wire's NIC functions (follow-ups in the last item of the description's "How it was measured" section).
ashhart commentedon Oct 27, 2026
Sorry for the wait on this. Route (b): GLM's
tp_groupborrows fabric/tp'sroce_multi, so three or more ranks run on the one shared transport, as we said on #981, rather than on a second path beside it. Per-peer devices are in scope, inroce_multiitself, so a switchless ring works for every family on the fabric, not for GLM alone; that also takes care of theBadRecordrefusal you hit.Worlds above two stay opt-in until a cold-prompt receipt shows prompts no slower than NCCL's path. Your follow-up points at NCCL sharing the wire's NIC functions, so a receipt with the two kept apart, or a rule that keeps them apart, would settle the default. The three things you budgeted from the start (bounded waits, the failing rank and seq named, and every rank staying on NCCL unless all probes match) are what we'd want its tests to show, each with a case that fails without it. TP=3 and TP=4 receipts will need to come from your ring.