Skip to content

roce_multi on a switchless ring of four: start-up warm-up and the relay #1971

Open

Description

@ashhart

Split out of #1288, whose two questions are answered and on main: per-peer devices in roce_multi (#1516) and GLM's opt-in TF_GLM_ROCE_MULTI=1 (#1524). Thanks @Bizuayeu for both and for the splits. Two pieces of work are left, both ours:

  1. Start-up on a switchless ring of four. warmPeers in glm/cuda_native.zig connects every rank to every other, so a ring without opposite links fails at start. @vdalesandro hit this. Warm-up should touch only the links the ring has.
  2. The relay for rings without opposite links, in roce_multi so every family on the fabric gets it, not GLM alone. Opt-in, with the wire's tests: bounded waits, the failing rank and sequence named in errors, and NCCL unless every probe matches. @Bizuayeu's single-poller and grid change (GLM-5.3-Flash at TP=3: its small all-gathers stay on NCCL, and fabric/tp's three-to-eight-rank RoCE (#1209) refuses a switchless ring with BadRecord. Which route should GLM take, and would per-peer devices be in scope? #1288, 31 Oct) can come in with the relay's rewrite if it helps there, with the relay's tests.

Worlds above two stay on NCCL by default, for the reasons in #1288.

Activity

  1. Bizuayeu commented on Nov 2, 2026

    @Bizuayeu
    Contributor

    @Bizuayeu's single-poller and grid change (#1288, 31 Oct) can come in with the relay's rewrite if it helps there, with the relay's tests.

    #1971

    It's one commit on main a0d24896f (30 Oct), not pushed anywhere yet. Say when the relay wants it and I'll rebase it and put it on my fork. The numbers are in #1288: 0 to 1% in serving, so it earns its place only if the relay needs the single poller or the grid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions