[jschmied]

What this changes

This adds zig/src/fabric/stage, the pipeline stage link from #2005. It's a TCP ring of ranks that carries each round's plan and hidden rows, builds on Linux and macOS, and uses no CUDA or Metal module. Each link proves a shared key both ways after the hello, before any plan or rows frame. Nothing calls it yet: the Qwen 27B stage mode (step 2) is the first user.

• Issue: Refs RFC: a layer-range (pipeline) layout across mixed nodes: one hidden row per token over Ethernet, GB10 and Mac in one model #2005 (step 1).
• Stacked on: main. Six commits on ef9fa70: the link, the shared key, the stop and refusals, the frame magic (fe6aeebc7), the non-blocking sockets for macOS (43ea71321) and zig fmt on secret.zig and frame.zig (27f01d6).
• atc overlap --json:
  • The one shared file is build.zig, which gets one added line in the test step list. Most open PRs touch build.zig, and they merge cleanly with this one.
  • Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835 conflicts because it adds its own steps on the same line. That's an adjacent-line conflict, and whichever lands second keeps both lines.

Why a module of its own: #2005 first proposed reusing fabric/tp's control plane. But fabric/tp/config.zig imports cuda (for graph_cache), and zig/build/tp.zig builds tp only beside a CUDA module, so a Metal process can't import it. A check on an M3 Pro confirmed that bootstrap, fate, sock and log compile for aarch64-macos, and that planlink fails only in its Linux CPU-pinning helpers. A module that needs only libc avoids changing the tensor-parallel code that GLM and DeepSeek-V4.1 serve with. Folding fabric/tp/sock.zig onto this module's net.zig can follow once the control plane is split.

Design:

• Topology: each rank listens on its own address only (a wildcard 0.0.0.0 is refused, WildcardAddress), connects to the next rank, and accepts the one before. Rows go round the ring, back to rank 0 for the head.
• Hello: both ends of every link check the world size, that they're neighbours, and the checkpoint's and layout's fingerprint. A mismatch is refused by name (OtherWorld, NotNeighbour, OtherFingerprint).
• Shared key: TF_STAGE_SECRET_FILE names a file of 16 bytes to 4 KiB that only its owner may read (mode 0600); the key is its SHA-256. When a link forms, both ends send a fresh nonce from /dev/urandom, then an HMAC-SHA256 over a label, their role, rank, world, the fingerprint and both nonces, checked in constant time. The key never crosses the wire and a proof can't be replayed. A rank without the key is refused by both neighbours (BadSecret), and a stranger that connects and talks first is refused at its first frame (BadMagic).
• Frames: a 32-byte little-endian header (magic, version, kind, round, rows, payload bytes), then the payload. The magic is 0x54534654, so its bytes on the wire read TFST. The payload can be sent as several parts, for example e.h and e.r back to back. Kinds are hello, plan, rows and stop. Every send and receive has a deadline.
• Refusals: a frame of another kind or round, or larger than the receiver's buffer, is refused before its payload is read. Any failed call leaves the link broken (error.Broken from then on), since its stream may be mid-frame.
• Failure: a stop on a broken link hangs up both links instead of sending a header that could complete a half-sent frame. A rank that fails closes both its links, so each neighbour's next call fails and the ring ends as one. That's RFC: a layer-range (pipeline) layout across mixed nodes: one hidden row per token over Ethernet, GB10 and Mac in one model #2005's fail-fast option (b).
• Settings: TF_STAGE_RANK, TF_STAGE_WORLD, TF_STAGE_PEERS (host:port,..., in rank order), TF_STAGE_SECRET_FILE and TF_STAGE_TIMEOUT_S, each refused by name when missing or malformed.
• Not covered: the key authenticates a link when it forms. It doesn't encrypt, and it doesn't protect each frame after the handshake, so the link is for a trusted network.

Checks on this head

• Head commit: 27f01d6.
• Lean checker: 424 on the base, 424 on this head, 0 new.
• Tests: zig build test-stage passes 20 of 20 on the GB10 (Linux aarch64), counting 18 named test blocks and two that pull the files in. The previous head (19 of 19) also ran 20 repeats clean on the GB10 and natively on macOS 27.2 on an M3 Pro (3 of 3 runs); this head wasn't rerun on the Mac. The tests run rings of two and three ranks on loopback threads:
  • a plan and rows split into parts go round three rounds exact;
  • a slow middle stage changes only the timing;
  • another fingerprint, another world, a missing key and a stranger are refused;
  • a wrong round or kind, or an oversized frame, is refused and the link is broken after it;
  • a stop ends the next rank's wait, and a stop after a send that failed mid-payload hangs up, so the next rank sees Closed (this test fails on the previous stop);
  • a failed middle rank is seen by both neighbours;
  • the settings and key-file refusals (too short, too long, not owner-only) and a wildcard listen address.
• test-stage-compile builds the tests for aarch64 Linux and macOS. zig build test (which now includes both), the CI set (test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon) and git diff --check pass.
• CI: green locally as above.
• Not run: two hosts. The tests are loopback only; a GB10 and a Mac over Ethernet come with step 2's stage tool.

Receipt

n/a: nothing reaches inference yet. No engine, server or CLI path calls the module.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @jschmied. An independent review on current main (d8b80bd):
Verdict: CHANGES (one, small), then LAND.

• It's groundwork only. A new module, zig/src/fabric/stage/, plus its build step. Nothing serves through it yet.
• Existing TP paths are untouched.
• Failure handling is bounded and named.
• One real hazard: a stop() after a failed send can complete a half-sent frame with header bytes.

Applying it: both commits apply cleanly on main d8b80bd. The lean check stays at 424 (5,063 files, against
5,055 on main). Both author lines are @jschmied's real noreply. Nothing was built or run here (no Zig).

What it changes on existing paths: nothing

• The only file outside the new folder is build.zig: one added line,
  @import("zig/build/stage.zig").steps(b, test_step);.
• zig/build/stage.zig (new) adds test-stage (host tests on Linux and macOS only) and test-stage-compile
  (aarch64 Linux and macOS). Both are added to zig build test.
• No engine, server, CLI or fabric/tp file imports fabric/stage. The other hits for "stage" in the tree are
  DeepSeek's and GLM's own host_stage.zig and cuda_vision_stage.zig, which are unrelated.
• So there's no change to NCCL or RoCE, rank agreement, fingerprints, start-up order or hot-path env reads. The
  module uses only libc and std: no CUDA or Metal module.

Failure handling

• Every wait is bounded.
  • Connect retries every 50 ms until the ring's deadline.
  • Accept, send and receive each poll against a deadline, in steps of at most 1 s (net.zig:80-157).
  • The ring forms within one deadline (TF_STAGE_TIMEOUT_S, default 60 s). Each later send or receive gets its own.
  • TCP keepalive (30 s idle, 10 s interval, 6 probes) also catches a host that vanishes.
• Peer death: a hang-up makes recv return 0, which is error.Closed. A rank that fails closes both links
  (Ring.close), so both neighbours' next calls fail and the ring ends as one. That's tested on a ring of three.
• Partial reads and writes: sendAll and recvExact loop until every byte is through, and retry on EINTR and
  EAGAIN. A send uses MSG_NOSIGNAL (or SO_NOSIGPIPE on macOS), so a dead peer can't raise SIGPIPE.
• Frames: a 32-byte header (magic TFST, version 1, kind, round, rows, bytes).
  • decode refuses a wrong magic, version or kind.
  • It also refuses a payload larger than min(max_bytes, out.len) (default 256 MiB), before reading the payload
    (frame.zig:35-42, ring.zig:117).
  • Payloads land in the caller's buffer, so no allocation is sized by an untrusted length.
  • A wrong kind or round is refused (WrongKind, OutOfOrder). Any failed call sets broken, so later calls return
    error.Broken. All of this is tested.
• The handshake:
  • join exchanges hellos (rank, world, fingerprint) and checks OtherWorld, NotNeighbour and
    OtherFingerprint.
  • Then prove runs: fresh 32-byte nonces from /dev/urandom each way, then HMAC-SHA256 proofs. Each proof is tied
    to a label, role (connector or acceptor), rank, world, fingerprint and both nonces, and compared in constant time
    (timing_safe.eql). A replayed or reflected proof fails.
• Settings: each one is refused by name when missing or malformed. World is 2 to 64, rank is below world, there is
  one host:port per rank, and the timeout is at least 1 s. The key file must be owner-only (mode & 077 == 0, via
  statx or fstat) and hold at least 16 bytes. The key is that file's SHA-256, and the buffer is zeroed after.

Security

• Bind: each rank listens on its own entry's IPv4 address only (net.listen), not on every interface. But the
  config takes any host string, so an operator who writes 0.0.0.0:port gets every interface (small point 2).
• Authentication: a shared key, proven both ways before any plan or rows frame. The PR is explicit that there's
  no encryption or per-frame integrity after the handshake, so it's for a trusted network.
• What a hostile host on that network can do:
  • Stop the ring forming. accept takes the first connection, so a stranger who connects first is refused
    (BadMagic or BadSecret). The ring then fails rather than waiting for the real neighbour. That's
    denial-of-service, not entry.
  • Read the hello. The hello (rank, world, the 32-byte checkpoint fingerprint) is sent before the key proof
    (ring.zig:62-68). So an unauthenticated connector learns those values. The second commit's subject ("proves the
    ring's shared key both ways before any frame") overstates this: the key is proven before any plan or rows frame,
    not before the hello.
  • Tamper after the handshake. It could change frames in flight, which the PR's "trusted network" note covers.

Change

1. stop() after a failed send can complete a half-sent frame.
   • Ring.stop (ring.zig:95-97) writes a 32-byte stop header without checking broken.
   • If a send timed out partway through a payload, the next rank is still waiting for the rest of that payload.
     If 32 or fewer bytes were missing, the stop header fills the gap, and the next rank returns a full, wrong rows
     frame. A pipeline stage would then compute a round on corrupted hidden rows.
   • Ask: make stop() close the links instead when broken, and add a test where a send fails mid-payload and is
     followed by stop().

Small points (not blocking)

1. Exchange the key proof before the hello, so nothing is sent to an unauthenticated peer. Or reword the commit and
   the PR body to "before any plan or rows frame".
2. Refuse a wildcard listen address (0.0.0.0) for the rank's own entry, unless explicitly opted in, so "never every
   interface" holds whatever the operator writes.
3. A key file over 4,096 bytes is silently cut to its first 4,096. Refuse it, or say so.
4. The PR says "19 of 19". The module has 17 test blocks; the difference is likely the build's root blocks.

Tests

There are 17 test blocks across frame, net, secret, config and ring_test:

• rings of two and three on loopback threads, carrying split payloads exactly over three rounds;
• a slow middle stage;
• another fingerprint or world, a missing key on either side, and a stranger;
• a wrong round or kind, followed by Broken;
• an oversized frame, refused before its payload;
• stop;
• a failed middle rank, seen by both neighbours;
• the settings and key-file refusals.

It's a new module, so there's nothing for the tests to fail against on main. The author ran them on Linux aarch64
(20 repeats) and on macOS (3 runs).

Device cells

• Linux aarch64 host and macOS host | no model | zig build test-stage and test-stage-compile, plus one case
  with stop() after a send that fails mid-payload (after the change): all pass, the next rank sees Closed, never
  a full frame.
• GB10 and an Apple Silicon Mac on one Ethernet segment | no model | a two-rank ring over real TCP with
  TF_STAGE_SECRET_FILE on both: three rounds exact, then a wrong key refused by name. This comes with the next
  step's stage tool, as the PR says.

[jschmied]

Thanks @ashhart. Fixed in c68421b, a third commit on top (no rebase).

The change: Ring.stop() no longer writes a header on a broken link. It hangs up both links instead (shutdown, descriptors kept for close), and a stop whose own send fails does the same. So the next rank sees error.Closed, never a whole frame made from a half-sent one.

• New test: a 64 MiB rows send times out partway (past loopback's socket buffers), then stop(); the next rank's recv returns Closed. On the old stop() the same test fails (expected error.Closed, found error.Timeout).

Small points:

1. Reworded rather than reordered: the key option's comment and the PR body now say the key is proven after the hello, before any plan or rows frame. The second commit's subject still says "before any frame"; I left it, since rewriting it would mean a force-push under review. Say if you'd rather have the proof first; it's a small change, but a peer with another fingerprint would then be refused as BadSecret rather than OtherFingerprint.
2. A rank's own address that resolves to 0.0.0.0 is now refused (WildcardAddress), so "never every interface" holds whatever the operator writes. There's no opt-in; one can come with a real need.
3. A key file over 4 KiB is now refused (SecretTooLong) instead of being cut. Tested.
4. Right: the module had 17 named test blocks and has 18 now. The runner reports 20, because it also counts the two unnamed test {} blocks that pull the files in.

Checks on c68421b: zig build test-stage 20 of 20 and test-stage-compile (aarch64 Linux and macOS) on the GB10; zig build test, the CI set (test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon) and git diff --check pass; lean 424 on both, 0 new. Not rerun natively on the Mac for this commit.

[jschmied]

@ashhart, one more commit before your Mac run: 43ea71321. I ran test-stage on an M3 Pro (macOS 27.2, zig 0.17.0) ahead of your run, and the test "a stop after a send that failed mid-payload hangs up" blocked until it was killed.

The cause:

• macOS ignores MSG_DONTWAIT on a blocking socket. sendAll polls for POLLOUT and then sends the 64 MiB payload; on macOS that send waits until the peer has taken every byte, so the deadline never fires.
• The link connected non-blocking but set the socket back to blocking afterwards, and an accepted socket was never made non-blocking.
• A small C reproducer shows the same thing: on a blocking descriptor the send waits forever, and with O_NONBLOCK it makes a partial send and then returns EAGAIN.

The fix: every link socket stays non-blocking after connect and accept. The send and receive loops already poll and take partial transfers.

Results:

• M3 Pro: test-stage 20/20, and the stop test takes 0.22 s (the 200 ms timeout plus teardown). test-stage-compile 3/3. With the same patch on top, fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060's head passes test-stage 35/35.
• GB10 (Linux): lean is unchanged (0 new), and zig build test, the CI set and diff-check are green.

Without the fix, any send larger than the socket buffer to a stalled peer would ignore its timeout on macOS. #2060 picks this commit up when it's rebased onto #2012.

[ashhart]

Thanks @jschmied, everything from our last review is in. stop() now hangs up instead of writing a header on a broken link, and the new mid-payload test shows the next rank sees Closed rather than a whole frame. The four small points are done too: the key wording, the 0.0.0.0 refusal, the 4 KiB key limit, and the test counts. Thanks for fixing the frame magic: 0x54534654 puts "TFST" on the wire, and the test checks those bytes.

Two tiny things for the body: it still says "Head commit: c68421b" and "Three commits" (it's four now, ending at fe6aeebc). We'll reword the second commit's subject ("before any plan or rows frame") when we squash. Before merging, we'll run test-stage on a GB10 and on a Mac, since the last two commits haven't run natively on macOS. Then it lands, and the stage-mode PR can rebase onto it.

[jschmied]

Thanks @ashhart. zig fmt on secret.zig and frame.zig is in 27f01d6cb, so fmt should pass here and, after the rebase, on #2060. The body now names six commits and the new head. Verified before the push: zig build test, the CI set, lean (0 new) and the server goldens.

[ashhart]

Thanks @jschmied, the macOS socket fix and the fmt commit close it. The non-blocking sockets after connect and
accept explain the hang exactly, and the 0.22 s stop test shows it. We'll squash with the second commit's subject reworded
as discussed, and it lands after our host run of test-stage. #2060 can rebase onto it after that.

[ashhart]

Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @jschmied! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3