[jschmied]

What this changes

Before this PR, the boot and round loop of a pipeline stage lived in #2115's qwen3_5/cuda_stage.zig, written against the Qwen CUDA engine. This PR moves that code, which has no CUDA calls in it, into zig/src/fabric/stage/driver.zig as Driver(B), generic over an engine binding B. A Metal or XPU stage then writes only its binding.

• Issue: Refs RFC: a layer-range (pipeline) layout across mixed nodes: one hidden row per token over Ethernet, GB10 and Mac in one model #2005 (step 2's shared half; the RFC's section 6.4 protocol, unchanged)
• Stacked on: fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060 (9f679f852, the protocol with the mode byte). This PR adds two commits:
  • 8e3dbcfa3: a stop that a stage receives no longer marks its link broken, so a middle stage passes the stop on. Without it, the last stage of a ring of three exits with Closed. This commit was in qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model at 2 and 3 stages (RFC #2005 step 2, stacked on #2241) #2115 and moves here, because the driver's 3-stage test needs it.
  • 27e4d7e33: the driver and its test.
• atc overlap --json: driver.zig and driver_test.zig are new files. Of the base stack's 66 changed files, one conflicts, with Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835, and that file comes from the base stack, not from these commits. Most other overlaps are unknown because of the tool's PR-files limit. Receipt: atc-overlap-stage-driver.json.

What the driver holds (driver.zig)

• bootRing: the boot over the ring, in rounds 1 and 2. Each stage appends its description, then its ready report, and rank 0 checks every one against the layout with boot.check and checkLayout.
• fingerprint(version, config, ranges).
• Slots: the same code on every rank, so all ranks apply a plan's open, keep, close and generation checks alike.
• Lead: rank 0's rounds. It encodes the plan, applies it to its own slots, sends it, checks the last stage's echo, and runs the forward with the hop.
• serve: a follower's loop. It receives a plan, passes it on, applies it, and runs the forward with the hop.
• Hop, Window and Entry.

What a binding provides

• The types Engine, Cache and Part. A Part has cache and rows fields.
• version, backend and most_streams.
• boundary(e), shapes(parallel, context) and kvDtype(e).
• open and close for a slot's cache, and reset for a stream opened on that slot.
• position(cache) and window(cache), the cache's last window that no plan has settled yet.
• rollback(e, cache, accepted), which keeps that window's first accepted rows.
• forward(e, parts, ids, mode, hop): the binding's layer range over the round's rows. mode is the plan's window or prompt, and hop is null on a single engine.

Checks on this head

• Head commit: 27e4d7e33
• Lean checker, on the base and on this head: 424 on the base and 424 on this head, 0 new
• Tests run:
  • zig test zig/src/fabric/stage/stage.zig -lc, the test-stage module: 44/44 pass. That is 41 named tests: 35 from the base and 6 new in driver_test.zig.
  • zig build test: RC 0, 0 errors
• CI: the host set test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon: RC 0, 0 errors. git diff --check: clean, 0 lines.
• Compiles for macOS on aarch64 (--test-no-exec, which is what test-stage-compile does). Not run on a Mac.

driver_test.zig: a fake engine on loopback rings

The fake engine's layer l mixes l and a per-stream running state into every row, so the output depends on the layers' order and on each stream's state. The test runs 13 rounds:

• a 2-chunk prompt;
• steps;
• a second stream beside the first;
• a keep-alive;
• windows that keep 3, 0, 5 and 2 of 5 rows;
• closes.

What it shows:

• Byte-equal: the fake engine's head bytes from 2 stages (0:3,3:6) and from 3 stages (0:2,2:5,5:6) equal those of one engine.
• Planted faults:
  • a stage that runs one layer short of its declared range changes the bytes;
  • a rollback that moves the position but not the state changes the bytes, on the middle stage and on the last;
  • a dropped rollback is refused as OtherPosition, and rank 0 ends;
  • a stage that describes a range other than the layout's is refused at boot as OtherRange.

Receipt

• n/a for inference speed: this commit moves host code and adds a host test. qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model at 2 and 3 stages (RFC #2005 step 2, stacked on #2241) #2115, rebased on this PR, is the inference receipt: Qwen3.8-27B's logits there are byte-equal to the whole model at 2 and 3 stages, with bf16 and fp8 KV, through this driver.
• Environment: GB10 (aarch64 Linux), Zig 0.17.0.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. The planted faults fail as planted.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

Questions

1. bootRing expects every rank on one backend, as qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model at 2 and 3 stages (RFC #2005 step 2, stacked on #2241) #2115 did. A mixed CUDA and Metal ring would need a backend for each rank in the layout. Should that go into the layout setting, or into the description alone?
2. Lead.step takes the plan's mode, and the binding maps it to its own keep: the last row's logits for a prompt, every row's for a window. Rank-0 sampling on the GPU would need a third keep. Should that be a binding option, or should it wait for the step that adds it?

[ashhart]

Thanks @jschmied. Pulling the boot and rounds out of #2115 into a driver with no CUDA in it, and testing it over a fake engine whose output depends on layer order and stream state, is a clean way to share it. This is host-side only, so it can follow #2060 when that lands. The engine bindings that would use it, #2115 and any Metal or XPU stage, still wait on the decision on #2005, so this one isn't urgent.