qwen27 metal: the 27B as one stage of fabric/stage's ring, its binding of the shared driver (RFC #2005 v1) by jschmied · Pull Request #2375 · ashhart/TensorFold

jschmied · 2026-11-06T15:32:09Z

What this changes

Qwen3.8-27B on Metal can now run as one stage of an RFC #2005 ring, as #2115 does on CUDA. zig/src/native/qwen27_stage.zig is its binding of #2241's Driver(B). The driver itself is unchanged. With no ring, nothing on the served path changes.

Boundary: two bf16 planes of 5,120 values a row: plane 0 h, the residual stream before the add, and plane 1 r, the pending block output. These mean the same as CUDA's planes: tf_q35_norm computes H = bf16(H + R), and Metal's norm bfloat(float(hidden) + float(pending)).
Forward: forward.Span runs layers [first, end). A later stage takes h and r through the host before its first layer. Each stage gives its rows on after its last layer, waiting for its command buffer first. Rank 0 then takes the last stage's rows back for the final norm and the head.
Load checks: laneOrder and uniformMlp read only the stage's own layers through Weights.mine(), and a stage takes no Neural Engine share.
Keeps: a stage keeps only its own layers. Runner.keep skips the others, and keepDevice refuses a stage runner by name.
Weights: Model.loadStage and gpu_weights.load(..., owned_layers) read only the stage's own layers' projections. The other layers' slots carry only their mixer kind. Each 27B stage peaks at 12.8 GB instead of about 19.6 GB for the whole model.
Error path: Runner.keep now ends its encoder when it fails part way. The same one-line fix is offered against main on its own in a separate PR.
v1 limit: Metal's runner must keep a window before its next pass, but the protocol reports a stream's accepted rows only in that stream's next round. So when a round runs without a stream, that stream's window is kept whole. A later report of fewer rows is refused by name (WindowKeptWhole). The scenario's one-row windows and drafted windows, continued in the next round, are unaffected.
Issue: Refs RFC: a layer-range (pipeline) layout across mixed nodes: one hidden row per token over Ethernet, GB10 and Mac in one model #2005.
Stacked on: fabric stage: one stage driver over an engine binding, so CUDA, Metal and XPU stages share the boot and the rounds (#2005, stacked on #2060) #2241 (stage-driver 27e4d7e), two commits on top.
Overlap: a file-level check against open PRs finds the qwen27 files shared with qwen27 metal: keys and values in core pool pages behind TF_QWEN27_METAL_KV_PAGES=1 (#1293) #2283, qwen27 metal: qk projections, flat glue and Metal 4 verify passes, each bit-identical and switchable #2342, Living Weights on the dense Qwen3.8-27B on Metal: the pocket and the native trainer (#1568) #2023, Metal: serve the dense Qwen3.8-27B straight from a GGUF file (Q4_K_M, Q8_0) (#1920) #2078 and others. forward.zig, decode_round.zig and gpu_weights.zig will need a rebase once those land.

Checks on this head

Head commit: 19350ca
Lean checker on the base (27e4d7e) and on this head: 424 and 424, new findings 0.
Tests run:
- zig build test --summary all: 623/626 steps, 222/227 tests (4 skipped). The one failure is ring_test's "a stop after a send that failed mid-payload hangs up…", which is already in the base. It hangs in net.sendAll at 0 % CPU and was stopped after 5 minutes. zig build test-stage on the untouched base 27e4d7e hangs the same way on this M3 Pro, so it isn't this change. The cause is the older fabric/stage/net.zig on fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060's branch: its sockets are blocking, so on macOS a send to a stalled peer never reaches its deadline. Main fixed this in 5d1bf95. With only main's net.zig swapped into 27e4d7e, test-stage passes 44/44, and main itself 20/20. This branch inherits the fix when fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060 is rebased;
- zig fmt --check on the touched files and git diff --check: clean;
- zig build test-qwen27: 145/145 (the gpu_weights.zig tests now run in a step; they ran in none before);
- gate 1, below, rerun on this head: cmp-equal, 47 rows, 23,342,080 bytes, both ranks exit 0.
CI: not run yet.
Not run, and why: gate 2 (a GB10 and a Mac over LAN) is blocked by the open question below.

Receipt

Environment: this commit, Zig 0.17.0, Apple M3 Pro (no tensor units), macOS 27.2, TensorFold/Qwen3.8-27B-MLX-4bit at 70ae7fac.
Gate 1: two stage processes on one Mac, TF_STAGE_LAYOUT=0:32,32:64 over a loopback ring. Every logits row of tf-qwen27-stage's scenario (47 rows) is byte-equal (cmp) to the single Metal engine's.
- The scenario is qwen35_stage's, with the prompt in 128-row chunks: a 600-token prompt, eight one-row windows, a second stream's 300-token prompt, eight two-stream rounds, a keep-alive, four drafted 5-row windows (one rolled back whole), a last window, then close.
- A one-process run over all 64 layers with the boundary through the host and back (TF_STAGE_LOOP=64) is byte-equal too.
Paths reached: Metal before the M5. The binding has not run on an M5, so the laneOrder path that needs tensor units hasn't either.
Exactness: logits bytes against the single engine, as above. No served path changes.
Speed: not a goal of this step. The ring's speed against the best single machine is RFC: a layer-range (pipeline) layout across mixed nodes: one hidden row per token over Ethernet, GB10 and Mac in one model #2005's later gate.
Licence of new weights or drafter: n/a.

Open question: gate 2 (GB10 + Mac)

boot.check requires every rank to report rank 0's version and backend (bootRing passes one backend for the ring) and identical shapes. This binding reports prompt_chunk = 128, the pre-M5 prompt pass, and CUDA's is larger. So a mixed CUDA + Metal ring is refused at boot today. A per-rank backend and agreed shapes need a change in boot or Driver, and that change waits for #2060 and #2241 to land. Across backends the tokens can legitimately differ from either single machine, so I read "tokens equal per placement" there as one placement giving the same tokens on repeat, at any chunking, and across streams. Is that what you meant, @ashhart?

Checklist

The title and description match the head commit.
Comments are one line.
No personal data, machine names or local paths. No AI attribution lines.
Gate 2 and speed against the best single machine: open, as above.

…und's plan and hidden rows (Linux and macOS) zig/src/fabric/stage is the first step of ashhart#2005: each rank listens on its own address only, connects to the next rank and accepts the one before. A hello checks that both ends of every link agree on the world, on being neighbours, and on the checkpoint's and layout's fingerprint. Frames carry a 32-byte header (magic, version, kind, round, rows, payload bytes) and every send and receive has a deadline. A frame of another kind or round, or past the receiver's buffer, is refused, and any failed call leaves the link broken, since its stream may be mid-frame. A rank that fails closes both links, so its neighbours see it and the ring ends as one. It uses libc sockets only, with no CUDA or Metal module. fabric/tp's control plane imports cuda through config.zig, so this does not reuse it. TF_STAGE_RANK, TF_STAGE_WORLD, TF_STAGE_PEERS and TF_STAGE_TIMEOUT_S name a rank's settings. `zig build test-stage` runs rings of two and three ranks on loopback threads, and `test-stage-compile` builds the tests for aarch64 Linux and macOS.

… any frame A stage rank opens a port, so a ring needs a key: TF_STAGE_SECRET_FILE names a file of at least 16 bytes that only its owner may read, and the key is its SHA-256. When a link forms, both ends send a fresh nonce from /dev/urandom, then an HMAC-SHA256 over a label, their role (connector or acceptor), rank, world, the fingerprint and both nonces. Each end checks the other's proof in constant time, so the key never crosses the wire and a proof can't be replayed. A rank without the key is refused by both its neighbours (error.BadSecret), and a stranger that connects and talks first is refused at its first frame. This authenticates the ring when it forms. It doesn't encrypt, and it doesn't protect each frame after that, so it's for a trusted network.

…wildcard listen and a key file over 4 KiB are refused

…y report, rank 0's round plan, the rows layout and the keep-alive, each mismatch refused by its name

…yte-stable and each refused by its name, for a CUDA or Metal stage

…ytes on the wire read TFSP; fixtures rewritten

…d the keep-alive beats mid-round too RFC ashhart#2005 section 6.1 has a stage refuse rows whose generation is not its own, but the plan carried none. A stream is now 28 bytes: the generation u32 at 24, 0 for an open (BadOpen otherwise), and plan.sameGeneration refuses another by name (OtherGeneration). The keep-alive is due once 2 s pass without a plan, not only when idle, so a stage's long forward elsewhere in the ring never reads as a dead ring; a stage already counts any frame as heard. Ready's warm means executable for the declared shapes, not graphs captured. The plan fixtures are rewritten (140 bytes for the round) and plan_open_generation added.

… prompt chunk, so a stage picks the engine's kernels A Qwen stage runs a prompt chunk through the prompt GEMMs and a window through the one-row-exact path, and the bits differ, so a follower must know which one rank 0 runs. A keep-alive is a window; any other mode byte is BadMode. Fixtures plan_prompt, plan_bad_mode and plan_keepalive_mode added; plan_bad_reserved pokes byte 21.

…e passes it on to the next recv marked the link broken on any error, a received stop too, so the rank that got it could only hang up: in a ring of three the last stage saw Closed where rank 0 had stopped cleanly. A stop is a whole frame; only other errors break the link now.

… binding, so CUDA, Metal and XPU stages share them driver.zig takes the backend-neutral half of ashhart#2115's Qwen CUDA stage: the boot over the ring (descriptions, then ready reports, rank 0 checking each), the fingerprint, the slots that apply a plan's open, keep, close and generation checks alike on every rank, rank 0's rounds with the plan's echo, and a follower's serve loop. `Driver(B)` is generic over a binding that names its engine, cache and forward part and supplies the boundary, the shapes, the cache's position, window and rollback, and the forward of its layer range given the round's mode. driver_test.zig runs it over a fake engine whose every layer mixes its index and a per-stream state into the rows: 2- and 3-stage loopback rings give one engine's bytes, a stage one layer short and a rollback that moves the position but not the state both change the bytes, a dropped rollback is refused as OtherPosition, and a range described against the layout is refused at boot as OtherRange.

…g of the shared driver (RFC ashhart#2005 v1) native/qwen27_stage.zig binds Driver(B) for Metal: the boundary is h and r as bf16 planes of 5120, the same meaning as CUDA's (the stream before the add, and the pending block output). A stage loads only its own layers' projections (Model.loadStage), its forward runs layers [first, end) and hands the boundary through the host at the range's edges, and its keeps touch only those layers. A window a stream has not reported when another round runs is kept whole, and a later report of fewer rows is refused by name (v1). Gate 1, two stages on one Mac (layers 0:32 and 32:64 over loopback): every logits row of tf-qwen27-stage's scenario is byte-equal to the single engine's, across prompt chunks, two streams and drafted windows.

ashhart · 2026-11-06T16:08:38Z

Thanks @jschmied. Logits byte-equal to the single engine at two stages and over the host loop is the exactness half of the #2005 gate. With no stage built, span stays null and the served path reads unchanged.

As our 13:15 note on #2005 says, engine stages wait until a placement shows a gain, with its speed beside the best single machine that can hold it, so this stays open until then.

Three things for the stage path meanwhile:

Undefined layers. Other stages' layers are undefined (unowned), but laneOrder walks every layer at load on tensor-unit Macs, and uniformMlp() does too for the Neural Engine check. So a stage on an M5 would read them. Could both skip layers the stage doesn't own?
The errdefer. Please take qwen27 metal: a keep that fails part way ends its encoder, so the error returns instead of aborting the process #2376's guarded errdefer once it's settled, rather than a second copy.
Rebase. Please rebase after fabric stage: one stage driver over an engine binding, so CUDA, Metal and XPU stages share the boot and the rounds (#2005, stacked on #2060) #2241 and fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060, so fabric/stage picks up main's net.zig.

On gate 2: across backends, we'd read "tokens equal per placement" as you do, with one addition. That placement's tokens should agree with each single machine's except where the top two are near-tied, with every difference listed with its margin.

…t Macs and for the Neural Engine check too Other stages' layers hold no tensors, but laneOrder walked every layer on a Mac with tensor units and uniformMlp walked every layer for the Neural Engine check, so a stage on an M5 would read them. Both now walk Weights.mine(), the stage's own layers. A stage also takes no Neural Engine share: the share builds fp16 copies for every layer, its own and the others'. qwen27's test block now runs gpu_weights' tests, which no test step reached before. Refs ashhart#2005.

jschmied · 2026-11-06T17:38:15Z

Thanks @ashhart. On 19350ca7e, a new commit on the same base, not rebased yet:

Undefined layers: laneOrder and uniformMlp() now walk Weights.mine(), the stage's own layers, with a new test that the uniform MLP check reads only those. The M5 path itself hasn't run, because this is an M3 Pro.
The errdefer: waits for the rebase. qwen27 metal: a keep that fails part way ends its encoder, so the error returns instead of aborting the process #2376's guarded form comes in then, with no second copy here. Until that lands this branch keeps its old line.
Rebase: after fabric stage: one stage driver over an engine binding, so CUDA, Metal and XPU stages share the boot and the rounds (#2005, stacked on #2060) #2241 and fabric stage: the protocol layer, boot description, round plan, rows layout and keep-alive, with golden bytes for every message (#2005 step 1b, stacked on #2012) #2060 land.

One addition you didn't ask for: a stage takes no Neural Engine share, since ane_mlp.Share.init builds programs for every layer. The binding never calls loadShare today, so this is only a guard. I can drop it if you'd rather keep this minimal.

qwen27.zig's test block now imports gpu_weights.zig, whose tests ran in no step before. test-qwen27: 145/145, was 142. Gate 1 again on this head: cmp-equal to the single engine, 47 rows, both ranks exit 0. Lean checker 424 and 424, new findings 0. zig fmt --check and git diff --check are clean. The full zig build test still stops at ring_test's hang from the old net.zig, as before; nothing in qwen27 failed before that point.

On gate 2, understood: near-ties excepted, with every difference listed with its margin.

ashhart · 2026-11-06T20:53:47Z

Thanks @jschmied. mine() covers both walks, and the new test is the check we wanted. Keep the Neural Engine guard: a stage holds only part of the MLPs, so it shouldn't build a share. One note for the rebase: #2060 is on main now, and main's share condition in model.zig gained !c.moe(), so your stage == null goes beside it. The rest waits as agreed: #2241 for the rebase, #2376 for the errdefer, and a placement that shows a gain for the stage path itself.

jschmied added 11 commits November 3, 2026 09:15

fabric stage: a stop on a broken link hangs up instead of sending, a …

c68421b

…wildcard listen and a key file over 4 KiB are refused

fabric stage: the protocol layer, a stage's boot description and read…

b38728d

…y report, rank 0's round plan, the rows layout and the keep-alive, each mismatch refused by its name

fabric stage: golden bytes of every protocol message, good and bad, b…

8fbb30c

…yte-stable and each refused by its name, for a CUDA or Metal stage

fabric stage: the payload magic is 0x50534654, so its little-endian b…

05a8853

…ytes on the wire read TFSP; fixtures rewritten

Navigation Menu

Repository navigation

qwen27 metal: the 27B as one stage of fabric/stage's ring, its binding of the shared driver (RFC #2005 v1) - #2375

qwen27 metal: the 27B as one stage of fabric/stage's ring, its binding of the shared driver (RFC #2005 v1)#2375
jschmied wants to merge 12 commits into
ashhart:mainfrom
jschmied:metal-stage-v1

jschmied commented Nov 6, 2026 •

edited

Loading

Uh oh!

ashhart commented Nov 6, 2026

Uh oh!

jschmied commented Nov 6, 2026

Uh oh!

ashhart commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jschmied commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Checks on this head

Receipt

Open question: gate 2 (GB10 + Mac)

Checklist

Uh oh!

ashhart commented Nov 6, 2026

Uh oh!

jschmied commented Nov 6, 2026

Uh oh!

ashhart commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jschmied commented Nov 6, 2026 •

edited

Loading