Skip to content

qwen27: document two-Mac 27B serving and the seam's known tie - #1437

Open
akol1 wants to merge 6 commits into
ashhart:mainfrom
akol1:qwen27/pair-docs
Open

qwen27: document two-Mac 27B serving and the seam's known tie#1437
akol1 wants to merge 6 commits into
ashhart:mainfrom
akol1:qwen27/pair-docs

Conversation

@akol1

@akol1 akol1 commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this changes

Documents the two-Mac 27B path where a reader will look for it. docs/recipes/qwen3.8-27b.md gains a "Two Macs over Thunderbolt" subsection: rank 1 first, then rank 0, the settings files, the --parallel 1 requirement, the per-rank split widths, and every flag the pair refuses. docs/known-issues.md records the bf16 reduction-order tie at the seam and the pair's refusals as known limits, naming the two fixtures that hit the tie. RUNBOOK.md gains the pair's two-line delta. No code, test or gate behaviour changes.

Refs #1236 and #808 — the record says what this stack does, and in the same section what #808 asked for that it does not do. Stacked on #1432, #1433, #1434, #1435, #1436; the diff against main carries them too, so review this pull request's own commit, cb488f24.

Receipt

  • Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: documentation only, so no build belongs to this commit. The facts it records came from the runs in the two pull requests above: macOS 27.0.1 on an Apple M4 Pro (Mac16,11), Metal only (applegpu_g16, no tensor units), checkpoint TensorFold/Qwen3.8-27B-MLX-4bit at revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, 4-bit affine with group 64. The recipe's commands are the ones those runs used.
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a for a documentation change. The record cites both sides: the single Mac reproduces all four oracle shas, and the pair matches three and records chat-256 at 58daf9743077 because of the tie, so a reader is not left to discover that from a test log.
  • Decode speed before and after, from tools/bench_openai.py: n/a — no code changed. The recipe points at the harness rather than quoting timings, and the receipt it refers to has one Mac at 12.3 to 13.6 tok/s on these fixtures and the pair at 16.8 to 17.1; the recipe says plainly that two Macs buy capacity for a model that does not fit one, not two times the speed on a model that does.
  • Prompt speed before and after: n/a — the recipe says prompt processing stays on each rank's own rows and makes no prompt-speed claim; the pair's fixtures have 14 and 31-token prompts, where one Mac reports 0.198 to 0.382 s to first token and the pair 0.131 to 0.229 s.
  • Tests run, with pass, fail and skip counts:zig build test test-golden at this tree: 7116 passed, 45 skipped, 0 failed of 7161, with 535 of 535 build steps; the 45 skips are the image-codec and pixel cases that need libjpeg-turbo 3.1.4.1 or libwebp 1.6.0, plus SKIP qwen27-tp-check: no --model DIR. This commit adds no test and the commands it prints were run by hand on the two Macs.
  • Not run, and why: documentation only, so nothing to run; the commands the recipe prints were exercised on the two Macs named above and are not presented as checked on any other platform, and the pair flags it lists are the ones the served path refuses by name.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. — documentation only: no code changed, and the recipe describes the pair path the previous pull requests build.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — documentation only; the record states which of these the pair refuses by name instead of claiming them.
  • No precision traded for speed. If the bits change, the description says which sums change and why. — nothing traded; the known-issues entry names the seam's reduction-order tie and both contenders rather than leaving it to be found.
  • A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. — this commit changes no kernel and quotes no new SUMMARY; the pair's filtered SUMMARY stays in the first pull request.
  • Prompt processing is no slower than the last release. — n/a, documentation only: the recipe makes no prompt-speed claim and says why.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. — the recipe names macOS and two Metal Macs over Thunderbolt 5, and says the rest was not run.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. — documentation only, no tests added; the red-before receipts for the code are in the two pull requests above.
  • Comments and docstrings are one line. No measurements or history in the source. — prose only; the lean check is unchanged at 428 problems, 428 on the base, 0 new.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines. — the added text uses no machine name, user path, peer address or serial; the settings fields are placeholders filled from the environment, and the checkpoint is named by revision.

akol1 added 6 commits October 26, 2026 18:18
@akol1
Add the pair seam and register its kernels with the kernel contract
137cf63 
The seam carries one round's partials between two Macs: pair.metal writes a
frame, the host relay owns the exchange instead of spinning on device
counters (which only works while both ranks share a launch group), and
pair_sum completes the partial on the far rank. The payload is f32, the
seam's accumulation, and simd_qmm grows the matching store. pair.zig splits
into pair_layout and pair_loopback so each module stays under the 600-line
rule.

Nothing registered the seam before: three of the four kernels already ran
under the older ABI on the shipped glm_dsa speed-up path, but there was no
record and no spec, so no gate covered the kernels this stack changes. Add
the four launch records, the seam spec with its four probes, the
`metal pair ` route in the limits table, and the fp32-out simd_qmm variant,
so the transport and family changes that follow have a SUMMARY to post.

The glm_dsa call sites and the two Pipeline.load test call sites are forced
by the shared signatures; the family changes themselves come later.
@akol1
Rank-slice the qwen27 family and gather the whole embedding table
ba27237 
On a pair rank the embedding table is global while the vocabulary in the
split config is the shard's width, so the gather bound took the shard's
number and read the wrong rows; embedBound now takes the whole table and
refuses a bound below it. Rank-sliced projections, the head seam, the GDN
partials and the attention plan follow the same split, and the drafts-off
path keeps the two ranks in lockstep.

The regression test pins the old behaviour (124160) against the new
(248320) and fails when the split width is used again.
@akol1
Advertise the pair late, warm the first turn, and die with the peer
95c9452 
A supervised pair has to survive an unattended night: rank 0 now waits for
rank 1's handshake before it advertises /health, the first turn is warmed
so the client does not pay the load, and either rank exiting takes the
other down instead of leaving it serving alone. The launch script starts
and stops both, with the kill/restart drill recorded in the receipt.

The benchmark and TF_DUMP_IDS receipts in server/ are the support code for
those runs; dump.zig and input.zig are their two halves.
@akol1
Pin the pair against the single host in the test suite
dff7857 
tf-qwen27-tp-check runs rank-1 parity bit for bit and then joins two
rank-local shards through pair.Loopback on one Mac, comparing ids against
the single-host load and the forced replay. It SKIPs without a checkpoint,
so the suite stays clean on hosts that have none. The module registration
is the only build change.
@akol1
Add the pair acceptance and supervision scripts
53a0d27 
pair_accept.py walks a live pair through the fixture set and compares each
token_sha against the recorded single-host oracle, so the check needs no
second reference machine. pair_supervise.sh starts rank 1 on the peer,
waits for the handshake, starts rank 0, polls both health endpoints,
restarts either rank if it dies, and stops both on SIGTERM. Hosts and paths
come from arguments or environment only.
@akol1
Document two-host 27B serving, the defect record, and the runbook delta
cb488f2 
Add the recipe for a supervised 27B pair and carry the runbook delta the
lifecycle needs. known-issues records the embedding gather defect (the
split vocabulary used as the gather bound) and what the fix leaves behind:
the code-128, code-256 and chat-128 fixtures now match the single host bit
for bit, and the two differences still seen are bf16 reduction-order ties,
recorded with their receipts rather than called parity. No code.
@akol1
akol1 marked this pull request as ready for review October 27, 2026 04:32
@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner
  • known-issues.md and the recipe present the chat-256 divergence as accepted behaviour, which amounts to a new tie policy. Please drop that once the seam is exact (see core: add the pair seam and register its Metal kernels with the kernel contract #1432).
  • RUNBOOK says that when rank 0 stops, rank 1 "keeps running until you end it". The code (qwen27_link.watch) exits rank 1 within about 3 s. The code matches the requirement, so please fix the text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@akol1 @ashhart