qwen27: document two-Mac 27B serving and the seam's known tie

RUNBOOK.md

@@ -277,6 +277,12 @@ Each settings file names `rank`, `library` and `links`; each link names its peer
 [Speed-up settings](docs/speed-up-mode.md#4-write-the-settings-files) has the schema and link setup.
 That guide's two-Mac Flash Next commands load the checkpoint directly, as one Mac does.
 
+Qwen3.8-27B serves the same way: the same `$HOME/models/qwen3.8-27b` checkpoint on both Macs, one settings file per rank, rank 1 started first and rank 0 within five minutes, with `--parallel 1` because a pair serves one greedy stream.
+Send requests to rank 0 only, and expect no drafter, no prompt cache and no learning: `--drafter`, `--prompt-cache-gib` and `--learn` are refused on a pair.
+Each rank waits up to five minutes for the other's ready word, then exits with `the peer never signalled ready in five minutes`.
+When rank 0 stops, rank 1 stops following but keeps running until you end it, and it exits by itself only when it cannot replay a round, so restart both after a link failure: that fails the request in flight and leaves the pair down until both ranks run again.
+[Qwen3.8-27B on two Macs](docs/recipes/qwen3.8-27b.md#two-macs-over-thunderbolt) has the commands and what the split covers.
+
 On one Mac, GLM loads only when its weights and caches fit in 70% of RAM, 179.2 GB on a 256 GB Mac, and leaves the MTP head on disk when only the head is over.
 `GLM_LOAD_LIMIT_GB=N` sets that limit in GB, up to the GPU's recommended working set; a value that isn't a number above 0 stops the server.
 Past 70% of RAM the server warns at startup: a load that wires most of the RAM can stall macOS until its watchdog restarts the Mac, so run nothing else large beside it.

docs/kernel-contract.md

@@ -127,10 +127,14 @@ float64 output class, without an inherited exception.
 ## Coverage
 
 Metal: softmax, RMS norm, bf16 top-k, the row projection (3, 4, 6 and 8 bits), affine_mm (4 and 8 bits), simd_qmm
-(4 bits) and simd_qmm_bits (3, 6 and 8 bits), and the lane projection (6-bit groups of 32 on every chip; 4- and 8-bit
-formats on the M5's tensor units, where its reg kernel must equal the cooperative one bit for bit), and tile_qmm's
-prompt tiles and their decode twin (4, 6 and 8 bits; the twin bit-equal to the tiles). Gate 5 also holds the rest of
-tile_qmm's pipelines and GLM-5.3-Flash's hc_expand_10.
+(4 bits, and its fp32 store: the same tiles compiled with `TF_OUT` set to float, so a rank can write its exact part
+into a seam) and simd_qmm_bits (3, 6 and 8 bits), and the lane projection (6-bit groups of 32 on every chip; 4- and
+8-bit formats on the M5's tensor units, where its reg kernel must equal the cooperative one bit for bit), and
+tile_qmm's prompt tiles and their decode twin (4, 6 and 8 bits; the twin bit-equal to the tiles), and the TP seam's
+four exchange kernels (`pair_add_rms`, `pair_sum`, `pair_argmax` and `pair_pick`; the input is one rank's own part
+plus the peer's bytes, so an exchange is a function of those bytes alone, `pair_sum`'s fp32 parts meet in the
+frame's one bf16 rounding, and `pair_argmax` and `pair_pick` state their own NaN and tie rules). Gate 5 also holds
+the rest of tile_qmm's pipelines and GLM-5.3-Flash's hc_expand_10.
 CUDA: the RMS norm rows (16-byte form and the fp32-out tree), the bf16 residual's norm tails (MLX's rounded rows and
 the once-rounded ones, each checked on its residual and its output, with four controls each dropping a rounding point), Nemotron's router logits (and their 128-row tiles,
 bit-equal to them) and top-k, the latent top-k's decode form (bit-equal to `tf_super_topk`), the latent combine and

docs/known-issues.md

@@ -37,7 +37,17 @@ release fixes it, and the release notes say which one.
   since 1.0.10's faster products. `FZ_M3_LANES=1` keeps the earlier kernels and their tokens.
 - @Bizuayeu's regression run on shipped 1.1.0 loaded the whole GLM-5.3-Flash NVFP4 checkpoint on two GB10s at TP=2 in 302.1 s, kept one token hash per prompt across three runs and found a passphrase in 199,307 tokens; scored prompts and more than two ranks were not covered ([#1040](https://github.com/ashhart/TensorFold/issues/1040)).
 - DeepSeek-V4.1 CUDA pieces land in this release, but no release binary serves DeepSeek checkpoints until its whole-model gate ([#827](https://github.com/ashhart/TensorFold/pull/827)) passes.
-
+- Qwen3.8-27B in speed-up mode, two Macs over Thunderbolt: a pair serves one greedy stream, drafts nothing, keeps no
+  prompt cache and does not learn, because rank 1 has no drafter checkpoint and each Mac holds only its own halves of a
+  slot's state. `--drafter`, `--prompt-cache-gib`, `--learn` and `--parallel` above 1 are refused, and a request that
+  samples or carries a grammar fails.
+- Qwen3.8-27B in speed-up mode, two candidates whose seam sums land within one bf16 ulp of each other: the order the two
+  Macs add their halves can decide between them, so a reply can differ from the single Mac's by one token and in the text
+  after it. Seen twice. In the `n4-hi` raw-id case (ids `[124160,124161,124162,124163]`, 4 rows, 8 tokens) the single
+  host's last token is `98135` and the pair's is `135917`. In the chat fixture at 256 tokens the single host takes `17077`
+  with a 0.125 logit lead over `369`, the split sum makes the two logits equal and the lower id `369` wins at token 236,
+  so the reply parts from there. A difference is accepted only between the same two contenders when both margins are
+  inside that gap; every other difference must match. The code-128, code-256 and chat-128 fixtures are bit-identical.
 
 ## Living Weights (`--slide`)
 

docs/recipes/qwen3.8-27b.md

@@ -72,6 +72,55 @@ least 256 tokens beyond the previous chunk start, or after 2,048 tokens if no ea
 Prefix reuse resumes only at these chunk starts, so a follow-up can reuse the state before its previous
 reply. The plan comes from rendered tokens; a template without detected markers uses 2,048-token chunks.
 
+### Two Macs over Thunderbolt
+
+Speed-up mode splits one Qwen3.8-27B reply across two Macs over a Thunderbolt 5 cable: both Macs hold the same
+`TensorFold/Qwen3.8-27B-MLX-4bit` checkpoint, each serves it from its own directory with its own settings file, and
+rank 1 starts before rank 0.
+[Speed-up mode](../speed-up-mode.md) builds the server, sets up MCDMA and writes the two settings files; this
+subsection covers what the 27B pair serves and what it leaves out.
+Send requests to rank 0 only; rank 1 holds the other half of the weights and replays rank 0's rounds.
+
+```bash
+# rank 1, on the second Mac: start it first
+zig-out/native/bin/tensorfold-native serve ~/models/qwen3.8-27b \
+  --speed-up rank1.json --port 4101 --parallel 1 --context 32768 --no-update-check
+# rank 0, on the first Mac: start it within five minutes of rank 1
+zig-out/native/bin/tensorfold-native serve ~/models/qwen3.8-27b \
+  --speed-up rank0.json --port 4100 --parallel 1 --context 32768 --no-update-check
+```
+
+`--parallel 1` is required: a pair serves one greedy stream, the default is 8, and any other value stops with
+`--speed-up serves one greedy stream: pass --parallel 1`.
+`--context` defaults to 32768 for this model, and `--no-update-check` is the compatibility switch it is on one Mac,
+so beyond `--parallel` and the settings file each command is what that Mac runs on its own.
+
+Each rank runs every layer and splits the even widths: 12 of the 24 attention heads, 2 of the 4 key/value heads,
+half of the linear attention's heads and value channels, 8,704 of the 17,408-wide MLP, and 124,160 of the untied
+lm_head's 248,320 rows.
+A layer then sums two 5120-wide partials over the link, one from its attention or linear attention output projection
+and one from its MLP's down projection, so a prompt block of up to 128 rows makes two exchanges a layer.
+Embeddings and norms stay whole on both Macs, so the 248,320-row table is not split and a token id means the same
+thing on both ranks.
+At each decided token, each rank takes the argmax of its own 124,160 logits and the pair keeps the better of the two,
+the lower id on a tie.
+
+The pair refuses what needs one Mac's whole state rather than quietly serving without it: `--drafter`, because rank 1
+has no drafter checkpoint, and `--prompt-cache-gib` and `--learn`, because each Mac holds only its own halves of a
+slot's state.
+A pair therefore keeps no prompt cache, learns nothing and drafts nothing.
+A request that samples or carries a grammar fails too, since a round's token comes from that argmax and not from the
+two halves' logits.
+
+Pair replies are bit-identical to one Mac's on three of the four runs the acceptance oracle ships: a code prompt at 128
+and at 256 tokens and a chat prompt at 128 tokens.
+The fourth, the chat prompt at 256 tokens, parts from the single Mac at token 236: there one Mac picks id `17077` with a
+0.125 logit lead over `369`, and on the pair the two halves' sums round to the same logit for both, so the lower id `369`
+wins and the reply differs from there.
+The `n4-hi` raw-id case is the same class, ids `124160` through `124163` in four rows, where the last of eight tokens is
+`135917` on the pair and `98135` on one Mac.
+[Known issues](../known-issues.md) records that boundary and how a difference there is accepted.
+
 ## Weights other than 4-bit
 
 The M5 lane kernels accept MLX affine 2-, 3-, 4-, 5-, 6- and 8-bit projections in groups of 64.

tools/qwen27/pair_accept.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Accept a paired TensorFold serve against the checked-in single-host receipts.
+
+run   send every oracle case twice; both replies must be identical, and either the oracle's token_sha or the
+      case's recorded pair_known one
+warm  one chat turn, then that conversation's next turn: the reply must report cached prompt tokens
+
+The oracle (qwen38_27b_oracle.json, next to this file) is the single-Mac, undrafted, temperature-0 record.
+A pair passes when the same token ids come out, so neither Mac's rank count nor the seam can show up in the text.
+Where a case records pair_known, the pair is accepted at that one value and a third value fails: the pair's
+seam sums can decide between two candidates closer together than a bf16 logit step.
+"""
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+
+DEFAULT_MODEL = "qwen3.8-27b"
+
+
+def post(base, path, body, timeout):
+    request = urllib.request.Request(
+        base.rstrip("/") + path,
+        data=json.dumps(body).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(request, timeout=timeout) as response:
+        return json.loads(response.read())
+
+
+def reply_text(reply, endpoint):
+    choice = reply["choices"][0]
+    if endpoint.endswith("/chat/completions"):
+        return choice.get("message", {}).get("content") or ""
+    return choice.get("text", "")
+
+
+def token_sha(reply):
+    return reply.get("tensorfold", {}).get("token_sha")
+
+
+def request_body(case, model):
+    body = {
+        "model": model,
+        "max_tokens": case["max_tokens"],
+        "temperature": 0.0,
+        "seed": 1234,
+    }
+    if case["endpoint"].endswith("/chat/completions"):
+        body["messages"] = [{"role": "user", "content": case["prompt"]}]
+        body["chat_template_kwargs"] = {"enable_thinking": bool(case.get("thinking", False))}
+    else:
+        body["prompt"] = case["prompt"]
+    return body
+
+
+def load_oracle(path):
+    with open(path, "r", encoding="utf-8") as handle:
+        return json.load(handle)
+
+
+def run(base, oracle, model, repeat, timeout, only):
+    cases = [c for c in oracle["cases"] if only is None or c["name"] in only]
+    if not cases:
+        print("pair_accept: no case matches --only", file=sys.stderr)
+        return 2
+    failures = 0
+    recorded = 0
+    for case in cases:
+        body = request_body(case, model)
+        shas = []
+        texts = []
+        for _ in range(repeat):
+            reply = post(base, case["endpoint"], body, timeout)
+            shas.append(token_sha(reply))
+            texts.append(reply_text(reply, case["endpoint"]))
+        same = len(set(shas)) == 1 and len(set(texts)) == 1
+        known = (case.get("pair_known") or {}).get("token_sha")
+        if same and shas[0] == case["token_sha"]:
+            print("PASS {} token_sha={} ({} identical runs)".format(case["name"], shas[0], repeat))
+        elif same and known is not None and shas[0] == known:
+            recorded += 1
+            print("KNOWN {} token_sha={} is the pair value recorded for this case, not the oracle's {}".format(case["name"], shas[0], case["token_sha"]))
+        else:
+            failures += 1
+            if not same:
+                why = "the {} runs differ".format(repeat)
+            elif known is None:
+                why = "token_sha {} != oracle {}".format(shas[0], case["token_sha"])
+            else:
+                why = "token_sha {} is neither the oracle's {} nor the recorded pair value {}".format(shas[0], case["token_sha"], known)
+            print("FAIL {} {} shas={}".format(case["name"], why, ",".join(str(s) for s in shas)))
+    total = len(cases)
+    print("pair_accept: {}/{} cases match the oracle, {} recorded as different, {} unexpected on {}".format(total - failures - recorded, total, recorded, failures, base))
+    return 0 if failures == 0 else 1
+
+
+def cached_tokens(reply):
+    usage = reply.get("usage") or {}
+    details = usage.get("prompt_tokens_details") or {}
+    value = details.get("cached_tokens")
+    return value if value is not None else usage.get("cached_tokens")
+
+
+def warm(base, oracle, model, timeout):
+    case = next((c for c in oracle["cases"] if c["name"].startswith("chat")), None)
+    if case is None:
+        print("pair_accept: the oracle has no chat case to build a conversation from", file=sys.stderr)
+        return 2
+    first = post(base, case["endpoint"], request_body(case, model), timeout)
+    used = cached_tokens(first)
+    text = reply_text(first, case["endpoint"])
+    if not text:
+        print("FAIL warm: the first turn returned no text", file=sys.stderr)
+        return 1
+    second_body = {
+        "model": model,
+        "max_tokens": 32,
+        "temperature": 0.0,
+        "seed": 1234,
+        "messages": [
+            {"role": "user", "content": case["prompt"]},
+            {"role": "assistant", "content": text},
+            {"role": "user", "content": "Now give the same example for a 3 by 3 matrix."},
+        ],
+        "chat_template_kwargs": {"enable_thinking": False},
+    }
+    second = post(base, case["endpoint"], second_body, timeout)
+    cached = cached_tokens(second)
+    print("warm: turn 1 prefill marked {}; turn 2 cached={}".format(used, cached))
+    if cached is None or cached <= 0:
+        print("FAIL warm: the next turn reused no prompt state (cached={})".format(cached))
+        return 1
+    print("PASS warm: the next turn resumed {} cached prompt tokens".format(cached))
+    return 0
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Accept a paired TensorFold serve against single-host receipts")
+    sub = parser.add_subparsers(dest="command", required=True)
+    for name in ("run", "warm"):
+        p = sub.add_parser(name)
+        p.add_argument("--base-url", default="http://127.0.0.1:4100")
+        p.add_argument("--oracle", default="tools/qwen27/qwen38_27b_oracle.json")
+        p.add_argument("--model", default=DEFAULT_MODEL)
+        p.add_argument("--timeout", type=float, default=600.0)
+        if name == "run":
+            p.add_argument("--repeat", type=int, default=2)
+            p.add_argument("--only", nargs="*", default=None)
+    args = parser.parse_args()
+    oracle = load_oracle(args.oracle)
+    try:
+        if args.command == "run":
+            return run(args.base_url, oracle, args.model, args.repeat, args.timeout, args.only)
+        return warm(args.base_url, oracle, args.model, args.timeout)
+    except urllib.error.URLError as err:
+        print("pair_accept: {} did not answer: {}".format(args.base_url, err), file=sys.stderr)
+        return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())