[ThinkOffApp]

What this changes

Before, nothing in the tree could check the dumps a mixed-TP run writes. This adds tf-tp-check, a tool under zig/tests/ with its own build target, as agreed on #1236. tf-tp-check reference, repeat and agree check rules 1, 2 and 4 over each rank's dumps. It reads fabric/ without changing it, and it enables no mode.

• Issue: Refs Tensor parallel across Metal and CUDA ranks (Mac + NVIDIA): tracker #1236. This is step 1, the checker as a tool only. The dense mode follows after MCDMA over a ConnectX link, and serve --cluster on real hosts (transport only) #1594's step 2a, behind a named cluster-config key.
• Stacked on: main.
• atc overlap --json: run on this head against main at 53311e7. ATC marks 4 open pull requests as conflicting, 6 clean and 78 unknown, and reports the scan as incomplete (otherPullRequestsLimit, pullFilesLimit, pullFilesUnavailable, timeLimit). The 4 are gemma4 Metal: Gemma 4 E4B (NVIDIA Nemotron 3.5 Bolt E4B) on M1 to M4, per-layer inputs and shared KV #2273, gfx12 WMMA dense tile: 64-deep swizzled K step, LDS-only barrier (same bytes) #2280, core: Gemma 4 and Qwen 3.5 on CUDA load with direct reads, 28-31 % faster cold; Flash Next shares the read-ahead #2282 and profile: register the pool-page switches for Dory, Gemma 4, Spark2.5 and Nemotron's Metal paging #2306, one file each. From their file lists, the only path this PR shares with them is CHANGELOG.md.

Checks on this head

• Head commit: d334c45, on base 53311e7.
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 252 and 252, all inherited, 0 new.
• Tests run on this head and base, with the command and pass, fail and skip counts, on an Apple M5 Max:
  • zig build test-tp-check: 28 passed, 0 failed.
  • zig build test test-golden: 723 of 723 steps succeeded. 6,629 passed, 67 skipped, 0 failed.
  • zig fmt --check build.zig zig/build zig/tests/tp_check*.zig and git diff --check: both clean.
  • zig build tf-tp-check -Dtarget=aarch64-linux-gnu and -Dtarget=x86_64-linux-gnu: both build. The start on a GB10 was on the previous head and was not repeated.
• CI: all seven jobs passed on this head.
• Not run, and why: the checker's tests on Linux or Windows, which CI covers. zig build dev-tools, because no API the tf-* tools call changed. The checks over a live run on this head, because nothing on main writes these dumps yet. The dense mode adds that writer.

Receipt

n/a: this tool reads files and cannot reach inference, and no serving path, kernel or arithmetic changes. Outside the new files, this adds one line to build.zig and makes runtime() in zig/build/tp.zig public.

The host evidence for the tool itself:

• Same results as the checker it replaces. On 84 runs over 77 real dumps from our mixed-TP runs, this binary matched both earlier private builds in exit code, stdout and stderr. That covers reference, repeat and agree, with exit 0 and exit 1, and dump versions 1 and 3. A second person replayed all 84 and rehashed the dumps.
• Dump versions: 1, 2 and 3. Version 2 adds phases, and version 3 adds canonical bootstrap.
• Refusals: foreign magic, unknown versions, every cut of a valid dump, phase bytes a version does not allow, and a writer asked for a phase or version the format lacks.
• Negative controls: making version 1 accept any phase, version 2 accept bootstrap, or the parser accept version 4 each fails a test.

Limits:

• No real version 2 dump survives, so version 2 is covered by synthetic tests only.
• 76 of the 77 dumps have their hash in their own earlier receipt. One does not, so treat that one as unanchored.
• The dumps are not in this pull request. The tests here use synthetic fixtures.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

The inference checklist does not apply: the Receipt is n/a.

[ashhart]

Thanks @ThinkOffApp. This is the shape we agreed on #1236: its own tool under zig/tests/ with its own target,
reading fabric/tp without changing it, and no mode. Lean is 425 before and after, and nothing serves differently.
The planted faults in the tests (a flipped bit, a changed dtype, layer or phase, a split, a gap, a wrong fed hash,
every cut of a dump) cover the rules well. Four small things before it lands:

1. agree exits 0 when it compared nothing. If every position is skipped (vocab sizes differ, or non-finite
   logits), compared is 0 but tp_check_main.zig still reaches ok = true and prints zero divergence. Please
   fail when compared == 0, as the usage text promises. Worth considering too: NaN or Inf logits from the mixed run
   as a failure rather than a skip.
2. repeat passes two empty dumps ("PASS: all 0 records identical"). reference already fails on no
   collectives; please do the same here.
3. Buffer lengths that aren't whole elements. Nothing checks send.len and recv.len against
   dtype.size(). For an all-reduce, reference.accumulate's assert then panics instead of exiting 2. For an
   all-gather or exchange, compare walks only whole elements, so a difference in a trailing partial element
   passes. One refusal in the parser covers both, with a test.
4. The 2 GiB loader test runs in every zig build test, on every host, holding about 2 GiB while it runs (and
   reserving 2 GiB of disk on Windows). Please keep it under test-tp-check only. Also: does std's
   readPositionalAll already cap each read on Darwin? If it does, the custom loader may not be needed.

Could you paste the Windows and Linux CI results once they finish? This is the first time zig build test on
Windows compiles the tp module for the Windows host. And please add a CHANGELOG line, for example "Tests:
zig build tf-tp-check builds a checker for the dumps each rank of a mixed Metal and CUDA tensor-parallel run
writes ... (#2321, part of #1236)."

One thing for step 2, not this PR: the dense mode's writer will live in engine code, and engine code shouldn't
import from zig/tests/. When the writer arrives, it may be cleanest for the format (tp_check_dump.zig) to move
beside it, with the checker importing it from there.

[ThinkOffApp]

Thank you @ashhart! All four are done on the new head. The first three each have a test that fails without the change.

1. agree now fails when it compared nothing, and when the mixed run's logits are not finite. Non-finite logits in the solo run stay a skip.
2. repeat fails when the dumps hold no records.
3. The parser refuses a send or recv that isn't whole elements of its dtype, with BadLength, so it exits 2 instead of panicking or missing a trailing partial element.
4. The 2 GiB loader test now runs only under test-tp-check. I also checked the Darwin read: with one read of the whole file, that test fails on this Mac at Zig 0.17, so the bounded loader stays.

CHANGELOG line added, in your wording.

CI on the previous head, all green: zig 5m2s, zig-windows 7m3s, zig-macos-dev-tools, fmt, release-paths (ubuntu-24.04), benchmark-clients on Linux and Windows. The new head's run will follow.

Noted on the format moving beside the writer in step 2.

[ThinkOffApp]

The new head d334c45 is up, on main at 53311e7, and CI passed on all seven jobs.

zig (Linux), 15m19s: zig build test, test-kernel-contract, test-golden, test-server-security test-pull-security, fuzz-cortheon and the NFKD generator tests all passed.
zig-windows, 19m52s: zig build test, test-golden and fuzz-cortheon all passed. That covers the tp module compiling for the Windows host, which you asked about.
zig-macos-dev-tools, fmt, release-paths (ubuntu-24.04) and benchmark-clients on Linux and Windows passed.
The CI logs don't print pass counts for these steps, so these are step results. Both jobs ran longer than on the previous head (5m2s and 7m3s), and I haven't looked into why.

Two things I should have put more plainly in my last comment:

In Zig 0.17, File.readPositionalAll has no per-read cap of its own. It loops on readPositional with whatever is left of the buffer. The loader calls it in 8 MiB pieces. As one call over the whole 2 GiB + 17 byte file, the sparse test fails on this Mac, and only the read size differs, since the buffer is allocated once either way. So the loader stays.
For BadLength, the test covers the parser's refusal. The exit code 2 comes from bad() in main. I checked that by hand on one such dump, and no test asserts it.
The description is updated for this head.

[ashhart]

/atc exchange f312be89192aae591dc3d1d95c10f1fd2dcfbd7577f518f06a7bc0ffc5df7759