[akol1]

What this changes

zig/src/core/tokenizer/nfkd_data.zig is 24,459 generated lines, and nothing in the tree generates or checks it. Its provenance header named .agents/tools/gen_nfkd_tables.py, which is not in this repository, so the table could not be regenerated or verified from a clean checkout — and CI would not have noticed if it drifted from the Unicode database it claims to come from.

This adds that generator as tools/zig/nfkd_tables.py, wires its --check into the no-GPU workflow, and covers it with tests.

The generator reproduces the committed table exactly. Regenerating changes no data: the diff is the header and three reflowed comments, plus one added declaration:

• the header names a path that exists, instead of the missing .agents/ one;
• pub const version = "16.0.0";, which unicode_data.zig already carries from its own generator;
• the comment reflow keeps each one to a single line within 120 columns, per the lean rule.

So this is a provenance repair rather than a data change. Reviewers can confirm the tables are untouched by diffing the file: every sequences, decompositions, cccs, lower_sequences, lowercases and marks line is byte-identical.

It follows unicode_tables.py's conventions deliberately — the same ROOT/OUT/VERSION shape, the same single-line provenance header, the same pub const version declaration, and the same refusal to run on an interpreter whose unicodedata is not pinned to 16.0.0. That guard is what stops a newer Python from silently emitting tables from a different UCD release.

What it does not change

No runtime behaviour, no model code, no build graph. The Zig source the engine compiles is unchanged apart from the added version constant, which nothing reads.

Receipt

Environment. macOS on Apple silicon, Zig 0.17.0, Python 3.14 with unicodedata 16.0.0.
Commit. f2ef56403 on pr-nfkd-tables, two commits on top of main at d1018b318.
Checkpoint and revision. None; this touches no model path.

Exactness. nfkd_data.zig is unchanged as data. tools/zig/nfkd_tables.py --check exits 0 against the committed file and exits 1 when a single stray declaration is appended, so the check is known to bite rather than merely to pass.

Tests. zig build test: all 115 test binaries exit 0 when run directly. tools/zig/test_nfkd_tables.py: 5 tests pass — reproduction, header, pinned version, refusal on the wrong unicodedata, and that the refusal happens before the table is written. Removing the guard from the generator fails two of the five, which is how I know the tests are not vacuous. tools/zig/unicode_tables.py --check and tools/zig/nfkd_tables.py --check both exit 0 on this branch. lean_check.py reports no finding in the touched files; git diff --check is clean.

CI. The zig job passes, including both steps this PR adds, "NFKD tables are current" and "NFKD generator tests".

The first CI run failed, and the failure is worth knowing about: ubuntu-latest's python3 is 3.12, whose unicodedata is 15.0.0, so the generator's version guard refused to run rather than emit tables from the wrong UCD release. The guard did exactly its job; my workflow step had assumed an interpreter it did not have. Python 3.14 carries 16.0.0 (3.12 is 15.0.0, 3.13 is 15.1.0), so the second commit pins 3.14 via setup-python and reaches it through that action's python-path output, leaving python3 for the other steps as it was.

zig-windows fails, and it is not this change. It fails inside zig build test on zig/src/fabric/tp/{bootstrap,fate,sock,fate_test}.zig and zig/src/core/digest_key.zig with Windows calling-convention and ws2_32 member errors, which is std-library socket code compiled for a target it does not support. The name nfkd appears nowhere in that log, and no file this PR touches is in the error list. The same zig-windows: failure with zig: success signature appears on main's own ornith-main run, so this is pre-existing separate work.

Not run, and why. No GPU step is affected and this change touches no kernel, so no device run was warranted. I did not reproduce that failure locally either; the ornith-main run is the evidence that it predates this branch.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (Not applicable: no inference path is touched.)
• No change to a GPU kernel, so no kernel-contract registration is needed.
• Generated files carry a generator that reproduces them, and CI now enforces it.
• Comments are single lines within 120 columns; no file approaches 600 lines.

Suggested by @ashhart in the #825 stack review as a change that could land on its own.

[ashhart]

Thanks @akol1 for giving nfkd_data.zig a generator that reproduces it with no data change, in the same shape as unicode_tables.py, and for the CI step that checks it under the pinned Unicode. We are carrying both commits onto current main ourselves, kept as yours, with a separate commit that makes the reproduction test (tools/zig/test_nfkd_tables.py:10) skip when unicodedata.unidata_version is not 16.0.0, as the test at :26-27 already does, pins actions/setup-python to the commit main's pack-compare.yml already uses, and shortens the 129-column comment at .github/workflows/no-gpu.yml:36. It touches no inference path, so it needs no device runs: once CI is green on current main, which should also clear the zig-windows failure you traced to the older base, it lands. You need not rebase; if you are in the middle of a change, please tell us so we do not collide.

- Synthema, Ash's AI Assistant.

[ashhart]

Thanks @akol1, this is landed on main as 65b4f13 and 9a52175 with you as the author, plus our small commit bd26436 on top, as promised: the reproduction test now skips on an interpreter whose Unicode is not 16.0.0, actions/setup-python is pinned to the same commit as main's other workflow, and the long workflow comment is within 120 columns. The generator reproduces nfkd_data.zig with no data change, and the host suite, kernel contract, golden, security and fuzz gates all pass on main with it; CI now checks the table under the pinned Unicode on every run.

- Synthema, Ash's AI Assistant.