[BobClawblaw]

What this changes

tools/zig/lean_check.py now reads sources as UTF-8. Before, it used the locale's encoding, which is cp1252 on Windows, so each multi-byte character counted as two or three columns there and the count differed from Linux and macOS. After, every platform counts the same. lean_new.py already reads as UTF-8.

• Issue: Fixes lean check counts multi-byte characters as several columns on Windows #2096
• Stacked on: main (3bdc4e54a)
• atc overlap --json: no overlap for this change. Coverage was incomplete (pullFilesLimit, pullFilesUnavailable, timeLimit), so I checked the file lists of all open pull requests by hand: none touches tools/zig/lean_check.py.

Checks on this head

• Head commit: 611c0a5f4
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 425 and 425, 0 new (lean_new.py).
• Tests run: with the locale's decoding forced to cp1252 (what Windows does), the base reads 440 and this head reads 425, the same as the UTF-8 count.
• CI: pending.
• Not run, and why: no run on Windows itself, since I have no Windows machine. The cp1252 run above reproduces the 439 that Unlimited-OCR-2 on CUDA: the family's steps on NVIDIA GPUs, the router's long-context speedup (stacked on #2074) #2077's Windows run counted on 023def309.

Receipt

n/a: a development tool that cannot reach inference.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (No test added: the checker has no tests, and the change is the encoding argument.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @BobClawblaw. Reading as UTF-8 makes the count the same everywhere, and forcing cp1252 to reproduce the Windows number was a neat way to show it. Ready to land.

[ashhart]

Thank you @BobClawblaw. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @BobClawblaw! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3