Skip to content

Fix joiner-aware text normalization and version MNG canonical output - #12

Closed
satsragclaw wants to merge 3 commits into
Satsrag:mainfrom
satsragclaw:refactor/phase-0-contracts
Closed

Fix joiner-aware text normalization and version MNG canonical output#12
satsragclaw wants to merge 3 commits into
Satsrag:mainfrom
satsragclaw:refactor/phase-0-contracts

Conversation

@satsragclaw

@satsragclaw satsragclaw commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • use one Mongolian word-run classifier for strict validation and normalize_text(), preserving Nirugu/ZWJ joining context
  • freeze the current MNG canonical selection policy as mng-canonical/1, expose it through MongolianShaper.canonical_version, embed it in the normalize table, and reject mismatched table versions
  • add reproducible compatibility fixtures:
    • 15 ordered phase-trace vectors covering every MNG shaping rule transition
    • 1,993 byte-for-byte canonical vectors covering every current corpus shape group
  • add scripts/gen_compat_goldens.py with write and --check modes; serialization is key-sorted for Python 3.6 portability
  • fail on unknown corpus aliases, freeze the 1,993-vector cardinality, and require every phase vector to trigger its declared witness rule
  • include test data, goldens, and generator scripts in source distributions

Why

normalize_text() previously split runs at Nirugu and ZWJ even though normalize() accepted both as word characters. Splitting at a joiner changes neighboring positions and could produce a different canonical form.

The goldens lock the current shaping transitions and canonical Unicode outputs so later refactors cannot change them silently. They were replayed against unmodified upstream/main@8d43521962ef489443e18f900491ba9f5ea72e05 with zero differences.

Verification

  • python3 -m compileall -q mongol_norm scripts tests
  • python3 scripts/gen_compat_goldens.py --check
  • python3 -m unittest discover -s tests -p 'test_*.py'
    • 157 tests passed
    • 177/177 core-hud direct cases
    • 3507/3507 eac-hud direct cases, 5 documented UTN xfails skipped
    • 225/225 CORE-HUD round trips
    • 3513/3513 EAC-HUD round trips
    • 2237/2237 prefix-stability pairs
    • 1993/1993 shape groups converge
  • normalize-table regeneration SHA-256 unchanged across repeated generation
  • uv build succeeded for wheel and sdist
  • full 157-test suite passed from the unpacked sdist
  • installed-wheel smoke test passed on Python 3.14
  • unmodified-base replay: 1993 canonical vectors, 0 differences; 15 phase vectors, 0 differences

Closes #10

@satsragclaw satsragclaw changed the title Add Phase 0 behavior contracts Fix joiner-aware text normalization and version MNG canonical output Jul 14, 2026
@satsragclaw
satsragclaw force-pushed the refactor/phase-0-contracts branch from c3df3f6 to f0c2cff Compare July 14, 2026 14:38
@satsragclaw

Copy link
Copy Markdown
Contributor Author

Superseded by #13 and #14 so the joiner bug fix and the canonical-version/compatibility work can be reviewed and merged independently. The final content is preserved across those two PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Version the canonical normalization format to ensure index stability

1 participant