Skip to content

Fix normalize_text handling of Nirugu and ZWJ - #13

Merged
Satsrag merged 1 commit into
Satsrag:mainfrom
satsragclaw:fix/joiner-aware-normalize-text
Jul 14, 2026
Merged

Fix normalize_text handling of Nirugu and ZWJ#13
Satsrag merged 1 commit into
Satsrag:mainfrom
satsragclaw:fix/joiner-aware-normalize-text

Conversation

@satsragclaw

Copy link
Copy Markdown
Contributor

Summary

  • use one Mongolian word-run classifier for strict validation and normalize_text()
  • keep Nirugu and ZWJ inside the same Mongolian run as their neighboring letters
  • add regression tests for standalone and mixed-text normalization

Bug

normalize() accepted Nirugu and ZWJ as word characters, but normalize_text() split runs at those characters. Splitting at a joiner changes neighboring letters' isol/init/medi/fina positions, so the same word could receive a different canonical result depending on which API was used.

Verification

  • python3 -m compileall -q mongol_norm tests
  • python3 -m unittest discover -s tests -p 'test_*.py'
    • 145 tests passed
    • 177/177 core-hud direct cases
    • 3507/3507 eac-hud direct cases, with 5 documented UTN xfails skipped
    • 225/225 CORE-HUD round trips
    • 3513/3513 EAC-HUD round trips
    • 2237/2237 prefix-stability pairs
    • 1993/1993 shape groups converge

@Satsrag
Satsrag merged commit 9ce7887 into Satsrag:main Jul 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants