Expose written-unit normalization API - #15
Conversation
|
Add this api to CLI |
| _PUBLIC_STRUCTURAL_TOKENS = { | ||
| 'MVS': 'mvs', | ||
| 'Nirugu': 'nirugu', | ||
| 'ZWJ': 'zwj', | ||
| } |
There was a problem hiding this comment.
Should we standardize the capitalization of these control words? Should the first letter be capitalized?
There was a problem hiding this comment.
Updated in 704b73f. All public written-unit names now use PascalCase. shape() emits Mvs, Nirugu, and Zwj; normalize_written_units() accepts those exact names and rejects the old lowercase/all-uppercase control aliases. The internal lowercase aliases used by particle-source data remain internal only.
|
Implemented in 704b73f. The CLI now exposes |
Summary
MongolianShaper.normalize_written_units()for callers that already have an ordered written-unit sequenceshape()tokens plus stableMVS,Nirugu, andZWJcontrol aliasesContract
Sequence[str]TypeError; unknown or non-round-trippable sequences raiseValueErrorVerification
python3 -m unittest discover -s tests -p 'test_*.py'— 156 tests passedpython3 -m compileall -q mongol_norm tests scriptsuv buildproduced wheel and sdist