Skip to content

Expose written-unit normalization API - #15

Merged
Satsrag merged 2 commits into
Satsrag:mainfrom
satsragclaw:feat/normalize-written-units
Jul 26, 2026
Merged

Expose written-unit normalization API#15
Satsrag merged 2 commits into
Satsrag:mainfrom
satsragclaw:feat/normalize-written-units

Conversation

@satsragclaw

Copy link
Copy Markdown
Contributor

Summary

  • expose MongolianShaper.normalize_written_units() for callers that already have an ordered written-unit sequence
  • accept direct shape() tokens plus stable MVS, Nirugu, and ZWJ control aliases
  • reuse the existing canonical table encoder, structural chain handling, velar-feminine refinement, and exact reshape verification
  • reject malformed containers, non-string or unknown units, and unencodable sequences without partial output or inferred controls
  • document the API contract in English, Chinese, and the language-neutral data-format guide

Contract

  • input is an ordered Sequence[str]
  • positions are inferred from sequence order and explicit structural controls
  • explicit position records are not accepted
  • MVS, Nirugu, and ZWJ are never inferred or inserted automatically
  • empty input returns an empty string
  • malformed input raises TypeError; unknown or non-round-trippable sequences raise ValueError

Verification

  • python3 -m unittest discover -s tests -p 'test_*.py' — 156 tests passed
  • all 1,993 corpus shape groups pass through the public API
  • all 177 core-hud and 3,507 eac-hud direct cases passed (5 documented UTN xfails skipped)
  • 225/225 CORE-HUD and 3,513/3,513 EAC-HUD round trips passed
  • 2,237/2,237 prefix-stability pairs passed
  • normalize-table regeneration is byte-identical
  • python3 -m compileall -q mongol_norm tests scripts
  • uv build produced wheel and sdist
  • installed-wheel API smoke test passed

@Satsrag

Satsrag commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Add this api to CLI

Comment thread mongol_norm/shaper.py Outdated
Comment on lines +1144 to +1148
_PUBLIC_STRUCTURAL_TOKENS = {
'MVS': 'mvs',
'Nirugu': 'nirugu',
'ZWJ': 'zwj',
}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we standardize the capitalization of these control words? Should the first letter be capitalized?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 704b73f. All public written-unit names now use PascalCase. shape() emits Mvs, Nirugu, and Zwj; normalize_written_units() accepts those exact names and rejects the old lowercase/all-uppercase control aliases. The internal lowercase aliases used by particle-source data remain internal only.

@satsragclaw

Copy link
Copy Markdown
Contributor Author

Implemented in 704b73f. The CLI now exposes normalize-written-units with inline, stdin, file, and batch I/O. It accepts either compact PascalCase streams (for example BZwj) or explicit + boundaries (B+Zwj), rejects ambiguous compact parsing, and still requires an exact canonical shape round trip. shape CLI output uses the same PascalCase control names and can be piped back directly. Verified with 174 tests, all 1,993 corpus shape groups, installed-wheel smoke tests, and a real Python 3.6.8 CLI/API smoke test.

@Satsrag
Satsrag merged commit 3effbc6 into Satsrag:main Jul 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants