mongol-norm

English

Status

Both halves of the library — shaping and normalization (MNG / Hudum) — are verified against the same upstream corpora. CI runs the full suite on Python 3.9 – 3.13 on every push.

✅ Shaping

shape() and same_shape() are cross-validated against two upstream TSV suites:

Suite	Cases	Pass	Notes
`mongfontbuilder/core-hud.tsv`	225	100%	curated regression set
`mongfontbuilder/eac-hud.tsv` (GB/T 25914-2023)	3513	100%	5 cases excluded as UTN ↔ EAC xfail, matching mongfontbuilder's own `pytest.mark.xfail` set
Hand-written unit tests	—	100%	shape / same_shape / joiner tokens (nirugu, ZWJ)

✅ Normalization — same guarantees, machine-checked

normalize() / normalize_text() are a pure function of shape with invariants checked in CI over every corpus encoding:

Property	Result
Round-trip — `shape(normalize(x)) == shape(x)`	3757 / 3757 corpus encodings (100%)
Shape-canonicity — same shape ⟹ same Unicode output	1993 / 1993 shape groups (100%)
Prefix-stability — word and word+suffix share their prefix encoding	99.87% of real corpus pairs

Scope note: normalization is implemented for MNG (Hudum) only — Todo / Sibe / Manchu load shaping rules but have no normalizer yet.

This project was generated with Claude Code (AI-assisted coding). The tests and key parts of the core code have been manually reviewed, and test coverage is extensive (corpus round-trip / shape-canonicity / prefix-stability plus the upstream cross-implementation suites). Treat this as a preview release — it should be fine for normal use; if you hit a problem, please open an issue or PR. Shaping logic is derived from UTN #57 v4 and mongfontbuilder.

Why This Project Exists

Traditional Mongolian script in Unicode has a fundamental problem: the same visible word can be encoded in multiple different Unicode sequences. This happens because:

Letters share glyphs — A and E look identical in medial and final positions; O/U and OE/UE share forms; QA and GA share forms depending on vowel harmony.
Multiple encoding paths — The same tooth glyph (I) can be encoded as I, YA+FVS1, or even two separate I characters.
Redundant FVS usage — Free Variation Selectors (FVS1–FVS4) can create equivalent sequences that render identically.
Joining controls — nirugu (U+180A, the visible stem extender) and ZWJ (U+200D) force letters into their joined forms, and inside those joined contexts even more letters collapse to the same glyph (nirugu+o and nirugu+u render identically).
Suffix particles after MVS/NNBSP — the same rendered suffix can be spelled with different letters (MVS+a and MVS+e both render the chachlag form; MVS+u and MVS+ue render the same connector).

This means:

Search fails: Searching for "sain" (one encoding) won't find the same word in another encoding, even though they look identical.
Deduplication breaks: The same word has multiple Unicode representations.
Indexing is unreliable: Different encodings of the same word produce different keys.

What This Project Does

This is a shaping-aware normalizer for Traditional Mongolian. It:

Shapes the input using the full UTN #57 v4 shaping process (5-step conditional mapping)
Compares glyph sequences to detect identical visual forms
Normalizes to one canonical, FVS-pinned Unicode encoding — same shape ⟹ same Unicode, and it always round-trips

Example: All five of these encode the word "sain" (good) and look identical:

How It Works

The normalizer implements a lightweight Mongolian shaping engine — equivalent to what HarfBuzz does with a font file, but using only the rule data from UTN #57 v4 and the mongfontbuilder project. No font files needed.

Shaping Pipeline (UTN #57 v4 Mongolian-Specific Phase)

Chachlag — Suffix forms for A/E after MVS (Mongolian Vowel Separator)
Syllabic — Consonant/vowel context: onset, devsger, marked, masculine/feminine harmony, dotless
Particle — MVS particle dictionary lookup for specific suffix words
Devsger — I after a vowel (vowel_devsger) gets double-tooth form: I → I+I
Post-bowed — Vowel forms change after bowed consonants (G, B, K, P, F)

Normalization Strategy

normalize is a pure function of shape: any two encodings that shape identically produce the same Unicode output, and the output always round-trips — shape(normalize(x)) == shape(x). It is also prefix-stable. When these goals conflict the priority is round-trip > prefix-stable > shortest.

Per word:

shape the input into its written-unit sequence. Structural characters — MVS, nirugu, ZWJ — appear verbatim as mvs / nirugu / zwj tokens (nirugu renders a visible stem; all three are the evidence for a neighbour's init/medi/fina form). Split the shape at these tokens into chains; the tokens themselves are copied through unchanged.
encode each chain (right-to-left, so appending a suffix can't disturb what precedes it):
1. partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up (position, written-unit) → (letter, FVS) in an FVS-pinned table. Each value renders its unit regardless of neighbours, so the result is a deterministic, O(N), prefix-stable function of the shape.
2. velar-feminine refinement — a G/Gx velar's forward-coupled vowel (a/o/u) is swapped to its feminine partner (e/oe/ue) for clean output.
3. verify — reshape the candidate in full context; accept only if it equals the target chain shape.
4. no search fallback — the table is total over the corpus (with FVS-first selection there are no gap chains left). If an out-of-corpus shape ever misses the table, the safety net returns the input unchanged (round-trip preserved, never a silent mis-encoding). A letter next to a joiner simply looks its unit up at the shifted (joined) position.
post-MVS suffix rule — a chain directly after MVS takes its standalone canonical (drop the MVS, normalize, re-attach), so the spelling never depends on MVS. One exception: chachlag Aa after MVS is written the bare letter a. (The isolate-I → i+FVS1 spelling is pinned in the table itself — no post-processing pass exists.)

Prefix-stability means: if word A = word B + a suffix and their shapes share a prefix, the shared region encodes identically except the single boundary unit whose position changes (final in B → medial in A). The per-unit table delivers this for free — each unit's encoding depends only on its own position, never on its neighbours.

How the table is built (the selection method): offline, a context-independence battery fills each (position, written-unit) slot with the (letter, FVS) that renders exactly that unit in every probed neighbour context (the probes include a bowed consonant, so post-bowed effects can't hide). Candidate order is letter-major, FVS-first within the letter — an FVS exists precisely to pin a form against context, so the pinned variant of the right letter always beats its context-sensitive bare form. The result is exported and shipped as JSON; the battery lives in scripts/gen_normalize_table.py.

Note: output is FVS-pinned, not bare — each unit carries the selector that fixes its form independent of context. This is what makes "same shape ⟹ same Unicode" and prefix-stability hold. The per-unit table is exported as language-agnostic JSON (mongol_norm/data/MNG.normalize.json); schema + consuming algorithm are in docs/data-format.md, so ports in other languages can implement normalize with just a JSON parser.

Installation

mongol-norm is a single self-contained package on PyPI — the shaping/normalize data is bundled, no runtime dependencies:

pip install mongol-norm

Or from source:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

Usage

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")  # Hudum Traditional Mongolian

# Shape: get written-unit sequence
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# Compare: are two encodings visually identical?
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False

# Normalize: one canonical, FVS-pinned Unicode (same shape ⟹ same Unicode)
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

Full-text normalization

normalize() operates on single words. For sentences, paragraphs, or mixed-script text, use normalize_text() — it normalizes each Mongolian word independently while preserving spaces, punctuation, and non-Mongolian text verbatim.

# Normalize a sentence (each Mongolian word normalized independently)
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'

# Mixed script: non-Mongolian text preserved as-is
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

Batch normalization example

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} inputs → {len(unique)} unique form(s): {unique}")
# 4 inputs → 1 unique form(s): {'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}

Command line

After pip install mongol-norm, the mongol-norm command is on PATH (or run python -m mongol_norm.shaper ... without installing).

# Inline text
mongol-norm shape 'ᠰᠠᠢᠨ'                   # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ'               # canonical form
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'    # mixed-script

# Pipe / stdin (use `-` as the text)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
cat doc.txt | mongol-norm normalize-text -

# File in / out
mongol-norm normalize-text -i in.txt -o out.txt

# Batch: one word per line in, one canonical per line out
mongol-norm normalize --batch -i words.txt -o canonical.txt

# Visual-identity check (exit 0 if same, 1 if different)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'

normalize (single-word) skips non-Mongolian characters, so feeding it a multi-line file treats the whole thing as one word. Use --batch for one-word-per-line files, or normalize-text for free-form text.

Running Tests

cd mongol-norm

# Shaping + same_shape + normalize unit tests
python -m unittest tests.test_shaper -v

# Normalize properties: round-trip + shape-canonicity + prefix-stability
python -m unittest tests.test_round_trip

# Normalize-table export (compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud (225) + GB/T 25914-2023 eac-hud (3513, 5 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# Or all together (auto-discovers every tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

The hand-written suite covers:

Test class	What it checks
`TestShape`	`shape()` returns correct written-unit sequence (sain variants, the 5 shaping phases step-by-step, UTN-vs-EAC divergences)
`TestSameShape`	`same_shape()` correctly identifies visually identical vs. distinct encodings
`TestNormalize`	`normalize()` produces canonical output; idempotency; normalized result matches original visually
`TestNormalizeText`	`normalize_text()` handles multi-word, mixed-script, punctuation, empty input; idempotency; word independence
`TestNNBSP`	NNBSP ↔ MVS equivalence (UTN model)

Current totals: 142 tests (unit + property + 225 core-hud + 3513 eac-hud corpus runners), all green on Python 3.9 – 3.13.

Use Cases

Search & Retrieval — Index Mongolian text with unique keys per visual word
Deduplication — Detect identical words encoded differently
Spell Checking — Normalize before dictionary lookup
Corpus Linguistics — Consistent word frequency counts
OCR Post-processing — Standardize OCR output that may use inconsistent encodings
Input Method Engines — Validate and normalize user input

Project Structure

mongol-norm/                          # the repo = the package (single, self-contained)
├── .github/workflows/test.yml        # CI: Python 3.9-3.13 on every push
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # the 5 shaping phases (iii1..iii5) mirroring iii.py
│   ├── _data.py                      # loaders for the bundled JSON
│   └── data/                         # bundled shaping + normalize data
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # per-unit normalize table
├── scripts/                          # dev-only generators (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, for other-language ports
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # vendored from mongfontbuilder

mongol-norm has no runtime dependencies — the shaping/normalize JSON is bundled in mongol_norm/data/. Install with pip install mongol-norm.

Data Sources & Acknowledgments

UTN #57 v4 — Unicode Technical Note: Encoding and Shaping of the Mongolian Script. The authoritative specification for Mongolian shaping rules.
mongfontbuilder by Kushim Jiang — Source for the bundled flat variant tables in mongol_norm/data/ (preprocessed from data.variants / data.particles) and for the core-hud.tsv / eac-hud.tsv regression suites we vendor into tests/data/. Both UTN #57 and mongfontbuilder are authored by the same person.
GB/T 25914—2023 — China national standard for Traditional Mongolian nominal characters; source of the EAC compliance test set.
Claude Code — This project was developed with AI assistance. The shaping rules are derived from the above sources; Claude Code was used to implement and structure the engine.

Supported Locales

Locale	Script	Status
MNG	Hudum (Traditional Mongolian)	✅ Full shaping + normalization
TOD	Todo	⬜ Shaping rules generated, normalization WIP
SIB	Sibe	⬜ Shaping rules generated, normalization WIP
MCH	Manchu	⬜ Shaping rules generated, normalization WIP

Requirements

Python 3.6+ (CI-tested on 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
No runtime dependencies (shaping/normalize data is bundled)

License

MIT License — see LICENSE.

The shaping rules and bundled data are derived from mongfontbuilder (MIT) and UTN #57. Their required notices are retained in NOTICE.

中文

状态

库的两部分 —— 整形与规范化(MNG / Hudum)—— 都对照同一批上游语料验证。CI 在每次 push 上对 Python 3.9 – 3.13 跑完整套件。

✅ Shaping(整形)

shape() 和 same_shape() 对照两套上游 TSV 套件交叉验证:

套件	用例数	通过	说明
`mongfontbuilder/core-hud.tsv`	225	100%	精选回归集
`mongfontbuilder/eac-hud.tsv` (GB/T 25914-2023)	3513	100%	5 个 UTN ↔ EAC 分歧 case 跳过(跟 mongfontbuilder 自己的 `pytest.mark.xfail` 列表一致)
手写单元测试	—	100%	shape / same_shape / joiner token(nirugu、ZWJ)

✅ Normalization(规范化)— 与整形同级保障,机器验证

normalize() / normalize_text() 是 shape 的纯函数,以下不变量在 CI 中对每一条语料编码逐一验证:

性质	结果
往返 —— `shape(normalize(x)) == shape(x)`	3757 / 3757 语料编码(100%)
同形同码 —— shape 相同 ⟹ 输出 Unicode 相同	1993 / 1993 shape 组(100%)
前缀稳定 —— 词与词+后缀共享前缀编码	99.87% 真实语料词对

范围说明:规范化目前只实现了 MNG(Hudum)—— Todo / 锡伯文 / 满文已加载 shaping 规则,尚无规范化。

本项目由 Claude Code(AI 辅助编码)生成;测试与部分核心代码经人工审核,测试覆盖比较充分(语料往返 / 同形同码 / 前缀稳定 + 上游跨实现套件)。当前为预览版,正常使用应无问题;遇到问题欢迎提 issue 和 PR。Shaping 逻辑源自 UTN #57 v4 和 mongfontbuilder。

为什么做这个项目

传统蒙古文在 Unicode 中存在一个根本性问题：同一个可见词形可以用多种不同的 Unicode 序列编码。原因是：

字母共享字形 — A 和 E 在中间和尾部位置外形完全相同；O/U、OE/UE 共享形态；QA 和 GA 根据元音和谐共享形态。
多种编码路径 — 同一个齿形字形可以编码为 I、YA+FVS1，甚至两个独立的 I 字符。
冗余的 FVS 使用 — 自由变体选择符（FVS1–FVS4）可以创建渲染结果完全相同的等价序列。
连接控制符 — nirugu(U+180A,可见的连笔延长符)和 ZWJ(U+200D)会强制字母取连接形,而连接语境下更多字母塌缩成同一字形(nirugu+o 和 nirugu+u 渲染完全相同)。
MVS/NNBSP 后的后缀词 — 同一个渲染出的后缀可以用不同字母拼写(MVS+a 和 MVS+e 都渲染 chachlag 形;MVS+u 和 MVS+ue 同形)。

这意味着：

搜索失效：搜索同一个词的某种编码，找不到另一种编码，尽管它们外形完全一样。
去重失败：同一个词有多种 Unicode 表示。
索引不可靠：同一个词的不同编码产生不同的索引键。

这个项目做什么

这是一个形态感知的蒙古文规范化器。它：

使用完整的 UTN #57 v4 shaping 过程（5 步条件映射）对输入进行字形化
通过比较字形序列来检测视觉上相同的词形
规范化为唯一的、FVS 钉死的 canonical Unicode 编码 —— 同 shape ⟹ 同 Unicode,且必定往返还原

示例：以下五种编码都表示 "sain"（好的），外形完全相同：

工作原理

本规范化器实现了一个轻量级蒙古文 shaping 引擎——功能相当于 HarfBuzz 配合字体文件所做的事情，但仅使用 UTN #57 v4 的规则数据和 mongfontbuilder 项目的变体数据。不需要字体文件。

Shaping 管线（UTN #57 v4 蒙古文特定阶段）

步骤	名称	说明
1	Chachlag	MVS（蒙古文元音分隔符）后的 a/e 后缀形态
2	Syllabic	辅音/元音上下文：onset/devsger/marked/阴阳和谐/dotless
3	Particle	MVS 小品词词典查找
4	Devsger	元音后的 i 获得双齿形态：`I → I+I`（vowel_devsger）
5	Post-bowed	弓形辅音(G/B/K/P/F)后的元音形态变化

规范化策略

normalize 是 shape 的纯函数:任意两个 shape 相同的编码,normalize 输出相同,且始终往返成立 —— shape(normalize(x)) == shape(x),同时前缀稳定。三者冲突时优先级:往返 > 前缀稳定 > 最短。

逐词:

shape 成书写单元序列。结构字符 —— MVS、nirugu、ZWJ —— 原样输出为 mvs / nirugu / zwj token(nirugu 是可见的连笔字形;三者都是邻居字母 init/medi/fina 形的依据)。按这些 token 切成 chain,token 本身原样拷贝。
逐 chain 编码(从右往左,这样加后缀不影响前面):
1. 划分 + 查表(主路径):每个位置优先取单单元(输出干净),否则取最长多单元,查 (位置, 书写单元) → (字母, FVS) 的 FVS 钉死表。每个值不依赖邻居就渲染出该单元 → 确定性、O(N)、前缀稳定。
2. velar 阴性微调:G/Gx 前向耦合的元音(a/o/u)换成阴性(e/oe/ue),输出更干净。
3. 校验:在完整上下文里重新 shape,只接受与目标 chain shape 一致的结果。
4. 没有搜索兜底:FVS 优先的选择下,表对全部语料 chain 是完备的(缺口为零)。语料外的 shape 万一查不到表,安全网原样返回输入(保住往返,绝不静默错编)。紧邻 joiner 的字母只是按移动后的连接位置查表。
MVS 后缀规则:紧跟 MVS 的 chain 用其 standalone canonical(去掉 MVS、归一、再拼回),拼写不依赖 MVS。唯一例外:MVS 后的 chachlag Aa 写裸字母 a。(孤立 I → i+FVS1 的拼写已钉进表本身 —— 不存在后处理。)

前缀稳定的含义:若词 A = 词 B + 后缀,且二者 shape 共享前缀,则共享部分编码完全一致,只有那个位置发生变化的边界单元不同(在 B 里是词尾、在 A 里变词中)。逐单元表天然保证这点 —— 每个单元的编码只取决于它自己的位置,与邻居无关。

表是怎么来的(选择方法):离线跑一个 context 无关性电池 —— 对每个 (位置, 书写单元),挑出在所有探测邻居上下文里都恰好渲染出该单元的 (字母, FVS)(探针含弓形辅音,post-bowed 效应藏不住)。候选顺序是字母优先、字母内 FVS 优先 —— FVS 的意义就是把字形从 context 里隔离出来,所以正确字母的钉死形永远优于其受感染的裸形。结果导出成 JSON 随包发布;电池在 scripts/gen_normalize_table.py。

注意:输出是 FVS 钉死而非 bare —— 每个单元都带着把字形固定住、不受上下文影响的选择符,这正是"同 shape ⟹ 同 Unicode"和前缀稳定成立的原因。逐单元表导出为语言无关的 JSON(mongol_norm/data/MNG.normalize.json),schema 与消费算法见 docs/data-format.md;其他语言只需一个 JSON 解析器即可实现 normalize。

安装

mongol-norm 是单一自包含包,已发布到 PyPI —— shaping/normalize 数据内置,零运行时依赖:

pip install mongol-norm

或从源码安装:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

使用方法

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")  # Hudum 传统蒙文

# 字形化：获取书写单元序列
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# 比较：两个编码视觉上是否相同？
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False

# 规范化:唯一的 FVS 钉死 canonical(同 shape ⟹ 同 Unicode)
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

全文规范化

normalize() 作用于单个词。对于句子、段落或混合文字文本，使用 normalize_text() ——它独立规范化每个蒙古文词，同时原样保留空格、标点和非蒙古文文本。

# 规范化句子（每个蒙古文词独立规范化）
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'

# 混合文字：非蒙古文文本原样保留
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

批量规范化示例

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} 个输入 → {len(unique)} 个唯一形态：{unique}")
# 4 个输入 → 1 个唯一形态：{'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}

命令行

pip install mongol-norm 之后,mongol-norm 命令即在 PATH 上(不安装也可用 python -m mongol_norm.shaper ...)。

# 直接传文本
mongol-norm shape 'ᠰᠠᠢᠨ'                   # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ'               # 输出 canonical
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'    # 混合文字

# 管道 / 标准输入(文本位置写 `-`)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
cat doc.txt | mongol-norm normalize-text -

# 文件输入 / 输出
mongol-norm normalize-text -i in.txt -o out.txt

# 批量:一行一词输入,一行一个 canonical 输出
mongol-norm normalize --batch -i words.txt -o canonical.txt

# 视觉等价检查(相同退出码 0,不同退出码 1)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'

normalize(单词模式)会跳过非蒙古文字符,多行文件直接喂给它会被当成一整个词;一行一词的文件请用 --batch,自由文本请用 normalize-text。

运行测试

cd mongol-norm

# 手写 shaper / same_shape / normalize 测试
python -m unittest tests.test_shaper -v

# normalize 性质:往返 + 同 shape 同输出 + 前缀稳定
python -m unittest tests.test_round_trip

# normalize 表导出(compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud(225)+ GB/T 25914-2023 eac-hud(3513,5 个 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# 或一次跑全部(自动发现所有 tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

手写套件覆盖范围:

测试类	测试内容
`TestShape`	`shape()` 输出正确的书写单元序列(sain 变体、5 步 shaping 分步测试、UTN-vs-EAC 分歧)
`TestSameShape`	`same_shape()` 正确识别外形相同 vs 不同的编码
`TestNormalize`	`normalize()` 输出规范结果; 幂等性; 规范化后与原始词形视觉相同
`TestNormalizeText`	`normalize_text()` 处理多词、混合文字、标点、空输入; 幂等性; 词独立性
`TestNNBSP`	NNBSP ↔ MVS 等价性(UTN 模型)

当前总数: 142 个测试(单元 + 性质 + 225 core-hud + 3513 eac-hud 语料跑批), 在 Python 3.9 – 3.13 上全绿。

应用场景

搜索与检索 — 为每个可见词形建立唯一索引键
文本去重 — 检测编码不同但外形相同的词
拼写检查 — 规范化后再查词典
语料库语言学 — 一致的词频统计
OCR 后处理 — 标准化可能使用不一致编码的 OCR 输出
输入法引擎 — 验证和规范化用户输入

项目结构

mongol-norm/                          # 仓库 = 包(单一自包含)
├── .github/workflows/test.yml        # CI: 每次 push 跑 Python 3.9-3.13
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # 5 步 shaping 阶段 (iii1..iii5) 镜像 iii.py
│   ├── _data.py                      # 内置 JSON 的加载器
│   └── data/                         # 内置 shaping + normalize 数据
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # 逐单元 normalize 表
├── scripts/                          # 仅开发用的生成脚本 (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, 供其他语言移植
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # 来自 mongfontbuilder

mongol-norm 没有运行时依赖 —— shaping/normalize JSON 内置在 mongol_norm/data/。安装:pip install mongol-norm。

数据来源与致谢

UTN #57 v4 — Unicode 技术注释：蒙古文编码与字形化。蒙古文 shaping 规则的权威规范。
mongfontbuilder(Kushim Jiang)— mongol_norm/data/ 内置扁平变体表的来源(从 data.variants / data.particles 预处理而来), 同时也是我们 vendor 进 tests/data/ 的 core-hud.tsv / eac-hud.tsv 回归套件的来源。UTN #57 和 mongfontbuilder 的作者是同一人。
GB/T 25914—2023 — 中国国家标准：传统蒙古文名义字符、表现字符和控制字符使用规则; EAC 一致性测试集的来源。
Claude Code — 本项目使用 AI 辅助开发。shaping 规则来源于上述数据；Claude Code 用于实现和组织引擎代码。

支持的语种

Locale	文字	状态
MNG	Hudum（传统蒙文）	✅ 完整 shaping + 规范化
TOD	Todo（托忒文）	⬜ shaping 规则已生成，规范化开发中
SIB	Sibe（锡伯文）	⬜ shaping 规则已生成，规范化开发中
MCH	Manchu（满文）	⬜ shaping 规则已生成，规范化开发中

环境要求

Python 3.6+(CI 实测矩阵: 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
无运行时依赖(shaping/normalize 数据已内置)

许可证

MIT License —— 见 LICENSE。

整形规则与内置数据派生自 mongfontbuilder(MIT)和 UTN #57,其许可证要求的署名保留在 NOTICE 中。

Name		Name	Last commit message	Last commit date
Latest commit History 78 Commits
.github/workflows		.github/workflows
assets		assets
docs		docs
mongol_norm		mongol_norm
scripts		scripts
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation