feat(asr): add opt-in canonical written-form CER route (cer_canonical) - #2
Open
liutaocode wants to merge 2 commits into
Open
feat(asr): add opt-in canonical written-form CER route (cer_canonical)#2liutaocode wants to merge 2 commits into
liutaocode wants to merge 2 commits into
Conversation
Correctness (scores change only where results were previously wrong): - aispeech_norm: keep rows whose recognition text is empty so they are scored as deletions; previously "key\t" rows were dropped and the matching reference rows were skipped downstream, understating error rates (up to ~50% on sets with many empty outputs). Count malformed rows in the node trace (row_stats) and fail when a file yields no parseable <key>\t<text> rows instead of scoring a perfect 0.0. - wenet scoring wrapper: score reference utterances missing from the hypothesis as empty hypotheses (pure deletions) instead of silently skipping them; report utterance coverage (num_ref_utts, num_hyp_utts, num_hyp_missing_utts, num_hyp_extra_utts, key samples) in the scoring result; raise instead of reporting 0.0 when zero reference tokens are covered. Code-switch zh/en side scores stay lenient since monolingual subsets legitimately cover zero tokens. Performance (bit-identical scores, regression-tested): - wenet_compute_cer: reset only the DP submatrix an utterance uses; resetting every previously grown row made all utterances after one long utterance pay O(longest^2) (17x on a synthetic corpus, ~10x end-to-end on a real 1.4k-utterance code-switch run). - aispeech_norm: preload num2words map tables once per node call and share the number cache instead of re-globbing and re-reading map files per line/token (~40x per-call overhead). Also fix a latent NameError in asr_num2words' fallback map loading (load_and_sort_map was referenced but never defined; the path only survived because the default map dir has no root-level .map files). Tests: nine regression cases covering empty/missing hypotheses, coverage reporting, zero-token rejection, order-independent totals, and monolingual code-switch side scores; legacy parity tests now subset-compare so scoring results may add coverage fields while every legacy field must still match exactly.
Add a zh CER variant that canonicalizes numbers into the written space before scoring, so different written forms of the same speech stop scoring as errors while real errors stay errors: - normalization/canonical_itn: NFKC+lowercase -> mask numeral-bearing idioms / percent / unit words -> cn2an ITN (spoken -> written, many-to-one: 2024 == 二零二四 == 两千零二十四) -> span-wise second pass for leftover CJK numerals -> 百分之X -> X% -> exact mixed-number expansion -> punctuation replaced by spaces with % $ ¥ ° and digit-context . / - protected. The ITN engine is required and its version is recorded in the node trace of every run; per-string fallbacks are counted, never silent. - scoring/token_cer: token-level corpus micro-average CER (CJK per char, latin words, digits per char) with S/D/I decomposition from minimal edit operations (rapidfuzz); inherits the utterance-coverage policy (missing hypotheses scored as deletions, zero covered reference tokens raise instead of reporting 0.0). - Route asr.zh.cer_canonical.canonical_itn.token_cer coexists with the default WeNet-compatible cer and never replaces it. Requires the new [canonical] extra (cn2an + rapidfuzz). - Docs, pipeline catalog entry, and a synthetic judge-pair test suite: equivalent written forms must score 0, real differences must stay errors, plus coverage / zero-token / route-contract cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The default zh CER normalizes in the TN direction (digits -> readings), which must pick one reading per number and therefore mis-scores every other legitimate written form: 百分之五十 vs
50%, 二零二四年 vs2024年, 四千五百亿 vs4500亿all count as errors today. Inverse text normalization (spoken -> written) is a many-to-one mapping, so all reading variants collapse into one canonical form — insensitive to writing conventions, while real errors (magnitude confusions, dropped negatives, decimal misplacement) stay errors.What's added
normalization/canonical_itn— NFKC+lowercase -> mask numeral-bearing idioms / 百分之 / unit words -> cn2an ITN -> span-wise second pass for leftover CJK numerals (万/亿 cut points preferred; a dangling 点 is never swallowed) -> 百分之X -> X% -> exact mixed-number expansion (Decimal) -> punctuation replaced by spaces with% $ ¥ °and digit-context. -protected. The ITN engine is required and its version is recorded in every run's node trace; per-string fallbacks are counted, never silent — availability-dependent fallbacks would change the metric between environments.scoring/token_cer— token-level corpus micro-average CER (CJK per char, latin words, digits per char) with S/D/I from minimal edit operations (rapidfuzz); inherits the utterance-coverage policy from fix(asr): stop silently dropping utterances and remove scoring hot spots #1 (missing hypotheses scored as deletions, zero covered reference tokens raise instead of reporting a perfect 0.0).asr.zh.cer_canonical.canonical_itn.token_cer— opt-in via--metric cer_canonical; the default WeNet-compatibleceris unchanged and never replaced. Requires the new[canonical]extra (cn2an + rapidfuzz); a missing extra fails fast with an install hint.docs/tasks/asr.md), pipeline catalog entry, task manifest/contract entries.Tests
A synthetic judge-pair suite (20 cases): written-form variants must score exactly 0 (year/quantity readings, percent forms, mixed magnitudes, numeral-bearing idioms, punctuation/width folding), real differences must stay errors (magnitude, value, dropped qualifier, decimal placement, plain substitution), plus route-contract, coverage, zero-token rejection, and >100% insertion cases. Tests skip cleanly when the
[canonical]extra is not installed, so base CI is unaffected.Determinism
Identical scores require an identical cn2an version; the engine and rules version are recorded in
report.json's pipeline trace, so any two runs can be compared by fingerprint. Known normalization limitations are documented in the node README and are frozen: changing them means a rules-version bump and a new route id, never an in-place behavior change.