feat(asr): complete the canonical family with mer_canonical and wer_canonical - #3
Open
liutaocode wants to merge 3 commits into
Open
feat(asr): complete the canonical family with mer_canonical and wer_canonical#3liutaocode wants to merge 3 commits into
liutaocode wants to merge 3 commits into
Conversation
Correctness (scores change only where results were previously wrong): - aispeech_norm: keep rows whose recognition text is empty so they are scored as deletions; previously "key\t" rows were dropped and the matching reference rows were skipped downstream, understating error rates (up to ~50% on sets with many empty outputs). Count malformed rows in the node trace (row_stats) and fail when a file yields no parseable <key>\t<text> rows instead of scoring a perfect 0.0. - wenet scoring wrapper: score reference utterances missing from the hypothesis as empty hypotheses (pure deletions) instead of silently skipping them; report utterance coverage (num_ref_utts, num_hyp_utts, num_hyp_missing_utts, num_hyp_extra_utts, key samples) in the scoring result; raise instead of reporting 0.0 when zero reference tokens are covered. Code-switch zh/en side scores stay lenient since monolingual subsets legitimately cover zero tokens. Performance (bit-identical scores, regression-tested): - wenet_compute_cer: reset only the DP submatrix an utterance uses; resetting every previously grown row made all utterances after one long utterance pay O(longest^2) (17x on a synthetic corpus, ~10x end-to-end on a real 1.4k-utterance code-switch run). - aispeech_norm: preload num2words map tables once per node call and share the number cache instead of re-globbing and re-reading map files per line/token (~40x per-call overhead). Also fix a latent NameError in asr_num2words' fallback map loading (load_and_sort_map was referenced but never defined; the path only survived because the default map dir has no root-level .map files). Tests: nine regression cases covering empty/missing hypotheses, coverage reporting, zero-token rejection, order-independent totals, and monolingual code-switch side scores; legacy parity tests now subset-compare so scoring results may add coverage fields while every legacy field must still match exactly.
Add a zh CER variant that canonicalizes numbers into the written space before scoring, so different written forms of the same speech stop scoring as errors while real errors stay errors: - normalization/canonical_itn: NFKC+lowercase -> mask numeral-bearing idioms / percent / unit words -> cn2an ITN (spoken -> written, many-to-one: 2024 == 二零二四 == 两千零二十四) -> span-wise second pass for leftover CJK numerals -> 百分之X -> X% -> exact mixed-number expansion -> punctuation replaced by spaces with % $ ¥ ° and digit-context . / - protected. The ITN engine is required and its version is recorded in the node trace of every run; per-string fallbacks are counted, never silent. - scoring/token_cer: token-level corpus micro-average CER (CJK per char, latin words, digits per char) with S/D/I decomposition from minimal edit operations (rapidfuzz); inherits the utterance-coverage policy (missing hypotheses scored as deletions, zero covered reference tokens raise instead of reporting 0.0). - Route asr.zh.cer_canonical.canonical_itn.token_cer coexists with the default WeNet-compatible cer and never replaces it. Requires the new [canonical] extra (cn2an + rapidfuzz). - Docs, pipeline catalog entry, and a synthetic judge-pair test suite: equivalent written forms must score 0, real differences must stay errors, plus coverage / zero-token / route-contract cases.
…anonical Add mixed and English token error rates that share one normalization chain, one tokenizer, and one scorer with cer_canonical, so the family degenerates consistently by construction: text without latin letters scores identically under mer_canonical and cer_canonical, and text without CJK scores identically under mer_canonical and wer_canonical. Both identities are locked by property tests. - normalization/canonical_itn now accepts zh/en/cs. For en/cs it whisper-normalizes latin spans first (vendored Whisper English normalizer: contraction expansion, spoken numbers to digits including percent/currency, spoken-filler removal, British->American spelling), with two robustness passes measured on real system output styles: unambiguous bare contractions are restored before expansion (dont == don't == do not, so apostrophe-dropping systems are not falsely penalized), and 's is collapsed instead of expanded (it's == its, john's == johns; 's is possessive/is/has-ambiguous, and expanding it mis-scores every possessive). The one forgone equivalence (it's vs it is) is documented in tests as the intentional trade-off forced by contraction non-transitivity. Spans without latin letters pass through unchanged, preserving pure-Chinese degeneration. - scoring/token_mer applies the exact same token-level scorer as scoring/token_cer via a shared implementation; the family difference lives entirely in normalization. - The shared scorer gains a deterministic word-spacing repair: a latin word token whose letters exactly equal the concatenation of 2-4 consecutive word tokens on the other side is split (tenthe == ten the), so pure spacing artifacts -- pervasive in some system outputs -- never score as errors, while any letter difference stays fully scored. Repair counts are reported as spacing_repairs. This is per-utterance and reference-anchored: no corpus statistics, no frozen artifacts, no batch dependence. - Routes asr.cs.mer_canonical.canonical_itn.token_mer and asr.en.wer_canonical.canonical_itn.token_mer; legacy cer/wer/mer routes unchanged. Docs, catalog entries, and a 24-case synthetic suite (degeneration properties, contraction/spelling/filler/number equivalences, spacing repair both directions, CJK-leakage and letter-difference errors, contract rejections).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's added
Mixed and English token error rates that share one normalization chain, one tokenizer, and one scorer with
cer_canonical, so the family degenerates consistently by construction: text without latin letters scores identically undermer_canonicalandcer_canonical(token for token), and text without CJK scores identically undermer_canonicalandwer_canonical. Both identities are locked by property tests.normalization/canonical_itnnow accepts zh/en/cs. For en/cs it whisper-normalizes latin spans first, reusing the vendored Whisper English normalizer (contraction expansion, spoken numbers → digits including percent/currency, spoken-filler removal, British→American spelling), plus two robustness passes motivated by real ASR output styles:dont→don't→do not), so systems that drop apostrophes are not systematically penalized against apostrophed references;'scollapse instead of expansion (it's≡its,john's≡johns):'sis three-ways ambiguous (possessive / is / has), and expanding it mis-scores every possessive. The one forgone equivalence (it'svsit is) is documented in tests as the trade-off forced by contraction non-transitivity — no deterministic mapping can hold all three equivalences at once.scoring/token_merapplies the exact same token-level scorer asscoring/token_certhrough a shared implementation — the family difference lives entirely in normalization.tenthe≡ten the, and the reversesome thing≡something), so pure spacing artifacts never score as errors while any letter difference stays fully scored. Repairs are counted and reported asspacing_repairs, keeping spacing quality visible as a separate signal. Reference-anchored and per-utterance: no corpus statistics, no frozen artifacts, no dependence on what else is being evaluated.asr.cs.mer_canonical.canonical_itn.token_merandasr.en.wer_canonical.canonical_itn.token_mer; legacycer/wer/merroutes are untouched. Docs, catalog entries.Tests
24 synthetic cases: degeneration properties (pure-Chinese ≡ cer_canonical token-for-token; pure-English ≡ wer_canonical), contraction/spelling/filler/number equivalences, spacing repair in both directions plus a letter-difference case where repair must NOT fire, CJK-leakage errors, and contract rejections. Tests skip cleanly without the
[canonical]extra, so base CI is unaffected.