fix(asr): stop silently dropping utterances and remove scoring hot spots - #1
Open
liutaocode wants to merge 1 commit into
Open
fix(asr): stop silently dropping utterances and remove scoring hot spots#1liutaocode wants to merge 1 commit into
liutaocode wants to merge 1 commit into
Conversation
Correctness (scores change only where results were previously wrong): - aispeech_norm: keep rows whose recognition text is empty so they are scored as deletions; previously "key\t" rows were dropped and the matching reference rows were skipped downstream, understating error rates (up to ~50% on sets with many empty outputs). Count malformed rows in the node trace (row_stats) and fail when a file yields no parseable <key>\t<text> rows instead of scoring a perfect 0.0. - wenet scoring wrapper: score reference utterances missing from the hypothesis as empty hypotheses (pure deletions) instead of silently skipping them; report utterance coverage (num_ref_utts, num_hyp_utts, num_hyp_missing_utts, num_hyp_extra_utts, key samples) in the scoring result; raise instead of reporting 0.0 when zero reference tokens are covered. Code-switch zh/en side scores stay lenient since monolingual subsets legitimately cover zero tokens. Performance (bit-identical scores, regression-tested): - wenet_compute_cer: reset only the DP submatrix an utterance uses; resetting every previously grown row made all utterances after one long utterance pay O(longest^2) (17x on a synthetic corpus, ~10x end-to-end on a real 1.4k-utterance code-switch run). - aispeech_norm: preload num2words map tables once per node call and share the number cache instead of re-globbing and re-reading map files per line/token (~40x per-call overhead). Also fix a latent NameError in asr_num2words' fallback map loading (load_and_sort_map was referenced but never defined; the path only survived because the default map dir has no root-level .map files). Tests: nine regression cases covering empty/missing hypotheses, coverage reporting, zero-token rejection, order-independent totals, and monolingual code-switch side scores; legacy parity tests now subset-compare so scoring results may add coverage fields while every legacy field must still match exactly.
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Three silent-failure paths in the ASR scoring pipeline understate error rates or report perfect scores on broken inputs:
aispeech_normkeeps only rows that split into two tab-separated fields, so an empty recognition result (key\t) is discarded, andcompute_werthen skips the matching reference row (fid not in rec_set). The worst utterances — where the model produced nothing — are excluded from CER/WER/MER entirely. On a benchmark set where ~half the hypotheses are legitimately empty, corpus CER is understated by up to ~50%.all == 0maps to a perfect 0.0 error rate with no warning.Changes
normalization/aispeech_norm: keep empty-text rows (scored as deletions downstream), reportrow_stats(rows / empty / dropped-malformed) in the node trace, and raise when a file yields no parseable<key>\t<text>rows.scoring/wenet_*wrapper: score reference utterances missing from the hypothesis as empty hypotheses (pure deletions), expose utterance coverage (num_ref_utts,num_hyp_utts,num_hyp_missing_utts,num_hyp_extra_utts, key samples) in the scoring result, and refuse to report 0.0 when zero reference tokens were covered. Code-switch zh/en side scores stay lenient (monolingual subsets legitimately cover zero tokens).compute_wer(one long utterance previously made every following utterance payO(longest²); 17x on a synthetic corpus, ~10x end-to-end on a real 1.4k-utterance code-switch run), and preload normalization map tables once per node call instead of re-globbing/re-reading per line or token (~40x per-call overhead).NameErrorinasr_num2words' fallback map loading (load_and_sort_mapreferenced but never defined).Tests
Nine new regression cases (
tests/test_asr_scoring_coverage.py) covering empty/missing hypotheses, coverage reporting, zero-token rejection, order-independent totals, and monolingual code-switch side scores. Legacy parity tests now subset-compare: scoring results may add coverage fields, but every legacy field must still match exactly.