Skip to content

fix(asr): stop silently dropping utterances and remove scoring hot spots - #1

Open
liutaocode wants to merge 1 commit into
PigeonDan1:mainfrom
liutaocode:fix/asr-empty-hyp-and-coverage
Open

fix(asr): stop silently dropping utterances and remove scoring hot spots#1
liutaocode wants to merge 1 commit into
PigeonDan1:mainfrom
liutaocode:fix/asr-empty-hyp-and-coverage

Conversation

@liutaocode

Copy link
Copy Markdown
Contributor

Problem

Three silent-failure paths in the ASR scoring pipeline understate error rates or report perfect scores on broken inputs:

  1. Empty hypothesis text is dropped. aispeech_norm keeps only rows that split into two tab-separated fields, so an empty recognition result (key\t) is discarded, and compute_wer then skips the matching reference row (fid not in rec_set). The worst utterances — where the model produced nothing — are excluded from CER/WER/MER entirely. On a benchmark set where ~half the hypotheses are legitimately empty, corpus CER is understated by up to ~50%.
  2. Missing hypothesis utterances are skipped silently, with no utterance counts anywhere in the report, so a partial or key-mismatched hypothesis file passes as a clean run.
  3. Malformed input scores 0.0. A space-separated (Kaldi-style) file parses to zero rows, and all == 0 maps to a perfect 0.0 error rate with no warning.

Changes

  • normalization/aispeech_norm: keep empty-text rows (scored as deletions downstream), report row_stats (rows / empty / dropped-malformed) in the node trace, and raise when a file yields no parseable <key>\t<text> rows.
  • scoring/wenet_* wrapper: score reference utterances missing from the hypothesis as empty hypotheses (pure deletions), expose utterance coverage (num_ref_utts, num_hyp_utts, num_hyp_missing_utts, num_hyp_extra_utts, key samples) in the scoring result, and refuse to report 0.0 when zero reference tokens were covered. Code-switch zh/en side scores stay lenient (monolingual subsets legitimately cover zero tokens).
  • Performance, bit-identical scores: reset only the DP submatrix each utterance uses in the vendored compute_wer (one long utterance previously made every following utterance pay O(longest²); 17x on a synthetic corpus, ~10x end-to-end on a real 1.4k-utterance code-switch run), and preload normalization map tables once per node call instead of re-globbing/re-reading per line or token (~40x per-call overhead).
  • Fix a latent NameError in asr_num2words' fallback map loading (load_and_sort_map referenced but never defined).

Tests

Nine new regression cases (tests/test_asr_scoring_coverage.py) covering empty/missing hypotheses, coverage reporting, zero-token rejection, order-independent totals, and monolingual code-switch side scores. Legacy parity tests now subset-compare: scoring results may add coverage fields, but every legacy field must still match exactly.

Correctness (scores change only where results were previously wrong):
- aispeech_norm: keep rows whose recognition text is empty so they are
  scored as deletions; previously "key\t" rows were dropped and the
  matching reference rows were skipped downstream, understating error
  rates (up to ~50% on sets with many empty outputs). Count malformed
  rows in the node trace (row_stats) and fail when a file yields no
  parseable <key>\t<text> rows instead of scoring a perfect 0.0.
- wenet scoring wrapper: score reference utterances missing from the
  hypothesis as empty hypotheses (pure deletions) instead of silently
  skipping them; report utterance coverage (num_ref_utts, num_hyp_utts,
  num_hyp_missing_utts, num_hyp_extra_utts, key samples) in the scoring
  result; raise instead of reporting 0.0 when zero reference tokens are
  covered. Code-switch zh/en side scores stay lenient since monolingual
  subsets legitimately cover zero tokens.

Performance (bit-identical scores, regression-tested):
- wenet_compute_cer: reset only the DP submatrix an utterance uses;
  resetting every previously grown row made all utterances after one
  long utterance pay O(longest^2) (17x on a synthetic corpus, ~10x
  end-to-end on a real 1.4k-utterance code-switch run).
- aispeech_norm: preload num2words map tables once per node call and
  share the number cache instead of re-globbing and re-reading map
  files per line/token (~40x per-call overhead).

Also fix a latent NameError in asr_num2words' fallback map loading
(load_and_sort_map was referenced but never defined; the path only
survived because the default map dir has no root-level .map files).

Tests: nine regression cases covering empty/missing hypotheses,
coverage reporting, zero-token rejection, order-independent totals,
and monolingual code-switch side scores; legacy parity tests now
subset-compare so scoring results may add coverage fields while every
legacy field must still match exactly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant