Skip to content

fix: stop DER frame counts from saturating float16 - #2043

Open
Atishyy27 wants to merge 1 commit into
pyannote:developfrom
Atishyy27:fix/der-frame-count-overflow
Open

fix: stop DER frame counts from saturating float16#2043
Atishyy27 wants to merge 1 commit into
pyannote:developfrom
Atishyy27:fix/der-frame-count-overflow

Conversation

@Atishyy27

@Atishyy27 Atishyy27 commented Jul 30, 2026

Copy link
Copy Markdown

Closes #2042.

discrete_diarization_error_rate casts reference and hypothesis to np.half before counting frames, and np.sum accumulates in the array dtype. float16 tops out at 65504, so on a recording with more active frames than that the total saturates to inf and the error rate is reported as 0.0, roughly 18 minutes of single-speaker speech at the usual frame resolution, so most real recordings are affected, and a wrong output silently scores as a perfect one.

This accumulates the frame counts in float64. The np.half cast is left in place, so the comparison and memory behaviour are unchanged.

The same function divided by the total without checking it, so a reference with no speech gave inf or nan (reported in #1790, closed unfixed). The three divisions now go through a small helper that follows the convention already used by pyannote.metrics.detection: an empty reference scores 0 when nothing was detected either, and 1 otherwise.

Testing

Two tests in tests/test_metrics.py, both failing on develop:

  • test_discrete_der_long_recording_does_not_overflow: 450000 frames (about two hours) with 1000 false-alarm frames. Returns 0.0 with total = inf before, 0.00222 with total = 450000.0 after.
  • test_discrete_der_empty_reference: an all-zero reference returns nan / inf before, 0.0 / 1.0 after.

Also checked that a short recording is unaffected: 1000 frames with 100 false-alarm frames gives 0.1 before and after.

Reference and hypothesis are cast to float16 before the frame counts are
computed, and np.sum accumulates in the array dtype. float16 tops out at
65504, so on a recording with more active frames than that the total
saturates to infinity and the error rate is reported as 0.0. At the usual
frame resolution that threshold is reached after about 18 minutes of
single-speaker speech, so most real recordings are affected and a wrong
output silently scores as a perfect one.

Accumulate the frame counts in float64. The float16 cast is left alone,
so the comparison and memory behaviour are unchanged.

The same function also divided by the total without checking it, so a
reference containing no speech produced inf or nan. Route the three
divisions through a helper that follows the convention already used by
pyannote.metrics.detection: an empty reference scores 0 when nothing was
detected either, and 1 otherwise.
@Atishyy27
Atishyy27 force-pushed the fix/der-frame-count-overflow branch from 1b1f0c7 to 5e402a8 Compare July 30, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant