Skip to content

Accumulate discrete DER frame counts in float64 - #2049

Open
Kayvan-Zahiri wants to merge 1 commit into
pyannote:developfrom
Kayvan-Zahiri:fix/2042-der-float16-saturation
Open

Accumulate discrete DER frame counts in float64#2049
Kayvan-Zahiri wants to merge 1 commit into
pyannote:developfrom
Kayvan-Zahiri:fix/2042-der-float16-saturation

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown

Closes #2042.

reference and hypothesis are cast to float16 before the frame counts are
computed, and np.sum accumulates in the array dtype. float16 tops out at
65504, so once a recording has more active frames than that the total saturates
to inf and the rate comes back as 0.0. At the usual frame resolution that is
roughly 18 minutes of a single speaker, so most real recordings are past it, and
the failure is silent: an obviously wrong hypothesis scores as a perfect one.

Reproduced on develop, two hours with 1000 frames of false alarm:

before   der = 0.0          total = inf
after    der = 0.0022222    total = 450000.0

The four reductions now accumulate in float64. The float16 cast is
untouched, so the comparison and memory behaviour are unchanged, which is what
the issue suggested.

Tests are parametrized across the boundary: 1,000 and 65,000 frames pass either
way, 70,000 and 450,000 fail without the change. That pins the float16 cap
rather than just asserting one large number.

Not addressed here: the second problem in the issue, dividing by total without
checking it, which returns inf or nan on a reference with no speech (also
reported in #1790). Deciding what DER should be when there is nothing to score
is a judgement call rather than a correctness fix, so it seemed better left to
you than bundled in.

black and isort --profile black are clean. tests/test_metrics.py,
test_clustering.py and test_import_lib.py go from 10 to 12 passing.

reference and hypothesis are cast to float16 before the frame counts are
computed, and np.sum accumulates in the array dtype. float16 tops out at
65504, so a recording with more active frames than that sums to inf and the
error rate comes back as 0.0. At the usual frame resolution that is about 18
minutes of a single speaker, so most real recordings are past it, and an
obviously wrong hypothesis scores as a perfect one.

The four reductions now accumulate in float64. The float16 cast is left alone,
so the comparison and memory behaviour do not change.

Closes pyannote#2042

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnFJxiG7r2xeshM4G8rm3y
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant