Accumulate discrete DER frame counts in float64 - #2049
Open
Kayvan-Zahiri wants to merge 1 commit into
Open
Conversation
reference and hypothesis are cast to float16 before the frame counts are computed, and np.sum accumulates in the array dtype. float16 tops out at 65504, so a recording with more active frames than that sums to inf and the error rate comes back as 0.0. At the usual frame resolution that is about 18 minutes of a single speaker, so most real recordings are past it, and an obviously wrong hypothesis scores as a perfect one. The four reductions now accumulate in float64. The float16 cast is left alone, so the comparison and memory behaviour do not change. Closes pyannote#2042 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnFJxiG7r2xeshM4G8rm3y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2042.
referenceandhypothesisare cast tofloat16before the frame counts arecomputed, and
np.sumaccumulates in the array dtype.float16tops out at65504, so once a recording has more active frames than that the total saturates
to
infand the rate comes back as0.0. At the usual frame resolution that isroughly 18 minutes of a single speaker, so most real recordings are past it, and
the failure is silent: an obviously wrong hypothesis scores as a perfect one.
Reproduced on
develop, two hours with 1000 frames of false alarm:The four reductions now accumulate in
float64. Thefloat16cast isuntouched, so the comparison and memory behaviour are unchanged, which is what
the issue suggested.
Tests are parametrized across the boundary: 1,000 and 65,000 frames pass either
way, 70,000 and 450,000 fail without the change. That pins the float16 cap
rather than just asserting one large number.
Not addressed here: the second problem in the issue, dividing by
totalwithoutchecking it, which returns
infornanon a reference with no speech (alsoreported in #1790). Deciding what DER should be when there is nothing to score
is a judgement call rather than a correctness fix, so it seemed better left to
you than bundled in.
blackandisort --profile blackare clean.tests/test_metrics.py,test_clustering.pyandtest_import_lib.pygo from 10 to 12 passing.