Repository navigation
Take the run's language for clips Whisper is unsure of - #189
Merged
Merged
Conversation
With Auto-detect, each clip's language came from its first chunk, and on faint background voices that is close to a guess: a street clip in a Polish project was captioned "¡Vamos a ver, vamos!". Every clip's language is now detected first with its probability, and a clip under 0.5 takes the language Whisper is surest of in the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #188
With Spoken language: Auto-detect, each clip's language came from its own first chunk. On a clip with only faint voices in the background, that is close to a guess, and the clip got captions in that language: "¡Vamos a ver, vamos!" in a Polish project.
Now, when Auto-detect runs over more than one clip:
Whisper.detectLanguage(new JNI call on whisper.cpp'swhisper_pcm_to_melandwhisper_lang_auto_detect), on 30 s of the clip's sound after any opening quiet, within its first minute;LanguageChoice.pick(ml:whisper) keeps the language of every clip Whisper is sure of (0.5 or more), and gives the others the language Whisper is surest of over the run;Settling the languages takes the first 5% of the progress. It costs one decode of up to a minute and one encoder pass per clip.
Checked on the emulator with the project of #188 (Whisper Base, Main clips, six clips). The
Languages heardlog line, with clip ids replaced by what the clips are:Every clip was transcribed in Polish. The café clip got "Omy naszą cześć, ja mam." (rough, the voices are faint) instead of "¡Vamos a ver, vamos!" twice. The run took 31.5 s for 38.7 s of sound, as before. Undo brought the project back.
Tests:
LanguageChoiceTest(a guess takes the language heard clearly; two languages spoken clearly stay apart; with nothing sure, the likeliest language goes for all).🤖 Generated with Claude Code