Skip to content

Take the run's language for clips Whisper is unsure of - #189

Merged
bpacholek merged 1 commit into
mainfrom
188-run-language
Nov 1, 2026
Merged

bpacholek merged 1 commit into
mainfrom
188-run-language

Conversation

@bpacholek

Copy link
Copy Markdown
Contributor

Closes #188

With Spoken language: Auto-detect, each clip's language came from its own first chunk. On a clip with only faint voices in the background, that is close to a guess, and the clip got captions in that language: "¡Vamos a ver, vamos!" in a Polish project.

Now, when Auto-detect runs over more than one clip:

  • every clip's language is detected first, with how sure Whisper is: Whisper.detectLanguage (new JNI call on whisper.cpp's whisper_pcm_to_mel and whisper_lang_auto_detect), on 30 s of the clip's sound after any opening quiet, within its first minute;
  • LanguageChoice.pick (ml:whisper) keeps the language of every clip Whisper is sure of (0.5 or more), and gives the others the language Whisper is surest of over the run;
  • each clip is then transcribed in its language. A run of one clip works as before.

Settling the languages takes the first 5% of the progress. It costs one decode of up to a minute and one encoder pass per clip.

Checked on the emulator with the project of #188 (Whisper Base, Main clips, six clips). The Languages heard log line, with clip ids replaced by what the clips are:

talk pl 0.997, talk pl 0.996, harbour pl 0.76, street ko 0.23, street ko 0.35, café es 0.11

Every clip was transcribed in Polish. The café clip got "Omy naszą cześć, ja mam." (rough, the voices are faint) instead of "¡Vamos a ver, vamos!" twice. The run took 31.5 s for 38.7 s of sound, as before. Undo brought the project back.

Tests: LanguageChoiceTest (a guess takes the language heard clearly; two languages spoken clearly stay apart; with nothing sure, the likeliest language goes for all).

🤖 Generated with Claude Code

With Auto-detect, each clip's language came from its first chunk, and on
faint background voices that is close to a guess: a street clip in a
Polish project was captioned "¡Vamos a ver, vamos!". Every clip's
language is now detected first with its probability, and a clip under
0.5 takes the language Whisper is surest of in the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bpacholek
bpacholek merged commit 4226675 into main Nov 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-detect picks a random language for clips with faint speech

1 participant