Repository navigation
Start transcription where the speech starts - #177
Merged
Merged
Conversation
Whisper times the first words of a recording that opens with quiet as if they began at once. On a talk clip whose in-point sits 0.8 s before the speech, "Witam" was highlighted 0.64 s before it was said, and the words after it drifted early too, past the 0.4 s reach of WordTiming.refine. Each chunk now goes to Whisper from 0.1 s before the first 0.1 s of sound over the quiet (WordTiming.speechStart), and the words are moved back by that offset. On the same clip the first word now starts within 0.05 s of the voice and the later words land after their pauses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while making a demo video with the karaoke captions on the 4K talking-head test clip (Polish, Whisper Small).
Problem
The clip's in-point was 0.8 s before the speech. Whisper timed the first words as if the speech began at once: "Witam" was highlighted 0.64 s before it was said, and the words after it were early too.
WordTiming.refineonly snaps edges within 0.4 s, so it could not fix it.Change
WordTiming.speechStartfinds where the speech starts. It looks for the first 0.1 s of sound over the quiet, so a click in the quiet doesn't count, and returns 0.1 s before that. It returns 0 when the sound opens with speech or is too noisy to trust.Transcribergives Whisper each chunk from that point and moves the words back by the same offset.Result on the same clip (source seconds)
Tests
WordTimingTest.speechStartsAfterTheOpeningQuietAndItsClicks.:ml:whisper:testDebugUnitTestpasses andspotlessCheckis clean.🤖 Generated with Claude Code