Repository navigation
fix: detect the language on the offset window without VAD - #78
Merged
Merged
Conversation
whisper.cpp v1.9.4 auto-detects the language at offset 0 before it reads offset_ms, so transcribe/3 with :offset_ms and no VAD decoded the window in the language of the first 30 s of the buffer. Cut the window from the samples in Rust on both paths, stop passing offset_ms/duration_ms to whisper.cpp, and shift segment and word times back by the window start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before
Without VAD,
build_paramshands the window to whisper.cpp:The two new integration tests on the
origin/mainRust (source-built NIF):Problem
whisper.cpp v1.9.4 detects the language at offset 0 (
whisper_full_with_state,whisper_lang_auto_detect_with_state(ctx, state, 0, ...)). It readsoffset_msonly after detection. Sotranscribe/3withoffset_ms: 30_000decoded the window in the language of the first 30 s of the buffer. A:duration_mswindow also returned segments past its end. The VAD path andtranscribe_slice/4already detect on the window, so the three ways to ask for one window gave different results.After
transcribe_onecuts the window from the samples in Rust on both paths. whisper.cpp sees only the window.origin/mainjfk,offset_ms: 30_000"nn""en"offset_ms: 2_000, duration_ms: 6_000Cause, design, tests
Cause.
native/whisper_cpp_native/src/transcribe.rs:build_paramspassedset_offset_ms/set_duration_msto whisper.cpp when VAD was off. whisper.cpp windows the decode with them, but not the language detection.Design. The VAD path already cut the window from the samples before it ran. Now the non-VAD path does the same:
samples.truncate(win_end); samples.drain(..win_start). One code path owns the window, so detection and decode cannot disagree.windowmoved fromvad.rstotranscribe.rs, because VAD no longer owns it. Theset_offset_ms/set_duration_msblock is gone.win_start / 16000, so they stay absolute on the whole buffer. The VAD ranges already carry the window origin.TranscriptionResult::emptybefore whisper runs. The mel reflect pad reads out of bounds on 0 samples.duration_sstays the duration of the whole buffer, as before.Tests.
test/integration_test.exs: "offset_ms detects the language on the window, not on the start of the buffer". On the multilingual tiny model, 30 s of seeded white noise beforejfk.transcribe/3withoffset_ms: 30_000must return"en"and the same text astranscribe_slice/4on the same window. Digital silence does not show the bug, because tiny detects"en"on silence.test/integration_test.exs: "offset and duration window the timeline without VAD".offset_ms: 2_000, duration_ms: 6_000givesfirst.start >= 2.0andlast.end <= 8.5.window_selects_offset_and_duration_in_samples.origin/mainand pass on the branch.task checkpasses: fmt, compile with warnings as errors, credo, clippy-D warnings, 195 mix tests, cargo tests, zizmor.Docs: the
:offset_ms/:duration_msentry inlib/whisper_cpp.ex, and an[Unreleased]Fixed entry inCHANGELOG.md.