Skip to content

fix: detect the language on the offset window without VAD - #78

Merged
rubas-agent[bot] merged 3 commits into
mainfrom
fix-offset-window-language-detection
Oct 16, 2026
Merged

rubas-agent[bot] merged 3 commits into
mainfrom
fix-offset-window-language-detection

Conversation

@rubas-agent

@rubas-agent rubas-agent Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Before

Without VAD, build_params hands the window to whisper.cpp:

if req.vad_model_path.is_none() {
    if let Some(o) = req.offset_ms {
        params.set_offset_ms(u32_to_i32(o));
    }
    if let Some(d) = req.duration_ms {
        params.set_duration_ms(u32_to_i32(d));
    }
}

The two new integration tests on the origin/main Rust (source-built NIF):

1) test offset_ms detects the language on the window, not on the start of the buffer
   language: "nn"            # expected "en"
2) test offset and duration window the timeline without VAD
   List.last(segs).end = 10.399999618530273   # limit 8.5

Problem

whisper.cpp v1.9.4 detects the language at offset 0 (whisper_full_with_state, whisper_lang_auto_detect_with_state(ctx, state, 0, ...)). It reads offset_ms only after detection. So transcribe/3 with offset_ms: 30_000 decoded the window in the language of the first 30 s of the buffer. A :duration_ms window also returned segments past its end. The VAD path and transcribe_slice/4 already detect on the window, so the three ways to ask for one window gave different results.

After

transcribe_one cuts the window from the samples in Rust on both paths. whisper.cpp sees only the window.

Check origin/main Branch
Language of a 30 s noise prefix plus jfk, offset_ms: 30_000 "nn" "en"
Last segment end, offset_ms: 2_000, duration_ms: 6_000 10.4 s at most 8.5 s
Integration suite, source-built NIF 2 failures 86 passed
Cause, design, tests

Cause. native/whisper_cpp_native/src/transcribe.rs:build_params passed set_offset_ms/set_duration_ms to whisper.cpp when VAD was off. whisper.cpp windows the decode with them, but not the language detection.

Design. The VAD path already cut the window from the samples before it ran. Now the non-VAD path does the same: samples.truncate(win_end); samples.drain(..win_start). One code path owns the window, so detection and decode cannot disagree.

  • window moved from vad.rs to transcribe.rs, because VAD no longer owns it. The set_offset_ms/set_duration_ms block is gone.
  • Without VAD, segment and word times shift by win_start / 16000, so they stay absolute on the whole buffer. The VAD ranges already carry the window origin.
  • An empty window returns TranscriptionResult::empty before whisper runs. The mel reflect pad reads out of bounds on 0 samples.
  • duration_s stays the duration of the whole buffer, as before.

Tests.

  • test/integration_test.exs: "offset_ms detects the language on the window, not on the start of the buffer". On the multilingual tiny model, 30 s of seeded white noise before jfk. transcribe/3 with offset_ms: 30_000 must return "en" and the same text as transcribe_slice/4 on the same window. Digital silence does not show the bug, because tiny detects "en" on silence.
  • test/integration_test.exs: "offset and duration window the timeline without VAD". offset_ms: 2_000, duration_ms: 6_000 gives first.start >= 2.0 and last.end <= 8.5.
  • Cargo test window_selects_offset_and_duration_in_samples.
  • Both integration tests fail on origin/main and pass on the branch. task check passes: fmt, compile with warnings as errors, credo, clippy -D warnings, 195 mix tests, cargo tests, zizmor.

Docs: the :offset_ms/:duration_ms entry in lib/whisper_cpp.ex, and an [Unreleased] Fixed entry in CHANGELOG.md.

rubas added 3 commits October 16, 2026 16:00
whisper.cpp v1.9.4 auto-detects the language at offset 0 before it reads
offset_ms, so transcribe/3 with :offset_ms and no VAD decoded the window
in the language of the first 30 s of the buffer. Cut the window from the
samples in Rust on both paths, stop passing offset_ms/duration_ms to
whisper.cpp, and shift segment and word times back by the window start.
@rubas-agent
rubas-agent Bot merged commit 51ccf58 into main Oct 16, 2026
1 check passed
@rubas-agent
rubas-agent Bot deleted the fix-offset-window-language-detection branch October 16, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant