Diarization fails in the bundled service: torchcodec native libs can't load (missing FFmpeg shared libraries)

[julien731]

Problem

In the bundled (PyInstaller) service, speaker diarization fails — transcription completes but every segment is labelled UNKNOWN. The dev (non-frozen) service diarizes correctly; this is specific to the frozen build.

Surfaced while running the native macOS app (#94) end-to-end.

Root cause

torchcodec (transformers'/pyannote's audio backend) ships native libraries libtorchcodec_core{4,5,6,7}.dylib that dynamically link against FFmpeg shared libraries (libavcodec, libavformat, libavutil, libswresample, …). The bundle vendors only the static ffmpeg/ffprobe binaries (scripts/vendor_ffmpeg.sh), not the FFmpeg shared libs, so torchcodec can't dlopen any of its core libs:

pyannote/audio/core/io.py:48: UserWarning:
torchcodec is not installed correctly so built-in audio decoding will fail.
[start of libtorchcodec loading traceback]
FFmpeg version 7: Failed to load dynlib '.../torchcodec/libtorchcodec_core7.dylib'.
FFmpeg version 6: Failed to load dynlib '.../torchcodec/libtorchcodec_core6.dylib'.
...

Transcription still works because WhisperX decodes audio via the ffmpeg subprocess, but pyannote's diarization pipeline relies on torchcodec for audio I/O and degrades.

Options (pick one)

1. Vendor FFmpeg shared libraries matching a torchcodec-supported version (4–7) into the bundle and make them resolvable by torchcodec (rpath/@loader_path), in addition to the static binaries. Most faithful; heaviest.
2. Route pyannote audio through the in-memory fallback — load the waveform ourselves (e.g. via the ffmpeg subprocess / soundfile) and pass pyannote a {'waveform': tensor, 'sample_rate': int} dict, avoiding torchcodec entirely (pyannote documents this fallback in the warning).
3. Swap torchcodec for torchaudio's decoder where possible / pin the audio backend so torchcodec isn't used.

Option 2 is likely the smallest, most self-contained fix.

Scope

Service-side / bundled-service packaging (local-transcription-service.md, #95) — the native macOS app is unaffected structurally; it just surfaces the degraded result. Verify with a diarization-enabled transcription (HF token set) in the frozen build after the fix.

[julien731]

Investigation: stated root cause disproven — torchcodec is not why speakers are UNKNOWN

Dug into this before implementing. The torchcodec/FFmpeg-shared-lib theory does not explain the symptom.

Evidence

1. The proposed fix (Option 2) is already the current behavior. _diarize_and_assign (backend/services/transcriber.py:99) hands whisperx a numpy waveform, and whisperx wraps it as {'waveform', 'sample_rate'} before calling pyannote (whisperx/diarize.py:133-136) — identical in the venv and the frozen bundle.
2. torchcodec is only on pyannote's file-path branch, never the waveform branch. grep torchcodec across pyannote hits only core/io.py; Audio.__call__/crop/get_duration all short-circuit on "waveform" in file and never touch torchcodec.
3. Direct reproduction. Forcing TORCHCODEC_AVAILABLE=False (simulating the frozen build's exact warning) and running the real pipeline on the same audio the frozen build labelled UNKNOWN → full segmentation + embedding + clustering ran with no exception and produced a real SPEAKER_00.
4. Full dev run. A ~23-min meeting transcribed end-to-end via the web build diarized into 3 distinct speakers (SPEAKER_00/01/02), no torchcodec error in the log.

Conclusion: the torchcodec is not installed correctly warning is real but harmless given in-memory waveform input. The UNKNOWN-everywhere result is specific to the frozen PyInstaller bundle and has a different, currently-uncaptured cause — some diarization failure swallowed by the broad except Exception at backend/services/transcriber.py:100-105, which silently degrades to no speaker labels.

Why we can't name the real cause yet

The frozen build's diarization traceback isn't persisted anywhere: ServiceSupervisor (macos/.../Service/ServiceSupervisor.swift:143) reads service stderr only to parse the startup handshake, and nothing logs to disk. So the actual exception is lost.

Recommendation

Don't implement this issue as written — Option 2 changes nothing, and Options 1/3 target a non-cause. Instead:

1. Add diagnostics first: narrow the swallowed except and persist the diarization failure reason (job/metadata + a log file). Small, correct, and it captures the real frozen traceback.
2. Run the PyInstaller bundle once with a diarization-enabled transcription to capture that traceback.
3. Re-scope the fix from the real error.

Will re-title / re-scope this issue once the frozen traceback is in hand.