Skip to content

Add optional TwelveLabs video segmentation backend (Marengo + Pegasus) - #13

Open
mohit-twelvelabs wants to merge 1 commit into
roothch:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

Add optional TwelveLabs video segmentation backend (Marengo + Pegasus)#13
mohit-twelvelabs wants to merge 1 commit into
roothch:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

What this adds

An opt-in segmentation backend that analyzes the video frames and audio directly via TwelveLabs, as an alternative to the default Whisper-transcript + LLM pipeline:

  • Marengo (search) — semantic retrieval to locate clips relevant to the user's prompt.
  • Pegasus (analyze) — generates the per-segment summary and tags.

It returns the exact same {start, end, summary, tags} structure the rest of the app already consumes, so the analysis table, re-analyze tab, clipping, and export flows all work unchanged.

Why it helps PreenCut

The current pipeline keys off speech (Whisper → transcript → LLM). For speech-light footage — sports, b-roll, surveillance, music videos, silent demos — there's little transcript to segment on. Segmenting straight from the visual + audio content covers that gap.

Opt-in / non-breaking

Disabled by default. The default SEGMENTER_TYPE=llm path is untouched. To enable:

export SEGMENTER_TYPE=twelvelabs
export TWELVELABS_API_KEY=your_key
# optional: TWELVELABS_INDEX_NAME, TWELVELABS_MAX_CLIPS

New config lives alongside the existing LLM_MODEL_OPTIONS in config.py; the processor (modules/twelvelabs_processor.py) mirrors LLMProcessor.segment_video's contract and is swapped in processing_queue.py only when opted in. twelvelabs>=1.2.8 added to requirements.txt.

How it was tested

  • No-network unit tests for the JSON/segment parsing and the overlap filter (tests/test_twelvelabs_processor.py), runnable as plain Python or via pytest.
  • A live smoke test (gated on TWELVELABS_API_KEY, skipped when unset) confirming a Marengo text embedding returns a vector.
  • Verified live against the SDK that Pegasus analyze returns well-formed {start, end, summary, tags} segments and Marengo search returns relevant time windows that the overlap filter applies correctly. The slow upload/index step uses the documented SDK signatures (tasks.create / wait_for_done).

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Add an opt-in segmentation backend that analyzes video frames and audio
directly via TwelveLabs (Marengo retrieval + Pegasus analysis), instead of
relying on Whisper transcripts. Useful for speech-light content. Returns the
same {start, end, summary, tags} structure so the existing clip/export flow is
unchanged. Enabled via SEGMENTER_TYPE=twelvelabs; default behavior unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant