Skip to content

Edit a full multicam episode: sync every camera and mic, cut to the speaker - #226

Merged
nmbrthirteen merged 1 commit into
mainfrom
multicam-edit
Oct 1, 2026
Merged

nmbrthirteen merged 1 commit into
mainfrom
multicam-edit

Conversation

@nmbrthirteen

Copy link
Copy Markdown
Owner

podcli could only start from one finished video. A podcast recorded on several cameras and recorders had to be synced and cut in another tool first. This adds the full-episode edit: point podcli at one episode's folder and it maps, syncs, cuts, and delivers.

What changes

  • Mapping. Each file's role is guessed from its name: a person's name, "wide", or Zoom Tr1/Tr2 numbering. A single stereo recorder is split one person per side. Guesses are flagged until someone confirms them.
  • Sync. Onset envelopes are cross-correlated over the full length of both files, then refined with GCC-PHAT at several checkpoints and fitted to a line. The line gives the offset plus the clock drift between devices, which the render and both timelines correct.
    • A file with no sound in common with the reference is retried against every file that did sync, so split camera parts line up without a continuous mic.
    • Audio that starts late inside a video file is anchored to container time.
    • Offsets set by hand survive a re-sync unless it is forced.
  • Cut. Each person's mic decides who is talking. Levels are normalized to that mic's own speech level, so bleed doesn't pick the camera. Crosstalk goes wide, long monologues get a wide cutaway, and short blips fold into their neighbours. A shared-mic show falls back to speaker detection.
  • Deliver. An MP4 (libx264, loudness-normalized mic mix), one WAV per person, and a Premiere XML (xmeml v4) or FCPXML 1.10 timeline that points at the original files.
    • Stereo sides stay on separate tracks.
    • Camera timecode sets the asset start.
    • Phone frame rates such as 29.92 snap to broadcast rates.
  • Three surfaces, one edit.
    • The studio has a new Multicam edit page.
    • The CLI adds podcli multicam <folder>, with --set FILE=ROLE:NAME, --export, --no-render and --resync.
    • Agents get a new manage_multicam MCP tool, bringing the total to 27.
    • All three share one saved edit per set of files. Re-running reuses the sync and the cut, and an unchanged render is skipped.
  • Safety. A running job holds its edit, and a second job or an edit made meanwhile gets a 409 instead of being lost. /api/multicam/file serves only files inside a multicam output folder. Folder scans stop at 40 media files or 20,000 entries.
  • Shared fix. fcpxml.file_uri now uses Path.as_uri(), so Windows paths import in Resolve. This also fixes the existing DaVinci export.

Verification

  • Python: 814 passed. The one failure, test_ai_fallback::test_find_cli_falls_back_to_shell_lookup, also fails on main.
  • JS: 332 passed. Typecheck, build, and the docs drift check pass. Go build, vet and test pass.
  • 34 new multicam tests. They generate real media with ffmpeg and cover:
    • sync accuracy within 5 ms
    • drift
    • audio starting 0.4 s late
    • split camera parts and mic bleed
    • crosstalk and wide inserts
    • planning a 3-hour episode in under 0.2 s
    • render timing, render skip, and manual offsets across a re-sync
    • stereo channels and timecode in both exports
    • CLI resume, delete, and errors
  • Live runs on a generated 3-camera, 2-mic episode:
    • the studio page, clicked through in a headless browser
    • the CLI, including the interactive review in a pseudo-terminal
    • manage_multicam, driven through the real MCP server with the MCP client SDK

Not verified: opening the exported timelines in Premiere, Resolve or Final Cut, and rendering multi-hour 4K footage.

…peaker

podcli could only start from one finished video. A podcast recorded on three
cameras and two recorders had to be synced and cut somewhere else first.

Point podcli at the folder of one episode's recordings. It guesses who each
file belongs to from its name, syncs everything by audio, and cuts cameras to
whoever is talking. Crosstalk and long monologues go to the wide shot. The
result renders as an MP4 with one WAV per person, or exports as a Premiere XML
or FCPXML timeline that points at the original files.

Sync cross-correlates onset envelopes over the full length of both files,
then refines with GCC-PHAT at several checkpoints and fits a line through
them. The line gives the offset and the clock drift between devices, which
the render and both timelines correct. A file that shares no sound with the
reference gets a second try against every file that did sync, so split camera
parts line up without a continuous mic. Audio that starts late inside a video
file is anchored to container time, and offsets set by hand survive a re-sync.

Who is talking comes from each person's mic, normalized to that mic's own
speech level so bleed and a hot preamp don't decide the cut. Shows with one
shared mic fall back to speaker detection.

The same edit runs from the studio (Multicam edit), the CLI
(podcli multicam <folder>) and the manage_multicam MCP tool. All three share
one saved edit per set of files, so re-running reuses the sync and the cut,
and an unchanged render is skipped. A job holds its edit until it finishes,
so a second job or an edit made meanwhile is refused instead of lost.
@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 47 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: bbc92b8c-6926-4e28-9b0f-a69366399695

📥 Commits

Reviewing files that changed from the base of the PR and between 7978b9e and 4ceece0.

📒 Files selected for processing (27)
  • CLAUDE.md
  • README.md
  • backend/cli.py
  • backend/main.py
  • backend/services/integrations/_shared/fcpxml.py
  • backend/services/integrations/_shared/multicam_xml.py
  • backend/services/multicam.py
  • backend/services/multicam_signal.py
  • cli/main.go
  • src/models/index.ts
  • src/server.ts
  • src/ui/client/Layout.tsx
  • src/ui/client/icons.tsx
  • src/ui/client/main.tsx
  • src/ui/client/multicam-cut.tsx
  • src/ui/client/multicam-deliver.tsx
  • src/ui/client/multicam-look.tsx
  • src/ui/client/multicam-page.tsx
  • src/ui/client/multicam-person-chip.tsx
  • src/ui/client/multicam-sources.tsx
  • src/ui/client/multicam-start.tsx
  • src/ui/client/multicam-sync.tsx
  • src/ui/client/multicam-types.ts
  • src/ui/public/css/styles.css
  • src/ui/web-server.ts
  • tests/test_multicam.py
  • tests/test_multicam_signal.py
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nmbrthirteen
nmbrthirteen merged commit 7d6a67d into main Oct 1, 2026
14 checks passed
@nmbrthirteen nmbrthirteen mentioned this pull request Oct 1, 2026
nmbrthirteen added a commit that referenced this pull request Oct 1, 2026
Version bump only. Ships multicam editing from #226.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant