Offline video export with ffmpeg muxing and deterministic audio - #10
Merged
Conversation
Every one-shot SFX body moves into a RECIPES table scheduling onto an
AudioSink { ctx, bus, t } — live playback targets the shared context's
master bus at currentTime through a single emit() choke point, so the
audible behavior is unchanged. The win/pickup fanfares now schedule on
the audio clock instead of setTimeout, and the noise buffer is cached
per-context.
New: startSfxCapture/setSfxCaptureTime/stopSfxCapture record (name,
arg, t) entries instead of sounding, and renderExportAudio() replays a
tape (plus the battle-music bed) into a 48 kHz OfflineAudioContext at
fixed reference levels — the groundwork for video exports with sound.
audioBufferToWav (replay/wav.ts) turns the render into ffmpeg input.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX
@ffmpeg/ffmpeg 0.12.15 + the single-threaded @ffmpeg/core 0.12.10 (no SharedArrayBuffer, so no COOP/COEP requirements). replay/ffmpeg.ts owns one lazily-loaded shared instance: the library arrives via dynamic import, the core js/wasm are bundled same-origin assets (?url), the class worker goes through Vite's worker pipeline (?worker&url), and logs land in a ring buffer for actionable failure messages. Terminating (export cancel) rejects in-flight execs and drops the cache. replay/bitstream.ts adds the elementary-stream glue: AVCC→Annex-B conversion for encoders that can't emit Annex-B directly, and IVF framing (timebase 1/fps) for the VP9 fallback. Nothing imports these yet — the pipeline swap lands next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX
…h removed exportVideo now runs one offline pipeline: WebCodecs encodes each tick-stepped frame into a raw elementary stream (Annex-B H.264 when the encoder supports it, AVCC converted via bitstream.ts otherwise, IVF for the VP9 fallback), the SFX fired along the way land on the audio tape at exact frame times, and ffmpeg muxes the stream at a declared framerate together with the OfflineAudioContext-rendered AAC soundtrack into a constant-frame-rate MP4 with +faststart. Container timing no longer depends on encode pacing or per-frame timestamp rounding — the VFR output of the old mp4-muxer path (never told a frame rate) is what made later parts of long clips stutter in normal players. The realtime WebM path is gone with everything that existed for it (captureAudioStream, MediaRecorder pump, format picker); it was wall-clock paced and tab-visibility dependent, and only existed because it was the sole path with sound. Export now requires WebCodecs and the dialog says so. New QUALITY (standard/high bitrate) and SOUND (on/off) chips replace the FORMAT group. Countdown beeps now fire during exports (gate widened to include the paused exporting transport, reset made deterministic in beginExport). Cancel mid-mux terminates the ffmpeg worker (in-flight exec rejects and the next export reloads the core); cancel after completion is a no-op so closing the dialog keeps the warm instance. The e2e spec now proves the fixes byte-level: single stts run (true CFR) and mp4a present or absent per the sound toggle, across two consecutive exports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX
The progress bar now says which stage is running (rendering frames / rendering audio / encoding mp4), and opening the export dialog warms the ffmpeg core download while the user trims — the first render no longer pays the fetch at mux time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX
CI's Chromium exports through the H.264 path (the local container only
offers VP9), and there the muxed MP4 came out variable-frame-rate
(stts=49 over 60 frames) even though the offline frame stepping is
uniform — the platform H.264 encoder embeds VUI/timing derived from the
per-frame VideoFrame timestamps, and ffmpeg's H.264 parser preferred
that over the -framerate demux hint.
Close every door the jitter can enter:
- Encoder timestamps are now exact integer multiples of a rounded
usPerFrame, so anything the encoder derives from them is uniform too.
- The raw-H.264 input gets '-r {fps}' (discards parsed/VUI timestamps
and regenerates constant 1/fps) alongside the existing -framerate.
- Both containers set -video_track_timescale {fps*1000}, so frame
durations are whole numbers and the stts table is a single run by
construction. (Confirmed locally on the VP9 path: stts=1.)
Also adds an onCodec diagnostic threaded to lastExport.codec and the
export spec's logs, so the encoder path each environment took is visible
at a glance the next time browsers diverge.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactored video export to use a unified offline pipeline with ffmpeg-wasm for muxing, replacing the dual mp4/webm approach. Audio is now captured deterministically via a "recipe" system that records SFX calls during export and replays them through an OfflineAudioContext, enabling frame-perfect synchronized sound in exported videos.
Key Changes
Audio recipe system: Converted all SFX functions to recipes—pure functions that schedule audio nodes onto an
AudioSink(context, bus, time). Live playback points the sink at the shared context's master bus; export mode records(name, arg, t)entries to a tape and replays them offline into an OfflineAudioContext for deterministic, synchronized audio.Removed real-time WebM export: Eliminated
captureAudioStream()and the real-time MediaRecorder pipeline. All exports now use WebCodecs for video encoding into elementary streams (H.264 Annex-B or VP9 IVF format).ffmpeg-wasm integration: Added
ffmpeg.tswith lazy-loaded, shared FFmpeg instance that muxes video and audio into constant-frame-rate MP4. Includes log ring buffer for error diagnostics and instance lifecycle management (warm reuse, cancel-safe termination).Bitstream utilities: New
bitstream.tsmodule handles AVCC→Annex-B conversion for H.264 (when encoder can't emit Annex-B directly) and IVF framing for VP9, allowing flexible WebCodecs output routing to ffmpeg.WAV writer: Added
wav.tsto convert OfflineAudioContext renders to RIFF/WAVE format for ffmpeg input.Export UI refinement: Replaced "format" (mp4/webm) selector with "quality" (standard/high) and "sound" (on/off) toggles. Updated messaging to reflect unified offline pipeline and background-tab capability.
Noise buffer lifecycle: Changed from a single global noise buffer to a
WeakMap<BaseAudioContext, AudioBuffer>, allowing offline renders to have their own GC-able buffers.Deterministic timing: Fixed procedural sequences (e.g.,
winfanfare,pickuptwo-tone) to schedule on the audio clock instead ofsetTimeout, making them replay-safe in offline renders.Test updates: E2E test now validates MP4 container structure (ftyp magic, single stts entry for true CFR) and audio track presence/absence per sound toggle.
Implementation Details
AudioSinkinterface ({ ctx, bus, t }) abstracts the destination for audio scheduling, enabling both live and offline rendering with identical recipe code.tapevariable; when non-null,emit()records instead of playing, ensuring no audio leaks during export setup.runRecipe()dispatcher andRECIPESmap centralize all SFX logic, eliminating duplication between live and export paths.https://claude.ai/code/session_01ERaEpotVycgrLwqfcau7NX