Repository navigation
Steady each frame by its own correction - #176
Merged
Merged
Conversation
A clip shaken on purpose kept two thirds of its shake once stabilized, and more as it went on. The compositor looked the correction up at the project's time, not at the time of the frame it showed; phone footage at 30.06 fps in a 30 fps project drifted a frame off over a few seconds, so the corrections stopped matching the shake. The camera path and the person masks now follow the frame shown, and the analyses (stabilize, cutout, follow face) take each frame's time from the container instead of spreading the duration evenly. Measured on a street clip with added 4 to 7 Hz shake: per-frame jitter 7.2 px before, 2.9 px stabilized before this, 0.17 px now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while testing stabilization on the user's street footage with a known shake added (ffmpeg: 4.3, 7.1, 3.7 and 6.3 Hz sines, up to ±42 px at 1080p) and measuring the export with phase correlation.
Before: Stabilize removed the shake at the start of the clip and less and less after: jitter per second went 0.3, 0.8, 1.3 … 5.0 px over 10 s (shaky input 7.6). In all, 7.2 → 2.9 px per frame horizontally.
Cause: the compositor looked the correction up at the project's time (
mediaTimeAt(clip, timeUs)), but shows the decoder's frame at or before that time. The user's phone footage runs at 30.06 fps (15000/499) in a 30 fps project, so the frame shown drifts up to a frame from the project's time. Its correction was interpolated for a moment it doesn't show, which moved it out of step with its own shake. Variable-rate footage (most phone video) and 24/25 fps clips in a 30 fps project do the same. The person cutout masks (#50) were looked up the same way.Fix:
ClipFramecarries the media time of the frame shown (VideoFrameSource.frameTimeUs). The stabilization correction and the cutout mask are looked up at that time.FrameTimesreads each frame's presentation time from the container (MediaExtractor, no decoding). The stabilize and cutout analyses record real frame times instead ofindex × duration / count, and follow face maps media times to the nearest real frame. With the even spread as a fallback.After: 0.1 to 0.3 px per second all through; 0.17 / 0.11 px per frame (x / y) against 7.2 / 4.6 shaky, 98% removed.
Test plan
StabilizeTimingTest: a 24 fps pattern clip in a 30 fps timeline, corrections alternating frame to frame. Without the compositor change it fails (frame 2 not moved #FF0000), with it it passes. Also checksFrameTimesagainst the file.StabilizeRenderTest,StabilizerTest,CutoutsTest,CutoutRenderTest,FaceTracksTest,HeldFrameRenderTestpass.spotlessCheck,:core:engine:testDebugUnitTest,:core:model:test.Caches of earlier analyses keep their evenly spread times until the media's analysis runs again; their frames are at most a frame off, as before.
🤖 Generated with Claude Code