Skip to content

test(core): fix flaky calibration-flash e2e assertion - #54

Merged
jbrecht merged 2 commits into
mainfrom
fix-flash-detection-flake
Jun 15, 2026
Merged

jbrecht merged 2 commits into
mainfrom
fix-flash-detection-flake

Conversation

@jbrecht

@jbrecht jbrecht commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Root cause

The e2e gate intermittently failed at assert.ok(result.videoClockOffsetMs > 0, 'calibration flash was detected') (~1 run in 3). It is not a detection-robustness problem — #46's range tolerance is fine.

The flash (#ff00ff) is painted right after clock.zero(). recordVideo starts at context creation, but VP8 coalesces the identical pre-flash about:blank frames, so the first encoded frame is sometimes the flash itself at pts_time:0. Detection then correctly returns offset 0 — and the assertion's > 0 treated that valid detection as a failure. The offset 0 is also the correct value (the flash genuinely is at clock zero).

I confirmed this against the raw recording from a failed run: chroma at frame 0 = UAVG 201 / VAVG 219 (clean magenta), pts_time 0, so detectFlashOffsetMs returns 0.

detected offset: 0
old assertion (> 0):   FAIL   <- the flake
new assertion (!=null): PASS

Fix (test-only)

  • e2e: re-detect the flash on the raw recording and assert it was found (!== null) and that videoClockOffsetMs equals that detected offset — instead of asserting the offset is positive. A genuine miss (null) still fails the gate, so the original intent (catch real misses) is preserved.
  • Unit: add a parseFlashFromMetadata regression test pinning that a first-frame magenta flash parses to 0, not null.

Production code is unchanged — the pipeline already distinguished null (warn, fall back) from a found offset, and 0 is the right answer when the flash is on frame 0.

Verification

  • 159 core unit tests pass; pnpm build clean.
  • Full e2e integration gate run: exit 0.
  • Proven against the exact recording that previously failed.

Note

This adds an ## Unreleased CHANGELOG entry; #53 does too, so whichever merges second needs a trivial one-line merge to combine the bullets under a single ## Unreleased. Both should land before the next version bump.

🤖 Generated with Claude Code

jbrecht and others added 2 commits June 15, 2026 11:42
The e2e gate asserted `videoClockOffsetMs > 0`, which conflated "flash
detected at offset 0" with "flash not found (fell back to 0)". When
recordVideo's first encoded frame IS the magenta flash — VP8 coalesces
the identical pre-flash blank frames — the flash sits at pts_time 0 and
detection correctly returns offset 0, tripping the `> 0` check ~1 run in
3 despite a perfect detection.

Verified against the exact recording that failed: detectFlashOffsetMs
returns 0 (non-null). Now re-detect on the raw recording and assert the
flash was found (non-null) and that the pipeline consumed that offset;
null still fails, so genuine misses are still caught. Add a unit test
pinning that a first-frame flash parses to 0, not null.

Production behavior was already correct — this is a test-only fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jbrecht
jbrecht merged commit 67afdca into main Jun 15, 2026
1 check passed
jbrecht added a commit that referenced this pull request Jun 15, 2026
Lockstep minor bump. Carries #52 (chapters now respect YouTube's 10s
floor and Vimeo's 50-char title limit; new exports enforceMinChapterDuration
and YOUTUBE_MIN_CHAPTER_MS; default title cap 60→50) and the flaky
calibration-flash e2e fix (#54). Minor rather than patch: two additive
public exports plus the title-cap behavior change.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant