Harden A/V sync CI, cursor timing, and Windows hardware encoding

[richiemcilroy]

Follow-up to #1977, addressing the nightly sync-tests failure plus two field-reported defects.

1. Nightly sync-matrix flake (run 28697093459, Windows)

The randomized case random/2/video-14fps-.../41ts failed the absolute-pts check with 0.285s of error. Root cause is a harness expectation bug, not a pipeline bug: the case's drop lottery dropped the first 3 nominal frames, so the first frame the pipeline ever saw was sent at t=0.216s. The pipeline (by design) zeroes each track's timeline at its first delivered frame — cross-track alignment comes from the per-track start_time persisted by the recorder — so every muxed pts carried a structural 0.216s offset against the test's absolute origin, plus ~70ms of encoder-queue jitter on the shared runner. The relative-structure check (the sync-bearing one) passed.

Fix: the absolute check now measures against the origin-normalized sent timeline (first sent frame ⇔ first muxed pts), and the absolute tolerance is floored at the relative tolerance since it contains the same per-frame jitter. Verified locally with the failing seed (CAP_SYNC_MATRIX_SEED=1783145408979603400): all 28 cases pass; random/2's error drops from 285ms structural to pure scheduler jitter.

2. Cursor visibly lagging the video

The default cursor-smoothing spring (tension 470, friction 70) chases the raw position at the same timestamp, and a spring tracking a moving target trails it by friction/tension at steady state — a systematic 149ms cursor delay behind the video, identical in preview and export. (The old 500ms click look-ahead used to mask this near clicks; its reduction to 140ms in 529a71580 exposed it.)

Fix: the spring's target now samples the raw path friction/tension ahead per active profile (149ms default, 40ms drag, ~75ms click-window), one-pole-smoothed across profile switches so the target never pops. Each regime cancels its own lag, so the smoothed cursor lands where the real cursor was at render time while keeping the smoothing aesthetic. The drawn cursor icon still samples un-led time. Locked by smoothed_cursor_tracks_moving_target_without_lag, which fails at ~149ms of trail without the compensation (verified by reverting).

Clock domains, pause excision, and preview/export parity were audited and are correct; remaining known offsets are sub-frame (≤21ms cross-track start snap vs raw cursor epochs, ~8–16ms poll quantization) and are documented for follow-up rather than touched here.

3. Windows studio recordings showing a solid green display

A solid green display with intact editor background is the render of zeroed YUV (Y=U=V=0 through the BT.601 shader). The editor decodes with software FFmpeg on Windows (deterministic on every machine), which localizes the reported machine-dependent green to encode time: the default Windows studio path picks a vendor hardware encoder (h264_nvenc/h264_amf/h264_qsv/h264_mf) with no output validation and no runtime fallback — unlike the non-fragmented MediaFoundation path, which validates and falls back to software. Some driver stacks open a session successfully and then emit zeroed frames with no error surfaced.

Fix: on Windows, every hardware H264 candidate must now pass a pre-flight round trip — encode a short neutral-gray clip, decode it with the software decoder, verify the gray survives — before it can be selected, in both the muxed and standalone builder paths. Failures log a warning and fall through the existing priority list, terminating at libx264, so NVIDIA/AMD/Intel/CPU-only machines all end up on a proven-working encoder. Results are cached per (encoder, resolution, mode); CAP_DISABLE_ENCODER_SELF_TEST=1 is the escape hatch. macOS is exempt (single-vendor VideoToolbox, and a measured ~2.5s session+encode cost would tax recording start for a failure mode never observed there); Linux records via libx264 already.

The verifier rejects both zeroed (green) and black round-trip output; covered by unit tests plus a full round trip through a real encoder.

4. Capture gaps landing on a segment cut were collapsed (multi-second desync)

Found by running the randomized matrix at nightly scale locally (40 cases): two case shapes failed deterministically with 2.4–3.1s relative pts errors. Box-level analysis of the produced fragments showed the mechanism: the DASH/mov fragmenter anchors each fragment's tfdt at the accumulated duration of the previous fragment, and the last sample of a fragment takes its packet duration verbatim — which we always synthesized as the nominal frame period. When a capture gap (static screen) crosses a segment cut, the first post-gap frame is pulled back onto the pre-gap timeline (one frame period after the last pre-gap frame) and its content plays during the gap; everything muxed after it is placed correctly, leaving that frame seconds out of sync and the viewer seeing post-gap content during the hold. Curated gap cases never caught this because their gap frame is not a keyframe, so the muxer never cuts there.

Fix: EncoderBase now holds one packet and stamps it with the real dts delta to its successor before writing (only when the duration was synthesized), so the fragmenter's accumulated timeline includes the gap and the pre-gap frame correctly holds through it. Locked by gap_crossing_segment_cut_preserves_post_gap_pts, which fails on the old code with the exact production signature (post-gap frame at 0.396s instead of 3.0s). This covers the macOS/Windows/Linux studio fragmented paths and instant mode, which all sit on SegmentedVideoEncoder.

A second hardening in the same class: the anomaly tracker confirmed real >2s gaps only via arrival spacing at the mux loop, which encoder backpressure can defeat (pre-gap backlog and post-gap frame drain back-to-back). A frame can never be stamped ahead of the wall clock, so forward jumps landing at-or-behind it are now accepted as real gaps regardless of arrival bunching; future-stamped jumps still collapse as glitches. Locked by bunched_real_gap_behind_wall_clock_is_accepted / future_stamped_jump_is_still_collapsed.

With both fixes, the full 62-case matrix passes on the failing seed (CAP_SYNC_MATRIX_SEED=1783149530744270000), the previously failing cases dropping from 2.4–3.1s errors to ~65ms scheduler jitter.

Greptile Summary

This PR hardens recording sync, cursor timing, and Windows H.264 encoder selection. The main changes are:

• Adds Windows hardware encoder round-trip self-tests before selection.
• Caches self-test results by the encoder session inputs that shape output.
• Fails closed when decoded self-test frames use unexpected pixel formats.
• Holds one video packet so real durations survive segment-cut gaps.
• Updates timestamp gap handling for bunched post-gap frames.
• Normalizes sync-matrix absolute PTS checks to the first delivered frame.
• Adds cursor phase-lead compensation for spring smoothing.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
crates/enc-ffmpeg/src/video/h264.rs	Adds Windows hardware H.264 self-tests and tightens cache keys plus decoded-format validation.
crates/enc-ffmpeg/src/base.rs	Adds one-packet holdback so synthesized durations can be replaced with real DTS deltas.
crates/enc-ffmpeg/src/mux/segmented_stream.rs	Retries init-segment notification and adds coverage for gaps crossing segment cuts.
crates/recording/src/output_pipeline/core.rs	Uses wall-clock elapsed time to distinguish real capture gaps from future-stamped jumps.
crates/recording/tests/sync_matrix.rs	Compares absolute PTS values against the origin-normalized sent timeline.
crates/rendering/src/cursor_interpolation.rs	Adds spring lead compensation so smoothed cursor motion tracks the raw path timing.

_{Reviews (3): Last reviewed commit: "Include pixel format in the self-test ca..." | Re-trigger Greptile}

Context used:

• Context used - CLAUDE.md (source)
• Context used - AGENTS.md (source)

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[tembo]

I think the absolute check is still implicitly assuming pts[0] == 0. Since read_video_pts() returns raw container PTS, would it be safer to origin-normalize the muxed side too (to match the doc above + the relative check)?

Suggested change

	max_abs = max_abs.max((p - (s - sent_origin)).abs());
	max_abs = max_abs.max(((p - pts[0]) - (s - sent_origin)).abs());

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[richiemcilroy]

hey @greptileai, please re-review the PR

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[richiemcilroy]

hey @greptileai, please re-review the PR

[tembo]

Minor cache-hit improvement: Dictionary::iter() order can vary depending on how options are constructed, so the same logical option set could generate multiple keys and re-run the self-test. Sorting by option key before appending makes the cache deterministic.

Suggested change

	}
	let mut opts: Vec<_> = encoder_options.iter().collect();
	opts.sort_by(|(a, _), (b, _)| a.cmp(b));
	for (k, v) in opts {
	let _ = write!(key, "|{k}={v}");
	}