Map reliable model testing and reviewer evidence in ProofShot #8

Destination

Produce an approved, implementation-ready workflow for model-led UI verification grounded in real agent-browser exploration, distilled into concise human testing guidance and fresh reviewer evidence, recoverable under interruption and concurrency, and assigned to the correct owner without making ProofShot the behavioral-testing authority.

Decisions and outcomes

• Audit remaining ProofShot gaps after the reliability and workflow updates identified two managed-runtime defects, missing replay enforcement, and the live-reliability evidence frontier.
• Measure the exploration-to-review workflow on representative UI changes found exploration dominates elapsed time while stabilized ProofShot replay adds about eight seconds; representative reviewer videos were 4.3s and 5.6s.
• Verify interrupted and concurrent ProofShot sessions on the deployed release found and resolved an undrained-FFmpeg-progress finalization deadlock. A fresh managed soak produced two overlapping PASS recordings at 77s and 80s, with B remaining usable after A stopped.
• Choose ownership for browser exploration, proof replay, and E2E coverage keeps behavioral exploration with the owning model and agent-browser, scenario/E2E content with the target repository, generic replay/evidence with ProofShot, and routing guidance with agent configuration.
• Define the reviewer evidence and human testing handoff requires high-level human instructions, one concise replay per distinct use case, explicit assertions, curated screenshots, canonical verdicts, action disclosure, and private raw evidence.
• Prototype a deterministic fresh replay contract for stabilized flows proved a versioned stable-selector case and became ProofShot PR #17 — feat(replay): add stable-selector evidence replay contracts.
• Restore safe modes for managed ProofShot launchers and config was resolved by explicit projection modes and the direct-native browser deployment.
• Make ProofShot start compatible with the managed agent-browser policy was resolved by validating and invoking the exact pinned native agent-browser binary.

Shipped implementation

• ProofShot PR #17 — feat(replay): add stable-selector evidence replay contracts.
• ProofShot PR #18 — fix(browser): prevent concurrent recording finalization deadlocks.
• Agent configuration PRs fix(browser): prevent concurrent recording finalization deadlocks #18, docs(readme): fix repo URLs, add --force flag, update viewer description AmElmo/proofshot#19, and docs: add footer credit to AmElmo AmElmo/proofshot#21–pr command fails with github where gh is auth'd with OAuth AmElmo/proofshot#24 for secure projection, E2E-profile guidance, native browser deployment, replay pinning, and final recording deployment.

Not yet specified

None.

Out of scope

• no-mistakes pipeline, model-routing, and PR-body work; those have separate owners and plans.
• Upstream contribution strategy; this map targets the deployed alvarohulse/proofshot fork.
• Implementing target-application E2E tests; target repositories own those tests and their flake proof.
• Unrelated dependency-vulnerability triage.