Skip to content

Replay entry point, cache hit through replay, profile-aware path tests (#790 #788 #787) - #821

Merged
casoon merged 1 commit into
mainfrom
claude/790-replay-entry-cache-hit
Oct 31, 2026
Merged

casoon merged 1 commit into
mainfrom
claude/790-replay-entry-cache-hit

Conversation

@casoon

@casoon casoon commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Continues #816 (replayable snapshot), #818 (support matrix) and #787 (measurement profile).

What changed

  • Cache hit = replay. --reuse-cache answers through artifacts::cached_report → replay_report_as(snapshot, "cache"). The lossy to_audit_report fallback is gone from the CLI path. Every non-hit is a CacheMiss with a printed reason: unreadable, incomplete, version-mismatched snapshot, or different audit configuration (signature, which already contains session policy, state_id and storage-state digest). load_artifacts keeps its API.
  • auditmysite replay <snapshot.json | cache dir | URL>. A subcommand rather than a flag on the single-URL run: it takes no URL to fetch, starts no browser and does no reachability check, and its inputs and refusals differ from an audit. It yields a verdict exit code like an audit (source: replay).
  • Profile flags are assertions on replay. --session (only when given explicitly), --state-id, --storage-state must match the stored measurement profile, else ReplayError::OtherMeasurement (field, stored, asserted). Flags go before the subcommand (auditmysite --state-id x replay URL), as for every other top-level flag.
  • Support matrix. Mode Replay is now a real mode: all four extra-check flags are Rejected for it (a replay shows what the stored run recorded; it cannot measure), validated in Args::validate.
  • Tests. tests/replay_paths_test.rs (Chrome, local fixture, throw-away $HOME, added to the CI browser job): fresh vs cache hit vs replay give one model except timestamp, duration_ms, environment.source, with the server stopped before cache/replay. SPA fixture (content changes after pushState without reload, driven by the declared storage state): same profile twice → same content hash; other state → other content, and audit::profile_differences names state_id and storage_state; the plain run does not get the in-state cache entry; replay asserting another profile is refused. Unit tests for the profile check and the matrix; unchecked_normalize.txt baseline lowered (cache path now uses normalize_checked).

Checks

cargo test, cargo clippy --all-features --all-targets -- -D warnings, cargo check --all-features, cargo test --test replay_paths_test -- --ignored (local Chrome) green; manual CLI run of hit/replay/refusals. (One cargo test run showed 7 failures in one suite while I was editing in parallel; two reruns were clean, I did not reproduce it.)

Open

  • to_audit_report is unused by the CLI now; kept because it is public API.
  • No CLI-level (binary) test of replay; covered by library tests and a manual run.
  • Replay of a batch (several snapshots) does not exist; single URL only.
  • Studio (../auditmysite_studio, not touched): needs session, storage_state (Option<StorageState> from StorageState::load) and state_id in its PipelineConfig literals, a UI/setting for them, and should include them in any cache key it keeps; it can use audit::cached_report/CacheMiss to show why a stored result was not reused.

@casoon
casoon merged commit 337a90e into main Oct 31, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant