feat(replay): finding replay / regression (/replay) — re-verify past findings

[shivsin25]

What & why

docs/CAPABILITY-GAPS.md (agent runtime) lists: "No finding-replay/regression to re-verify past bugs on a target." This closes it — and it's the natural second half of the PoC bundler (#144): once /poc has captured a finding, /replay re-sends that exact request and tells you whether the bug is still live. Retests pay again and programs increasingly reward regression reports; this turns a one-shot PoC into a monitorable asset.

Two modes

• run <bundle-dir> [--marker "signal"] — replay one bundle.
• all [root] — sweep every poc-bundle/1 under root (default findings/) into a ranked regression-report.md + regression.json. This is "retest day" in one command.

Verdicts

🔴 STILL_VULNERABLE	the bug's signal is still present
🟢 FIXED	the signal is gone
🟡 CHANGED	response differs from the PoC — review
⚫ UNREACHABLE	endpoint no longer answers
⚪ INDETERMINATE	no marker + no baseline, or a missing secret
⏭ SKIPPED_UNSAFE	mutating method — needs --confirm-unsafe

How it decides (and where it's honest about limits)

• Marker (confidence: confirmed) — a string that proves the bug (leaked email, reflected payload, error text). Present → still vulnerable; absent → fixed. Markers persist as marker.txt so later sweeps reuse them automatically.
• Heuristic (confidence: heuristic) — with no marker, it compares status + response SHA-256. Byte-identical → still vulnerable (unchanged); different → CHANGED. It never claims FIXED from a hash alone (dynamic content also changes) — deliberately conservative.

Reuses existing infra (no new surface)

• Re-sends through the repo's SSRF-guarded safe_http.safe_urlopen (via poc_bundler.capture).
• Parses the bundle with poc_bundler.parse_raw_request, hashes with the same sha256_hex.
• Trusts the manifest URL for scheme/host, so an http:// finding isn't replayed over https (avoids a false UNREACHABLE — caught and fixed during testing, with a regression test).

Safety

• PUT/DELETE/PATCH skipped without --confirm-unsafe — replaying a mutating request could re-trigger the action.
• Secrets are read from env vars only (like the bundle's repro.sh) and never written into replay.json or the sweep report (asserted by a test).
• --fail-if-vulnerable exits 3 when anything is still vulnerable — for scheduled monitoring / CI.
• Console output reconfigures to UTF-8 so verdict icons don't crash a non-UTF-8 terminal.

Testing

• 25 tests in tests/test_replay.py — every verdict branch, header/secret resolution, bundle discovery + I/O, offline replay (via monkeypatched capture), and both CLI modes.
• Pure core (decide_verdict, resolve_headers, render_report) tested without a socket; verified live end-to-end against the repo's demo/app.py (STILL_VULNERABLE while up, UNREACHABLE while down).
• Additive only: 4 files changed, 758 insertions(+).

Try it

# after /poc has captured a finding:
tools/replay.py run findings/target.com-idor/evidence/F-12 --marker "victim@corp.com"
tools/replay.py all          # sweep everything, write a regression report

🤖 Generated with Claude Code

[shuvonsec]

Solid, well-scoped - pure core (decide_verdict/resolve_headers/render_report) is separated and genuinely tested, secrets are env-only with a leak test, unsafe methods gated. Verified 25 tests pass. Two asks (both about re-issuing live traffic):

1. (scope) replay_one and especially replay all re-attack every stored target with no scope check, which contradicts Critical Rule #1 (READ FULL SCOPE before touching any asset). The repo already ships scope_checker.py - please gate the send through it, or at minimum warn/confirm per target, especially for all.
2. (SSRF) The initial hop is unguarded: safe_urlopen only validates redirect hops, and load_bundle trusts bundle.json's exchange.url verbatim. A tampered/untrusted bundle makes replay all hit 169.254.169.254 / localhost on the first request. Validate the initial host before calling capture.
3. (minor) Guard/redirect URLErrors are caught as verdict UNREACHABLE, which reads like "endpoint gone / fixed" - a bug that now 302s to an internal host gets masked. Worth distinguishing guard/redirect errors from real unreachability.

[shivsin25]

Great review, @shuvonsec — all three are about re-issuing live traffic responsibly, and you're right on each. Fixed in 73fd55d:

1. Scope (Critical Rule #1). replay now gates every send through scope_checker.py. Pass --scope '*.target.com' (repeatable) or --scope-file <file>; out-of-scope targets get an OUT_OF_SCOPE verdict and are never sent. replay all now refuses to run with no scope unless you pass --no-scope to explicitly assert every stored target is in scope; run warns when no scope is supplied. (Live: all with no scope → exit 2; a bundle for api.acme.com under --scope other-company.com → 🚫 OUT_OF_SCOPE, no request issued.)

2. SSRF on the initial hop. You nailed it — load_bundle trusts bundle.json's exchange.url and safe_urlopen only guards redirects. Added initial_host_blocked(), which validates the initial host with the same blocklist safe_http uses on redirects (metadata / private / loopback / link-local) before any request. A tampered bundle pointing at 169.254.169.254 now returns 🛑 BLOCKED and nothing is sent. (Verified live with a bundle whose URL is http://169.254.169.254/latest/meta-data.)

3. Guard/redirect errors masked as UNREACHABLE. Fixed: a new BLOCKED verdict is returned for SSRF-guard / too-many-redirects URLErrors (distinct from UNREACHABLE), so a bug that now 302s to an internal host reads as a live signal to investigate, not "endpoint gone / fixed." Real connection errors still map to UNREACHABLE.

+9 tests (34 total pass), covering scope in/out + all refusal, the initial-host guard on literals (169.254 / localhost / 127.0.0.1 blocked, public IP allowed), a tampered-bundle replay that returns BLOCKED without sending, and redirect-to-internal → BLOCKED vs a real connection error → UNREACHABLE. Ready for another look