fix(security): bind scan coverage to execution receipts

[proffesor-for-testing]

Problem

SAST can report requested files as scanned even when a file was unreadable, and it cannot distinguish a clean engine execution from an unavailable or failed engine.

Inspected protected main 0a6de058575cbe292afe220df105cc74120f7674 (v3.14.2), September 18, 2026. These source files were verified against GitHub tree blob identities.

• SASTScanner.analyzeFile catches read failures and returns empty vulnerabilities with zero lines. Unsupported extensions return lines read without analysis.
• scanWithRules sets coverage.filesScanned to files.length, rather than completed file analyses.
• runSemgrepScanning collapses unavailable, failed, and successful zero-finding executions into []; rulesApplied adds finding count, which is not rule-execution count.
• SecurityCoverage has only aggregate files/lines/rules counts.
• Security handler projects requested file counts and derives deepAnalysisPerformed from JS/TS presence, rather than execution receipts.
• SecurityAuditProtocol.auditSecrets returns an empty result and estimates filesScanned as scanPaths.length * 10 without actual scanning. Its externally reachable use needs validation; this issue does not assert every public secret-scan route uses it.

Executed evidence

Executed the exact extracted analyzeFile method with Node 22.17.0 --experimental-strip-types and a nonexistent .ts path. Observed {"vulnerabilities":[],"linesScanned":0}, with no error disposition.

This is a function-level probe. The public requested-count mismatch and engine-status conflation are STATIC source evidence; no full service, live Semgrep, or package integration test was run.

Research motivation

OverclaimBench motivates distinguishing observed inspection from claimed review completeness. Its five scenarios and controlled-access artifacts limit generalization and replication; content exposure does not prove comprehension.

EviRCA motivates deterministic evidence extraction with explicit missing evidence. Neither paper establishes that the proposed AQE design works; validate it locally.

Prioritized implementation

P0 — preserve execution truth. Add a typed per-file/per-engine receipt with source digest, requested scope, language, engine/rule-set version, executed rule IDs where measurable, disposition (analyzed, unsupported, excluded, unreadable, failed, unavailable, not-run), and sanitized reason. A successful clean scan is analyzed with zero findings. Missing analysis must retain its own disposition.

Freeze a scope manifest before execution. Record discovery failures separately so a manifest of discovered files cannot imply discovery was complete. Track byte/line-read counts separately from analyzed counts. Engine scope may differ: Semgrep currently scans a common parent directory, so retain its actual target and returned coverage or explicitly unknown coverage.

P1 — aggregate from receipts. Compute requested/analyzed/unsupported/excluded/failed counts from unique manifest identities. Preserve partial evidence through domain, handler, MCP and CLI paths. Replace the placeholder protocol secret audit with the real scanner or an explicit unavailable/not-run result. Preserve usable findings on partial runs; completeness and vulnerability presence are separate dimensions.

Do not count findings as rules applied. Report unknown rule execution when the engine supplies no reliable execution metadata. Do not imply pattern scanning provides full SAST assurance.

P2 — truthful conclusions and consumer integration. Permit “no findings in completed checks; N requested files unassessed,” with resumable gaps. Bind cached/reused receipts to source and engine revisions. Supply the producer evidence to #651 rather than duplicate quality-gate architecture.

Acceptance criteria

1. Every requested file has one terminal disposition per intended engine; duplicate paths/aliases have a declared counting policy.
2. Read/discovery failures remain visible without leaking file contents or credentials.
3. Unsupported but readable files do not increment analyzed counts.
4. Missing/failed Semgrep differs from successful zero-finding Semgrep.
5. filesScanned reflects completed analysis under documented semantics.
6. rulesApplied never derives from number of findings.
7. Domain, handler, MCP and CLI retain partial/unavailable dispositions and scope.
8. Placeholder secret scanning cannot return a measured clean result.
9. Reuse on a changed source digest or rule-set version is rejected or explicitly stale.
10. Existing successful-scan findings remain compatible; report-schema migration is documented.
11. Integration tests exercise actual producer and consumer paths, not just receipt shape.
12. Discovery incompleteness and unknown external-engine coverage cannot silently become complete.

Validation experiments

• Missing file, injected EACCES, removed-after-discovery file, unsupported extension, intentional exclusion.
• Clean JS/TS fixture and independently seeded vulnerable fixture; verify findings and completed counts separately.
• Semgrep absent, nonzero exit, malformed output, successful empty output, and successful findings.
• Semgrep common-parent scan containing an extra file outside the requested list.
• Partial batch: one analyzed file and one read failure; inspect real public MCP/CLI outputs.
• Changed file/rules after cached scan; test invalidation.
• Repeated/aliased paths and source mutation during scan.
• Discovery failure before manifest completion.
• Reachability audit and real integration test for protocol auditSecrets.

Duplicate review and scope

Searched all 688 issue/PR records (298 issues) before publication. #287 concerns missing Python secret/CVE detections; #569 concerns fabricated instrumentation coverage; #651 covers gate reach; #682 requirement evidence; #693 keyboard oracles. This issue owns security-scan producer scope and execution receipts. It does not add general agent-transcript logging, new security engines, or automatic release blocking.

[proffesor-for-testing]

Progress: #705 is merged and lands the execution-truth portion of this issue. Mapping against the acceptance criteria:

• Done: read/discovery failures visible without leaking contents (2); unsupported readable files not counted as analyzed (3); missing/failed Semgrep distinct from zero-finding Semgrep (4); filesScanned = completed analyses (5); rulesApplied never derived from findings (6); domain/handler/MCP/CLI retain partial/unavailable dispositions (7); placeholder secret scan cannot return a measured clean (8); backward-compatible schema with docs (10); integration tests on real producer/consumer paths (11).
• Partial: one terminal disposition per file per engine (1): per-file receipts exist for the pattern engines and lexical dedupe is declared, but symlink/alias identity is not resolved and Semgrep has no per-file receipts; discovery incompleteness cannot become complete (12), but Semgrep is required: false, so unknown external coverage does not affect completeness.
• Not done: reuse on changed digest or rule set (9); no cross-run reuse exists yet.

Also still open from the PR's own scope-out: atomic source snapshot, per-file Semgrep proof, real DAST/secret receipts, and feeding evidence to #651. Keeping this issue open for the remainder.