2.0 foundation: sample scope, coverage, session policy, snapshot fingerprint (#791 #778 #787 #790)

CHANGELOG.md

@@ -9,6 +9,11 @@ and verification for each change is in the
 
 ### Added
 
+- **JSON:** `pages[].detail.module_detail` says what `detail.modules` holds (`full`, `omitted` in a batch, `not_restored` in a cache replay), so an empty module block is not read as "nothing found".
+- **Architecture:** `tests/architecture_test.rs` fails when an analysis module imports `cli`, `output` or the audit orchestration beyond a shrinking baseline.
+- **Session:** `--storage-state FILE` (Playwright-style cookies and localStorage) starts every page from a declared login, consent or SPA state; `--state-id` names it. `audit_scope.measurement_profile` records the name, the file's SHA-256 and the counts — never the values — and the state is part of the cache key.
+- **Session:** `--session isolated|shared` (default `isolated`) chooses whether the pages of a run share cookies and storage; isolated pages run in a browser context of their own, so audit order and repetition cannot change a result. The choice is stored in `audit_scope.measurement_profile` (`version`, `session_policy`, `read_only`, `state_changing_journeys`) and in the cache key; `--interactive basic` is a read-only run, `full` lists the journeys that can submit forms or change a cart.
+- **Batch:** `summary.score_spread` (min, median, max, standard deviation, five worst pages) and `summary.selection_profile` (audited pages by page type and language, unknowns counted) accompany `average_score`; a `first_n` selection is documented as valid for the audited pages only, not as a representative sample of the site.
 - **AI transparency:** `ai_transparency.coverage` counts attempted, checked and limit-skipped images and failed fetches by reason (`redirect_not_followed`, `oversized`, `timeout`, `host_not_public`, …).
 - **Security:** `security.fetch` records where the analysed headers came from (`http_probe`, method, status, requested and final URL, rejected `HEAD` status); `security.document_match` records whether the probe ended on the document the browser audited.
 - **Audit contracts:** A format-independent validation boundary checks normalized page and batch data before live results are returned, final runs are persisted, or CLI reports and library exports are generated. Contract failures provide stable codes and data paths; honestly qualified partial results remain usable.
@@ -71,6 +76,9 @@ and verification for each change is in the
 
 ### Fixed
 
+- **Cache:** The content fingerprint covers every AX node (role, name, value, properties) and the page-describing SEO parts instead of the node count and 100 ids, so a changed label on an unchanged tree shape is detected; measured timings no longer enter it. Entries record the fingerprint version and older ones are not compared.
+- **Audit quality:** Each requested shared rule needs an execution outcome (`audit_scope.requested_rules`); a missing one fails validation. An errored rule check is counted once (`failed_rule_checks`), no longer again as a partial accessibility module, and the counter must match the outcomes.
+- **Coverage:** The PDF principle coverage is derived from the rules that ran, like the JSON, instead of from the static catalog: an empty run is no longer 100 %, rules that errored, were disabled or lacked a capability count as not passed, and rules without anything to check on the page leave the denominator.
 - **Browser:** A run that was killed from outside (SIGKILL, a supervisor or test timeout) no longer leaves its headless Chrome and profile directory behind for good: the next run ends a Chrome still holding an orphaned `auditmysite-chrome-*` profile and removes the profile.
 - **AI visibility:** The batch result aggregates overall score, dimension scores and signals over the same page set instead of taking the first page's details; signals are majority outcomes with an `example_url` for their values, and the result no longer depends on page order (#785).
 - **AI transparency:** An image fetch that redirects is counted as failed instead of being dropped silently, and a page whose image fetches all failed reports the check as failed rather than "no AI-generated images".

CLAUDE.md

@@ -128,9 +128,11 @@ Lokal vor dem Tag, falls `cargo-docs-rs` installiert ist: `cargo docs-rs` baut m
 # 1. Single page audit (all modules) — tiefe Analyse einer konkreten Seite
 ./target/release/auditmysite https://example.com --full --format pdf --output reports/example-audit.pdf
 
-# 2. Sample batch audit — 20 Seiten als repräsentativer Durchschnitt
-# Ideal um template-weite Probleme (fehlendes ARIA, Struktur, SEO-Muster)
-# von seitenspezifischen Fehlern zu trennen. Liefert stabile Durchschnittswerte.
+# 2. Sample batch audit — die ersten 20 Seiten der Sitemap (first_n)
+# Aussage gilt nur für diese erfolgreich geprüfte Auswahl, nicht für die Website:
+# first_n ist nach Reihenfolge gewählt, nicht nach Seitentyp/Template/Sprache.
+# Brauchbar für eine erste Einschätzung template-weiter Probleme; Streuung und
+# Worst Cases stehen in summary.score_spread, die Auswahl in summary.selection_profile.
 ./target/release/auditmysite --sitemap https://example.com/sitemap.xml --full --format pdf --output reports/example-sample-audit.pdf --max-pages 20
 
 # 3. Full sitemap batch audit — alle Seiten, domain-weit

docs/OUTPUT_CONTRACT.md

@@ -117,6 +117,12 @@ Binary evidence is not embedded in the JSON. The top-level `artifacts` array poi
   - `measurement = "estimated_lab"` — derived/estimated from other lab signals (INP, TTI, Speed Index). These are approximations, not direct measurements, and must not be read as field metrics.
 - In rendered reports, estimated metrics carry a localized "(lab estimate)" / "(Lab-Schätzung)" suffix, and the performance section states that all values are lab data, not field data.
 
+## Measurement Profile
+
+- `audit_scope.measurement_profile` states how the pages were measured: `version` (meaning of the profile), `session_policy` (`isolated` or `shared`; `unknown` for reports from before the field), `read_only`, and `state_changing_journeys` (`form_error_submit`, `add_to_cart`, `quantity_stepper` when `--interactive full` permits them).
+- `state_id` and `storage_state` (`sha256`, `cookies`, `origins`) name a declared browser state (`--storage-state`, `--state-id`). Values never enter the report. A different state file is a different `sha256` and so another measurement.
+- A result measured under another profile is another measurement. Validation rejects a profile whose `read_only` or journey list contradicts `interactive_mode`.
+
 ## Batch Sample Metadata
 
 - Batch reports may include a top-level `sample` block describing how the audited URL set was discovered and sampled:
@@ -127,6 +133,9 @@ Binary evidence is not embedded in the JSON. The top-level `artifacts` array poi
   - `selection` — `"first_n"` (discovery order), `"all"`, or `"path_filter"` (narrowed by `--include-path`/`--exclude-path`, then capped by `sample_limit` in discovery order).
   - `is_sample` — `true` when fewer URLs were audited than discovered.
 - The block is omitted when no sampling metadata was recorded (e.g. cached single reports).
+- A selection (`first_n`, `path_filter`) is no evidence of representativeness. Batch statements hold for the successfully audited pages (`summary.coverage.succeeded`); failed and blocked URLs are counted in `summary.coverage`, never in the averages.
+- `summary.score_spread` reports `min`, `median`, `max`, `std_dev` and the `worst_pages` (up to 5, worst first) next to `average_score`.
+- `summary.selection_profile` counts the audited pages by canonical `page_types` and primary-subtag `languages`; pages without the signal are counted in `unclassified` / `language_unknown`.
 
 ## Batch Site Analysis
 

docs/json-report.schema.json

@@ -154,6 +154,67 @@
             "type": "string"
           }
         },
+        "requested_subchecks": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "requested_rules": {
+          "description": "Shared rules the run must account for with an outcome (#778).",
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "measurement_profile": {
+          "description": "How the pages were measured: session policy and state-changing journeys (#787).",
+          "type": "object",
+          "required": [
+            "version",
+            "session_policy",
+            "read_only"
+          ],
+          "additionalProperties": false,
+          "properties": {
+            "version": {
+              "type": "integer",
+              "minimum": 0
+            },
+            "session_policy": {
+              "type": "string",
+              "enum": [
+                "isolated",
+                "shared",
+                "unknown"
+              ]
+            },
+            "read_only": {
+              "type": "boolean"
+            },
+            "state_changing_journeys": {
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
+            "state_id": {
+              "description": "Name of the declared state the run started from (--state-id).",
+              "type": "string"
+            },
+            "storage_state": {
+              "description": "Identity of the declared browser state: digest and counts, never values.",
+              "type": "object",
+              "required": ["sha256", "cookies", "origins"],
+              "additionalProperties": false,
+              "properties": {
+                "sha256": { "type": "string" },
+                "cookies": { "type": "integer", "minimum": 0 },
+                "origins": { "type": "integer", "minimum": 0 }
+              }
+            }
+          }
+        },
         "full_audit": {
           "type": "boolean"
         },
@@ -845,6 +906,11 @@
             }
           }
         },
+        "module_detail": {
+          "description": "What detail.modules holds: full (collected data), omitted (compact batch projection) or not_restored (cache replay without raw module data). Empty modules mean 'nothing found' only when this is full.",
+          "type": "string",
+          "enum": ["full", "omitted", "not_restored"]
+        },
         "viewport_scores": {
           "type": [
             "object",

src/a11y_journey/mod.rs

@@ -82,13 +82,30 @@ pub const DEFAULT_BUDGET_MS: u64 = 15_000;
 /// ohne Hänger lag bei 5,6 s (Modal).
 const JOURNEY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
 
+/// Journeys that click controls able to change page or account state: a
+/// submit button, an add-to-cart trigger, a quantity stepper (#787). Only
+/// `--interactive full` runs them; `basic` and `off` are read-only. The
+/// measurement profile lists the permitted ones, with the ids the JSON uses.
+const STATE_CHANGING: [(JourneyKind, &str); 3] = [
+    (JourneyKind::FormErrorSubmit, "form_error_submit"),
+    (JourneyKind::AddToCart, "add_to_cart"),
+    (JourneyKind::QuantityStepper, "quantity_stepper"),
+];
+
+/// Ids of the state-changing journeys `mode` permits; empty for a read-only
+/// run.
+pub fn state_changing_journeys(mode: InteractiveMode) -> Vec<&'static str> {
+    STATE_CHANGING
+        .iter()
+        .filter(|(kind, _)| journey_allowed(mode, *kind))
+        .map(|(_, id)| *id)
+        .collect()
+}
+
 fn journey_allowed(mode: InteractiveMode, journey: JourneyKind) -> bool {
     match mode {
         InteractiveMode::Off => false,
-        InteractiveMode::Basic => !matches!(
-            journey,
-            JourneyKind::FormErrorSubmit | JourneyKind::AddToCart | JourneyKind::QuantityStepper
-        ),
+        InteractiveMode::Basic => !STATE_CHANGING.iter().any(|(kind, _)| *kind == journey),
         InteractiveMode::Full => true,
     }
 }
@@ -477,6 +494,21 @@ pub(super) async fn eval_string(page: &Page, js: &str) -> Option<String> {
 
 #[cfg(test)]
 mod tests {
+    #[test]
+    fn only_a_full_run_may_change_state() {
+        assert!(state_changing_journeys(InteractiveMode::Off).is_empty());
+        assert!(state_changing_journeys(InteractiveMode::Basic).is_empty());
+        assert_eq!(
+            state_changing_journeys(InteractiveMode::Full),
+            ["form_error_submit", "add_to_cart", "quantity_stepper"]
+        );
+        // The profile list and the gate are one fact.
+        for (kind, id) in STATE_CHANGING {
+            assert!(!journey_allowed(InteractiveMode::Basic, kind), "{id}");
+            assert!(journey_allowed(InteractiveMode::Full, kind), "{id}");
+        }
+    }
+
     use super::*;
 
     fn candidate(confidence: f32, backend: i64) -> JourneyCandidate {

src/audit/artifacts.rs

@@ -56,6 +56,19 @@ pub struct CacheMeta {
     /// this field existed — such entries are never reused.
     #[serde(default)]
     pub audit_signature: String,
+    /// What `content_hash` covered when this entry was written
+    /// ([`FINGERPRINT_VERSION`]). 0 for entries from before the field: their
+    /// hash does not compare with a current one.
+    #[serde(default)]
+    pub fingerprint_version: u32,
+}
+
+impl CacheMeta {
+    /// Whether this entry's content hash can be compared with one computed
+    /// now.
+    pub fn fingerprint_comparable(&self) -> bool {
+        self.fingerprint_version == FINGERPRINT_VERSION
+    }
 }
 
 /// Whether a cached entry was produced with an audit configuration compatible
@@ -132,6 +145,7 @@ pub fn save_artifacts(url: &str, wcag_level: &str, artifacts: &AuditArtifacts) -
         cached_at: Utc::now(),
         content_hash: artifacts.content_hash.clone(),
         audit_signature: artifacts.meta.audit_signature.clone(),
+        fingerprint_version: FINGERPRINT_VERSION,
     };
     fs::write(dir.join("meta.json"), serde_json::to_vec_pretty(&meta)?)?;
 
@@ -198,6 +212,7 @@ pub fn load_artifacts(url: &str) -> Result<Option<AuditArtifacts>> {
             cached_at: fetch.fetched_at,
             content_hash: content_hash.clone(),
             audit_signature: String::new(),
+            fingerprint_version: 0,
         }
     };
 
@@ -211,44 +226,75 @@ pub fn load_artifacts(url: &str) -> Result<Option<AuditArtifacts>> {
     }))
 }
 
-/// FNV-1a fingerprint of the snapshot's AXTree structure and key SEO signals.
+/// Version of what [`content_hash`] covers. A hash of another version never
+/// compares equal to this one's: it measured something else.
+pub const FINGERPRINT_VERSION: u32 = 2;
+
+/// FNV-1a fingerprint of the snapshot's content: every AX node with role,
+/// name, value, properties, links and DOM anchor, plus the SEO parts that
+/// describe the page (meta, headings, social tags, technical SEO, structured
+/// data).
+///
+/// Used for delta detection between two runs of the same URL, not as a cache
+/// key. Stable across processes and platforms (unlike DefaultHasher) and
+/// independent of map iteration order. Measured timings (LCP, TBT, …) are
+/// not content and stay out: a slower run of an unchanged page is no change.
 ///
-/// Used for delta detection between two runs of the same URL, not as a cache key.
-/// Stable across processes and platforms (unlike DefaultHasher).
+/// Version 1 hashed the node count, the first 100 node ids and a few SEO
+/// strings, so a changed label or text on an unchanged tree shape went
+/// undetected (#790).
 pub fn content_hash(snapshot: &SnapshotArtifact) -> String {
-    let mut input = String::with_capacity(512);
-
-    // AXTree structural fingerprint: node count + root + sorted node IDs (first 100)
-    let node_count = snapshot.ax_tree.len();
-    input.push_str(&node_count.to_string());
-    input.push(':');
-
-    let mut node_ids: Vec<&str> = snapshot.ax_tree.nodes.keys().map(String::as_str).collect();
-    node_ids.sort_unstable();
-    for id in node_ids.iter().take(100) {
+    let mut input = String::with_capacity(4096);
+    let mut nodes: Vec<_> = snapshot.ax_tree.nodes.iter().collect();
+    nodes.sort_unstable_by(|a, b| a.0.cmp(b.0));
+    for (id, node) in nodes {
         input.push_str(id);
-        input.push(',');
+        input.push('=');
+        push_canonical(&mut input, &serde_json::to_value(node).unwrap_or_default());
+        input.push('\n');
     }
-
-    // SEO signals
-    if let Some(ref seo) = snapshot.seo {
-        if let Some(ref t) = seo.meta.title {
-            input.push_str(t);
-        }
-        if let Some(ref d) = seo.meta.description {
-            input.push_str(d);
+    if let Some(seo) = &snapshot.seo {
+        for part in [
+            serde_json::to_value(&seo.meta),
+            serde_json::to_value(&seo.headings),
+            serde_json::to_value(&seo.social),
+            serde_json::to_value(&seo.technical),
+            serde_json::to_value(&seo.structured_data),
+        ] {
+            push_canonical(&mut input, &part.unwrap_or_default());
+            input.push('\n');
         }
-        input.push_str(&seo.headings.total_count.to_string());
     }
+    format!("{:016x}", fnv1a(input.as_bytes()))
+}
 
-    // Performance: LCP as a change indicator
-    if let Some(ref perf) = snapshot.performance {
-        if let Some(ref lcp) = perf.vitals.lcp {
-            input.push_str(&(lcp.value as u64).to_string());
+/// JSON with object keys sorted, so equal content hashes equal whatever the
+/// order a map was filled in.
+fn push_canonical(out: &mut String, value: &serde_json::Value) {
+    use serde_json::Value;
+    match value {
+        Value::Object(map) => {
+            let mut keys: Vec<&String> = map.keys().collect();
+            keys.sort_unstable();
+            out.push('{');
+            for key in keys {
+                out.push_str(key);
+                out.push(':');
+                push_canonical(out, &map[key]);
+                out.push(',');
+            }
+            out.push('}');
         }
+        Value::Array(items) => {
+            out.push('[');
+            for item in items {
+                push_canonical(out, item);
+                out.push(',');
+            }
+            out.push(']');
+        }
+        other => out.push_str(&other.to_string()),
     }
-
-    format!("{:016x}", fnv1a(input.as_bytes()))
 }
 
 pub fn to_audit_report(artifacts: &AuditArtifacts, locale: &str) -> AuditReport {
@@ -436,6 +482,7 @@ mod tests {
             cached_at: Utc::now(),
             content_hash: "deadbeef".to_string(),
             audit_signature: signature.to_string(),
+            fingerprint_version: 0,
         }
     }
 
@@ -469,6 +516,34 @@ mod tests {
         ));
     }
 
+    fn labelled(name: Option<&str>, text_value: Option<&str>) -> SnapshotArtifact {
+        let mut snapshot = snapshot_with_unlabelled_field(&[]);
+        let field = snapshot.ax_tree.nodes.get_mut("field").unwrap();
+        field.name = name.map(str::to_string);
+        field.value = text_value.map(str::to_string);
+        snapshot
+    }
+
+    #[test]
+    fn changed_content_on_an_unchanged_tree_shape_changes_the_hash() {
+        let before = content_hash(&labelled(Some("Email"), None));
+        // Same node count, same ids: only the label differs.
+        assert_ne!(
+            before,
+            content_hash(&labelled(Some("E-Mail address"), None))
+        );
+        assert_ne!(before, content_hash(&labelled(Some("Email"), Some("x"))));
+        assert_eq!(before, content_hash(&labelled(Some("Email"), None)));
+    }
+
+    #[test]
+    fn an_entry_hashed_by_an_older_fingerprint_is_not_comparable() {
+        let mut meta = meta_with_signature("sig");
+        assert!(!meta.fingerprint_comparable());
+        meta.fingerprint_version = FINGERPRINT_VERSION;
+        assert!(meta.fingerprint_comparable());
+    }
+
     fn snapshot_with_unlabelled_field(excluded: &[&str]) -> SnapshotArtifact {
         let node = |id: &str, role: &str, children: Vec<String>, parent: Option<&str>| {
             a11y_perception::AXNode {

src/audit/batch.rs

@@ -47,6 +47,7 @@ impl From<&Args> for BatchConfig {
             browser_options: BrowserOptions {
                 // Parallel pages share one window; only then are tabs hidden.
                 focus_emulation: args.effective_concurrency() > 1,
+                session: args.session,
                 chrome_path: args.chrome_path.clone(),
                 no_sandbox: args.no_sandbox,
                 disable_images: args.disable_images,