diff --git a/CHANGELOG.md b/CHANGELOG.md index 40b6b489..f1515b1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ and verification for each change is in the ### Added +- **JSON:** `pages[].detail.module_detail` says what `detail.modules` holds (`full`, `omitted` in a batch, `not_restored` in a cache replay), so an empty module block is not read as "nothing found". +- **Architecture:** `tests/architecture_test.rs` fails when an analysis module imports `cli`, `output` or the audit orchestration beyond a shrinking baseline. +- **Session:** `--storage-state FILE` (Playwright-style cookies and localStorage) starts every page from a declared login, consent or SPA state; `--state-id` names it. `audit_scope.measurement_profile` records the name, the file's SHA-256 and the counts — never the values — and the state is part of the cache key. +- **Session:** `--session isolated|shared` (default `isolated`) chooses whether the pages of a run share cookies and storage; isolated pages run in a browser context of their own, so audit order and repetition cannot change a result. The choice is stored in `audit_scope.measurement_profile` (`version`, `session_policy`, `read_only`, `state_changing_journeys`) and in the cache key; `--interactive basic` is a read-only run, `full` lists the journeys that can submit forms or change a cart. +- **Batch:** `summary.score_spread` (min, median, max, standard deviation, five worst pages) and `summary.selection_profile` (audited pages by page type and language, unknowns counted) accompany `average_score`; a `first_n` selection is documented as valid for the audited pages only, not as a representative sample of the site. - **AI transparency:** `ai_transparency.coverage` counts attempted, checked and limit-skipped images and failed fetches by reason (`redirect_not_followed`, `oversized`, `timeout`, `host_not_public`, …). - **Security:** `security.fetch` records where the analysed headers came from (`http_probe`, method, status, requested and final URL, rejected `HEAD` status); `security.document_match` records whether the probe ended on the document the browser audited. - **Audit contracts:** A format-independent validation boundary checks normalized page and batch data before live results are returned, final runs are persisted, or CLI reports and library exports are generated. Contract failures provide stable codes and data paths; honestly qualified partial results remain usable. @@ -71,6 +76,9 @@ and verification for each change is in the ### Fixed +- **Cache:** The content fingerprint covers every AX node (role, name, value, properties) and the page-describing SEO parts instead of the node count and 100 ids, so a changed label on an unchanged tree shape is detected; measured timings no longer enter it. Entries record the fingerprint version and older ones are not compared. +- **Audit quality:** Each requested shared rule needs an execution outcome (`audit_scope.requested_rules`); a missing one fails validation. An errored rule check is counted once (`failed_rule_checks`), no longer again as a partial accessibility module, and the counter must match the outcomes. +- **Coverage:** The PDF principle coverage is derived from the rules that ran, like the JSON, instead of from the static catalog: an empty run is no longer 100 %, rules that errored, were disabled or lacked a capability count as not passed, and rules without anything to check on the page leave the denominator. - **Browser:** A run that was killed from outside (SIGKILL, a supervisor or test timeout) no longer leaves its headless Chrome and profile directory behind for good: the next run ends a Chrome still holding an orphaned `auditmysite-chrome-*` profile and removes the profile. - **AI visibility:** The batch result aggregates overall score, dimension scores and signals over the same page set instead of taking the first page's details; signals are majority outcomes with an `example_url` for their values, and the result no longer depends on page order (#785). - **AI transparency:** An image fetch that redirects is counted as failed instead of being dropped silently, and a page whose image fetches all failed reports the check as failed rather than "no AI-generated images". diff --git a/CLAUDE.md b/CLAUDE.md index 2edf8cdb..cd8da5a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -128,9 +128,11 @@ Lokal vor dem Tag, falls `cargo-docs-rs` installiert ist: `cargo docs-rs` baut m # 1. Single page audit (all modules) — tiefe Analyse einer konkreten Seite ./target/release/auditmysite https://example.com --full --format pdf --output reports/example-audit.pdf -# 2. Sample batch audit — 20 Seiten als repräsentativer Durchschnitt -# Ideal um template-weite Probleme (fehlendes ARIA, Struktur, SEO-Muster) -# von seitenspezifischen Fehlern zu trennen. Liefert stabile Durchschnittswerte. +# 2. Sample batch audit — die ersten 20 Seiten der Sitemap (first_n) +# Aussage gilt nur für diese erfolgreich geprüfte Auswahl, nicht für die Website: +# first_n ist nach Reihenfolge gewählt, nicht nach Seitentyp/Template/Sprache. +# Brauchbar für eine erste Einschätzung template-weiter Probleme; Streuung und +# Worst Cases stehen in summary.score_spread, die Auswahl in summary.selection_profile. ./target/release/auditmysite --sitemap https://example.com/sitemap.xml --full --format pdf --output reports/example-sample-audit.pdf --max-pages 20 # 3. Full sitemap batch audit — alle Seiten, domain-weit diff --git a/docs/OUTPUT_CONTRACT.md b/docs/OUTPUT_CONTRACT.md index b763ae78..90d0711c 100644 --- a/docs/OUTPUT_CONTRACT.md +++ b/docs/OUTPUT_CONTRACT.md @@ -117,6 +117,12 @@ Binary evidence is not embedded in the JSON. The top-level `artifacts` array poi - `measurement = "estimated_lab"` — derived/estimated from other lab signals (INP, TTI, Speed Index). These are approximations, not direct measurements, and must not be read as field metrics. - In rendered reports, estimated metrics carry a localized "(lab estimate)" / "(Lab-Schätzung)" suffix, and the performance section states that all values are lab data, not field data. +## Measurement Profile + +- `audit_scope.measurement_profile` states how the pages were measured: `version` (meaning of the profile), `session_policy` (`isolated` or `shared`; `unknown` for reports from before the field), `read_only`, and `state_changing_journeys` (`form_error_submit`, `add_to_cart`, `quantity_stepper` when `--interactive full` permits them). +- `state_id` and `storage_state` (`sha256`, `cookies`, `origins`) name a declared browser state (`--storage-state`, `--state-id`). Values never enter the report. A different state file is a different `sha256` and so another measurement. +- A result measured under another profile is another measurement. Validation rejects a profile whose `read_only` or journey list contradicts `interactive_mode`. + ## Batch Sample Metadata - Batch reports may include a top-level `sample` block describing how the audited URL set was discovered and sampled: @@ -127,6 +133,9 @@ Binary evidence is not embedded in the JSON. The top-level `artifacts` array poi - `selection` — `"first_n"` (discovery order), `"all"`, or `"path_filter"` (narrowed by `--include-path`/`--exclude-path`, then capped by `sample_limit` in discovery order). - `is_sample` — `true` when fewer URLs were audited than discovered. - The block is omitted when no sampling metadata was recorded (e.g. cached single reports). +- A selection (`first_n`, `path_filter`) is no evidence of representativeness. Batch statements hold for the successfully audited pages (`summary.coverage.succeeded`); failed and blocked URLs are counted in `summary.coverage`, never in the averages. +- `summary.score_spread` reports `min`, `median`, `max`, `std_dev` and the `worst_pages` (up to 5, worst first) next to `average_score`. +- `summary.selection_profile` counts the audited pages by canonical `page_types` and primary-subtag `languages`; pages without the signal are counted in `unclassified` / `language_unknown`. ## Batch Site Analysis diff --git a/docs/json-report.schema.json b/docs/json-report.schema.json index 6ae376d3..94990b9c 100644 --- a/docs/json-report.schema.json +++ b/docs/json-report.schema.json @@ -154,6 +154,67 @@ "type": "string" } }, + "requested_subchecks": { + "type": "array", + "items": { + "type": "string" + } + }, + "requested_rules": { + "description": "Shared rules the run must account for with an outcome (#778).", + "type": "array", + "items": { + "type": "string" + } + }, + "measurement_profile": { + "description": "How the pages were measured: session policy and state-changing journeys (#787).", + "type": "object", + "required": [ + "version", + "session_policy", + "read_only" + ], + "additionalProperties": false, + "properties": { + "version": { + "type": "integer", + "minimum": 0 + }, + "session_policy": { + "type": "string", + "enum": [ + "isolated", + "shared", + "unknown" + ] + }, + "read_only": { + "type": "boolean" + }, + "state_changing_journeys": { + "type": "array", + "items": { + "type": "string" + } + }, + "state_id": { + "description": "Name of the declared state the run started from (--state-id).", + "type": "string" + }, + "storage_state": { + "description": "Identity of the declared browser state: digest and counts, never values.", + "type": "object", + "required": ["sha256", "cookies", "origins"], + "additionalProperties": false, + "properties": { + "sha256": { "type": "string" }, + "cookies": { "type": "integer", "minimum": 0 }, + "origins": { "type": "integer", "minimum": 0 } + } + } + } + }, "full_audit": { "type": "boolean" }, @@ -845,6 +906,11 @@ } } }, + "module_detail": { + "description": "What detail.modules holds: full (collected data), omitted (compact batch projection) or not_restored (cache replay without raw module data). Empty modules mean 'nothing found' only when this is full.", + "type": "string", + "enum": ["full", "omitted", "not_restored"] + }, "viewport_scores": { "type": [ "object", diff --git a/src/a11y_journey/mod.rs b/src/a11y_journey/mod.rs index 890dd105..5eef55fb 100644 --- a/src/a11y_journey/mod.rs +++ b/src/a11y_journey/mod.rs @@ -82,13 +82,30 @@ pub const DEFAULT_BUDGET_MS: u64 = 15_000; /// ohne Hänger lag bei 5,6 s (Modal). const JOURNEY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); +/// Journeys that click controls able to change page or account state: a +/// submit button, an add-to-cart trigger, a quantity stepper (#787). Only +/// `--interactive full` runs them; `basic` and `off` are read-only. The +/// measurement profile lists the permitted ones, with the ids the JSON uses. +const STATE_CHANGING: [(JourneyKind, &str); 3] = [ + (JourneyKind::FormErrorSubmit, "form_error_submit"), + (JourneyKind::AddToCart, "add_to_cart"), + (JourneyKind::QuantityStepper, "quantity_stepper"), +]; + +/// Ids of the state-changing journeys `mode` permits; empty for a read-only +/// run. +pub fn state_changing_journeys(mode: InteractiveMode) -> Vec<&'static str> { + STATE_CHANGING + .iter() + .filter(|(kind, _)| journey_allowed(mode, *kind)) + .map(|(_, id)| *id) + .collect() +} + fn journey_allowed(mode: InteractiveMode, journey: JourneyKind) -> bool { match mode { InteractiveMode::Off => false, - InteractiveMode::Basic => !matches!( - journey, - JourneyKind::FormErrorSubmit | JourneyKind::AddToCart | JourneyKind::QuantityStepper - ), + InteractiveMode::Basic => !STATE_CHANGING.iter().any(|(kind, _)| *kind == journey), InteractiveMode::Full => true, } } @@ -477,6 +494,21 @@ pub(super) async fn eval_string(page: &Page, js: &str) -> Option { #[cfg(test)] mod tests { + #[test] + fn only_a_full_run_may_change_state() { + assert!(state_changing_journeys(InteractiveMode::Off).is_empty()); + assert!(state_changing_journeys(InteractiveMode::Basic).is_empty()); + assert_eq!( + state_changing_journeys(InteractiveMode::Full), + ["form_error_submit", "add_to_cart", "quantity_stepper"] + ); + // The profile list and the gate are one fact. + for (kind, id) in STATE_CHANGING { + assert!(!journey_allowed(InteractiveMode::Basic, kind), "{id}"); + assert!(journey_allowed(InteractiveMode::Full, kind), "{id}"); + } + } + use super::*; fn candidate(confidence: f32, backend: i64) -> JourneyCandidate { diff --git a/src/audit/artifacts.rs b/src/audit/artifacts.rs index 28cdfd89..5831e25b 100644 --- a/src/audit/artifacts.rs +++ b/src/audit/artifacts.rs @@ -56,6 +56,19 @@ pub struct CacheMeta { /// this field existed — such entries are never reused. #[serde(default)] pub audit_signature: String, + /// What `content_hash` covered when this entry was written + /// ([`FINGERPRINT_VERSION`]). 0 for entries from before the field: their + /// hash does not compare with a current one. + #[serde(default)] + pub fingerprint_version: u32, +} + +impl CacheMeta { + /// Whether this entry's content hash can be compared with one computed + /// now. + pub fn fingerprint_comparable(&self) -> bool { + self.fingerprint_version == FINGERPRINT_VERSION + } } /// Whether a cached entry was produced with an audit configuration compatible @@ -132,6 +145,7 @@ pub fn save_artifacts(url: &str, wcag_level: &str, artifacts: &AuditArtifacts) - cached_at: Utc::now(), content_hash: artifacts.content_hash.clone(), audit_signature: artifacts.meta.audit_signature.clone(), + fingerprint_version: FINGERPRINT_VERSION, }; fs::write(dir.join("meta.json"), serde_json::to_vec_pretty(&meta)?)?; @@ -198,6 +212,7 @@ pub fn load_artifacts(url: &str) -> Result> { cached_at: fetch.fetched_at, content_hash: content_hash.clone(), audit_signature: String::new(), + fingerprint_version: 0, } }; @@ -211,44 +226,75 @@ pub fn load_artifacts(url: &str) -> Result> { })) } -/// FNV-1a fingerprint of the snapshot's AXTree structure and key SEO signals. +/// Version of what [`content_hash`] covers. A hash of another version never +/// compares equal to this one's: it measured something else. +pub const FINGERPRINT_VERSION: u32 = 2; + +/// FNV-1a fingerprint of the snapshot's content: every AX node with role, +/// name, value, properties, links and DOM anchor, plus the SEO parts that +/// describe the page (meta, headings, social tags, technical SEO, structured +/// data). +/// +/// Used for delta detection between two runs of the same URL, not as a cache +/// key. Stable across processes and platforms (unlike DefaultHasher) and +/// independent of map iteration order. Measured timings (LCP, TBT, …) are +/// not content and stay out: a slower run of an unchanged page is no change. /// -/// Used for delta detection between two runs of the same URL, not as a cache key. -/// Stable across processes and platforms (unlike DefaultHasher). +/// Version 1 hashed the node count, the first 100 node ids and a few SEO +/// strings, so a changed label or text on an unchanged tree shape went +/// undetected (#790). pub fn content_hash(snapshot: &SnapshotArtifact) -> String { - let mut input = String::with_capacity(512); - - // AXTree structural fingerprint: node count + root + sorted node IDs (first 100) - let node_count = snapshot.ax_tree.len(); - input.push_str(&node_count.to_string()); - input.push(':'); - - let mut node_ids: Vec<&str> = snapshot.ax_tree.nodes.keys().map(String::as_str).collect(); - node_ids.sort_unstable(); - for id in node_ids.iter().take(100) { + let mut input = String::with_capacity(4096); + let mut nodes: Vec<_> = snapshot.ax_tree.nodes.iter().collect(); + nodes.sort_unstable_by(|a, b| a.0.cmp(b.0)); + for (id, node) in nodes { input.push_str(id); - input.push(','); + input.push('='); + push_canonical(&mut input, &serde_json::to_value(node).unwrap_or_default()); + input.push('\n'); } - - // SEO signals - if let Some(ref seo) = snapshot.seo { - if let Some(ref t) = seo.meta.title { - input.push_str(t); - } - if let Some(ref d) = seo.meta.description { - input.push_str(d); + if let Some(seo) = &snapshot.seo { + for part in [ + serde_json::to_value(&seo.meta), + serde_json::to_value(&seo.headings), + serde_json::to_value(&seo.social), + serde_json::to_value(&seo.technical), + serde_json::to_value(&seo.structured_data), + ] { + push_canonical(&mut input, &part.unwrap_or_default()); + input.push('\n'); } - input.push_str(&seo.headings.total_count.to_string()); } + format!("{:016x}", fnv1a(input.as_bytes())) +} - // Performance: LCP as a change indicator - if let Some(ref perf) = snapshot.performance { - if let Some(ref lcp) = perf.vitals.lcp { - input.push_str(&(lcp.value as u64).to_string()); +/// JSON with object keys sorted, so equal content hashes equal whatever the +/// order a map was filled in. +fn push_canonical(out: &mut String, value: &serde_json::Value) { + use serde_json::Value; + match value { + Value::Object(map) => { + let mut keys: Vec<&String> = map.keys().collect(); + keys.sort_unstable(); + out.push('{'); + for key in keys { + out.push_str(key); + out.push(':'); + push_canonical(out, &map[key]); + out.push(','); + } + out.push('}'); } + Value::Array(items) => { + out.push('['); + for item in items { + push_canonical(out, item); + out.push(','); + } + out.push(']'); + } + other => out.push_str(&other.to_string()), } - - format!("{:016x}", fnv1a(input.as_bytes())) } pub fn to_audit_report(artifacts: &AuditArtifacts, locale: &str) -> AuditReport { @@ -436,6 +482,7 @@ mod tests { cached_at: Utc::now(), content_hash: "deadbeef".to_string(), audit_signature: signature.to_string(), + fingerprint_version: 0, } } @@ -469,6 +516,34 @@ mod tests { )); } + fn labelled(name: Option<&str>, text_value: Option<&str>) -> SnapshotArtifact { + let mut snapshot = snapshot_with_unlabelled_field(&[]); + let field = snapshot.ax_tree.nodes.get_mut("field").unwrap(); + field.name = name.map(str::to_string); + field.value = text_value.map(str::to_string); + snapshot + } + + #[test] + fn changed_content_on_an_unchanged_tree_shape_changes_the_hash() { + let before = content_hash(&labelled(Some("Email"), None)); + // Same node count, same ids: only the label differs. + assert_ne!( + before, + content_hash(&labelled(Some("E-Mail address"), None)) + ); + assert_ne!(before, content_hash(&labelled(Some("Email"), Some("x")))); + assert_eq!(before, content_hash(&labelled(Some("Email"), None))); + } + + #[test] + fn an_entry_hashed_by_an_older_fingerprint_is_not_comparable() { + let mut meta = meta_with_signature("sig"); + assert!(!meta.fingerprint_comparable()); + meta.fingerprint_version = FINGERPRINT_VERSION; + assert!(meta.fingerprint_comparable()); + } + fn snapshot_with_unlabelled_field(excluded: &[&str]) -> SnapshotArtifact { let node = |id: &str, role: &str, children: Vec, parent: Option<&str>| { a11y_perception::AXNode { diff --git a/src/audit/batch.rs b/src/audit/batch.rs index cf89e61a..82d5df1f 100644 --- a/src/audit/batch.rs +++ b/src/audit/batch.rs @@ -47,6 +47,7 @@ impl From<&Args> for BatchConfig { browser_options: BrowserOptions { // Parallel pages share one window; only then are tabs hidden. focus_emulation: args.effective_concurrency() > 1, + session: args.session, chrome_path: args.chrome_path.clone(), no_sandbox: args.no_sandbox, disable_images: args.disable_images, diff --git a/src/audit/mod.rs b/src/audit/mod.rs index b44e8fe3..6239e116 100644 --- a/src/audit/mod.rs +++ b/src/audit/mod.rs @@ -56,11 +56,11 @@ pub use report::{ BatchReport, BatchSummary, BrokenLink, BrokenLinkSeverity, ConsentAuditState, ConsentCookieSignal, ConsentPrivacySnapshot, CrawlDepthDiagnostics, CrawlDepthEntry, CrawlDiagnostics, DiscoverabilitySection, DualViewportResults, ExecutionEnvironment, - ExecutionStatus, ExperienceSection, ModuleRun, NavigationSnapshot, PageScreenshots, - PerformanceResults, RedirectChain, RobotsSitemapConflict, RunContext, SampleMetadata, - ScreenshotStatus, SitemapDiagnostics, SitemapHttpIssue, SubcheckRun, ThrottledPerfResult, - ViewportAuditData, ViewportDefinition, ViewportMetaRefresh, ViewportScoreSet, ViewportScores, - ViewportScreenshot, + ExecutionStatus, ExperienceSection, MeasurementProfile, ModuleRun, NavigationSnapshot, + PageScreenshots, PerformanceResults, RedirectChain, RobotsSitemapConflict, RunContext, + SampleMetadata, ScreenshotStatus, SitemapDiagnostics, SitemapHttpIssue, StorageStateRef, + SubcheckRun, ThrottledPerfResult, ViewportAuditData, ViewportDefinition, ViewportMetaRefresh, + ViewportScoreSet, ViewportScores, ViewportScreenshot, MEASUREMENT_PROFILE_VERSION, }; pub(crate) use scoring::{default_impact, score_from_penalties}; pub use scoring::{AccessibilityScorer, CoverageRatio, PrincipleCoverage, ViolationStatistics}; diff --git a/src/audit/pipeline/assembly.rs b/src/audit/pipeline/assembly.rs index 2608bef3..f6bce54d 100644 --- a/src/audit/pipeline/assembly.rs +++ b/src/audit/pipeline/assembly.rs @@ -443,7 +443,30 @@ pub(super) fn audit_scope_from_config(config: &PipelineConfig) -> crate::audit:: crate::audit::AuditScope { requested_modules, requested_subchecks, + requested_rules: crate::wcag::shared::SHARED_RULES + .iter() + .map(|rule| rule.id.to_string()) + .collect(), full_audit: config.full_audit, + measurement_profile: crate::audit::MeasurementProfile { + version: crate::audit::MEASUREMENT_PROFILE_VERSION, + session_policy: config.session.as_str().to_string(), + read_only: crate::a11y_journey::state_changing_journeys(config.interactive).is_empty(), + state_changing_journeys: crate::a11y_journey::state_changing_journeys( + config.interactive, + ) + .iter() + .map(|j| j.to_string()) + .collect(), + state_id: config.state_id.clone(), + storage_state: config.storage_state.as_ref().map(|state| { + crate::audit::StorageStateRef { + sha256: state.digest.clone(), + cookies: state.cookie_count(), + origins: state.origin_count(), + } + }), + }, interactive_mode: format!("{:?}", config.interactive).to_lowercase(), journey_budget_ms: config.journey_budget_ms, throttling_profiles: if config.check_performance { @@ -634,10 +657,13 @@ pub(super) fn update_audit_quality(report: &mut AuditReport) { .module_runs .iter() .filter(|run| { - matches!( - run.status, - crate::audit::ExecutionStatus::Partial | crate::audit::ExecutionStatus::Failed - ) + // The accessibility run is partial *because of* the errored rule + // checks counted above — one fact, one counter (#778). + run.reason_code.as_deref() != Some("one_or_more_rule_checks_failed") + && matches!( + run.status, + crate::audit::ExecutionStatus::Partial | crate::audit::ExecutionStatus::Failed + ) }) .count(); let missing_capabilities = report @@ -739,6 +765,222 @@ mod tests { } } + fn quality_of(report: &mut AuditReport) -> crate::audit::AuditQualityStatus { + update_audit_quality(report); + report.accessibility.execution.quality.status + } + + #[test] + fn execution_state_matrix_qualifies_the_audit() { + use crate::audit::AuditQualityStatus::{Complete, Insufficient, Partial}; + use crate::interaction::stability::{StabilityProvenance, StabilityStatus}; + use crate::wcag::{NotRun, RuleRun}; + + let with_rule = |run: RuleRun| { + let mut report = report_with_run("accessibility"); + report.accessibility.wcag_results.rule_outcomes.push(run); + report + }; + let with_stability = |status| { + let mut report = report_with_run("accessibility"); + report + .accessibility + .execution + .navigation + .stability + .push(StabilityProvenance { + viewport: "desktop".into(), + status, + ..Default::default() + }); + report + }; + let cases = [ + ("ran", with_rule(RuleRun::ran("r", 0)), Complete), + ( + "not applicable", + with_rule(RuleRun::not_run("r", NotRun::NotApplicable)), + Complete, + ), + ( + "disabled by configuration", + with_rule(RuleRun::not_run("r", NotRun::Disabled)), + Complete, + ), + ( + "capability missing", + with_rule(RuleRun::not_run("r", NotRun::CapabilityMissing)), + Partial, + ), + ( + "errored", + with_rule(RuleRun::not_run("r", NotRun::Errored)), + Insufficient, + ), + ("stable", with_stability(StabilityStatus::Stable), Complete), + ( + "ongoing animation", + with_stability(StabilityStatus::OngoingAnimation), + Complete, + ), + ( + "fallback", + with_stability(StabilityStatus::Fallback), + Partial, + ), + ( + "budget exhausted", + with_stability(StabilityStatus::BudgetExhausted), + Partial, + ), + ]; + for (name, mut report, expected) in cases { + assert_eq!(quality_of(&mut report), expected, "{name}"); + crate::audit::normalize_checked(&report) + .unwrap_or_else(|e| panic!("{name} must stay valid data: {e}")); + } + } + + #[test] + fn measurement_profile_must_agree_with_the_interactive_depth() { + use clap::Parser; + let config_scope = |interactive: crate::cli::InteractiveMode| { + let mut config = crate::audit::PipelineConfig::from_args_and_config( + &crate::cli::Args::parse_from(["auditmysite", "https://example.com"]), + None, + ); + config.interactive = interactive; + audit_scope_from_config(&config) + }; + let full = config_scope(crate::cli::InteractiveMode::Full); + assert!(!full.measurement_profile.read_only); + assert!(!full.measurement_profile.state_changing_journeys.is_empty()); + assert_eq!(full.measurement_profile.session_policy, "isolated"); + let basic = config_scope(crate::cli::InteractiveMode::Basic); + assert!(basic.measurement_profile.read_only); + + // A read-only mode claiming a state-changing journey is rejected. + let mut report = report_with_run("accessibility"); + report.accessibility.execution.scope = basic; + report + .accessibility + .execution + .scope + .measurement_profile + .state_changing_journeys = vec!["add_to_cart".into()]; + update_audit_quality(&mut report); + let Err(err) = crate::audit::normalize_checked(&report) else { + panic!("contradiction must be rejected"); + }; + assert!(err + .violations + .iter() + .any(|v| v.code == "measurement_profile_read_only" + || v.code == "measurement_profile_journeys")); + } + + #[test] + fn declared_state_enters_the_profile_by_identity_not_by_value() { + use clap::Parser; + let mut config = crate::audit::PipelineConfig::from_args_and_config( + &crate::cli::Args::parse_from(["auditmysite", "https://example.com"]), + None, + ); + config.state_id = Some("logged-in".into()); + config.storage_state = Some( + crate::browser::StorageState::from_bytes( + br#"{"cookies":[{"name":"sid","value":"TOPSECRET"}],"origins":[]}"#, + ) + .unwrap(), + ); + let scope = audit_scope_from_config(&config); + let json = serde_json::to_string(&scope).unwrap(); + assert!(json.contains("logged-in") && json.contains("\"cookies\":1")); + assert!(!json.contains("TOPSECRET"), "{json}"); + let without = audit_scope_from_config(&crate::audit::PipelineConfig { + storage_state: None, + state_id: None, + ..config.clone() + }); + assert_ne!( + serde_json::to_string(&without).unwrap(), + json, + "another state is another measurement" + ); + assert_ne!( + config.audit_signature(), + crate::audit::PipelineConfig { + storage_state: None, + ..config.clone() + } + .audit_signature() + ); + } + + #[test] + fn an_errored_rule_is_counted_once() { + let mut report = report_with_run("probe"); + report + .accessibility + .wcag_results + .rule_outcomes + .push(crate::wcag::RuleRun::not_run( + "r", + crate::wcag::NotRun::Errored, + )); + ensure_requested_module_runs(&mut report); + update_audit_quality(&mut report); + let quality = &report.accessibility.execution.quality; + assert_eq!(quality.failed_rule_checks, 1); + assert_eq!(quality.partial_or_failed_modules, 0); + } + + #[test] + fn a_complete_claim_over_incomplete_execution_is_rejected() { + use crate::wcag::{NotRun, RuleRun}; + for run in [ + RuleRun::not_run("r", NotRun::Errored), + RuleRun::not_run("r", NotRun::CapabilityMissing), + ] { + let mut report = report_with_run("accessibility"); + report.accessibility.wcag_results.rule_outcomes.push(run); + update_audit_quality(&mut report); + // Tamper: claim completeness the execution data contradicts. + report.accessibility.execution.quality.status = + crate::audit::AuditQualityStatus::Complete; + report.accessibility.execution.quality.qualified_results = false; + let Err(err) = crate::audit::normalize_checked(&report) else { + panic!("contradiction must be rejected"); + }; + assert!(err + .violations + .iter() + .any(|v| v.code == "execution_completeness")); + } + } + + #[test] + fn a_requested_rule_without_outcome_is_a_gap() { + let mut report = report_with_run("accessibility"); + report.accessibility.execution.scope.requested_rules = vec!["lang/missing".into()]; + update_audit_quality(&mut report); + let Err(err) = crate::audit::normalize_checked(&report) else { + panic!("contradiction must be rejected"); + }; + assert!(err + .violations + .iter() + .any(|v| v.code == "requested_rule_outcome")); + + report + .accessibility + .wcag_results + .rule_outcomes + .push(crate::wcag::RuleRun::ran("lang/missing", 0)); + update_audit_quality(&mut report); + crate::audit::normalize_checked(&report).unwrap(); + } + #[test] fn foundation_merge_preserves_concrete_rule_identity() { let finding = |rule: &str, node: i64| { diff --git a/src/audit/pipeline/mod.rs b/src/audit/pipeline/mod.rs index 0dd896c6..ad0ba501 100644 --- a/src/audit/pipeline/mod.rs +++ b/src/audit/pipeline/mod.rs @@ -372,6 +372,14 @@ pub struct PipelineConfig { pub interactive: crate::cli::InteractiveMode, /// Wall-clock budget for the interactive phase per URL (milliseconds). pub journey_budget_ms: u64, + /// Session policy of the browser the pages run in; part of the + /// measurement profile (#787). + pub session: crate::browser::SessionPolicy, + /// Declared browser state every page starts from (#787). Its values are + /// never reported; the profile carries digest and counts. + pub storage_state: Option, + /// Name of that state, stored in the measurement profile. + pub state_id: Option, /// Report locale ("de" / "en") — used for i18n stopword loading. pub lang: String, /// Display mode of the data-display convention to audit in (#653). @@ -428,13 +436,13 @@ impl PipelineConfig { // 20 for the display-mode scope and detection fields (#653). // 21 for the frame coverage in the execution record (#715). // 22 for the run context and the SEO indexability check (#780). - const CACHE_FMT: u8 = 22; + const CACHE_FMT: u8 = 24; let mut disabled = self.rule_filter.disabled_rules.clone(); disabled.sort(); let mut enabled_only = self.rule_filter.enabled_only_rules.clone(); enabled_only.sort(); format!( - "v={};fmt={};level={};perf={};seo={};sec={};mobile={};dark={};design_quality={};html_conform={};ai_transparency={};dns={};isolate_tp_impact={};ssr_content={};stack={};consent={};interactive={:?};journey_budget_ms={};lang={};disabled={};enabled_only={};exclude={};display={};indexable={:?}", + "v={};fmt={};level={};perf={};seo={};sec={};mobile={};dark={};design_quality={};html_conform={};ai_transparency={};dns={};isolate_tp_impact={};ssr_content={};stack={};consent={};interactive={:?};journey_budget_ms={};session={};state_id={};state={};lang={};disabled={};enabled_only={};exclude={};display={};indexable={:?}", env!("CARGO_PKG_VERSION"), CACHE_FMT, self.wcag_level, @@ -453,6 +461,9 @@ impl PipelineConfig { self.dismiss_consent as u8, self.interactive, self.journey_budget_ms, + self.session.as_str(), + self.state_id.as_deref().unwrap_or(""), + self.storage_state.as_ref().map_or("", |s| s.digest.as_str()), self.lang, disabled.join(","), enabled_only.join(","), @@ -569,6 +580,11 @@ impl PipelineConfig { .collect(), interactive: args.interactive, journey_budget_ms, + session: args.session, + storage_state: args + .load_storage_state() + .expect("--storage-state is checked by Args::validate"), + state_id: args.state_id.clone(), lang: args.lang.clone(), display_mode: args.display.and_then(|d| d.mode()), indexability: { @@ -839,6 +855,11 @@ pub async fn audit_page( // behind `display/init-missing`. Persists across both viewport passes. crate::display::prepare_page(page, config.display_mode).await; + // ── Pre-navigation: declared browser state (#787) ───────────────────────── + if let Some(state) = &config.storage_state { + state.apply(page, url).await; + } + // ── Desktop pass ────────────────────────────────────────────────────────── info!("Desktop pass starting for {}", url); set_viewport(page, Viewport::Desktop).await?; @@ -1969,6 +1990,7 @@ pub(crate) fn persist_artifacts( cached_at: report.timestamp, content_hash: hash.clone(), audit_signature: config.audit_signature(), + fingerprint_version: crate::audit::artifacts::FINGERPRINT_VERSION, }; // Persist the full report so cache hits render every module faithfully. // Screenshots reference temp files that won't exist on reload, so strip them. @@ -2155,6 +2177,9 @@ mod tests { dismiss_consent: false, exclude_selector: Vec::new(), interactive: crate::cli::InteractiveMode::Off, + session: Default::default(), + storage_state: None, + state_id: None, report_level: crate::cli::ReportLevel::Standard, lang: "de".to_string(), display: None, @@ -2335,6 +2360,9 @@ mod tests { exclude_selectors: Vec::new(), interactive: crate::cli::InteractiveMode::Off, journey_budget_ms: crate::a11y_journey::DEFAULT_BUDGET_MS, + session: Default::default(), + storage_state: None, + state_id: None, lang: "de".to_string(), display_mode: None, indexability: Default::default(), diff --git a/src/audit/report.rs b/src/audit/report.rs index 82929e70..ec3b58e3 100644 --- a/src/audit/report.rs +++ b/src/audit/report.rs @@ -103,7 +103,16 @@ pub struct AuditScope { /// Each carries a subcheck run in its module's `module_runs` (#788). #[serde(default, skip_serializing_if = "Vec::is_empty")] pub requested_subchecks: Vec, + /// Shared rules this run is expected to account for with an outcome — run, + /// disabled by filter or level, or errored — in at least one viewport + /// (#778). Empty for reports from before the field. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub requested_rules: Vec, pub full_audit: bool, + /// Measurement profile: under which session policy the pages ran and + /// which journeys may change page or account state (#787). + #[serde(default)] + pub measurement_profile: MeasurementProfile, pub interactive_mode: String, pub journey_budget_ms: u64, #[serde(default)] @@ -119,6 +128,57 @@ pub struct AuditScope { pub display_mode: crate::display::AuditedDisplayMode, } +/// Version of the measurement profile's meaning. A report of another +/// version is another contract, never the same measurement. +pub const MEASUREMENT_PROFILE_VERSION: u32 = 1; + +/// How the pages of a run were measured beyond the viewport (#787). +/// +/// `read_only` runs observe; a run with `state_changing_journeys` also +/// clicks controls that can submit a form or change a cart. The distinction +/// is stored with the result so it is never an invisible side effect. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MeasurementProfile { + pub version: u32, + /// Cookie/storage/cache sharing between the pages of the run: + /// `isolated` or `shared`. `unknown` for reports from before the field. + pub session_policy: String, + /// True when no journey that can change page or account state may run. + pub read_only: bool, + /// Journeys permitted to change state (submit clicks, cart triggers). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub state_changing_journeys: Vec, + /// Name of the declared state the run started from (`--state-id`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub state_id: Option, + /// The declared browser state by identity: digest and counts, never + /// values. `None` when every page started from an empty profile. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub storage_state: Option, +} + +/// Identity of a declared browser state (`--storage-state`). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct StorageStateRef { + /// SHA-256 of the state file, hex. + pub sha256: String, + pub cookies: usize, + pub origins: usize, +} + +impl Default for MeasurementProfile { + fn default() -> Self { + Self { + version: 0, + session_policy: "unknown".to_string(), + read_only: false, + state_changing_journeys: Vec::new(), + state_id: None, + storage_state: None, + } + } +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ViewportDefinition { pub name: String, @@ -763,7 +823,9 @@ pub struct BatchReport { #[serde(skip_serializing_if = "Option::is_none", default)] pub consistency: Option, /// How the audited URLs were discovered and sampled. Lets consumers tell a - /// representative sample apart from full domain coverage (issue #261). + /// partial selection apart from full domain coverage (issues #261, #791). + /// A selection is no evidence of representativeness: statements hold for + /// the successfully audited pages only. #[serde(skip_serializing_if = "Option::is_none", default)] pub sample: Option, /// Total execution time @@ -1080,6 +1142,113 @@ pub struct BatchSummary { pub root_causes: Vec, #[serde(default)] pub audit_quality: AuditQuality, + /// Spread and worst cases of the page scores next to the mean (#791). + /// `None` when no page produced a report. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub score_spread: Option, + /// What the successfully audited pages are: page types and languages + /// (#791). Describes the selection; it does not make it representative. + #[serde(default)] + pub selection_profile: SelectionProfile, +} + +/// Distribution of the page scores behind `average_score`. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ScoreSpread { + pub min: u32, + pub median: f64, + pub max: u32, + /// Population standard deviation of the page scores. + pub std_dev: f64, + /// Lowest-scoring pages, worst first (at most [`ScoreSpread::WORST_CASES`]). + pub worst_pages: Vec, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct WorstPage { + pub url: String, + pub score: u32, +} + +impl ScoreSpread { + pub const WORST_CASES: usize = 5; + + fn from_pages(pages: &[(String, u32)]) -> Option { + if pages.is_empty() { + return None; + } + let mut sorted: Vec<&(String, u32)> = pages.iter().collect(); + sorted.sort_by(|a, b| a.1.cmp(&b.1).then_with(|| a.0.cmp(&b.0))); + let n = sorted.len(); + let mean = sorted.iter().map(|p| p.1 as f64).sum::() / n as f64; + let variance = sorted + .iter() + .map(|p| (p.1 as f64 - mean).powi(2)) + .sum::() + / n as f64; + let median = if n % 2 == 1 { + sorted[n / 2].1 as f64 + } else { + (sorted[n / 2 - 1].1 + sorted[n / 2].1) as f64 / 2.0 + }; + Some(Self { + min: sorted[0].1, + median, + max: sorted[n - 1].1, + std_dev: variance.sqrt(), + worst_pages: sorted + .iter() + .take(Self::WORST_CASES) + .map(|p| WorstPage { + url: p.0.clone(), + score: p.1, + }) + .collect(), + }) + } +} + +/// Page types and languages of the pages that produced a report. Keys are +/// canonical (snake_case page type, lower-case primary language subtag), so +/// the JSON stays language-independent. Pages without the signal count under +/// `unclassified` / `language_unknown`. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SelectionProfile { + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub page_types: BTreeMap, + #[serde(default)] + pub unclassified: usize, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub languages: BTreeMap, + #[serde(default)] + pub language_unknown: usize, +} + +impl SelectionProfile { + fn from_reports(reports: &[AuditReport]) -> Self { + let mut profile = Self::default(); + for report in reports { + let seo = report.discoverability.seo.as_ref(); + match seo + .and_then(|s| s.content_profile.as_ref()) + .and_then(|p| serde_json::to_value(&p.page_classification.primary_type).ok()) + .and_then(|v| v.as_str().map(str::to_string)) + { + Some(kind) => *profile.page_types.entry(kind).or_default() += 1, + None => profile.unclassified += 1, + } + match seo + .and_then(|s| s.technical.lang.as_deref()) + .and_then(|l| l.split(['-', '_']).next()) + .map(|l| l.trim().to_ascii_lowercase()) + .filter(|l| !l.is_empty()) + { + Some(lang) => *profile.languages.entry(lang).or_default() += 1, + None => profile.language_unknown += 1, + } + } + profile + } } impl BatchSummary { @@ -1221,6 +1390,14 @@ impl BatchReport { 0.0 }; + let score_spread = ScoreSpread::from_pages( + &normalized_reports + .iter() + .map(|r| (r.url.clone(), r.score)) + .collect::>(), + ); + let selection_profile = SelectionProfile::from_reports(&reports); + let total_violations = normalized_reports .iter() .map(|r| r.occurrence_counts.total) @@ -1311,6 +1488,8 @@ impl BatchReport { decision: super::verdict::VerdictResult::undecided(), root_causes, audit_quality, + score_spread, + selection_profile, }, crawl_diagnostics: None, sitemap_diagnostics: None, @@ -1366,6 +1545,47 @@ mod tests { use super::*; use crate::wcag::WcagResults; + #[test] + fn score_spread_exposes_worst_cases_next_to_the_mean() { + let pages: Vec<(String, u32)> = [("a", 100), ("b", 100), ("c", 100), ("d", 20)] + .iter() + .map(|(u, s)| (u.to_string(), *s)) + .collect(); + let spread = ScoreSpread::from_pages(&pages).unwrap(); + assert_eq!((spread.min, spread.max), (20, 100)); + assert_eq!(spread.median, 100.0); + assert!(spread.std_dev > 30.0, "{}", spread.std_dev); + assert_eq!(spread.worst_pages[0].url, "d"); + assert!(ScoreSpread::from_pages(&[]).is_none()); + } + + #[test] + fn selection_profile_counts_unknown_instead_of_dropping_pages() { + let mut with_lang = AuditReport::new( + "https://e.com/a".into(), + WcagLevel::AA, + WcagResults::new(), + 1, + ); + with_lang.discoverability.seo = Some(crate::seo::SeoAnalysis { + technical: crate::seo::technical::TechnicalSeo { + lang: Some("de-AT".into()), + ..Default::default() + }, + ..Default::default() + }); + let bare = AuditReport::new( + "https://e.com/b".into(), + WcagLevel::AA, + WcagResults::new(), + 1, + ); + let profile = SelectionProfile::from_reports(&[with_lang, bare]); + assert_eq!(profile.languages.get("de"), Some(&1)); + assert_eq!(profile.language_unknown, 1); + assert_eq!(profile.unclassified, 2); + } + #[test] fn test_audit_report_new() { let results = WcagResults::new(); diff --git a/src/audit/scoring.rs b/src/audit/scoring.rs index 18241de5..e34e3df0 100644 --- a/src/audit/scoring.rs +++ b/src/audit/scoring.rs @@ -5,17 +5,8 @@ use serde::{Deserialize, Serialize}; use crate::taxonomy::{ criterion_for_rule, principle_for_criterion, RuleLookup, Scaling, ScoreImpact, WcagPrinciple, }; -use crate::wcag::coverage::automated_criteria; use crate::wcag::types::{Severity, Violation}; -/// Number of automatically-checked WCAG criteria belonging to a principle. -fn checked_criteria_count(principle: WcagPrinciple) -> usize { - automated_criteria() - .iter() - .filter(|(c, _)| principle_for_criterion(c) == Some(principle)) - .count() -} - /// Calculates accessibility scores and grades based on WCAG violations pub struct AccessibilityScorer; @@ -155,19 +146,32 @@ impl AccessibilityScorer { /// Missing, failed and review-only checks must never become passes merely /// because no violation was retained. This is not WCAG conformance. pub fn calculate_execution_coverage(results: &crate::wcag::WcagResults) -> PrincipleCoverage { - let mut criteria: BTreeMap = BTreeMap::new(); + // Per criterion: `Some(true)` ran and found nothing, `Some(false)` did + // not pass (finding, error, disabled, capability missing), `None` has + // nothing to check on this page and leaves the denominator (#778). + let mut criteria: BTreeMap> = BTreeMap::new(); for run in &results.rule_outcomes { let ids = if run.wcag.is_empty() { criterion_for_rule(&run.rule_id).into_iter().collect() } else { run.wcag.clone() }; + let state = match run.not_run { + None => Some(true), + Some(a11y_report::NotRun::NotApplicable) => None, + Some(_) => Some(false), + }; for id in ids { - let passed = run.not_run.is_none(); criteria .entry(id) - .and_modify(|p| *p &= passed) - .or_insert(passed); + .and_modify(|p| { + *p = match (*p, state) { + (Some(false), _) | (_, Some(false)) => Some(false), + (Some(true), _) | (_, Some(true)) => Some(true), + _ => None, + } + }) + .or_insert(state); } } for finding in results @@ -177,9 +181,13 @@ impl AccessibilityScorer { .chain(&results.not_testables) { if let Some(id) = criterion_for_rule(&finding.rule) { - criteria.insert(id, false); + criteria.insert(id, Some(false)); } } + let criteria: BTreeMap = criteria + .into_iter() + .filter_map(|(id, state)| state.map(|passed| (id, passed))) + .collect(); let ratio_for = |principle| { let entries: Vec<_> = criteria .iter() @@ -310,44 +318,6 @@ impl AccessibilityScorer { crate::registry::CERTIFICATE.label(score, false) } - /// Calculate WCAG principle coverage — a purely informative secondary - /// indicator that does *not* feed into the numeric score. - /// - /// For each of the four WCAG principles it reports how many of the - /// criteria the tool checks passed (had no violation). - pub fn calculate_coverage(violations: &[Violation]) -> PrincipleCoverage { - let failed: HashSet = violations - .iter() - .filter_map(|v| criterion_for_rule(&v.rule)) - .collect(); - - let ratio_for = |principle: WcagPrinciple| -> CoverageRatio { - let total = checked_criteria_count(principle) as u32; - let failed_in_principle = failed - .iter() - .filter(|c| principle_for_criterion(c) == Some(principle)) - .count() as u32; - let passed = total.saturating_sub(failed_in_principle); - CoverageRatio { - passed, - total, - ratio: if total == 0 { - 1.0 - } else { - let r = passed as f32 / total as f32; - (r * 10000.0).round() / 10000.0 - }, - } - }; - - PrincipleCoverage { - perceivable: ratio_for(WcagPrinciple::Perceivable), - operable: ratio_for(WcagPrinciple::Operable), - understandable: ratio_for(WcagPrinciple::Understandable), - robust: ratio_for(WcagPrinciple::Robust), - } - } - /// Calculate detailed statistics for a set of violations pub fn calculate_statistics(violations: &[Violation]) -> ViolationStatistics { let total = violations.len(); @@ -661,39 +631,50 @@ mod tests { assert!(apply_soft_floor(-60.0) < 1.0); } + fn run_results(runs: Vec) -> crate::wcag::WcagResults { + let mut results = crate::wcag::WcagResults::new(); + results.rule_outcomes = runs; + results + } + #[test] - fn test_coverage_all_pass_when_no_violations() { - let coverage = AccessibilityScorer::calculate_coverage(&[]); + fn test_coverage_of_an_empty_run_is_not_a_pass() { + let coverage = AccessibilityScorer::calculate_execution_coverage(&run_results(vec![])); for ratio in [ &coverage.perceivable, &coverage.operable, &coverage.understandable, &coverage.robust, ] { - assert_eq!(ratio.passed, ratio.total); - assert!(ratio.total > 0); - assert_eq!(ratio.ratio, 1.0); + assert_eq!((ratio.passed, ratio.total), (0, 0)); + assert_eq!(ratio.ratio, 0.0); } } #[test] - fn test_coverage_reflects_violations() { - use crate::cli::WcagLevel; - - let violations = vec![Violation::new( - "1.1.1", - "Non-text Content", - WcagLevel::A, + fn test_coverage_counts_only_what_ran() { + use crate::wcag::RuleRun; + use a11y_report::NotRun; + let mut results = run_results(vec![ + RuleRun::ran("a", 0).with_wcag(["1.1.1"]), + RuleRun::ran("b", 2).with_wcag(["1.4.3"]), + RuleRun::not_run("c", NotRun::CapabilityMissing).with_wcag(["1.4.11"]), + RuleRun::not_run("d", NotRun::Errored).with_wcag(["1.4.4"]), + RuleRun::not_run("e", NotRun::NotApplicable).with_wcag(["1.2.1"]), + ]); + results.violations.push(Violation::new( + "1.4.3", + "Contrast", + crate::cli::WcagLevel::AA, Severity::High, - "Missing alt", + "Low contrast", "n1", - )]; - let coverage = AccessibilityScorer::calculate_coverage(&violations); - // One Perceivable criterion failed. - assert_eq!(coverage.perceivable.passed, coverage.perceivable.total - 1); - // Other principles untouched. - assert_eq!(coverage.operable.passed, coverage.operable.total); - assert_eq!(coverage.robust.passed, coverage.robust.total); + )); + let coverage = AccessibilityScorer::calculate_execution_coverage(&results); + // 1.1.1 passed; 1.4.4 errored, 1.4.11 capability missing: not passed; + // 1.2.1 has nothing to check and leaves the denominator. + assert_eq!(coverage.perceivable.total, 4); + assert_eq!(coverage.perceivable.passed, 1); } #[test] diff --git a/src/audit/summary.rs b/src/audit/summary.rs index af59fb61..cadd591a 100644 --- a/src/audit/summary.rs +++ b/src/audit/summary.rs @@ -751,7 +751,7 @@ mod tests { interactive_high_issues: 0, summary: String::new(), }, - principle_coverage: crate::audit::scoring::AccessibilityScorer::calculate_coverage(&[]), + principle_coverage: Default::default(), } } diff --git a/src/audit/validation.rs b/src/audit/validation.rs index 3ca14e88..6b92a7e9 100644 --- a/src/audit/validation.rs +++ b/src/audit/validation.rs @@ -272,6 +272,43 @@ impl NormalizedReport { "each requested module needs exactly one explicit execution outcome".into(), ); } + // The measurement profile and the requested interaction depth tell + // one story: a read-only profile lists no state-changing journey, and + // a run that lists one says so (#787). + let profile = &self.execution.scope.measurement_profile; + if profile.version > 0 { + check( + profile.read_only == profile.state_changing_journeys.is_empty(), + "measurement_profile_read_only", + "execution.scope.measurement_profile".into(), + "read_only must match the listed state-changing journeys".into(), + ); + let expected = crate::cli::InteractiveMode::from_scope_name( + &self.execution.scope.interactive_mode, + ) + .map(crate::a11y_journey::state_changing_journeys); + if let Some(expected) = expected { + check( + profile.state_changing_journeys == expected, + "measurement_profile_journeys", + "execution.scope.measurement_profile".into(), + format!( + "interactive mode `{}` permits {expected:?}", + self.execution.scope.interactive_mode + ), + ); + } + } + // Every requested rule has an outcome; its absence is a gap, not a + // pass (#778). + for rule in &self.execution.scope.requested_rules { + check( + self.rule_outcomes.iter().any(|run| &run.rule_id == rule), + "requested_rule_outcome", + format!("rule_outcomes[{rule}]"), + "each requested rule needs an explicit execution outcome".into(), + ); + } // A module that lost a scoring collection was not fully measured: its // run is not `Completed` and its score neither claims a measurement // nor feeds the overall score (#777). @@ -498,6 +535,20 @@ impl NormalizedReport { Some(crate::wcag::NotRun::Errored | crate::wcag::NotRun::CapabilityMissing) ) }); + let errored = self + .rule_outcomes + .iter() + .filter(|run| crate::wcag::rule_run_errored(run)) + .count(); + check( + q.failed_rule_checks == errored, + "failed_rule_checks", + "execution.quality.failed_rule_checks".into(), + format!( + "quality counts {} failed rule checks, outcomes show {errored}", + q.failed_rule_checks + ), + ); check( !(incomplete || gaps.stability_fallbacks > 0 diff --git a/src/audit/verdict.rs b/src/audit/verdict.rs index 5a12f667..a0e18d83 100644 --- a/src/audit/verdict.rs +++ b/src/audit/verdict.rs @@ -571,6 +571,8 @@ mod tests { partial_or_failed_modules: 1, reasons: vec!["module_runs_incomplete".to_string()], }, + score_spread: None, + selection_profile: Default::default(), }; let result = compute_batch_verdict(&summary, &DecisionProfile::default()); diff --git a/src/browser/manager.rs b/src/browser/manager.rs index 9a1fcced..4002eca3 100644 --- a/src/browser/manager.rs +++ b/src/browser/manager.rs @@ -9,6 +9,10 @@ use std::sync::Arc; use std::time::Duration; use chromiumoxide::browser::{Browser, BrowserConfig, HeadlessMode}; +use chromiumoxide::cdp::browser_protocol::browser::BrowserContextId; +use chromiumoxide::cdp::browser_protocol::target::{ + CreateBrowserContextParams, CreateTargetParams, +}; use chromiumoxide::Page; use futures::StreamExt; use tokio::sync::Mutex; @@ -27,6 +31,44 @@ const PROFILE_PREFIX: &str = "auditmysite-chrome-"; /// Orphaned profiles are reaped once per process, before the first launch. static REAP_ORPHANS: std::sync::Once = std::sync::Once::new(); +/// Which cookies, storage and cache pages of one run share (#787). +/// +/// The choice is part of the measurement profile and is stored in the report: +/// the same URL measured under another policy is another measurement. +#[derive( + clap::ValueEnum, + Debug, + Clone, + Copy, + Default, + PartialEq, + Eq, + serde::Serialize, + serde::Deserialize, +)] +#[serde(rename_all = "snake_case")] +pub enum SessionPolicy { + /// Every URL runs in a browser context of its own: no cookies, storage + /// or cache survive from the pages audited before it, so order and + /// repetition cannot change a result. + #[default] + #[value(name = "isolated")] + Isolated, + /// All pages share one browser profile; a consent choice, login or cart + /// created by one page is seen by the next. + #[value(name = "shared")] + Shared, +} + +impl SessionPolicy { + pub fn as_str(self) -> &'static str { + match self { + Self::Isolated => "isolated", + Self::Shared => "shared", + } + } +} + /// Browser configuration options #[derive(Debug, Clone)] pub struct BrowserOptions { @@ -51,6 +93,8 @@ pub struct BrowserOptions { /// Report every page as focused and visible (CDP focus emulation). Only /// parallel batch pages need it; see [`BrowserManager::new_page`]. pub focus_emulation: bool, + /// Whether the pages of a pool share cookies and storage (#787). + pub session: SessionPolicy, } impl Default for BrowserOptions { @@ -66,6 +110,7 @@ impl Default for BrowserOptions { verbose: false, user_agent_override: None, focus_emulation: false, + session: SessionPolicy::default(), } } } @@ -336,17 +381,59 @@ impl BrowserManager { Ok(dir) } - /// Create a new page (tab) in the browser + /// The session policy this browser was launched with. + pub fn session_policy(&self) -> SessionPolicy { + self.options.session + } + + /// Create a new page (tab) in the browser's default context. pub async fn new_page(&self) -> Result { + self.new_page_in(None).await + } + + /// Create a page in a browser context of its own: no cookies, storage or + /// cache are shared with any other page (session policy `isolated`). + /// Dispose the context with [`Self::dispose_context`] when the page is + /// done. + pub async fn new_isolated_page(&self) -> Result<(Page, BrowserContextId)> { + let context = self + .browser + .create_browser_context(CreateBrowserContextParams::default()) + .await + .map_err(|e| AuditError::BrowserLaunchFailed { + reason: format!("Failed to create isolated browser context: {e}"), + })?; + match self.new_page_in(Some(context.clone())).await { + Ok(page) => Ok((page, context)), + Err(e) => { + self.dispose_context(context).await; + Err(e) + } + } + } + + /// Drop a context created by [`Self::new_isolated_page`] with everything + /// it stored. + pub async fn dispose_context(&self, context: BrowserContextId) { + if let Err(e) = self.browser.dispose_browser_context(context).await { + warn!("Failed to dispose browser context: {e}"); + } + } + + async fn new_page_in(&self, context: Option) -> Result { use chromiumoxide::cdp::browser_protocol::emulation::SetFocusEmulationEnabledParams; use chromiumoxide::cdp::browser_protocol::network::{Headers, SetExtraHttpHeadersParams}; use chromiumoxide::cdp::browser_protocol::page::AddScriptToEvaluateOnNewDocumentParams; - let page = self.browser.new_page("about:blank").await.map_err(|e| { - AuditError::BrowserLaunchFailed { - reason: format!("Failed to create new page: {}", e), - } - })?; + let mut target = CreateTargetParams::new("about:blank"); + target.browser_context_id = context; + let page = + self.browser + .new_page(target) + .await + .map_err(|e| AuditError::BrowserLaunchFailed { + reason: format!("Failed to create new page: {}", e), + })?; // Tabs of one window are hidden except the active one: a hidden page // never runs requestAnimationFrame and has no focus. In a batch that diff --git a/src/browser/mod.rs b/src/browser/mod.rs index 025b249a..3efa14a0 100644 --- a/src/browser/mod.rs +++ b/src/browser/mod.rs @@ -20,6 +20,7 @@ pub mod mock; mod pool; mod registry; pub mod resolver; +pub mod state; pub mod throttle; pub mod types; @@ -34,7 +35,8 @@ pub use types::{ // Legacy API (still used by main.rs and manager.rs) pub use detection::{detect_chrome, find_chrome, ChromeInfo}; -pub use manager::{BrowserManager, BrowserOptions, MetaRefresh, MetaRefreshOutcome}; +pub use manager::{BrowserManager, BrowserOptions, MetaRefresh, MetaRefreshOutcome, SessionPolicy}; pub(crate) use pool::PAGE_RETURN_BUDGET_SECS; pub use pool::{BrowserPool, PoolConfig, PoolStats, PooledPage}; +pub use state::StorageState; pub use throttle::ThrottleProfile; diff --git a/src/browser/pool.rs b/src/browser/pool.rs index c958467d..05b6de16 100644 --- a/src/browser/pool.rs +++ b/src/browser/pool.rs @@ -92,6 +92,14 @@ struct BrowserPoolInner { pages: Mutex>, /// Semaphore for limiting concurrent pages semaphore: Semaphore, + /// Browser contexts of isolated pages, by target id: disposed with the + /// page, so nothing a URL stored reaches the next one (#787). + contexts: Mutex< + std::collections::HashMap< + String, + chromiumoxide::cdp::browser_protocol::browser::BrowserContextId, + >, + >, /// Total pages created pages_created: AtomicUsize, /// Maximum pages allowed @@ -105,6 +113,12 @@ struct BrowserPoolInner { impl BrowserPoolInner { /// Return a page to the pool async fn return_page(&self, page: Page) { + // An isolated page is never reused: its tab and context go, the next + // URL gets a fresh one. + if self.browser.session_policy() == super::SessionPolicy::Isolated { + self.discard_page(page).await; + return; + } // Reset the page for reuse. Guard with a short timeout — some pages become // unresponsive after navigating to heavy sites and the goto call would hang // indefinitely, starving the semaphore. @@ -143,6 +157,11 @@ impl BrowserPoolInner { /// the bookkeeping happens either way so a hung close can't reintroduce /// the same exhaustion bug. async fn discard_page(&self, page: Page) { + let context = self + .contexts + .lock() + .await + .remove(AsRef::::as_ref(page.target_id())); match tokio::time::timeout(PAGE_CLOSE_TIMEOUT, page.close()).await { Ok(Ok(())) => {} Ok(Err(e)) => warn!("Failed to close discarded page: {}", e), @@ -151,6 +170,9 @@ impl BrowserPoolInner { PAGE_CLOSE_TIMEOUT.as_secs() ), } + if let Some(context) = context { + self.browser.dispose_context(context).await; + } self.pages_created.fetch_sub(1, Ordering::SeqCst); self.semaphore.add_permits(1); } @@ -180,6 +202,7 @@ impl BrowserPool { let inner = Arc::new(BrowserPoolInner { browser, pages: Mutex::new(Vec::with_capacity(config.max_pages)), + contexts: Mutex::new(Default::default()), semaphore: Semaphore::new(config.max_pages), pages_created: AtomicUsize::new(0), max_pages: config.max_pages, @@ -250,19 +273,30 @@ impl BrowserPool { // the rest of the run and later pages wait for a slot that never // frees up (#651). Bounded, because a Chrome saturated by heavy pages // can leave the page creation unanswered for good. - let created = tokio::time::timeout( - self.inner.page_create_timeout, - self.inner.browser.new_page(), - ) - .await - .unwrap_or_else(|_| { - Err(AuditError::BrowserLaunchFailed { - reason: format!( - "Failed to create new page: no answer after {} seconds", - self.inner.page_create_timeout.as_secs() - ), - }) - }); + let isolated = self.inner.browser.session_policy() == super::SessionPolicy::Isolated; + let creation = async { + if isolated { + let (page, context) = self.inner.browser.new_isolated_page().await?; + self.inner + .contexts + .lock() + .await + .insert(AsRef::::as_ref(page.target_id()).to_string(), context); + Ok(page) + } else { + self.inner.browser.new_page().await + } + }; + let created = tokio::time::timeout(self.inner.page_create_timeout, creation) + .await + .unwrap_or_else(|_| { + Err(AuditError::BrowserLaunchFailed { + reason: format!( + "Failed to create new page: no answer after {} seconds", + self.inner.page_create_timeout.as_secs() + ), + }) + }); let page = match created { Ok(page) => page, Err(e) => { diff --git a/src/browser/state.rs b/src/browser/state.rs new file mode 100644 index 00000000..deb03e3a --- /dev/null +++ b/src/browser/state.rs @@ -0,0 +1,205 @@ +//! Declared browser state (#787): cookies and localStorage a run starts +//! from, read from a Playwright-style `storage-state` JSON file. +//! +//! The state is part of the measurement: its digest and counts go into the +//! report's measurement profile, its values never do — the file may hold a +//! login. + +use std::fmt; +use std::path::Path; + +use chromiumoxide::cdp::browser_protocol::network::{CookieParam, SetCookiesParams}; +use chromiumoxide::cdp::browser_protocol::page::AddScriptToEvaluateOnNewDocumentParams; +use chromiumoxide::Page; +use serde::Deserialize; +use sha2::{Digest, Sha256}; +use tracing::warn; + +use crate::error::{AuditError, Result}; + +#[derive(Deserialize)] +struct StateFile { + #[serde(default)] + cookies: Vec, + #[serde(default)] + origins: Vec, +} + +#[derive(Deserialize)] +struct StateCookie { + name: String, + value: String, + #[serde(default)] + domain: Option, + #[serde(default)] + path: Option, + #[serde(default)] + secure: Option, + #[serde(default, rename = "httpOnly")] + http_only: Option, +} + +#[derive(Deserialize)] +struct StateOrigin { + origin: String, + #[serde(default, rename = "localStorage")] + local_storage: Vec, +} + +#[derive(Deserialize)] +struct StateItem { + name: String, + value: String, +} + +/// A loaded state. `Debug` shows the identity, never a value. +#[derive(Clone)] +pub struct StorageState { + file: std::sync::Arc, + /// SHA-256 of the file bytes, hex. + pub digest: String, +} + +impl fmt::Debug for StorageState { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("StorageState") + .field("digest", &self.digest) + .field("cookies", &self.cookie_count()) + .field("origins", &self.origin_count()) + .finish() + } +} + +impl StorageState { + pub fn load(path: &Path) -> Result { + let bytes = std::fs::read(path).map_err(|e| { + AuditError::ConfigError(format!("cannot read storage state {}: {e}", path.display())) + })?; + Self::from_bytes(&bytes) + } + + pub fn from_bytes(bytes: &[u8]) -> Result { + let file: StateFile = serde_json::from_slice(bytes).map_err(|e| { + // The message names the position, not the content. + AuditError::ConfigError(format!( + "storage state is not valid (expected cookies[] and origins[]): line {}, column {}", + e.line(), + e.column() + )) + })?; + Ok(Self { + file: std::sync::Arc::new(file), + digest: format!("{:x}", Sha256::digest(bytes)), + }) + } + + pub fn cookie_count(&self) -> usize { + self.file.cookies.len() + } + + pub fn origin_count(&self) -> usize { + self.file.origins.len() + } + + /// Script run on every new document: writes the declared localStorage + /// entries of the document's own origin. + fn storage_script(&self) -> Option { + let by_origin: Vec = self + .file + .origins + .iter() + .filter(|o| !o.local_storage.is_empty()) + .map(|o| { + serde_json::json!({ + "origin": o.origin, + "items": o.local_storage.iter() + .map(|i| [i.name.as_str(), i.value.as_str()]) + .collect::>(), + }) + }) + .collect(); + if by_origin.is_empty() { + return None; + } + Some(format!( + "(function () {{ try {{ var all = {}; for (var i = 0; i < all.length; i++) {{ \ + if (all[i].origin !== location.origin) continue; \ + for (var j = 0; j < all[i].items.length; j++) {{ \ + localStorage.setItem(all[i].items[j][0], all[i].items[j][1]); }} }} }} \ + catch (e) {{}} }})();", + serde_json::Value::Array(by_origin) + )) + } + + /// Before navigation: set the declared cookies and register the storage + /// script. A cookie the browser rejects is logged by name only. + pub async fn apply(&self, page: &Page, url: &str) { + let cookies: Vec = self + .file + .cookies + .iter() + .map(|c| { + let mut p = CookieParam::new(c.name.clone(), c.value.clone()); + match &c.domain { + Some(domain) => p.domain = Some(domain.clone()), + None => p.url = Some(url.to_string()), + } + p.path = Some(c.path.clone().unwrap_or_else(|| "/".to_string())); + p.secure = c.secure; + p.http_only = c.http_only; + p + }) + .collect(); + if !cookies.is_empty() { + if let Err(e) = page.execute(SetCookiesParams::new(cookies)).await { + warn!("Declared cookies were not set: {e}"); + } + } + if let Some(script) = self.storage_script() { + if let Err(e) = page + .execute(AddScriptToEvaluateOnNewDocumentParams::new(script)) + .await + { + warn!("Declared localStorage was not registered: {e}"); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const FILE: &str = r#"{"cookies":[{"name":"sid","value":"SECRET","domain":"example.com"}], + "origins":[{"origin":"https://example.com","localStorage":[{"name":"k","value":"v"}]}]}"#; + + #[test] + fn debug_and_errors_never_show_values() { + let state = StorageState::from_bytes(FILE.as_bytes()).unwrap(); + assert_eq!((state.cookie_count(), state.origin_count()), (1, 1)); + assert!(!format!("{state:?}").contains("SECRET")); + let err = StorageState::from_bytes(b"{\"cookies\": [SECRET").unwrap_err(); + assert!(!err.to_string().contains("SECRET")); + } + + #[test] + fn digest_follows_the_bytes() { + let a = StorageState::from_bytes(FILE.as_bytes()).unwrap(); + let b = StorageState::from_bytes(FILE.replace("\"v\"", "\"w\"").as_bytes()).unwrap(); + assert_ne!(a.digest, b.digest); + assert_eq!( + a.digest, + StorageState::from_bytes(FILE.as_bytes()).unwrap().digest + ); + } + + #[test] + fn storage_script_is_bound_to_the_declared_origin() { + let state = StorageState::from_bytes(FILE.as_bytes()).unwrap(); + let script = state.storage_script().unwrap(); + assert!(script.contains("https://example.com") && script.contains("location.origin")); + let cookies_only = + StorageState::from_bytes(br#"{"cookies":[{"name":"a","value":"b"}]}"#).unwrap(); + assert!(cookies_only.storage_script().is_none()); + } +} diff --git a/src/cli/args.rs b/src/cli/args.rs index 0debd4b4..e2dae420 100644 --- a/src/cli/args.rs +++ b/src/cli/args.rs @@ -265,13 +265,42 @@ pub struct Args { #[arg(long = "expect-indexable", value_name = "BOOL")] pub expect_indexable: Option, + /// Which cookies, storage and cache the audited pages share. + /// + /// `isolated` (default) — every URL runs in a browser context of its own, + /// so audit order and repetition cannot change a result. + /// `shared` — all pages use one profile: a consent choice, login or cart + /// created by one page reaches the next. Stored in the report's + /// measurement profile. + #[arg(long, value_enum, default_value = "isolated")] + pub session: crate::browser::SessionPolicy, + + /// Start every page from a declared browser state: a Playwright-style + /// `storage-state` JSON (`cookies[]`, `origins[].localStorage[]`), for a + /// reproducible login, consent or SPA state. + /// + /// Digest and counts — never values — are stored in the report's + /// measurement profile. The file may hold credentials: keep it out of + /// version control. + #[arg(long = "storage-state", value_name = "FILE")] + pub storage_state: Option, + + /// Name of the declared state (for example `logged-in`, `consent-all`), + /// stored in the measurement profile so a result names the state it was + /// measured in. + #[arg(long = "state-id", value_name = "ID")] + pub state_id: Option, + /// Run the Accessibility-Journey-Layer for interactive checks /// (tab walk, modal focus trap, skip-link verification, …). /// /// `off` — no interactive phase, fastest. /// `basic` — tab walk + skip link + obvious pattern journeys. /// `full` (default) — adds SPA navigation, form-error announcement, - /// link-text inventory and outline export. + /// link-text inventory and outline export. Also clicks submit + /// buttons and add-to-cart triggers, which can change page or + /// account state: listed in the report's measurement profile + /// (`state_changing_journeys`). Use `basic` for a read-only run. /// /// Adds a few seconds per URL; recommended for single-URL audits. #[arg(long, value_enum, default_value = "full")] @@ -560,6 +589,17 @@ pub enum InteractiveMode { } impl InteractiveMode { + /// The mode named in `audit_scope.interactive_mode`; `None` for a name + /// this version does not know. + pub fn from_scope_name(name: &str) -> Option { + match name { + "off" => Some(Self::Off), + "basic" => Some(Self::Basic), + "full" => Some(Self::Full), + _ => None, + } + } + pub fn is_enabled(self) -> bool { !matches!(self, InteractiveMode::Off) } @@ -770,11 +810,20 @@ impl Args { } /// Validate arguments + /// The declared browser state, if `--storage-state` names one. + pub fn load_storage_state(&self) -> Result, String> { + self.storage_state + .as_deref() + .map(|path| crate::browser::StorageState::load(path).map_err(|e| e.to_string())) + .transpose() + } + pub fn validate(&self) -> Result<(), String> { // Subcommands don't need URL validation if self.command.is_some() { return Ok(()); } + self.load_storage_state()?; // At least one input source required (unless --detect-chrome) if !self.detect_chrome @@ -1024,6 +1073,9 @@ mod tests { dismiss_consent: false, exclude_selector: Vec::new(), interactive: InteractiveMode::Off, + session: Default::default(), + storage_state: None, + state_id: None, display: None, report_level: ReportLevel::Standard, lang: "de".to_string(), diff --git a/src/cli/runners.rs b/src/cli/runners.rs index e31486cb..462312ac 100644 --- a/src/cli/runners.rs +++ b/src/cli/runners.rs @@ -191,6 +191,9 @@ async fn run_single(args: &Args, config: &Option) -> R .then(|| BOT_USER_AGENT.to_string()), // One page, the active tab (see `BrowserManager::new_page`). focus_emulation: false, + // A single run starts a browser with a fresh profile: nothing else + // shares it. + session: args.session, }; if !args.quiet { diff --git a/src/lib.rs b/src/lib.rs index e49b2f42..aa5e9e83 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -52,6 +52,9 @@ //! exclude_selectors: Vec::new(), //! interactive: auditmysite::cli::InteractiveMode::Off, //! journey_budget_ms: auditmysite::a11y_journey::DEFAULT_BUDGET_MS, +//! session: Default::default(), +//! storage_state: None, +//! state_id: None, //! lang: "de".to_string(), //! display_mode: None, //! indexability: Default::default(), diff --git a/src/output/json.rs b/src/output/json.rs index 6bfccb76..6dfcee49 100644 --- a/src/output/json.rs +++ b/src/output/json.rs @@ -623,6 +623,15 @@ pub struct BatchContentProfile { pub top_issues: Vec, } +/// How complete `detail.modules` is. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ModuleDetailState { + Full, + Omitted, + NotRestored, +} + /// Per-page module detail blob — single reports only. #[derive(Debug, Serialize)] pub struct PageDetail { @@ -645,6 +654,12 @@ pub struct PageDetail { /// (#581). Single reports only. #[serde(skip_serializing_if = "Vec::is_empty")] pub security_category_scores: Vec, + /// What `modules` holds: collected data (`full`), nothing because the + /// projection is compact by design (`omitted`), or nothing because the + /// run is a cache replay without raw module data (`not_restored`). An + /// empty `modules` is never to be read as "the modules found nothing" + /// unless this says `full` (#790). + pub module_detail: ModuleDetailState, pub modules: ModuleBlob, #[serde(skip_serializing_if = "Vec::is_empty")] pub confidence_summary: Vec, diff --git a/src/output/json/detail.rs b/src/output/json/detail.rs index 4512db8c..86a9bd4b 100644 --- a/src/output/json/detail.rs +++ b/src/output/json/detail.rs @@ -87,6 +87,7 @@ pub(super) fn build_batch_detail(normalized: &NormalizedReport) -> PageDetail { // Batch detail stays compact (CLAUDE.md "Report Intent"). accessibility_subcategory_scores: Vec::new(), security_category_scores: Vec::new(), + module_detail: ModuleDetailState::Omitted, modules: ModuleBlob::default(), confidence_summary: Vec::new(), capabilities: Vec::new(), @@ -110,6 +111,7 @@ pub(super) fn build_detail_cached( bik_guide: build_bik_guide_annex(normalized, &[], &[], false), accessibility_subcategory_scores: normalized.accessibility_subcategory_scores.clone(), security_category_scores: normalized.security_category_scores.clone(), + module_detail: ModuleDetailState::NotRestored, modules: ModuleBlob::default(), confidence_summary: Vec::new(), capabilities: Vec::new(), @@ -360,6 +362,7 @@ pub(super) fn build_detail(ctx: &AuditContext<'_>, detail_ctx: DetailContext) -> ), accessibility_subcategory_scores: normalized.accessibility_subcategory_scores.clone(), security_category_scores: normalized.security_category_scores.clone(), + module_detail: ModuleDetailState::Full, modules, confidence_summary: vm .methodology diff --git a/src/output/json/tests.rs b/src/output/json/tests.rs index d17dee90..a7aa19e8 100644 --- a/src/output/json/tests.rs +++ b/src/output/json/tests.rs @@ -1329,3 +1329,39 @@ fn test_batch_certificate_is_risk_gated() { ); assert_eq!(certificate, "EINGESCHRÄNKT"); } + +#[test] +fn empty_module_detail_says_why_it_is_empty() { + let report = AuditReport::new( + "https://example.com".to_string(), + WcagLevel::AA, + WcagResults::new(), + 500, + ); + let normalized = normalize(&report); + let state = |json: String| -> String { + let value: serde_json::Value = serde_json::from_str(&json).unwrap(); + value["pages"][0]["detail"]["module_detail"] + .as_str() + .unwrap() + .to_string() + }; + // A fresh run collected its modules; a cache replay has no raw data and + // must not look like modules that found nothing (#790). + assert_eq!( + state( + UnifiedReport::single(&normalized, &report) + .to_json(false) + .unwrap() + ), + "full" + ); + assert_eq!( + state(format_json_cached(&normalized.normalized, false).unwrap()), + "not_restored" + ); + let batch = BatchReport::from_reports(vec![report], vec![], 1); + let value: serde_json::Value = + serde_json::from_str(&format_json_batch(&batch, false).unwrap()).unwrap(); + assert_eq!(value["pages"][0]["detail"]["module_detail"], "omitted"); +} diff --git a/src/output/pdf/wcag_coverage.rs b/src/output/pdf/wcag_coverage.rs index 5926a499..4225ea78 100644 --- a/src/output/pdf/wcag_coverage.rs +++ b/src/output/pdf/wcag_coverage.rs @@ -26,11 +26,11 @@ pub(super) fn render_wcag_coverage_section( // Principle coverage — informative secondary indicator (#99). let coverage = - AccessibilityScorer::calculate_coverage(&report.accessibility.wcag_results.violations); + AccessibilityScorer::calculate_execution_coverage(&report.accessibility.wcag_results); let cov_title = if en { - "Principle coverage (criteria passed)" + "Principle coverage (executed criteria passed)" } else { - "Prinzip-Abdeckung (bestandene Kriterien)" + "Prinzip-Abdeckung (ausgeführte Kriterien bestanden)" }; let fmt_ratio = |r: &CoverageRatio| format!("{}/{} ({:.0} %)", r.passed, r.total, r.ratio * 100.0); diff --git a/src/tech_stack/module.rs b/src/tech_stack/module.rs index 31841e86..beeafa11 100644 --- a/src/tech_stack/module.rs +++ b/src/tech_stack/module.rs @@ -70,6 +70,9 @@ mod tests { exclude_selectors: Vec::new(), interactive: InteractiveMode::Off, journey_budget_ms: crate::a11y_journey::DEFAULT_BUDGET_MS, + session: Default::default(), + storage_state: None, + state_id: None, lang: "de".to_string(), display_mode: None, indexability: Default::default(), diff --git a/tests/architecture_test.rs b/tests/architecture_test.rs new file mode 100644 index 00000000..ffce7606 --- /dev/null +++ b/tests/architecture_test.rs @@ -0,0 +1,182 @@ +//! Direction of dependencies (#790, plan 96): analysis modules do not reach +//! up into `cli`, `output` or the orchestration of `audit`. +//! +//! Existing backward imports are a baseline that may only shrink +//! (`tests/fixtures/architecture/backward_imports.txt`, `path count`). A new +//! one — in a new file or beyond a file's count — fails this test, as does a +//! baseline entry that no longer exists, so the list stays honest. + +use std::collections::BTreeMap; +use std::fs; +use std::path::{Path, PathBuf}; + +/// Modules that collect or evaluate; they sit below `audit` orchestration and +/// `output`. +const ANALYSIS_MODULES: &[&str] = &[ + "a11y_journey", + "accessibility", + "ai_visibility", + "best_practices", + "commerce", + "content_visibility", + "dark_mode", + "design_quality", + "journey", + "mobile", + "network", + "patterns", + "performance", + "screen_reader", + "security", + "seo", + "tech_stack", + "ux", + "wcag", +]; + +fn rust_files(dir: &Path, out: &mut Vec) { + for entry in fs::read_dir(dir).unwrap().flatten() { + let path = entry.path(); + if path.is_dir() { + rust_files(&path, out); + } else if path.extension().is_some_and(|e| e == "rs") { + out.push(path); + } + } +} + +/// Code lines (not comments) containing any of `needles`, per file. +fn count_imports(needles: &[&str]) -> BTreeMap { + let root = Path::new(env!("CARGO_MANIFEST_DIR")); + let mut counts = BTreeMap::new(); + for module in ANALYSIS_MODULES { + let mut files = Vec::new(); + let dir = root.join("src").join(module); + if dir.is_dir() { + rust_files(&dir, &mut files); + } + for file in files { + let n = fs::read_to_string(&file) + .unwrap() + .lines() + .filter(|l| !l.trim_start().starts_with("//")) + .filter(|l| needles.iter().any(|n| l.contains(n))) + .count(); + if n > 0 { + let rel = file + .strip_prefix(root) + .unwrap() + .to_string_lossy() + .to_string(); + counts.insert(rel, n); + } + } + } + counts +} + +fn baseline() -> BTreeMap { + let path = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures/architecture/backward_imports.txt"); + fs::read_to_string(path) + .unwrap() + .lines() + .filter(|l| !l.trim().is_empty() && !l.starts_with('#')) + .map(|l| { + let (path, n) = l.rsplit_once(' ').expect("`path count`"); + (path.to_string(), n.parse().expect("count")) + }) + .collect() +} + +/// Differences between the imports found and the baseline. +fn ratchet_problems( + actual: &BTreeMap, + allowed: &BTreeMap, +) -> Vec { + let mut problems = Vec::new(); + for (file, n) in actual { + match allowed.get(file) { + None => problems.push(format!("{file}: {n} new import(s) of cli/output")), + Some(max) if n > max => { + problems.push(format!("{file}: {n} imports of cli/output, baseline {max}")) + } + Some(max) if n < max => problems.push(format!( + "{file}: now {n}, baseline {max} — lower the baseline" + )), + _ => {} + } + } + for file in allowed.keys().filter(|f| !actual.contains_key(*f)) { + problems.push(format!( + "{file}: no import left — remove it from the baseline" + )); + } + problems +} + +#[test] +fn analysis_modules_do_not_gain_imports_of_cli_or_output() { + let problems = ratchet_problems( + &count_imports(&["crate::cli", "crate::output"]), + &baseline(), + ); + assert!( + problems.is_empty(), + "dependency direction (plan 96):\n{}", + problems.join("\n") + ); +} + +#[test] +fn the_ratchet_rejects_new_and_grown_imports_and_stale_entries() { + let map = |items: &[(&str, usize)]| -> BTreeMap { + items.iter().map(|(k, v)| (k.to_string(), *v)).collect() + }; + let allowed = map(&[("src/seo/a.rs", 1), ("src/seo/gone.rs", 1)]); + assert!(ratchet_problems( + &map(&[("src/seo/a.rs", 1), ("src/seo/gone.rs", 1)]), + &allowed + ) + .is_empty()); + // A new file importing cli/output. + assert_eq!( + ratchet_problems( + &map(&[ + ("src/seo/a.rs", 1), + ("src/seo/gone.rs", 1), + ("src/ux/new.rs", 1) + ]), + &allowed + ) + .len(), + 1 + ); + // A file importing more than its baseline. + assert_eq!( + ratchet_problems( + &map(&[("src/seo/a.rs", 2), ("src/seo/gone.rs", 1)]), + &allowed + ) + .len(), + 1 + ); + // A baseline entry nothing imports any more. + assert_eq!( + ratchet_problems(&map(&[("src/seo/a.rs", 1)]), &allowed).len(), + 1 + ); +} + +#[test] +fn analysis_modules_do_not_import_orchestration_or_persistence() { + let found = count_imports(&[ + "crate::audit::pipeline", + "crate::audit::batch", + "crate::audit::artifacts", + ]); + assert!( + found.is_empty(), + "analysis modules must not import audit orchestration: {found:?}" + ); +} diff --git a/tests/easy_language_detection_test.rs b/tests/easy_language_detection_test.rs index 79b180d3..28da7f1b 100644 --- a/tests/easy_language_detection_test.rs +++ b/tests/easy_language_detection_test.rs @@ -68,6 +68,9 @@ fn default_config() -> PipelineConfig { exclude_selectors: Vec::new(), interactive: auditmysite::cli::InteractiveMode::Off, journey_budget_ms: auditmysite::a11y_journey::DEFAULT_BUDGET_MS, + session: Default::default(), + storage_state: None, + state_id: None, lang: "de".to_string(), display_mode: None, indexability: Default::default(), diff --git a/tests/fixtures/architecture/backward_imports.txt b/tests/fixtures/architecture/backward_imports.txt new file mode 100644 index 00000000..4b99fb9e --- /dev/null +++ b/tests/fixtures/architecture/backward_imports.txt @@ -0,0 +1,32 @@ +src/a11y_journey/mod.rs 1 +src/ai_visibility/mod.rs 1 +src/content_visibility/mod.rs 1 +src/dark_mode/mod.rs 2 +src/seo/headings.rs 1 +src/seo/topics.rs 1 +src/tech_stack/module.rs 1 +src/wcag/engine.rs 1 +src/wcag/rules/accessible_authentication.rs 1 +src/wcag/rules/accessible_name.rs 1 +src/wcag/rules/display_modes.rs 1 +src/wcag/rules/focus_not_obscured_enhanced.rs 1 +src/wcag/rules/focus_not_obscured_minimum.rs 1 +src/wcag/rules/focus_visible.rs 1 +src/wcag/rules/html_content_model.rs 1 +src/wcag/rules/iframe_rules.rs 3 +src/wcag/rules/keyboard.rs 1 +src/wcag/rules/media_alternative.rs 1 +src/wcag/rules/media_rules.rs 3 +src/wcag/rules/modern_attributes.rs 3 +src/wcag/rules/on_input.rs 3 +src/wcag/rules/orientation.rs 1 +src/wcag/rules/page_rules.rs 1 +src/wcag/rules/page_titled.rs 3 +src/wcag/rules/pointer_cancellation.rs 1 +src/wcag/rules/reflow.rs 1 +src/wcag/rules/text_alternatives.rs 1 +src/wcag/rules/text_spacing.rs 3 +src/wcag/rules/timing_adjustable.rs 1 +src/wcag/rules/visual_presentation.rs 1 +src/wcag/shared.rs 1 +src/wcag/types.rs 1 diff --git a/tests/integration_test.rs b/tests/integration_test.rs index 14b1be70..14fb8425 100644 --- a/tests/integration_test.rs +++ b/tests/integration_test.rs @@ -405,6 +405,9 @@ fn default_config() -> PipelineConfig { exclude_selectors: Vec::new(), interactive: auditmysite::cli::InteractiveMode::Off, journey_budget_ms: auditmysite::a11y_journey::DEFAULT_BUDGET_MS, + session: Default::default(), + storage_state: None, + state_id: None, lang: "de".to_string(), display_mode: None, indexability: Default::default(), diff --git a/tests/session_policy_test.rs b/tests/session_policy_test.rs new file mode 100644 index 00000000..ea677f42 --- /dev/null +++ b/tests/session_policy_test.rs @@ -0,0 +1,133 @@ +//! Session policy (#787): pages of one run share cookies and storage only +//! when the measurement profile says so. Local fixture only — the page +//! writes a cookie and a localStorage entry and reports what it found +//! before doing so. +//! +//! cargo test --test session_policy_test -- --ignored + +mod common; + +use auditmysite::browser::{BrowserOptions, BrowserPool, PoolConfig, SessionPolicy}; + +const PAGE: &str = "State

State

\ +"; + +/// What the n-th page of a one-slot pool found when it loaded. +async fn seen_by_consecutive_pages(session: SessionPolicy, pages: usize) -> Vec { + let fixture = common::fixture_server::serve_html(PAGE.to_string()); + let pool = BrowserPool::new(PoolConfig { + max_pages: 1, + browser_options: BrowserOptions { + no_sandbox: std::env::var("CI").is_ok(), + session, + ..Default::default() + }, + ..Default::default() + }) + .await + .expect("pool"); + let mut seen = Vec::new(); + for _ in 0..pages { + let pooled = pool.acquire().await.expect("page"); + let page = pooled.page().expect("page"); + pool.browser() + .navigate(page, &fixture.url) + .await + .expect("navigate"); + let value: String = page + .evaluate("JSON.stringify(window.__seen)") + .await + .expect("evaluate") + .into_value() + .expect("value"); + seen.push(value); + drop(pooled); + // The page goes back to the pool (or is discarded) asynchronously. + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + } + pool.close().await.ok(); + fixture.stop(); + seen +} + +#[tokio::test] +#[ignore = "needs Chrome"] +async fn isolated_pages_do_not_see_what_the_page_before_stored() { + let seen = seen_by_consecutive_pages(SessionPolicy::Isolated, 2).await; + assert_eq!(seen[0], seen[1], "order must not change the measurement"); + assert!( + !seen[1].contains("k=1") && !seen[1].contains("\"1\""), + "{seen:?}" + ); +} + +#[tokio::test] +#[ignore = "needs Chrome"] +async fn shared_pages_carry_state_to_the_next_page() { + let seen = seen_by_consecutive_pages(SessionPolicy::Shared, 2).await; + assert_ne!(seen[0], seen[1]); + assert!(seen[1].contains("k=1"), "{seen:?}"); +} + +/// What a page finds at load when the run starts from `state`, then what the +/// next page finds when it starts from `next` (a login/consent switch). +async fn seen_with_states( + first: Option<&auditmysite::browser::StorageState>, + next: Option<&auditmysite::browser::StorageState>, +) -> Vec { + let fixture = common::fixture_server::serve_html(PAGE.to_string()); + let pool = BrowserPool::new(PoolConfig { + max_pages: 1, + browser_options: BrowserOptions { + no_sandbox: std::env::var("CI").is_ok(), + ..Default::default() + }, + ..Default::default() + }) + .await + .expect("pool"); + let mut seen = Vec::new(); + for state in [first, next] { + let pooled = pool.acquire().await.expect("page"); + let page = pooled.page().expect("page"); + if let Some(state) = state { + state.apply(page, &fixture.url).await; + } + pool.browser() + .navigate(page, &fixture.url) + .await + .expect("navigate"); + seen.push( + page.evaluate("JSON.stringify(window.__seen)") + .await + .expect("evaluate") + .into_value::() + .expect("value"), + ); + drop(pooled); + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + } + pool.close().await.ok(); + fixture.stop(); + seen +} + +#[tokio::test] +#[ignore = "needs Chrome"] +async fn declared_state_is_present_on_each_page_and_does_not_leak_to_the_next() { + // The fixture server's port is not known before it runs, so the state + // names the host and path scope and the origin is matched at runtime + // through a cookie only; localStorage is checked for the origin-bound + // script separately in the unit tests. + let state = auditmysite::browser::StorageState::from_bytes( + br#"{"cookies":[{"name":"login","value":"user-a","domain":"127.0.0.1","path":"/"}]}"#, + ) + .unwrap(); + let seen = seen_with_states(Some(&state), None).await; + assert!(seen[0].contains("login=user-a"), "{seen:?}"); + assert!( + !seen[1].contains("login"), + "a login switch must not carry the previous state: {seen:?}" + ); +}