[lanyue-llk]

Summary

Extract the Web Search benchmark from Browser Eval into an independently installable package. This repository owns the 149 scenario queries, the pinned factual retrieval probes, provenance and source qualifications, URL/answer rules, the four-grade annotation rubric, and deterministic ranking metrics. Browser Eval retains providers, collection, model/human annotation orchestration, latency/billing evidence, audits, and reports.

Retain the original task fields and historical input hashes. Add explicit attribution for every case, distinguish linkable community originals from unattributed screenshot transcriptions, and preserve upstream MIT notices. The factual set is documented as a retrieval adaptation, not an official SimpleQA Verified score.

Provide English documentation, citation metadata, offline inspection/export/scoring commands, locked dependencies, and CI with an installed-wheel test outside the checkout. This package has no runtime dependencies and makes no provider or model calls.

Validation

• 52 behavior tests pass in the checkout and from an installed wheel outside both repositories.
• Dataset hash/tamper checks, complete source coverage, grade boundaries, missing/duplicate results, regression controls, CLI export and scoring are covered.
• Lint, formatting, sdist and wheel builds pass.
• Browser Eval integration tests verify that resolving the split inputs reproduces the exact original collection-profile hashes.

Companion integration: lexmount/browser-eval#58

[lanyue-llk]

Review: repository initialization and Web Search extraction (with lexmount/browser-eval#58)

This PR sets up lexbench-websearch and moves the Web Search datasets and deterministic scoring out of Browser Eval. lexmount/browser-eval#58 is the consumer and pins this PR's head commit. I reviewed both together and posted a separate review there. There are 20 inline findings: 5 major, 15 minor.

What is solid

• Setup and packaging:
  • uv lock --check is clean, and lint, format and pytest pass (52 passed).
  • sdist and wheel both build, and the wheel ships every file under definitions/.
  • The CI step that tests the installed wheel outside the checkout passes. The matrix really uses each Python version, because setup-uv sets UV_PYTHON.
  • CITATION.cff is valid against schema 1.2.0.
• Extraction fidelity:
  • Old Browser Eval code and this package were compared on about 4,000 synthetic cases across normalize, evaluate, factual_evidence, factual_sensitivity (all rules), task_acceptance (all metrics, cutoffs 3/5/10), classify, match_reference_answer, reference_rule over all 831 answers, query_metrics, candidate_packet, and the regression/revision functions. There were no differences.
  • rebuild_dataset reproduces factual-v1.json byte for byte from the pinned SimpleQA CSV, so the earlier "echo-only builder" finding is fixed.
  • The provenance.json source hashes reproduce.
• Consumer boundary: Browser Eval imports only public functions from this package (scoring, relevance, revisions, evidence_quality, inputs, contracts, rubric).

Major

1. The current reference set is split between hashed and unhashed files (reference-additions.json). The 30 qualified anchors in reference-additions.json and the 4 aliases in identity-correction.json aren't part of load_dataset('scenarios-v1') and aren't covered by registry.json. They only apply when saved rows are rescored, so a fresh run scores against an older reference set than the published one. Reproduced: applying reference-additions.json under reference_scope: identity produces cases that validate_cases rejects, and nothing runs that check.
2. The report-correction workflow is in the benchmark package (revisions.py:211). Rescoring measured rows (elapsed_ms, cost) and approving report-prose revisions by hash is report workflow that Browser Eval owns. browser-eval#58 never reads these packaged revision files, so the copies are unused and will drift.
3. The regression controls bypass the pass/fail function (revisions.py:33). The controls call match_reference_answer/evaluate/is_benchmark_page instead of task_acceptance/classify. Reproduced: removing the benchmark-mirror exclusion from factual acceptance leaves every control and test green. lexbench-webfetch runs its controls through its public scoring path. Ten of 26 one-line mutations to acceptance branches also go undetected.
4. provenance.json cites an orphaned commit. Source commit 6eee6d18 can be fetched by SHA, but no browser-eval branch or PR contains it; it is a pre-rebase copy. The execution-*.json paths exist only in open browser-eval#58. Re-point it after browser-eval#58 lands, or cite a reachable commit.
5. SimpleQA constants are still in code (simpleqa.py:9). The source SHA/URL, the 128-character limit and the 1000/831 counts are literals rather than a dataset definition.

Minor (selected)

• Licensing: NOTICE.txt lacks OpenAI's original SimpleQA MIT copyright line ("Copyright (c) 2024 OpenAI"). The SimpleQA Verified card itself has no holder.
• Fixed defaults:
  • The contamination rule hard-codes simple qa|super gpqa.
  • Factual cutoffs default to 5, and evaluate always uses [:5].
  • The judge rubric requires Chinese reasons.
  • implementation_sources builds Browser Eval's receipt layout into the package.
• Duplication and API: the re-evaluation loop is duplicated in revisions.py. definition_path/digest differ from lexbench-webfetch's API.
• Repository files and CI:
  • MANIFEST.in leaves AGENTS/CONTRIBUTING/SECURITY/uv.lock/Makefile out of the sdist; the sibling repo ships them.
  • CI has no permissions block, doesn't pin the uv version, and never runs the installed CLI.
  • CITATION.cff encodes LexMount as a person's surname and doesn't reference the upstream datasets.
• Dataset provenance and balance: 28 screenshot-transcribed queries cite an image nobody else can access. 13 of 149 scenario queries name DeepSeek, one of the measured providers, and three are anchored to its own docs.

Merge order

Merge this PR first. Then browser-eval#58 should re-pin to this repo's main, since it currently pins the unmerged 2711961, and extend its pin tooling to cover this package.
🤖 Generated with Claude Code

[lanyue-llk]

[major] Provenance source commit is an orphaned browser-eval commit; materialized paths exist only in an unmerged PR

source_commit 6eee6d18ef4ae2e456fa168f0d2e6da6ad4c10b2 can be fetched by SHA, and the source_files hashes do reproduce from it. However, git branch -r --contains finds it on no remote branch and the GitHub API reports no associated pull request. It looks like a pre-rebase copy of c4931943 ('Record the shared-publisher Search report release'), which is the commit browser-eval PR #58 actually builds on. Unreferenced commits can be garbage-collected, and then this provenance cannot be checked. The materialized_profiles paths (evaluation/web-search/execution-primary.json, execution-factual.json) exist in neither that commit nor browser-eval main; they appear only in open PR #58. Point source_commit at a commit reachable from browser-eval main (after PR #58 merges), and state the browser-eval revision that contains the materialized profiles.

[lanyue-llk]

[major] Current reference set is split between the hashed dataset and unhashed post-hoc revision files

All 30 cases listed in reference-additions.json still have no anchors in load_dataset('scenarios-v1'). The four identity aliases in identity-correction.json are also absent from the dataset cases. These qualified references are not covered by registry.json and are never applied by load_dataset or resolve_profile. They take effect only when apply_report_revision / apply_identity_correction rescore saved rows. A harness that follows docs/integration.md and loads scenarios-v1 for a fresh run therefore scores against a different, older reference set than the published revisions, while the package presents scenarios-v1 as the frozen benchmark. Fold the qualified anchors and aliases into a new hash-bound dataset version (e.g. scenarios-v2), or bind these files in the registry and apply them in the loader, so that one dataset ID means one acceptance set.

[lanyue-llk]

[major] Scoring-regression controls bypass the task predicate they are meant to protect

validate_regressions runs each control against a low-level helper: answer calls match_reference_answer, anchor reads evaluate()["anchor_hit_5"] and benchmark_page calls is_benchmark_page. None of them goes through task_acceptance/classify, the predicate that decides pass or fail. Mutation check: replacing factual_sensitivity with plain factual_evidence in task_acceptance (scoring.py line 238) stops excluding benchmark-mirror pages from factual acceptance, yet every regression control and all 52 tests still pass. lexbench-webfetch runs its controls through the public scoring entry point and task predicate. Express each control as a case plus results with an expected task_acceptance verdict, and keep the helper-level checks as extra assertions only.

[lanyue-llk]

[major] Report-correction machinery and data copies duplicate Browser Eval's authority

apply_report_revision/apply_identity_correction take measured rows (with elapsed_ms, cost, metrics) and return before/after rescoring receipts. validate_revision_binding approves 'reader prose' revisions by SHA-256 through assessment-revisions.json. That is report and result-correction workflow, which the README and CONTRIBUTING assign to Browser Eval. Browser Eval PR #58 does not read the packaged reference-additions.json or identity-correction.json. It reads the revision from external authored report content (authored['assessment_revision']) and the identity correction from a --identity-correction CLI path. The packaged copies are therefore unused second copies that can drift from what reports apply. Keep the provider-independent pieces here (equivalence/qualification predicates and regression controls). Move result-rescoring, receipt and approval logic plus the report-specific approval list to Browser Eval, or make Browser Eval load these files from the package as the single source.

[lanyue-llk]

[major] SimpleQA population constants are still embedded in code

Partly fixed: the five required method IDs are gone, and rebuild_dataset now derives the cases from the pinned CSV. Still open: the source revision, SHA and URL (lines 9-11), the <= 128 selection rule (line 49), the 1000/831 counts (line 59), the factual-v1 identity, the license and the scope text (lines 63-70) are all literals. A SimpleQA revision, a different query limit or another public factual dataset still needs a code change or a copy of this module. Move the source binding, selection rule and expected counts into a definition file such as a factual-v1 source manifest, and keep only a generic CSV-to-cases transform here.

[lanyue-llk]

[minor] The re-evaluation loop is still duplicated, and annotation revisions are never validated

apply_report_revision (lines 166-174) and apply_identity_correction (lines 196-204) still repeat the same re-evaluate-and-diff loop, and only the identity path calls validate_cases. Repro: apply the shipped reference-additions.json to scenarios-v1 under the consumer's reference_scope: identity. The revision adds 30 anchors without reference_checks, and validate_cases(..., reference_scope='identity') then raises 'Identity references require nonempty publisher assertions'. Nothing runs that check today, so the revised profile is used silently. Extract one helper that validates and re-evaluates for both paths, and decide whether qualified annotation sources satisfy the identity-assertion contract or are explicitly exempt; a naive validate call would reject the approved revision.

[lanyue-llk]

[minor] Assessment-revision authority: schema and provenance are still unchecked

Fixed: a positive test now accepts an approved revision and rejects a changed one, so an always-reject mutation fails the suite. Still open: validate_revision_binding never checks the authority's schema == "search-assessment-authority/1", and it never compares the authority's provenance entries (root_cause, regression_ids, provider_calls, derived_from_sha256) with the approved revision. The test uses a monkeypatched authority with no schema or provenance, and no test binds the shipped reference-additions.json to the shipped authority (it does hash to the second approved entry today). Validate the schema and require a matching provenance entry for every approved hash, and add a test over the shipped pair.

[lanyue-llk]

[minor] The frozen judge rubric requires Chinese reasons

JUDGE_PROMPT asks for a "reason":"brief explanation in Chinese". That language choice belongs to one report's audience, not to a provider-independent benchmark rubric, and English-language consumers would get reasons in the wrong language. Changing it later also changes the frozen prompt identity. Make the reason language a parameter of the consumer's judge request, or require a language-neutral reason, and version the rubric when it changes.

[lanyue-llk]

[minor] The factual-dataset rebuild has no entry point and no reproduction check

rebuild_dataset is reachable only from Python: the CLI offers list/validate/export/score, and the tests run it only on synthetic CSVs. Rebuilding from the pinned publisher CSV currently reproduces factual-v1.json byte for byte (registry SHA b3422267...), but nothing in the repository demonstrates or protects that. Add a lexbench-websearch rebuild factual-v1 --csv <file> command that compares the output with the registry hash, and document it in docs/data.md.

[lanyue-llk]

[minor] Key acceptance branches have no test that fails when they change

Mutation testing over 26 single-line changes left 10 survivors in the package's own suite. The survivors were: site acceptance ignoring the cutoff (scoring.py line 226), quality_excluded ignored in task_acceptance (line 199), relevance acceptance lowered from grade >=2 to >=1 (line 205), relevance success lowered from grade 3 to >=2 (relevance.py line 82), the factual contamination exclusion removed from acceptance (scoring.py line 238), _qualified accepting non-HTTPS anchors (revisions.py line 122), identity corrections no longer requiring the original target (line 188), and validate_cases no longer checking alias targets (contracts.py line 23). task_acceptance is tested only through the empty/API-error classify case and one anchor scenario. Add direct pass/fail pairs for each primary metric (answer, source, site, relevance) at cutoffs 5 and 10, and for quality exclusion.