Skip to content

fix(scoring): verify saved responses and revision evidence - #7

Open
lanyue-llk wants to merge 21 commits into
mainfrom
codex/notion-scoring-fixes-current
Open

lanyue-llk wants to merge 21 commits into
mainfrom
codex/notion-scoring-fixes-current

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem and result

Live and saved responses could receive different verdicts, and a task revision could name regression controls without executing them. This PR gives Browser Eval and other callers the same deterministic scoring and revision-export contract.

Task scoring, retained evidence and reusable reports

Changes

  • Use the public scoring orchestrator and task-result predicate for live and saved responses. Keep binary acceptance separate from text-quality diagnostics.
  • Reject incomplete body facts, word fragments, unrelated destinations, fields combined across separate records and invalid file evidence. Accept equivalent link and Markdown serialization when the required content is present. Resource aliases allow only declared asset-preserving parameters.
  • Preserve frozen task definitions and historical dataset identities. Explicit revision exports verify the parent, changed leaf fields, independent source hashes and executable positive/negative controls before producing a new identity. inspect-revision exposes those checks.
  • Verify retained reference answers against the same scorer and reject accepted error pages. Missing original page bodies remain visible in inspection; a hash alone does not establish complete source coverage.
  • Keep revision JSON and scoring-regression JSON as installed, executable benchmark inputs. Report instances, provider outputs and publication targets stay outside this package.
  • Document contribution and title conventions, and update the shared banner in both README editions.
  • Use Apache 2.0 for project-owned code and definitions. Both source and wheel distributions ship the canonical license, LexMount copyright notice and third-party notices.
  • Check English maintainer identifiers, comments and docstrings in the installed package, while preserving original-language task evidence.

The branch includes current main's article acceptance contracts. Expanded main-content references and unified mixed requirements remain in draft #8. Execution, retained evidence and all three report pipelines are handled by Browser Eval #56.

Validation

  • make check: 2,541 tests and 321 subtests passed after integrating current main.
  • Python 3.11–3.14 CI passes at 6bd263ed; the PR has no merge conflict.
  • Revision inspection executes registered controls and rejects missing review records, incorrect field differences, unbound sources, empty reasons, one-direction controls and failing checks. Seven of ten historical corrections have executable retained-source controls; the other three retain source hashes without the original page body.
  • The harness pins this exact commit. Its retained real-data report instances cover Web Fetch (3,024 results), Browser Use (180) and Browser Trajectory Replay (450).

Historical report scope

The published 430-task report retains its frozen snapshot and scorer binding. Its complete comparison against the audited e1b6efc1 definitions contains 286 matching contracts, 91 retrieval-hint-only differences and 53 other differences. Original/revised-definition pass counts are 378/360/229/223/217/190 and 385/367/233/227/222/194 respectively. LZNOIP's eight canonical English navigation aliases are preserved by the report's bound revision. These saved results are not recomputed when this package's source identity changes.

The full scope and source-copy audit are attached to the Notion review. The existing report keeps its normal version presentation and permissions.

Scoring and partial reruns

Scoring corrections and partial reruns

@lanyue-llk lanyue-llk changed the title Fix acceptance isolation and explicit page identity contracts fix(scoring): require body facts and recompute stale verdicts Oct 20, 2026
@lanyue-llk lanyue-llk changed the title fix(scoring): require body facts and recompute stale verdicts fix(scoring): isolate answer evidence and enforce task requirements Oct 20, 2026
@lanyue-llk lanyue-llk changed the title fix(scoring): isolate answer evidence and enforce task requirements fix(scoring): verify saved responses and revision evidence Oct 22, 2026
Comment thread src/lexbench_webfetch/definitions/annotation-revisions/reviews.json
Comment thread README.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant