Skip to content

feat(versioning): freeze benchmark editions and compare contracts - #10

Merged
lanyue-llk merged 4 commits into
mainfrom
codex/benchmark-release-versions
Oct 23, 2026
Merged

lanyue-llk merged 4 commits into
mainfrom
codex/benchmark-release-versions

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Reports need a readable benchmark edition that stays tied to the actual tasks, references and scorer when the task package changes. This PR freezes that contract, compares editions by their fingerprints, and gives the installed catalog a verifiable release manifest.

  • Add release, compare-releases and version to the existing CLI. Keep Python package versions separate from benchmark editions.
  • Require a major edition for changed populations, requests, output scope or routes. Corrected scoring/reference rules require unchanged inputs and common-edition rescoring; saved outputs can be reused.
  • Bind full scorer commits and source hashes. Include ordered-tree reference hashes, tree thresholds and all supported list identity, naming, scope and price controls in the shared public task projection.
  • Create benchmark-vMAJOR.MINOR.PATCH tags only in LexBench. Browser Eval uses full commits; reports name their benchmark edition and hosted build ID.

The current catalog is benchmark-v2.0.1. Its 550 tasks, reference bindings and scorer are unchanged from 2.0.0; the patch fixes compatibility checking. The historical 430-task report remains on edition 1.0.0.

Validation: make check passed 2,579 tests and 321 subtests. The new controls execute actual scorer verdict flips in both directions for tree references and thresholds, plus a list-name rule. Each requires a minor edition, rejects relaxed verification against the old edition, and selects rescoring. The real frozen 430-task edition still verifies unchanged. CI passed.

Based on main after #7 merged. Companion execution/report binding: browser-eval #64.

Simplify both README citation sections to one sentence and BibTeX, and correct CITATION.cff to Apache-2.0. Retain the historical 1.0.0 benchmark source and current 2.0.1 catalog; remove temporary and superseded tags.

@lanyue-llk
lanyue-llk changed the base branch from codex/notion-scoring-fixes-current to main October 23, 2026 07:58
@lanyue-llk
lanyue-llk force-pushed the codex/benchmark-release-versions branch from 98dd342 to 04c2b99 Compare October 23, 2026 07:59

@lanyue-llk lanyue-llk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR separates benchmark editions from package versions and adds contract-based compatibility checks. The shared versioning module and data-driven catalog are appropriate; I found no new report-instance-specific renderer or execution branch, and adding classes would not address the defect below.

One correctness issue needs fixing before merge: the compatibility fingerprint omits inputs consumed by the public scorer. A reproduced reference change flips a saved answer from pass to fail while compare-releases still returns direct.

Validation at 56aca0c: make check passed 2,574 tests and 321 subtests. I also exercised create_release, compare_releases, verify_release and evaluate_response with a changed ordered-tree reference; that negative control exposes the missing coverage.

Comment thread src/lexbench_webfetch/versions.py
@lanyue-llk
lanyue-llk merged commit 291c619 into main Oct 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant