Repository navigation
feat(versioning): freeze benchmark editions and compare contracts - #10
Conversation
98dd342 to
04c2b99
Compare
lanyue-llk
left a comment
There was a problem hiding this comment.
This PR separates benchmark editions from package versions and adds contract-based compatibility checks. The shared versioning module and data-driven catalog are appropriate; I found no new report-instance-specific renderer or execution branch, and adding classes would not address the defect below.
One correctness issue needs fixing before merge: the compatibility fingerprint omits inputs consumed by the public scorer. A reproduced reference change flips a saved answer from pass to fail while compare-releases still returns direct.
Validation at 56aca0c: make check passed 2,574 tests and 321 subtests. I also exercised create_release, compare_releases, verify_release and evaluate_response with a changed ordered-tree reference; that negative control exposes the missing coverage.
Reports need a readable benchmark edition that stays tied to the actual tasks, references and scorer when the task package changes. This PR freezes that contract, compares editions by their fingerprints, and gives the installed catalog a verifiable release manifest.
release,compare-releasesandversionto the existing CLI. Keep Python package versions separate from benchmark editions.benchmark-vMAJOR.MINOR.PATCHtags only in LexBench. Browser Eval uses full commits; reports name their benchmark edition and hosted build ID.The current catalog is benchmark-v2.0.1. Its 550 tasks, reference bindings and scorer are unchanged from 2.0.0; the patch fixes compatibility checking. The historical 430-task report remains on edition 1.0.0.
Validation:
make checkpassed 2,579 tests and 321 subtests. The new controls execute actual scorer verdict flips in both directions for tree references and thresholds, plus a list-name rule. Each requires a minor edition, rejects relaxed verification against the old edition, and selects rescoring. The real frozen 430-task edition still verifies unchanged. CI passed.Based on main after #7 merged. Companion execution/report binding: browser-eval #64.
Simplify both README citation sections to one sentence and BibTeX, and correct CITATION.cff to Apache-2.0. Retain the historical 1.0.0 benchmark source and current 2.0.1 catalog; remove temporary and superseded tags.