Repository navigation
feat(reporting): bind reports and calls to benchmark editions - #64
Conversation
8d6684b to
69829ce
Compare
lanyue-llk
left a comment
There was a problem hiding this comment.
This PR carries the selected benchmark edition through the existing runner and report pipeline so package upgrades can preserve historical measurement identities. The separate report_version module and external profile binding are a reasonable reusable design. I found no new report-instance-specific branch or copied renderer; additional object-oriented structure is unnecessary here.
Two reproduced gaps need fixing before merge: ordinary builds drop the original-call edition's execution binding, and the package/publication boundaries do not consistently enforce the new edition check. Both can be fixed in the existing shared build and validation owners.
Validation at 4221ea4: 172 relevant report, certification, dependency and packaging tests passed, 1 skipped, and 2 subtests passed. The local HTTP smoke test passed after installing the declared reference-audit dependencies. Additional negative controls reproduced both findings. In the packaging control, changing only the release version to 99.0.0 made check_release reject the report, but report_package.build and publication_check.check both succeeded. The companion lexbench-webfetch #10 also needs its scorer-input fingerprint coverage fixed.
An existing report must retain its benchmark edition when the installed task package changes. This PR carries the selected edition through the current certified runner, report builder, checker, packer and publisher.
benchmark-vMAJOR.MINOR.PATCHtags. Browser Eval uses full commits; report rebuilds use hosting version IDs.Validation: full local
make cipassed, including 6,430 Python tests, 2,912 subtests, UI/native checks, fixture execution and installed-wheel/package checks. Boundary scenarios cover ordinary build → check → rebuild with distinct call/scoring editions, original-file tampering and binding deletion, and missing/changed sidecars, dropped bindings and wrong labels at ordinary and frozen package/publication boundaries. Existing no-edition reports remain supported.The real 430-task, 2,580-result report passes the current checker without new calls or changed scores. Its benchmark source is benchmark-v1.0.0, with unchanged report access. The separate 550-task catalog is benchmark-v2.0.1; the patch changes version validation, not tasks or scorer rules.
Based on main after #56 merged. Companion: lexbench-webfetch #10. Main-content acceptance work remains separate in lexbench-webfetch #8.
Keep the README citation to one sentence and a BibTeX entry in both languages. Move dependency and method maintenance instructions to CONTRIBUTING, and align CITATION.cff with Apache-2.0.