Skip to content

feat(reporting): bind reports and calls to benchmark editions - #64

Merged
lanyue-llk merged 5 commits into
mainfrom
codex/benchmark-release-bindings
Oct 23, 2026
Merged

lanyue-llk merged 5 commits into
mainfrom
codex/benchmark-release-bindings

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

An existing report must retain its benchmark edition when the installed task package changes. This PR carries the selected edition through the current certified runner, report builder, checker, packer and publisher.

  • Verify the selected edition before provider calls, retaining the original call edition in execution metadata.
  • Preserve its file and path/hash binding through ordinary builds and rebuilds. The report manifest also records the original binding so deletion or substitution is detected after rescoring selects a newer edition.
  • Run the shared edition validator before every packaging early return, including frozen presentation and snapshot modes. Presentation validation uses the same check.
  • Link the edition beside the home-page task scope and keep the surrounding text translatable in both languages.
  • Pin the exact LexBench commit and catalog manifest hash. Only LexBench creates benchmark-vMAJOR.MINOR.PATCH tags. Browser Eval uses full commits; report rebuilds use hosting version IDs.

Validation: full local make ci passed, including 6,430 Python tests, 2,912 subtests, UI/native checks, fixture execution and installed-wheel/package checks. Boundary scenarios cover ordinary build → check → rebuild with distinct call/scoring editions, original-file tampering and binding deletion, and missing/changed sidecars, dropped bindings and wrong labels at ordinary and frozen package/publication boundaries. Existing no-edition reports remain supported.

The real 430-task, 2,580-result report passes the current checker without new calls or changed scores. Its benchmark source is benchmark-v1.0.0, with unchanged report access. The separate 550-task catalog is benchmark-v2.0.1; the patch changes version validation, not tasks or scorer rules.

Based on main after #56 merged. Companion: lexbench-webfetch #10. Main-content acceptance work remains separate in lexbench-webfetch #8.

Keep the README citation to one sentence and a BibTeX entry in both languages. Move dependency and method maintenance instructions to CONTRIBUTING, and align CITATION.cff with Apache-2.0.

@lanyue-llk
lanyue-llk changed the base branch from codex/notion-webfetch-fixes-current to main October 23, 2026 08:02
@lanyue-llk
lanyue-llk force-pushed the codex/benchmark-release-bindings branch from 8d6684b to 69829ce Compare October 23, 2026 08:02

@lanyue-llk lanyue-llk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR carries the selected benchmark edition through the existing runner and report pipeline so package upgrades can preserve historical measurement identities. The separate report_version module and external profile binding are a reasonable reusable design. I found no new report-instance-specific branch or copied renderer; additional object-oriented structure is unnecessary here.

Two reproduced gaps need fixing before merge: ordinary builds drop the original-call edition's execution binding, and the package/publication boundaries do not consistently enforce the new edition check. Both can be fixed in the existing shared build and validation owners.

Validation at 4221ea4: 172 relevant report, certification, dependency and packaging tests passed, 1 skipped, and 2 subtests passed. The local HTTP smoke test passed after installing the declared reference-audit dependencies. Additional negative controls reproduced both findings. In the packaging control, changing only the release version to 99.0.0 made check_release reject the report, but report_package.build and publication_check.check both succeeded. The companion lexbench-webfetch #10 also needs its scorer-input fingerprint coverage fixed.

Comment thread harness/web_fetch/report_version.py
Comment thread harness/web_fetch/report_version.py
@lanyue-llk
lanyue-llk merged commit 809c6fa into main Oct 23, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant