Skip to content

feat(reporting): identify report builds and prepare v0.2.0 - #66

Merged
lanyue-llk merged 4 commits into
mainfrom
codex/tag-citation-cleanup
Oct 23, 2026
Merged

lanyue-llk merged 4 commits into
mainfrom
codex/tag-citation-cleanup

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Reports need to identify both the software that built them and the benchmark edition that owns their tasks and scoring. This change prepares Browser Eval 0.2.0 and records those identities separately.

  • Add one shared build identity for Web Fetch, Browser Use and trajectory replay, including the software version, full checkout commit and a source hash covering Python, runtime scripts, workers, catalogs, templates and configuration. Bytecode caches are excluded. Installed packages without a checkout retain the version and source hash.
  • Show compact Browser Eval and bench version links together when a benchmark edition is selected. Check their JSON bindings and labels through the existing report gates. Rebuilding preserves the selected edition and saved results.
  • Record the Web Fetch execution version separately from its report builder. Pin the complete bench release commit and updated catalog hash.
  • Reduce both READMEs to purpose, runnable entry points, outputs and documentation links. Describe the framework as extensible to Web Search and other workflows; use Agent browser use for the agent-driven scope. Update citation metadata to Apache-2.0.

Scenario coverage includes newer builders with unchanged benchmark inputs, both languages, repeated builds, installed packages outside Git, and missing/changed version files, labels and bindings. A rebuild of the real 430-task, 2,580-result report passes its existing checker with byte-identical dataset and result files and zero new calls.

Companion benchmark tag naming: lexmount/lexbench-webfetch#11.

Validation: final make ci passed 6,441 Python tests and 2,912 subtests, plus UI/native checks, fixture execution and installed-wheel checks. Version links honor the HTML document base in ordinary, bilingual and tokenized-preview layouts. Python 3.12–3.14 base CLI CI is green.

Full CI passed at 30645af1ba2520b4d089cbef32979e088cffe277.

Latest review fix at 76c2c6b0: 174 targeted tests passed across all three report owners. Ten runtime-asset mutation controls fail against the previous implementation and pass after the fix; cache creation preserves the identity. Lint and the complete installed-wheel package check passed. Current CI.

@lanyue-llk lanyue-llk changed the title docs: simplify citations and keep benchmark tags upstream feat(reporting): identify report builds and prepare v0.2.0 Oct 23, 2026

@lanyue-llk lanyue-llk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared build-identity module is a sensible extension of the existing report owners, and I found no new report-instance-specific branch or copied renderer. The earlier benchmark-fingerprint, original-call binding and package-validation fixes are present in this revision and covered by the relevant tests.

One reproducible provenance gap remains: the new software source hash omits packaged JavaScript, workers and translation catalogs. Installed builds can therefore have different behavior while recording identical software identities. The inline comment gives the reproduction and a small shared fix.

Validation at 30645af: 374 relevant Web Fetch, Browser Use, replay, language, dependency and packaging tests passed; 1 skipped and 2 subtests passed. The offline fixture smoke test also passed. I separately checked the installed-package identity with changed non-Python assets and a Python-file control.

Comment thread harness/reporting/versions.py Outdated
@lanyue-llk
lanyue-llk merged commit 13d17de into main Oct 23, 2026
7 checks passed
lanyue-llk added a commit that referenced this pull request Oct 23, 2026
* docs: simplify citations and keep benchmark tags upstream

* feat(reporting): identify report builds and prepare v0.2.0

* fix(reporting): resolve version links against each document base

* fix(reporting): include runtime assets in software fingerprints
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant