Repository navigation
feat(reporting): identify report builds and prepare v0.2.0 - #66
Conversation
lanyue-llk
left a comment
There was a problem hiding this comment.
The shared build-identity module is a sensible extension of the existing report owners, and I found no new report-instance-specific branch or copied renderer. The earlier benchmark-fingerprint, original-call binding and package-validation fixes are present in this revision and covered by the relevant tests.
One reproducible provenance gap remains: the new software source hash omits packaged JavaScript, workers and translation catalogs. Installed builds can therefore have different behavior while recording identical software identities. The inline comment gives the reproduction and a small shared fix.
Validation at 30645af: 374 relevant Web Fetch, Browser Use, replay, language, dependency and packaging tests passed; 1 skipped and 2 subtests passed. The offline fixture smoke test also passed. I separately checked the installed-package identity with changed non-Python assets and a Python-file control.
* docs: simplify citations and keep benchmark tags upstream * feat(reporting): identify report builds and prepare v0.2.0 * fix(reporting): resolve version links against each document base * fix(reporting): include runtime assets in software fingerprints
Reports need to identify both the software that built them and the benchmark edition that owns their tasks and scoring. This change prepares Browser Eval 0.2.0 and records those identities separately.
Scenario coverage includes newer builders with unchanged benchmark inputs, both languages, repeated builds, installed packages outside Git, and missing/changed version files, labels and bindings. A rebuild of the real 430-task, 2,580-result report passes its existing checker with byte-identical dataset and result files and zero new calls.
Companion benchmark tag naming: lexmount/lexbench-webfetch#11.
Validation: final
make cipassed 6,441 Python tests and 2,912 subtests, plus UI/native checks, fixture execution and installed-wheel checks. Version links honor the HTML document base in ordinary, bilingual and tokenized-preview layouts. Python 3.12–3.14 base CLI CI is green.Full CI passed at
30645af1ba2520b4d089cbef32979e088cffe277.Latest review fix at
76c2c6b0: 174 targeted tests passed across all three report owners. Ten runtime-asset mutation controls fail against the previous implementation and pass after the fix; cache creation preserves the identity. Lint and the complete installed-wheel package check passed. Current CI.