feat(versioning): freeze benchmark editions and compare contracts

CONTRIBUTING.md

@@ -1,5 +1,9 @@
 # Contributing
 
+Task, reference and scoring releases follow [benchmark versions](docs/versions.md).
+Keep package releases separate from benchmark editions; a new package commit
+must not silently replace a report's frozen inputs.
+
 Task definitions and acceptance requirements belong here. Task models, acceptance scoring, file checks and compilation also belong here. Browser adapters, scheduling, reference-capture orchestration and report rendering belong in the selected execution harness. Include a task ID and a reproducible accepted/rejected example when reporting a rule problem.
 
 Use Python 3.11 or newer and [uv](https://docs.astral.sh/uv/). Run `uv sync --frozen`, then `make check`. Run `uv sync --frozen --extra test` to install the test tools. Tests use synthetic inputs and need no browser, network access or historical source bundle. Keep the English and Chinese READMEs aligned; other guides use English.

README.md

@@ -69,6 +69,18 @@ uv run --frozen lexbench-webfetch export \
 
 The export includes only manifest-listed files and prints the executable dataset path. Missing files, changed bytes and symlinks fail validation. Existing output directories are never reused.
 
+## Freeze a benchmark version
+
+Reports bind a benchmark edition, its frozen dataset and its scoring source.
+Package upgrades do not change historical results. Use `lexbench-webfetch release`
+to freeze an external `benchmark-release.json`, and `compare-releases` to check
+whether two editions support direct comparison, require rescoring or belong
+to separate major versions. See [benchmark versions](docs/versions.md).
+
+`lexbench-webfetch version` verifies and prints the installed catalog edition,
+including every suite's task count, frozen input hashes and scoring source.
+The immutable `benchmark-vMAJOR.MINOR.PATCH` tag identifies its repository revision.
+
 ## Score a saved answer
 
 The benchmark includes its own scorer. It needs no Browser Eval installation:

docs/README.zh-CN.md

@@ -53,10 +53,13 @@ cd lexbench-webfetch
 uv sync --frozen
 uv run --frozen lexbench-webfetch list
 uv run --frozen lexbench-webfetch validate
+uv run --frozen lexbench-webfetch version
 ```
 
 这一步检查公开任务定义，无需浏览器、API Key 或 Browser Eval。它不访问网站，也不计算评测分数。
 
+`version` 会校验并显示当前题库版本，包括各子集题数、冻结输入哈希和评分器哈希。`benchmark-v主版本.次版本.补丁版本` 标签固定对应的仓库提交；报告另行绑定实际使用的题目和评分规则。
+
 复现冻结任务集需要取得获准使用的原始输入。所需文件及哈希列在 `src/lexbench_webfetch/definitions/registry.json` 中。本仓库不分发第三方页面快照，目前也没有公开的完整输入下载包。
 
 ```sh
@@ -69,6 +72,13 @@ uv run --frozen lexbench-webfetch export \
 
 导出工具只复制清单内的文件，并打印可执行数据集路径。缺文件、内容变化或符号链接都会使校验失败，已有输出目录不会被复用。
 
+## 固定评测版本
+
+报告绑定评测版本、冻结题目和评分源码。升级软件包不会改变历史结果。
+用 `lexbench-webfetch release` 在外部输入目录生成 `benchmark-release.json`，
+用 `compare-releases` 检查两个版本能否直接比较、是否需要统一重评，或应按不同大版本分别展示。
+具体规则见[评测版本管理](versions.md)。
+
 ## 对保存的响应评分
 
 本仓库自带评分器，无需安装 Browser Eval：

docs/versions.md

@@ -0,0 +1,91 @@
+# Benchmark versions
+
+A benchmark edition fixes the task population, request contract, acceptance
+rules, frozen references and scoring source. Its version is independent of
+the Python package version. A package upgrade does not upgrade a report.
+
+Use a readable `MAJOR.MINOR.PATCH` together with immutable SHA-256 bindings.
+Keep the exact repository commit in the harness dependency lock. A Git tag
+such as `benchmark-v2.0.0` gives a released commit a durable name; never move
+or overwrite a published release tag. A tag alone does not identify external
+reference bytes or a report's selected task population.
+
+Use `benchmark-vMAJOR.MINOR.PATCH` in both the benchmark and report repositories.
+A report names the benchmark edition it uses; it does not introduce a separate
+report-version sequence. The hosting platform's immutable version ID identifies
+each published report build. Retain historical tags at their original commits.
+
+| Change | Version | Comparing results |
+| --- | --- | --- |
+| Documentation or presentation; measurement contracts and scorer unchanged | Patch | Direct, with the same execution controls |
+| Reviewed scoring/reference correction; original tasks and requests retained | Minor | Recompute both saved cohorts under one edition before comparison |
+| Added/removed tasks, different URLs, requests, output scope or retrieval route | Major | Start a separate benchmark series of results |
+
+A minor version is for correcting the existing measurement, not adding a new
+scoring dimension under the same name. Expanding stable-label checks into a
+main-content benchmark is a major change. The release tool checks concrete
+population and request differences; maintainers still review whether a rule
+change preserves the intended task. Existing annotation revisions retain
+their independent source evidence and executable correction controls.
+
+## Freeze an edition
+
+`lexbench-webfetch version` verifies the installed catalog's complete suite list,
+task definitions, frozen reference bindings and scorer against
+`definitions/benchmark-release.json`. The published `benchmark-vMAJOR.MINOR.PATCH`
+tag fixes that catalog. Full reference inputs remain separately hash-bound;
+the catalog does not replace the executable datasets or distribute their source
+captures. A report binds its actual selected population with the sidecar below;
+installing a newer catalog never expands that population.
+
+Compile or export the inputs through their existing owner first, including any
+registered annotation revision. Freeze the complete selected population before
+provider calls:
+
+```sh
+uv run --frozen lexbench-webfetch release \
+  --dataset /absolute/inputs/dataset.json \
+  --output /absolute/inputs/benchmark-release.json \
+  --benchmark lexbench-webfetch --version 2.0.0 \
+  --scorer-revision <full-40-character-scorer-commit>
+```
+
+For a successor, add `--previous /absolute/previous/benchmark-release.json`.
+The scorer commit locates the code; the independently computed scoring-source
+hash fixes the actual evaluator bytes. Scoring corrections also require a
+concrete `--change-note`. The tool rejects
+an insufficient version bump and an existing output path. Keep the sidecar
+alongside frozen inputs, outside Git. It contains no provider measurements or
+local paths. Commit reusable definitions and publish the code through the
+normal feature-branch and release process.
+
+```sh
+uv run --frozen lexbench-webfetch compare-releases \
+  /absolute/old/benchmark-release.json /absolute/new/benchmark-release.json
+```
+
+The result is `direct`, `rescore` or `separate`. It checks actual contract
+fingerprints in addition to version numbers. Reusing the same version for a
+different measurement is an error. A different major edition stays separate
+even if some task IDs overlap. An overlap analysis may be diagnostic, but it
+does not make the two full-population percentages comparable.
+
+## Reports and retained calls
+
+Report builds retain the selected edition, dataset digest and per-result
+scoring-source identities. Rebuilding presentation preserves that edition.
+Scoring corrections create a new edition and result revision, retaining the
+previous report. Do not replace task inputs from an installed package merely
+because it is newer.
+
+For a minor correction, reuse successful and failed saved outputs wherever
+the original request contract still applies, and recompute their scores under
+one scorer. Changed requests need new calls for the affected scope. Preserve
+all physical attempts and use the existing final-attempt selection policy.
+Execution budgets, method configurations and timing comparability remain
+owned by the harness; a matching dataset edition does not establish those.
+
+Historical reports without a readable edition keep their existing exact hash
+bindings. Naming one does not change its task snapshot, results, permissions
+or original scoring source. Never label a historical subset as the latest
+full dataset.

src/lexbench_webfetch/benchmark.py

@@ -24,6 +24,7 @@
     "source_review_url", "evaluation_url", "expected_final_url", "page_identity_required",
     "expected_document_language", "document_language_match",
     "expected_document_language_present", "target_language", "target_region",
+    "ordered_tree_min_similarity", "ordered_tree_reference_sha256",
 )
 
 
@@ -49,9 +50,16 @@ def public_task(case: dict) -> dict:
     task = {key: case[key] for key in PUBLIC_TASK_FIELDS if key in case}
     if "body_requirements" in task:
         task["body_requirements"] = reference_bindings(task["body_requirements"])
+    if case.get("ordered_tree_reference") is not None:
+        task["ordered_tree_reference_sha256"] = hashlib.sha256(json.dumps(
+            case["ordered_tree_reference"], ensure_ascii=False, sort_keys=True,
+            separators=(",", ":"), allow_nan=False).encode()).hexdigest()
     benchmark = case.get("source_benchmark", {})
     routing = {key: benchmark[key] for key in
-               ("mode", "hosts", "entry_path", "minimum_entries", "verification_scope")
+               ("mode", "hosts", "entry_path", "minimum_entries", "verification_scope",
+                "entry_identity_pattern", "identity_query_keys", "link_base_url", "entry_scope",
+                "name_validation", "record_name_exclusions", "title_exclusion_patterns",
+                "price_patterns", "price_exclusion_patterns", "price_unavailable_patterns")
                if key in benchmark}
     from .required_components import COMPONENTS
     for component in COMPONENTS:

src/lexbench_webfetch/cli.py

@@ -10,6 +10,19 @@ def main() -> None:
     parser = argparse.ArgumentParser(description="Inspect and verify LexBench Web Fetch inputs")
     sub = parser.add_subparsers(dest="command", required=True)
     sub.add_parser("list", help="List the versioned suites")
+    sub.add_parser("version", help="Show and verify the published benchmark catalog edition")
+    release = sub.add_parser("release", help="Freeze a dataset and scorer as a benchmark edition")
+    release.add_argument("--dataset", type=Path, required=True)
+    release.add_argument("--output", type=Path, required=True)
+    release.add_argument("--benchmark", required=True)
+    release.add_argument("--version", required=True)
+    release.add_argument("--scorer-revision", required=True,
+                         help="Full Git commit containing the selected scorer")
+    release.add_argument("--previous", type=Path)
+    release.add_argument("--change-note")
+    compare = sub.add_parser("compare-releases", help="Check whether two result editions can be compared")
+    compare.add_argument("left", type=Path)
+    compare.add_argument("right", type=Path)
     check = sub.add_parser("validate", help="Check all public definitions and bindings")
     check.add_argument("--suite")
     for name in ("verify-inputs", "export"):
@@ -43,7 +56,21 @@ def main() -> None:
     articles.add_argument("--evidence-home", type=Path, required=True)
     args = parser.parse_args()
     try:
-        if args.command == "inspect-revision":
+        if args.command == "version":
+            from .versions import catalog_release
+            result = catalog_release()
+        elif args.command == "release":
+            from .versions import freeze_release
+            previous = json.loads(args.previous.read_text()) if args.previous else None
+            result = freeze_release(args.dataset, args.output, benchmark=args.benchmark,
+                                    version=args.version, previous=previous,
+                                    change_note=args.change_note,
+                                    scorer_revision=args.scorer_revision)
+        elif args.command == "compare-releases":
+            from .versions import compare_releases
+            result = compare_releases(json.loads(args.left.read_text()),
+                                      json.loads(args.right.read_text()))
+        elif args.command == "inspect-revision":
             result = inspect_annotation_revision(args.suite, args.revision)
         elif args.command == "score":
             from .evaluation import evaluate_response

src/lexbench_webfetch/definitions/benchmark-release.json

@@ -0,0 +1,72 @@
+{
+  "schema_version": 1,
+  "kind": "benchmark_catalog_release",
+  "benchmark": "lexmount-webfetch",
+  "version": "2.0.1",
+  "tag": "benchmark-v2.0.1",
+  "scorer_revision": "d9e2f00c7bca0298960ee42e8d2493ab1e835bb0",
+  "scoring_source_sha256": "c97158df8941ea55d358f5defd9c3e97869f82ae8aba5ecb6468dc43f1ed1c7a",
+  "case_count": 550,
+  "suites": [
+    {
+      "suite": "webfetch",
+      "case_count": 150,
+      "dataset_sha256": "5f751146143dcbe9daf46f0fb2511da1147508b3c8c4cd3377a6f3cceb82264e",
+      "dataset_id": "webfetch",
+      "directory": "evaluation",
+      "files": {
+        "tasks.json": "ae3aec6eab8d2d470cdeb18c4f80d238e5dd79e1108a729589b77a80fe869405",
+        "dataset-catalog.json": "994fa53f7fc7c3170692a7bb493ebc5af9ad8ffb5b576c022fa123e6e6fb3838",
+        "source-inputs.json": "c202abc4624811b938a837402bbaf0ecbe716e375fbca70567346d2edb3b1708"
+      }
+    },
+    {
+      "suite": "popular-sites",
+      "case_count": 100,
+      "dataset_sha256": "85544512d4cfd9459c22af1f70b1eb400844742dbf3179807662b76db2dcfc2e",
+      "dataset_id": "popular-sites",
+      "directory": "evaluation/popular-sites",
+      "files": {
+        "tasks.json": "82859ffb301740bcf38fee4fe87a675e8fb21af8dd660f92f74e4cbaede35b33",
+        "dataset-catalog.json": "f2b9dfb550230cdf8dbbba544d58158ac957620c1d064173790ead70754c178d",
+        "source-inputs.json": "32c31cfd5458a1dfe06d370fc2411a1eea5d1a05da8ef88e0dcd4fd3b0f3c810"
+      }
+    },
+    {
+      "suite": "dynamic-content",
+      "case_count": 100,
+      "dataset_sha256": "0da2bb30aaabe3ca2b93ec17b4cb88cdc1cf3ea46336bb3777fdb21c1f211b84",
+      "dataset_id": "dynamic-qualified-100-moli-native-local-audit-20260924",
+      "directory": "evaluation/dynamic-content-100-moli-webfetch",
+      "files": {
+        "tasks.json": "139c62a141183d305d8fa9c3658c546927731461d86aeba5ad3091649bd4ecd7",
+        "dataset-catalog.json": "ca1572aa80fc2324b2f00800a79b053cfd584d1831ffb16e6906ae0981522c6c",
+        "source-inputs.json": "710f9c91705b29bb9e4aa15af97dcc856e64e406baa4b9966446408d7e3d26a8"
+      }
+    },
+    {
+      "suite": "access-restrictions",
+      "case_count": 100,
+      "dataset_sha256": "165b888133e859304b035d74183dd40bf7ab609cac78bb6ae6e75b645c1de426",
+      "dataset_id": "risk-control-audited-100",
+      "directory": "evaluation/risk-control/current",
+      "files": {
+        "tasks.json": "c67b60ccf5ee0b14e4e23b3efc282e625005f8e096f0bfc978772d312e8f50c2",
+        "dataset-catalog.json": "52ec968b2ada26e5e061ccb25cbe4a73747539e556fa88f31bece6d7fb79b3cb",
+        "source-inputs.json": "c61d6825abbff2a92098592f83055347cd0b0101374794023046c0fd0323dd5f"
+      }
+    },
+    {
+      "suite": "articles",
+      "case_count": 100,
+      "dataset_sha256": "1929d5f702a2f7a0a1248694ca6843fba02ecaf7c559c6bf232865a8fb3c4bb6",
+      "dataset_id": "articles-v11",
+      "directory": "evaluation/articles",
+      "files": {
+        "tasks.json": "171241ea0bf3cb3f9482de14a7201885733fa717af56d7ea2b28d86a4516a3bb",
+        "dataset-catalog.json": "8e413e683c309674abf3d297df3f378e76d65ead1e2aa343828e4b0e1d90764a",
+        "source-inputs.json": "0d9c729f14dae8abb3323a2cf4734736c4733957c9d74d5484033d204642b91b"
+      }
+    }
+  ]
+}