Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c74477b
Add article-suite authoring and independent post verification
Oct 19, 2026
845c1cb
Qualify the complete initial Smithsonian article
Oct 19, 2026
dff7cd1
Qualify five approved full-article replacements
Oct 19, 2026
0ec43e1
Remove four declined cases and qualify the Zhihu article
Oct 19, 2026
dba0be1
Qualify the original Substack article after modal dismissal
Oct 19, 2026
49ac184
Complete and register the 100-case article suite
Oct 19, 2026
79a3592
Require initial titles, semantic body sections and complete displayed…
Oct 20, 2026
554880f
Cap article body regions at five and clarify dynamic comparison limits
Oct 20, 2026
c8c1126
Use evidenced fixed coverage for initially displayed comments
Oct 20, 2026
2ef8ade
Require eighty percent comment coverage with complete small samples
Oct 20, 2026
e664d7f
Use fixed reusable coverage for article post collections
Oct 20, 2026
b1b3afe
Bound article comment coverage and revise brittle anchors
Oct 20, 2026
a01aaa0
Match sliced comment boundaries inside complete article text
Oct 20, 2026
323f766
Merge current main into article suite branch
Oct 22, 2026
55a30c1
Simplify article acceptance components and preserve visible content b…
Maisiqi Oct 22, 2026
41388ad
Move site and case extraction policy from code into versioned data
Maisiqi Oct 22, 2026
664b4d6
Bind article labels to frozen task contracts and protect literal reco…
Maisiqi Oct 22, 2026
668bbd0
Require explicit article labels and simplify input authoring
Maisiqi Oct 22, 2026
4c425ff
Clarify article publication steps and normalization semantics
Maisiqi Oct 22, 2026
0fe482a
Check article authoring and published contract consistency
Maisiqi Oct 22, 2026
416a7cc
Add Chinese visual overview of article benchmark PR
Maisiqi Oct 22, 2026
a016744
Focus PR overview on purpose and deliverables
Maisiqi Oct 22, 2026
e18f1e8
Explain article rule authoring in three-panel overview
Maisiqi Oct 22, 2026
4dd9a95
Reject code-literal components and metadata-only post authors
Maisiqi Oct 22, 2026
b6b0105
Add four approved archive-based Weixin article rules
Maisiqi Oct 22, 2026
f38bccb
Require all article regions and reviewed endings; deduplicate review …
Maisiqi Oct 22, 2026
4edfd2b
Clarify structural coverage limits from follow-up review
Maisiqi Oct 22, 2026
c584082
Record decision to retain changing collection cases
Maisiqi Oct 22, 2026
ba7bdae
Document public reference distribution limits and follow-up scope
Maisiqi Oct 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,14 @@ flowchart LR
| `dynamic-content` | 100 | Content and records loaded after the initial page response |
| `popular-sites` | 100 | Task-relevant content on widely used websites |
| `access-restrictions` | 100 | URLs where access restrictions were previously observed |
| `articles` | 100 | Displayed article bodies; fixed initial comment and post coverage rules |

These are four separately versioned populations, with 450 task entries in total. They are not a deduplicated 450-site leaderboard. The access-restriction label describes historical observations; a URL may behave differently today. WebMainBench, WCXB and WebArena remain separate upstream benchmarks.
These are five separately versioned populations, with 550 task entries in total. They are not a deduplicated 550-site leaderboard. The access-restriction label describes historical observations; a URL may behave differently today. WebMainBench, WCXB and WebArena remain separate upstream benchmarks.

Each suite ships a readable `tasks.json`, a catalog with original task-definition hashes, and a manifest binding the exact executable dataset and reference files. See [data layout and rights](docs/data.md). Task wording and frozen acceptance rules retain their original language.

The [article suite](src/lexbench_webfetch/definitions/evaluation/articles/README.md), `articles-v11`, adds 95 fixed articles and five changing post collections. It checks displayed article content and required page components, including initial comment and post coverage, to catch title-only or body-only extraction. Fixed articles require 80% coverage in every declared body region and the reviewed ending. This tolerates formatting differences but does not prove every small module or its order. Changing comments and post lists use structural checks (record counts, attribution and nonempty text), not verification of the full contents of each record. Complete references remain outside Git.

## Quickstart

Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then:
Expand Down Expand Up @@ -83,6 +86,7 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa

The first answer passes; the second fails. Use `--metadata /absolute/path/metadata.json` to include structured answer fields such as `links` or `images`; the file must contain a JSON object. Metadata cannot supply a file-acceptance verdict. Supply the complete task JSON from a verified dataset for a real result. Public task projections bind reference bodies by hash and cannot replace those complete rules. File tasks use repeated `--file /absolute/path/file` arguments; acceptance is derived from the actual bytes, signature, length and declared hash, not a supplied pass flag. Optional ordered-tree tasks require `uv sync --frozen --extra tree`.


Compile the audited access-restriction rules with independently captured references:

```sh
Expand Down
6 changes: 5 additions & 1 deletion docs/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,14 @@ flowchart LR
| `dynamic-content` | 100 | 初始页面响应后加载的正文与记录 |
| `popular-sites` | 100 | 常用网站上与任务相关的内容 |
| `access-restrictions` | 100 | 曾观察到访问限制的网址 |
| `articles` | 100 | 已展示的文章正文;固定的初始评论和帖子覆盖规则 |

四组任务分别固定版本,共 450 条任务记录,不代表去重后的 450 个网站,也不直接合成总榜。访问限制描述的是历史观察,网址今天的表现可能不同。WebMainBench、WCXB 和 WebArena 仍是各自独立的上游 benchmark。
五组任务分别固定版本,共 550 条任务记录,不代表去重后的 550 个网站,也不直接合成总榜。访问限制描述的是历史观察,网址今天的表现可能不同。WebMainBench、WCXB 和 WebArena 仍是各自独立的上游 benchmark。

每组提供可读的 `tasks.json`、包含原始完整任务哈希的目录,以及绑定可执行数据集和参考文件的清单。详见[数据结构与使用权](data.md)。任务原文和冻结的验收规则保留原语言。

[文章分类](../src/lexbench_webfetch/definitions/evaluation/articles/README.md)的 `articles-v11` 增加了 95 个固定文章和 5 个变化的帖子集合案例。它检查已展示的文章内容及必需页面组成部分,包括初始评论和帖子覆盖,识别只抓到标题或只抓到正文的遗漏。固定文章要求每个正文分区及核对过的正文结尾均达到 80% 覆盖率。这允许格式差异,但不保证每个小节及其顺序完整。变化的评论和帖子列表检查记录数量、作者及非空正文等基本结构,不验证每条记录的正文是否完整。完整参考保存在 Git 之外。

## 快速开始

安装 [uv](https://docs.astral.sh/uv/getting-started/installation/),然后运行:
Expand Down Expand Up @@ -83,6 +86,7 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa

第一份响应通过,第二份失败。若回答含有结构化链接或图片等字段,可通过 `--metadata /absolute/path/metadata.json` 提供,文件内容须为 JSON 对象。元数据不能指定文件验收结果。实际评分时,请使用已校验数据集中的完整任务 JSON。公开任务文件中的参考正文只有哈希绑定,不能替代完整验收输入。文件任务通过可重复的 `--file /absolute/path/file` 提供真实文件;评分器检查字节签名、长度和声明的哈希,不接受调用者自报的通过标记。可选的树结构任务需要先运行 `uv sync --frozen --extra tree`。


使用独立采集的参考输入编译已审定的访问受限任务规则:

```sh
Expand Down
77 changes: 77 additions & 0 deletions docs/article-review-decisions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Article coverage review decisions

This records the response to the [100-case review](https://github.com/lexmount/lexbench-webfetch/pull/5#issuecomment-6336835161) against `4dd9a95` and the approved `articles-v10`/`articles-v11` revisions.

- **Missing profile content in 038/041: accepted and corrected.** Both saved accessibility observations contain the bio, category, location, website and joined date. The rules now require these fields using the existing shared stable-text matcher. Paired public-scorer controls accept complete profiles and reject each missing field. No site-specific scoring branch was added.
- **Dynamic records with a one-character body: reproduced, not claimed fixed.** All five collection contracts accept distinct attributed links with nonempty text, including `x`. This is a real limit of structural coverage. The user explicitly chose reusable fixed floors without per-run source acquisition and excluded reply relationships. Reintroducing contemporary references or reply matching would change that approved contract. A minimum character count would reject legitimate short posts while accepting unrelated padding. No such patch is applied, and the suite must not be presented as proving semantic or exhaustive dynamic-body completeness.
- **Static coverage: adopt the user-approved minimal scheme 2 from the v7 review.** Version 4 requires every existing body region and the separately bound ending to reach 80%. No new module layer or website branch is published. The experimental module layer was withdrawn: 24 of 25 omission controls failed as intended, but repeated text still allowed one omission. Scheme 2 addresses the weaker 60% region allowance and unprotected tail; it does not solve every middle-module omission, ordering, table-cell or speaker association. Those stronger proposals are not claimed implemented. Exact 20% character deletion need not fail an 80% fragment rule. Format and truncation controls must substantiate the narrower contract.
- **Repeated review prose: accepted.** Per-case defaults deduplicate basis/date fields; every row retains its ID, source region and explicit exceptions. Expanded reviews and acceptance/title/comment/post contracts are unchanged by this storage change. Input version 11 preserves the already published v10 Weixin revision.
- **Retrievable evidence: valid reproducibility requirement, external distribution still outstanding.** Full input and capture bundles exist in local external storage and are hash-bound. Local paths are not a public download service. Do not commit third-party captures or claim a public bundle exists. Public rules plus a hash alone do not let a reviewer reconstruct the frozen bodies. A separately provisioned evidence location remains necessary for external reproduction.

## Weixin revision

Four user-approved articles replace 001, 002, 003 and 077, each previously the only case from its platform and without a comment contract. The population stays at 100: 95 single articles/posts and five collections; all 20 existing comment contracts remain unchanged. The new population covers 72 platform names and 75 URL hosts.

The evidence consists of four user-supplied webarchives and the user's confirmation that they were saved after normal automatic loading without interaction. It is not an independently successful live browser visit. Source captures remain external. Main-body boundaries exclude attribution, promotional cards and related-article lists; source links remain. Ordinary stable attribution/profile/control text is declared separately with per-region rationale. Hidden comment dialogs do not become visible comment requirements. The existing compiler, qualifier and scorer are reused unchanged.

## Version 11 scope

The four approved Weixin cases and all 100 URLs are preserved. Body compilation changes for 95 fixed articles; five changing collections retain their contracts. Review metadata storage changes do not relax any acceptance threshold. Prior frozen bytes remain unchanged. The v7 proposal referenced v9/v10; current publication instead advances v10 to v11 to avoid reusing the already bound version.

## Version 11 validation and limits

All 95 fixed reference bodies passed body-only checks in plain text, Markdown
blockquote and HTML-derived text (285 positive controls). Truncating the last
24% of characters failed for all 95. Removing only the declared ending failed
for 93/95; 069 and 100 still passed because matching fragments occur elsewhere.
Removing the last four blank-line-delimited paragraphs failed for 94/95; 069
still passed. These are source-based body-layer controls, not provider runs or
a claim that all 100 cases are fixed articles. No special exception was added
for these cases. Scheme 2 is a stricter coverage contract, not an ordered
end-position or semantic completeness guarantee.

Deduplication removes 7,671 repeated review fields: rules.json decreases from
3,078,113 to 2,211,902 bytes. Expanding defaults reproduces every prior review
row; acceptance, title, comment and collection contracts remain unchanged.

## Follow-up at f38bccb

The [follow-up review](https://github.com/lexmount/lexbench-webfetch/pull/5#issuecomment-6340008998) correctly distinguishes coverage limits from newly introduced bugs. Reproduction through the public scorer again accepts attributed distinct records with body `x` for all five collections, while removing their records fails. The main READMEs now disclose the structural scope directly rather than requiring readers to find it in the detailed contract. No score, floor or case is changed by that clarification.

The user explicitly retained the five changing collections; replacing them with fixed articles/posts is no longer proposed. Retention does not establish acceptance of a stronger content-completeness claim. Per-run independent reference acquisition would change the previously requested reusable-reference workflow and has not been authorized. The user has now confirmed retaining the structural contract for this PR. [Issue #9](https://github.com/lexmount/lexbench-webfetch/issues/9) tracks investigation of stronger dynamic-body coverage; it does not claim that coverage is implemented or that the reviewer approved the scope. The user has explicitly retained the five-region/80%-plus-ending fixed-body scope rather than adding per-module checks.

A private handoff package has been prepared outside Git: it contains the frozen v11 scoring input for all 100 cases, the four original Weixin webarchives, their qualified extracts and scope evidence, and a checksum manifest. All 17 data files were checked after packaging. It does not contain the remaining 96 historical browser captures. It is not yet delivered to the reviewer or hosted at a shared location. Raw archives can contain session-bearing resource URLs; preparing a local package is not authorization to publish them.


## Public reproduction follow-up

Private delivery to one reviewer would not provide public reproducibility. The
required scoring input and the original captures serve different purposes:
`dataset.json` supplies the frozen scoring references; captures support auditing
what the source displayed. A verified local scoring package exists separately
from the larger capture package, but neither is currently publicly distributed.

The v11 input passes `verify-inputs` for all 100 cases. A preliminary scan found
no token/ticket/session/authentication parameter names in its extracted URLs;
this is not a complete privacy audit or distribution clearance. The input does
not include a per-source redistribution permission manifest.

A source-policy check on 2026-10-22 found materially different conditions:
[Our World in Data](https://ourworldindata.org/faqs),
[PLOS](https://plos.org/terms-of-use/) and
[Distill](https://distill.pub/journal/) provide reuse terms that require checking
the specific content and attribution. [Quanta](https://www.quantamagazine.org/terms-conditions/)
and [Pew](https://www.pewresearch.org/about/terms-and-conditions/) do not provide
blanket permission for the proposed full-text bundle in the reviewed terms.
No separate permission for that distribution is recorded. This is not a legal
determination of statutory exceptions, and the remaining sources have not all
been reviewed. The code license does not resolve these differences.

Public distribution therefore remains open. A release must have a documented
distribution basis, appropriate notices, a stable download location and matching
hashes before it can be advertised as a reproducible public input package. Moving
the same files to Release assets does not itself supply that basis. Publishing a
permitted subset would not reproduce the complete 100-case v11 suite. Do not
silently remove cases, replace references, alter frozen bytes, or present private
delivery as completion of this requirement. No scoring changes are justified by
this distribution gap.
6 changes: 3 additions & 3 deletions docs/assets/overview.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/pr5-distribution-zh.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/pr5-overview-zh.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/protocol.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Evaluation protocol

A sample is one declared method–task logical result. Choose the complete task population, methods, versions, repetitions, timeout, concurrency and retry policy before calls. Keep the four suites separate unless a combined population and deduplication policy were declared in advance.
A sample is one declared method–task logical result. Choose the complete task population, methods, versions, repetitions, timeout, concurrency and retry policy before calls. Keep the suites separate unless a combined population and deduplication policy were declared in advance.

## Inputs and delivery

Expand Down
Loading
Loading