diff --git a/README.md b/README.md index ffd47d9..cd5de7e 100644 --- a/README.md +++ b/README.md @@ -41,11 +41,14 @@ flowchart LR | `dynamic-content` | 100 | Content and records loaded after the initial page response | | `popular-sites` | 100 | Task-relevant content on widely used websites | | `access-restrictions` | 100 | URLs where access restrictions were previously observed | +| `articles` | 100 | Displayed article bodies; fixed initial comment and post coverage rules | -These are four separately versioned populations, with 450 task entries in total. They are not a deduplicated 450-site leaderboard. The access-restriction label describes historical observations; a URL may behave differently today. WebMainBench, WCXB and WebArena remain separate upstream benchmarks. +These are five separately versioned populations, with 550 task entries in total. They are not a deduplicated 550-site leaderboard. The access-restriction label describes historical observations; a URL may behave differently today. WebMainBench, WCXB and WebArena remain separate upstream benchmarks. Each suite ships a readable `tasks.json`, a catalog with original task-definition hashes, and a manifest binding the exact executable dataset and reference files. See [data layout and rights](docs/data.md). Task wording and frozen acceptance rules retain their original language. +The [article suite](src/lexbench_webfetch/definitions/evaluation/articles/README.md), `articles-v11`, adds 95 fixed articles and five changing post collections. It checks displayed article content and required page components, including initial comment and post coverage, to catch title-only or body-only extraction. Fixed articles require 80% coverage in every declared body region and the reviewed ending. This tolerates formatting differences but does not prove every small module or its order. Changing comments and post lists use structural checks (record counts, attribution and nonempty text), not verification of the full contents of each record. Complete references remain outside Git. + ## Quickstart Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then: @@ -83,6 +86,7 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa The first answer passes; the second fails. Use `--metadata /absolute/path/metadata.json` to include structured answer fields such as `links` or `images`; the file must contain a JSON object. Metadata cannot supply a file-acceptance verdict. Supply the complete task JSON from a verified dataset for a real result. Public task projections bind reference bodies by hash and cannot replace those complete rules. File tasks use repeated `--file /absolute/path/file` arguments; acceptance is derived from the actual bytes, signature, length and declared hash, not a supplied pass flag. Optional ordered-tree tasks require `uv sync --frozen --extra tree`. + Compile the audited access-restriction rules with independently captured references: ```sh diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index a34e08b..0b090f9 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -41,11 +41,14 @@ flowchart LR | `dynamic-content` | 100 | 初始页面响应后加载的正文与记录 | | `popular-sites` | 100 | 常用网站上与任务相关的内容 | | `access-restrictions` | 100 | 曾观察到访问限制的网址 | +| `articles` | 100 | 已展示的文章正文；固定的初始评论和帖子覆盖规则 | -四组任务分别固定版本，共 450 条任务记录，不代表去重后的 450 个网站，也不直接合成总榜。访问限制描述的是历史观察，网址今天的表现可能不同。WebMainBench、WCXB 和 WebArena 仍是各自独立的上游 benchmark。 +五组任务分别固定版本，共 550 条任务记录，不代表去重后的 550 个网站，也不直接合成总榜。访问限制描述的是历史观察，网址今天的表现可能不同。WebMainBench、WCXB 和 WebArena 仍是各自独立的上游 benchmark。 每组提供可读的 `tasks.json`、包含原始完整任务哈希的目录，以及绑定可执行数据集和参考文件的清单。详见[数据结构与使用权](data.md)。任务原文和冻结的验收规则保留原语言。 +[文章分类](../src/lexbench_webfetch/definitions/evaluation/articles/README.md)的 `articles-v11` 增加了 95 个固定文章和 5 个变化的帖子集合案例。它检查已展示的文章内容及必需页面组成部分，包括初始评论和帖子覆盖，识别只抓到标题或只抓到正文的遗漏。固定文章要求每个正文分区及核对过的正文结尾均达到 80% 覆盖率。这允许格式差异，但不保证每个小节及其顺序完整。变化的评论和帖子列表检查记录数量、作者及非空正文等基本结构，不验证每条记录的正文是否完整。完整参考保存在 Git 之外。 + ## 快速开始 安装 [uv](https://docs.astral.sh/uv/getting-started/installation/)，然后运行： @@ -83,6 +86,7 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa 第一份响应通过，第二份失败。若回答含有结构化链接或图片等字段，可通过 `--metadata /absolute/path/metadata.json` 提供，文件内容须为 JSON 对象。元数据不能指定文件验收结果。实际评分时，请使用已校验数据集中的完整任务 JSON。公开任务文件中的参考正文只有哈希绑定，不能替代完整验收输入。文件任务通过可重复的 `--file /absolute/path/file` 提供真实文件；评分器检查字节签名、长度和声明的哈希，不接受调用者自报的通过标记。可选的树结构任务需要先运行 `uv sync --frozen --extra tree`。 + 使用独立采集的参考输入编译已审定的访问受限任务规则： ```sh diff --git a/docs/article-review-decisions.md b/docs/article-review-decisions.md new file mode 100644 index 0000000..990055f --- /dev/null +++ b/docs/article-review-decisions.md @@ -0,0 +1,77 @@ +# Article coverage review decisions + +This records the response to the [100-case review](https://github.com/lexmount/lexbench-webfetch/pull/5#issuecomment-6336835161) against `4dd9a95` and the approved `articles-v10`/`articles-v11` revisions. + +- **Missing profile content in 038/041: accepted and corrected.** Both saved accessibility observations contain the bio, category, location, website and joined date. The rules now require these fields using the existing shared stable-text matcher. Paired public-scorer controls accept complete profiles and reject each missing field. No site-specific scoring branch was added. +- **Dynamic records with a one-character body: reproduced, not claimed fixed.** All five collection contracts accept distinct attributed links with nonempty text, including `x`. This is a real limit of structural coverage. The user explicitly chose reusable fixed floors without per-run source acquisition and excluded reply relationships. Reintroducing contemporary references or reply matching would change that approved contract. A minimum character count would reject legitimate short posts while accepting unrelated padding. No such patch is applied, and the suite must not be presented as proving semantic or exhaustive dynamic-body completeness. +- **Static coverage: adopt the user-approved minimal scheme 2 from the v7 review.** Version 4 requires every existing body region and the separately bound ending to reach 80%. No new module layer or website branch is published. The experimental module layer was withdrawn: 24 of 25 omission controls failed as intended, but repeated text still allowed one omission. Scheme 2 addresses the weaker 60% region allowance and unprotected tail; it does not solve every middle-module omission, ordering, table-cell or speaker association. Those stronger proposals are not claimed implemented. Exact 20% character deletion need not fail an 80% fragment rule. Format and truncation controls must substantiate the narrower contract. +- **Repeated review prose: accepted.** Per-case defaults deduplicate basis/date fields; every row retains its ID, source region and explicit exceptions. Expanded reviews and acceptance/title/comment/post contracts are unchanged by this storage change. Input version 11 preserves the already published v10 Weixin revision. +- **Retrievable evidence: valid reproducibility requirement, external distribution still outstanding.** Full input and capture bundles exist in local external storage and are hash-bound. Local paths are not a public download service. Do not commit third-party captures or claim a public bundle exists. Public rules plus a hash alone do not let a reviewer reconstruct the frozen bodies. A separately provisioned evidence location remains necessary for external reproduction. + +## Weixin revision + +Four user-approved articles replace 001, 002, 003 and 077, each previously the only case from its platform and without a comment contract. The population stays at 100: 95 single articles/posts and five collections; all 20 existing comment contracts remain unchanged. The new population covers 72 platform names and 75 URL hosts. + +The evidence consists of four user-supplied webarchives and the user's confirmation that they were saved after normal automatic loading without interaction. It is not an independently successful live browser visit. Source captures remain external. Main-body boundaries exclude attribution, promotional cards and related-article lists; source links remain. Ordinary stable attribution/profile/control text is declared separately with per-region rationale. Hidden comment dialogs do not become visible comment requirements. The existing compiler, qualifier and scorer are reused unchanged. + +## Version 11 scope + +The four approved Weixin cases and all 100 URLs are preserved. Body compilation changes for 95 fixed articles; five changing collections retain their contracts. Review metadata storage changes do not relax any acceptance threshold. Prior frozen bytes remain unchanged. The v7 proposal referenced v9/v10; current publication instead advances v10 to v11 to avoid reusing the already bound version. + +## Version 11 validation and limits + +All 95 fixed reference bodies passed body-only checks in plain text, Markdown +blockquote and HTML-derived text (285 positive controls). Truncating the last +24% of characters failed for all 95. Removing only the declared ending failed +for 93/95; 069 and 100 still passed because matching fragments occur elsewhere. +Removing the last four blank-line-delimited paragraphs failed for 94/95; 069 +still passed. These are source-based body-layer controls, not provider runs or +a claim that all 100 cases are fixed articles. No special exception was added +for these cases. Scheme 2 is a stricter coverage contract, not an ordered +end-position or semantic completeness guarantee. + +Deduplication removes 7,671 repeated review fields: rules.json decreases from +3,078,113 to 2,211,902 bytes. Expanding defaults reproduces every prior review +row; acceptance, title, comment and collection contracts remain unchanged. + +## Follow-up at f38bccb + +The [follow-up review](https://github.com/lexmount/lexbench-webfetch/pull/5#issuecomment-6340008998) correctly distinguishes coverage limits from newly introduced bugs. Reproduction through the public scorer again accepts attributed distinct records with body `x` for all five collections, while removing their records fails. The main READMEs now disclose the structural scope directly rather than requiring readers to find it in the detailed contract. No score, floor or case is changed by that clarification. + +The user explicitly retained the five changing collections; replacing them with fixed articles/posts is no longer proposed. Retention does not establish acceptance of a stronger content-completeness claim. Per-run independent reference acquisition would change the previously requested reusable-reference workflow and has not been authorized. The user has now confirmed retaining the structural contract for this PR. [Issue #9](https://github.com/lexmount/lexbench-webfetch/issues/9) tracks investigation of stronger dynamic-body coverage; it does not claim that coverage is implemented or that the reviewer approved the scope. The user has explicitly retained the five-region/80%-plus-ending fixed-body scope rather than adding per-module checks. + +A private handoff package has been prepared outside Git: it contains the frozen v11 scoring input for all 100 cases, the four original Weixin webarchives, their qualified extracts and scope evidence, and a checksum manifest. All 17 data files were checked after packaging. It does not contain the remaining 96 historical browser captures. It is not yet delivered to the reviewer or hosted at a shared location. Raw archives can contain session-bearing resource URLs; preparing a local package is not authorization to publish them. + + +## Public reproduction follow-up + +Private delivery to one reviewer would not provide public reproducibility. The +required scoring input and the original captures serve different purposes: +`dataset.json` supplies the frozen scoring references; captures support auditing +what the source displayed. A verified local scoring package exists separately +from the larger capture package, but neither is currently publicly distributed. + +The v11 input passes `verify-inputs` for all 100 cases. A preliminary scan found +no token/ticket/session/authentication parameter names in its extracted URLs; +this is not a complete privacy audit or distribution clearance. The input does +not include a per-source redistribution permission manifest. + +A source-policy check on 2026-10-22 found materially different conditions: +[Our World in Data](https://ourworldindata.org/faqs), +[PLOS](https://plos.org/terms-of-use/) and +[Distill](https://distill.pub/journal/) provide reuse terms that require checking +the specific content and attribution. [Quanta](https://www.quantamagazine.org/terms-conditions/) +and [Pew](https://www.pewresearch.org/about/terms-and-conditions/) do not provide +blanket permission for the proposed full-text bundle in the reviewed terms. +No separate permission for that distribution is recorded. This is not a legal +determination of statutory exceptions, and the remaining sources have not all +been reviewed. The code license does not resolve these differences. + +Public distribution therefore remains open. A release must have a documented +distribution basis, appropriate notices, a stable download location and matching +hashes before it can be advertised as a reproducible public input package. Moving +the same files to Release assets does not itself supply that basis. Publishing a +permitted subset would not reproduce the complete 100-case v11 suite. Do not +silently remove cases, replace references, alter frozen bytes, or present private +delivery as completion of this requirement. No scoring changes are justified by +this distribution gap. diff --git a/docs/assets/overview.svg b/docs/assets/overview.svg index 3276128..613aa61 100644 --- a/docs/assets/overview.svg +++ b/docs/assets/overview.svg @@ -1,14 +1,14 @@ LexBench Web Fetch -Benchmark definitions: 450 TASK ENTRIES — 150 general · 100 dynamic · 100 popular · 100 restricted; ACCEPTANCE RULES — Body text · Structured records · Resources · Files; FROZEN INPUTS — Versioned catalogs · SHA-256 reference bindings +Benchmark definitions: 550 TASK ENTRIES — 150 general · 100 dynamic · 100 popular · 100 restricted · 100 articles; ACCEPTANCE RULES — Body text · Structured records · Resources · Files; FROZEN INPUTS — Versioned catalogs · SHA-256 reference bindings BENCHMARK DEFINITIONS LexBench Web Fetch -450 TASK ENTRIES +550 TASK ENTRIES 150 general · 100 dynamic -100 popular · 100 restricted +100 popular · 100 restricted · 100 articles ACCEPTANCE RULES diff --git a/docs/images/pr5-distribution-zh.png b/docs/images/pr5-distribution-zh.png new file mode 100644 index 0000000..af2ae81 Binary files /dev/null and b/docs/images/pr5-distribution-zh.png differ diff --git a/docs/images/pr5-overview-zh.png b/docs/images/pr5-overview-zh.png new file mode 100644 index 0000000..c25250b Binary files /dev/null and b/docs/images/pr5-overview-zh.png differ diff --git a/docs/protocol.md b/docs/protocol.md index 3e77a8e..126a569 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -1,6 +1,6 @@ # Evaluation protocol -A sample is one declared method–task logical result. Choose the complete task population, methods, versions, repetitions, timeout, concurrency and retry policy before calls. Keep the four suites separate unless a combined population and deduplication policy were declared in advance. +A sample is one declared method–task logical result. Choose the complete task population, methods, versions, repetitions, timeout, concurrency and retry policy before calls. Keep the suites separate unless a combined population and deduplication policy were declared in advance. ## Inputs and delivery diff --git a/src/lexbench_webfetch/article_comments.py b/src/lexbench_webfetch/article_comments.py new file mode 100644 index 0000000..cb9c3e8 --- /dev/null +++ b/src/lexbench_webfetch/article_comments.py @@ -0,0 +1,286 @@ +"""Check fixed initial comment coverage and explicit record ownership.""" +from __future__ import annotations + +import re + +import soupsieve + +from .article_text import (author_key, article_html, without_markdown_images, is_article_html, + article_markdown, TIMESTAMP_LINE, mask_markdown_code, label_key, + validate_labels, declared_labels) +from .body_coverage import tokens +from .dynamic_lists import visible_html_text +from .reference_quality import markdown_links, visible_markdown_links + + +def validate_comment_contract(contract): + """Validate the current fixed comment coverage contract.""" + if not isinstance(contract, dict) or type(contract.get("version")) is not int or contract["version"] != 2: + raise ValueError("only fixed comment coverage version 2 is supported") + allowed = {'version', 'minimum_comments', 'record_selector', 'author_selector', + 'body_selector', 'section_markers', 'id_attribute', 'text_scope', + 'ignored_labels', 'unavailable_notices'} + if set(contract) - allowed or type(contract.get('minimum_comments')) is not int or contract['minimum_comments'] < 1: + raise ValueError('comment coverage requires a positive minimum') + validate_labels(contract) + scope = contract.get('text_scope') + if scope is not None: + if (not isinstance(scope, dict) or set(scope) != {'body_end', 'end_markers'} + or not isinstance(scope['body_end'], str) or not tokens(scope['body_end']) + or not isinstance(scope['end_markers'], list) + or any(not isinstance(marker, str) or not tokens(marker) for marker in scope['end_markers'])): + raise ValueError('invalid comment text scope') + elif 'text_scope' in contract: + raise ValueError('invalid comment text scope') + try: + for key in ('record_selector', 'author_selector', 'body_selector'): + if not isinstance(contract[key], str) or not contract[key].strip(): + raise ValueError('empty comment selector') + soupsieve.compile(contract[key]) + if 'id_attribute' in contract and not re.fullmatch(r'[a-zA-Z_][\w:-]*', contract['id_attribute']): + raise ValueError('invalid comment ID attribute') + markers = contract['section_markers'] + if not isinstance(markers, list) or not markers or any(not isinstance(x, str) or not x.strip() for x in markers): + raise ValueError('comment section markers required') + except (KeyError, TypeError, re.error, soupsieve.SelectorSyntaxError) as error: + raise ValueError('invalid comment contract') from error + + +def _html_records(content, contract): + soup = article_html(content) + cards = soup.select(contract['record_selector']) + rows = [] + for card in cards: + def owner(node): + return node.find_parent(lambda x: soupsieve.match(contract['record_selector'], x)) + fields = [[node for node in card.select(contract[key]) if owner(node) is card] + for key in ('author_selector', 'body_selector')] + parent = card.find_parent(lambda x: soupsieve.match( + contract.get('parent_record_selector', contract['record_selector']), x)) + parent_index = next((i for i, candidate in enumerate(cards) if candidate is parent), + -1 if parent is not None else None) + row = {'parent': parent_index, 'source_id': card.get(contract.get('id_attribute', 'id'))} + if any(len(nodes) != 1 for nodes in fields): + rows.append(row) + continue + author, body = [nodes[0] for nodes in fields] + copy = article_html(str(body)) + for nested in list(copy.select(contract['record_selector'])): + if nested.parent is not None: + nested.decompose() + row.update(author=visible_html_text(str(author)), text=visible_html_text(str(copy))) + rows.append(row) + return rows + + +def _section_marker_tokens(line): + """Ignore changing count decorations only on a comment section heading.""" + label = re.sub(r'^\s*#{1,6}\s+', '', line).strip().rstrip(':：') + label = re.sub(r'^[0-9]+(?:\s*(?:条|则|件)\s*|\s+)', '', label) + label = re.sub(r'\s*(?:\(\s*[0-9]+\s*(?:条|则|件)?\s*\)|（\s*[0-9]+\s*(?:条|则|件)?\s*）)\s*$', '', label) + return tuple(tokens(label)) + + +# These are interface controls, not author identities or comment bodies. Match +# whole labels only: a real comment can discuss logging in or loading a page. + + +def valid_comment_records(rows, contract): + controls = declared_labels(contract, 'ignored_labels') + unavailable = declared_labels(contract, 'unavailable_notices') + valid, seen = [], set() + for row in rows: + author, body = row.get('author'), row.get('text') + if (author_key(author) is None or not isinstance(body, str) + or author_key(body) is None + or label_key(author) in controls + or label_key(body) in controls + or label_key(body) in unavailable): + continue + # Preserve separately identified equal comments. Without source IDs, + # repeated identical records provide no evidence of additional coverage. + identity = ('id', row.get('source_id') or row.get('id')) if row.get('source_id') or row.get('id') else ( + 'text', author_key(author), label_key(body)) + if identity not in seen: + seen.add(identity) + valid.append(row) + return valid + + +def _comment_time_metadata(line): + """Recognize standalone serializer timestamps, not date mentions in prose. + + Plain output consisting only of an author and date cannot distinguish a + timestamp from a date-only comment. Native HTML body ownership remains the + unambiguous representation for that unusual case. + """ + stamp = visible_markdown_links(line).strip() + return bool(TIMESTAMP_LINE.fullmatch(stamp)) + + +def _coverage_text_start(lines, contract): + """Prefer the reviewed body ending over an earlier navigation label. + + The ending is a scope hint, not an additional exact-body pass requirement. + If it is absent, an explicit comment heading can independently establish + the component. A bare navigation label cannot do so for scoped contracts. + """ + markers = {_section_marker_tokens(x) for x in contract['section_markers']} + scope = contract.get('text_scope') + end_markers = {_section_marker_tokens(x) for x in scope['end_markers']} if scope else set() + first = 0 + found_ending = False + if scope: + expected = ' '.join(tokens(visible_markdown_links(scope['body_end']))) + fragments, owners = [], [] + for index, line in enumerate(lines): + normalized = ' '.join(tokens(visible_markdown_links(line))) + if not normalized: + continue + fragments.append(normalized) + owners.extend([index] * (len(normalized) + 1)) + # Frozen endings are character slices, so their first or last word + # may be partial. Match normalized text, not whole-token sequences. + # Retain character ownership to find the actual ending line even when + # a serializer wraps the same visible prose onto multiple lines. + normalized_content = ' '.join(fragments) + offset = normalized_content.find(expected) + if offset >= 0: + # A real section heading disambiguates earlier body repetitions + # without allowing a later comment quoting the ending to move the + # start boundary past already displayed comments. + section = next((index for index, line in enumerate(lines) + if index > owners[offset + len(expected) - 1] + and re.match(r'\s*#{1,6}\s+', line) + and _section_marker_tokens(line) in markers), None) + if section is not None: + candidate = normalized_content.find(expected, offset + 1) + while candidate >= 0 and owners[candidate + len(expected) - 1] < section: + offset = candidate + candidate = normalized_content.find(expected, candidate + 1) + if offset >= 0: + first = owners[offset + len(expected) - 1] + 1 + found_ending = True + for index in range(first, len(lines)): + # A reviewed end boundary wins over a historical start label. Forms + # below the last comment must never open a new discussion component. + if _section_marker_tokens(lines[index]) in end_markers: + break + if _section_marker_tokens(lines[index]) not in markers: + continue + heading = re.match(r'\s*(#{1,6})\s+', lines[index]) + if not scope or found_ending or heading: + # Do not skip earlier records just because a comment form appears + # below them. Only a marker before any content is the start marker. + if found_ending and any(line.strip() for line in lines[first:index]): + break + return index + 1, len(heading[1]) if heading else 0, bool(heading) + return (first, 0, True) if found_ending else (None, 0, False) + + +def _text_author(stripped, heading, heading_authors, explicit_scope, new_paragraph): + """Recognize one author format without treating inline prose as a new record.""" + if heading: + return heading[2], '', True + if heading_authors: + return None, '', True + links = [link for link in markdown_links(stripped) if not link.is_image] + if len(links) == 1 and links[0].start == 0 and links[0].end == len(stripped): + return links[0].label, '', False + bold = re.fullmatch(r'\*\*(.+?)\*\*[:：]?', stripped) + if bold: + return bold[1], '', False + colon = re.fullmatch(r'([^\n:：]{1,120})[:：]\s*(.*)', stripped) + if explicit_scope and colon: + if new_paragraph: + return colon[1], colon[2], False + if not colon[2]: + return colon[1], '', False + return None, '', False + + +def _coverage_text_records(content, contract): + """Read explicit author/body formatting without a frozen author list. + + An unlabelled prose block cannot establish author attribution. Subheadings, + linked/bold author lines and colon-labelled records are supported; ambiguous + bare names are deliberately not guessed from arbitrary prose. Colon-style + authors additionally require a section heading or reviewed body ending: a + plain navigation label must not turn article captions into authors. + """ + # Code may be comment text, but cannot introduce fake record boundaries. + content, fences = mask_markdown_code(content, retain=True) + prefix = '\ue000coverage-paragraph' + while prefix in content: + prefix += '_' + content = without_markdown_images(content) + paragraph_marker = prefix + 'paragraph' + content = re.sub(r'\n[ \t]*\n', '\n' + paragraph_marker + '\n', content) + content = visible_html_text(str(article_html(content)), preserve_source_whitespace=True, preserve_links=True) + lines = content.replace(paragraph_marker, '').splitlines() + markers = {_section_marker_tokens(x) for x in contract['section_markers']} + start, section_level, explicit_scope = _coverage_text_start(lines, contract) + if start is None: + return [] + end_markers = {_section_marker_tokens(x) for x in contract.get('text_scope', {}).get('end_markers', [])} + controls = declared_labels(contract, "ignored_labels") + rows, current = [], None + heading_authors = False + for offset, line in enumerate(lines[start:], start): + previous_blank = offset == start or not lines[offset - 1].strip() + stripped = line.strip() + if stripped in fences: + if current is not None: + current['text'] += '\n' + fences[stripped] + continue + if _section_marker_tokens(line) in end_markers: + break + heading = re.fullmatch(r'\s*(#{1,6})\s+(.+?)\s*', line) + if heading and section_level and len(heading[1]) <= section_level: + break + if label_key(visible_markdown_links(stripped)) in controls: + current = None + continue + if _section_marker_tokens(line) in markers: + if not rows and heading: + # Metadata may separate the reviewed ending from the section + # heading. Retain its level before reading any author records. + section_level = len(heading[1]) + explicit_scope = True + current = None + continue + pingback = re.fullmatch(r'\s*Pingback:\s+(.+?)\s*', line) + if pingback: + label = visible_markdown_links(pingback[1]) + rows.append({'author': label, 'text': visible_markdown_links(line).strip()}) + current = None + continue + author, body, heading_authors = _text_author( + stripped, heading, heading_authors, explicit_scope, + current is None or previous_blank) + if author is not None: + current = {'author': visible_markdown_links(author), 'text': visible_markdown_links(body)} + rows.append(current) + elif current is not None and not _comment_time_metadata(line): + current['text'] += '\n' + visible_markdown_links(line) + return rows + + +def _verify_comment_coverage(content, contract): + if is_article_html(content): + native = article_html(content).select(contract['record_selector']) + rows = (_html_records(content, contract) if native + else _coverage_text_records(article_markdown(content), contract)) + else: + rows = _coverage_text_records(content, contract) + valid = valid_comment_records(rows, contract) + return {'article_comments_mode': 'fixed_coverage', + 'article_comments_pass': len(valid) >= contract['minimum_comments'], + 'article_comment_count': len(valid), + 'article_comment_minimum': contract['minimum_comments']} + + +def verify_article_comments(content, contract): + """Check fixed visible-comment coverage without a runtime reference.""" + validate_comment_contract(contract) + return _verify_comment_coverage(content, contract) diff --git a/src/lexbench_webfetch/article_dataset.py b/src/lexbench_webfetch/article_dataset.py new file mode 100644 index 0000000..79a4312 --- /dev/null +++ b/src/lexbench_webfetch/article_dataset.py @@ -0,0 +1,283 @@ +"""Compile article contracts against independently reviewed, external full bodies. + +The authoring inventory retains unresolved cases. Compilation is fail-closed: +every approved case needs its own complete, hash-bound reference and contract. +No provider, browser adapter or capture orchestration belongs in this module. +""" +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from urllib.parse import urlsplit + +from .body_coverage import body_requirements, compact_ngrams +from .models import EvalCase, validate_case +from .paths import contained_path +from .stable_tasks import validate_acceptance_contract + +DEFINITIONS = Path(__file__).parent / "definitions/evaluation/articles" + + +def _acceptance(rule: dict) -> dict: + acceptance = rule["acceptance_contract"] + validate_acceptance_contract(acceptance) + if acceptance["schema"] != "body_text" or not acceptance.get("page_patterns"): + raise ValueError("Article rules require body text and target identity") + if any(re.compile(pattern).search("") for pattern in acceptance["page_patterns"]): + raise ValueError("Article identity patterns cannot match empty text") + if acceptance.get("stable_text_role", "required") != "required": + raise ValueError("Declared article stable text must be required") + reviews = rule.get("stable_text_review") + if not isinstance(reviews, list) or not rule.get("excluded_regions"): + raise ValueError("Article rules need stable-region review and exclusions") + defaults = rule.get("stable_text_review_defaults", {}) + if (not isinstance(defaults, dict) or set(defaults) - {"basis", "observed_on"} + or any(not isinstance(v, str) or not v.strip() for v in defaults.values())): + raise ValueError("Invalid stable review defaults") + reviews = [{**defaults, **review} if isinstance(review, dict) else review + for review in reviews] + anchor_ids = [anchor["id"] for anchor in acceptance.get("stable_text_anchors", [])] + if (any(not isinstance(review, dict) for review in reviews) + or len(reviews) != len(anchor_ids) + or {review.get("id") for review in reviews} != set(anchor_ids) + or any(not isinstance(review.get(key), str) or not review[key].strip() + for review in reviews for key in ("region", "basis", "observed_on"))): + raise ValueError("Every declared stable group requires its own source-region review") + return acceptance + + +def _base_case(case: dict, rule: dict, acceptance: dict) -> dict: + task = { + "id": case["id"], "name": case["name"], "url": case["url"], + "category": "article", "track": "articles", "task_kind": "body", + "task_prompt": case["task_prompt"], "expected_any": [], "min_content_chars": 1, + "acceptance_contract": acceptance, + "rule_review": {"status": "reviewed", "reason": rule["reason"]}, + } + components = {} + if "title_contract" in rule: + from .article_title import validate_title_contract + validate_title_contract(rule["title_contract"]) + components["title_contract"] = rule["title_contract"] + if "comment_contract" in rule: + from .article_comments import validate_comment_contract + validate_comment_contract(rule["comment_contract"]) + components["comment_contract"] = rule["comment_contract"] + if components: + task["source_benchmark"] = components + validate_case(EvalCase(**task)) + return task + + + +def _qualify_components(case: dict, rule: dict, task: dict, evidence_home: Path) -> dict: + """Validate historical qualification without certifying a future live run.""" + review = rule.get("comment_coverage_review") + if "comment_contract" not in rule and review is None: + return task + from .article_qualification import qualify_article_comments + contract = rule.get("comment_contract") + if review is not None: + if (not isinstance(review, dict) or review.get("status") not in ( + "visible_text", "observed_absent", "visible_placeholders_only") + or not isinstance(review.get("basis"), str) or not review["basis"].strip()): + raise ValueError("Invalid comment presence review") + if ((review["status"] == "visible_text") != (contract is not None) + or contract is not None and contract.get("version") != 2): + raise ValueError("Comment presence review and coverage contract disagree") + if "historical_contract" in review and ( + review["status"] != "visible_placeholders_only" + or not isinstance(review["historical_contract"], dict) + or review["historical_contract"].get("version") != 1): + raise ValueError("Only placeholders may retain a legacy authoring contract") + elif contract is not None and contract.get("version") == 2: + raise ValueError("Comment coverage requires a presence review") + if contract is None: + if not isinstance(review, dict) or review.get("status") not in ( + "observed_absent", "visible_placeholders_only"): + raise ValueError("Omitted comment rules require reviewed absence evidence") + + proof = rule.get("comment_qualification_reference", {}) + if not isinstance(proof.get("path"), str) or not proof.get("sha256"): + raise ValueError("Comment contracts require independent qualification evidence") + raw = contained_path(evidence_home, proof["path"], reject_symlinks=True).read_bytes() + if hashlib.sha256(raw).hexdigest() != proof["sha256"]: + raise ValueError("Changed article comment qualification reference") + reference = json.loads(raw) + checked = qualify_article_comments( + contract, reference, case_id=case["id"], url=case["url"]) + if checked["status"] != "qualified": + raise ValueError("Article comments require a complete independent observation") + if review is not None: + if not isinstance(review, dict) or not reference.get("source_evidence"): + raise ValueError("Comment presence review requires source evidence") + if review.get("status") == "visible_placeholders_only": + placeholders = review.get("placeholder_texts", []) + records = reference.get("records", []) + if ("comment_contract" in rule or not records + or not isinstance(placeholders, list) or not placeholders + or any(not isinstance(text, str) or not text.strip() for text in placeholders) + or any(row["text"].strip() not in placeholders for row in records)): + raise ValueError("Placeholder-only review must explain every observed record") + elif review.get("status") == "observed_absent" and reference.get("records"): + raise ValueError("Absent comment review contains observed records") + elif review.get("status") == "visible_text" and "comment_contract" not in rule: + raise ValueError("Visible comment text requires a coverage contract") + for source in reference.get("source_evidence", []): + data = contained_path(evidence_home, source["path"], reject_symlinks=True).read_bytes() + if hashlib.sha256(data).hexdigest() != source["sha256"]: + raise ValueError("Changed article comment source evidence") + task.setdefault("source_benchmark", {})["comment_qualification_proof"] = proof + if review is not None: + task["source_benchmark"]["comment_presence_review"] = { + "status": review["status"], "basis": review["basis"], + "observed_record_count": len(reference["records"]), + "qualification_only": True, + } + return task + + +def _compile_post_collection(case: dict, rule: dict, evidence_home: Path) -> dict: + from .article_posts import validate_post_contract + from .article_qualification import qualify_article_posts + if case.get("page_type") != "collection": + raise ValueError("Post collection rules require an explicitly reviewed collection") + contract = rule["post_contract"] + validate_post_contract(contract) + proof = rule["qualification_reference"] + raw = contained_path(evidence_home, proof["path"], reject_symlinks=True).read_bytes() + if hashlib.sha256(raw).hexdigest() != proof["sha256"]: + raise ValueError("Changed article collection qualification reference") + reference = json.loads(raw) + records = reference.get("records", []) + counts = reference.get("source_observation_counts", []) + if (reference.get("version") != 1 or reference.get("case_id") != case["id"] + or reference.get("url") != case["url"] + or reference.get("scope") != "initial_load_only" + or not isinstance(counts, list) or len(counts) < 2 + or any(type(count) is not int or count < 1 for count in counts) + or not isinstance(records, list) or not records or len(records) != counts[-1]): + raise ValueError("Article collections require independent initial-load record review") + sources = rule.get("post_coverage_review", {}).get("source_evidence", []) + if not sources: + raise ValueError("Static post coverage requires source evidence") + for source in sources: + captured = contained_path(evidence_home, source["path"], reject_symlinks=True).read_bytes() + if hashlib.sha256(captured).hexdigest() != source["sha256"]: + raise ValueError("Changed article collection source evidence") + qualification = qualify_article_posts( + contract, reference, case_id=case["id"], url=case["url"]) + if qualification["status"] != "qualified": + raise ValueError("Every source record needs complete owned post and quote text") + task = _base_case(case, rule, _acceptance(rule)) + task.update(ground_truth_main_content="", source_benchmark={ + **task.get("source_benchmark", {}), + "mode": "article_post_collection", "post_contract": contract, + "qualification_proof": proof, + }) + return _qualify_components(case, rule, task, evidence_home) + + +def compile_case(case: dict, rule: dict, evidence_home: Path) -> dict: + """Require a reviewed complete body; never turn a missing reference into a pass.""" + if rule.get("id") != case["id"] or rule.get("status") != "verified": + raise ValueError(f"Unresolved article rule: {case['id']}") + if rule.get("target_kind") == "post_collection": + return _compile_post_collection(case, rule, evidence_home) + if rule.get("target_kind") != "fixed_article" or case.get("page_type") == "collection": + raise ValueError("Collection/post-feed contracts require separate record review") + proof = rule["body_reference"] + path = contained_path(evidence_home, proof["path"], reject_symlinks=True) + raw = path.read_bytes() + if hashlib.sha256(raw).hexdigest() != proof["sha256"]: + raise ValueError(f"Changed article reference: {case['id']}") + body = raw.decode("utf-8") + if len(body) != proof["chars"] or not body.strip(): + raise ValueError("Empty or incorrectly bound article body") + scope = rule["reference_scope"] + if (scope.get("complete_initial_body") is not True + or not scope.get("start") or not scope.get("end") + or not body.lstrip().startswith(scope["start"]) + or not body.rstrip().endswith(scope["end"])): + raise ValueError("Article scope must bind the complete opening and ending") + acceptance = _acceptance(rule) + lines = body.splitlines() + ranges = rule.get("body_regions", []) + if not isinstance(ranges, list) or not ranges: + raise ValueError("Article scope requires reviewed contiguous semantic regions") + contract = body_requirements(body) + segmentation_version = rule.get("body_segmentation", {}).get("version") + if segmentation_version == 3 and len(ranges) > 5: + raise ValueError("Current article scope allows at most five body regions") + semantic = segmentation_version in (2, 3) + reviews = rule.get("body_region_review", []) + if semantic and (len(reviews) != len(ranges) + or rule["body_segmentation"].get("reference_sha256") != proof["sha256"] + or any(row.get("lines") != bounds + or not isinstance(row.get("label"), str) or not row["label"].strip() + for row, bounds in zip(reviews, ranges))): + raise ValueError("Every semantic region needs a source-bound readable label") + regions, next_line = [], 1 + for index, bounds in enumerate(ranges, 1): + if (not isinstance(bounds, list) or len(bounds) != 2 + or any(type(value) is not int for value in bounds) + or bounds[0] != next_line or not bounds[0] <= bounds[1] <= len(lines)): + raise ValueError("Article regions must cover every source line once in order") + source = "\n".join(lines[bounds[0] - 1:bounds[1]]).strip() + part = body_requirements(source, structure_hint="whole_declared_component") + if not part["regions"] or not compact_ngrams(source, 6): + raise ValueError("Article body cannot yield an empty coverage region") + region = part["regions"][0] + region.update(id=f"body-region-{index}", segmentation_method=( + "reviewed_source_semantics" if semantic else "reviewed_source_lines")) + if semantic: + region.update(label=reviews[index - 1]["label"], source_line_range=bounds) + regions.append(region) + next_line = bounds[1] + 1 + if next_line != len(lines) + 1: + raise ValueError("Article regions must include the complete source ending") + contract.update(version=4, ending={"text": scope["end"]}, + regions=regions, segmentation_method=( + "reviewed_source_semantics" if semantic else "reviewed_source_lines"), + minimum_region_passes=len(regions), + minimum_floor=contract["minimum_coverage"]) + task = { + **_base_case(case, rule, acceptance), + "ground_truth_main_content": body, + "body_requirements": contract, + } + return _qualify_components(case, rule, task, evidence_home) + + +def compile_dataset(evidence_home: Path, definition_dir: Path = DEFINITIONS) -> dict: + """Compile the whole approved population, refusing unresolved or omitted cases.""" + sources = {name: (definition_dir / name).read_bytes() + for name in ("cohort.json", "rules.json")} + cohort = json.loads(sources["cohort.json"]) + document = json.loads(sources["rules.json"]) + cases, rules = cohort["cases"], document["rules"] + ids = [case["id"] for case in cases] + if (cohort.get("schema_version") != 1 or document.get("schema_version") != 1 + or cohort.get("case_count") != len(cases) + or not ids or len(ids) != len(set(ids)) + or ids != [rule["id"] for rule in rules]): + raise ValueError("Article cohort and rule inventory differ") + compiled = [ + compile_case(case, rule, evidence_home) for case, rule in zip(cases, rules) + ] + return { + "schema_version": 1, "id": cohort.get("id", "articles-v1"), "name": "Article retrieval", + "tool": "web_fetch", "case_count": len(compiled), + "site_count": len({urlsplit(case["url"]).hostname for case in cases}), + "runs_per_case": 1, "timeout_ms": 45000, "workers": 1, + "retry_policy": {"declared_retries": 0, + "authoritative_response": "final_physical_call"}, + "interaction_policy": "initial_load_only_no_click_input_scroll_pagination_login_or_ocr", + "source_bindings": { + name.replace(".json", "_sha256"): hashlib.sha256(raw).hexdigest() + for name, raw in sources.items() + }, + "cases": compiled, + } diff --git a/src/lexbench_webfetch/article_posts.py b/src/lexbench_webfetch/article_posts.py new file mode 100644 index 0000000..fdfab93 --- /dev/null +++ b/src/lexbench_webfetch/article_posts.py @@ -0,0 +1,323 @@ +"""Post ownership checks for fixed initial coverage contracts.""" +import re +import unicodedata +from urllib.parse import urljoin, urlsplit + +import soupsieve + +from .body_coverage import tokens +from .dynamic_lists import named_entry_identity, visible_html_text +from .article_text import (article_html, is_article_html, article_markdown, TIMESTAMP_LINE, + mask_markdown_code, author_key, label_key, validate_labels, declared_labels) +from .reference_quality import markdown_links, visible_markdown_links + + +def validate_post_contract(contract): + """Raise ValueError for malformed frozen post extraction contracts.""" + try: + allowed = {'version', 'hosts', 'entry_path', 'html_record_selector', + 'html_permalink_selector', 'html_author_selector', 'html_body_selector', + 'html_quote_selector', 'quote_markers', + 'allow_empty_outer_with_quotes', 'minimum_posts', 'ignored_labels', 'unavailable_notices', 'non_author_labels'} + if not isinstance(contract, dict) or set(contract) - allowed: + raise ValueError('unknown frozen post contract fields') + if type(contract['version']) is not int or contract['version'] != 2 or not isinstance(contract['hosts'], list) or not contract['hosts']: + raise ValueError('invalid version or hosts') + if any(not isinstance(h, str) or not h or '/' in h for h in contract['hosts']): + raise ValueError('invalid hosts') + re.compile(contract['entry_path']) + for key in ('html_record_selector', 'html_permalink_selector', 'html_author_selector', 'html_body_selector'): + if not isinstance(contract[key], str) or not contract[key].strip(): + raise ValueError('empty selector') + soupsieve.compile(contract[key]) + soupsieve.compile(contract.get('html_quote_selector', 'blockquote')) + if type(contract.get('minimum_posts')) is not int or contract['minimum_posts'] < 1: + raise ValueError('invalid fixed coverage bound') + if type(contract.get('allow_empty_outer_with_quotes', False)) is not bool: + raise ValueError('invalid empty outer opt-in') + validate_labels(contract, ('ignored_labels', 'unavailable_notices', 'non_author_labels')) + markers = contract.get('quote_markers', []) + if not isinstance(markers, list) or any(not isinstance(x, str) or not x.strip() for x in markers): + raise ValueError('invalid quote markers') + except (KeyError, TypeError, re.error, soupsieve.SelectorSyntaxError) as error: + raise ValueError('invalid post contract') from error + + +def post_identity(value, contract, base): + destination = urljoin(base, value) + p = urlsplit(destination) + p.port # Force validation; hostname alone accepts malformed port text. + if (p.scheme not in {'http', 'https'} or p.hostname not in contract['hosts'] + or p.username or p.password or not re.fullmatch(contract['entry_path'], p.path.rstrip('/'))): + raise ValueError('invalid post URL') + return named_entry_identity(destination, contract) + + +def _owned_body_text(node, contract): + surface = article_html(str(node)) + # Field-node ownership alone does not exclude a nested card's text. + for nested in list(surface.select(contract['html_record_selector'])): + if nested.parent is not None: + nested.decompose() + return visible_html_text(str(surface)) + + +def _html_records(content, contract): + soup, rows = article_html(content), [] + selector = contract.get('html_quote_selector', 'blockquote') + for card in soup.select(contract['html_record_selector']): + def owned(node): + return node.find_parent(lambda x: soupsieve.match(contract['html_record_selector'], x)) is card + quotes = [x for x in card.select(selector) if owned(x) + and not x.find_parent(lambda p: p is not card and soupsieve.match(selector, p))] + def outer(node): + return owned(node) and not any(node is q or q in node.parents for q in quotes) + fields = [[x for x in card.select(contract[key]) if outer(x)] for key in + ('html_permalink_selector', 'html_author_selector', 'html_body_selector')] + optional_body = contract.get('allow_empty_outer_with_quotes', False) + if (any(len(x) != 1 for x in fields[:2]) + or len(fields[2]) != 1 and not (optional_body and not fields[2])): + rows.append({}) + continue + link, author = [x[0] for x in fields[:2]] + body = fields[2][0] if fields[2] else None + quote_text = [_owned_body_text(x, contract) for x in quotes] + quote_authors = [[a.get_text(' ', strip=True) + for a in x.select(contract['html_author_selector']) + if owned(a) and a.find_parent( + lambda p: soupsieve.match(selector, p)) is x] + for x in quotes] + quote_bodies = [] + for quote in quotes: + # Prefer an explicit quote body field. If the site's quote uses + # a different shape, exclude attribution and interface fields + # before accepting its remaining visible text as prose. + bodies = [node for node in quote.select(contract['html_body_selector']) + if owned(node) and node.find_parent( + lambda parent: soupsieve.match(selector, parent)) is quote] + copy = article_html(str(bodies[0] if len(bodies) == 1 else quote)) + metadata = (contract['html_author_selector'] + ', ' + + contract['html_permalink_selector'] + + ', time, button, [role="button"]') + for field in list(copy.select(metadata)): + if field.parent is not None: + field.decompose() + quote_bodies.append(_owned_body_text(copy, contract)) + quote.extract() + rows.append({'url': link.get('href', ''), 'author': author.get_text(' ', strip=True), + 'post': _owned_body_text(body, contract) if body is not None else '', 'quote': quote_text, + 'quote_bodies_after_author': quote_bodies, + 'quote_authors': quote_authors}) + return rows + + +def _post_headers(lines, contract, url): + """Identify owned permalink headings before parsing their body ranges.""" + headers = [] + header_level = None + for index, line in enumerate(lines): + heading = re.match(r'^\s*(#{1,6})\s+', line) + links = [link for link in markdown_links(line) if not link.is_image] + if not links or line.lstrip().startswith('>'): + if heading and header_level is not None and len(heading[1]) <= header_level: + break + continue + if header_level is None and heading: + header_level = len(heading[1]) + if len(links) > 1: + first, last = links[0], links[-1] + prefix, suffix = line[:first.start].strip(), line[last.end:].strip() + if (not re.match(r'^(?:#{1,6}|[-*+]|\d+[.)])(?:\s|$)', prefix) + or suffix.strip(' |—–-:*_')): + continue + author = visible_markdown_links(first.label) + destination = 'about:invalid-post-heading' + # A linked author profile may precede exactly one owning post + # permalink. Other explicit linked headings still end the prior + # record, but cannot supply a valid record of their own. + if (len(links) == 2 and re.fullmatch(r'(?:#{1,6}|[-*+]|\d+[.)])', prefix) + and not line[first.end:last.start].strip(' |·—–-:*_')): + try: + profile = urlsplit(urljoin(url, first.destination)) + profile.port + if (profile.scheme in {'http', 'https'} + and profile.hostname in contract['hosts'] + and not profile.username and not profile.password + and not re.fullmatch(contract['entry_path'], profile.path.rstrip('/'))): + post_identity(last.destination, contract, url) + destination = last.destination + except ValueError: + pass + headers.append((index, destination, visible_markdown_links(line), author)) + continue + link = links[0] + prefix, suffix = line[:link.start].strip(), line[link.end:].strip() + explicit_header = bool(re.match(r'^(?:#{1,6}|[-*+]|\d+[.)])(?:\s|$)', prefix)) + if suffix.strip(' |—–-:*_') or (prefix and not explicit_header): + continue + try: + post_identity(link.destination, contract, url) + except ValueError: + # An explicit linked post heading still ends the previous post, + # even when its own URL is invalid. Ordinary inline links do not. + if not explicit_header: + continue + header = visible_markdown_links(line) + author = re.sub(r'^(?:#{1,6}|[-*+]|\d+[.)])(?:\s+|$)', '', prefix).strip() + if not author: + author = visible_markdown_links(link.label) + if not prefix and index and lines[index-1].strip() and not lines[index-1].startswith('>'): + author = visible_markdown_links(lines[index-1]) + header = author+'\n'+header + headers.append((index, link.destination, header, author)) + return headers + + +def _markdown_records(content, contract, url): + # Fenced source snippets cannot introduce post identities or supply post prose. + # Comments keep code as owned text; posts deliberately require unfenced prose. + content, _ = mask_markdown_code(content) + # HTML visibility projection drops empty lines. Preserve Markdown paragraph + # boundaries across that projection so a blockquote cannot absorb a later + # unquoted paragraph. Choose a marker absent from the answer itself. + paragraph_marker = '\ue000post-paragraph-break\ue001' + while paragraph_marker in content: + paragraph_marker += '_' + content = re.sub(r'\n[ \t]*\n', '\n'+paragraph_marker+'\n', content) + content = visible_html_text(content, preserve_source_whitespace=True, preserve_links=True) + content = content.replace(paragraph_marker, '') + lines = content.splitlines() + headers = _post_headers(lines, contract, url) + rows = [] + markers = {x.casefold().rstrip(':：') for x in contract.get('quote_markers', [])} + for n, (start, destination, header, author) in enumerate(headers): + end = headers[n+1][0] if n+1 < len(headers) else len(lines) + post, quotes, current = [], [], None + quote_mode, paragraph_break = None, False + header_heading = re.match(r'^\s*(#{1,6})\s+', lines[start]) + for line in lines[start+1:end]: + heading = re.match(r'^\s*(#{1,6})\s+', line) + if heading and header_heading and len(heading[1]) <= len(header_heading[1]): + break + if line.strip().strip('*_').rstrip(':：').casefold() in markers: + if current is not None: + quotes.append('\n'.join(current)) + current = [] + quote_mode, paragraph_break = 'marker', False + elif line.lstrip().startswith('>'): + if current is None: + current = [] + quote_mode = 'blockquote' + paragraph_break = False + current.append(re.sub(r'^\s*>\s?', '', line)) + elif current is not None: + if quote_mode == 'blockquote' and paragraph_break and line.strip(): + # Lazy continuation belongs to the current paragraph; a + # new unquoted paragraph belongs to the owning outer post. + quotes.append('\n'.join(current)) + current, quote_mode = None, None + post.append(line) + else: + current.append(line) + if not line.strip(): + paragraph_break = True + else: + post.append(line) + if current is not None: + quotes.append('\n'.join(current)) + rows.append({'url': destination, 'author_header': header, 'author': author, + 'whole_post': visible_markdown_links('\n'.join(lines[start+1:end])), + 'post': visible_markdown_links('\n'.join(post)), + 'quote': [visible_markdown_links(x) for x in quotes], + 'quote_authors': [[visible_markdown_links(next( + (line.strip() for line in x.splitlines() if line.strip()), ''))] + for x in quotes], + 'quote_bodies_after_author': [visible_markdown_links('\n'.join( + x.lstrip().splitlines()[1:])) for x in quotes]}) + return rows + + +def _initial_surface(content): + """Apply shared closed-disclosure visibility to mixed representations.""" + return str(article_html(content)) if re.search(r'= contract['minimum_posts'], + 'article_post_results': checks, 'article_post_count': len(rows), + 'article_post_confirmed_count': confirmed, + 'article_post_minimum_count': contract['minimum_posts']} + + +def verify_article_posts(content, contract, *, url): + """Check fixed visible-post coverage without a runtime reference.""" + validate_post_contract(contract) + return _verify_fixed_posts(content, contract, url) diff --git a/src/lexbench_webfetch/article_qualification.py b/src/lexbench_webfetch/article_qualification.py new file mode 100644 index 0000000..170ea00 --- /dev/null +++ b/src/lexbench_webfetch/article_qualification.py @@ -0,0 +1,192 @@ +"""Pure historical article evidence qualification, independent of scoring.""" +from datetime import date, datetime +import re +from .article_comments import validate_comment_contract, valid_comment_records +from .article_posts import validate_post_contract, post_identity +from .article_text import author_key +from .body_coverage import body_requirements, compact_ngrams, tokens + +def _utc(value): + stamp = datetime.fromisoformat(value.replace('Z', '+00:00')) + if stamp.tzinfo is None or stamp.utcoffset().total_seconds() != 0: + raise ValueError('UTC timestamp required') + return stamp + +def _comment_evidence_status(reference, case_id, url): + if reference is None: + return 'missing' + try: + if (type(reference['version']) is not int or reference['version'] != 1 + or reference['case_id'] != case_id or reference['url'] != url + or reference['scope'] != 'initial_load_only' + or reference['observation_complete'] is not True + or not isinstance(reference['context_id'], str) or not reference['context_id']): + return 'invalid' + records, counts = reference['records'], reference['source_observation_counts'] + if (not isinstance(records, list) or not isinstance(counts, list) + or len(counts) < 1 + or any(type(n) is not int or n < 0 for n in counts) or counts[-1] != len(records)): + return 'invalid' + evidence = reference.get('source_evidence') + if len(counts) == 1: + if not isinstance(evidence, list) or len(evidence) != 1: + return 'invalid' + if evidence is not None and (not isinstance(evidence, list) or not evidence + or any(not isinstance(item, dict) or not isinstance(item.get('path'), str) + or not item['path'].strip() or not isinstance(item.get('sha256'), str) + or not re.fullmatch(r'[0-9a-f]{64}', item['sha256']) + or item.get('scope') != 'complete_initial_page' for item in evidence)): + return 'invalid' + seen = set() + for row in records: + if (not isinstance(row['id'], str) or not row['id'] or row['id'] in seen + or author_key(row['author']) is None + or not isinstance(row['text'], str) + or (not row['text'].strip() and row.get('empty_body') is not True) + or ('empty_body' in row and (row['empty_body'] is not True or row['text'] != '')) + or row['complete_initial_text'] is not True + or row['parent_id'] is not None and row['parent_id'] not in seen): + return 'invalid' + if ('image_only' in row or 'source_media_count' in row) and ( + row.get('image_only') is not True or row.get('empty_body') is not True + or row['text'] != '' or type(row.get('source_media_count')) is not int + or row['source_media_count'] < 1): + return 'invalid' + seen.add(row['id']) + by_id = {row['id']: row for row in records} + for row in records: + if row.get('empty_body') is not True: + continue + # Independent rendered-media evidence can explain an image-only + # leaf without inventing OCR prose. Otherwise an empty parent must + # own at least one displayed descendant with ordinary text. + if row.get('image_only') is True: + continue + descendant_text = False + for child in records: + parent = child['parent_id'] + while parent is not None: + if parent == row['id'] and child['text'].strip(): + descendant_text = True + break + parent = by_id[parent]['parent_id'] + if descendant_text: + break + if not descendant_text: + return 'invalid' + if not records and any(counts): + return 'invalid' + if 'observed_at' not in reference: + # Historical date-only evidence establishes authoring provenance, + # never a clock value, freshness, or an execution context match. + observed_on = reference.get('observed_on') + if (not isinstance(observed_on, str) or not re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}', observed_on) + or date.fromisoformat(observed_on).isoformat() != observed_on or not evidence): + return 'invalid' + else: + _utc(reference['observed_at']) + except (KeyError, ValueError, TypeError, AttributeError): + return 'invalid' + return 'verified' + +def qualify_article_comments(contract, reference, *, case_id, url): + """Validate historical evidence only, without runtime clocks or answers.""" + if contract is not None: + validate_comment_contract(contract) + status = _comment_evidence_status(reference, case_id, url) + result = {'status': 'qualified' if status == 'verified' else status, + 'observation_count': len(reference.get('source_observation_counts', [])) if isinstance(reference, dict) and isinstance(reference.get('source_observation_counts'), list) else 0, + 'comment_count': len(reference.get('records', [])) if isinstance(reference, dict) and isinstance(reference.get('records'), list) else 0} + if status == 'verified': + if contract is not None: + valid_count = len(valid_comment_records(reference['records'], contract)) + result['valid_comment_count'] = valid_count + if valid_count < contract['minimum_comments']: + result['status'] = 'insufficient_comments' + if 'observed_at' not in reference: + result['observed_on'] = reference['observed_on'] + return result + +def _part_contract(part): + text = part['text'] + contract = body_requirements(text) + if 'source_line_ranges' in part: + lines, cursor, regions = text.splitlines(), 1, [] + ranges = part['source_line_ranges'] + if not isinstance(ranges, list) or not 1 <= len(ranges) <= 5: + raise ValueError('invalid source ranges') + for span in ranges: + if (not isinstance(span, list) or len(span) != 2 or any(type(n) is not int for n in span) + or span[0] != cursor or not span[0] <= span[1] <= len(lines)): + raise ValueError('ranges must partition the whole body part') + region = body_requirements('\n'.join(lines[span[0]-1:span[1]]), + structure_hint='whole_declared_component')['regions'] + if len(region) != 1: + raise ValueError('empty source region') + regions.append({**region[0], 'id': f'post-region-{len(regions)+1}'}) + cursor = span[1] + 1 + if cursor != len(lines)+1: + raise ValueError('ranges omit the end') + contract['regions'] = regions + if any(not compact_ngrams(r['text'], 6) for r in contract['regions']): + raise ValueError('body regions require six-character matching units') + contract['minimum_region_passes'] = len(contract['regions']) + contract['minimum_floor'] = contract['minimum_coverage'] + return contract + +def _post_evidence_status(reference, contract, case_id, url): + if reference is None: + return 'missing' + try: + if (type(reference['version']) is not int or reference['version'] != 1 or reference['case_id'] != case_id or reference['url'] != url + or reference['scope'] != 'initial_load_only' + or not isinstance(reference['context_id'], str) or not reference['context_id']): + return 'invalid' + _utc(reference['observed_at']) + records, seen = reference['records'], set() + if not isinstance(records, list) or not records: + return 'invalid' + for row in records: + key = post_identity(row['url'], contract, url) + if key in seen or not isinstance(row['author'], str) or not tokens(row['author']): + return 'invalid' + seen.add(key) + parts = row['body_parts'] + if (not isinstance(parts, list) or not parts or parts[0].get('kind') != 'post' + or any(p.get('kind') != 'quote' for p in parts[1:])): + return 'invalid' + for index, part in enumerate(parts): + if (part['kind'] not in {'post', 'quote'} or part['complete_initial_text'] is not True + or not isinstance(part['text'], str)): + return 'invalid' + if 'author' in part and (not isinstance(part['author'], str) or not tokens(part['author'])): + return 'invalid' + if 'empty_outer_text' in part: + # This independent observation cannot excuse omitted prose or + # an empty quote. Every owned quote still needs attribution. + if (not contract.get('allow_empty_outer_with_quotes', False) + or part['empty_outer_text'] is not True or index != 0 + or part['text'] != '' or 'source_line_ranges' in part or len(parts) < 2 + or any(not isinstance(p.get('author'), str) or not tokens(p['author']) + for p in parts[1:])): + return 'invalid' + continue + if not tokens(part['text']): + return 'invalid' + _part_contract(part) + except (KeyError, TypeError, ValueError, AttributeError): + return 'invalid' + return 'verified' + +def qualify_article_posts(contract, reference, *, case_id, url): + """Validate complete historically observed post ownership and count.""" + validate_post_contract(contract) + status = _post_evidence_status(reference, contract, case_id, url) + if status == 'verified': + counts = reference.get('source_observation_counts') + if (not isinstance(counts, list) or len(counts) < 2 + or any(type(n) is not int or n < 1 for n in counts) + or counts[-1] != len(reference['records']) + or min(counts) < contract['minimum_posts']): + status = 'invalid' + return {'status': 'qualified' if status == 'verified' else status} diff --git a/src/lexbench_webfetch/article_text.py b/src/lexbench_webfetch/article_text.py new file mode 100644 index 0000000..267b3bd --- /dev/null +++ b/src/lexbench_webfetch/article_text.py @@ -0,0 +1,177 @@ +"""Shared article-only visible text and identity policy.""" +import re +import unicodedata +from .dynamic_lists import scoped_html +from .body_coverage import tokens +from .reference_quality import is_html_document_or_fragment, markdown_links + +def author_key(value): + """Keep literal symbol-only identities instead of comparing empty tokens.""" + if not isinstance(value, str): + return None + value = unicodedata.normalize('NFKC', value) + # Joiners matter in rendered names and emoji sequences. Other invisible + # format/control characters cannot create a username by themselves. + value = ''.join(char for char in value + if not unicodedata.category(char).startswith('C') or char in '\u200c\u200d') + value = ' '.join(value.split()) + if not any(not char.isspace() and not unicodedata.category(char).startswith('C') for char in value): + return None + words = tokens(value) + return ('tokens', value.startswith('@'), tuple(words)) if words else ('literal', value) + +def article_html(content): + """Apply shared visibility and remove article-only fallback text.""" + soup = scoped_html(content) + for fallback in list(soup.select('noscript')): + if fallback.parent is not None: + fallback.decompose() + return soup + +def without_markdown_images(content): + """Image alt labels are metadata, including images nested in normal links.""" + pieces, cursor = [], 0 + for link in markdown_links(content): + if link.start < cursor: + continue + pieces.append(content[cursor:link.start]) + if not link.is_image: + original = content[link.start:link.end] + pieces.append(original.replace(link.label, without_markdown_images(link.label), 1)) + cursor = link.end + pieces.append(content[cursor:]) + return ''.join(pieces) + +def is_article_html(content): + """Classify HTML after complete leading non-content blocks only.""" + remaining = content.lstrip() + # Attributes may contain a greater-than sign inside quoted values. + attributes = r"(?:[^>\"']|\"[^\"]*\"|'[^']*')*" + while True: + prefix = re.match( + r'|<(noscript|style|script)\b' + attributes + + r'>.*?|<(?:meta|link)\b' + attributes + r'>', + remaining, re.I | re.S, + ) + if not prefix: + return is_html_document_or_fragment(remaining) + remaining = remaining[prefix.end():].lstrip() + + +TIMESTAMP_LINE = re.compile( + r'(?:\d{4}[-/]\d{1,2}[-/]\d{1,2}(?:[ T]\d{1,2}:\d{2}' + r'(?::\d{2})?(?:Z|[+-]\d{2}:?\d{2})?)?' + r'|\d{1,2}:\d{2}(?::\d{2})?(?:\s*[AP]M)?' + r'|\d+\s+(?:seconds?|minutes?|hours?|days?|weeks?|months?|years?)\s+ago' + r'|\d+\s*(?:秒|分钟|小时|天|周|个月|年)前|刚刚)', re.I) + + +def mask_markdown_code(content, *, retain=False): + """Shield fenced and indented literals before article record recognition. + + Retained blocks may belong to an existing comment, but their headings and + links cannot introduce a title, section or author. Tabs use four-column stops. + """ + prefix = '\ue000article-code' + while prefix in content: + prefix += '_' + lines = content.splitlines(keepends=True) + output, blocks = [], {} + i = 0 + while i < len(lines): + start = i + line = lines[i] + fence = re.match(r'^ {0,3}(`{3,}|~{3,})(.*)$', line.rstrip('\r\n')) + if fence and not (fence[1][0] == '`' and '`' in fence[2]): + run = fence[1] + i += 1 + closing = re.compile(r'^ {0,3}' + re.escape(run[0]) + '{' + str(len(run)) + r',}[ \t]*$') + while i < len(lines): + end = bool(closing.fullmatch(lines[i].rstrip('\r\n'))) + i += 1 + if end: + break + elif line.strip() and line.expandtabs(4).startswith(' '): + # Treat serializer indentation as literal, never as record syntax. + i += 1 + while i < len(lines) and (not lines[i].strip() or lines[i].expandtabs(4).startswith(' ')): + i += 1 + else: + output.append(line) + i += 1 + continue + key = prefix + str(len(blocks)) + blocks[key] = ''.join(lines[start:i]) + output.append('\n' + (key if retain else '') + '\n') + return ''.join(output), blocks + + +def article_markdown(content): + """Project cleaned HTML while retaining headings, links and ownership stops. + + This is used only when no reviewed native record containers remain. It is + not a competing parser whose more permissive count can override native DOM + ownership. Navigation and explicit related-content landmarks are excluded. + """ + from bs4 import NavigableString, Tag + soup = article_html(content) + for node in list(soup.select('nav, aside, footer, [role="navigation"], [role="complementary"]')): + if node.parent is not None: + node.decompose() + def render(node): + if isinstance(node, NavigableString): + return str(node) + if not isinstance(node, Tag): + return '' + if node.name == 'pre': + literal = node.get_text() + longest = max((len(run) for run in re.findall(r'`+', literal)), default=0) + fence = '`' * max(3, longest + 1) + return '\n\n' + fence + '\n' + literal + '\n' + fence + '\n\n' + if node.name == 'code': + # Inline literal syntax must not create a record or section boundary. + literal = node.get_text().replace('\n', ' ') + return '`' + literal.replace('`', '\\`') + '`' + if node.name == 'tr': + cells = [render(child).replace('\n', ' ').strip() + for child in node.children if isinstance(child, Tag) + and child.name in ('td', 'th')] + return '\n| ' + ' | '.join(cells) + ' |\n' + text = ''.join(render(child) for child in node.children) + if node.name == 'img': + return '' + if node.name == 'a': + return '[' + text.strip() + '](' + node.get('href', '') + ')' + if node.name in ('strong', 'b'): + return '**' + text.strip() + '**' + if re.fullmatch(r'h[1-6]', node.name): + return '\n\n' + '#' * int(node.name[1]) + ' ' + text.strip() + '\n\n' + if node.name == 'blockquote': + return '\n\n' + '\n'.join('> ' + line for line in text.strip().splitlines()) + '\n\n' + if node.name == 'br': + return '\n' + if node.name in ('p', 'div', 'section', 'article', 'ul', 'ol', 'li'): + return '\n\n' + text.strip() + '\n\n' + return text + return render(soup) + + +def label_key(text): + """Normalize whole-label formatting identically for rules and observations.""" + return ' '.join(unicodedata.normalize('NFKC', text).split()).casefold().strip(':：*_').strip() + + +def validate_labels(contract, fields=('ignored_labels', 'unavailable_notices')): + """Require explicit task labels; an empty list is valid, an omission is not.""" + for field in fields: + labels = contract.get(field) + if (not isinstance(labels, list) or any(not isinstance(label, str) + or not label_key(label) for label in labels)): + raise ValueError('Invalid or missing task-declared ' + field) + if len({label_key(label) for label in labels}) != len(labels): + raise ValueError('Duplicate task-declared ' + field) + + +def declared_labels(contract, field): + """Read the validated, explicit task list without runtime defaults.""" + return frozenset(label_key(label) for label in contract[field]) diff --git a/src/lexbench_webfetch/article_title.py b/src/lexbench_webfetch/article_title.py new file mode 100644 index 0000000..7ea0243 --- /dev/null +++ b/src/lexbench_webfetch/article_title.py @@ -0,0 +1,82 @@ +"""Verify the independently reviewed article title as a separate component.""" +import re + +import soupsieve + +from .article_text import article_html, is_article_html, without_markdown_images, mask_markdown_code +from .metrics import normalize_for_match +from .reference_quality import visible_mixed_html_text + + +def _title_key(text): + normalized = normalize_for_match(text).strip() + # DOM line breaks around punctuation are layout, not different wording. + return re.sub(r'\s*([，。！？、；：,.!?;:])\s*', r'\1', normalized) + + +def validate_title_contract(contract): + """Reject malformed frozen title contracts before evaluating a response.""" + try: + if (not isinstance(contract, dict) + or set(contract) - {'version', 'alternatives', 'html_selector', + 'excluded_html_selectors'} + or not {'version', 'alternatives', 'html_selector'} <= set(contract) + or type(contract['version']) is not int or contract['version'] != 1): + raise ValueError('invalid fields or version') + alternatives = contract['alternatives'] + if (not isinstance(alternatives, list) or not alternatives + or any(not isinstance(text, str) or not text.strip() + or '\n' in text or '\r' in text or not _title_key(text) + for text in alternatives)): + raise ValueError('nonempty observed single-line titles required') + if not isinstance(contract['html_selector'], str) or not contract['html_selector'].strip(): + raise ValueError('reviewed title selector required') + soupsieve.compile(contract['html_selector']) + exclusions = contract.get('excluded_html_selectors', []) + if (not isinstance(exclusions, list) + or any(not isinstance(selector, str) or not selector.strip() + for selector in exclusions)): + raise ValueError('reviewed hidden title selectors must be a list') + for selector in exclusions: + soupsieve.compile(selector) + except (KeyError, TypeError, soupsieve.SelectorSyntaxError) as error: + raise ValueError('invalid article title contract') from error + + +def verify_article_title(content, contract): + """Require a visible selected HTML title or an exact standalone text line. + + Explicit visibility follows the shared HTML projection. External stylesheet + visibility must be reviewed when authoring source contracts. Plain text + cannot establish DOM ownership, but a title mention inside prose cannot + substitute for the standalone title component. + """ + validate_title_contract(contract) + if not isinstance(content, str): + raise ValueError('article title content must be text') + expected = {_title_key(text) for text in contract['alternatives']} + candidates = [] + if is_article_html(content): + soup = article_html(content) + for selector in contract.get('excluded_html_selectors', []): + for node in list(soup.select(selector)): + if node.parent is not None: + node.decompose() + candidates = [node.get_text(' ', strip=True) + for node in soup.select(contract['html_selector'])] + surface = 'reviewed_html_title_selector' + else: + content, _ = mask_markdown_code(content) + if re.search(r'

Archive content.

", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_task_pass": true + } + }, + { + "id": "visible-control-hidden-button", + "category": "answer-surface", + "root_cause": "visible_control_attribute_scope", + "failure_mode": "false-positive", + "case": { + "id": "visible-control-regression", + "url": "https://example.test/archive", + "task_kind": "body", + "min_content_chars": 1, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_role": "required", + "stable_text_anchors": [ + { + "id": "control", + "alternatives": [ + "Search archive" + ], + "visible_control": { + "selector": "input", + "attribute": "value" + } + } + ] + } + }, + "artifact": "

Archive content.

", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_task_pass": false + } + }, + { + "id": "native-disclosure-closed-details", + "category": "answer-surface", + "root_cause": "closed_disclosure_visibility", + "failure_mode": "false-positive", + "case": { + "id": "native-disclosure-regression", + "url": "https://example.test/article", + "task_kind": "body", + "min_content_chars": 1, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_role": "required", + "stable_text_anchors": [ + { + "id": "visible-label", + "alternatives": [ + "Visible explanation" + ] + } + ] + } + }, + "artifact": "
OverviewVisible explanation
", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_task_pass": false + } + }, + { + "id": "native-disclosure-open-details", + "category": "answer-surface", + "root_cause": "closed_disclosure_visibility", + "failure_mode": "false-negative", + "case": { + "id": "native-disclosure-regression", + "url": "https://example.test/article", + "task_kind": "body", + "min_content_chars": 1, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_role": "required", + "stable_text_anchors": [ + { + "id": "visible-label", + "alternatives": [ + "Visible explanation" + ] + } + ] + } + }, + "artifact": "
OverviewVisible explanation
", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_task_pass": true + } + }, + { + "id": "native-disclosure-closed-summary", + "category": "answer-surface", + "root_cause": "closed_disclosure_visibility", + "failure_mode": "false-negative", + "case": { + "id": "native-disclosure-regression", + "url": "https://example.test/article", + "task_kind": "body", + "min_content_chars": 1, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_role": "required", + "stable_text_anchors": [ + { + "id": "visible-label", + "alternatives": [ + "Visible explanation" + ] + } + ] + } + }, + "artifact": "
Visible explanationHidden body
", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_task_pass": true + } } ], "superseded_cases": { diff --git a/src/lexbench_webfetch/definitions/registry.json b/src/lexbench_webfetch/definitions/registry.json index fccf498..450781f 100644 --- a/src/lexbench_webfetch/definitions/registry.json +++ b/src/lexbench_webfetch/definitions/registry.json @@ -44,6 +44,17 @@ "source-inputs.json": "c61d6825abbff2a92098592f83055347cd0b0101374794023046c0fd0323dd5f" }, "directory": "evaluation/risk-control/current" + }, + { + "id": "articles", + "dataset_id": "articles-v11", + "case_count": 100, + "directory": "evaluation/articles", + "files": { + "tasks.json": "171241ea0bf3cb3f9482de14a7201885733fa717af56d7ea2b28d86a4516a3bb", + "dataset-catalog.json": "8e413e683c309674abf3d297df3f378e76d65ead1e2aa343828e4b0e1d90764a", + "source-inputs.json": "0d9c729f14dae8abb3323a2cf4734736c4733957c9d74d5484033d204642b91b" + } } ] } diff --git a/src/lexbench_webfetch/dynamic_lists.py b/src/lexbench_webfetch/dynamic_lists.py index 4bb5bb9..f4b76c9 100644 --- a/src/lexbench_webfetch/dynamic_lists.py +++ b/src/lexbench_webfetch/dynamic_lists.py @@ -59,7 +59,7 @@ def _is_named_label(title,url,parts,nonempty=False): return label not in {_label_key(value) for value in route_labels} -def _canonical_entry_path(path): +def canonical_entry_path(path): """Normalize IRI text without decoding path separators or delimiters.""" def decode(match): encoded = match[0] @@ -92,7 +92,7 @@ def _markdown_safe_destination(destination): def named_entry_identity(url, contract): """Return the contract's canonical linked-record identity.""" parts = urlsplit(url) - identity = _canonical_entry_path(parts.path).rstrip('/') + identity = canonical_entry_path(parts.path).rstrip('/') if pattern := contract.get('entry_identity_pattern'): match = re.search(pattern, identity) if not match or not match.groups(): @@ -119,7 +119,7 @@ def named_entry_source_pair_hash(title, url, contract): return hashlib.sha256((_label_key(title) + '\0' + identity).encode()).hexdigest() -def _scoped_html(content): +def scoped_html(content): """Exclude explicitly hidden nodes and image metadata from visible names. This is not a CSS renderer: external stylesheet visibility still needs @@ -149,6 +149,15 @@ def _scoped_html(content): or declarations.get('visibility') in {'hidden', 'collapse'} or declarations.get('opacity') in {'0', '0.0', '0%'}): node.decompose() + # Native closed disclosures render only the first direct summary. Apply + # this before extracting any body, anchor or record text. + for details in list(soup.select('details:not([open])')): + if details.parent is None: + continue + summary = details.find('summary', recursive=False) + for child in list(details.contents): + if child is not summary: + child.extract() # An image description is not a visible news/product title. Explicit # title_selector extraction below already uses actual text nodes only. for node in soup.select('img'): @@ -170,7 +179,7 @@ def visible_html_text( blocks = HTML_BLOCK_ELEMENTS parts = [] - stack = [(_scoped_html(content), False, False)] + stack = [(scoped_html(content), False, False)] while stack: node, closing, preformatted = stack.pop() if isinstance(node, Comment): @@ -223,7 +232,7 @@ def _named_candidate(url, title, contract, base_url, excluded_titles, diagnostic or parts.hostname not in contract['hosts'] or not re.fullmatch( contract['entry_path'], - _canonical_entry_path(parts.path).rstrip('/'), + canonical_entry_path(parts.path).rstrip('/'), )): return None title = re.sub(r'<[^>]+>', ' ', unescape(title)) @@ -358,7 +367,7 @@ def _breadcrumb_scope_end(content, contract, base_url): from .reference_quality import markdown_links, visible_markdown_links page = urlsplit(base_url) def route(path): - return re.sub(r'/index\.html?$', '', _canonical_entry_path(path)).rstrip('/') + return re.sub(r'/index\.html?$', '', canonical_entry_path(path)).rstrip('/') current = route(page.path) offset = 0 chain = [] @@ -407,7 +416,7 @@ def _scope_named_content(content, scope, *, contract=None, base_url=''): """Apply one declared HTML or text scope and return owned name/link pairs.""" owned_items = [] if _has_scoped_html(content): - soup = _scoped_html(content) + soup = scoped_html(content) owners = soup.select(scope['html_selector']) # Inline HTML labels in Markdown do not make a document HTML. # A real document still owns its entries through the HTML selector. @@ -540,12 +549,12 @@ def _owned_model_caption(content, links, index): finish = next((other.start for other in links[index + 1:] if not other.is_image), len(content)) block = content[link.end:finish] - identity = _canonical_entry_path(urlsplit(link.destination).path).rstrip('/').rsplit('/', 1)[-1] + identity = canonical_entry_path(urlsplit(link.destination).path).rstrip('/').rsplit('/', 1)[-1] if not identity or re.search(r"`+|~{3,}", block): return '' images = [image for image in markdown_links(block) if image.is_image and re.search(rf"(? dict: + list_reference=None, + status_code=None, title=None, final_url=None) -> dict: metadata = dict(metadata or {}) # Acceptance of delivered files can only be derived from actual bytes. metadata.pop('file_evaluation', None) if case.get('task_kind') == 'file' or files: metadata['file_evaluation'] = evaluate_files(case, list(files)) metrics = score_attempt(case, response, ok=ok, metadata=metadata, - list_reference=list_reference, status_code=status_code, + list_reference=list_reference, + status_code=status_code, title=title, final_url=final_url) return {'passed': task_passed(metrics.get('request_ok', ok), metrics, case.get('task_kind', 'body')), 'measured': task_measured(metrics, case.get('task_kind', 'body')), diff --git a/src/lexbench_webfetch/metrics.py b/src/lexbench_webfetch/metrics.py index 9664199..3a86997 100644 --- a/src/lexbench_webfetch/metrics.py +++ b/src/lexbench_webfetch/metrics.py @@ -445,6 +445,8 @@ def task_passed(ok: bool, metrics: dict, task_kind: str | None = None) -> bool: if (metrics.get('page_identity_required') and metrics.get('page_identity_status') != 'verified'): return False + if any(value is not True for value in metrics.get('required_components', {}).values()): + return False if not task_measured(metrics, task_kind) or metrics.get('deadline_overrun'): return False if "stable_task_pass" in metrics: diff --git a/src/lexbench_webfetch/models.py b/src/lexbench_webfetch/models.py index ff1a711..a569dde 100644 --- a/src/lexbench_webfetch/models.py +++ b/src/lexbench_webfetch/models.py @@ -8,7 +8,7 @@ TRACKS = { 'web': '普通网页正文', 'dynamic': '动态网站正文', 'challenge': '热门网站正文', - 'risk_control': '风控网页正文', + 'risk_control': '风控网页正文', 'articles': '文章内容', 'document': '文档正文', 'image': '图片获取', 'video': '视频获取', } diff --git a/src/lexbench_webfetch/required_components.py b/src/lexbench_webfetch/required_components.py new file mode 100644 index 0000000..c719d1c --- /dev/null +++ b/src/lexbench_webfetch/required_components.py @@ -0,0 +1,32 @@ +"""Explicit optional acceptance components shared by scoring and task export.""" +from typing import Callable, NamedTuple +from .article_title import verify_article_title +from .article_comments import verify_article_comments +from .article_posts import verify_article_posts + + +class Component(NamedTuple): + name: str + verify: Callable + pass_metric: str + digest_metric: str + + +def _title(content, contract, url): + return verify_article_title(content, contract) + + +def _comments(content, contract, url): + return verify_article_comments(content, contract) + + +def _posts(content, contract, url): + # Relative permalinks require the task URL, unlike title/comment text. + return verify_article_posts(content, contract, url=url) + + +COMPONENTS = { + "title_contract": Component("article_title", _title, "article_title_pass", "article_title_contract_sha256"), + "comment_contract": Component("article_comments", _comments, "article_comments_pass", "article_comment_contract_sha256"), + "post_contract": Component("article_posts", _posts, "article_posts_pass", "article_post_contract_sha256"), +} diff --git a/src/lexbench_webfetch/scoring.py b/src/lexbench_webfetch/scoring.py index 3f20cf9..04312a1 100644 --- a/src/lexbench_webfetch/scoring.py +++ b/src/lexbench_webfetch/scoring.py @@ -21,7 +21,8 @@ "list_verification.py", "reference_quality.py", "dynamic_entry_matching.py", "response_input.py", "body_coverage.py", "structured_tables.py", "ordered_tree.py", "composite_tasks.py", "dynamic_reference.py", - "file_acceptance.py", "evaluation.py", "models.py", "paths.py", + "file_acceptance.py", "evaluation.py", "models.py", "paths.py", "article_posts.py", + "article_comments.py", "article_title.py", "article_text.py", "required_components.py", "resource_urls.py", )) SCORING_SOURCE_SHA256 = hashlib.sha256(b"\0".join( @@ -33,7 +34,7 @@ def _case_dict(case: EvalCase | dict[str, Any]) -> dict[str, Any]: def load_list_reference(case: EvalCase | dict[str, Any], source: Path) -> dict | None: - """Load only evidence required by the legacy list verifier, never by stable tasks.""" + """Load declared independent list/post evidence, never infer it from an answer.""" case = _case_dict(case) benchmark = case.get("source_benchmark", {}) if (not case.get("acceptance_contract") @@ -58,7 +59,7 @@ def load_list_reference(case: EvalCase | dict[str, Any], source: Path) -> dict | "acceptable_final_urls", "ordered_tree", "ordered_tree_similarity", "composite_acceptance_sha256", - "body_requirements_sha256", + "body_requirements_sha256", "required_components", } @@ -123,7 +124,7 @@ def score_attempt( answer_input(artifact, metadata, ok=ok, status_code=status_code)) result = {key: value for key, value in (previous_metrics or {}).items() if key not in _CONDITIONAL_FIELDS - and not key.startswith(("stable_", "list_", "page_context_"))} + and not key.startswith(("stable_", "list_", "page_context_", "article_"))} for retired in ('acceptance_policy_version', 'forbidden_hits', 'reference_term_exemptions'): result.pop(retired, None) keys = ("required_facts", "forbidden_any", "diagnostic_any", "min_content_chars", @@ -300,6 +301,22 @@ def score_attempt( result["resource_acceptance_contract_sha256"] = hashlib.sha256(json.dumps( resource_contract, sort_keys=True, ensure_ascii=False).encode()).hexdigest() benchmark = case.get("source_benchmark", {}) + if rule_review.get("status") != "unresolved": + from .required_components import COMPONENTS + required = {} + for field, component in COMPONENTS.items(): + if field not in benchmark: + continue + component_contract = benchmark[field] + component_result = component.verify(content, component_contract, case["url"]) + result.update(component_result) + required[component.name] = component_result[component.pass_metric] is True + result[component.digest_metric] = hashlib.sha256(json.dumps( + component_contract, sort_keys=True, ensure_ascii=False).encode()).hexdigest() + if required: + result["content_decision_basis"] = "base_acceptance_and_required_components" + result["required_components"] = required + result["content_pass"] = bool(result["content_pass"] and all(required.values())) if (rule_review.get("status") != "unresolved" and not contract and benchmark.get("mode") == "dynamic_list_structure"): result.update(evaluate_dynamic_list(content, benchmark, base_url, ok, links)) diff --git a/src/lexbench_webfetch/stable_tasks.py b/src/lexbench_webfetch/stable_tasks.py index 0be8d7a..aa085fb 100644 --- a/src/lexbench_webfetch/stable_tasks.py +++ b/src/lexbench_webfetch/stable_tasks.py @@ -139,8 +139,8 @@ def _visible(text, schema=None, *, normalized_html=False): text = visible_html_text(text) if schema == "name_description" and not normalized_html and not projected_html and re.search(r"<(?:div|section|article|span|script|template)\b", text, re.I): - from .dynamic_lists import _scoped_html - text = _scoped_html(text).get_text(separator="\n") + from .dynamic_lists import scoped_html + text = scoped_html(text).get_text(separator="\n") projected_html = True text = clean_reference_text(text) # An actual inline HTML break in Markdown is layout. Preserve escaped @@ -626,7 +626,7 @@ def _non_record_visible(content, contract, records, base_url): """Remove owned record links, without erasing identical navigation text.""" from urllib.parse import urljoin - from .dynamic_lists import _scoped_html, named_entry_identity, _canonical_entry_path, visible_html_text + from .dynamic_lists import scoped_html, named_entry_identity, canonical_entry_path, visible_html_text from .reference_quality import markdown_links, visible_markdown_links, visible_mixed_html_text identities = {named_entry_identity(r['fields']['url'], contract) @@ -641,7 +641,7 @@ def is_record(url): return (named_entry_identity(absolute, contract) in identities or (parts.hostname in contract['hosts'] and re.fullmatch(contract['entry_path'], - _canonical_entry_path(parts.path).rstrip('/')) is not None)) + canonical_entry_path(parts.path).rstrip('/')) is not None)) except ValueError: return False @@ -662,7 +662,7 @@ def is_record(url): if target.scheme in {'http', 'https'} and target.hostname and not is_record(owner['href']): image.name = 'span' image.string = image['alt'] - soup = _scoped_html(str(soup)) + soup = scoped_html(str(soup)) for node in list(soup.select('a[href]')): if node.parent is None or node.attrs is None: continue @@ -786,6 +786,46 @@ def field_matches(record, fields, field): value, registered_words)] +def _validate_visible_control(control): + """Validate the narrowly opted-in ordinary control attribute surface.""" + if (not isinstance(control, dict) or set(control) != {"selector", "attribute"} + or not isinstance(control["selector"], str) or not control["selector"].strip() + or not isinstance(control["attribute"], str) + or control["attribute"] not in {"placeholder", "value"}): + raise ValueError("Invalid visible control declaration") + import soupsieve + try: + soupsieve.compile(control["selector"]) + except (soupsieve.SelectorSyntaxError, NotImplementedError) as error: + raise ValueError("Invalid visible control selector") from error + + +def _ordinary_control_text(soup, control): + """Read only declared rendered-label attributes, never accessibility metadata. + + Explicit hidden nodes are removed by scoped_html. External stylesheet + visibility must be established by the independent source review. + """ + parts = [] + for node in soup.select(control["selector"]): + kind = str(node.get("type", "text")).lower() + attribute = control["attribute"] + # A populated control does not display its placeholder, even when the + # current value contains only whitespace. + if attribute == "placeholder" and ( + node.get("value", "") if node.name == "input" else node.get_text()): + continue + permitted = ( + attribute == "placeholder" and ( + node.name == "textarea" or (node.name == "input" and kind in { + "text", "search", "email", "url", "tel", "number"}))) or ( + attribute == "value" and node.name == "input" + and kind in {"button", "submit", "reset"}) + if permitted and isinstance(node.get(attribute), str): + parts.append(node[attribute]) + return "\n".join(parts) + + def validate_acceptance_contract(contract: dict) -> None: """Fail closed before a caller chooses stable scoring instead of snapshots.""" if (not isinstance(contract, dict) or type(contract.get("version")) is not int @@ -882,6 +922,7 @@ def validate_acceptance_contract(contract: dict) -> None: or set(anchor) - { "id", "alternatives", "match_scope", "source_location", "observation_status", "source_evidence_sha256", + "visible_control", } or ('match_scope' in anchor and (anchor['match_scope'] != 'non_record_lines' or contract['schema'] != 'named_link')) or ('source_location' in anchor and ( @@ -897,6 +938,8 @@ def validate_acceptance_contract(contract: dict) -> None: or not isinstance(anchor.get("alternatives"), list) or not anchor["alternatives"] or any(not isinstance(value, str) or not _visible(value).strip() for value in anchor["alternatives"])): raise ValueError("Invalid or duplicate stable text anchor group") + if "visible_control" in anchor: + _validate_visible_control(anchor["visible_control"]) ids.add(anchor["id"]) if contract["schema"] == "body_text" and not anchors and not patterns: raise ValueError("Stable body-text contracts require text anchors or page patterns") @@ -1051,6 +1094,9 @@ def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligi same-site detail-link parser. Raw URL identity is not factual verification. anchor_text may reuse the caller's normalized visible HTML prose; document-title metadata cannot supply page text or structural records. + An anchor may opt into visible_control {selector, attribute} to recognize + independently reviewed ordinary input placeholders or button values in HTML. + This surface belongs only to that anchor, not page identity or body coverage. """ validate_acceptance_contract(contract) schema = contract["schema"] @@ -1175,6 +1221,12 @@ def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligi ) ) anchors = [] + control_soup = None + if any("visible_control" in a for a in contract.get("stable_text_anchors", [])): + from .reference_quality import is_html_document_or_fragment + if is_html_document_or_fragment(content): + from .dynamic_lists import scoped_html + control_soup = scoped_html(content) non_record_visible = None if schema == 'named_link' and any(a.get('match_scope') == 'non_record_lines' for a in contract.get('stable_text_anchors', [])): @@ -1183,8 +1235,15 @@ def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligi for value in anchor["alternatives"] if re.fullmatch(r"[A-Za-z]+", value)) for anchor in contract.get("stable_text_anchors", []): + selected_surface = anchor_visible + selected_non_record = non_record_visible + if control_soup is not None and "visible_control" in anchor: + control_text = _ordinary_control_text(control_soup, anchor["visible_control"]) + selected_surface = (anchor_visible if anchor_visible is not None else visible) + "\n" + control_text + if non_record_visible is not None: + selected_non_record = non_record_visible + "\n" + control_text matched = _anchor_matches(anchor, records, visible, schema, - non_record_visible, anchor_visible, registered_words) + selected_non_record, selected_surface, registered_words) if original_markdown_visible is not None: literal_matches = _anchor_matches( anchor, records, visible, schema, non_record_visible, diff --git a/src/lexbench_webfetch/structured_tables.py b/src/lexbench_webfetch/structured_tables.py index d78888b..cc7a630 100644 --- a/src/lexbench_webfetch/structured_tables.py +++ b/src/lexbench_webfetch/structured_tables.py @@ -221,8 +221,8 @@ def table_text_surface(content): from bs4 import BeautifulSoup content = _without_example_fences(content) if _contains_rendered_html_table(content): - from .dynamic_lists import _scoped_html - content = str(_scoped_html(content)) + from .dynamic_lists import scoped_html + content = str(scoped_html(content)) content = re.sub( r"]*>[\s\S]*?", lambda m: "\n" + BeautifulSoup(m[0], "html.parser").get_text(" ", strip=True) + "\n", @@ -343,8 +343,8 @@ def parse_table_records(content, visible, contract): root = None if (re.search(r'<(?:html|body|main|div|section|article|ul|ol)\b', content, re.I) or _contains_rendered_html_table(content)): - from .dynamic_lists import _scoped_html - soup = _scoped_html(content) + from .dynamic_lists import scoped_html + soup = scoped_html(content) root = soup if scope.get('root_selector'): roots = soup.select(scope['root_selector']) diff --git a/tests/article_contract_fixtures.py b/tests/article_contract_fixtures.py new file mode 100644 index 0000000..96b3742 --- /dev/null +++ b/tests/article_contract_fixtures.py @@ -0,0 +1,13 @@ +"""Explicit synthetic label contracts for parser controls; not runtime defaults.""" +from copy import deepcopy + +COMMENT_LABELS = {'ignored_labels': ['add a comment', 'leave a reply', 'like', 'likes', 'load more', 'log in', 'login', 'read more', 'register', 'replies', 'reply', 'report', 'share', 'show more', 'sign in', '加载更多', '发表评论', '回复', '展开', '展开回复', '查看更多', '注册', '添加评论', '点赞', '登录', '登录后评论'], 'unavailable_notices': ['[deleted]', '[removed]', '[内容不可见]', '[已删除]', 'this comment has been hidden', 'this comment has been hidden (marked as spam)']} +POST_LABELS = {'ignored_labels': ['bookmark', 'bookmarks', 'like', 'likes', 'load more', 'log in', 'permalink', 'read more', 'replies', 'reply', 'report', 'repost', 'reposts', 'share', 'show more', 'sign in', 'today', 'view', 'views', 'yesterday', '今天', '分享', '加载更多', '回复', '展开', '收藏', '昨天', '查看', '查看更多', '注册', '点赞', '登录', '转发'], 'unavailable_notices': [], 'non_author_labels': ['today', 'yesterday', 'date', 'time', 'permalink', 'view post', 'post', '今天', '昨天', '详情', '查看帖子']} + + +def comment_labels(): + return deepcopy(COMMENT_LABELS) + + +def post_labels(): + return deepcopy(POST_LABELS) diff --git a/tests/test_article_cleaned_html.py b/tests/test_article_cleaned_html.py new file mode 100644 index 0000000..6a336ae --- /dev/null +++ b/tests/test_article_cleaned_html.py @@ -0,0 +1,106 @@ +"""Cleaned HTML retains semantic boundaries, never competes with native records.""" +from article_contract_fixtures import comment_labels, post_labels +from copy import deepcopy +import pytest +from lexbench_webfetch.article_comments import verify_article_comments +from lexbench_webfetch.article_posts import verify_article_posts + +COMMENTS = {**comment_labels(), 'version': 2, 'minimum_comments': 2, 'record_selector': '.comment', + 'author_selector': '.author', 'body_selector': '.body', 'section_markers': ['Comments'], + 'text_scope': {'body_end': 'Article ending.', 'end_markers': ['Related articles']}} +POSTS = {**post_labels(), 'version': 2, 'minimum_posts': 2, 'hosts': ['example.test'], 'entry_path': r'/post/\d+', + 'html_record_selector': 'article.post', 'html_permalink_selector': 'a.permalink', + 'html_author_selector': '.author', 'html_body_selector': '.body'} + + +def comments(text): + return verify_article_comments(text, COMMENTS) + + +def posts(text): + return verify_article_posts(text, POSTS, url='https://example.test/feed') + + +CLEAN_COMMENTS = '

Article ending.

Comments

Alice

First reply.

Bob

Second reply.

' +CLEAN_POSTS = '

Alice Today

First post.

Bob Today

Second post.

' + + +def test_cleaned_heading_comment_html_matches_natural_markdown(): + assert comments(CLEAN_COMMENTS)['article_comments_pass'] + assert comments('Article ending.\n## Comments\n### Alice\nFirst reply.\n### Bob\nSecond reply.')['article_comments_pass'] + + +def test_cleaned_linked_post_html_preserves_author_and_destination(): + assert posts(CLEAN_POSTS)['article_posts_pass'] + + +@pytest.mark.parametrize('wrapper', ['', '', '', + '', '
More{}
']) +def test_cleaned_hidden_and_unrelated_landmarks_cannot_fill_floors(wrapper): + assert not comments(wrapper.format(CLEAN_COMMENTS))['article_comments_pass'] + assert not posts(wrapper.format(CLEAN_POSTS))['article_posts_pass'] + + +def test_native_malformed_records_cannot_be_overruled_by_cleaned_fallback(): + bad_comment = '
A
' + assert not comments(bad_comment + CLEAN_COMMENTS)['article_comments_pass'] + bad_post = '
A
' + assert not posts(bad_post + CLEAN_POSTS)['article_posts_pass'] + + +def test_cleaned_recommendation_section_does_not_supply_comments_or_posts(): + assert not comments('

Article ending.

Comments

Alice

Real reply.

' + '

Related articles

Bob

Story summary.

')['article_comments_pass'] + assert not posts('

Alice Today

Real post.

' + '

Related articles

' + CLEAN_POSTS)['article_posts_pass'] + + +def test_missing_post_body_cannot_borrow_following_section_text(): + assert not posts('

Alice Today

' + '

Related articles

Story summary.

' + CLEAN_POSTS)['article_posts_pass'] + + +def test_repeated_body_ending_before_explicit_heading_does_not_open_article_body(): + text = ('Article ending.\n### A caption\nImage credit.\n### Another caption\nImage credit.\n' + 'Article ending.\n## Comments') + assert comments(text)['article_comment_count'] == 0 + + +def test_comment_quoting_body_ending_does_not_move_start_past_real_comments(): + text = ('Article ending.\n## Comments\n### Alice\nFirst reply.\n' + '### Bob\nI quote: Article ending.') + assert comments(text)['article_comments_pass'] + + +def test_unlabelled_comment_subheading_collision_is_documented_not_guessed(): + # The same string can mean a second author "Explanation" or a subheading + # inside Alice's comment. The current text contract cannot disambiguate it. + text = 'Article ending.\n## Comments\n### Alice\nFirst reply.\n### Explanation\nMore words.' + assert comments(text)['article_comment_count'] == 2 + + +def test_mixed_author_like_body_link_does_not_inflate_heading_records(): + # This can also be a second linked author. Lacking ownership metadata, the + # conservative parser keeps it with the established heading-owned comment. + text = ('Article ending.\n## Comments\n### Alice\nFirst reply.\n\n' + '[Documentation](https://example.test/docs)\nMore words.') + assert comments(text)['article_comment_count'] == 1 + + +@pytest.mark.parametrize('literal', ['### Fake author\nNot a second comment.', + '```\n### Fake author\nNot a second comment.\n```']) +def test_cleaned_pre_cannot_create_comment_records(literal): + html = '

Article ending.

Comments

Alice

Real reply.

' + literal + '
' + assert comments(html)['article_comment_count'] == 1 + assert not comments(html)['article_comments_pass'] + + +def test_cleaned_table_cell_cannot_create_comment_record(): + html = ('

Article ending.

Comments

Alice

Real reply.

' + '
### Fake author
Not a comment.
') + assert comments(html)['article_comment_count'] == 1 + + +def test_partial_native_fields_cannot_borrow_unowned_cleaned_records(): + html = '

Alice

Unidentified field.

' + CLEAN_COMMENTS + assert not comments(html)['article_comments_pass'] diff --git a/tests/test_article_cli.py b/tests/test_article_cli.py new file mode 100644 index 0000000..3aba687 --- /dev/null +++ b/tests/test_article_cli.py @@ -0,0 +1,84 @@ +"""The standalone scorer must preserve the independent reference boundary.""" +from article_contract_fixtures import post_labels +import json +import subprocess +import sys + +import pytest + + +@pytest.mark.parametrize("include_body,passed", [(True, True), (False, False)]) +def test_cli_scores_fixed_posts_without_runtime_reference(tmp_path, include_body, passed): + body = ("Orchard workers recorded every tree before sunrise. They inspected " + "the irrigation channels and published the complete harvest schedule.") + task = { + "id": "post-feed", "name": "Article feed", "url": "https://example.test/feed", + "category": "article", "track": "articles", "task_kind": "body", "expected_any": [], + "acceptance_contract": {"version": 1, "schema": "body_text", "minimum_records": 0, + "page_patterns": ["Article feed"]}, + "source_benchmark": {"mode": "article_post_collection", "post_contract": {**post_labels(), + "version": 2, "hosts": ["example.test"], "entry_path": r"/post/[0-9]+", + "html_record_selector": "article", "html_permalink_selector": "a.time", + "html_author_selector": ".author", "html_body_selector": ".body", + "minimum_posts": 1, + }}, + } + (tmp_path / "task.json").write_text(json.dumps(task)) + (tmp_path / "answer.txt").write_text( + "# Article feed\n\n## Alice [Today](https://example.test/post/1)\n\n" + (body if include_body else "")) + command = [sys.executable, "-m", "lexbench_webfetch.cli", "score", + "--task", str(tmp_path / "task.json"), + "--response", str(tmp_path / "answer.txt")] + result = subprocess.run(command, cwd=tmp_path, text=True, capture_output=True) + assert result.returncode == 0, result.stderr + scored = json.loads(result.stdout) + assert scored["passed"] is passed + assert scored["measured"] is True + assert "article_reference_status" not in scored["metrics"] + + +@pytest.mark.parametrize("argument", ["--comments-reference", "--reference-context"]) +def test_cli_rejects_removed_reference_arguments(tmp_path, argument): + result = subprocess.run([ + sys.executable, "-m", "lexbench_webfetch.cli", "score", + "--task", str(tmp_path / "unused-task.json"), + "--response", str(tmp_path / "unused-answer.txt"), + argument, str(tmp_path / "unused-reference.json"), + ], cwd=tmp_path, text=True, capture_output=True) + assert result.returncode == 2 + assert "unrecognized arguments" in result.stderr + + +def test_compile_articles_cli_and_output_protection(tmp_path, monkeypatch, capsys): + from lexbench_webfetch import cli, article_dataset + dataset = {"case_count": 1, "cases": [{"id": "synthetic"}]} + def compile_candidate(evidence, definitions): + assert evidence == tmp_path / "evidence" + return dataset + monkeypatch.setattr(article_dataset, "compile_dataset", compile_candidate) + output = tmp_path / "candidate.json" + monkeypatch.setattr(sys, "argv", ["lexbench-webfetch", "compile-articles", + "--evidence-home", str(tmp_path / "evidence"), "--output", str(output)]) + cli.main() + assert json.loads(output.read_text()) == dataset + assert json.loads(capsys.readouterr().out)["case_count"] == 1 + with pytest.raises(SystemExit) as error: + cli.main() + assert error.value.code == 1 + assert "already exists" in capsys.readouterr().err + assert json.loads(output.read_text()) == dataset + output.unlink() + (tmp_path / "source-inputs.json").write_text("{}") + with pytest.raises(SystemExit): + cli.main() + assert "immutable" in capsys.readouterr().err + assert not output.exists() + + +def test_compile_articles_cli_rejects_missing_evidence(tmp_path): + output = tmp_path / "candidate.json" + result = subprocess.run([sys.executable, "-m", "lexbench_webfetch.cli", + "compile-articles", "--evidence-home", str(tmp_path / "missing"), + "--output", str(output)], text=True, capture_output=True) + assert result.returncode == 1 + assert not output.exists() diff --git a/tests/test_article_comment_boundaries.py b/tests/test_article_comment_boundaries.py new file mode 100644 index 0000000..6be526e --- /dev/null +++ b/tests/test_article_comment_boundaries.py @@ -0,0 +1,136 @@ +"""Reviewed text boundaries prevent unrelated page records from counting.""" +from article_contract_fixtures import comment_labels +from copy import deepcopy + +import pytest + +from lexbench_webfetch.article_comments import validate_comment_contract, verify_article_comments + +CONTRACT = {**comment_labels(), + 'version': 2, 'minimum_comments': 2, 'record_selector': '.comment', + 'author_selector': '.author', 'body_selector': '.body', + 'section_markers': ['Comments', 'Post your reply'], + 'text_scope': {'body_end': 'The article ends right here.', + 'end_markers': ['Recommended articles', 'About this site']}, +} + + +def check(content, contract=CONTRACT): + return verify_article_comments(content, contract) + + +COMMENTS = '\n\n**Alice**\nFirst actual reply.\n\n**Bob**\nSecond actual reply.' + + +@pytest.mark.parametrize('ending', [ + 'The article ends right here.', + 'The article ends\nright here.', + 'The **article** ends [right here](https://example.test/end).', +]) +def test_body_end_accepts_replies_without_input_placeholder(ending): + assert check(ending + COMMENTS)['article_comment_count'] == 2 + + +@pytest.mark.parametrize('heading', ['Comments', '## Comments', 'Post your reply']) +def test_optional_marker_after_body_ending_still_works(heading): + assert check('The article ends right here.\n' + heading + COMMENTS)['article_comments_pass'] + + +def test_navigation_and_article_captions_cannot_fill_comment_floor(): + content = ('Comments\n\n**Photo author**\nArticle image caption.\n\n' + '**Other author**\nArticle quotation.\nThe article ends right here.') + assert check(content)['article_comment_count'] == 0 + + +@pytest.mark.parametrize('footer', ['Recommended articles', '## Recommended articles', 'About this site']) +def test_plain_or_heading_footer_does_not_supply_fake_comments(footer): + content = ('The article ends right here.\nComments\n\n**Alice**\nOne actual reply.\n\n' + + footer + '\n\n[News author](https://example.test/other)\nOther article summary.') + result = check(content) + assert result['article_comment_count'] == 1 + assert not result['article_comments_pass'] + + +def test_missing_body_ending_is_not_an_extra_exact_text_requirement(): + assert check('Different or truncated article ending.\n## Comments' + COMMENTS)['article_comments_pass'] + + +def test_missing_body_and_missing_heading_fail_closed(): + assert not check('Home\nComments' + COMMENTS)['article_comments_pass'] + + +def test_comment_form_below_records_does_not_discard_earlier_comments(): + assert check('The article ends right here.' + COMMENTS + '\nPost your reply')['article_comments_pass'] + + +def test_explicit_comment_heading_stops_at_same_level_even_without_known_footer(): + content = ('The article ends right here.\n## Comments\n### Alice\nOne reply.\n' + '## Unrelated section\n### News author\nOther article summary.') + assert check(content)['article_comment_count'] == 1 + + +def test_colon_records_supported_after_reviewed_body_ending_without_ui_marker(): + content = 'The article ends right here.\n\nAlice: First actual reply.\n\nBob: Second actual reply.' + assert check(content)['article_comments_pass'] + + +def test_known_footer_inside_fenced_comment_is_literal_prose(): + content = ('The article ends right here.\n## Comments\n### Alice\n' + '```\nRecommended articles\n```\n### Bob\nSecond reply.') + assert check(content)['article_comments_pass'] + + +@pytest.mark.parametrize('scope', [None, {}, {'body_end': 'end'}, + {'body_end': '', 'end_markers': []}, {'body_end': 'end', 'end_markers': ['']}, + {'body_end': 'end', 'end_markers': 'Footer'}, + {'body_end': 'end', 'end_markers': [], 'extra': True}]) +def test_scope_contract_rejects_malformed_boundaries(scope): + contract = deepcopy(CONTRACT) + contract['text_scope'] = scope + with pytest.raises(ValueError): + validate_comment_contract(contract) + + +def test_metadata_between_body_and_comment_heading_preserves_heading_stop(): + contract = deepcopy(CONTRACT) + contract['text_scope']['end_markers'] = [] + content = ('The article ends right here.\n12 likes\n## Comments\n' + '### Alice\nOne real reply.\n## Related\n### Other author\nOther summary.') + result = check(content, contract) + assert result['article_comment_count'] == 1 + assert not result['article_comments_pass'] + + +@pytest.mark.parametrize('ending_present', [True, False]) +@pytest.mark.parametrize('footer', ['Post your reply', '## Post your reply']) +def test_reviewed_end_marker_wins_over_overlapping_start_marker(ending_present, footer): + contract = deepcopy(CONTRACT) + contract['text_scope']['end_markers'].append('Post your reply') + prefix = 'The article ends right here.\n' if ending_present else '' + result = check(prefix + footer + COMMENTS, contract) + assert result['article_comment_count'] == 0 + assert not result['article_comments_pass'] + + +def test_earlier_comments_still_count_when_later_form_is_also_start_marker(): + contract = deepcopy(CONTRACT) + contract['text_scope']['end_markers'].append('Post your reply') + result = check('The article ends right here.' + COMMENTS + '\nPost your reply' + COMMENTS, contract) + assert result['article_comments_pass'] + assert result['article_comment_count'] == 2 + + +@pytest.mark.parametrize('ending,full_body', [ + ('t really coding - I just see stuff, and it mostly works.', + "A longer post. It's not really coding - I just see stuff, and it mostly works."), + ('t really coding - I just see stuff, and it mostly wor', + "A longer post. It's not really coding - I just see stuff, and it mostly works."), + ('t really coding - I just see stuff, and it mostly works.', + "A longer post. It's not **really** coding - I just see\nstuff, and it mostly works."), +]) +def test_character_sliced_ending_matches_inside_full_body_tokens(ending, full_body): + contract = deepcopy(CONTRACT) + contract['text_scope']['body_end'] = ending + result = check(full_body + COMMENTS, contract) + assert result['article_comments_pass'] + assert result['article_comment_count'] == 2 diff --git a/tests/test_article_comment_coverage.py b/tests/test_article_comment_coverage.py new file mode 100644 index 0000000..c96269d --- /dev/null +++ b/tests/test_article_comment_coverage.py @@ -0,0 +1,152 @@ +"""Static initial-comment coverage does not freeze changing discussion text.""" +from article_contract_fixtures import comment_labels +from copy import deepcopy + +import pytest + +from lexbench_webfetch.article_comments import validate_comment_contract, verify_article_comments +from lexbench_webfetch.article_qualification import qualify_article_comments + +CONTRACT = {**comment_labels(), 'version': 2, 'minimum_comments': 2, 'record_selector': '.comment', + 'author_selector': '.author', 'body_selector': '.text', + 'section_markers': ['Comments', '评论'], 'id_attribute': 'data-id'} + + +def check(content, contract=CONTRACT): + return verify_article_comments(content, contract) + + +def card(author, body, identity=''): + return f'
{author}
{body}
' + + +@pytest.mark.parametrize('content', [ + card('New user', 'New comment', '1') + card('Another user', 'Another comment', '2'), + '## Comments\n### New user\nNew comment\n### Another user\nAnother comment', + '## Comments\n### New user\nNew comment\n#### Another user\nAnother comment', + '## Comments\n\nNew user:\nNew comment\n\nAnother user:\nAnother comment', + '## Comments\n\nNew user: New comment\n\nAnother user: Another comment', + 'Comments\n\n[New user](https://example.test/u)\nNew comment\n\n**Another user**\nAnother comment', + 'Comments\nPingback: [First source](https://a.test)\nPingback: [Second source](https://b.test)', + card('New user', 'New comment' + card('Another user', 'Another comment', '2'), '1'), +]) +def test_static_coverage_accepts_changed_names_text_and_flat_or_nested_records(content): + result = check(content) + assert result['article_comments_pass'] + assert result['article_comment_count'] == 2 + assert result['article_comment_minimum'] == 2 + assert result['article_comments_mode'] == 'fixed_coverage' + assert 'article_comments_reference_status' not in result + + +@pytest.mark.parametrize('content', [ + card('Only one', 'Text'), + card('', 'Text') + card('Name', ''), + '', + '
Comments' + card('A', 'One') + card('B', 'Two') + '
', + card('A', 'text') + card('B', ''), + '## Comments\n### Login\nSign in\n### Load more\nClick here', + '## Comments\n### A\n\n### B\n', + '## Comments\n```\n### A\nFake\n### B\nFake\n```', + '## Comments\n### A\nOne\n## Related articles\n### B\nTwo', + '## Comments\nJust a paragraph with no author.\nAnother paragraph.', + card('A', 'One', 'same') + card('B', 'Two', 'same'), + '## Comments\n### A\nOne\n### A\nOne', + card('A', card('B', 'Two', 'child'), 'parent'), +]) +def test_missing_hidden_control_unattributed_and_duplicate_records_fail(content): + assert not check(content)['article_comments_pass'] + + +def test_equal_comments_with_distinct_source_ids_are_separate_records(): + assert check(card('A', 'Same text', '1') + card('A', 'Same text', '2'))['article_comments_pass'] + + +def test_more_records_than_minimum_pass_and_symbol_content_is_allowed(): + result = check(card('????', '👍', '1') + card('A', 'One', '2') + card('B', 'Two', '3')) + assert result['article_comments_pass'] + assert result['article_comment_count'] == 3 + + +@pytest.mark.parametrize('changes', [ + {**comment_labels(), 'minimum_comments': 0}, {**comment_labels(), 'minimum_comments': True}, {**comment_labels(), 'minimum_comments': 1.5}, + {'parent_record_selector': '.comment'}, {'max_reference_age_seconds': 900}, + {'mode': 'observed_absent'}, {'record_selector': '['}, {'section_markers': []}, +]) +def test_static_contract_rejects_invalid_floor_and_legacy_requirements(changes): + with pytest.raises(ValueError): + validate_comment_contract(dict(CONTRACT, **changes)) + + +def test_authoring_qualification_requires_enough_nonempty_textual_records(): + reference = {'version': 1, 'case_id': 'case', 'url': 'https://example.test/', + 'scope': 'initial_load_only', 'observation_complete': True, + 'context_id': 'authoring', 'observed_on': '2026-10-19', + 'source_observation_counts': [2], + 'source_evidence': [{'path': 'capture.json', 'sha256': 'a' * 64, 'scope': 'complete_initial_page'}], + 'records': [ + {'id': '1', 'author': 'A', 'text': 'One', 'parent_id': None, 'complete_initial_text': True}, + {'id': '2', 'author': 'B', 'text': 'Two', 'parent_id': '1', 'complete_initial_text': True}]} + result = qualify_article_comments(CONTRACT, reference, case_id='case', url=reference['url']) + assert result['status'] == 'qualified' + assert result['valid_comment_count'] == 2 + changed = deepcopy(reference) + changed['records'][1].update(text='', empty_body=True, image_only=True, source_media_count=1) + result = qualify_article_comments(CONTRACT, changed, case_id='case', url=reference['url']) + assert result['status'] == 'insufficient_comments' + assert result['valid_comment_count'] == 1 + + +@pytest.mark.parametrize('placeholder', ['[内容不可见]', '[deleted]', 'This comment has been hidden', 'This comment has been hidden (marked as Spam)']) +def test_unavailable_content_notice_is_not_comment_prose(placeholder): + assert not check(card('A', placeholder, '1') + card('B', placeholder, '2'))['article_comments_pass'] + # A quoted notice inside an actual comment is not discarded. + assert check(card('A', placeholder + ' I disagree with this.', '1') + card('B', 'This is my comment.', '2'))['article_comments_pass'] + + +def test_links_and_colon_prose_inside_one_heading_comment_do_not_inflate_count(): + text = ('## Comments\n### Alice\nOne real comment\n\n' + '[Documentation](https://example.test/docs)\nUseful text\n\n' + 'Example: this is still the same comment\n\n' + 'Note:\nMore of the same comment') + result = check(text) + assert result['article_comment_count'] == 1 + assert not result['article_comments_pass'] + + +@pytest.mark.parametrize('stamp', ['2026-10-20 10:00', '2026-10-20', '10:20 AM', '3 hours ago', '3小时前', '刚刚']) +def test_author_and_timestamp_without_comment_text_does_not_pass(stamp): + text = f'## Comments\n### Alice\n{stamp}\nReply\n### Bob\n{stamp}\nReply' + assert check(text)['article_comment_count'] == 0 + with_text = text.replace('\nReply', '\nActual comment text.\nReply') + assert check(with_text)['article_comments_pass'] + # An HTML-owned body is actual comment text, even when it is just a date. + assert check(card('Alice', stamp, '1') + card('Bob', stamp, '2'))['article_comments_pass'] + + + +def test_plain_navigation_comment_label_does_not_make_article_captions_comments(): + text = ('Home\nComments\nArticle title\nFull article opening.\n\n' + 'Image source: Example magazine\nMore article prose.\n\n' + 'Photo: Example photographer\nStill article prose.\n\n' + '▲ 图片来自: Example site\nArticle ending.') + assert check(text)['article_comment_count'] == 0 + assert not check(text)['article_comments_pass'] + + +@pytest.mark.parametrize('number,expected', [ + (5,6),(9,1),(14,3),(17,4),(23,8),(25,204),(27,1),(30,18), + (32,20),(44,24),(47,3),(49,1),(52,4),(53,4),(68,11),(92,12), + (93,16),(95,56),(98,40),(103,20), +]) +def test_published_comment_floors_accept_boundary_and_reject_one_fewer(number, expected): + import json + from lexbench_webfetch.article_dataset import DEFINITIONS + rules = json.loads((DEFINITIONS / 'rules.json').read_text())['rules'] + rule = next(row for row in rules if row['id'] == f'article-{number:03}') + contract = rule['comment_contract'] + assert contract['minimum_comments'] == expected + records = [f'### Current reader {i}\nCurrent comment prose {i}.' for i in range(expected)] + marker = '## ' + contract['section_markers'][0] + '\n' + assert check(marker + '\n\n'.join(records), contract)['article_comments_pass'] + assert not check(marker + '\n\n'.join(records[:-1]), contract)['article_comments_pass'] diff --git a/tests/test_article_comments.py b/tests/test_article_comments.py new file mode 100644 index 0000000..1a72438 --- /dev/null +++ b/tests/test_article_comments.py @@ -0,0 +1,88 @@ +"""Historical comment qualification is separate from the current runtime.""" +from article_contract_fixtures import comment_labels +from copy import deepcopy +import pytest +from lexbench_webfetch.article_comments import validate_comment_contract, verify_article_comments +from lexbench_webfetch.article_qualification import qualify_article_comments + +CONTRACT = {**comment_labels(), 'version': 2, 'minimum_comments': 1, 'record_selector': '.comment', + 'author_selector': '.author', 'body_selector': '.text', 'section_markers': ['Comments']} +REFERENCE = {'version': 1, 'case_id': 'article', 'url': 'https://example.test/article', + 'scope': 'initial_load_only', 'observation_complete': True, + 'context_id': 'historical', 'observed_on': '2026-10-19', + 'source_observation_counts': [1], + 'source_evidence': [{'path': 'capture.json', 'sha256': 'a' * 64, 'scope': 'complete_initial_page'}], + 'records': [{'id': '1', 'author': 'Alice', 'text': 'Actual comment.', + 'parent_id': None, 'complete_initial_text': True}]} + + +def qualify(reference, contract=CONTRACT): + return qualify_article_comments(contract, reference, case_id='article', url=REFERENCE['url']) + + +def test_historical_date_only_evidence_qualifies_without_runtime_clock(): + assert qualify(REFERENCE)['status'] == 'qualified' + timestamped = deepcopy(REFERENCE) + timestamped['observed_at'] = '2020-01-01T00:00:00Z' + assert qualify(timestamped)['status'] == 'qualified' + + +@pytest.mark.parametrize('change', [ + {'version': True}, {'case_id': 'other'}, {'url': 'https://other.test'}, + {'scope': 'expanded'}, {'observation_complete': False}, {'context_id': ''}, + {'source_observation_counts': []}, {'source_observation_counts': None}, + {'source_observation_counts': [True]}, {'source_observation_counts': [2]}, + {'source_evidence': []}, {'observed_on': '2026-02-31'}, {'observed_at': 'bad'}, + {'records': None}, +]) +def test_malformed_or_inconsistent_historical_evidence_fails(change): + ref = deepcopy(REFERENCE) + ref.update(change) + assert qualify(ref)['status'] != 'qualified' + + +@pytest.mark.parametrize('change', [ + {'id': ''}, {'author': ''}, {'author': '\u200b'}, {'text': ''}, + {'parent_id': 'missing'}, {'complete_initial_text': False}, + {'text': '', 'empty_body': True}, + {'text': '', 'empty_body': True, 'image_only': True, 'source_media_count': 0}, +]) +def test_invalid_historical_owned_records_fail(change): + ref = deepcopy(REFERENCE) + ref['records'][0].update(change) + assert qualify(ref)['status'] != 'qualified' + + +def test_image_only_records_are_evidence_but_do_not_meet_text_floor(): + ref = deepcopy(REFERENCE) + ref['records'][0].update(text='', empty_body=True, image_only=True, source_media_count=1) + assert qualify(ref, None)['status'] == 'qualified' + assert qualify(ref)['status'] == 'insufficient_comments' + + +def test_zero_evidence_qualifies_presence_review_without_fabricated_contract(): + ref = deepcopy(REFERENCE) + ref.update(records=[], source_observation_counts=[0]) + assert qualify(ref, None)['status'] == 'qualified' + assert qualify(ref)['status'] == 'insufficient_comments' + + +def test_repeated_ids_and_bad_source_binding_fail(): + ref = deepcopy(REFERENCE) + ref['records'] *= 2 + ref['source_observation_counts'] = [2] + assert qualify(ref)['status'] == 'invalid' + ref = deepcopy(REFERENCE) + ref['source_evidence'][0]['sha256'] = 'bad' + assert qualify(ref)['status'] == 'invalid' + + +def test_runtime_explicitly_rejects_legacy_contracts_and_reference_argument(): + legacy = dict(CONTRACT, version=1, max_reference_age_seconds=900) + legacy.pop('minimum_comments') + with pytest.raises(ValueError): + validate_comment_contract(legacy) + with pytest.raises(ValueError): + verify_article_comments('', legacy) + with pytest.raises(TypeError): + verify_article_comments('', CONTRACT, REFERENCE) diff --git a/tests/test_article_component_integration.py b/tests/test_article_component_integration.py new file mode 100644 index 0000000..5c29b67 --- /dev/null +++ b/tests/test_article_component_integration.py @@ -0,0 +1,120 @@ +"""Additive component acceptance must apply through every public scoring boundary.""" +from article_contract_fixtures import comment_labels +import copy +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from lexbench_webfetch.benchmark import public_task +from lexbench_webfetch.evaluation import evaluate_response +from lexbench_webfetch.scoring import score_attempt + + +class ArticleComponentIntegration(unittest.TestCase): + def setUp(self): + self.comment = {**comment_labels(), + "version": 2, "record_selector": ".comment", "author_selector": ".author", + "body_selector": ".text", "section_markers": ["Comments"], + "minimum_comments": 1, + } + self.title = {"version": 1, "alternatives": ["Life and dignity"], + "html_selector": "h1"} + self.case = {"id": "article-test", "name": "Life and dignity", + "url": "https://example.test/article", "category": "article", + "expected_any": [], "task_kind": "body", + "acceptance_contract": {"version": 1, "schema": "body_text", + "minimum_records": 0, "page_patterns": ["Life and dignity"], + "stable_text_role": "required", "stable_text_anchors": []}, + "source_benchmark": {"title_contract": self.title, + "comment_contract": self.comment}} + self.complete = "# Life and dignity\n\nThe author writes Life and dignity again.\n\nComments\n\n## Reader\nA thoughtful book." + + def run_score(self, content): + return evaluate_response(self.case, content.encode()) + + def test_title_body_and_comments_are_independently_required(self): + self.assertTrue(self.run_score(self.complete)["passed"]) + self.assertFalse(self.run_score(self.complete.split("Comments")[0])["passed"]) + self.assertFalse(self.run_score(self.complete.replace("# Life and dignity\n", ""))["passed"]) + self.assertFalse(self.run_score(self.complete.replace("A thoughtful book.", ""))["passed"]) + # Original repetition is valid and remains untouched. + self.assertIn("The author writes Life and dignity again.", self.complete) + + def test_removed_runtime_reference_arguments_are_rejected(self): + for argument in ('comments_reference', 'reference_context'): + with self.subTest(argument=argument), self.assertRaises(TypeError): + evaluate_response(self.case, self.complete.encode(), **{argument: {}}) + + def test_public_projection_keeps_component_contracts(self): + projected = public_task(self.case) + self.assertEqual(projected["source_benchmark"], self.case["source_benchmark"]) + + def test_old_case_without_components_is_unchanged(self): + original = copy.deepcopy(self.case) + original.pop("source_benchmark") + result = evaluate_response(original, b"Life and dignity") + self.assertTrue(result["passed"]) + self.assertTrue(result["measured"]) + self.assertNotIn("article_comments_pass", result["metrics"]) + + def test_rescore_does_not_retain_previous_component_metrics(self): + previous = self.run_score(self.complete)["metrics"] + original = copy.deepcopy(self.case) + original.pop("source_benchmark") + metrics = score_attempt(original, b"Life and dignity", ok=True, + previous_metrics=previous) + self.assertNotIn("article_comments_reference_status", metrics) + self.assertNotIn("article_title_pass", metrics) + + + +class StaticCommentCoverageIntegration(unittest.TestCase): + def setUp(self): + original = ArticleComponentIntegration() + original.setUp() + self.case = copy.deepcopy(original.case) + self.contract = self.case["source_benchmark"]["comment_contract"] + self.contract.update(version=2, minimum_comments=2) + self.complete = "# Life and dignity\n\nArticle prose.\n\nComments\n\n## New reader\nA new comment today.\n\n## Another reader\nA different response." + + def test_static_coverage_needs_no_runtime_reference(self): + result = evaluate_response(self.case, self.complete.encode()) + self.assertTrue(result["passed"]) + self.assertTrue(result["measured"]) + self.assertNotIn("article_comments_reference_status", result["metrics"]) + self.assertEqual(result["metrics"]["content_decision_basis"], + "base_acceptance_and_required_components") + + def test_missing_comment_section_and_below_floor_are_measured_failures(self): + for content in (self.complete.split("Comments")[0], + self.complete.split("## Another reader")[0]): + with self.subTest(content=content): + result = evaluate_response(self.case, content.encode()) + self.assertTrue(result["measured"]) + self.assertFalse(result["passed"]) + self.assertFalse(result["metrics"]["article_comments_pass"]) + + def test_comment_names_text_and_reply_depth_are_not_frozen(self): + changed = self.complete.replace("New reader", "Changed author").replace( + "A new comment today.", "Entirely different current prose.").replace( + "## Another reader", "### Another reader") + self.assertTrue(evaluate_response(self.case, changed.encode())["passed"]) + + def test_no_comment_contract_has_no_comment_requirement(self): + self.case["source_benchmark"].pop("comment_contract") + result = evaluate_response(self.case, b"# Life and dignity\n\nArticle prose.") + self.assertTrue(result["passed"]) + self.assertNotIn("article_comments_pass", result["metrics"]) + + def test_cli_scores_static_comments_without_reference_arguments(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "task.json").write_text(json.dumps(self.case)) + (root / "answer.md").write_text(self.complete) + result = subprocess.run([sys.executable, "-m", "lexbench_webfetch.cli", "score", + "--task", str(root / "task.json"), "--response", str(root / "answer.md")], + text=True, capture_output=True, check=True) + self.assertTrue(json.loads(result.stdout)["passed"]) diff --git a/tests/test_article_control_text.py b/tests/test_article_control_text.py new file mode 100644 index 0000000..d9773d5 --- /dev/null +++ b/tests/test_article_control_text.py @@ -0,0 +1,119 @@ +"""Opt-in ordinary form labels; synthetic inputs without browser or network.""" +from copy import deepcopy +import unittest + +from lexbench_webfetch.stable_tasks import ( + evaluate_stable_task, validate_acceptance_contract, +) + + +def control_contract(attribute="placeholder", selector="#search", label="Search articles…"): + return { + "version": 1, "schema": "body_text", "minimum_records": 0, + "stable_text_anchors": [{ + "id": "control", "alternatives": [label], + "visible_control": {"selector": selector, "attribute": attribute}, + }], + } + + +class ArticleControlTextTests(unittest.TestCase): + def passes(self, content, rules): + return evaluate_stable_task(content, rules)["stable_task_pass"] + + def test_full_html_and_markdown_preserve_ordinary_labels(self): + rules = control_contract() + html = '

Article text

' + self.assertTrue(self.passes(html, rules)) + self.assertTrue(self.passes('Article text\n\nSearch articles…', rules)) + self.assertTrue(evaluate_stable_task(html, rules, anchor_text="Article text")["stable_task_pass"]) + + def test_allowed_control_types(self): + for kind in ("text", "search", "email", "url", "tel", "number"): + with self.subTest(kind=kind): + self.assertTrue(self.passes( + f'
', + control_contract())) + self.assertTrue(self.passes('
', control_contract())) + for kind in ("button", "submit", "reset"): + with self.subTest(kind=kind): + self.assertTrue(self.passes( + f'
', + control_contract("value", "#submit", "Post Comment"))) + + def test_undeclared_control_and_missing_selector_fail(self): + html = '
' + self.assertFalse(self.passes(html, control_contract())) + self.assertFalse(self.passes('
Article text
', control_contract())) + rules = control_contract() + del rules["stable_text_anchors"][0]["visible_control"] + self.assertFalse(self.passes(html, rules)) + + def test_populated_controls_do_not_display_placeholders(self): + for value in ("Already entered", " "): + with self.subTest(value=value): + self.assertFalse(self.passes( + f'
', + control_contract())) + self.assertFalse(self.passes( + f'
', + control_contract())) + self.assertTrue(self.passes( + '
', + control_contract())) + + def test_control_text_cannot_supply_another_anchor_or_page_pattern(self): + rules = control_contract() + rules["stable_text_anchors"].append({"id": "separate", "alternatives": ["Search articles…"]}) + html = '
' + result = evaluate_stable_task(html, rules) + self.assertIn("missing_stable_anchor:separate", result["stable_task_errors"]) + rules["stable_text_anchors"].pop() + rules["page_patterns"] = ["Search articles"] + self.assertFalse(self.passes(html, rules)) + + def test_hidden_controls_and_ancestors_fail(self): + control = '' + for attr in ('hidden', 'style="display:none"', 'style="visibility:hidden"', + 'style="visibility:collapse"', 'style="opacity:0"'): + with self.subTest(attr=attr): + self.assertFalse(self.passes(f'
{control}
', control_contract())) + self.assertFalse(self.passes(f'
', control_contract())) + + def test_password_hidden_and_non_button_values_fail(self): + for kind in ("password", "hidden", "text", "email", "checkbox"): + with self.subTest(kind=kind): + self.assertFalse(self.passes( + f'
', + control_contract("value", "#submit", "Post Comment"))) + for kind in ("password", "hidden", "submit", "checkbox"): + with self.subTest(kind=kind): + self.assertFalse(self.passes( + f'
', + control_contract())) + + def test_accessibility_and_metadata_attributes_cannot_supply_text(self): + for attribute in ("aria-label", "title", "alt"): + with self.subTest(attribute=attribute): + self.assertFalse(self.passes( + f'
', control_contract())) + self.assertFalse(self.passes('
', control_contract())) + + def test_invalid_declarations_fail_closed(self): + invalid = [None, {}, {"selector": "#search", "attribute": "aria-label"}, + {"selector": "#search", "attribute": "title"}, + {"selector": "#search", "attribute": []}, + {"selector": "[", "attribute": "placeholder"}, + {"selector": ":not-a-real-pseudo", "attribute": "placeholder"}, + {"selector": " ", "attribute": "placeholder"}, + {"selector": "#search", "attribute": "value", "unknown": True}] + for declaration in invalid: + with self.subTest(declaration=declaration): + rules = deepcopy(control_contract()) + rules["stable_text_anchors"][0]["visible_control"] = declaration + with self.assertRaises(ValueError): + validate_acceptance_contract(rules) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_article_dataset.py b/tests/test_article_dataset.py new file mode 100644 index 0000000..13bd12e --- /dev/null +++ b/tests/test_article_dataset.py @@ -0,0 +1,592 @@ +"""Article authoring must reject incomplete references and incomplete answers.""" +from article_contract_fixtures import comment_labels, post_labels +import copy +import hashlib +import html +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from lexbench_webfetch.article_dataset import compile_case, compile_dataset, DEFINITIONS +from lexbench_webfetch.benchmark import public_task +from lexbench_webfetch.evaluation import evaluate_response +from lexbench_webfetch.models import EvalCase, validate_case +from lexbench_webfetch.scoring import load_list_reference, score_attempt + + +class ArticleContracts(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.paragraphs = [ + "A comet travels beyond the distant planets. Astronomers measure its icy " + "nucleus through reflected sunlight while spectrometers distinguish carbon " + "molecules from frozen dust. Their observations establish the opening survey.", + "Across mountain valleys, botanists catalogue alpine flowers. Tiny seeds " + "survive winter beneath protective snow before spring temperatures awaken " + "their roots. Field notebooks describe every habitat in this second study.", + "An underwater microphone records whale songs near the coastline. Divers " + "compare low frequency pulses across migrating families and carefully mark " + "the changing tide. This acoustic experiment occupies the central section.", + "Inside a pottery workshop, artisans shape clay into cooking vessels. " + "Kiln temperatures transform fragile surfaces into durable ceramics, and " + "glazes produce vivid copper colours. Excavations explain the fourth investigation.", + "Finally, railway engineers inspect a suspended bridge. Steel cables " + "distribute heavy carriage loads between stone towers as instruments record " + "wind pressure. The closing assessment recommends regular structural inspections.", + ] + self.body = "\n\n".join(self.paragraphs) + "\n" + raw = self.body.encode() + (self.root / "body.txt").write_bytes(raw) + self.case = {"id": "article-001", "name": "Comet field report", + "url": "https://example.test/article", "task_prompt": "Return the full article"} + self.rule = { + "id": "article-001", "status": "verified", "target_kind": "fixed_article", + "reason": "Complete synthetic article used to exercise the public scorer.", + "body_reference": {"path": "body.txt", "sha256": hashlib.sha256(raw).hexdigest(), + "chars": len(self.body)}, + "reference_scope": {"complete_initial_body": True, + "start": self.paragraphs[0], "end": self.paragraphs[-1]}, + "body_regions": [[1, 2], [3, 4], [5, 6], [7, 8], [9, 9]], + "acceptance_contract": { + "version": 1, "schema": "body_text", "minimum_records": 0, + "page_patterns": ["Comet field report"], "stable_text_role": "required", + "stable_text_anchors": [{"id": "archive", "alternatives": ["Article archive"]}], + }, + "stable_text_review": [{"id": "archive", "region": "navigation", + "basis": "Fixed archive label in the synthetic page", + "observed_on": "2026-10-19"}], + "excluded_regions": ["Unrelated changing recommendations and comments."], + } + self.task = compile_case(self.case, self.rule, self.root) + self.prefix = "# Comet field report\n\nArticle archive\n\n" + + def score(self, text): + return evaluate_response(self.task, text.encode())["passed"] + + def test_compiler_requires_every_region_and_binds_ending(self): + body = self.task["body_requirements"] + self.assertEqual(body["version"], 4) + self.assertEqual(body["minimum_floor"], 0.8) + self.assertEqual(body["minimum_region_passes"], len(body["regions"])) + self.assertEqual(body["ending"]["text"], self.rule["reference_scope"]["end"]) + self.assertTrue(self.score(self.prefix + self.body)) + self.assertTrue(self.score(self.prefix + "\n\n".join("**" + p + "**" for p in self.paragraphs))) + self.assertFalse(self.score(self.prefix + "\n\n".join(self.paragraphs[:-1]))) + public = public_task(self.task)["body_requirements"]["ending"] + self.assertNotIn("text", public) + self.assertEqual(public["text_binding"]["chars"], len(self.paragraphs[-1])) + + def test_review_defaults_preserve_acceptance_and_explicit_overrides(self): + rule = copy.deepcopy(self.rule) + original = rule["stable_text_review"][0] + rule["stable_text_review_defaults"] = {k: original.pop(k) for k in ("basis", "observed_on")} + self.assertEqual(compile_case(self.case, rule, self.root), self.task) + rule["stable_text_review"][0]["basis"] = "" + with self.assertRaises(ValueError): + compile_case(self.case, rule, self.root) + del rule["stable_text_review"][0]["basis"] + rule["stable_text_review_defaults"]["region"] = "Should remain explicit" + with self.assertRaises(ValueError): + compile_case(self.case, rule, self.root) + + def test_one_region_below_eighty_no_longer_uses_sixty_percent_allowance(self): + text = self.prefix + "\n\n".join([self.paragraphs[0][:int(len(self.paragraphs[0]) * .72)], *self.paragraphs[1:]]) + legacy = copy.deepcopy(self.task) + legacy["body_requirements"].update(version=3, minimum_floor=.6, minimum_region_passes=4) + del legacy["body_requirements"]["ending"] + self.assertTrue(evaluate_response(legacy, text.encode())["passed"]) + self.assertFalse(self.score(text)) + + def test_legacy_regions_remain_readable_but_current_regions_are_capped(self): + paragraphs = self.paragraphs + [ + "A desert observatory records meteor paths across the darkest sky.", + "Marine researchers preserve specimens in a coastal laboratory.", + ] + raw = ("\n".join(paragraphs) + "\n").encode() + (self.root / "body.txt").write_bytes(raw) + rule = copy.deepcopy(self.rule) + rule["body_reference"].update(sha256=hashlib.sha256(raw).hexdigest(), chars=len(raw.decode())) + rule["reference_scope"].update(start=paragraphs[0], end=paragraphs[-1]) + rule["body_regions"] = [[i, i] for i in range(1, 8)] + task = compile_case(self.case, rule, self.root) + self.assertEqual(len(task["body_requirements"]["regions"]), 7) + current = copy.deepcopy(rule) + current["body_segmentation"] = {"version": 3} + with self.assertRaisesRegex(ValueError, "at most five"): + compile_case(self.case, current, self.root) + self.assertTrue(evaluate_response(task, (self.prefix + raw.decode()).encode())["passed"]) + for omitted in range(7): + with self.subTest(omitted=omitted): + incomplete = self.prefix + "\n".join(p for i, p in enumerate(paragraphs) if i != omitted) + self.assertFalse(evaluate_response(task, incomplete.encode())["passed"]) + + def test_semantic_regions_keep_labels_and_reject_misbound_reviews(self): + rule = copy.deepcopy(self.rule) + rule["body_segmentation"] = {"version": 3, "method": "source_sections", + "reference_sha256": rule["body_reference"]["sha256"]} + rule["body_region_review"] = [ + {"lines": bounds, "label": f"Observed section {i}"} + for i, bounds in enumerate(rule["body_regions"], 1)] + task = compile_case(self.case, rule, self.root) + self.assertEqual(task["body_requirements"]["regions"][0]["label"], "Observed section 1") + self.assertEqual(task["body_requirements"]["regions"][0]["source_line_range"], [1, 2]) + for field, value in [("label", ""), ("lines", [2, 3])]: + bad = copy.deepcopy(rule) + bad["body_region_review"][0][field] = value + with self.assertRaisesRegex(ValueError, "source-bound readable label"): + compile_case(self.case, bad, self.root) + + def test_historical_comment_qualification_binds_actual_source_bytes(self): + source = b"

Comet field report

Independent complete initial page.

" + (self.root / "source.html").write_bytes(source) + reference = {"version": 1, "case_id": self.case["id"], "url": self.case["url"], + "scope": "initial_load_only", "context_id": "historical-authoring", + "observed_on": "2026-10-19", "observation_complete": True, + "source_observation_counts": [0], "records": [], + "source_evidence": [{"path": "source.html", + "sha256": hashlib.sha256(source).hexdigest(), + "scope": "complete_initial_page"}]} + raw = (json.dumps(reference) + "\n").encode() + (self.root / "comments.json").write_bytes(raw) + rule = copy.deepcopy(self.rule) + rule["comment_coverage_review"] = {"status": "observed_absent", + "basis": "Independent initial page has no comments."} + with self.assertRaisesRegex(ValueError, "independent qualification evidence"): + compile_case(self.case, rule, self.root) + rule["comment_qualification_reference"] = { + "path": "comments.json", "sha256": hashlib.sha256(raw).hexdigest()} + task = compile_case(self.case, rule, self.root) + # Historical qualification establishes absence without a runtime reference. + unavailable = evaluate_response(task, (self.prefix + self.body).encode()) + self.assertTrue(unavailable["passed"]) + self.assertTrue(unavailable["measured"]) + (self.root / "source.html").write_bytes(source + b"changed") + with self.assertRaisesRegex(ValueError, "Changed article comment source evidence"): + compile_case(self.case, rule, self.root) + (self.root / "source.html").write_bytes(source) + (self.root / "comments.json").write_bytes(raw + b" ") + with self.assertRaisesRegex(ValueError, "Changed article comment qualification reference"): + compile_case(self.case, rule, self.root) + + def test_absence_review_keeps_evidence_without_runtime_comment_contract(self): + source = b"
Complete source without comments.
" + (self.root / "source.html").write_bytes(source) + reference = {"version": 1, "case_id": self.case["id"], "url": self.case["url"], + "scope": "initial_load_only", "context_id": "historical-authoring", + "observed_on": "2026-10-19", "observation_complete": True, + "source_observation_counts": [0], "records": [], + "source_evidence": [{"path": "source.html", + "sha256": hashlib.sha256(source).hexdigest(), + "scope": "complete_initial_page"}]} + raw = (json.dumps(reference) + "\n").encode() + (self.root / "comments.json").write_bytes(raw) + rule = copy.deepcopy(self.rule) + rule["comment_coverage_review"] = { + "status": "observed_absent", "basis": "Complete initial source reviewed."} + with self.assertRaisesRegex(ValueError, "qualification evidence"): + compile_case(self.case, rule, self.root) + rule["comment_qualification_reference"] = { + "path": "comments.json", "sha256": hashlib.sha256(raw).hexdigest()} + task = compile_case(self.case, rule, self.root) + self.assertNotIn("comment_contract", task["source_benchmark"]) + self.assertEqual(task["source_benchmark"]["comment_presence_review"]["status"], + "observed_absent") + result = evaluate_response(task, (self.prefix + self.body).encode()) + self.assertTrue(result["passed"]) + self.assertTrue(result["measured"]) + for status in ("unknown", None, "visible_text"): + invalid = copy.deepcopy(rule) + invalid["comment_coverage_review"]["status"] = status + with self.assertRaises(ValueError): + compile_case(self.case, invalid, self.root) + invalid = copy.deepcopy(rule) + invalid["comment_contract"] = {**comment_labels(), + "version": 2, "minimum_comments": 1, "record_selector": ".comment", + "author_selector": ".author", "body_selector": ".body", + "section_markers": ["Comments"]} + invalid.pop("comment_coverage_review") + with self.assertRaisesRegex(ValueError, "presence review"): + compile_case(self.case, invalid, self.root) + (self.root / "source.html").write_bytes(source + b"changed") + with self.assertRaisesRegex(ValueError, "Changed article comment source evidence"): + compile_case(self.case, rule, self.root) + + def test_placeholder_only_review_cannot_hide_actual_comment_text(self): + source = b"
Comment hidden
" + (self.root / "source.html").write_bytes(source) + reference = {"version": 1, "case_id": self.case["id"], "url": self.case["url"], + "scope": "initial_load_only", "context_id": "historical-authoring", + "observed_on": "2026-10-19", "observation_complete": True, + "source_observation_counts": [1], + "records": [{"id": "c1", "author": "Reader", "text": "Comment hidden", + "parent_id": None, "complete_initial_text": True}], + "source_evidence": [{"path": "source.html", + "sha256": hashlib.sha256(source).hexdigest(), + "scope": "complete_initial_page"}]} + rule = copy.deepcopy(self.rule) + rule["comment_coverage_review"] = { + "status": "visible_placeholders_only", "basis": "Only a moderation notice is displayed.", + "placeholder_texts": ["Comment hidden"], "historical_contract": { + "version": 1, "record_selector": ".comment", "author_selector": ".author", + "body_selector": ".body", "section_markers": ["Comments"], + "max_reference_age_seconds": 900}} + for text, accepted in [("Comment hidden", True), ("Actual comment prose", False)]: + reference["records"][0]["text"] = text + raw = (json.dumps(reference) + "\n").encode() + (self.root / "comments.json").write_bytes(raw) + rule["comment_qualification_reference"] = { + "path": "comments.json", "sha256": hashlib.sha256(raw).hexdigest()} + if accepted: + task = compile_case(self.case, rule, self.root) + self.assertNotIn("comment_contract", task["source_benchmark"]) + for placeholders in ("Comment hidden", [""], [None]): + invalid = copy.deepcopy(rule) + invalid["comment_coverage_review"]["placeholder_texts"] = placeholders + with self.assertRaisesRegex(ValueError, "Placeholder-only"): + compile_case(self.case, invalid, self.root) + else: + with self.assertRaisesRegex(ValueError, "Placeholder-only"): + compile_case(self.case, rule, self.root) + + def test_complete_plain_and_markdown_articles_pass(self): + validate_case(EvalCase(**self.task)) + self.assertTrue(self.score(self.prefix + self.body)) + self.assertTrue(self.score(self.prefix + "\n\n".join( + f"## Section {i}\n\n{p}" for i, p in enumerate(self.paragraphs)))) + + def test_each_missing_whole_paragraph_fails(self): + for omitted in range(len(self.paragraphs)): + with self.subTest(omitted=omitted): + text = "\n\n".join(p for i, p in enumerate(self.paragraphs) if i != omitted) + self.assertFalse(self.score(self.prefix + text)) + + def test_title_navigation_opening_repetition_and_wrong_article_fail(self): + for text in (self.prefix, self.prefix + self.paragraphs[0], + self.prefix + (self.paragraphs[0] + "\n") * 8, + self.prefix + "An unrelated article discusses a different research project."): + with self.subTest(text=text[:80]): + self.assertFalse(self.score(text)) + + def test_complete_body_cannot_replace_required_identity_or_stable_text(self): + self.assertFalse(self.score(self.body)) + self.assertFalse(self.score("Comet field report\n" + self.body)) + + def test_article_identity_cannot_match_empty_text(self): + for pattern in ("^", ".*", r"\s*", "(?:Comet field report)?"): + with self.subTest(pattern=pattern): + rule = copy.deepcopy(self.rule) + rule['acceptance_contract']['page_patterns'] = [pattern] + with self.assertRaisesRegex(ValueError, "cannot match empty"): + compile_case(self.case, rule, self.root) + rule = copy.deepcopy(self.rule) + rule['acceptance_contract']['page_patterns'] = ["("] + with self.assertRaisesRegex(ValueError, "Invalid stable acceptance regex"): + compile_case(self.case, rule, self.root) + rule = copy.deepcopy(self.rule) + rule['acceptance_contract']['page_patterns'] = ["Comet(?: field)? report"] + task = compile_case(self.case, rule, self.root) + self.assertTrue(evaluate_response(task, (self.prefix + self.body).encode())['passed']) + + def test_invalid_task_fields_fail_during_compilation(self): + for field, value in (("url", "javascript:alert(1)"), ("url", "/article"), + ("id", "../article-001"), ("name", " ")): + with self.subTest(field=field, value=value): + case = {**self.case, field: value} + rule = {**self.rule, "id": case['id']} + with self.assertRaises(ValueError): + compile_case(case, rule, self.root) + directory = self.root / "invalid-definitions" + directory.mkdir() + (directory / "cohort.json").write_text(json.dumps({ + "schema_version": 1, "case_count": 1, + "cases": [{**self.case, "url": "javascript:alert(1)"}]})) + (directory / "rules.json").write_text(json.dumps({ + "schema_version": 1, "rules": [self.rule]})) + with self.assertRaisesRegex(ValueError, "HTTP"): + compile_dataset(self.root, directory) + + def test_public_projection_binds_the_entire_reference_without_copying_it(self): + projected = public_task(self.task) + encoded = json.dumps(projected) + for paragraph in self.paragraphs: + self.assertNotIn(paragraph, encoded) + self.assertEqual(len(projected['body_requirements']['regions']), 5) + self.assertEqual(sum(r['source_text_binding']['chars'] for r in + projected['body_requirements']['regions']), + sum(len(p) for p in self.paragraphs)) + + def test_unresolved_scope_changed_bytes_and_escaping_paths_fail(self): + changes = [ + ("status", "unresolved"), + ("reference_scope", {**self.rule['reference_scope'], "complete_initial_body": False}), + ("reference_scope", {**self.rule['reference_scope'], "end": "Invented ending"}), + ("body_reference", {**self.rule['body_reference'], "sha256": "0" * 64}), + ("body_reference", {**self.rule['body_reference'], "path": "../body.txt"}), + ("target_kind", "article_feed"), + ("body_regions", [[1, 2], [4, 9]]), + ("body_regions", [[1, 2], [2, 9]]), + ("body_regions", [[1, 8]]), + ("body_regions", []), + ("stable_text_review", []), + ("stable_text_review", [{**self.rule['stable_text_review'][0], "id": "other"}]), + ("stable_text_review", [{**self.rule['stable_text_review'][0], "basis": ""}]), + ] + for key, value in changes: + with self.subTest(key=key, value=value): + rule = copy.deepcopy(self.rule) + rule[key] = value + with self.assertRaises(ValueError): + compile_case(self.case, rule, self.root) + + def test_body_region_requires_scorable_text(self): + raw = b"Tiny\n" + (self.root / "body.txt").write_bytes(raw) + rule = copy.deepcopy(self.rule) + rule.update(body_reference={"path": "body.txt", + "sha256": hashlib.sha256(raw).hexdigest(), "chars": 5}, + reference_scope={"complete_initial_body": True, + "start": "Tiny", "end": "Tiny"}, + body_regions=[[1, 1]]) + with self.assertRaisesRegex(ValueError, "empty coverage region"): + compile_case(self.case, rule, self.root) + + def test_collection_cannot_freeze_current_feed_as_fixed_article(self): + with self.assertRaisesRegex(ValueError, "separate record review"): + compile_case({**self.case, "page_type": "collection"}, self.rule, self.root) + + def test_post_collection_uses_fixed_coverage_with_independent_qualification(self): + case = {**self.case, "url": "https://example.test/feed", "page_type": "collection"} + contract = {**post_labels(), + "version": 2, "hosts": ["example.test"], "entry_path": r"/research/[a-z]+", + "html_record_selector": "article", "html_permalink_selector": "a.permalink", + "html_author_selector": ".author", "html_body_selector": ".post-body", + "html_quote_selector": "blockquote", "quote_markers": ["Quote"], + "minimum_posts": 2, + } + reference = { + "version": 1, "case_id": case['id'], "url": case['url'], + "scope": "initial_load_only", "context_id": "independent-control-1", + "observed_at": "2026-10-19T00:00:00Z", "source_observation_counts": [2, 2], + "records": [{"url": f"https://example.test/research/{name}", "author": author, + "body_parts": [{"kind": "post", "text": body, + "complete_initial_text": True}]} + for name, author, body in [ + ("one", "Dr Rivera", self.paragraphs[0]), + ("two", "Dr Chen", self.paragraphs[2])]], + } + raw = json.dumps(reference).encode() + (self.root / 'qualification.json').write_bytes(raw) + rule = {key: value for key, value in self.rule.items() + if key not in {'body_reference', 'reference_scope', 'body_regions'}} + rule.update(target_kind='post_collection', post_contract=contract, + qualification_reference={'path': 'qualification.json', + 'sha256': hashlib.sha256(raw).hexdigest()}) + source = self.root / 'independent-source.txt' + source.write_text('Two independently displayed posts with visible author and text.') + rule['post_coverage_review'] = {'source_evidence': [ + {'path': source.name, 'sha256': hashlib.sha256(source.read_bytes()).hexdigest()}]} + task = compile_case(case, rule, self.root) + validate_case(EvalCase(**task)) + projected = public_task(task) + self.assertEqual(projected['source_benchmark']['post_contract'], contract) + for record in reference['records']: + self.assertNotIn(record['url'], json.dumps(projected)) + self.assertNotIn(record['body_parts'][0]['text'], json.dumps(projected)) + self.assertNotIn('qualification_proof', projected['source_benchmark']) + refs = self.root / 'list-references' + refs.mkdir() + self.assertIsNone(load_list_reference(task, self.root)) + (refs / f"{case['id']}.json").write_bytes(raw) + self.assertIsNone(load_list_reference(task, self.root)) + + def response(records): + text = '

Comet field report

' + for record in records: + text += (f'
' + f'
{record["author"]}
' + f'
{html.escape(record["body_parts"][0]["text"])}
' + '
') + return text.encode() + + original = response(reference['records']) + complete = evaluate_response(task, original) + self.assertTrue(complete['passed']) + self.assertTrue(complete['measured']) + insufficient = evaluate_response(task, response(reference['records'][:1])) + self.assertFalse(insufficient['passed']) + self.assertTrue(insufficient['measured']) + updated = copy.deepcopy(reference['records']) + for row in updated: + row['body_parts'][0]['text'] = 'New visible discussion text.' + self.assertTrue(evaluate_response(task, response(updated))['passed']) + + rescored = score_attempt(self.task, (self.prefix + self.body).encode(), ok=True, + previous_metrics=complete['metrics']) + self.assertNotIn('article_reference_status', rescored) + self.assertNotIn('article_posts_pass', rescored) + + static = copy.deepcopy(rule) + static['post_contract'] = {key: value for key, value in contract.items() + if key not in {'required_ratio', 'max_reference_age_seconds'}} + static['post_contract'].update(version=2, minimum_posts=2, + allow_empty_outer_with_quotes=True) + source = self.root / 'independent-source.txt' + source.write_text('Two independently displayed posts with visible author and text.') + static['post_coverage_review'] = {'source_evidence': [ + {'path': source.name, 'sha256': hashlib.sha256(source.read_bytes()).hexdigest()}]} + static_task = compile_case(case, static, self.root) + self.assertEqual(static_task['source_benchmark']['post_contract']['minimum_posts'], 2) + static['post_contract']['minimum_posts'] = 3 + with self.assertRaises(ValueError): + compile_case(case, static, self.root) + static['post_contract']['minimum_posts'] = 2 + source.write_text('Changed capture') + with self.assertRaisesRegex(ValueError, 'Changed article collection source'): + compile_case(case, static, self.root) + static['post_coverage_review']['source_evidence'] = [] + with self.assertRaisesRegex(ValueError, 'requires source evidence'): + compile_case(case, static, self.root) + + source.write_text('Two independently displayed posts with visible author and text.') + incomplete = copy.deepcopy(reference) + incomplete['records'][0]['body_parts'][0]['complete_initial_text'] = False + raw = json.dumps(incomplete).encode() + (self.root / 'qualification.json').write_bytes(raw) + rule['qualification_reference']['sha256'] = hashlib.sha256(raw).hexdigest() + with self.assertRaisesRegex(ValueError, 'complete owned post'): + compile_case(case, rule, self.root) + + def test_whole_population_cannot_silently_drop_or_ignore_pending_cases(self): + directory = self.root / "definitions" + directory.mkdir() + second = {**self.case, "id": "article-002"} + (directory / "cohort.json").write_text(json.dumps({ + "schema_version": 1, "case_count": 2, "cases": [self.case, second]})) + path = directory / "rules.json" + path.write_text(json.dumps({"schema_version": 1, "rules": [self.rule]})) + with self.assertRaisesRegex(ValueError, "inventory"): + compile_dataset(self.root, directory) + path.write_text(json.dumps({"schema_version": 1, "rules": [self.rule, { + "id": second['id'], "status": "unresolved"}]})) + with self.assertRaisesRegex(ValueError, "Unresolved"): + compile_dataset(self.root, directory) + + # Resolving the same approved inventory produces a bound suite; it does + # not require dropping a difficult case or rewriting an existing suite. + second_rule = {**self.rule, "id": second['id']} + path.write_text(json.dumps({"schema_version": 1, + "rules": [self.rule, second_rule]})) + dataset = compile_dataset(self.root, directory) + self.assertEqual(dataset['case_count'], 2) + self.assertEqual(dataset['site_count'], 1) + self.assertEqual(dataset['source_bindings']['rules_sha256'], + hashlib.sha256(path.read_bytes()).hexdigest()) + self.assertEqual([case['id'] for case in dataset['cases']], + [self.case['id'], second['id']]) + + def test_approved_inventory_keeps_all_cases_and_disclosed_removals(self): + cohort = json.loads((DEFINITIONS / "cohort.json").read_text()) + rules = json.loads((DEFINITIONS / "rules.json").read_text())['rules'] + self.assertEqual(cohort['case_count'], 100) + self.assertEqual([c['id'] for c in cohort['cases']], [r['id'] for r in rules]) + self.assertEqual([c['id'] for c in cohort['cases']], [ + f'article-{number:03}' for number in range(1, 108) + if number not in {4, 42, 43, 62, 83, 97, 105}]) + self.assertEqual(cohort['provenance']['removed_candidate_ids'], + ['article-candidate-043', 'article-candidate-105']) + revision = next(r for r in cohort['revisions'] if r['revision'] == 3) + self.assertEqual({change['id'] for change in revision['changes']}, + {'article-042', 'article-062', 'article-083', 'article-097'}) + self.assertTrue(all(change['action'] == 'remove' + and change['previous']['id'] == change['id'] + for change in revision['changes'])) + self.assertTrue(all(c['platform'] != '搜狗微信' for c in cohort['cases'])) + final = next(r for r in cohort['revisions'] if r['revision'] == 4) + self.assertEqual(cohort['revisions'][-1]['revision'], 14) + self.assertEqual({c['id'] for c in final['changes'] if c['action'] == 'replace'}, + {'article-001', 'article-002', 'article-003', 'article-079', 'article-082'}) + self.assertEqual([c['id'] for c in final['changes'] if c['action'] == 'remove'], + ['article-004']) + self.assertTrue(all(rule['status'] == 'verified' for rule in rules)) + self.assertEqual(sum(rule['target_kind'] == 'post_collection' for rule in rules), 5) + fixed = [r for r in rules if r['target_kind'] == 'fixed_article'] + self.assertEqual(cohort['id'], 'articles-v11') + self.assertEqual(len(fixed), 95) + self.assertEqual(sum('title_contract' in r for r in fixed), 90) + self.assertEqual(sum('comment_contract' in r for r in fixed), 20) + self.assertTrue(all(r['comment_contract']['version'] == 2 + for r in fixed if 'comment_contract' in r)) + self.assertTrue(all('comment_qualification_reference' in r + and 'comment_coverage_review' in r for r in fixed)) + self.assertTrue(all(r['body_segmentation']['version'] == 3 for r in fixed)) + self.assertTrue(all(r['body_segmentation']['reference_sha256'] == r['body_reference']['sha256'] + for r in fixed)) + for rule in fixed: + self.assertEqual([r['lines'] for r in rule['body_region_review']], rule['body_regions']) + self.assertTrue(all(r['label'].strip() for r in rule['body_region_review'])) + self.assertTrue(all(1 <= len(r['body_regions']) <= 5 for r in fixed)) + self.assertEqual(len(next(r for r in cohort['revisions'] if r['revision'] == 6)['changes']), 57) + revised = {c['id']: c['action'] for c in next(r for r in cohort['revisions'] if r['revision'] == 5)['changes']} + self.assertEqual(revised, {'article-049': 'revise_entry_url', + 'article-090': 'revise_source_reference', + 'article-061': 'replace', + 'article-038': 'require_all_initial_posts', + 'article-040': 'require_all_initial_posts', + 'article-041': 'require_all_initial_posts'}) + posts = [r for r in rules if r['target_kind'] == 'post_collection'] + self.assertTrue(all(r['post_contract']['version'] == 2 for r in posts)) + self.assertEqual([r['post_contract']['minimum_posts'] for r in posts], [4, 8, 4, 4, 16]) + self.assertTrue(all(r['post_coverage_review']['source_evidence'] for r in posts)) + + def test_published_contracts_match_authoring_rules(self): + rules = json.loads((DEFINITIONS / "rules.json").read_text())["rules"] + tasks = json.loads((DEFINITIONS / "tasks.json").read_text())["cases"] + by_id = {task["id"]: task for task in tasks} + self.assertEqual(len(by_id), len(tasks), "Duplicate public task IDs") + self.assertEqual(len({rule["id"] for rule in rules}), len(rules)) + self.assertEqual({rule["id"] for rule in rules}, set(by_id)) + for rule in rules: + task = by_id[rule["id"]] + with self.subTest(case=rule["id"]): + self.assertEqual(rule["acceptance_contract"], task["acceptance_contract"]) + for component in ("title_contract", "comment_contract", "post_contract"): + self.assertEqual(rule.get(component), + task.get("source_benchmark", {}).get(component)) + + def test_published_tasks_bind_current_authoring_bytes(self): + tasks = json.loads((DEFINITIONS / "tasks.json").read_text()) + for name in ("cohort", "rules"): + with self.subTest(source=name): + self.assertEqual(tasks["source_bindings"][name + "_sha256"], + hashlib.sha256((DEFINITIONS / (name + ".json")).read_bytes()).hexdigest()) + + def test_definition_hashes_bind_the_parsed_snapshot(self): + directory = self.root / "snapshot-definitions" + directory.mkdir() + cohort = {"schema_version": 1, "case_count": 1, "cases": [self.case]} + rules = {"schema_version": 1, "rules": [self.rule]} + original = {"cohort.json": json.dumps(cohort).encode(), + "rules.json": json.dumps(rules).encode()} + for name, raw in original.items(): + (directory / name).write_bytes(raw) + + def replace_sources_then_compile(case, rule, evidence_home): + (directory / "cohort.json").write_text(json.dumps({**cohort, "case_count": 0})) + (directory / "rules.json").write_text(json.dumps({ + "schema_version": 1, + "rules": [{**self.rule, "status": "unresolved"}]})) + return compile_case(case, rule, evidence_home) + + with patch("lexbench_webfetch.article_dataset.compile_case", + side_effect=replace_sources_then_compile): + dataset = compile_dataset(self.root, directory) + self.assertEqual(dataset['case_count'], 1) + for name, raw in original.items(): + key = name.replace('.json', '_sha256') + self.assertEqual(dataset['source_bindings'][key], hashlib.sha256(raw).hexdigest()) + self.assertNotEqual(dataset['source_bindings'][key], hashlib.sha256( + (directory / name).read_bytes()).hexdigest()) diff --git a/tests/test_article_labels.py b/tests/test_article_labels.py new file mode 100644 index 0000000..97ec94a --- /dev/null +++ b/tests/test_article_labels.py @@ -0,0 +1,126 @@ +"""Language labels are frozen task data, not scorer-wide site onboarding.""" +from article_contract_fixtures import post_labels +from copy import deepcopy +import pytest +from lexbench_webfetch.article_comments import validate_comment_contract, verify_article_comments +from lexbench_webfetch.article_posts import validate_post_contract, verify_article_posts +from lexbench_webfetch.article_text import validate_labels + +C = {'version': 2, 'minimum_comments': 1, 'record_selector': '.comment', + 'author_selector': '.author', 'body_selector': '.body', 'section_markers': ['Comments'], + 'ignored_labels': ['返信'], 'unavailable_notices': ['非表示のコメント']} + + +def test_new_language_labels_are_task_local_and_whole_label_only(): + def check(text, contract): + return verify_article_comments('
Reader

'+text+'

', contract)['article_comments_pass'] + assert not check('返信', C) + assert not check('非表示のコメント', C) + assert check('返信について説明します。', C) + assert check('Real response.', C) + other = {**C, 'ignored_labels': [], 'unavailable_notices': []} + assert check('返信', other) + assert not check('', other) + + +@pytest.mark.parametrize('bad', [None, 'Reply', [1], [''], ['Reply', 'Reply']]) +def test_invalid_declared_labels_fail_closed(bad): + with pytest.raises(ValueError): + validate_comment_contract({**C, 'ignored_labels': bad}) + + +def test_post_labels_are_not_global_or_substring_filters(): + contract = {**post_labels(), 'version': 2, 'minimum_posts': 1, 'hosts': ['example.test'], 'entry_path': r'/p/\d+', + 'html_record_selector': 'article', 'html_permalink_selector': 'a', + 'html_author_selector': 'b', 'html_body_selector': 'p', + 'ignored_labels': ['返信'], 'unavailable_notices': ['非表示']} + def check(body): + html = '
PostAuthor

'+body+'

' + return verify_article_posts(html, contract, url='https://example.test/feed')['article_posts_pass'] + assert not check('返信') + assert not check('非表示') + assert check('返信について説明します。') + assert not check('') + + +@pytest.mark.parametrize('field', ['ignored_labels', 'unavailable_notices']) +def test_missing_comment_labels_are_rejected(field): + contract = deepcopy(C) + del contract[field] + with pytest.raises(ValueError, match='missing task-declared'): + validate_comment_contract(contract) + + +def test_normalized_duplicate_labels_are_rejected(): + with pytest.raises(ValueError, match='Duplicate'): + validate_comment_contract({**C, 'ignored_labels': ['Log In', ' **ＬＯＧ　ＩＮ** ']}) + + +def test_whitespace_emphasis_and_width_use_the_same_label_key(): + contract = {**C, 'ignored_labels': [' ＬＯＧ　　ＩＮ '], 'unavailable_notices': [' Not Available ']} + for text in ['**log in**', '_LOG IN_', ':not available:']: + html = '
Reader

'+text+'

' + assert not verify_article_comments(html, contract)['article_comments_pass'] + + +def test_only_relevant_current_contracts_declare_labels(): + import json + from lexbench_webfetch.article_dataset import DEFINITIONS + rules = json.loads((DEFINITIONS / 'rules.json').read_text())['rules'] + comments = [r['comment_contract'] for r in rules if 'comment_contract' in r] + posts = [r['post_contract'] for r in rules if 'post_contract' in r] + assert (len(rules), len(comments), len(posts)) == (100, 20, 5) + for contract in comments: + validate_comment_contract(contract) + for contract in posts: + validate_post_contract(contract) + assert not (DEFINITIONS / 'labels.json').exists() + + +def test_task_declared_non_author_labels_do_not_reject_actual_attribution(): + contract = {**post_labels(), 'version': 2, 'minimum_posts': 1, + 'hosts': ['example.test'], 'entry_path': r'/p/\d+', + 'html_record_selector': 'article', 'html_permalink_selector': 'a', + 'html_author_selector': 'b', 'html_body_selector': 'p', + 'non_author_labels': ['返信']} + def check(text): + return verify_article_posts(text, contract, url='https://example.test/feed')['article_posts_pass'] + assert not check('## [返信](/p/1)\nVisible post body.') + assert check('## Real Author [返信](/p/1)\nVisible post body.') + assert not check('## [2026-10-22](/p/1)\nVisible post body.') + assert check('## [Reader](/p/1)\nVisible post body.') + for field in ['ignored_labels', 'unavailable_notices', 'non_author_labels']: + incomplete = deepcopy(contract) + del incomplete[field] + with pytest.raises(ValueError, match='missing task-declared'): + validate_post_contract(incomplete) + + +def test_post_whole_labels_share_rule_and_observation_normalization(): + contract = {**post_labels(), 'version': 2, 'minimum_posts': 1, + 'hosts': ['example.test'], 'entry_path': r'/p/\d+', + 'html_record_selector': 'article', 'html_permalink_selector': 'a', + 'html_author_selector': 'b', 'html_body_selector': 'p', + 'ignored_labels': [' ＬＯＧ　　ＩＮ '], + 'unavailable_notices': [' Not Available '], + 'non_author_labels': [' View Post ']} + def check(body, author='Reader'): + return verify_article_posts('## ['+author+'](/p/1)\n'+body, contract, + url='https://example.test/feed')['article_posts_pass'] + for text in ['**log in**', '_LOG IN_', ':not available:']: + assert not check(text) + assert check('How to log in securely.') + assert not check('Visible body.', 'ＶＩＥＷ　　ＰＯＳＴ') + assert check('Visible body.', 'View Post researcher') + + +def test_comment_emphasis_normalization_preserves_distinct_source_ids(): + from lexbench_webfetch.article_comments import valid_comment_records + contract = {**C, 'ignored_labels': ['reply']} + assert valid_comment_records([{'author': 'Reader', 'text': '_reply_'}], contract) == [] + rows = [{'author': 'Reader', 'text': 'Useful observation'}, + {'author': 'Reader', 'text': '_Useful observation_'}] + assert len(valid_comment_records(rows, contract)) == 1 + identified = [dict(row, source_id=str(i)) for i, row in enumerate(rows)] + assert len(valid_comment_records(identified, contract)) == 2 + assert len(valid_comment_records([{'author': 'Reader', 'text': 'A useful reply.'}], contract)) == 1 diff --git a/tests/test_article_literal_boundaries.py b/tests/test_article_literal_boundaries.py new file mode 100644 index 0000000..470bf57 --- /dev/null +++ b/tests/test_article_literal_boundaries.py @@ -0,0 +1,93 @@ +"""Code samples and serializer metadata cannot manufacture required components.""" +import re +from textwrap import indent + +import pytest + +from lexbench_webfetch.evaluation import evaluate_response +from test_article_post_coverage_integration import fixture +import test_article_component_integration as component_fixture + + +@pytest.mark.parametrize('number', [38, 39, 40, 41, 101]) +@pytest.mark.parametrize('representation', ['real', 'spaces', 'tabs', 'fenced', 'unclosed']) +def test_collection_code_samples_do_not_supply_posts(number, representation): + case, stable, posts = fixture(number) + text = '\n\n'.join(posts) + if representation in ('spaces', 'tabs'): + text = indent(text, ' ' if representation == 'spaces' else '\t') + elif representation == 'fenced': + text = '```markdown\n'+text+'\n```' + elif representation == 'unclosed': + text = '```markdown\n'+text + result = evaluate_response(case, (stable+'\n\n'+text).encode()) + assert result['passed'] is (representation == 'real') + + +@pytest.mark.parametrize('number', [38, 39, 40, 41, 101]) +@pytest.mark.parametrize('author', ['Today', '2026-10-22', '12:34', '\u200b', 'Actual reader', '🙂']) +@pytest.mark.parametrize('route', ['previous_line', 'prefix', 'linked_profile']) +def test_collection_author_validation_after_every_extraction_route(number, author, route): + case, stable, posts = fixture(number) + records = [] + for post in posts: + url = re.search(r'\]\(([^)]+)\)', post)[1] + if route == 'previous_line': + header = author+'\n[Today]('+url+')' + elif route == 'prefix': + header = '## '+author+' [Today]('+url+')' + else: + profile = 'https://'+case['source_benchmark']['post_contract']['hosts'][0]+'/reader' + header = '## ['+author+']('+profile+') [Today]('+url+')' + records.append(header+'\n\nVisible post body.') + result = evaluate_response(case, (stable+'\n\n'+'\n\n'.join(records)).encode()) + assert result['passed'] is (author in ['Actual reader', '🙂']) + + +@pytest.mark.parametrize('component', ['title', 'comments']) +@pytest.mark.parametrize('prefix', [' ', '\t']) +def test_code_cannot_supply_title_or_comment_authors(component, prefix): + fixture = component_fixture.ArticleComponentIntegration() + fixture.setUp() + assert fixture.run_score(fixture.complete)['passed'] + text = fixture.complete + if component == 'title': + text = text.replace('# Life and dignity', prefix+'# Life and dignity') + else: + text = text.replace('## Reader\nA thoughtful book.', indent('## Reader\nA thoughtful book.', prefix)) + assert not fixture.run_score(text)['passed'] + + +@pytest.mark.parametrize('literal', [' ### Fake reader\n Literal example.', '\t### Fake reader\n\tLiteral example.', '```\n### Fake reader\n```']) +def test_code_remains_body_of_existing_comment(literal): + fixture = component_fixture.ArticleComponentIntegration() + fixture.setUp() + text = fixture.complete.replace('A thoughtful book.', '\n'+literal) + result = fixture.run_score(text) + assert result['passed'] + assert result['metrics']['article_comment_count'] == 1 + + +@pytest.mark.parametrize('spaces', [0, 1, 2, 3]) +def test_ordinary_markdown_heading_indentation_remains_valid(spaces): + fixture = component_fixture.ArticleComponentIntegration() + fixture.setUp() + text = fixture.complete.replace('# Life and dignity', ' '*spaces+'# Life and dignity') + assert fixture.run_score(text)['passed'] + + +@pytest.mark.parametrize('author,expected', [('返信', False), ('2026-10-22T12:34:00Z', False), ('\u200b', False), ('Reply researcher', True), ('🙂', True)]) +@pytest.mark.parametrize('quoted', [False, True]) +def test_native_and_quote_authors_follow_task_policy(author, expected, quoted): + from test_article_cleaned_html import POSTS + from copy import deepcopy + case, stable, _ = fixture(38) + contract = deepcopy(POSTS) + contract.update(minimum_posts=1, non_author_labels=['返信'], allow_empty_outer_with_quotes=True) + case['source_benchmark']['post_contract'] = contract + if quoted: + fields = 'Real reader
'+author+'

Visible quote.

' + else: + fields = ''+author+'

Visible body.

' + content = '

'+stable+'

'+fields+'
' + assert evaluate_response(case, content.encode())['passed'] is expected diff --git a/tests/test_article_post_coverage_integration.py b/tests/test_article_post_coverage_integration.py new file mode 100644 index 0000000..75fa610 --- /dev/null +++ b/tests/test_article_post_coverage_integration.py @@ -0,0 +1,55 @@ +"""Current collection floors work through scoring without live references.""" +import json +import subprocess +import sys + +import pytest + +from lexbench_webfetch.article_dataset import DEFINITIONS +from lexbench_webfetch.evaluation import evaluate_response + + +def fixture(number): + case = next(c for c in json.loads((DEFINITIONS / 'tasks.json').read_text())['cases'] + if c['id'] == f'article-{number:03}') + stable = '\n\n'.join(a['alternatives'][0] for a in case['acceptance_contract']['stable_text_anchors']) + count = case['source_benchmark']['post_contract']['minimum_posts'] + prefix = 'https://xueqiu.com/123/' if number == 101 else 'https://x.com/browser_use/status/' + posts = [f'## [Current author {i}]({prefix}{90000+i})\n\nNew visible post text number {i}.' for i in range(count)] + return case, stable, posts + + +@pytest.mark.parametrize('number,minimum', [(38,4),(39,8),(40,4),(41,4),(101,16)]) +def test_current_collection_coverage_without_runtime_reference(number, minimum): + case, stable, posts = fixture(number) + assert case['source_benchmark']['post_contract']['minimum_posts'] == minimum + full = evaluate_response(case, (stable+'\n\n'+'\n\n'.join(posts)).encode()) + assert full['passed'] and full['measured'] + assert 'article_reference_status' not in full['metrics'] + assert full['metrics']['content_decision_basis'] == 'base_acceptance_and_required_components' + for text in (stable, stable+'\n\n'+'\n\n'.join(posts[:-1]), '\n\n'.join(posts)): + result = evaluate_response(case, text.encode()) + assert not result['passed'] + assert result['measured'] + + +def test_cli_accepts_collection_without_reference_arguments(tmp_path): + case, stable, posts = fixture(38) + task, answer = tmp_path/'task.json', tmp_path/'answer.md' + task.write_text(json.dumps(case)) + answer.write_text(stable+'\n\n'+'\n\n'.join(posts)) + result = subprocess.run([sys.executable, '-m', 'lexbench_webfetch.cli', 'score', + '--task', str(task), '--response', str(answer)], + capture_output=True, text=True, check=True) + assert json.loads(result.stdout)['passed'] + + +@pytest.mark.parametrize('number', [38, 41]) +def test_required_profile_fields_cannot_be_replaced_by_posts(number): + case, stable, posts = fixture(number) + full = stable + '\n\n' + '\n\n'.join(posts) + assert evaluate_response(case, full.encode())['passed'] + fields = ['Agents that use the Browser', 'Science & Technology', + 'San Francisco', 'browser-use.com', 'Joined November 2024'] + for field in fields: + assert not evaluate_response(case, full.replace(field, '').encode())['passed'] diff --git a/tests/test_article_posts.py b/tests/test_article_posts.py new file mode 100644 index 0000000..9568b8a --- /dev/null +++ b/tests/test_article_posts.py @@ -0,0 +1,82 @@ +"""Historical post qualification never scores an empty runtime answer.""" +from article_contract_fixtures import post_labels +from copy import deepcopy +import pytest +from lexbench_webfetch.article_posts import validate_post_contract, verify_article_posts +from lexbench_webfetch.article_qualification import qualify_article_posts + +CONTRACT = {**post_labels(), 'version': 2, 'hosts': ['example.test'], 'entry_path': r'/post/\d+', + 'html_record_selector': '.post', 'html_permalink_selector': 'a', + 'html_author_selector': '.author', 'html_body_selector': '.body', + 'minimum_posts': 1, 'allow_empty_outer_with_quotes': True} +REFERENCE = {'version': 1, 'case_id': 'feed', 'url': 'https://example.test/feed', + 'scope': 'initial_load_only', 'context_id': 'historic', + 'observed_at': '2020-01-01T00:00:00Z', 'source_observation_counts': [1, 1], + 'records': [{'url': 'https://example.test/post/1', 'author': 'Alice', + 'body_parts': [{'kind': 'post', 'text': 'A complete displayed post body.', + 'complete_initial_text': True}]}]} + + +def qualify(ref, contract=CONTRACT): + return qualify_article_posts(contract, ref, case_id='feed', url=REFERENCE['url']) + + +def test_historical_posts_need_no_current_clock_or_answer(): + assert qualify(REFERENCE)['status'] == 'qualified' + + +@pytest.mark.parametrize('change', [ + {'version': True}, {'case_id': 'other'}, {'url': 'https://other.test'}, + {'scope': 'expanded'}, {'context_id': ''}, {'observed_at': 'bad'}, + {'source_observation_counts': []}, {'source_observation_counts': [1]}, + {'source_observation_counts': [1, 2]}, {'source_observation_counts': [True, 1]}, + {'records': []}, {'records': None}, +]) +def test_bad_post_observation_fails(change): + ref = deepcopy(REFERENCE) + ref.update(change) + assert qualify(ref)['status'] != 'qualified' + + +@pytest.mark.parametrize('change', [ + {'url': 'https://foreign.test/post/1'}, {'url': 'https://example.test:bad/post/1'}, + {'author': ''}, {'body_parts': []}, + {'body_parts': [{'kind': 'post', 'text': 'Hidden body', 'complete_initial_text': False}]}, + {'body_parts': [{'kind': 'post', 'text': '', 'complete_initial_text': True}]}, +]) +def test_bad_post_owned_fields_fail(change): + ref = deepcopy(REFERENCE) + ref['records'][0].update(change) + assert qualify(ref)['status'] != 'qualified' + + +def test_quote_only_observation_requires_explicit_empty_outer_and_quote_author(): + ref = deepcopy(REFERENCE) + ref['records'][0]['body_parts'] = [ + {'kind': 'post', 'text': '', 'empty_outer_text': True, 'complete_initial_text': True}, + {'kind': 'quote', 'author': 'Bob', 'text': 'Complete attributed quoted text.', 'complete_initial_text': True}] + assert qualify(ref)['status'] == 'qualified' + del ref['records'][0]['body_parts'][1]['author'] + assert qualify(ref)['status'] == 'invalid' + + +def test_ranges_cannot_omit_text_and_duplicate_posts_fail(): + ref = deepcopy(REFERENCE) + ref['records'][0]['body_parts'][0]['source_line_ranges'] = [[2, 2]] + assert qualify(ref)['status'] == 'invalid' + ref = deepcopy(REFERENCE) + ref['records'] *= 2 + ref['source_observation_counts'] = [2, 2] + assert qualify(ref)['status'] == 'invalid' + assert qualify(REFERENCE, dict(CONTRACT, minimum_posts=2))['status'] == 'invalid' + + +def test_runtime_rejects_legacy_post_contract_and_reference_argument(): + legacy = dict(CONTRACT, version=1, required_ratio=1, max_reference_age_seconds=900) + legacy.pop('minimum_posts') + with pytest.raises(ValueError): + validate_post_contract(legacy) + with pytest.raises(ValueError): + verify_article_posts('', legacy, url=REFERENCE['url']) + with pytest.raises(TypeError): + verify_article_posts('', CONTRACT, REFERENCE, url=REFERENCE['url']) diff --git a/tests/test_article_posts_fixed.py b/tests/test_article_posts_fixed.py new file mode 100644 index 0000000..0efbfe1 --- /dev/null +++ b/tests/test_article_posts_fixed.py @@ -0,0 +1,120 @@ +"""Fixed post coverage does not require historical content or a run reference.""" +from article_contract_fixtures import post_labels +from copy import deepcopy +import unittest + +from lexbench_webfetch.article_posts import validate_post_contract, verify_article_posts + + +class FixedPostTests(unittest.TestCase): + def setUp(self): + self.contract = {**post_labels(), + 'version': 2, 'hosts': ['example.com'], 'entry_path': r'/post/\d+', + 'html_record_selector': 'article.post', 'html_permalink_selector': 'a.time', + 'html_author_selector': '.author', 'html_body_selector': '.body', + 'html_quote_selector': 'blockquote', 'quote_markers': ['Quote', '引用'], + 'minimum_posts': 2, 'allow_empty_outer_with_quotes': True, + } + + def score(self, content): + return verify_article_posts(content, self.contract, url='https://example.com/feed') + + def html(self, number=1, author='New author', text='New content', quote=''): + return (f'
{author}' + f'Today' + f'
{text}
{quote}
') + + def md(self, number=1, author='New author', text='New content'): + return f'## {author} [Today](https://example.com/post/{number})\n\n{text}\n\n' + + def test_changed_posts_and_exact_floor(self): + for render in (self.html, self.md): + result = self.score(render(900, 'Different author', 'Different text') + render(901)) + self.assertTrue(result['article_posts_pass'], result) + self.assertNotIn('article_reference_status', result) + self.assertEqual(result['article_post_confirmed_count'], 2) + self.assertFalse(self.score(render(900))['article_posts_pass']) + + def test_invalid_records_do_not_count(self): + self.contract['minimum_posts'] = 1 + for bad in (self.html(author=''), self.html(text=''), + self.html().replace('/post/1', 'https://wrong.example/post/1'), + self.html().replace('/post/1', '/profile/1'), + self.html().replace('/post/1', 'https://user@example.com/post/1')): + with self.subTest(bad=bad): + self.assertFalse(self.score(bad)['article_posts_pass']) + self.assertTrue(self.score(bad + self.html(2))['article_posts_pass']) + + def test_permalink_label_is_not_an_author(self): + self.contract['minimum_posts'] = 1 + for label in ('Today', 'Permalink', '2026-10-20'): + self.assertFalse(self.score(f'## [{label}](https://example.com/post/1)\n\nBody') + ['article_posts_pass']) + self.assertTrue(self.score('## [Alice](https://example.com/post/1)\n\nBody') + ['article_posts_pass']) + + def test_duplicate_permalink(self): + for render in (self.html, self.md): + self.assertFalse(self.score(render() + render(text='Different words'))['article_posts_pass']) + + def test_hidden_and_closed_disclosure(self): + self.contract['minimum_posts'] = 1 + for wrapper in ('', '
{}
', + '
More{}
', + '
More
{}
'): + self.assertFalse(self.score(wrapper.format(self.html()))['article_posts_pass']) + self.assertTrue(self.score('
' + self.html() + '
')['article_posts_pass']) + + def test_quote_only_needs_attributed_nonempty_text(self): + self.contract['minimum_posts'] = 1 + good = '
Original authorOriginal text
' + for body in (self.html(text='', quote=good), + self.html(text='', quote=good).replace('
', ''), + self.md(text='> Original author\n> Original text')): + self.assertTrue(self.score(body)['article_posts_pass'], self.score(body)) + for quote in ('
Unattributed text
', + '
Only author
'): + self.assertFalse(self.score(self.html(text='', quote=quote))['article_posts_pass']) + self.assertFalse(self.score(self.md(text='> Only author'))['article_posts_pass']) + self.contract['allow_empty_outer_with_quotes'] = False + self.assertFalse(self.score(self.html(text='', quote=good))['article_posts_pass']) + + def test_metadata_only_bodies_and_quotes_do_not_count(self): + self.contract['minimum_posts'] = 1 + metadata = ('Original author' + 'Today' + 'Like') + self.assertFalse(self.score(self.html(text='', quote='
' + metadata + + '
'))['article_posts_pass']) + # A real declared body is accepted despite neighboring controls. + self.assertTrue(self.score(self.html(text='', quote='
' + metadata + + '
Real quote words
' + + '
'))['article_posts_pass']) + for metadata_text in ('2026-10-20\nReply\nLike', '2 hours ago\n3 Likes | Reply', + '今天\n回复 | 点赞', '2026-10-20T12:00:00Z'): + self.assertFalse(self.score(self.md(text=metadata_text))['article_posts_pass']) + self.assertFalse(self.score(self.md(text='> Original author\n> ' + + metadata_text.replace('\n', '\n> '))) + ['article_posts_pass']) + self.assertTrue(self.score(self.md(text='I like the reply about 2026-10-20.')) + ['article_posts_pass']) + + def test_nested_cards_cannot_supply_outer_fields(self): + outer = '
OuterDate
' + self.html() + '
' + result = self.score(outer) + self.assertFalse(result['article_posts_pass']) + self.assertEqual(result['article_post_confirmed_count'], 1) + + def test_contract_bounds_and_version_fields(self): + validate_post_contract(self.contract) + for value in (0, -1, True, 1.5, None): + contract = {**post_labels(), **self.contract, 'minimum_posts': value} + with self.assertRaises(ValueError): + validate_post_contract(contract) + for key in ('required_ratio', 'max_reference_age_seconds'): + with self.assertRaises(ValueError): + validate_post_contract({**self.contract, key: 1}) + legacy = deepcopy(self.contract) + legacy.update(version=1, required_ratio=1, max_reference_age_seconds=900) + with self.assertRaises(ValueError): + validate_post_contract(legacy) diff --git a/tests/test_article_title.py b/tests/test_article_title.py new file mode 100644 index 0000000..fc8ab42 --- /dev/null +++ b/tests/test_article_title.py @@ -0,0 +1,111 @@ +"""The title is an independent component, not a substring of article prose.""" +import unittest + +from lexbench_webfetch.article_title import validate_title_contract, verify_article_title + + +class ArticleTitleTests(unittest.TestCase): + def setUp(self): + self.contract = {'version': 1, 'alternatives': ['Why We Read'], + 'html_selector': 'h1.article-title'} + + def passes(self, content): + return verify_article_title(content, self.contract)['article_title_pass'] + + def test_plain_and_markdown_titles_pass(self): + for text in ['Why We Read', '# Why We Read', '## **Why We Read** ##', + '[Why We Read](https://example.test/story)', + '# [Why We Read](https://example.test/story)', + 'Why We Read\n==========\n\nThe opening paragraph.', + '# Why We Read\n\nThe opening paragraph.']: + with self.subTest(text=text): + self.assertTrue(self.passes(text)) + + def test_body_mentions_wrong_title_and_quoted_snippets_fail(self): + for text in ['The essay Why We Read explains the value of books.', + '# Another Article\nWhy We Read explains the value of books.', + '> Why We Read', '- Why We Read', 'Prefix Why We Read', + 'Why We Read suffix', '```text\nWhy We Read\n```', + '~~~\nWhy We Read\n~~~']: + with self.subTest(text=text): + self.assertFalse(self.passes(text)) + + def test_selected_visible_html_heading_passes(self): + self.assertTrue(self.passes('

Why We Read

' + '

The complete prose follows.

')) + self.assertTrue(self.passes('

Why We Read

')) + + def test_html_prose_and_unselected_heading_cannot_supply_title(self): + for text in ['

Why We Read

', + '

Why We Read

', + '

Another Article

Why We Read

', + '

Why We Read extra

', + 'Why We ReadProse']: + with self.subTest(text=text): + self.assertFalse(self.passes(text)) + + def test_hidden_html_metadata_and_image_alt_cannot_supply_title(self): + for text in ['

Why We Read

', + '

Why We Read

', + '

Why We Read

', + '

Why We Read

', + '

Why We Read

', + '

', + '# ', + '# Header\n', + '![Why We Read](https://example.test/image.png)', + '[![Why We Read](https://example.test/image.png)](https://example.test/story)']: + with self.subTest(text=text): + self.assertFalse(self.passes(text)) + + def test_closed_details_title_is_hidden_but_summary_and_open_content_are_visible(self): + self.assertFalse(self.passes('
More' + '

Why We Read

' + '
')) + self.assertTrue(self.passes('
More' + '

Why We Read

' + '
')) + self.assertFalse(self.passes('# Other title\n
More\n' + 'Why We Read\n
')) + contract = {**self.contract, 'html_selector': 'details > summary'} + self.assertTrue(verify_article_title( + '
Why We Read

Hidden

', contract + )['article_title_pass']) + + def test_reviewed_css_hidden_duplicate_cannot_supply_title(self): + contract = {**self.contract, 'html_selector': '.text-title h1,h1,h2,[role="heading"]', + 'excluded_html_selectors': ['h1.screen-reader-only']} + hidden = '

Why We Read

' + self.assertFalse(verify_article_title(hidden, contract)['article_title_pass']) + self.assertTrue(verify_article_title( + hidden + '

Why We Read

', contract + )['article_title_pass']) + self.assertTrue(verify_article_title('

Why We Read

', contract)['article_title_pass']) + self.assertFalse(verify_article_title( + hidden + '

Why We Read is mentioned in the original body.

', contract + )['article_title_pass']) + for exclusions in ['h1', [None], [''], ['['], True]: + with self.subTest(exclusions=exclusions): + with self.assertRaises(ValueError): + validate_title_contract({**self.contract, + 'excluded_html_selectors': exclusions}) + + def test_alternative_and_whitespace_normalization(self): + contract = {**self.contract, 'alternatives': ['Why We Read', '为何阅读']} + self.assertTrue(verify_article_title('# 为何阅读', contract)['article_title_pass']) + self.assertTrue(self.passes(' WHY WE READ ')) + chinese = {**self.contract, 'alternatives': ['从产业细微处 ， 看见周期起落']} + self.assertTrue(verify_article_title('从产业细微处，看见周期起落', chinese)['article_title_pass']) + + def test_contract_rejects_invalid_inputs(self): + for changes in [{'version': True}, {'version': 2}, {'alternatives': []}, + {'alternatives': ['']}, {'alternatives': ['Why\nRead']}, + {'alternatives': [None]}, {'html_selector': ''}, + {'html_selector': '['}, {'unknown': 1}]: + with self.subTest(changes=changes): + with self.assertRaises(ValueError): + validate_title_contract({**self.contract, **changes}) + with self.assertRaises(ValueError): + validate_title_contract(None) + with self.assertRaises(ValueError): + verify_article_title(b'Why We Read', self.contract) diff --git a/tests/test_benchmark.py b/tests/test_benchmark.py index 0a352e5..e0305bd 100644 --- a/tests/test_benchmark.py +++ b/tests/test_benchmark.py @@ -18,8 +18,8 @@ def write(path, value): class PublicDefinitions(unittest.TestCase): def test_all_shipped_suites_are_consistent(self): rows = [bench.validate(row['id']) for row in bench.registry()['suites']] - self.assertEqual(len(rows), 4) - self.assertEqual(sum(row['case_count'] for row in rows), 450) + self.assertEqual(len(rows), 5) + self.assertEqual(sum(row['case_count'] for row in rows), 550) def test_nested_reference_bodies_are_bound_without_being_distributed(self): body = 'Third-party reference paragraph. ' * 400 diff --git a/tests/test_body_ending.py b/tests/test_body_ending.py new file mode 100644 index 0000000..dfc5f29 --- /dev/null +++ b/tests/test_body_ending.py @@ -0,0 +1,42 @@ +"""Strict body contracts protect the declared ending without changing legacy ones.""" +import copy +import pytest +from lexbench_webfetch.body_coverage import body_requirements, evaluate_body_requirements + +INTRO = "Scientists map distant galaxies and record their motion across the sky. " * 12 +ENDING = "The closing recommendation requires an independent replication before deployment." + + +def contract(): + rule = body_requirements(INTRO + ENDING, structure_hint="whole_declared_component") + rule.update(version=4, ending={"text": ENDING}, minimum_floor=.8, minimum_region_passes=1) + return rule + + +def test_ending_rejects_truncation_that_whole_region_tolerates(): + strict = contract() + legacy = copy.deepcopy(strict) + legacy["version"] = 3 + del legacy["ending"] + assert evaluate_body_requirements(INTRO, legacy)["body_coverage_pass"] + assert not evaluate_body_requirements(INTRO, strict)["body_coverage_pass"] + for text in [INTRO + ENDING, INTRO + "\n**" + ENDING + "**"]: + assert evaluate_body_requirements(text, strict)["body_coverage_pass"] + + +@pytest.mark.parametrize("ending", [None, {}, {"text": ""}, {"text": 3}]) +def test_invalid_or_missing_ending_fails_closed(ending): + rule = contract() + if ending is None: + del rule["ending"] + else: + rule["ending"] = ending + with pytest.raises(ValueError): + evaluate_body_requirements(INTRO + ENDING, rule) + + +def test_strict_contract_cannot_allow_a_weak_region(): + rule = contract() + rule["minimum_floor"] = .6 + with pytest.raises(ValueError): + evaluate_body_requirements(INTRO + ENDING, rule) diff --git a/tests/test_required_components.py b/tests/test_required_components.py new file mode 100644 index 0000000..74938d4 --- /dev/null +++ b/tests/test_required_components.py @@ -0,0 +1,32 @@ +"""Optional components must participate in the shared verdict and rescore path.""" +from lexbench_webfetch.metrics import task_passed +from lexbench_webfetch.scoring import score_attempt +from lexbench_webfetch.benchmark import public_task + + +def test_arbitrary_required_component_cannot_be_bypassed_by_stable_pass(): + metrics = {'stable_task_pass': True, 'content_pass': True, 'length_pass': True, + 'required_components': {'future_component': False}} + assert not task_passed(True, metrics) + metrics['required_components']['future_component'] = True + assert task_passed(True, metrics) + assert not task_passed(False, metrics) + + +def test_component_failure_is_removed_when_rescoring_unrelated_task(): + task = {'id': 'synthetic', 'url': 'https://example.test/article', + 'min_content_chars': 1, 'expected_any': []} + result = score_attempt(task, b'Ordinary visible body text.', ok=True, + previous_metrics={'required_components': {'retired': False}, + 'article_comments_pass': False}) + assert 'required_components' not in result + assert 'article_comments_pass' not in result + assert task_passed(True, result) + + +def test_registered_component_contracts_are_public_without_private_evidence(): + from lexbench_webfetch.required_components import COMPONENTS + benchmark = {key: {'version': 2} for key in COMPONENTS} + benchmark['comment_qualification_proof'] = {'path': 'private-capture.json'} + task = public_task({'id': 'synthetic', 'source_benchmark': benchmark}) + assert set(task['source_benchmark']) == set(COMPONENTS) diff --git a/tests/test_shared_disclosure_visibility.py b/tests/test_shared_disclosure_visibility.py new file mode 100644 index 0000000..5e7318b --- /dev/null +++ b/tests/test_shared_disclosure_visibility.py @@ -0,0 +1,34 @@ +"""Bodies, stable labels and article components share native disclosure visibility.""" +import pytest +from lexbench_webfetch.body_coverage import body_requirements +from lexbench_webfetch.dynamic_lists import visible_html_text +from lexbench_webfetch.scoring import score_attempt + +TEXT = 'Complete authored explanation of the mountain expedition and its route.' + + +@pytest.mark.parametrize("wrapper", ["

{}

", "{}"] ) +def test_closed_body_does_not_supply_reference_coverage(wrapper): + contract = body_requirements(TEXT) + closed = '
Overview' + wrapper.format(TEXT) + '
' + opened = closed.replace('
', '
') + task = {'id': 'body-visibility', 'url': 'https://example.test/article', + 'body_requirements': contract, 'min_content_chars': 1} + assert not score_attempt(task, closed.encode(), ok=True)['body_coverage_pass'] + assert score_attempt(task, opened.encode(), ok=True)['body_coverage_pass'] + + +def test_only_first_direct_summary_survives_closed_ancestor(): + html = ('
Shown labelSecond hidden label' + '
Nested hidden labelHidden text
') + assert visible_html_text(html).strip() == 'Shown label' + + +def test_registered_visible_control_and_disclosure_verdicts(): + from lexbench_webfetch.scoring_regressions import load_scoring_regressions + controls = load_scoring_regressions() + from lexbench_webfetch.metrics import task_passed + for key, row in controls.items(): + if key.startswith(('visible-control-', 'native-disclosure-')): + metrics = score_attempt(row['case'], row['artifact'].encode(), ok=row['ok']) + assert task_passed(row['ok'], metrics) == row['expected_task_pass'], key