Repository navigation
Add 100 article cases with initial title, body and discussion checks - #5
Conversation
lanyue-llk
left a comment
There was a problem hiding this comment.
Reviewed at e18f1e8. Two reproducible scoring defects need changes; see inline comments.
The purpose is to add a separately versioned 100-case article population and make title/discussion requirements additive to the shared scorer. The component registry and task-declared selectors/labels are appropriate reuse boundaries: they avoid provider-specific rules and report dependencies. A class hierarchy would not improve this small registry. Existing suite input files remain unchanged, while the shared visibility change intentionally changes scorer identity.
Validation: definition validation succeeded; the existing suite passed with 2,125 tests and 310 subtests. Additional controls exercised all five current collection tasks through evaluate_response: five real-post controls passed, but ten negative controls incorrectly passed (five indented-code answers and five answers with dates instead of author attribution). The indented-code issue also reproduces for comment and title components. These are scoring defects, not provider evaluations or claims about historical measured results.
|
Reviewed all 100 article cases for main-content/ROI coverage at [P2] Protect the profile introduction in article-038 and article-041. Both pages visibly contain the Browser Use introduction, but their required stable anchors cover account identity, tabs and footer without the bio, category, location, website or joined date. Cases 039/040 already require those fields for the same account. Reuse a shared profile module across these page variants. Footer, navigation and login checks can remain; add independent protection for the main profile content. [P2] Record-count acceptance does not protect dynamic post content in article-038/039/040/041/101. Reproduced through public Use shared, composable checks for source readiness, distinct records, owned fields, reply/repost/quote relationships and per-record content coverage. Dynamic coverage needs an independent contemporaneous source scope rather than historical membership or literal wording. Structural/length checks can detect severe truncation but cannot alone prove semantic correctness; any semantic coverage judge needs paired valid, truncated, wrong-content and padding controls. Preserve genuinely short posts. For fixed long articles, keep the existing reference coverage and add independent checkpoints for each major module where large regions combine them (for example 022, 037, 066, 073). For tables and interviews, preserve field/speaker associations using shared structured-record checks. These are coverage improvements, not claimed reproductions of a false pass for each static case. All 100 URLs were attempted. Six browser failures/incomplete loads were supplemented with primary-site search retrieval, which establishes content scope but not contemporaneous rendering state. The external hash-bound reference bundle was not supplied, so this review does not claim byte-for-byte validation of those historical captures. Please provide a retrievable bundle location for reproducibility. 逐项 ROI 与验收建议:100 个案例现有页脚、侧栏和按钮要求均可保留。分区数量不等于主要模块已经逐一独立必过。折叠片段按已声明的首次展示范围验收;动态帖子与评论不使用历史文字匹配。article-090 已保留默认年龄表,article-082 已保留 Editorial Note,应继续保留这些正确范围。
|
|
Follow-up review at Verified improvements: 038/041 now require the missing profile fields, with paired public-scorer tests. All 95 fixed cases use body contract v4. Comparing the 100 rules with the previously reviewed version confirms that only 001/002/003/077 changed their URL/body/title references; comment and post contracts are unchanged. Expanding review defaults changes only the four replacements and two corrected profile cases. The complete package test run passed: 2,272 tests and 310 subtests. The following requirements remain open; the PR already documents most of these as deliberate limits, so they should not be reported as newly introduced implementation regressions:
Footer, sidebar and login checks remain valid supplementary requirements. Shared body/stable-text machinery is reused, and adding object-oriented layers solely for style would not address these acceptance gaps. No article-specific semantic scorer or provider calls were added by this review. |
|
针对本轮 review 的第 2 项(正文主要模块覆盖),补充明确的处理决定及理由: 认可指出的限制,但本次保留现有正文方案,不声称已经解决“每个主要模块都不可遗漏”。 95 个固定案例最多划分 5 个正文区,每区及独立结尾文字均须达到 80% 的覆盖率。一个较短的小节仍可能整体落在所属分区的容差内;结尾文字检查也不证明位置、顺序、表格单元格或发言人归属正确。 保持这一方案的技术理由:
因此本次选择保留经过验证的较窄覆盖要求,明确披露其局限,不再增加模块检查层,也不降低现有门槛。 这是在说明收益、代价和未覆盖情况后确认的范围决定,不是认定 reviewer 的发现错误,也不是为了让某个 Fetch 方法通过。已知反例同样保留:仅删除声明的结尾时,069/100 可能因其他位置的重复文字仍通过;这些结果不被表述为位置完整性。 暂不为正文分区另开 issue:当前没有确定要推进“所有主要模块逐一必过”的扩展,现有决策与验证限制已记录在 动态列表是另一项问题:目前缺少逐条正文覆盖检查,已由 #9 跟踪可行性验证。该 issue 不代表本次已经实现动态正文完整性,也不代表 reviewer 已认可放行。第 3 项参考材料的外部交付仍单独待完成,不因上述范围决定而视为解决。 |
|
针对第 3 项“Independent reproduction”,已核对旧测试集的实际处理方式,现明确本 PR 的范围决定: 本次沿用原有参考输入的交付方式,不在 PR #5 新增公开全文评分包,也不为此改造评分器。 核查依据:
选择沿用的理由是:PR #5 的目标是增加文章测试规则及相应评分支持;如果本次单独建立全文参考数据的公开托管、分发条件和维护机制,会扩大为新旧测试集共同的数据发布工作。现阶段保持既有输入边界,既不改变评分门槛,也不引入指纹格式或其他替代评分流程。用户已明确确认这一范围决定。 仍认可你指出的实际限制:没有获得匹配输入副本的外部使用者,无法仅凭公开仓库完整复现历史评分;哈希也不能代替原始参考材料的独立审查。 “沿用旧方式”并不证明这个缺口不存在。本次不声称已经提供公开下载入口、不声称原始微信页面已被 reviewer 独立核验,也不会把本地验证或 CI 通过当成公开复现完成。 因此,这项以“本 PR 保持既有外部输入模式,公开分发暂不纳入本次范围”回应,而不是以“已修复公开复现”回应。后续若建立公开参考输入发布机制,应统一考虑原有四套和新增文章套件。请据此评估本 PR 的范围;此回复不代表 reviewer 已同意该取舍。 |
|
P0/P1-focused follow-up at ba7bdae: no new P0 or P1 finding in the examined shared-scoring changes and acceptance boundaries. The latest change from f38bccb affects documentation only; scoring code, frozen case inputs and tests are unchanged. The complete package suite plus the 15 earlier independent record-recognition controls passed: 2,287 tests and 310 subtests. The known dynamic structural-coverage and fixed-region completeness limits remain disclosed; they are not being relabeled as newly discovered critical defects. No provider calls, historical result edits or publication were performed. This does not certify historical scores under the new scoring identity. |
中文图解
Change
Add an article category with 100 cases: 95 fixed articles/posts and five changing post collections. Current input version is
articles-v11, covering 72 platforms and 75 URL hosts. Four approved Weixin articles replace singleton-platform cases 001/002/003/077 without comment contracts. All 20 existing comment contracts are preserved.Use common compilation and scoring for displayed bodies, observed titles, stable page components and evidenced comments/posts. Task data supplies site/language exclusion labels; no provider adapters, browser execution, third-party captures or measured outputs are added to Git. Keep complete bodies and observations in external hash-bound inputs. The Weixin evidence is user-supplied initial-load webarchives with an explicit no-interaction attestation, not a successful independent live browser visit.
The shared scorer handles owned records, cleaned HTML, hidden disclosures, duplicate records, code/table literals, quote-only posts and invalid attribution. Public contracts and source bindings have consistency checks.
compile-articlesvalidates external evidence and refuses to overwrite pinned inputs.Current revision: scheme 2 and metadata deduplication
Following the user-approved v7 proposal, fixed articles retain at most five body regions. Every region must reach 80% fragment coverage, replacing the previous allowance for one region at 60%. Body contract version 4 additionally requires 80% coverage of the reviewed ending; a missing ending is invalid. No extra module-check layer is published. Legacy version-3 inputs retain their former scoring behavior.
Deduplicate repeated review basis/date fields into per-case defaults, preserving explicit regions and exceptions. Expanded review rows are identical, and acceptance/title/comment/post contracts are unchanged.
rules.jsondecreases from 3,078,113 to 2,211,902 bytes (7,671 repeated fields removed). Unlike a metadata-only revision, this version also intentionally tightens body acceptance.All 100 URLs and body references remain unchanged from v10. Preserve v10 rather than reusing its version: the four Weixin additions were already bound there. This revision advances to v11 and refreshes catalog, public tasks, input manifest, registry and bilingual documentation. Distribution counts are unchanged.
Validation
make check: 2,272 tests and 310 subtests passed, including complete/format-preserving responses, tail truncation, missing ending, the removed 60% allowance and review-default validation.verify-inputs --suite articles: all 100 v11 cases verified. Dataset SHA-256:1929d5f702a2f7a0a1248694ca6843fba02ecaf7c559c6bf232865a8fb3c4bb6.0372b25a89361cf4001630614a8cb7d20ddebb3b5a37c54bb95b330cb2c67a49. Existing suite input definitions remain unchanged, and legacy body-contract behavior is retained. Historical provider outputs for the other 450 cases were not available for rescoring; do not publish those historical results under the new scorer identity without rescoring.f38bccb51ad27a7d51b56dba1a52636eaca2c6e2: CI run. No merge or provider evaluation was performed.Review decisions and limits
See review decisions. Missing profile fields in 038/041 were corrected with paired controls. Independent module checks were investigated but not published; the approved minimal scheme retains a five-region design.
An 80% fragment threshold is not a precise limit on lost characters and cannot universally reject exactly 20% character deletion. It does not prove ordering, every intermediate module, table-cell/speaker association or the physical position of repeated ending text. Dynamic collection floors still measure structural coverage: distinct attributed links with one-character bodies can pass. Fixing semantic dynamic coverage would require a different approved reference/judging contract; no arbitrary character minimum is added to conceal this limitation.
Frozen input and raw-source evidence remain external. A local scoring-input package has been prepared, but no public evidence hosting location has been provisioned; this PR does not claim independent public reconstruction of historical browser state.