Skip to content

fix(scoring)!: combine declared acceptance requirements - #8

Draft
lanyue-llk wants to merge 13 commits into
codex/notion-scoring-fixes-currentfrom
codex/unified-webfetch-acceptance
Draft

lanyue-llk wants to merge 13 commits into
codex/notion-scoring-fixes-currentfrom
codex/unified-webfetch-acceptance

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Change

A task can require records, facts and a resource at the same time. Previously, a passing stable-record contract could hide a missing fact or resource. The final predicate selected a primary result by task kind, and stable content evaluation discarded the fact requirement.

The shared scorer now selects every declared requirement and combines their results. Literal and minimum-length checks follow their declared text applicability; explicit facts, body scope, patterns, resources and files remain independent requirements. Document-language checks follow declared identity fields across all tracks. Body, structure, resource and file tasks use that same combination. Undeclared checks are skipped; missing evidence for a required check is unavailable and cannot pass. An explicitly required tree similarity also contributes to acceptance instead of replacing other requirements. Full-text overlap remains diagnostic.

flowchart LR
    A[Task requirements] --> B[Shared score_attempt]
    C[Saved answer and delivered files] --> B
    B --> D[Facts, body, records, patterns, resources and files]
    D --> E{All declared requirements pass?}
    E -->|Yes| F[Pass]
    E -->|No| G[Fail or required evidence unavailable]
Loading

Existing field matchers remain shared; no site-specific scorer is added. Output labels do not add undeclared record checks. Dictionary and dataclass inputs apply the same default text requirement, and the low-level content checker retains declared literals and minimum length. Optional diagnostics cannot invalidate a body whose declared regions are fully delivered.

Main-content coverage

This PR also tracks the source-coverage work requested in the review. A shared scorer cannot compensate for a reference that only contains navigation. The registered popular-site suite has 57 body-text contracts and 43 named-link contracts, so the review's blanket “100 navigation-only cases” description is too broad. Neither count proves complete main-content coverage.

Independent HTTP collection attempted all 200 dynamic/popular-site tasks: 192 HTTP 200 responses, six HTTP 403 responses and two transport failures. The browser collector covered 200 tasks and retained every physical observation; after a declared retry, 193 acquired content and seven retained acquisition failures through the maintained capture entry. Acquisition success alone does not qualify main-content coverage. Coverage changes need source-qualified task versions; the historical dataset files and published scores remain unchanged. Real-source controls now accept an independently observed Yuque page as visible text and HTML, reject its navigation-only text, and reject all five single-section deletions through the shared scorer. These eight controls establish the mechanism for the five selected explanations. Five further real-source controls cover two independent Tabulator captures, its title-only scaffold, a list missing five rows, and records missing the explanatory body. These controls do not establish full coverage for the entire suite. A scoped Tabulator capture also identified valid table records rejected by historical page-shell anchors; DataPass body regions contain dashboard labels without their numeric values. Those task definitions need independently qualified revisions. This PR stays in draft while task-by-task main-content revisions are incomplete.

Compatibility

Required facts and applicable literal/length fields that stable scoring formerly treated as diagnostics now affect acceptance. Historical exports remain reproducible with their pinned scorer revision. Stored metrics retain their recorded legacy acceptance semantics when displayed; only a complete rescore generates the new requirement map. Newly scored responses use the normalized request status as well as every declared check. Adopting this change requires an audited offline rescore and source-qualified input revisions wherever the required content changes.

Validation

  • Reproduced missing facts/resources alongside a valid record contract, literal/length bypasses, and language/body checks that depended on track or output labels before fixing them.
  • make check: 2,720 tests and 321 subtests passed; all five registered suites validate (550 tasks).
  • Accepted/rejected controls cover mixed requirements across body, structure, resource and file deliveries, missing required evidence, and a perfect tree with a missing fact.
  • Paired controls are registered in the public scoring-regression inventory.
  • Compared frozen predicate projection on both real 430×6 matrices (original and revised definitions): 5,160 supported metric rows, zero verdict changes. This is a reader-compatibility check, not a rescore.
  • Existing task/catalog/input hashes are preserved. The scorer source identity changes; adopting it for benchmark results requires an audited offline recomputation, without new provider calls for unchanged inputs.

Depends on #7; the base branch keeps the new diff separate from the existing input/provenance fixes.

Base integration

Includes #7 through 6bd263ed, including current main's article contracts and the shared README banner. After integration, make check passed 2,720 tests and 321 subtests. The per-page main-content revisions described above remain incomplete, so this PR stays in draft.

New upstream components

Title, comment and post contracts now appear in the same acceptance_checks inventory as the other declared requirements. They already contributed to the final verdict; the shared diagnostic inventory had omitted them. Seventy-two complete/missing-title/missing-comment controls cover six reporting families and four delivery kinds. The five registered changing-post collections also check complete, missing-post and missing-profile-field answers. Apache licensing and the installed-package language boundary are synchronized from #7.

@lanyue-llk lanyue-llk changed the title fix(scoring): combine declared requirements and qualify main content fix(scoring)!: combine declared acceptance requirements Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant