Repository navigation
fix(scoring)!: combine declared acceptance requirements - #8
Draft
lanyue-llk wants to merge 13 commits into
Draft
lanyue-llk wants to merge 13 commits into
lanyue-llk wants to merge 13 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
A task can require records, facts and a resource at the same time. Previously, a passing stable-record contract could hide a missing fact or resource. The final predicate selected a primary result by task kind, and stable content evaluation discarded the fact requirement.
The shared scorer now selects every declared requirement and combines their results. Literal and minimum-length checks follow their declared text applicability; explicit facts, body scope, patterns, resources and files remain independent requirements. Document-language checks follow declared identity fields across all tracks. Body, structure, resource and file tasks use that same combination. Undeclared checks are skipped; missing evidence for a required check is unavailable and cannot pass. An explicitly required tree similarity also contributes to acceptance instead of replacing other requirements. Full-text overlap remains diagnostic.
flowchart LR A[Task requirements] --> B[Shared score_attempt] C[Saved answer and delivered files] --> B B --> D[Facts, body, records, patterns, resources and files] D --> E{All declared requirements pass?} E -->|Yes| F[Pass] E -->|No| G[Fail or required evidence unavailable]Existing field matchers remain shared; no site-specific scorer is added. Output labels do not add undeclared record checks. Dictionary and dataclass inputs apply the same default text requirement, and the low-level content checker retains declared literals and minimum length. Optional diagnostics cannot invalidate a body whose declared regions are fully delivered.
Main-content coverage
This PR also tracks the source-coverage work requested in the review. A shared scorer cannot compensate for a reference that only contains navigation. The registered popular-site suite has 57 body-text contracts and 43 named-link contracts, so the review's blanket “100 navigation-only cases” description is too broad. Neither count proves complete main-content coverage.
Independent HTTP collection attempted all 200 dynamic/popular-site tasks: 192 HTTP 200 responses, six HTTP 403 responses and two transport failures. The browser collector covered 200 tasks and retained every physical observation; after a declared retry, 193 acquired content and seven retained acquisition failures through the maintained capture entry. Acquisition success alone does not qualify main-content coverage. Coverage changes need source-qualified task versions; the historical dataset files and published scores remain unchanged. Real-source controls now accept an independently observed Yuque page as visible text and HTML, reject its navigation-only text, and reject all five single-section deletions through the shared scorer. These eight controls establish the mechanism for the five selected explanations. Five further real-source controls cover two independent Tabulator captures, its title-only scaffold, a list missing five rows, and records missing the explanatory body. These controls do not establish full coverage for the entire suite. A scoped Tabulator capture also identified valid table records rejected by historical page-shell anchors; DataPass body regions contain dashboard labels without their numeric values. Those task definitions need independently qualified revisions. This PR stays in draft while task-by-task main-content revisions are incomplete.
Compatibility
Required facts and applicable literal/length fields that stable scoring formerly treated as diagnostics now affect acceptance. Historical exports remain reproducible with their pinned scorer revision. Stored metrics retain their recorded legacy acceptance semantics when displayed; only a complete rescore generates the new requirement map. Newly scored responses use the normalized request status as well as every declared check. Adopting this change requires an audited offline rescore and source-qualified input revisions wherever the required content changes.
Validation
make check: 2,720 tests and 321 subtests passed; all five registered suites validate (550 tasks).Depends on #7; the base branch keeps the new diff separate from the existing input/provenance fixes.
Base integration
Includes #7 through
6bd263ed, including current main's article contracts and the shared README banner. After integration,make checkpassed 2,720 tests and 321 subtests. The per-page main-content revisions described above remain incomplete, so this PR stays in draft.New upstream components
Title, comment and post contracts now appear in the same
acceptance_checksinventory as the other declared requirements. They already contributed to the final verdict; the shared diagnostic inventory had omitted them. Seventy-two complete/missing-title/missing-comment controls cover six reporting families and four delivery kinds. The five registered changing-post collections also check complete, missing-post and missing-profile-field answers. Apache licensing and the installed-package language boundary are synchronized from #7.