Skip to content

Investigate reliable content coverage for changing article post lists #9

Open

Description

@Maisiqi

Problem

PR #5 retains five changing post-list cases: article-038/039/040/041/101. Their current acceptance checks cover required stable text and distinct attributed records with nonempty bodies, using count floors of 4/8/4/4/16. They do not establish completeness of each displayed post body.

The reviewer reproduced that all five cases pass when each qualifying post body is replaced with x; removing the records fails. See the review findings. This is a known limitation of the current contract, not a newly introduced regression.

Current scope decision

Retain the five lists and existing structural checks in PR #5. Do not describe a passing result as proof that every displayed post is complete. Do not replace these cases, lower existing thresholds, freeze historical post text, or add arbitrary minimum text lengths to make this limitation disappear. The user agreed to retain this scope for now; this issue does not assert reviewer approval or resolve the separate reference-material distribution concern.

Follow-up investigation

Prototype in the execution harness before changing this repository's contracts:

  • Assess whether an independent contemporaneous page reference can be collected reliably and compared with provider output. A separate page load may return different posts, order, or visible text; source disagreement must not automatically become a fetch failure or a pass.
  • Limit the reference to automatically displayed content. Do not click, expand, switch tabs, log in, or require hidden full-post text. Preserve legitimately short posts and quoted content.
  • Keep the acceptance procedure reusable while allowing the observed records to change. Never use the evaluated provider's own output as its reference.
  • Measure acquisition latency, failure rate, page disagreement, and operational cost on these five cases. Report whether the stronger check is practical before adopting it.
  • If justified by that experiment, define generic, data-driven per-record coverage in this repository; keep provider adapters, reference acquisition, and run evidence in the harness or external storage.

Completion evidence

Demonstrate paired controls for complete displayed text, genuinely short posts, truncated text, missing records, wrong content, duplicates/padding, and differing or unavailable source views. Document how mismatched source views are reported as inconclusive. State the precise guarantees and remaining limits; do not claim universal completeness.

Closing this issue requires a supported implementation or an evidence-backed decision to retain structural-only coverage, not merely adding a text-length threshold. No per-run reference workflow is implemented or authorized by filing this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions