Skip to content

Validation parity: close the first public-evidence certification gaps #9

Open

Description

@hanklin9188

Parent

What to deliver

Convert the first evidence-blocking rules in specs/validation_requirements.yaml from partial/planned to independently enforced, regression-tested requirements.

Blocked by

Required closure set

  • VR-002 raw-summary recomputation parity
  • VR-005 exact cross-runtime workload/token parity
  • VR-006 same-precision correctness recomputation
  • VR-010 quality recomputation and protocol parity
  • VR-014 search/holdout separation
  • VR-015 paired uncertainty and practical effect
  • VR-016 causal evidence promotion
  • VR-018 public-claim scope audit
  • VR-020 fairness artifact required for ranking
  • VR-021 raw-artifact immutability/transformation provenance
  • VR-022 capacity failures excluded from speed ranking

Implementation method for every requirement

  1. Name the public certification seam.
  2. Build the smallest red fixture that violates only the target rule.
  3. Confirm the fixture currently exposes the gap.
  4. Implement independent enforcement from primary artifacts where possible.
  5. Add valid, invalid, missing-evidence, and unrelated-regression cases.
  6. Change the matrix status to implemented in the same PR only after tests pass.
  7. Run Standards and Specification reviews independently.

Acceptance criteria

  • Each closed rule has an executable enforcement path and focused tests.
  • A misleading pass=true attestation cannot override corrupted underlying artifacts.
  • Missing correctness, quality, fairness, or holdout evidence cannot become policy/public eligible.
  • Outcome-only causal results cannot exceed E2.
  • OOM/timeout rows remain visible as capacity boundaries and never receive speed scores.
  • python scripts/audit_validation_parity.py passes.
  • python scripts/audit_validation_parity.py --strict-release --release-profile first_evidence passes before the first evidence release.
  • Full EdgeFlow CI remains green.

Out of scope

This issue does not run E04–E09. It creates the certification system those experiments require.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions