Skip to content

status_evidence: an artifact whose work SHIPPED can stay proposed — R3 cannot see manual: done-when, R4's subject convention no longer matches, R10 passes anyway #1250

Description

@avrabe

An artifact whose work has SHIPPED can sit at status: proposed indefinitely and no rule notices. RQ-66-PINDEBT is the live instance: it merged twice#1242 (the known_open_pins ratchet) and #1247 (the DictComp counter) — has a filled verified-by:, and is still proposed on main. status_evidence_check.py exits 0.

Three rules could each have caught it. Each misses for a different, checkable reason.

R3 cannot see a manual: done-when

R3 is the right rule: "a NON-claiming status whose evidence evaluates TRUE is red — the artifact's own done-signature exists on main while the status says otherwise."

But it can only fire on a predicate it can evaluate, and every v0.66 artifact uses a prose one:

$ grep -l 'done-when: "manual:' artifacts/release-v0.66/*.yaml | wc -l
8          # 8 of 8

A manual: done-when is invisible to R3 by construction, so R3 is vacuous for this entire release.

R4 expects a commit-subject convention this release does not use

R4 requires the subject to start with the artifact id, from a measured convention: "28 of 28 delivery commits on main at authoring" in the shape RQ-59-TIERCENSUS (#1021): .... Every v0.66 delivery commit uses a conventional-commit prefix instead:

fix(mutants):   RQ-66-POTENCY (#1189) — ...
fix(selector):  RQ-66-BOTHWRONG (#1210) — ...
feat(toolchain):RQ-66-VARVE (#1236) — ...
feat(gate):     RQ-66-PINDEBT (#242) — ...

The id is present but not first, so R4 never matches and never demands acknowledgement.

R10 passes, correctly, and that is the trap

R10 is the weaker completeness floor: a delivery-shaped commit must be attributable to some artifact, by an id anywhere in the subject. All six match. So the gate reports success — and its success is what makes the other two failures invisible.

Why this matters beyond one status field

The gate checks that delivery commits point at an artifact. Nothing checks the converse: that an artifact with delivered work says so. A release can therefore ship every artifact and leave them all proposed, with status_evidence green throughout, and the traceability gate at release time would then be reasoning about a set that understates what shipped.

It is the same shape as the other instrument defects this release found: the check measures a proxy (attributability) for the property it is named after (the status agrees with reality).

Suggested direction, not a prescription

  • Make R4's id-match position-independent, matching R10's, so the two rules cover the same commits rather than differing by an accident of commit style.
  • Give R3 something to evaluate when done-when is manual: — at minimum, require that an artifact with a merged PR named in landed: or with delivery commits attributed to it is not left at a non-claiming status.

Related: RQ-66-BOTHWRONG and RQ-66-VARVE are implemented with no landed: field at all; they satisfy attribution only through R10.

Found by the v0.66 release loop while deriving release notes from the artifact set. Attribution: the maintainer's autonomous loop (same avrabe account).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions