Skip to content

Re-apply #3784's binary-identity fix as a design: the ladder receipt names the CHECKOUT HEAD, not the binary that ran #3871

Closed as not planned

Description

@noahgift

Why a design ticket rather than "merge #3784"

PR #3784 (PMAT-3771-binary-identity) contains the right fix. It is not foldable, and the numbers that made it look foldable were computed against the wrong base.

git rev-list --left-right --count origin/release/0.69.1-batch-2...origin/PMAT-3771-binary-identity
  release-only: 140     head-only: 144     merge-base: a9502d992

Divergent, not ahead. The +689/−136 quoted in triage is the PR's diff against its own base branch (fix/0691-bump-fixture-libs) — #3784 does not target main. A number computed against a different base cannot answer "ahead or behind" for our line.

Cherry-picking fails too. Tested by comparing the files its key commit (68f5f0084) touches between that commit's parent and our release tip:

file parent vs release tip
scripts/model_ladder.sh DIVERGED → conflict
scripts/check_model_ladder.sh DIVERGED → conflict
contracts/model-capability-ladder-v1.yaml DIVERGED → conflict
scripts/check_release_bump_pr_body.sh identical → clean

And #3784 contains divergent copies of #3766's two commits (395756d06 / c08b9a89e vs 3e69272a1 / 36f77e0d6 — same subjects, different shas; 395756d06 is literally the parent of the commit above). Landing both as branches would land the Q4_K universe rewrite twice. Same hazard as #3804 vs #3807.

So: #3766 folds as a branch; this one re-applies as ~15 lines of intent.

The defect

scripts/model_ladder.sh on the release branch:

# apr_sha: the full 40-hex HEAD, which scripts/apr_bin.sh proved the binary was built from. The
APR_SHA=$(git rev-parse HEAD 2>/dev/null || echo unknown)

The receipt's binary identity is the checkout HEAD, justified by a comment asserting apr_bin.sh proved the binary matches. That is an assumption wearing a comment. Pinned, the two agree; unpinned (DOGFOOD_ALLOW_UNPINNED=1, or any external $APR) nothing ties them — #3784's own header records a run that recorded bfd99757c while it executed b89d126fc (PMAT-3724).

This bit us twice on 2026-09-22, including a session measuring the FP8 fix on a binary reporting apr 0.69.0 (85d8491f6) — a sha from a different branch than the tree being reasoned about.

Measured: the fail-closed decline matches 0 times on the release branch and 1 time on #3784.

The diff of intent — this is the whole change

Producer (scripts/model_ladder.sh), replacing the git rev-parse HEAD line:

# THE RECEIPT'S IDENTITY IS THE BINARY THAT RAN: what `apr --version` says — its version and the
# git sha it was built from — never the checkout's HEAD. The checkout's HEAD is kept as
# `checkout_sha`, which nothing judges.
APR_VERSION=$("$APR" --version 2>/dev/null | head -1)
SHA=$(sed -nE 's/.*\(([0-9a-f]{7,40})\).*/\1/p' <<< "$APR_VERSION")
[ -n "$SHA" ] || { echo "decline: \"$APR --version\" names no git sha ('${APR_VERSION:-<empty>}') — what ran cannot be identified, so no receipt can name it" >&2; exit 2; }
CHECKOUT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo unknown)
APR_SHA=$(git rev-parse --verify --quiet "${SHA}^{commit}" 2>/dev/null)

Note the <<< here-string rather than a pipe into sed — and keep it that way (scripts/check_no_pipe_into_grep_q.sh is the narrow cousin; #3864 is the general one).

Receipt schema: add checkout_sha as its own field. apr_sha becomes the binary's commit resolved in this repo, or empty.

Judge (scripts/check_model_ladder.sh), four refusals in order:

  1. receipt's apr_version carries no git sha → FAIL, "a checkout HEAD is not what ran"
  2. receipt's sha ≠ the sha of the binary it names → FAIL, "the identity must be what ran"
  3. binary sha ≠ the pinned release binary's sha (compared on the common prefix length) → FAIL, "a matching version string is not the same binary"
  4. the release binary itself cannot be pinned (scripts/apr_bin.sh declines) or names no sha in its --version → decline, exit 2

Refusal 3 is the load-bearing one: two builds of the same version string are not the same binary, and that is precisely how a stale binary passes a naive check.

done_when

  1. The producer derives the receipt's identity from apr --version and declines (exit 2, no receipt) when the binary names no sha. Fail-closed, not fail-plausible.
  2. checkout_sha is recorded separately and nothing judges it.
  3. The judge implements all four refusals above.
  4. Each of the four is proven by a planted-failure fixture — and the mutation is proven to have landed, not merely attempted. feat(ladder): a receipt names the binary that ran, never the checkout HEAD; the judge requires the release binary; GPU calls join gpu-q (#3771) #3784 carries 125 fixture files for this; reuse its case shapes rather than inventing them.
  5. IDENTITY_PROBE=1 (or equivalent) prints sha / checkout_sha / apr_version and exits, so the identity can be inspected without a full ladder run.

Explicitly not in scope

The Q4_K universe rewrite that #3784 also carries. That is #3766 / #3846 and it should land once, from there.

Refs

#3784 (the PR this is extracted from — do not merge it), #3771, #3766, #3846, #3712, #3724 (the recorded-vs-executed mismatch), #3863 (epic), #3864

Activity

  1. added this to the 0.70.0 milestone on Sep 22, 2026
  2. added
    bugSomething isn't working
    P1High priority
    on Sep 22, 2026
  3. modified the milestones: 0.70.0, 0.71.0 on Sep 23, 2026
  4. noahgift commented on Sep 23, 2026

    @noahgift
    MemberAuthor

    milestone → 0.71.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: model/backend correctness or the certified matrix: Don't Leave Behind (#3994)

  5. added a commit that references this issue on Sep 23, 2026
  6. modified the milestones: 0.71.0, backlog on Sep 25, 2026
  7. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    Parked by APR-EPIC-001: over S1 budget (rank 46/163). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).

  8. added
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritybugSomething isn't workingparkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions