main is RED to the merge queue: setfit conformance tolerances are stale, every merge_group ejects #3362

Activity

1. noahgift commented on Sep 16, 2026

   

   noahgift

   on Sep 16, 2026

   MemberAuthor

   More actions

   Correction — the root cause stated above is wrong. Two hypotheses measured and both refuted.

   1. "main is RED" — refuted. The test compares sha256(contracts/setfit-encoder-conformance-v1.yaml) against tol::CONTRACT_SHA256. Measured at origin/main 4b00761a3:

   contract sha256 (git blob): 16a6591788a6c693ad3d08845a20267e31d4a86ee663310a943c841d9e7b2b93
   contract sha256 (worktree): 16a6591788a6c693ad3d08845a20267e31d4a86ee663310a943c841d9e7b2b93
   CONTRACT_SHA256 constant:   16a6591788a6c693ad3d08845a20267e31d4a86ee663310a943c841d9e7b2b93
   

   Identical. That assertion passes at main. Also, contracts/setfit-encoder-conformance-v1.yaml has exactly one commit in its entire history (4aff4594d) — the same commit that last wrote tolerances_generated.rs — so "the contract has changed since generation" cannot be literally true.

   Do not run the regenerate command. It would bake whatever hash the runner computes into the constant and permanently hide the real defect.

   2. "a queued PR edits the contract and poisons every group" — refuted. A merge_group branch is main plus the entries ahead of it, so this was worth checking. It is not the cause: none of the eight queued PRs (#3320, #3302, #3278, #3270, #3266, #3265, #3238, #3205, #3354) touches setfit-encoder-conformance-v1.yaml or tolerances_generated.rs.

   What still stands, and is the real finding: this target runs only in merge_group. The step is gated if: steps.tier.outputs.tier == 'full', and PRs run the quick tier — so workspace-test reports SUCCESS on the PR page and the identical tree ejects from the queue. The queue is the first and only place anyone learns. That asymmetry is worth fixing regardless of what this particular failure turns out to be.

   Still open: why sha256_file(&path) disagrees inside the container when the repo-relative file matches. Leading candidates are contract_path() resolving to a different file in the container checkout, or an earlier step in the same job rewriting the YAML before it is hashed. Investigating; I will not propose a fix until the two halves of that comparison are measured inside the failing job rather than inferred from outside it.
2. noahgift commented on Sep 16, 2026

   

   noahgift

   on Sep 16, 2026

   MemberAuthor

   More actions

   Root cause confirmed — and it is #3320, which my previous comment wrongly cleared.

   My earlier check used gh pr view 3320 --json files, which caps at 100 entries. #3320 changes 824 files (823 under contracts/), so the cap hid the relevant one. Measured properly with gh api repos/paiml/aprender/pulls/3320/files --paginate:

   contracts/setfit-encoder-conformance-v1.yaml  +28 -14
   

   And the hashes:

   contract @ #3320 head 67359fead : fdfaeb5ce8fbd53d93e946169d5dca9dde352eeed06aced038ee20338b46f14d
   contract @ main 4b00761a3       : 16a6591788a6c693ad3d08845a20267e31d4a86ee663310a943c841d9e7b2b93
   CONTRACT_SHA256 constant        : 16a6591788a6c693ad3d08845a20267e31d4a86ee663310a943c841d9e7b2b93
   

   #3320 edits the contract and does not regenerate tolerances_generated.rs.

   Why it looked like a main defect. The Integration tests step is gated if: steps.tier.outputs.tier == 'full'. PRs run the quick tier, so this target never runs on the PR and #3320 shows workspace-test: SUCCESS. merge_group runs the full tier, so the group dies. A merge_group branch is main plus the entries ahead of it, which is why pr-3302-0cd4d940… (= main + #3320) failed too, and why PRs with nothing to do with contracts were ejected alongside it.

   Confirmed from the other direction: at origin/main 4b00761a3 the exact fragment command passes locally — conformance_tolerances_agree_with_the_contract ... ok.

   Fix (on #3320's branch, not on main):

   APRENDER_REGEN_TOLERANCES=1 cargo test -p aprender-core --features setfit,conformance-fixtures \
     --test setfit_conformance conformance_tolerances_regenerate -- --ignored
   

   then commit the regenerated crates/aprender-core/tests/setfit_conformance/tolerances_generated.rs. There the contract genuinely changed, so regeneration is correct — unlike on main, where it would have baked a false constant.

   The durable defect, which outlives this PR: a required check that runs only in merge_group means the queue is the first place anyone learns, it costs a full group build to find out, and it ejects innocent PRs batched behind the offender. Either this target should also run on PRs that touch contracts/**, or the quick-tier filter should treat a contract edit as tier-relevant. Recommending the latter — it is the cheap half.
3. noahgift

   added 2 commits that reference this issue on Sep 16, 2026

   fix(setfit): regenerate the conformance tolerances — this PR's contra…

   ...

   8f58592

   chore(pr): re-run the body guard after adding the no-close reason for #…

   ...

   693acc2
4. noahgift

   closed this as completedin #3364on Sep 17, 2026
5. guyernest

   added 2 commits that reference this issue on Sep 29, 2026

   contracts: 3,612 obligations had no id, so nothing could cite them (p…

   ...

   1edcb55

   docs(run): §7 interval 13:20Z — the queue block was one PR's stale to…

   ...

   94df988