Repository navigation
models: full ladder sits on the scheduled night verdict; the release ships on crux smoke — move the ladder to a nightly ticket-only job, measure each stage, andon at 2x #5147
Description
The models bundle is crux, then publish, then ladder, then publish. No stage has ever completed, and none has a measured duration. Even so, the full ladder sits on the scheduled night verdict:
.github/workflows/models-nightly.yml:123relays without--measure, so the default isladder(scripts/release/models_nightly.sh:29,64). The job then fails unless that run is green.scripts/release/nightly_train.sh:110makesmodelsa verdict lane.
This conflicts with the release policy in contracts/model-capability-ladder-v1.yaml, release_policy: crux-smoke, which says larger rows are nightly and cannot stop a release. The release path itself (rehearse.sh, autopilot.sh, release_readiness.sh, prepare_bump.sh, and the in-run train) already reads crux only.
Done when:
- The scheduled
modelsjob relays--measure cruxand stays a gate on crux. - The ladder relay is a separate scheduled job that only opens tickets. It has no
needs:edge from any verdict lane, no continue-on-error and no allow-failure row. - A RED mutation shows that a red crux turns
modelsred, and a red ladder leavesmodelsgreen. A test fails if any gateneeds:the ladder job. - The chain runs end to end every night. Each stage records its start, end and exit code in the bundle, and has an andon at 2× its first measured time. Until a time is measured, the stage reads not_measured.
Workflow edit: it needs two non-author reviews and green CI.
🤖 Generated with Claude Code
Activity
fix(release): nightly-train fetches the pinned yq for the gate/LAB test
fix(test): the gate/LAB test validates its --workflow path (bashrs SE…
fix(test): the gate/LAB test reads its LAB list with a here-string, n…
noahgift commentedon Oct 11, 2026
Add to the done-when list: no automatic ladder on a release commit (the chain fix).
What happens today: the crux unit lists the ladder in both
OnSuccess=andOnFailure=. Its--run --measure cruxwas started on a release commit throughoverride-commit(on 2026-10-11, frozen main812dbe3eb5). So when that crux ends, it starts a full ladder on the same commit. That ladder then:work/.lockfor up to 6h30, atscripts/release/models_nightly.sh:392-393, so a crux re-run or the 21:00Z timer refuses with rc 3;aprcall on both GPU hosts, atscripts/model_ladder.sh:94-98, so a release-day T-1 crux in that window can decline (exit 75).Since #5158 the ladder casts no vote. The
modelsverdict needs only the crux relay, at.github/workflows/models-nightly.yml:130,146-149. So on a release commit the chained ladder costs the GPU hosts and buys nothing. On 2026-10-11 it was stopped by hand after the crux publish.Done when: a crux run that measured an
override-commit(a release C) does not start the ladder. The scheduled night still runs the ladder after crux. The ladder unit keeps chaining publish. A test plants an override and proves no ladder starts, then removes the override and proves it does.keep-open: the 2x andon and this chain fix are still open after the release.
fix(release): models gates on crux; ladder and six night rows go to L…
noahgift commentedon Oct 14, 2026
Stage measurement for 0.72, read from main at 8c3b19c, the
nightly-evidencebranch and the models-nightly runs from 10-11 to 10-14. Nothing has been changed.Done-when 1–3 already hold on main. models-nightly now runs three jobs:
models-evidence, which relays--measure cruxand gates;models-ladder-tickets, which relays--measure ladderand only opens tickets; andmodels. I did not re-run the RED mutation for item 3.Done-when 4 is open, and the scheduled job has measured nothing on any night. In all 8 scheduled runs from 10-11 to 10-14, both relays print
MODELS-NIGHTLY RELAY not_measured at <sha>: no bundle for <sha> on nightly-evidence. The runs exit green within about 20 s, and the receipt and ticket steps are skipped. The relay asks for its ownGITHUB_SHA(8c73aa9, a881e99, cc71bde, 7f9a15a). The GPU-host producer publishes bundles for a different commit (812dbe3, 70f8c75, 56cf129). The two never meet. This is the producer-side exposure listed in the #5082 survey: this producer is not pinned to the night's C.Per-stage results, from the bundles that do exist:
model_ladder.sh wrote no receipt (rc 2)on both hosts. A releasemodels_t1.shrun for 0.71.2 was running at the same time (the bundle'sconcurrentfile)Durations: not_measured. The bundle (
verdict,models-t1.log,judge.log) records a result, an rc and a commit. It records no start or end time for any stage. The only measurable times are the branch commit times above. They bound the gap between stages, but they give no stage's own duration, so no 2× andon can be set yet. For 0.73, the fix comes in two parts:start=,end=andrc=intoverdict.Until (b) lands, every stage time stays not_measured.
noahgift commentedon Oct 14, 2026
Folded into #2876 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Agent: aprender-clerk