Release churn five-whys v0.70.0–v0.71.1 + retire 'freeze main to tag' (cut at fixed sha in own worktree) #5185

Activity

1. noahgift

   added this to the 0.71.1 milestone on Oct 12, 2026
2. noahgift commented on Oct 12, 2026

   

   noahgift

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Five whys: why a frozen release branch takes days (measured; a3)

   release	freeze→tag (h)	reds on the release path	never-green reds
   v0.70.0	129.57	4	2
   v0.70.1	21.24	5	3
   v0.70.2	8.79	7	2
   v0.70.3	17.04	4	2
   v0.71.0	7.74	10	1

   Main was frozen about 86 h of 354 h (24%). The best freeze→tag so far, 7.74 h, is 10× the 45-minute rule.

   1. Why days? Every red inside the window costs a full loop: cut a fix PR, merge it through the queue, re-run. There were 30 reds across 5 releases.
   2. Why are there reds in the window? The window runs long computations (model smoke 58.5 min, publish dry-run 14.5 min) for the first time on the cut sha.
   3. Why do they go red there? 10 of the 30 reds came from steps that had never been green. A step that has never passed can only stop a window.
   4. Why is that allowed? The release path had no rule that a step must have a last green and a measured time.
   5. Root cause: the release window does computation and discovery instead of checking receipts.

   0.71.1, measured: models 3625 s (GO). A missing release-notes input then stopped the train for 1073 s. That is 78 min before the tag, already over budget.

   The fix (owner a3, effective 0.71.2):

   • The nightly job runs the models smoke and the publish dry-run on the candidate sha and writes a receipt keyed by sha.
   • The window only checks that a receipt exists for the exact cut sha. If not, it STOPs; it never re-runs.
   • An inputs check runs before the cut: release notes and receipts must exist, or the window doesn't open.
   • The window prints elapsed time per step and an andon line past 45 min.
   • Never-green steps go to LAB.

   The budget comes to tag 0.2 + clean-room 25.7 + assets check 0.1 ≈ 26 min, which leaves about 19 min for publish through close. This is a gate change, so it needs a non-author C314 sign-off and 2 reviews.

   🤖 Generated with Claude Code
3. noahgift

   added a commit that references this issue on Oct 12, 2026

   feat(release): the 45-min window reads the nightly receipt, makes its…

   ...

   6b58508
4. noahgift

   added a parent issue on Oct 12, 2026

   • EPIC E1 0.71: Fast Train #3998