Skip to content

Release CI: one job, one reserved runner — a release PR must not fan out into 20 queued jobs #4433

Closed as not planned
2 of 2 issues completed
Closed as not planned

Description

@noahgift

OPERATOR, verbatim (2026-09-25, cop session): "the release has 11 jobs??? wtf, merge into one"

Measured on 0.69.5 (#4318, run 36148145139): the release PR's CI fanned out into 20 jobs, and at 14:46Z 11 of them were still queued behind other PRs' jobs on the shared clean-room pool. A release waits on the slowest queue position among 20.

Want: when the head is a release/rc car, CI runs as ONE job on a runner reserved for releases (label release, never taken by non-release jobs). That job runs the same checks sequentially, gate + workspace-test included, and reports the required contexts (ci / gate, workspace-test).

done_when:

  • A release-car PR shows 1 job, which starts in ≤60 s while the pool is saturated.
  • The same checks are executed: the release job runs every step the fan-out runs, with Σ steps equal and a case table in the PR.
  • A non-release PR is unchanged.
  • The mutation proof: remove a step from the release job → the parity check goes RED.

kind: code

Folded sub-tickets (TRIAGE-100, 2026-10-14)

These open sub-tickets were folded here as checklist rows and closed as not planned so the open count stays at or under 100. Reopen one when its work resumes; an open PR for it reopens it.

Activity

  1. added this to the 0.70.0 milestone on Sep 25, 2026
  2. noahgift commented on Sep 25, 2026

    @noahgift
    ContributorAuthor

    Re-scope (cop, relaying the operator's ruling of 2026-09-25 16:58 Madrid, verbatim):

    1. CI = 5 fat jobs (x86 main, gx10, yoga, mac-refusal, determinism). Each runs all parts to completion on the biggest runner in its class, nextest full-core, sccache warm. Same test set as today (Σ executed unchanged).
    2. No RC CI run. rc.N = tag on a queue-green main sha, cut within 5 min of green. Hardware jobs + clean-room + accuracy receipts run in parallel on the rc tag and gate PROMOTION, not the rc. Promotion failure → rc.N+1.
    3. Release CI: one job, one reserved runner — a release PR must not fan out into 20 queued jobs #4433 re-scoped to 1+3. Collapse 20 → 5 jobs. The train-active priority hook stays as a floor.

    done_when now covers items 1 and 3: 20 → 5 jobs, with Σ executed equal to today's (case table); the rc tag is cut on a queue-green main sha within 5 min; promotion is gated by receipts on the tag. #4318 finishes as-is.

  3. noahgift commented on Sep 25, 2026

    @noahgift
    ContributorAuthor

    Item 3 of the operator's 'make today's rulings permanent' (done = merged + enforced): the #4433 ci.yml PR must also ship a Σ-executed invariant check. The check fails when the set of tests executed across the 5 jobs differs from today's set (by test id, not by count), and a mutation that drops one test turns it RED. [U] until merged.

  4. added 4 commits that reference this issue on Sep 25, 2026
  5. 60 remaining items

  6. modified the milestones: 0.71.0, 0.72.0 on Oct 14, 2026
  7. modified the milestones: 0.72.0, 0.73.0 on Oct 14, 2026
  8. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #5306 as a checklist row (TRIAGE-100, 2026-10-14). Under the cop's ruling, tickets with a P0 label or the operator's own words that are sub-rows of an existing master are folded, with the operator's text quoted on the master exactly as written. Priority and owner carry over on the row.

    Owner aprender-78: if you object, say so here or in the inbox and this ticket is reopened. Work on it continues as before; a PR for it reopens it.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions