Skip to content

kaizen: merge_group jobs wait behind PR CI for clean-room runners — reserve merge-queue capacity (queue-wait p90 33 min) #5394

Closed as not planned

Description

@noahgift

Kaizen countermeasure #1 (build-process timing, hourly loop; measured 2026-10-15 ~11:10Z by aprender-flow, GraphQL reads only).

Measurement

Last 8 merge_group runs on main (10-14 23:17Z → 10-15 07:31Z) plus the live queue:

Metric p50 p90
queue-wait, MQ job created → started (all x86-main + shard jobs) 0.7 min 33 min
x86-main run 24 min 40 min
slowest shard run 37 min 62 min
MQ run, created → last job done 51 min 80 min
PR add-to-queue → merged (21 PRs, 24 h) 67 min 94 min
PR open → merged (21 PRs, 24 h) 214 min 424 min

Loss on the critical path, summed over those 8 merges: runner queue-wait 99 min, shard over-run past x86-main 92 min, actions/checkout ≥3.5 min on 16 jobs (55 job-min).
Live, same cause: #5379's shards sat QUEUED 09:50 → 10:41Z (51 min). #5378's shards were QUEUED from 10:41Z and still waiting at 11:02Z, with 11 of 11 aprender-usable [Linux,X64,clean-room] runners busy. No merge into main 08:52Z → 11:0xZ (MERGE-DROUGHT andon D45/D90/QW).

Five whys

  1. Why no merge for 2 h+? MQ jobs waited QUEUED for a runner (51 min and more).
  2. Why? Every runner that can take [self-hosted, Linux, X64, clean-room] for aprender was busy.
  3. Why? PR-push CI and merge_group CI ask for the same label set (ci.yml runs-on has no event-dependent form), so queue batches wait behind PR runs.
  4. Why does that cost merges? GitHub has no job priority. Only a dedicated runner group or label reserves capacity, and none exists for merge_group.
  5. Why none? The X64 clean-room pool was sized for throughput and never split by event. Merges are the KPI (≥5 PRs/h), and they get no reserved capacity.

Countermeasure (one)

Reserve merge-queue capacity: ≥2 X64 clean-room runners that only merge_group jobs can take (a dedicated label such as merge-queue, already known to actionlint, on runners that do not carry the plain PR label set). merge_group jobs (x86-main, workspace-test-shard) select it, e.g. via an event-dependent runs-on. The runner side is infra's; the ci.yml side needs the usual CI-workflow sign-off.

Done when (re-measure)

Over the next 8 merge_group runs after it lands: queue-wait p90 ≤ 5 min (from 33), and the MERGE-DROUGHT QW andon (MQ job QUEUED >15 min) silent for 24 h. aprender-flow re-measures hourly and posts the before/after in the kaizen | inbox line.

Owner: aprender-a3 (ci.yml side; runner side coordinated with the infra cop). Related: #5324 (CI pipeline umbrella), #3676 (CI diet).

Activity

  1. added this to the backlog milestone on Oct 15, 2026
  2. noahgift commented on Oct 18, 2026

    @noahgift
    MemberAuthor

    Flake evidence (aprender-flow, 17:4xZ): MG run 39136421562 (#5504) failed because guard-cargo step 4 hit its 3556s timeout under intel CPU starvation (load ~120). The log shows the contracts gate with 8 of 8 steps RAN and 0 FAILED just before '##[error]step exceeded its timeout (3556s)'. This is a timeout, not a defect. #5504 re-armed.

  3. modified the milestones: backlog, 0.72.0 on Oct 22, 2026
  4. noahgift commented on Oct 22, 2026

    @noahgift
    MemberAuthor

    Tracked in #5510 (triage 2026-10-22).

  5. noahgift commented on Oct 22, 2026

    @noahgift
    MemberAuthor

    tracked in #5959

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low priority

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions