You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
kaizen: merge_group jobs wait behind PR CI for clean-room runners — reserve merge-queue capacity (queue-wait p90 33 min) #5394
Last 8 merge_group runs on main (10-14 23:17Z → 10-15 07:31Z) plus the live queue:
Metric
p50
p90
queue-wait, MQ job created → started (all x86-main + shard jobs)
0.7 min
33 min
x86-main run
24 min
40 min
slowest shard run
37 min
62 min
MQ run, created → last job done
51 min
80 min
PR add-to-queue → merged (21 PRs, 24 h)
67 min
94 min
PR open → merged (21 PRs, 24 h)
214 min
424 min
Loss on the critical path, summed over those 8 merges: runner queue-wait 99 min, shard over-run past x86-main 92 min, actions/checkout ≥3.5 min on 16 jobs (55 job-min).
Live, same cause: #5379's shards sat QUEUED 09:50 → 10:41Z (51 min). #5378's shards were QUEUED from 10:41Z and still waiting at 11:02Z, with 11 of 11 aprender-usable [Linux,X64,clean-room] runners busy. No merge into main 08:52Z → 11:0xZ (MERGE-DROUGHT andon D45/D90/QW).
Five whys
Why no merge for 2 h+? MQ jobs waited QUEUED for a runner (51 min and more).
Why? Every runner that can take [self-hosted, Linux, X64, clean-room] for aprender was busy.
Why? PR-push CI and merge_group CI ask for the same label set (ci.ymlruns-on has no event-dependent form), so queue batches wait behind PR runs.
Why does that cost merges? GitHub has no job priority. Only a dedicated runner group or label reserves capacity, and none exists for merge_group.
Why none? The X64 clean-room pool was sized for throughput and never split by event. Merges are the KPI (≥5 PRs/h), and they get no reserved capacity.
Countermeasure (one)
Reserve merge-queue capacity: ≥2 X64 clean-room runners that only merge_group jobs can take (a dedicated label such as merge-queue, already known to actionlint, on runners that do not carry the plain PR label set). merge_group jobs (x86-main, workspace-test-shard) select it, e.g. via an event-dependent runs-on. The runner side is infra's; the ci.yml side needs the usual CI-workflow sign-off.
Done when (re-measure)
Over the next 8 merge_group runs after it lands: queue-wait p90 ≤ 5 min (from 33), and the MERGE-DROUGHT QW andon (MQ job QUEUED >15 min) silent for 24 h. aprender-flow re-measures hourly and posts the before/after in the kaizen | inbox line.
Owner: aprender-a3 (ci.yml side; runner side coordinated with the infra cop). Related: #5324 (CI pipeline umbrella), #3676 (CI diet).
Flake evidence (aprender-flow, 17:4xZ): MG run 39136421562 (#5504) failed because guard-cargo step 4 hit its 3556s timeout under intel CPU starvation (load ~120). The log shows the contracts gate with 8 of 8 steps RAN and 0 FAILED just before '##[error]step exceeded its timeout (3556s)'. This is a timeout, not a defect. #5504 re-armed.
Activity
ci: sign-off r1 fixes for the queue-owes-FULL change
ci: cover the determinism wait too; checker reads every wait and tear…
ci: pin the wait and teardown step counts in the cancel checker
ci: cancel checker parses steps as records, exact matches, exact counts
ci: PR-event X64 jobs run on the pr-only pool; merge_group stays on c…
ci: x86-main wait step is cancel-aware; teardown keeps always() (paim…
ci: the queue owes the FULL tier; no reuse of a quick PR green (BSE-1…
ci: mg-reuse runs on push only; x86-main asks for a runner at t=0 on …
noahgift commentedon Oct 18, 2026
Flake evidence (aprender-flow, 17:4xZ): MG run 39136421562 (#5504) failed because guard-cargo step 4 hit its 3556s timeout under intel CPU starvation (load ~120). The log shows the contracts gate with 8 of 8 steps RAN and 0 FAILED just before '##[error]step exceeded its timeout (3556s)'. This is a timeout, not a defect. #5504 re-armed.
ci(m5): build pv inside the CI image so the guard's cargo build is ch…
ci(m5): match the host build env in the container, pin the image dige…
ci(m5): fail above the 12-unit baseline, one mount when CARGO_HOME is…
fix(ci): m5 container PATH puts the toolchain bin first so the rustup…
chore(roadmap): PMAT-5737 T-rollup ticket (Refs #5737, #5394)
ci: drop the continue-on-error budget step from the PR path; make che…
ci: merge_group job timeout 75 min on x86-main and workspace-test-sha…
ci(m5): build pv in the CI image, guard on host (A2 option c) (paiml#…
noahgift commentedon Oct 22, 2026
Tracked in #5510 (triage 2026-10-22).
noahgift commentedon Oct 22, 2026
tracked in #5959