You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: MG shard jobs queue up to 28 min on 5 mq runners (12 shards + x86-main); shard count vs mq runner ratio #5819
Owner: aprender-a3 (shard count, two-way). One-way option routed to arbiter-cop for a quorum round.
Finding (measured 2026-10-21 00:12Z, MG run 39516781979, 12 shards): shards run 13-15 min (the #5794 split works) but wait 0-28 min in queue; x86-main waited 27 min.
Five whys
Why is the MG wall ~40 min with 13-15 min shards? Jobs sit queued up to 28 min.
Why queued? A merge_group entry runs on runs-on: [self-hosted, Linux, X64, mq] (ci.yml:444). 12 shards + x86-main = 13 jobs.
Why not on idle runners? Label mismatch, confirmed: online mq runners are intel-clean-room-11, 2, 3, 4, 9 (all busy) = 5. Idle ones lack the label: gx10-pool1/2/3 are ARM64 (clean-room, no X64, no mq); intel-clean-room-16 is clean-room,perf-solo,pr-only, no mq; intel-clean-room-15/13 are review-light and busy. intel-clean-room-14 carries mq but is offline.
Why only 5? mq runners were sized for the 3-shard split (3 jobs + x86-main).
Countermeasure (two-way, this ticket): set shards = 5 (draft work/shard-5, base #5794, union test unchanged, not armed until #5812 is measured): 5 shards + x86-main on 5 mq runners is one-two waves. Re-measure with aprender-flow-shards.sh merge_group 3. One-way options (not done here; arbiter-cop quorum, Noah): bring intel-clean-room-14 (mq, offline) back online as a 6th mq runner; add mq to intel-clean-room-16. No runner registration was changed.
Activity
merge work/shard-4-mg-guards; ci.yml matrix and aggregator keyed by e…
noahgift commentedon Oct 22, 2026
Tracked in #5510 (triage 2026-10-22).
noahgift commentedon Oct 22, 2026
tracked in #5961