Repository navigation
MG job pickup stalled with idle mq/clean-room runners online (run 39283827214) #5764
Description
Owner: aprender-a3 (runner side); measured by aprender-flow.
MG run 39283827214 (#5745), 4 of 5 mq-labelled jobs created 16:13:59Z were still queued at 16:28Z (14+ min): workspace-test-shard (2), (3), x86-main (label X64,mq) and mg-reuse-self-test (clean-room). Only shard (1) started (16:14:01Z).
Org runner snapshot 16:28Z: X64+mq 6 online (4 busy, 2 IDLE); clean-room (no X64) 4 online, all IDLE; X64+clean-room 13 busy of 17 online (4 offline). Idle runners of the needed labels exist while jobs wait, so the label is not starved by count: pickup is stalled. Intel host load ~125 (java cijob ANDON) is the likely cause (listener/worker cannot start), not measured per host.
Not measured: which host the 2 idle mq runners sit on; GitHub runner-to-job assignment lag. Countermeasure to try (two-way): find the host of the idle mq/clean-room runners, restart their listeners or move them off the saturated host; stop the java cijob. Falsifier: queued->started under 2 min for mq jobs. Related: #5394, #5462, #5746.
Activity
noahgift commentedon Oct 19, 2026
Owner change (aprender-57 ruling): owner is now infra-66, not aprender-a3. Restarting listeners / stopping jobs on runner hosts is an infra act (IC-1). Re-measure of MG pickup due 17:00Z by aprender-flow.
noahgift commentedon Oct 19, 2026
infra-cop diagnosis: queued shards 2/3 want
[self-hosted, Linux, X64, mq]. Idle intel-clean-room-2, -3 and -14 match those labels and sit in group 6 (aprender-reserved, which includes aprender, no workflow restriction). GitHub reports all three online and not busy. So the cause is on the runner side (listener not long-polling, a stale session, or a refusing pre-job hook, under intel load), not labels or scope. Fix: check the listener _diag/journal for those three units, then restart them per unit (restart-clean-room-runnersrefuses while jobs are in flight). That is a runner-host act, so it is operator-only under IC-1. No workflow edit is needed.