Skip to content

ci: MG shard jobs queue up to 28 min on 5 mq runners (12 shards + x86-main); shard count vs mq runner ratio #5819

Closed as not planned

Description

@noahgift

Owner: aprender-a3 (shard count, two-way). One-way option routed to arbiter-cop for a quorum round.

Finding (measured 2026-10-21 00:12Z, MG run 39516781979, 12 shards): shards run 13-15 min (the #5794 split works) but wait 0-28 min in queue; x86-main waited 27 min.

Five whys

  1. Why is the MG wall ~40 min with 13-15 min shards? Jobs sit queued up to 28 min.
  2. Why queued? A merge_group entry runs on runs-on: [self-hosted, Linux, X64, mq] (ci.yml:444). 12 shards + x86-main = 13 jobs.
  3. Why not on idle runners? Label mismatch, confirmed: online mq runners are intel-clean-room-11, 2, 3, 4, 9 (all busy) = 5. Idle ones lack the label: gx10-pool1/2/3 are ARM64 (clean-room, no X64, no mq); intel-clean-room-16 is clean-room,perf-solo,pr-only, no mq; intel-clean-room-15/13 are review-light and busy. intel-clean-room-14 carries mq but is offline.
  4. Why only 5? mq runners were sized for the 3-shard split (3 jobs + x86-main).
  5. Why did the split change not move the runner count? ci: split workspace-test-shard 3 -> 12 with one count source #5794 changed shards, nothing owns the shards-vs-mq-runners ratio. Waves = ceil(13/5) = 3 at ~14 min each.

Countermeasure (two-way, this ticket): set shards = 5 (draft work/shard-5, base #5794, union test unchanged, not armed until #5812 is measured): 5 shards + x86-main on 5 mq runners is one-two waves. Re-measure with aprender-flow-shards.sh merge_group 3.
One-way options (not done here; arbiter-cop quorum, Noah): bring intel-clean-room-14 (mq, offline) back online as a 6th mq runner; add mq to intel-clean-room-16. No runner registration was changed.

Activity

  1. added this to the 0.73.0 milestone on Oct 21, 2026
  2. modified the milestones: 0.73.0, 0.72.0 on Oct 22, 2026
  3. noahgift commented on Oct 22, 2026

    @noahgift
    MemberAuthor

    Tracked in #5510 (triage 2026-10-22).

  4. noahgift commented on Oct 22, 2026

    @noahgift
    MemberAuthor

    tracked in #5961

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions