Skip to content

perf(ci): X64 runner hosts 3x oversubscribed (intel 98.7/32 cores) while yoga is at 0.33x — rebalance worker counts #5454

Open

Description

@noahgift

Owner: infra-6a (runner hosts); measured by aprender-flow 2026-10-16 16:2xZ (read-only ssh, /proc/loadavg, nproc, ps Runner.Worker).

X64 runner hosts are unevenly loaded: intel/mac-server 32 cores, 9 workers, load1 98.7 (3.1x); framework16 16 cores, 8 workers, load1 44.9 (2.8x); yoga 22 cores, 4 workers, load1 7.3 (0.33x). gx10 (ARM64 only) 20 cores, 2 workers, load1 9.8. The #5417 merge-group shard(3) ran 45+ min on intel-clean-room-2 while yoga had ~15 idle cores.

Proposal (numbers only): intel 9->6 workers, framework16 8->6, yoga 4->6 then 8 after a re-measure. Do not unpark the 7 offline intel runners on mac-server.
Falsifiable: after step 1 intel load1 < 70 and shard(1) wall (30 min) falls over 5 MG groups; if not, thrash was not the cause.
Evidence: /mnt/nvme-raid0/cop-inbox/receipts/q-20x-1016-r11/rebalance-1640.md. Related: #5394, #5429, #5432.

Activity

  1. added this to the 0.73.0 milestone on Oct 16, 2026
  2. noahgift commented on Oct 22, 2026

    @noahgift
    ContributorAuthor

    Tracked in #5510 (triage 2026-10-22).

  3. modified the milestones: 0.73.0, 0.72.0 on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions