Repository navigation
Six concurrent full-tier workspace-test jobs starve every short job for 74 minutes — the merge queue head cannot advance #3258
Description
Six concurrent full-tier workspace-test jobs starve every short job for over an hour
Measured 2026-09-14T09:38Z, aprender only, across all non-completed CI runs:
QUEUED 68–74 min ci / coverage, ci / provenance, ci / security, ci / test, ci / lint (20+ jobs)
RUNNING (10) workspace-test ×6 guard-cargo ×2 guard-tree ×1 ci / lint ×1
gx10-build, gx10-pool1, yoga-build,
intel-clean-room, -2, -5, -9, -11, -12, -13
The oldest queued job was created at 08:23:48Z and had still not been assigned a runner at
09:38Z. In the same runs, workspace-test is executing — so this is not a dependency wait. The
short jobs are queued behind long ones.
Mechanism
workspace-test is the full tier: 82,217 tests across 73 binaries, ~37 min on a clean-room
runner (measured, job 103868350604). There are six running at once because the merge queue
builds max_entries_to_build: 3 groups in parallel and open PRs run their own.
This is the recorded "two concurrent workspace-tests starve the box" failure at scale — per-run
target dirs mean N concurrent runs are N full workspace builds, and 23 clean-room runners cannot
absorb six of them plus everything else.
The cost is not the tests. It is that a 2-minute ci / lint waits 74 minutes behind them, and
gate cannot conclude until every one of those reports — so the merge queue's head cannot
advance. main merged nothing between 08:2xZ and 09:4xZ.
The lever is the reuse path, not more runners
scripts/ci_test_tier.sh already has the right idea:
reuse merge_group only: HEAD^{tree} equals the PR head's tree AND that head's
workspace-test check-run concluded success — the same tree measured twice
is the definition of waste. Any doubt -> full.
It almost never fires. Every merge rebases the groups behind it, so the queue tree differs from
the PR-head tree and the tier falls to full. With a 10-deep queue that is close to never.
So the merge queue re-runs 82,217 tests that the PR head already passed, once per entry, three at a
time. That is the six.
Worth investigating, in rough order of value:
- Make
reusereachable. The current predicate is exact tree equality. A merge group whose
only difference from the tested PR head is other people's merged commits is not obviously
owed a full re-test — but "obviously" is doing a lot of work there and this needs a real
argument, not a loosened comparison. This is the big one. max_entries_to_build: 3 → 2. Fewer concurrent full tiers, less starvation, at the cost of
parallel merge capacity. Cheap to try, easy to revert, and measurable within an hour.- Do not shorten
workspace-testby cutting tests. The dark-surface work (1,424 of 1,469 integration targets run only when a PR touches their own crate — four of tonight's defects trace to this #3239, docs(spec): APR-DARK-TARGETS-001 — what the unrun test surface costs, holds, and when it should run #3249) is
pushing the other way for good reasons.
What this is NOT
Not a runner shortage in the sense of "buy more". At the same instant, 12 of 23 clean-room runners
were busy org-wide (aprender 8, rmedia 3, infra 1) — the fleet had headroom and the jobs still
waited, because the runners that were free could not shorten a 37-minute job already in flight.
And not a dispatch defect: jobs in the same run were executing. GitHub's scheduler is
work-conserving; the work is simply large.
Related: #3251 (fleet throughput), #3239 (the dark surface this must not be "fixed" by shrinking).
Activity
noahgift commentedon Sep 14, 2026
Correction: "six concurrent full tiers" is wrong, and so was my top recommendation
Two errors in the original report. Both mine, both from inferring rather than reading.
1. It is not six full tiers
I derived the tier from step states — "did
Workspace lib testsrun?" — and read six. Theauthoritative source is the
Decide the test tierstep, which printstier=. Reading thatinstead, for the same live runs:
Every readable
pull_requestrun is quick. Only the twomerge_groupruns were full tier.Historical check agrees: over the last completed PR runs I could read, 0 of 7 paid the full
tier.
So the load is 2 full tiers plus several quick ones (~18 min each: 39,615 tests + 10,333
tree-reader), not six 37-minute jobs. The aggregate still exceeds what the fleet drains quickly,
but the shape is different and the headline number was wrong.
2. "Make
reusereachable" was the wrong thing to rank firstI listed it as the big lever because it fires 0 of 7 times. Having thought about what it actually
certifies: 0 of 7 is correct behaviour, not a defect.
A merge queue exists to test
main + this PR, in this order.reusefires only when the queuetree is byte-identical to the tested PR-head tree — i.e. when main did not move. When main did
move, the merged tree is genuinely different and re-testing it is the entire point of having a
queue. Loosening that predicate would not be an optimisation; it would be turning off the thing
the queue is for, and semantic conflicts are exactly what it catches.
I withdraw that item. It should not be attempted.
What still stands, measured
Short jobs starve behind long ones, the queue head cannot advance, and that is real.
The lever list, revised
max_entries_to_build: 3 → 2. Cheap, reversible, measurable within an hour. Fewerconcurrent full tiers at the cost of parallel merge capacity — and unlike the reuse idea it
does not weaken what the queue certifies. Note this was deliberately raised 1→3 on 2026-09-10,
so it is a revisit of a decision, not an oversight.
exactly the commits main gained. Testing
PR's crates ∪ crates touched by those commitsisproportional and preserves what the queue certifies, unlike reuse. This is BSE-17 extended to
merge groups, and it is real work, not a flag.
workspace-test. 1,424 of 1,469 integration targets run only when a PR touches their own crate — four of tonight's defects trace to this #3239/docs(spec): APR-DARK-TARGETS-001 — what the unrun test surface costs, holds, and when it should run #3249 push the other way for good reasons.The method note, because it is the second time tonight
Both errors came from reading an adjacent signal instead of the authoritative one — step states for
the tier here, and
--status queuedcounting runs rather than jobs in the packer an hour ago,which sent me through three false hypotheses. When a number drives a decision, read the field that
states it.
noahgift commentedon Sep 24, 2026
Milestone 0.70.0 → 0.70.1 (gate triage). 0.70.0 now holds only the operator's release gate: "FULL ontology spec and all aprender binary working" + "0.70 to crates.io" (ONT-001, every [[bin]] build+smoke, the publish cascade), plus P0s and operator-ruled rows. This issue isn't needed for any of those. Evidence: evidence/m070-gate-triage-1835.md. If that's wrong, move it back to 0.70.0 and say which gate it serves.
noahgift commentedon Sep 24, 2026
Re-measured 2026-09-24 21:15Z: the mechanism this issue names no longer exists
Scope: every
CIrun created since 09:00Z today (33 runs, 644 started jobs), queue wait =started_at − created_at.workspace-testmax_entries_to_build: 3(ruleset "Merge Queue (main)") is still configured, but the batch-PR regime (≤10 open PRs, one PR per epic) means the queue almost never holds more than one entry. Changing 3→2 would have changed nothing today.gpu-touched,determinism, oneworkspace-test-shard). The wait was not caused by concurrent full tiers.No ci.yml or ruleset change is proposed: neither lever acts on the load that exists now. Recommend closing this issue as superseded (batch-PR regime + #3138 sharding). If the starvation needs chasing, open it as a runner-pool ticket against the arbiter, with the per-repo share of the 16 runners during 18–20Z as its first measurement.
noahgift commentedon Sep 24, 2026
Closing as superseded per the 21:15Z re-measurement above (batch-PR regime + #3138 sharding removed the mechanism). Runner-pool capacity, if it still bites, belongs to the arbiter.