Skip to content

Stop settle() spinning while a task is pending its first run - #90

Merged
mansbernhardt merged 1 commit into
mainfrom
claude/settle-pending-start-spin
Sep 29, 2026
Merged

mansbernhardt merged 1 commit into
mainfrom
claude/settle-pending-start-spin

Conversation

@mansbernhardt

Copy link
Copy Markdown
Collaborator

Problem

Reported from parallel-phoenix-apple CI (#2444, m2pro, run 36601393927). PathInteractionTests.overviewTapped() sat 23+ minutes in settle() at about 1000% CPU. 1157 of 1164 samples were in _driveToStableFixpoint → hasPendingStartWork → AnyContext.hasPendingStartTask → allChildren, and the 30 s trait cap never fired. The whole Mac was starved (load 265).

_driveToStableFixpoint holds its quiet window open while any registered task hasn't run yet. When that was the only pending work, with the executor, background queue and main-observation queue all idle, every wait in the loop resolved synchronously. A continuation resumed inside its own body doesn't suspend the task, so the loop re-checked in a tight spin without ever giving up its thread.

A task spawned without the harness executor (e.g. from a main-queue callback) starts on the cooperative pool. Once enough tests spin at the same time they hold every pool thread (≈ 10 on that machine, matching the ~1000% CPU). The pending tasks then never start, and no settle can finish. The _withTestTimeout watchdogs are task-group children on the same pool, so they starve too and never cancel the test. That's why the trait cap never fired: the second reported issue is a symptom of the first.

Fix

When a task that hasn't started is the only thing keeping the model from idle, the loop sleeps a 1 ms poll interval on the timer (_gtsSleep, a real suspension that frees the thread) before re-checking. It's a poll cadence, not a verdict; no wall-clock decision changes. Polling rather than waiting for an event is deliberate: a task spawned off the harness executor may not see this test's ModelAccess, so its start isn't reliably observable.

Validation

  • SettlePendingStartSpinTests reproduces the deadlock deterministically on one thread. settle() runs on a single-thread task executor, and a registered task that hasn't started needs that thread to get going. Before the fix it spun, confirmed by instrumenting the loop (pending=true with all queues idle), until the 30 s trait cap cancelled it. After the fix it passes in about 0.08 s.
  • scripts/test (parallel) and --no-parallel are both green: 747 + 124 + 7 + 29 tests. Wall time fell compared with earlier today on the same machine: parallel 27.6 → 15.8 s, serial 191 → 78 s.
  • Stress A/B (6 interleaved rounds, main vs. fix, load average 47–221 from other work on the machine): main failed 1 of 6 (fadeParksOnFrozenClockAndStepsWithIt, trait timeout); the fix failed 0 of 6 and was faster in 4 of 6 rounds. An earlier --loop 10 on the fix had 3 early iterations fail under load ~120–190. The failing tests, fadeParksOnFrozenClockAndStepsWithIt among them, also time out on main under that load, so I don't count them against the fix.

🤖 Generated with Claude Code

`_driveToStableFixpoint` holds its quiet window open while a registered
task has not started (`hasPendingStartWork`). When that was the only
pending work, every wait in the loop resolved synchronously on idle
queues, so the loop re-checked in a tight spin without giving up its
thread. A task spawned without the harness executor starts on the
cooperative pool; once enough tests spun they held every pool thread,
the pending tasks never started, and the pool-hosted trait-cap
watchdogs starved as well. Downstream this pinned ~10 cores for 23+
minutes in a single settle().

Sleep a 1 ms poll interval on the timer in that state instead, freeing
the thread.

SettlePendingStartSpinTests reproduces the deadlock on a single-thread
task executor: it hit its 30 s trait cap before the fix and passes in
under 0.1 s after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mansbernhardt
mansbernhardt merged commit 87d64c5 into main Sep 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant