You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: nextest list phase hung 82 min on the realizar lib test binary; no list-phase timeout #5294
Merge-group run 38534258555, workspace-test-shard (1): the nextest list phase of cargo nextest run --profile ci --workspace --lib ... --partition hash:1/3 never finished.
The realizar lib test binary (realizar-b4cc444c5fd6fd17 --list --format terse) sat in state S at
0% CPU for ~72 min. nextest has no list-phase timeout, so the job hung ~82 min until it was cancelled
by hand, and the merge queue was blocked ~95 min. The job's timeout-minutes: 110 would have been the first bound.
Measured so far (cause not named yet)
The binary's list output is large: a nearby build (realizar-90214a2505fa1d47, PR-head shard 1 of the same PR)
writes 1,182,878 bytes / 16,543 lines for --list --format terse, ~18x a 64 KiB pipe buffer. A child that
outlives its reader's reads sleeps in a pipe write, which matches state S, 0 CPU.
Same binary, same CI image (sovereign-ci:stable, cargo-nextest 0.9.143), standalone --list --format terse: 23/23 runs exit 0, max 136 ms (20 of them piped through a reader, --cpus 4).
ldd: libssl, libcrypto, libgcc_s, libm, libc only. No CUDA library is linked, so no driver init at list time.
No #[ctor], inventory, linkme or link_section in crates/aprender-serve.
The hung process was gone when this was read (the per-run target dir was cleaned), so no wchan or stack was captured.
So far the binary does not hang on its own. The open candidates are the reader side (nextest stops draining
a >1 MiB stdout) or an environment stall. The next hang should be caught with /proc/<pid>/wchan + /proc/<pid>/stack
for both the binary and cargo-nextest before cancelling.
Fix plan
List-phase timeout. nextest runs --list through the target runner (verified with cargo-nextest 0.9.146:
the runner receives <bin> --list --format terse and --list --format terse --ignored). A runner wrapper that
applies timeout to --list invocations only turns a list hang into a named failure. Probe: a simulated
list hang under a 5 s bound fails nextest with rc 104 in 5 s, naming the binary and exit code 124.
Test invocations pass through unchanged.
Code fix + RED test once the blocking point is identified from a captured stack. Inputs are varied before a cause is named.
keep-open: tracks the list hang cause and the list-phase timeout.
Merged into #5324 (TRIAGE-100, 2026-10-14): same theme, so this master's scope and every checklist row now live on #5324, and this ticket is closed as not planned. Reopen it if the theme splits again.
Activity
ci(nextest): enable wrapper scripts with the [experimental] table (Refs
ci(nextest): forward TERM/INT/HUP from the list-wrapper to the test b…
ci(nextest): enable wrapper scripts with the [experimental] table (Refs
fix(ci): keep the list wrapper alive when its stderr reader is gone (…
merge(ci): main 8c3b19c into the list-wrapper branch (Refs #5294)
noahgift commentedon Oct 14, 2026
Merged into #5324 (TRIAGE-100, 2026-10-14): same theme, so this master's scope and every checklist row now live on #5324, and this ticket is closed as not planned. Reopen it if the theme splits again.
Agent: aprender-clerk