Skip to content

ci: nextest list phase hung 82 min on the realizar lib test binary; no list-phase timeout #5294

Closed as not planned

Description

@noahgift

What happened

Merge-group run 38534258555, workspace-test-shard (1): the nextest list phase of
cargo nextest run --profile ci --workspace --lib ... --partition hash:1/3 never finished.
The realizar lib test binary (realizar-b4cc444c5fd6fd17 --list --format terse) sat in state S at
0% CPU for ~72 min. nextest has no list-phase timeout, so the job hung ~82 min until it was cancelled
by hand, and the merge queue was blocked ~95 min. The job's timeout-minutes: 110 would have been the first bound.

Measured so far (cause not named yet)

  • The binary's list output is large: a nearby build (realizar-90214a2505fa1d47, PR-head shard 1 of the same PR)
    writes 1,182,878 bytes / 16,543 lines for --list --format terse, ~18x a 64 KiB pipe buffer. A child that
    outlives its reader's reads sleeps in a pipe write, which matches state S, 0 CPU.
  • Same binary, same CI image (sovereign-ci:stable, cargo-nextest 0.9.143), standalone --list --format terse:
    23/23 runs exit 0, max 136 ms (20 of them piped through a reader, --cpus 4).
  • ldd: libssl, libcrypto, libgcc_s, libm, libc only. No CUDA library is linked, so no driver init at list time.
  • No #[ctor], inventory, linkme or link_section in crates/aprender-serve.
  • The hung process was gone when this was read (the per-run target dir was cleaned), so no wchan or stack was captured.

So far the binary does not hang on its own. The open candidates are the reader side (nextest stops draining
a >1 MiB stdout) or an environment stall. The next hang should be caught with /proc/<pid>/wchan + /proc/<pid>/stack
for both the binary and cargo-nextest before cancelling.

Fix plan

  1. List-phase timeout. nextest runs --list through the target runner (verified with cargo-nextest 0.9.146:
    the runner receives <bin> --list --format terse and --list --format terse --ignored). A runner wrapper that
    applies timeout to --list invocations only turns a list hang into a named failure. Probe: a simulated
    list hang under a 5 s bound fails nextest with rc 104 in 5 s, naming the binary and exit code 124.
    Test invocations pass through unchanged.
  2. Code fix + RED test once the blocking point is identified from a captured stack. Inputs are varied before a cause is named.

keep-open: tracks the list hang cause and the list-phase timeout.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions