Repository navigation
A shard killed by SIGNAL reports as a bare 'test run failed' — 11,757/40,750 not run on yoga-build3, nextest printed the number, nothing reads it; must report Unknown{Signal} with ran/not-run/total #3625
Description
A shard killed by signal reports as a bare test run failed — indistinguishable to every consumer from a test failure. nextest prints the number of tests it did not run and nothing reads it.
Measured on #3621's workspace-test-shard (1/3), attempt 2 of run 35530342053, job 106147852695, runner yoga-build3, started 20:55:09Z. The log, in order:
573: Cancelling due to signal: 4 tests still running
574: TRY 1 TERM [ 0.007s] aprender-serve gpu::tests::tests_06::test_forward_gpu_with_cache_out_of_bounds
576: (test aborted with signal 15: SIGTERM)
578: TRY 1 TERM [ 12.227s] … test_super_long_context_memory_bound
585: TRY 1 TERM [ 81.902s] aprender-core::spec_checklist_q_qwen_coder q1_qwen25_coder_imports
593: TRY 1 TERM [ 13.500s] … test_mega_long_context_memory_bound
601: Summary [ 242.884s] 28993/40750 tests run: 28993 passed, 217 skipped
602: warning: 11757/40750 tests were not run due to signal
Line 573 precedes every TERM. Something on the host sent SIGTERM to the nextest process itself ~4 minutes in — no runner-shutdown message, no step timeout, job conclusion failure not cancelled. The four TERMed tests are nextest killing its own children as part of cancelling; one of them (test_forward_gpu_with_cache_out_of_bounds) had run for 0.007 s. Nothing had failed.
The step then exits 100 with error: test run failed, and the shard is red.
What this is not
It is not fail-fast. nextest's fail-fast wording is Cancelling due to test failure; that string is absent. nextest cancels on SIGTERM regardless of [profile.ci] fail-fast. Setting fail-fast = false (#3587's remaining half, a separate row) would not have run one of the 11,757. An earlier reading on this board attributed them to fail-fast; the log's ordering refutes it.
It is not a test defect. It is the runner-lottery class (#3607): yoga-build3 is a laptop, two of the four in-flight tests were *_memory_bound, and the host killed the process — the systemd-oomd-on-build-pressure shape. Land the same shard on intel and it runs.
What it is
#3587's doctrine — skipped is Unknown, not Fail — applied to the receipt. A shard that was killed has a verdict of Unknown{Signal} over 11,757 tests and a verdict of Pass over 28,993. Today it reports one bit: red. A real test failure on intel and a SIGTERM on a starving laptop produce the same red, and every consumer — the fan-in workspace-test, gate, the merge queue, the person reading the rollup — treats them identically.
The number is already in the log. 11757/40750 tests were not run due to signal is nextest's own line. The reducer discards it.
done_when
- The shard's receipt carries
ran / not-run / totaland the reason for not-run (signal,timeout,test-failure), read from nextest's own summary lines rather than re-derived. - A shard cancelled by signal reports
Unknown{Signal}, exit 2 — notFail, notPass. The fan-inworkspace-testpropagates Unknown as Unknown. - Runner name in the receipt. A signal-kill without the runner is a red nobody can attribute; with it,
yoga-build3becomes a named line on infra#813 rather than a mystery. - Falsifier, both arms: send SIGTERM to a running shard in a fixture and require
Unknown{Signal}naming the count; a genuine test failure in the same fixture must still reportFail. Both, or the verdict is not discriminating. - The 11,757-class number appears in the fan-in's output, so the next person sees "11,757 unmeasured" instead of "shard 1/3 red".
Relationship to the other rows
- nextest fail-fast discards verdicts: one failure hid 15 unmeasured commands and the shard read as '1 failure', not '1 failure + 15 Unknown' #3587 / PMAT-3587: fail-fast discards verdicts — the default flips, and skipped becomes a number #3616: the explicit-commands runner now defaults to
--no-fail-fastwithran/skipped/total. This row is the same shape for the nextest shards. [profile.ci] fail-fast = false(separate, being built): prevents a test failure from cancelling the rest. Its number is "7 rounds for 4 tests", not this one.- #3540's red set varies across heads that differ by an 11-line quorum JSON: determinism (X64) and a PATH-resolution guard-cargo step — a runner lottery, not a flake #3607: the runner side. This is the receipt side.
Credit: the ordering reading and the correction of the attribution are the train/fleet lane's.
Activity
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over E1 budget (rank 22/93). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).