Skip to content

A shard killed by SIGNAL reports as a bare 'test run failed' — 11,757/40,750 not run on yoga-build3, nextest printed the number, nothing reads it; must report Unknown{Signal} with ran/not-run/total #3625

Closed as not planned

Description

@noahgift

A shard killed by signal reports as a bare test run failed — indistinguishable to every consumer from a test failure. nextest prints the number of tests it did not run and nothing reads it.

Measured on #3621's workspace-test-shard (1/3), attempt 2 of run 35530342053, job 106147852695, runner yoga-build3, started 20:55:09Z. The log, in order:

573:  Cancelling due to signal: 4 tests still running
574:  TRY 1 TERM [   0.007s] aprender-serve gpu::tests::tests_06::test_forward_gpu_with_cache_out_of_bounds
576:    (test aborted with signal 15: SIGTERM)
578:  TRY 1 TERM [  12.227s] … test_super_long_context_memory_bound
585:  TRY 1 TERM [  81.902s] aprender-core::spec_checklist_q_qwen_coder q1_qwen25_coder_imports
593:  TRY 1 TERM [  13.500s] … test_mega_long_context_memory_bound
601:     Summary [ 242.884s] 28993/40750 tests run: 28993 passed, 217 skipped
602:  warning: 11757/40750 tests were not run due to signal

Line 573 precedes every TERM. Something on the host sent SIGTERM to the nextest process itself ~4 minutes in — no runner-shutdown message, no step timeout, job conclusion failure not cancelled. The four TERMed tests are nextest killing its own children as part of cancelling; one of them (test_forward_gpu_with_cache_out_of_bounds) had run for 0.007 s. Nothing had failed.

The step then exits 100 with error: test run failed, and the shard is red.

What this is not

It is not fail-fast. nextest's fail-fast wording is Cancelling due to test failure; that string is absent. nextest cancels on SIGTERM regardless of [profile.ci] fail-fast. Setting fail-fast = false (#3587's remaining half, a separate row) would not have run one of the 11,757. An earlier reading on this board attributed them to fail-fast; the log's ordering refutes it.

It is not a test defect. It is the runner-lottery class (#3607): yoga-build3 is a laptop, two of the four in-flight tests were *_memory_bound, and the host killed the process — the systemd-oomd-on-build-pressure shape. Land the same shard on intel and it runs.

What it is

#3587's doctrine — skipped is Unknown, not Fail — applied to the receipt. A shard that was killed has a verdict of Unknown{Signal} over 11,757 tests and a verdict of Pass over 28,993. Today it reports one bit: red. A real test failure on intel and a SIGTERM on a starving laptop produce the same red, and every consumer — the fan-in workspace-test, gate, the merge queue, the person reading the rollup — treats them identically.

The number is already in the log. 11757/40750 tests were not run due to signal is nextest's own line. The reducer discards it.

done_when

  1. The shard's receipt carries ran / not-run / total and the reason for not-run (signal, timeout, test-failure), read from nextest's own summary lines rather than re-derived.
  2. A shard cancelled by signal reports Unknown{Signal}, exit 2 — not Fail, not Pass. The fan-in workspace-test propagates Unknown as Unknown.
  3. Runner name in the receipt. A signal-kill without the runner is a red nobody can attribute; with it, yoga-build3 becomes a named line on infra#813 rather than a mystery.
  4. Falsifier, both arms: send SIGTERM to a running shard in a fixture and require Unknown{Signal} naming the count; a genuine test failure in the same fixture must still report Fail. Both, or the verdict is not discriminating.
  5. The 11,757-class number appears in the fan-in's output, so the next person sees "11,757 unmeasured" instead of "shard 1/3 red".

Relationship to the other rows

Credit: the ordering reading and the correction of the attribution are the train/fleet lane's.

Refs #3587, #3616, #3607, #3621, infra#813.

Activity

  1. added this to the 0.70.0 milestone on Sep 20, 2026
  2. added
    bugSomething isn't working
    P1High priority
    on Sep 20, 2026
  3. modified the milestones: 0.70.0, 0.71.0 on Sep 24, 2026
  4. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    Parked by APR-EPIC-001: over E1 budget (rank 22/93). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).

  5. modified the milestones: 0.71.0, backlog on Sep 25, 2026
  6. added
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritybugSomething isn't workingparkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions