Repository navigation
FLAKE-0: check_guard_steps_run_all.sh self-test row "a timed-out step still runs its EXIT trap" races its 1 s step timeout on a loaded runner #4759
Description
Seen in #4755, run 37222539899, x86-main, guard-tree#m2 (752.8 s). One row of
scripts/check_guard_steps_run_all.sh --self-test failed:
FAIL a timed-out step still runs its EXIT trap (restores a mutant)
Every other row of that table was ok, including "CI: a timed-out step is TIMEOUT, not a crash" and
"CI: a hung step TIMES OUT and goes red".
Likely mechanism (the fix must confirm it): the fixture step is
trap "touch \"$RUNNER_TEMP/restored\"" EXIT; sleep 30, run with --step-timeout 1. On a loaded runner
the step's shell can be killed before it reaches the trap line, so no trap was ever armed. The row
cannot tell "armed and then skipped", which is the defect it exists to catch, from "never armed", where
the 1 s budget ran out first.
Rule: zero tolerance for flaky tests. Disable at once, then fix or remove.
- Quarantine (today). The row still runs and prints what it read, but it is labelled
QUARANTINED FLAKE-0 #<this ticket>. The table's summary counts it as not measured, never as ok,
and it cannot set the table's rc. Every other row runs unchanged. - Fix. The fixture writes an "armed" marker right after
trap, and the row judges the EXIT trap only
once the step was armed. If the step was killed before arming, the row retries with a longer step
timeout: 1 s, then 4 s, then 16 s. A step still not armed at 16 s is RED. Then the quarantine
comes off.
Proof: the row goes RED against a runner mutant that skips the EXIT trap on timeout (SIGKILL first),
and stays green under artificial load on the host running the self-test.
Activity
chore(pr-review): receipt for #4762 at 9c59de3 (FLAKE-0 #4759)
fix(guard): count a quarantined self-test row apart from not-measured
noahgift commentedon Oct 6, 2026
Seen again after #4772's ready handshake. Reopening because the handshake did not fix this row.
main89ef1b7 for this row.FAIL a timed-out step still runs its EXIT trap (restores a mutant) step-timeout 1s. Thearmedmarker was seen, so this is not the old wall-clock race;restoredwas never written.ok ... step-timeout 1s).#4830 (quarantine, open) is the stop-gap. The fix must keep the assertion: make the trap's write unkillable or gate the second TERM, then re-mutate (trap removed -> RED).
noahgift commentedon Oct 6, 2026
Third hit after the handshake: run 37369278101, job 112048440724 (PR #4676, the same head as before). The row failed the same way. The runner was much less loaded for this rerun (1-minute load about 58, versus about 96 before), so load is not enough to explain it. Because of this, reruns are no longer a workaround: #4676's gate stays red until the row is fixed or quarantined.
noahgift commentedon Oct 14, 2026
Closing on evidence: fixed by PR #4830, merged to main at 07188d3 (#4830). No open PR for this issue remains. Triage closer lane, brief item 7a.