Skip to content

fix(nightly): ci-pending is asked again the same night, then ends RED (#5053) - #5054

Merged
noahgift merged 3 commits into
mainfrom
fix/nightly-ci-pending-retry
Oct 10, 2026
Merged

noahgift merged 3 commits into
mainfrom
fix/nightly-ci-pending-retry

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Closes #5053

Defect. Nightly run 38016007449 gated main 2 minutes after a push. workspace-test was still running, so the gate decided ci-pending, skipped every build and the run ended green. Nothing retried it, and the nightly went 27h unbuilt.

Change.

  • New scripts/nightly_gate_retry.sh:
    • decide <tries> <wait> -- <gate cmd> asks the gate again while it answers ci-pending, for the same sha.
    • verdict <decision> is the run's exit status: built/reused give 0; anything else gives 1 with the reason.
  • nightly.yml gate step: 7 tries, 20 min apart (2h), with step timeout-minutes: 150.
  • The final verdict step calls verdict, so ci-pending after the last retry is red, never green.
  • A failing gate command (fetch error, rate limit) fails the step immediately, with no retry and no decision.

Proof.

  • bash scripts/nightly_gate_retry.sh --self-test: 12/12, run by the gate's Case table step. It includes the planted case: a ci-pending head on every try ends ci-pending, and its verdict is rc 1.
  • Each of three mutations turns the self-test red:
    • verdict ci-pending -> 0
    • no retry loop
    • dropping || return
  • Reviews: two non-author lanes (claude-sonnet-5-5 and claude-haiku-5-5) both PASS, bound to 34af559, with nothing BLOCKING.

Agent: fleetbins

🤖 Generated with Claude Code

noahgift and others added 2 commits October 10, 2026 13:54
…#5053)

Run 38016007449 gated main 2m after its push. workspace-test was still
running, so the gate decided ci-pending, skipped every build and the run
ended green. Nothing retried, and the nightly went 27h unbuilt.

The gate now asks again for the same sha: 7 tries 20 min apart (2h).
Still pending after the last try, the run's verdict step exits 1 with
the reason. scripts/nightly_gate_retry.sh --self-test carries the planted
ci-pending head (must end red), and both mutations (ci-pending -> 0, no
retry loop) turn it red.

Refs #5053

Agent: fleetbins
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…exit (#5053)

Review lanes A and B both asked for this case in the table. Adding it showed
that decide relied on set -e: called inside a command substitution, where bash
drops errexit, it went on past a failing gate command. It now returns the
failure explicitly. The case is pinned, and removing `|| return` turns it red.

Refs #5053

Agent: fleetbins
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=5054 head=05dd40359493a8f790965776a095e2783be5ca4c verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

C314 round 1 on #5054: the planted ci-pending case ran only inside the
nightly. It now sits beside its siblings in "Fleet hygiene case tables",
so a PR that breaks it is red before merge.

Refs #5053

Agent: fleetbins
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift
noahgift added this pull request to the merge queue Oct 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 10, 2026
@noahgift
noahgift added this pull request to the merge queue Oct 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 10, 2026
@noahgift
noahgift added this pull request to the merge queue Oct 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 10, 2026
@noahgift
noahgift added this pull request to the merge queue Oct 10, 2026
Merged via the queue into main with commit 7fc243b Oct 10, 2026
16 checks passed
@noahgift
noahgift deleted the fix/nightly-ci-pending-retry branch October 10, 2026 16:56
@noahgift noahgift mentioned this pull request Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly: ci-pending skips every build and ends green; retry the same night, else red

1 participant