Repository navigation
check_silicon_coverage [run] goes red on one GitHub 5xx: 75 live calls per x86-main #5212
Closed as not planned
Description
scripts/check_silicon_coverage.sh [run] fails x86-main on a single transient GitHub API 5xx. It is a GATE on the PR and merge-queue path through guard_tree.
Measured (2026-10-12)
| run | event | silicon [run] | cause |
|---|---|---|---|
| 38241398316 | pull_request | FAIL | listing FAILED: cuda-nightly.yml schedule page 1: HTTP 502 → NO-GO |
| 38222226230 | merge_group (pr-5168) | FAIL | jobs read FAILED (not_measured): run 36511430401: HTTP 502 → NOT_MEASURED rows |
| 38231876079 | merge_group (pr-5167) | PASS | |
| 38234720807 | pull_request | PASS | |
| 38224050802 | pull_request | PASS |
- Not main-wide: push runs on main skip x86-main, so main's green says nothing about this guard. 2 of the 5 sampled x86-main runs were red on it.
- Not a rate limit. The token is the job's GITHUB_TOKEN. Its two runner-listing reads return 403 "Resource not accessible by integration" on every run, and the guard tolerates them (rc 0). The fatal reads are plain 502 Server Errors.
- Volume: one
[run]makes 75 live API calls (the guard's ownapi calls:line, run 2026-10-12 at main 8c12fca): about 4 listings, 4 cross-checks, about 65 per-run jobs reads over the 30-day lookback, and 2 runner reads. Any single 5xx turns the whole guard red, so its failure rate grows with the number of nightly runs in the window.
Fix bounds
- A failed read stays not_measured. There is no third state and no pass on an unread page.
- Allowed: a bounded retry of a 5xx that then gets a real answer (the read is measured; the same shape as
scripts/ci/fetch_p6.sh's "HTTP 500 once, then an answer" row), fewer calls, an ETag, or data moved off the API. - This is a gate change, so it needs a non-author C314, two non-author reviews and green CI.
Refs #5107 (same owner queue; this one goes first)
Activity
noahgift commentedon Oct 12, 2026
Third sample, from #5206's run 38246271098: red on
listing FAILED: silicon-nightly.yml workflow_dispatch unfiltered cross-check: HTTP 502. So it is not tied to one diff. The 5xx can hit any of the 75 reads.A second defect turned up while reading these logs. In all three red runs, both runner reads fail with 403, yet the guard prints
online runners visible (org + this repo): 1. The job's GITHUB_TOKEN has no administration scope, so it cannot list self-hosted runners. On an HTTP error,gh apiwrites the error body to stdout, and the loop appends stdout with>> "$RUNNERS". Aftersort -u, the two identical error bodies read as one runner. The zero-runner probe ("Blind is a NO-GO") therefore never fires on a blind listing. The fix keeps stdout only from a read that succeeded.fix(silicon-coverage): retry a 5xx read (cap 3, failure stays not_mea…
noahgift commentedon Oct 14, 2026
Evidence, 2026-10-14 (no code change; this comment does not close the issue).
The fix is on
main: 6bd89b0 "fix(silicon-coverage): retry a 5xx read (cap 3, failure stays not_measured); a 403 body is not a runner (#5220)".gh_readinscripts/check_silicon_coverage.shretries only an HTTP 5xx, at mostGH_READ_TRIESreads in all. A read still failing after the last try stays rc 1 / not_measured with gh's words. A 4xx is never retried. Self-test rows cover a flaky route (st_flaky_route).Proposed: close as fixed by #5220. Closing is the triage owner's call, not this lane's.
Agent: aprender-a7
noahgift commentedon Oct 14, 2026
Folded into #2876 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch:
fix/5212-silicon-5xx-retry.Agent: aprender-clerk