Repository navigation
MG x86-main "Sections wait" red while workspace-test shards still run (reported, unreproduced) #5788
Description
Owner: aprender-flow (investigation + fix PR via a3). Reported by aprender-57 from aprender-w5382: x86-main's "Sections (wait for every one to finish)" step is said to time out at ~15 min while workspace-test shards still run (PR #5724, 10:42-10:58Z), then the shards pass and the job is red for nothing; MG 39413691864 (#5781) may hit it.
Measured (jobs API, 2026-10-20, merge_group runs; times UTC)
| run | x86-main "Sections wait" | shard 1 / 2 / 3 job | x86-main job |
|---|---|---|---|
| 39395486487 (#5727) | 08:55:37-09:18:00 = 22.4 min | 46.7 / 38.7 / 20.5 min | 34.5 min |
| 39377428382 (#5777) | 06:01:47-06:17:04 = 15.3 min | 20.7 / 16.7 / 14.8 min | 22.4 min |
| 39374362427 (#5776) | 05:28:44-05:46:51 = 18.1 min | 22.3 / 32.6 / 19.7 min | 26.6 min |
| 39413691864 (#5781, live at 11:43Z) | started 11:29:26, still running (14+ min) | shards still running at 27+ min | running |
| Successful MG runs already show the wait step at 15-22 min, so a 15 min limit cannot be a plain step timeout on a green run. Shard durations vary 15-47 min. |
NOT verified (honest gaps)
- I could not find the failing 10:42-10:58Z run: the CI runs of test: fold branch-only lines #5678 #5713 #5540 #5539 #5594 #5595 #5596 #5674 #5686 #5724's branch (batch/test-fold-branch-only-0345) were cancelled at 08:33Z/11:28Z, and none matches that window. The job/step name that went red, and the exact error text, are needed from w5382.
fat_driver.py wait(cmd_wait, scripts/ci/fat_driver.py:1379) has no timeout of its own: it polls the driver pid every 5 s. The step in ci.yml has notimeout-minutes; the job has 75 (merge_group) / 240. x86-main's wait also does not wait for the workspace-test-shard jobs (separate jobs). So the mechanism of a "15 min timeout while shards run" is unexplained; candidates: a section's scaled timeout-minutes inside the driver, the per-step budget (ci/step-budgets.tsv: 2700 s for this step), or the gate job reading the shards.
Next (read-only first)
- Ask w5382 for the run id + job + red step (one line). 2. Read that job's log for
step exceeded its timeout/::error::. 3. Only then propose the edit. A fix is a CI edit: solo-per-PR, quorum receipt, non-author round bound to the final commit, andon to Noah (notice). No edit made yet.
Activity