From 0ae57c033f518bc1e0cea034ac121fa2c9210393 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 4 Oct 2026 10:59:32 +0200 Subject: [PATCH 1/2] =?UTF-8?q?fix(ci):=20CUDA=20Nightly=20G0=20step=20abo?= =?UTF-8?q?rted=20on=20its=20own=20RED=20control=20=E2=80=94=20bash=20-e?= =?UTF-8?q?=20killed=20it=20before=20r=3D$=3F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The G0 step (aprender#4250) runs under GitHub's default `bash -e {0}`. Its `set -uo pipefail` does not turn -e off, so the per-token RED control's expected exit 101 aborted the step at `; r=$?`, before the assertions ran. Run 37161229337: the GREEN arm passed, then the step failed 101 with no ::error:: line. Turn -e off with `set +e`, as the perf041 step in this workflow already does. Every failure path is guarded explicitly (`|| exit 1` and the GREEN/RED checks), so the step accepts exactly what it was written to accept. Repro: bash -e -c 'set -uo pipefail; (exit 101); r=$?; echo reached' prints nothing, rc=101. Refs #4681 Agent: aprender-wmainred Co-Authored-By: Claude Opus 5.5 --- .github/workflows/cuda-nightly.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/cuda-nightly.yml b/.github/workflows/cuda-nightly.yml index 53dafd909d..c33e72b56d 100644 --- a/.github/workflows/cuda-nightly.yml +++ b/.github/workflows/cuda-nightly.yml @@ -707,6 +707,11 @@ jobs: if: ${{ !cancelled() && steps.decide.outputs.proceed == 'true' }} run: | set -uo pipefail + # -e OFF, as in the perf041 step: the default shell is `bash -e {0}`, so the + # RED control's expected exit 101 aborted the step at `; r=$?` before its + # assertions ran (run 37161229337: GREEN passed, step failed 101, no ::error::). + # Every failure path below is guarded explicitly. + set +e t=api::openai_handlers::qwen35_serve_tests::gpu_every_serve_route_prefills_through_the_batched_prefill model="$HOME/models/Qwen3.5-0.8B-Q4_K_M.gguf" [ -f "$model" ] || { echo "::error::#4250 UNMEASURABLE on ${{ matrix.name }} (provisioning): $model is absent"; exit 1; } From ee4057c0dad74d7e7569a95816638861f43f9e5e Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 4 Oct 2026 11:09:20 +0200 Subject: [PATCH 2/2] chore(roadmap): PMAT-4715 fragment (B2 parent #4715) Agent: aprender-wmainred Co-Authored-By: Claude Opus 5.5 --- docs/roadmaps/entries/PMAT-4715.yaml | 19 +++++++++++++++++++ docs/roadmaps/roadmap.yaml | 19 +++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 docs/roadmaps/entries/PMAT-4715.yaml diff --git a/docs/roadmaps/entries/PMAT-4715.yaml b/docs/roadmaps/entries/PMAT-4715.yaml new file mode 100644 index 0000000000..398c99140b --- /dev/null +++ b/docs/roadmaps/entries/PMAT-4715.yaml @@ -0,0 +1,19 @@ +- id: PMAT-4715 + github_issue: 4715 + item_type: task + title: 'B2: root-cause fixes for red nightlies on main' + status: inprogress + priority: critical + assigned_to: aprender-wmainred + created: 2026-10-04T00:00:00Z + updated: 2026-10-04T00:00:00Z + acceptance_criteria: + - Each red or cancelled scheduled workflow on main has a root cause named in its fix PR, one PR per defect, listed on issue #4715 + - Each fix PR changes no gate's acceptance; a threshold or acceptance change goes to the train lead's sign-off batch instead + - After each fix merges, the next scheduled or dispatched run of that workflow on main gets past the step that failed + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:ci + notes: 'Parent #4678 (0.70.2), bar B2: no check red on main for more than 24 h. Proposed by aprender-a7, minted by cop aprender-14.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index f65016f6ce..6d12cec68b 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -24060,3 +24060,22 @@ Not done, pending the ruling: moving the 31 fixture files to an obviously synthe labels: - kind:docs notes: 'Operator C197, 2026-10-01 12:40Z: re-date via the spec''s own procedure; check unchanged; not a waiver.' +- id: PMAT-4715 + github_issue: 4715 + item_type: task + title: 'B2: root-cause fixes for red nightlies on main' + status: inprogress + priority: critical + assigned_to: aprender-wmainred + created: 2026-10-04T00:00:00Z + updated: 2026-10-04T00:00:00Z + acceptance_criteria: + - Each red or cancelled scheduled workflow on main has a root cause named in its fix PR, one PR per defect, listed on issue #4715 + - Each fix PR changes no gate's acceptance; a threshold or acceptance change goes to the train lead's sign-off batch instead + - After each fix merges, the next scheduled or dispatched run of that workflow on main gets past the step that failed + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:ci + notes: 'Parent #4678 (0.70.2), bar B2: no check red on main for more than 24 h. Proposed by aprender-a7, minted by cop aprender-14.'