fold(B2): five nightly root-cause fixes in one PR (#4705 #4706 #4707 #4708 #4713) + one roadmap regen

.github/workflows/conleche-nightly.yml

@@ -42,7 +42,6 @@ jobs:
     # 2.6 min) -> T = max(15, 68, 65) = 68, rounded up for a shared host. The first three runs replace this basis.
     timeout-minutes: 90
     env:
-      CI_CACHE: /mnt/nvme-raid0/ci-cache/pvl-conleche
       ELAN_VERSION: v4.2.4
       ELAN_SHA256: 42b94d4244e8353142c456ec0e4ca6528fd898a6c604d4059f494e706e431f63
       LEAN_NUM_THREADS: '8'
@@ -55,6 +54,11 @@ jobs:
       - name: Pinned elan in the job's own directory (sha256-checked, never a host install)
         run: |
           set -euo pipefail
+          # The cache lives beside the runner's _work dir, as nightly.yml's cache-nightly does. The old
+          # /mnt/nvme-raid0/ci-cache path exists only on lambda; on the clean-room hosts /mnt/nvme-raid0 is
+          # root-owned, so mkdir failed and the job never ran (runs 37082537768, 37163229207).
+          CI_CACHE="$(cd "$GITHUB_WORKSPACE/../.." && pwd)/cache-conleche"
+          echo "CI_CACHE=$CI_CACHE" >> "$GITHUB_ENV"
           export ELAN_HOME="$CI_CACHE/elan"
           mkdir -p "$CI_CACHE"
           if [ ! -x "$ELAN_HOME/bin/elan" ] || ! "$ELAN_HOME/bin/elan" --version | grep -q "${ELAN_VERSION#v}"; then

.github/workflows/cuda-nightly.yml

@@ -707,6 +707,11 @@ jobs:
         if: ${{ !cancelled() && steps.decide.outputs.proceed == 'true' }}
         run: |
           set -uo pipefail
+          # -e OFF, as in the perf041 step: the default shell is `bash -e {0}`, so the
+          # RED control's expected exit 101 aborted the step at `; r=$?` before its
+          # assertions ran (run 37161229337: GREEN passed, step failed 101, no ::error::).
+          # Every failure path below is guarded explicitly.
+          set +e
           t=api::openai_handlers::qwen35_serve_tests::gpu_every_serve_route_prefills_through_the_batched_prefill
           model="$HOME/models/Qwen3.5-0.8B-Q4_K_M.gguf"
           [ -f "$model" ] || { echo "::error::#4250 UNMEASURABLE on ${{ matrix.name }} (provisioning): $model is absent"; exit 1; }

.github/workflows/mutants-nightly.yml

@@ -104,8 +104,9 @@ jobs:
                 }
               fi
               cargo mutants --version || { echo "::error::cargo-mutants not runnable after install"; exit 1; }
-              # the crates ci.yml workspace-test --excludes, derived from its nextest line (see ci.yml mutants)
-              EXCL=$(grep -m1 -E "^[[:space:]]+cargo nextest run --profile ci --workspace --lib" .github/workflows/ci.yml \
+              # the crates workspace-test --excludes, derived from its nextest line. That line moved from
+              # ci.yml into ci/sections.yml (#4655); reading the old file found nothing and failed every night.
+              EXCL=$(grep -m1 -E "^[[:space:]]+cargo nextest run --profile ci --workspace --lib" ci/sections.yml \
                 | grep -oE -- "--exclude [a-z0-9_-]+" | sed "s/^--exclude /--exclude-crate /" | tr "\n" " ")
               [ -n "$EXCL" ] || { echo "::error::no --exclude list found on the workspace-test nextest line"; exit 1; }
               # shellcheck disable=SC2086

.github/workflows/nightly.yml

@@ -80,7 +80,9 @@ jobs:
           ref: main
 
       - name: Case table
-        run: python3 scripts/nightly_manifest.py --self-test
+        run: |
+          python3 scripts/nightly_manifest.py --self-test
+          bash scripts/nightly_prune_release.sh --self-test
 
       - name: Decide (reused | build | red-ci | ci-pending)
         id: gate
@@ -259,6 +261,11 @@ jobs:
       - name: Build apr (host-native, pinned toolchain)
         run: |
           set -euo pipefail
+          # Every later step reads ./target. The mini's runner env redirects cargo's
+          # target dir (CARGO_TARGET_DIR=~/data/cargo-target), so cargo built
+          # elsewhere and the --version line found no apr (run 37171118510, exit
+          # 127). Same fix as binary-release.yml's mini lane (#4660).
+          export CARGO_TARGET_DIR="$GITHUB_WORKSPACE/target"
           rustup show active-toolchain
           cargo build --release --locked -p apr-cli --target "$T"
           "target/$T/release/apr" --version
@@ -321,6 +328,13 @@ jobs:
           cat nightly-manifest.json
           echo "decision=$(python3 -c 'import json;print(json.load(open("nightly-manifest.json"))["decision"])')" >> "$GITHUB_OUTPUT"
 
+      # publish lists one page of assets (100); the release overflowed it, so staged
+      # leftovers and old copies past page one made every upload 422 (run 37171118510).
+      - name: Prune the release to what publish can see
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: bash scripts/nightly_prune_release.sh dist nightly-manifest.json "$NIGHTLY_SHA"
+
       # REST, not `gh`: the fleet boxes carry no gh (binary-release.yml, run 34448908554).
       - name: Publish green arches, then nightly-manifest.json
         env:

ci/sections.yml

@@ -1394,6 +1394,7 @@ jobs:
           bash scripts/ci_resolve_dirty.sh --selftest
           python3 scripts/lib/roadmap_merge.py --selftest
           python3 scripts/nightly_manifest.py --self-test   # #4189 nightly gate/manifest/publish plan
+          bash scripts/nightly_prune_release.sh --self-test   # nightly publish overflowed page one (422)
       - name: "target-watch: before-cargo-install-bashrs"
         # aprender#2822 boundary probe (2 stat, 1 readdir, 1 df, 1 pgrep).
         # Instrumentation, never a gate: the script always exits 0.

docs/roadmaps/entries/GH-4718.yaml

@@ -0,0 +1,17 @@
+- id: GH-4718
+  github_issue: 4718
+  item_type: task
+  title: '0702-P1: nightly producers for the deep lanes (deep-doctests, deep-nodefault, deep-bins-smoke)'
+  status: inprogress
+  priority: critical
+  assigned_to: null
+  created: 2026-10-04T09:17:34.904751550+00:00
+  updated: 2026-10-04T09:17:34.904751550+00:00
+  spec: null
+  acceptance_criteria: []
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - P0
+  notes: null

docs/roadmaps/entries/PMAT-4715.yaml

@@ -0,0 +1,19 @@
+- id: PMAT-4715
+  github_issue: 4715
+  item_type: task
+  title: 'B2: root-cause fixes for red nightlies on main'
+  status: inprogress
+  priority: critical
+  assigned_to: aprender-wmainred
+  created: 2026-10-04T00:00:00Z
+  updated: 2026-10-04T00:00:00Z
+  acceptance_criteria:
+  - Each red or cancelled scheduled workflow on main has a root cause named in its fix PR, one PR per defect, listed on issue #4715
+  - Each fix PR changes no gate's acceptance; a threshold or acceptance change goes to the train lead's sign-off batch instead
+  - After each fix merges, the next scheduled or dispatched run of that workflow on main gets past the step that failed
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:ci
+  notes: 'Parent #4678 (0.70.2), bar B2: no check red on main for more than 24 h. Proposed by aprender-a7, minted by cop aprender-14.'

docs/roadmaps/roadmap.yaml

@@ -18483,6 +18483,23 @@ roadmap:
   estimated_effort: null
   labels: []
   notes: 'Before: v0.70.1 tag run completed 25 min after start with no ci / coverage job; the gate refused only at preflight, after the tag and the GitHub release were public (up to 90 tries x 60 s, 2 gh calls/try). After: refused at cut_tag, 0 gh calls, 0 s. NOTE: ci.yml today declares no ci / coverage job (coverage runs inside x86-main), so every release refuses at cut_tag until one is declared or the gate is retargeted.'
+- id: GH-4718
+  github_issue: 4718
+  item_type: task
+  title: '0702-P1: nightly producers for the deep lanes (deep-doctests, deep-nodefault, deep-bins-smoke)'
+  status: inprogress
+  priority: critical
+  assigned_to: null
+  created: 2026-10-04T09:17:34.904751550+00:00
+  updated: 2026-10-04T09:17:34.904751550+00:00
+  spec: null
+  acceptance_criteria: []
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - P0
+  notes: null
 - id: PMAT-3351
   github_issue: 3347
   item_type: task
@@ -24081,6 +24098,25 @@ Not done, pending the ruling: moving the 31 fixture files to an obviously synthe
   labels:
   - kind:docs
   notes: 'Operator C197, 2026-10-01 12:40Z: re-date via the spec''s own procedure; check unchanged; not a waiver.'
+- id: PMAT-4715
+  github_issue: 4715
+  item_type: task
+  title: 'B2: root-cause fixes for red nightlies on main'
+  status: inprogress
+  priority: critical
+  assigned_to: aprender-wmainred
+  created: 2026-10-04T00:00:00Z
+  updated: 2026-10-04T00:00:00Z
+  acceptance_criteria:
+  - Each red or cancelled scheduled workflow on main has a root cause named in its fix PR, one PR per defect, listed on issue #4715
+  - Each fix PR changes no gate's acceptance; a threshold or acceptance change goes to the train lead's sign-off batch instead
+  - After each fix merges, the next scheduled or dispatched run of that workflow on main gets past the step that failed
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:ci
+  notes: 'Parent #4678 (0.70.2), bar B2: no check red on main for more than 24 h. Proposed by aprender-a7, minted by cop aprender-14.'
 - id: PMAT-4719
   github_issue: 4719
   item_type: task