Skip to content

fix(ci): conleche-nightly cache path is lambda-only — mkdir denied on clean-room - #4708

Closed
noahgift wants to merge 2 commits into
mainfrom
fix/conleche-cache-path
Closed

noahgift wants to merge 2 commits into
mainfrom
fix/conleche-cache-path

Conversation

@noahgift

@noahgift noahgift commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

conleche-nightly has never gone green since #4655. CI_CACHE=/mnt/nvme-raid0/ci-cache/pvl-conleche exists only on lambda. On the clean-room runner (intel), /mnt/nvme-raid0 is root-owned, so the elan step fails with mkdir: Permission denied (runs 37082537768, 37163229207).

The fix computes the cache dir beside the runner's _work and exports it through GITHUB_ENV. PVL_CONLECHE_CACHE="$CI_CACHE/pins" is unchanged. No check changes.

Refs #4681 (B2: no check red on main >24 h)

🤖 Generated with Claude Code

… clean-room, job never ran

CI_CACHE was hard-coded to /mnt/nvme-raid0/ci-cache/pvl-conleche. The job runs on
[self-hosted, Linux, X64, clean-room], which lands on intel, where /mnt/nvme-raid0 is
root-owned: `mkdir: cannot create directory '/mnt/nvme-raid0/ci-cache': Permission denied`
(runs 37082537768, 37163229207). The workflow has never gone green since #4655.

The cache now lives beside the runner's _work dir and is exported via GITHUB_ENV, so the
con-leche step's PVL_CONLECHE_CACHE="$CI_CACHE/pins" is unchanged. No check changes.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift

noahgift commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-4715",
 "head": "232691cb53a6d544953d805c10af667c8c51e896",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "NO-VERDICT",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift

noahgift commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): three PASS — agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-4715",
 "head": "232691cb53a6d544953d805c10af667c8c51e896",
 "width": 3,
 "executor": "agy",
 "agreed": true,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift
noahgift enabled auto-merge October 4, 2026 09:37
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=4708 head=232691cb53a6d544953d805c10af667c8c51e896 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

noahgift commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #4777 (one PR for the five B2 nightly fixes). The fix commit is moved there unchanged; this branch is kept.

@noahgift noahgift closed this Oct 5, 2026
auto-merge was automatically disabled October 5, 2026 06:22

Pull request was closed

github-merge-queue Bot pushed a commit that referenced this pull request Oct 5, 2026
…4708 #4713) + one roadmap regen (#4777)

* fix(mutants-nightly): read the workspace-test excludes from ci/sections.yml, where the line now lives

The nightly derived its --exclude-crate list by grepping ci.yml for the
workspace-test nextest line. #4655 moved that line into ci/sections.yml
(ci.yml's PR shard already reads it there), so the grep matched nothing
and the step failed closed every night: run 37183004032, 'no --exclude
list found on the workspace-test nextest line'.

Same derivation, same fail-closed check, same exclusions
(aprender-gpu, aprender-cuda-edge, aprender-compute); only the file read
changes. Proof: old path -> empty, new path -> the three crates.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5dccaa4)

* fix(nightly): darwin leg built into the mini's redirected target dir, so --version found no apr

The mini runner's env sets CARGO_TARGET_DIR=~/data/cargo-target. cargo
built there and the next line, "target/$T/release/apr" --version, exited
127 (run 37171118510, job 'apr aarch64-apple-darwin on mini'). Package
reads the same ./target path.

Step-local export of CARGO_TARGET_DIR=$GITHUB_WORKSPACE/target, the fix
#4660 already made for binary-release.yml's mini lane. No check changes.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 25ce022)

* fix(nightly): publish read one page of release assets; staged leftovers past it made every upload 422

The nightly release holds 234 assets, 117 of them staged.* leftovers from
runs that died mid-publish. publish() listed assets with per_page=100 and
no paging, so it saw 100. The leftovers past page one were never deleted,
and staging the same name again failed HTTP 422 already_exists (run
37171118510, job publish). Since the cleanup can't see them, the failure
repeats every night.

release_assets() reads every page. Two self-test rows: 234 over 3 pages,
and an exact multiple of 100. Mutant 'page one only' fails both rows.

Local complexity hook skipped under C205(3) as extended: hook errors on
scripts/nightly_manifest.py 4 before -> 4 after (publish cognitive
192 -> 181; self_test cyclomatic 30 = base). SATD 4 <= 5 and the docs
check were run by hand, green. Logged in controls-ledger.md.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 99ac4c1)

* fix(ci): conleche-nightly cache path is lambda-only — mkdir denied on clean-room, job never ran

CI_CACHE was hard-coded to /mnt/nvme-raid0/ci-cache/pvl-conleche. The job runs on
[self-hosted, Linux, X64, clean-room], which lands on intel, where /mnt/nvme-raid0 is
root-owned: `mkdir: cannot create directory '/mnt/nvme-raid0/ci-cache': Permission denied`
(runs 37082537768, 37163229207). The workflow has never gone green since #4655.

The cache now lives beside the runner's _work dir and is exported via GITHUB_ENV, so the
con-leche step's PVL_CONLECHE_CACHE="$CI_CACHE/pins" is unchanged. No check changes.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 344d386)

* fix(ci): CUDA Nightly G0 step aborted on its own RED control — bash -e killed it before r=$?

The G0 step (aprender#4250) runs under GitHub's default `bash -e {0}`. Its
`set -uo pipefail` does not turn -e off, so the per-token RED control's expected
exit 101 aborted the step at `; r=$?`, before the assertions ran. Run 37161229337:
the GREEN arm passed, then the step failed 101 with no ::error:: line.

Turn -e off with `set +e`, as the perf041 step in this workflow already does. Every
failure path is guarded explicitly (`|| exit 1` and the GREEN/RED checks), so the
step accepts exactly what it was written to accept.

Repro: bash -e -c 'set -uo pipefail; (exit 101); r=$?; echo reached' prints nothing, rc=101.

Refs #4681

Agent: aprender-wmainred
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0ae57c0)

* chore(roadmap): PMAT-4715 + GH-4718 fragments, one aggregate regen for the B2 fold

The five B2 nightly fixes above carried the same PMAT-4715 fragment each; it lands
once here. GH-4718's fragment moves here too, so the queue holds one roadmap.yaml
regen. make roadmap-aggregate-check: ok, 299 fragments, idempotent.

Pmat-Ticket: PMAT-4715
Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Revert "fix(nightly): publish read one page of release assets; staged leftovers past it made every upload 422"

This reverts commit c356ec8. It adds Python to scripts/nightly_manifest.py,
and the build takes no new Python. The 422 fix is redone in bash under #4681.

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nightly): prune the release in bash before publish; one page of assets made every upload 422

The nightly release held 234 assets: 117 staged.* leftovers from runs that died
mid-publish and 117 real ones. publish lists one page (per_page=100), so the
leftovers past it were never deleted and staging the same name again failed
HTTP 422 already_exists (run 37171118510), every night. Purging leftovers alone
still leaves 117 real assets, 17 past page one, and renaming onto one of those
is the same 422.

scripts/nightly_prune_release.sh runs before publish and reads every page. It
deletes every staged.* leftover, and the old copy of each name this run
re-uploads that sits past publish's page one. A past-page-one asset the run
does not upload (another arch's last green) is kept. Any API failure exits
nonzero; a 404 release is the first run. Bash only: this replaces the Python
version of the fix, reverted above.

--self-test: 13 rows, PASS, stub API, no network. Mutants, each RED:
page-one-only listing (7 rows fail), never drop past-page-one copies (5),
never purge staged (6). Runs in the nightly gate job and in ci/sections.yml
beside nightly_manifest.py --self-test. bashrs lint: 0 errors. actionlint on
nightly.yml: 2 findings before and after.

Refs #4681
Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nightly): a non-array asset page is an unread listing, never the empty last page

nightly_prune_release.sh turned any jq failure on an assets page into an
empty batch, which ends the page loop. An HTTP 200 whose body is not a JSON
array (a proxy's HTML, an error object) therefore truncated the listing and
exited 0, and publish would still hit the 422 the script exists to prevent.
Check the body is an array, else die; an empty array still ends the loop.

New self-test row: page 2 answers 200 with HTML -> exit nonzero. Restoring the
old line turns it RED (got 0, want 1). 14 rows PASS; bashrs 0 errors.

Refs #4681

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nightly): prune deletes only names in publish's own plan; the array guard gets a row that can fail

Re-review of 50f74e8 found two gaps.

1. A past-page-one asset was deleted if its name was a file in DIST. publish
   uploads less than DIST: publish_plan keeps only targets green at SHA and
   built by this run. A file in DIST outside that plan lost its old copy and got
   no new one. The script now takes MANIFEST and SHA, mirrors publish_plan in
   jq, and deletes only a planned name that is present in DIST (plus the
   manifest). nightly.yml passes nightly-manifest.json and $NIGHTLY_SHA.

2. The non-array row answered HTML, which the extraction line rejects on its
   own, so deleting the guard left the row green. The stub now answers `{}`,
   which only the guard catches.

Self-test: 21 rows PASS. Ten mutants each turn it RED: no array guard (1 row),
DIST-only membership (6), plan read not fail-closed (1), no status / sha / run
filter (2 each), page-one-only listing (10), no staged purge (2), manifest not
planned (1), substring membership (5). bashrs: 0 errors.

Refs #4681

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* evidence(pr-review): receipt for #4777 head b5bf02e

DEGRADED, same-family: gemini empty, Claude lanes claude-sonnet-5-5 and
claude-haiku-4-5-20251001 (round 3 unanimous PASS, planted question rejected
by both). pmat query + duplication scan (8 needles, 0 hits over 1376/1376
branches); 10/10 guard mutants killed; cuda consulted on the cuda-nightly.yml
path trigger (no-authority-found: exit 101 is the Rust harness). Checked with
a throwaway minisign key: check_pr_review_receipt.sh ACCEPT.

Refs #4681

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* evidence(pr-review): #4777 receipt records each round-3 lane's verdict on its own

C312-9: a same-family answer stands only if unanimous. The receipt said so in
one summary result; it now carries one cited result per lane (exact model id,
PASS, planted Q3 rejected, own mutation RED), quoted verbatim from the quorum
record, and re-pins findings_ref.sha256. Throwaway-key check: ACCEPT.

Refs #4681

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nightly): self-test jq fixture avoids $all[] that bashrs 7.4.1 misreads as SC1087

CI's pinned bashrs 7.4.1 reports SC1087 at nightly_prune_release.sh:153 on the
jq expression $all[] inside a single-quoted program; 7.4.2 reports nothing. The
shell-lint ratchet counted 7 against a baseline of 6 on this PR merged with main;
main alone measures 6. [$all | .[] | ...] is the same jq, and 7.4.1 reports 0.
Self-test PASS.

Agent: aprender-a7
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* evidence(pr-review): receipt for #4777 head 4cc22e5 (C312-9 round 4)

DEGRADED, same-family: gemini empty. Round 4 on the self-test jq rewrite
(58b6456) and the main merge: claude-sonnet-5-5 PASS and
claude-haiku-4-5-20251001 PASS, unanimous, each with its own cited result;
the planted question (that 58b6456 touched the delete loop) was rejected
by both. Re-measured at head: self-test 21 rows PASS, 10/10 mutants RED,
duplication scan 0 hits over 1380/1380 branches. Throwaway-key check: ACCEPT.

Refs #4681

Agent: aprender-wtix-odd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift
noahgift deleted the fix/conleche-cache-path branch October 9, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant