From d978fe972c4ff7cc0e014eedfef2d5ece757b34b Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Fri, 16 Oct 2026 14:05:17 +0200 Subject: [PATCH 1/4] =?UTF-8?q?fix(pr-gate):=20run=20authorize=20on=20a=20?= =?UTF-8?q?pool=20every=20caller=20can=20reach=20=E2=80=94=20it=20queued?= =?UTF-8?q?=2024=20h=20on=20repos=20outside=20the=20intel=20groups?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `runs-on` in a reusable workflow resolves against the CALLER's runner groups. Every runner labelled `intel` sits in a group with a repo allow-list (groups 6, 8, 9), so callers outside those lists (bashrs, forjar-cookbook, pmat-book and most of the ~50 callers) had no eligible runner: the job waited GitHub's 24 h queue limit and then failed or was cancelled. Measured 2026-10-16: bashrs 8/8 queued 24 h then cancelled; forjar-cookbook #24/#27 1440 min each. The new label set `[self-hosted, Linux, clean-room, ephemeral]` matches exactly 7 runners (yoga-build x3, gx10-build, gx10-pool1..3), all in the Default group (visibility all, public repos allowed). Those containers carry GH_CONFIG_DIR=/opt/ci-gh. The authorization logic is unchanged; if the ambient auth cannot see private memberships, the existing instrument probe fails red in seconds and closes nothing. actionlint 1.7.12: rc=0; control without the `ephemeral` declaration: rc=1. Co-Authored-By: Claude Opus 5.5 --- .github/actionlint.yaml | 1 + .github/workflows/pr-gate.yml | 20 +++++++++++++++++++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml index 71a947e..e339bf7 100644 --- a/.github/actionlint.yaml +++ b/.github/actionlint.yaml @@ -28,3 +28,4 @@ self-hosted-runner: - yoga - perf-solo - merge-queue + - ephemeral diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml index 5b51986..7852dcb 100644 --- a/.github/workflows/pr-gate.yml +++ b/.github/workflows/pr-gate.yml @@ -31,8 +31,26 @@ jobs: # anywhere in this workflow. No PR-authored code is ever fetched or run here — # the job only reads `context.payload` and calls the API. If a checkout step is # ever added, this decision must be revisited. + # + # WHICH self-hosted pool: one EVERY caller is admitted to. This file is called + # from ~50 repos, but `runs-on` is resolved against the CALLER's runner groups. + # This used to be `[self-hosted, Linux, X64, clean-room, intel]`, and every + # runner carrying `intel` sits in a group with a repo allow-list (aprender; + # forjar/infra/paiml-implement/paiml-mcp-agent-toolkit; catalogue/rmedia). + # Any other caller (bashrs, forjar-cookbook, pmat-book, ...) had NO eligible + # runner, so the job sat queued for GitHub's 24 h limit and then failed or was + # cancelled — a 1440-minute false red on every PR of those repos, for a check + # whose work takes seconds (measured 2026-10-16: bashrs 8/8 queued 24 h then + # cancelled; forjar-cookbook #24/#27 1440 min each). + # + # The only group open to all repos (public included) is `Default`, and its + # `clean-room` + `ephemeral` runners are the yoga-build and gx10 pool + # containers. They carry GH_CONFIG_DIR=/opt/ci-gh, which is the ambient auth + # the step below relies on. If that auth cannot see private org memberships, + # the instrument probe below fails RED within seconds and closes nothing — a + # visible, immediate answer instead of a 24 h queue that answers nothing. timeout-minutes: 22 - runs-on: [self-hosted, Linux, X64, clean-room, intel] + runs-on: [self-hosted, Linux, clean-room, ephemeral] permissions: pull-requests: write steps: From f376fd7c171ad2d6650fc1a86f65d45f848609aa Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 18 Oct 2026 02:14:22 +0200 Subject: [PATCH 2/4] fix(pr-gate): refresh stale comments (#83 review nits) Co-Authored-By: Claude Sonnet 5.5 --- .github/actionlint.yaml | 3 ++- .github/workflows/pr-gate.yml | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml index e339bf7..605f596 100644 --- a/.github/actionlint.yaml +++ b/.github/actionlint.yaml @@ -10,9 +10,10 @@ # actually mean something. # # Sources, so this can be re-derived rather than guessed: -# gh api /orgs/paiml/actions/runners -> clean-room, intel +# gh api /orgs/paiml/actions/runners -> clean-room, intel, ephemeral (Default group: yoga-build and gx10 pool) # gh api repos/paiml/aprender/actions/runners -> ada, blackwell, cuda, gb10, rtx4090 # grep -rhoE 'runs-on: \[[^]]*\]' .github/workflows/ -> amd, gpu, lambda-labs, yoga +# .github/workflows/pr-gate.yml runs-on -> clean-room, ephemeral (the pool every caller is admitted to) self-hosted-runner: labels: - clean-room diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml index 7852dcb..34d1887 100644 --- a/.github/workflows/pr-gate.yml +++ b/.github/workflows/pr-gate.yml @@ -16,7 +16,7 @@ on: jobs: authorize: - # Runs on self-hosted intel. GitHub-hosted runners are BANNED fleet-wide: + # Runs on the self-hosted `clean-room` + `ephemeral` pool (see WHICH pool below). GitHub-hosted runners are BANNED fleet-wide: # they bill on private repos, and when the Actions spending limit is reached # the job does not start AT ALL — it reports `failure` with ZERO steps, which # is indistinguishable from this gate rejecting an author. That is exactly @@ -27,7 +27,7 @@ jobs: # # The original rationale for ubuntu-latest was that untrusted PR code must not # execute on self-hosted infra. That concern does not apply to THIS job: it has - # exactly one step, `actions/github-script`, and there is NO `actions/checkout` + # a single github-script step (`Close unauthorized PR`) plus a bash authorization step, and there is NO `actions/checkout` # anywhere in this workflow. No PR-authored code is ever fetched or run here — # the job only reads `context.payload` and calls the API. If a checkout step is # ever added, this decision must be revisited. From d6353c53c8c0706372b603da8f502127d7f32534 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 18 Oct 2026 04:24:29 +0200 Subject: [PATCH 3/4] pr-gate: drop credential path and host names from comments (comment-only) Cop sign-off condition: paiml/.github is public, so comments say only 'ambient auth on the Default-pool runners'. No code change. Co-Authored-By: Claude Sonnet 5.5 --- .github/actionlint.yaml | 2 +- .github/workflows/pr-gate.yml | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml index 605f596..d209e04 100644 --- a/.github/actionlint.yaml +++ b/.github/actionlint.yaml @@ -10,7 +10,7 @@ # actually mean something. # # Sources, so this can be re-derived rather than guessed: -# gh api /orgs/paiml/actions/runners -> clean-room, intel, ephemeral (Default group: yoga-build and gx10 pool) +# gh api /orgs/paiml/actions/runners -> clean-room, intel, ephemeral (Default group) # gh api repos/paiml/aprender/actions/runners -> ada, blackwell, cuda, gb10, rtx4090 # grep -rhoE 'runs-on: \[[^]]*\]' .github/workflows/ -> amd, gpu, lambda-labs, yoga # .github/workflows/pr-gate.yml runs-on -> clean-room, ephemeral (the pool every caller is admitted to) diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml index 34d1887..f3f142f 100644 --- a/.github/workflows/pr-gate.yml +++ b/.github/workflows/pr-gate.yml @@ -44,9 +44,9 @@ jobs: # cancelled; forjar-cookbook #24/#27 1440 min each). # # The only group open to all repos (public included) is `Default`, and its - # `clean-room` + `ephemeral` runners are the yoga-build and gx10 pool - # containers. They carry GH_CONFIG_DIR=/opt/ci-gh, which is the ambient auth - # the step below relies on. If that auth cannot see private org memberships, + # `clean-room` + `ephemeral` runners are the Default-pool runners. They + # carry ambient auth on the Default-pool runners, which is the auth the + # step below relies on. If that auth cannot see private org memberships, # the instrument probe below fails RED within seconds and closes nothing — a # visible, immediate answer instead of a 24 h queue that answers nothing. timeout-minutes: 22 From deb29d953fa648864b576a00e46f1af7f318b3f3 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sun, 18 Oct 2026 05:13:13 +0200 Subject: [PATCH 4/4] ci: run validate+gate on the Default pool so the required check can start This repo is on no intel runner group's allow-list, so the required 'gate' check stayed queued and the PR could never merge. Same stranded label defect as pr-gate.yml. X64 kept: actionlint download is amd64-only. Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/ci.yml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 33e0eac..8f4b7f1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,10 @@ jobs: # private repos, and at the spending limit a job does not start at all, # reporting `failure` with zero steps. See paiml/infra#231. timeout-minutes: 15 - runs-on: [self-hosted, Linux, X64, clean-room, intel] + # Default-pool runners: this repo is not on any intel runner group's allow-list, + # so the old label set left this required check queued forever. X64 stays + # because the actionlint download below is amd64-only. + runs-on: [self-hosted, Linux, X64, clean-room, ephemeral] steps: - uses: actions/checkout@v6 - name: Lint workflow YAML @@ -111,7 +114,10 @@ jobs: # against a gate that silently stops running on billing state, self-hosted # is the safer failure mode. timeout-minutes: 15 - runs-on: [self-hosted, Linux, X64, clean-room, intel] + # Default-pool runners: this repo is not on any intel runner group's allow-list, + # so the old label set left this required check queued forever. X64 stays + # because the actionlint download below is amd64-only. + runs-on: [self-hosted, Linux, X64, clean-room, ephemeral] if: always() needs: [validate] steps: