Skip to content

Release Rehearsal Nightly: pick job cannot fetch the nightly-evidence branch (not_measured) #5052

Closed as not planned

Description

@noahgift

LAB (nightly, not a merge or release gate). The scheduled "Release Rehearsal Nightly" run failed on 2026-10-10 at 11:12Z in its pick job.

Observed: rehearse.sh --pick-c exited 2 (not_measured). The log reads fatal: couldn't find remote ref refs/heads/nightly-evidence, then "nightly-evidence was not fetched from origin (absent, or the read failed)".

Expected: the nightly finds its evidence branch, or the job reports a clear not-yet-created state that is distinct from a failed read.

Open question for the owner: is the branch missing because it was never created or was deleted, or did the fetch fail? Related: #4959 (the rehearsal's design ticket).

Refs #4959

Activity

  1. added this to the 0.71.1 milestone on Oct 10, 2026
  2. noahgift commented on Oct 12, 2026

    @noahgift
    ContributorAuthor

    Evidence (aprender-wprodmodels, 2026-10-12). Three nights, three different causes:

    • 10-10, run 38047594158 (what this ticket was filed for): couldn't find remote ref refs/heads/nightly-evidence. That cause is gone. The branch exists now (a85548b, published by the models producer on 10-11 at 20:45Z).
    • 10-11, run 38135547280: pick's case table was red, 236/265 rows, with miswired: the train has 3 step-name captures, not 2. Fixed on main by 3906777, which counts distinct capture names. rehearse.sh --selftest at main 8c12fca is 269/269.
    • 10-12, run 38254162909 (new): the case table and mutants passed, then --pick-c exited 2 with not_measured: paiml/infra clean-room runs were not read. The rate_limit read under the infra token succeeded. With a different reader, the same three endpoints (clean-room.yml runs, run jobs, job logs) all return 200. So the failure is specific to reads made with the INFRA_ACTIONS_READ secret, most likely its scope or expiry on paiml/infra. I did not touch the secret.
      • Why the log could not say which read failed: infra_gh sent gh's stderr to /dev/null.
      • Branch fix/5052-infra-read-names-status @ 171d798 (no PR): on a failed read, infra_gh now prints one line with the API path, the HTTP status and gh's first error line. Every read that failed before still fails, so no decision changes. Drill output: infra read failed: repos/paiml/infra/actions/workflows/clean-room.yml/runs?per_page=1: HTTP 401: gh: Bad credentials (HTTP 401). release_lanes.sh --self-test is 116 ok, 0 broke.

    Leaving the ticket open.

  3. added a commit that references this issue on Oct 12, 2026
  4. modified the milestones: 0.71.1, 0.72.0 on Oct 14, 2026
  5. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #5306 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch: fix/5052-infra-read-names-status.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions