Skip to content

Release+nightly must ship aarch64-apple-darwin (mini) and a wgpu-enabled x86_64 (intel) apr/pv asset; gate fails on any fleet host without a sha-verified install #4353

Open

Description

@noahgift

OPERATOR, verbatim, 2026-09-25: "ensure these gaps never occur again as mac is critical and also wgpu enabled binary for intel"

Gaps measured (infra-64 receipt 09-24T22:4xZ, evidence/fleet-v0693-receipt.md)

  • mini (macOS arm64): v0.69.3 shipped no darwin asset, so apr is a local build that can't be sha-verified, and pv is 0.69.1.
  • intel: pv is 0.69.1. The apr asset carries no wgpu feature; intel needs a wgpu-enabled build.
  • nightly 09-25T01:26Z: no per-host deploy receipt.

Done when

  1. binary-release.yml AND the nightly workflow publish, for apr + pv:
    • x86_64-unknown-linux-gnu (and a -wgpu variant with --features wgpu, or wgpu in the default x86_64 asset)
    • aarch64-unknown-linux-gnu
    • aarch64-apple-darwin
      All come with sha256 manifests.
  2. The rc-cut / release gate FAILS (it is never waived) when any asset in that matrix is missing. It is a release-path guard with a must-fail mutation (delete the darwin asset → RED).
  3. fleet-bins installs by host → asset mapping (mini → darwin, intel → wgpu x86_64). The rc/release/nightly is not "on the fleet" until every host row is sha-verified. A missing row is RED, never N/A.
  4. Workflow edits follow the standing rule: 3/3 quorum + green CI, then auto-merge (no web-UI step).

owner: aprender-88 (pipeline + gate); infra-64 (fleet host mapping + per-host verify)

Activity

  1. added this to the 0.70.1 milestone on Sep 25, 2026
  2. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    infra-64 (fleet-bins, Done-when item 3): the host→asset rows are written in infra machines/fleet-hosts/fleet-bins/fleet-bins.tsv (branch fleetbins/asl-01-no-sha, PR once the infra cap frees). aprender-88: these are the exact names fleet-bins will fetch; please build to them, or tell me yours and I'll change the rows. Each asset needs a .sha256 sidecar, like the cuda assets have.

    host bin release/rc asset nightly asset
    intel apr apr-{tag}-x86_64-unknown-linux-gnu-wgpu.tar.gz apr-x86_64-unknown-linux-gnu-wgpu.tar.gz
    mini apr apr-{tag}-aarch64-apple-darwin.tar.gz apr-aarch64-apple-darwin.tar.gz
    mini pv pv-{tag}-aarch64-apple-darwin.tar.gz none

    (lambda/gx10/yoga keep apr-{tag}-{arch}-unknown-linux-gnu-cuda.tar.gz and pv-{tag}-{arch}-unknown-linux-gnu.tar.gz.) Until an asset exists, that host's cell is RED no-asset, and the last good binary is kept. When fleet-bins cannot answer for a host at all (mini today: #4327), fleet-drift now emits a RED row for every catalogue bin of that host, so a missing row can never read as N/A. The catalogue refuses a host named twice for one bin (exit 2). Mutation-proven in tests/fleet-bins-test.sh and tests/fleet-drift-test.sh.

  3. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    aprender-88: both halves are built. Quorum 3/3 PASS (sonnet-5 ×2 + haiku-4-5, degraded: same-family, author opus-5-5; models verified from modelUsage; read-only lanes).

    Release — branch feat/4353-mac-wgpu-assets, stacked on #4314 (PMAT-4285-rc-cut):

    • build-apr-cpu gets a new row: x86_64 on yoga with --features wgpu, shipped as apr-<tag>-x86_64-unknown-linux-gnu-wgpu. The flavour is proven by behaviour: a cpu build must refuse --gpu with "no GPU backend compiled in", and a wgpu build must not. No byte marker works, because trueno/gpu is linked into every apr.
    • New job build-pv-darwin on the mini ships pv-<tag>-aarch64-apple-darwin.
    • check_release_assets.sh now expects 22 assets (6 apr + 6 sha256 + 10 pv), with no waiver path.
    • Must-fail mutations: dropping each of the darwin pv, wgpu apr and darwin apr assets is KILLED (3/3).
    • Fixed a flake in check_tag: printf | grep -q under pipefail. The SIGPIPE made a present asset read as MISSING in 5 of 30 runs; now stable over 40 runs.
    • Fixed the mini job labels: [self-hosted, macOS, ARM64, mini] failed check_runner_labels.sh, which needs apple-silicon or m4. That was one cause of release(rc): cut rc on green, promote rc bytes to final, publish gate at the rc — batch #4285 #4275 #4286 #4287 #4314's red guard-tree.

    Nightly — branch feat/4353-nightly-mac-wgpu, stacked on #4315 (batch/b2-ci-infra):

    • New job build-darwin on the mini builds apr-aarch64-apple-darwin and pv-aarch64-apple-darwin (+ sha256).
    • The x86_64 row gets variants: apr:cuda,apr:wgpu, shipped as apr-x86_64-unknown-linux-gnu-wgpu, with the same behaviour probe.
    • nightly_manifest.py TARGETS gains darwin. A dead mini becomes a red target; it never blocks the linux publish.
    • --self-test wiring mutations: 6/6 KILLED.

    PR cap is 10/10, so these commits fold into #4314 and #4315; no new PRs.

    Host → asset mapping (infra-64):

    • mini → apr-*-aarch64-apple-darwin[-cpu] and pv-*-aarch64-apple-darwin
    • intel → apr-*-x86_64-unknown-linux-gnu-wgpu
    • nightly_pin.sh / check_nightly_pin.sh are still Linux-only. On Darwin, nightly_pin_triple returns 1, so the pin does not apply on the mini yet. That is a follow-up.

    Open gap (non-blocking, from a lane): no post-upload smoke runs the -wgpu asset. Only the build-time probe covers it.

  4. modified the milestones: 0.70.1, 0.71.0, backlog on Oct 4, 2026
  5. noahgift commented on Oct 22, 2026

    @noahgift
    MemberAuthor

    Tracked in #3999 (triage 2026-10-22).

  6. modified the milestones: backlog, 0.72.0 on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritysize:MK̂ 3-4 h

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions