Skip to content

ci(FLOW-010 P1): nextest archive — build the tree-reader set once, shards run from it (squashed; supersedes #5812) - #5854

Draft
noahgift wants to merge 11 commits into
mainfrom
recut/5812-sq
Draft

noahgift wants to merge 11 commits into
mainfrom
recut/5812-sq

Conversation

@noahgift

Copy link
Copy Markdown
Member

Supersedes #5812. Same content squashed into ONE commit on current main (merge queue is REBASE; force-push is banned). #5812 moves to draft, not closed. Seats carry by patch-id (Noah 11:41Z).

🤖 Generated with Claude Code

…ards run from it

Squashed from #5812 for the REBASE merge queue.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (w5404): APPROVE at head 615ea0f. Squashed re-cut of #5812; patch-id f7a8690519cc identical to the quorum-approved bf7060c (agy + Opus). Receipt: handoff/w5404-quorum-5854-615ea0f47c.md. Not armed.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2 (non-author, session aprender-wsweep, claude-sonnet-5-5) @ 615ea0f: APPROVE. Re-cut of #5812 as one commit on main (b9cb67a). git patch-id --stable of the PR vs its merge-base is f7a8690519cc2b8501922a22b588ff9283b701d1, identical to #5812 at bf7060c and baff7ca (9 files, +1317/-19), so the change w5404 reviewed is unchanged. check_guard_steps_isolated, check_explicit_test_commands, check_no_pipe_into_grep_q, check_ci_push_reuse, check_ci_wait_cancel all rc=0. No added continue-on-error key. Not armed.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=5854 head=fa23314b22cf7aa7c4d3ed4631a7d6aa4e696b36 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…he shards (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop gate sign-off @ce0bdf0ad9 (bound to this head; any new push voids it). Valid only together with a full quorum at this same head; it does not replace the quorum.

  • Delta reviewed: the new workspace-test-archive job (draft-skip if, mq lane on merge_group) is now in workspace-test-shard's needs, and the shards run from the archive via archive_bind_check.sh (sha/EXPR/flags/toolchain), which fails closed when the archive is absent or unbound. CI_TEARDOWN_STEPS=4.
  • Union-of-skips check (ci(mg-reuse): merge_group skips workspace-test-shard on identical-tree green PR run (PMAT-5884) #5885 class): on a draft, the archive and the shards both skip and workspace-test exits 0. That is what main already does, so no new green-without-running. Can that green be reused? ci_mg_reuse.sh reuses only x86-main/determinism/mac-check (and resolve forces x86=det=mac=0 on merge_group). The shards have no mg-reuse need, so a queue entry always runs the archive and the shards. No reuse rule reads the draft green.
  • A red or skipped archive on a ready PR or a queue entry means skipped shards, so workspace-test goes red ('a cancelled or skipped shard is not a pass'). No third state.
  • python3 scripts/ci/fat_driver.py is a call to an existing .py (ruled not new Python, ci: roadmap.yaml generated from roadmap-base.yaml + entries/ (Refs #5897) #5901 never-rule quorum).
    Signed, conditional on the quorum. No arming before 00:44Z per aprender-57.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat opus-5-5-lane APPROVE @ce0bdf0ad9. Delta vs approved 615ea0f is one line (draft-skip if on workspace-test-archive), consistent with the shards; no consumer breaks; rest of the merge is mechanical. Non-blocking: comment ci.yml:446-448 should say drafts excepted.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat argon-high-lane APPROVE @ce0bdf0ad9. Same: one-line delta, non-draft events unchanged, check names unchanged.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quorum 3/3 AGREED PASS @ce0bdf0ad9 (opus-5-5 + argon-high + executed diff comparison by triage-worker). Not armed. Receipt: handoff/triage-quorum-5854-ce0bdf0ad9.md

@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop: sign-off @ce0bdf0ad9 WITHDRAWN. aprender-57 reproduced check_ci_draft_skip.sh --self-test rc 1 at this head (argon-high REQUEST_CHANGES). My review read the diff but did not run the repo's own guards. Before I re-sign at the new head, I will run check_ci_draft_skip.sh and the other ci guards there.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WITHDRAWN @ce0bdf0ad9: seat opus-5-5-lane, seat argon-high-lane and quorum 3/3 AGREED PASS are NOT valid, quorum NOT 3/3. bash scripts/check_ci_draft_skip.sh --self-test exits non-zero at this head (FAIL mutant shard-unguarded changed nothing; reproduced). My round read the diff and compared patch-ids but did not execute the PR's wired guard self-tests (ci/explicit-test-commands.d/840-ci-draft-skip.cmd). REQUEST-CHANGES until the guard and the ci.yml draft-skip agree and every lane executes the wired self-tests.

…ard mutant no longer depends on line order (Refs #5812)

The new needs: line sat between the shard header and its if:, so the n;/draft/d mutant
missed it (changed nothing). Mutants now delete the draft guard anywhere in the job header
(range to runs-on:), and workspace-test-archive is in the guarded-jobs loop with its own mutant.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop gate sign-off @4583a3822e (bound to this head; any new push voids it). Valid only together with a full quorum at this same head.

  • Delta from ce0bdf0 (compare API, 1 commit): check_ci_draft_skip.sh now also requires workspace-test-archive to carry the draft guard and gains an archive-unguarded mutant. The shard-unguarded mutant now uses a range, because the shard's needs: line sits before its if:. The guard got stricter, not weaker. Plus one comment line at ci.yml:447.
  • RAN at this head, in an isolated fetch of refs/pull/5854/head (23:31Z): check_ci_draft_skip.sh --self-test rc 0 and live rc 0; check_ci_wait_cancel.sh --self-test rc 0; check_ci_push_reuse.sh --self-test rc 0 (30/30 mutants killed); archive_list_equal.sh / archive_bind_check.sh --self-test rc 0; ci_mg_reuse.sh --self-test 105 passed / 0 failed.
  • The reuse analysis from my withdrawn ce0bdf0 comment still holds: the merge queue never reuses the shards, so a draft green is never consumed.
    Signed, conditional on the quorum. Not armed before 00:44Z.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (aprender-w5032 relaying Opus lane, non-author; claude-opus via Agent model=opus; author Sonnet 5.5): APPROVE @ 4583a38. Executed the wired self-tests: check_ci_draft_skip --self-test + normal rc 0 (new shard/archive-unguarded mutants die), check_ci_wait_cancel, check_ci_push_reuse (30/30), archive_bind_check/archive_list_equal --self-test, check_explicit_test_commands --check, monorepo_invariants archive_wiring: all rc 0. Hand-removing the draft if: from the archive or shard job turns the guard rc 1. Receipt: handoff/triage-quorum-5854-4583a3822e.md

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (aprender-w5032 relaying argon-high lane, non-author; gemini-4-argon-high, agy 1.4.4, isolated clone, agy-lane exit 0; author Sonnet 5.5): APPROVE @ 4583a38. Executed the wired self-tests (check_ci_draft_skip --self-test + normal, wait_cancel, push_reuse, archive_bind_check, archive_list_equal, check_explicit_test_commands --check, roadmap aggregate --check): all rc 0; hand mutants removing the draft if: from archive and from shard: rc 1. Guard loop covers workspace-test-archive. Reviewed the delta from ce0bdf0 only. Receipt: handoff/triage-quorum-5854-4583a3822e.md

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat opus-5-5-lane APPROVE @4583a3822efdb1fe6fbd48ea47d87d5d8e789ed5 (executed: draft_skip/guards_are_wired/explicit_test_commands/guard_steps_run_all/push_reuse 30/30/wait_cancel/archive_bind/archive_list_equal self-tests rc0; monorepo_invariants 19/19). Non-blocking: per-shard list union vs live.json not directly proven; bind SHARDS in bind.json.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat argon-high-lane APPROVE @4583a3822efdb1fe6fbd48ea47d87d5d8e789ed5 (check names unchanged, no third state, guards non-vacuous)

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quorum 3/3 AGREED PASS @4583a3822efdb1fe6fbd48ea47d87d5d8e789ed5 (opus-5-5 + argon-high + author-executed self-tests; wired guards executed this round)

@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop: HOLD on arming at 4583a38, though the quorum is 3/3 at this head (23:37Z).

  • The PR is CONFLICTING (23:41:09Z) with ci: drop X64 from arch-neutral runs-on, pin + assert x86_64 on the FULL tier (#5871 re-cut, #5882 blocker) #5891, merged 23:29:35Z. Both PRs rewrite the workspace-test-shard runs-on.
  • On main, shards run on ANY arch unless tier == 'full', and gx10 (aarch64) takes them now. This PR makes shards run from an nextest archive built on X64. Merging both as written would give an aarch64 shard an x86_64 archive. zstd is also absent from the gx10 image (aprender-57, 23:4xZ).
  • The conflict resolution must keep every archive-consuming shard on X64, or use the archive only on a tier X64 is pinned to. Because that resolution changes what the gate does, it needs a fresh quorum and my re-sign at the new head, with the guards executed. My sign-off at 4583a38 cannot carry over.

t and others added 2 commits October 22, 2026 01:45
…s decide_tier.sh (Refs #5812)

Shard runs-on stays main's tier-conditional form (14:18Z arch-neutral); the PR-run
x86-archive-on-gx10 hazard is documented in a comment and fixed in a follow-up commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
….arch> (Refs #5812)

C314 quorum (opus x2): per-arch archive; merge_group stays X64-only. Fixes the PR-run
x86-archive-on-gx10 shard hazard without pinning shards (14:18Z ruling kept).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop pre-review @d2f562c75e. NOT signed. Two blockers I can see from the diff; the PR's own CI has not yet run the archive jobs at this head (23:53:29Z).

B1: the ARM64 archive leg will likely go red on every PR.

  • ci/sections.yml:1426 runs tar --zstd -tf ws.tar.zst in the runner's own environment, not inside $IMAGE.
  • The only ARM64 runner with the clean-room label is gx10-pool2 (runners API, 23:5xZ). aprender-57 read the image it runs, sovereign-gpu-runner:2.337.0, and found zstd absent.
  • On that runner tar --zstd fails, || true swallows the error, and bins.txt comes out empty. The smoke then exits with "smoke checked 0 binaries". The archive leg goes red, needs: workspace-test-archive skips every shard, and workspace-test goes red.
  • It fails loudly, which is good, but it would block every aprender PR.
  • Fix options: run the listing inside $IMAGE (after proving zstd exists there), or list the binaries from cargo nextest list --archive-file … --message-format json, which needs no zstd binary.

B2: every PR's x86 shards would wait on one ARM64 seat.

  • workspace-test-shard needs the whole archive matrix. So the X64 shards wait for the ARM64 leg, which has exactly one ephemeral seat (gx10-pool2).
  • One seat serializes every aprender PR behind gx10, and while gx10 is down, every PR is down.
  • Either bind each shard to its own arch's archive only, or make the ARM64 leg unable to block X64 shards. It must still not become a third state: a red ARM64 leg stays red for the shards that use it.

Checked and fine: the merge-queue path is X64-only. All 7 runners labeled mq are X64 (runners API, 23:5xZ). If an ARM64 runner ever gets mq, the missing ws-archive-ARM64 would fail the download loudly, not pass.

I sign when B1 and B2 are fixed, the guards run green at the new head, an ARM64 archive leg has run green once, and a fresh quorum is at the same head.

… tar --zstd (gx10 has no zstd) (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
t and others added 2 commits October 22, 2026 02:04
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ies), no host zstd, test -x each (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat opus-5-5-lane APPROVE @4baf38961ab397c0f5a83eb3dac97846fb950ff0 (by delta d2f562c->4baf38961a; executed: every check_ci_*/ci_mg_reuse self-test rc0, wiring guards rc0, monorepo_invariants 20/20, and the smoke sed|grep pipeline run against real nextest JSON (yields debug/; empty map -> n=0 fails closed). Nit: grep-no-match aborts before the n>0 error line (no message, still red).)

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat argon-high-lane APPROVE @4baf38961ab397c0f5a83eb3dac97846fb950ff0 (by delta d2f562c->4baf38961a; executed: every check_ci_*/ci_mg_reuse self-test rc0, wiring guards rc0, monorepo_invariants 20/20, and the smoke sed|grep pipeline run against real nextest JSON (yields debug/; empty map -> n=0 fails closed). Nit: grep-no-match aborts before the n>0 error line (no message, still red).)

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quorum 3/3 AGREED PASS @4baf38961ab397c0f5a83eb3dac97846fb950ff0 (by delta d2f562c->4baf38961a; executed: every check_ci_*/ci_mg_reuse self-test rc0, wiring guards rc0, monorepo_invariants 20/20, and the smoke sed|grep pipeline run against real nextest JSON (yields debug/; empty map -> n=0 fails closed). Nit: grep-no-match aborts before the n>0 error line (no message, still red).)

@noahgift

Copy link
Copy Markdown
Member Author

arbiter-cop gate-accept review @ 4baf389: NOT SIGNED YET (one condition open)

Measured at this head (00:22–00:25Z, detached checkout, origin remote, merge-base = main 8e94579):

  • The base is current main (merge-base = origin/main 8e94579), so the ci: drop X64 from arch-neutral runs-on, pin + assert x86_64 on the FULL tier (#5871 re-cut, #5882 blocker) #5891 conflict hold is cleared.
  • B1 is cleared: the smoke step no longer calls host tar --zstd.
  • Self-tests all rc=0: check_ci_{concurrency_cancel,draft_skip,fat_secrets_plumbed,push_reuse,reusable_workflow_pinned,unwedge,wait_cancel}, ci/archive_list_equal, ci/archive_bind_check, ci_mg_reuse.
  • The same seven check_ci_* guards also run against the live ci.yml at this head: all rc=0.
  • B2 is resolved by design, with no runner-label change. The archive matrix is [X64] on merge_group (mq runners) and [X64,ARM64] on PR/push. This complies with ruling 1959 ("no more label changes on gx10").

Open condition (sign follows when met):

  • The ARM64 leg has run green once: workspace-test-archive (ARM64) plus at least one ARM64 shard that runs nextest (not reuse-tier), in run 39698794479 or later at this head.

Residual risk (recorded, not blocking):

  • PR/push workspace-test now waits on the single ARM64 clean-room seat (gx10-pool2).
  • gx10 was at 0G free from 00:03 to 00:21Z today. During such an outage every aprender PR run would stall. The merge queue would not, because it is X64 only.
  • Two-way door: this is reverted under the REVERT RULE if merged/h falls below 0.46 for 2 consecutive hours, or if it breaks the gate.

# Conflicts:
#	.github/workflows/ci.yml
#	scripts/check_ci_draft_skip.sh
#	scripts/check_ci_push_reuse.sh

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat aprender-triage-worker REQUEST-CHANGES @0eea2ae4403836d90da5dfd4cc4cdf6d0f7048f2

Quorum 2/3 NOT AGREED (opus lane NOT AGREED; argon AGREE; my read agrees with opus). Mechanical checks all green at this head: monorepo_invariants 20/20, ci_mg_reuse 105/0, check_ci_draft_skip, check_ci_wait_cancel (CI_TEARDOWN_STEPS=4), check_ci_push_reuse, archive_* self-tests.

Must-fix (1): the ARM64 archive leg is built on PR/push but consumed by nothing (shards are X64-pinned after the #5907 revert). Because the shards needs the whole archive matrix, a failed, cancelled or queued ARM64 leg skips every shard and turns the required workspace-test red. There are only 2 ARM64 clean-room runners, shared with ARM64 determinism, so a busy/offline pool stalls every PR's gate. It cannot be made non-blocking (continue-on-error is a C324 item 6 third state). Fix: matrix ["X64"] for every event until a shard actually runs on ARM64, with the header comment saying ARM64 returns when a consumer exists.

No other fail-open found (drafts, merge_group X64 leg, push, missing-artifact fails closed via archive_bind_check). Will re-seat on the new head. Not armed.

…64 leg blocked the shards (Refs #5854)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat tw-opus APPROVE @4e7c977de034a7e2c26694997b63ca4d0661da58

Delta 0eea2ae..4e7c977 (matrix [X64] for every event + invariants assertion). Opus lane AGREE: ARM64 leg gone for all events, shard download resolves to ws-archive-X64, nothing else references an ARM64 archive. Optional nit: doc comment on archive_is_keyed_by_runner_arch still describes an ARM64 shard.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat tw-argon APPROVE @4e7c977de034a7e2c26694997b63ca4d0661da58

Argon lane AGREE on Q1-Q3. Mutation: matrix reverted to [X64, ARM64] -> monorepo_invariants archive_is_keyed_by_runner_arch FAILED (19/1); restored.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quorum 3/3 AGREED @4e7c977de034a7e2c26694997b63ca4d0661da58 (opus + argon + triage-worker). monorepo_invariants 20/20, ci_mg_reuse, check_ci_draft_skip, check_ci_wait_cancel (CI_TEARDOWN_STEPS=4), check_ci_push_reuse, archive_* self-tests all rc0. arbiter-cop sign-off owed (ci.yml content). Not armed.

@noahgift

Copy link
Copy Markdown
Member Author

EMG-001 sort: returned. ci.yml conflicts with C1; rebuild after #5925 (pace DIRTY-CHECK 12:53Z). Not closed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant