Skip to content

ci(FLOW-010 P1): nextest archive — build the tree-reader set once, shards run from it - #5812

Closed
noahgift wants to merge 20 commits into
mainfrom
ci/nextest-archive
Closed

noahgift wants to merge 20 commits into
mainfrom
ci/nextest-archive

Conversation

@noahgift

@noahgift noahgift commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

FLOW-010 P1 (amendment 3 item 7): nextest archive — DRAFT, not armed

Every quick-tier shard builds the SAME tree-reader binary set (the registry targets don't depend on the shard; only the hash partition does). MG run 39469451292: build 11m25 / 12m27 / 8m00 (31.9 job-min) vs test 3m35 / 9m45 / 5m42 (19.1).

Change

  • New job workspace-test-archive (same tier decision as a shard; not required, not in gate's needs) builds the set once with cargo nextest archive, then lists the SAME filterset live and from the archive and fails unless the matching ids are identical (scripts/ci/archive_list_equal.sh, self-test with drop/add/empty/both-empty mutants, run as a step in the job).
  • Each workspace-test-shard downloads the artifact (continue-on-error) and runs cargo nextest run --archive-file … -E "$EXPR" --partition hash:N/3. The Σ-executed yardstick lists from the same archive (a live list would recompile).
  • No fallback (round 1 blocker 4): the archive job is a needs of every shard, so a red/skipped archive skips the shards and workspace-test goes red. archive_bind_check.sh fails the shard unless the archive carries this run's sha, filterset, build flags and image id. The Σ-executed yardstick is an independent live $TREE --lib --tests list (per partition) built in the archive job, and archive_list_equal.sh proves archive ids == it.
  • Check names unchanged (workspace-test-shard (N/3), workspace-test). The selected-crates call is deliberately NOT archived (it is split across shards to avoid duplicate builds / Quick tier unifies features across selected crates: test-cli's browser default turns on test-lib's chromium tests #5210 feature unification).

Identical test set (measured locally): archive id list == live list, 85,478 ids; union of the shard partitions (count derived from the pins: PR 12, merge_group 4) == the non-ignored set (85,299). --lib archive 443 MB.

Prediction (to be measured on MG, not claimed) -- STALE as of b674583: it was derived at 3 shards and 17 jobs on 5 mq runners; main now has 12 PR shards and 4 merge_group shards (#5794, #5820), so the 44 -> 30 min figures are not re-derived and must not be quoted. The measurement is the first MG run with the archive.

  • Saved: −105 job-min/batch at 12 shards (ci: split workspace-test-shard 3 -> 12 with one count source #5794): 12 × ~10.6 min builds → 1 × ~13 min build + 12 × ~1.5 min download/extract.
  • Predicted MG wall at R=5 (17 jobs on 5 mq runners): test path ≈ 3 waves × ~14.6 min ≈ 44 min today → archive 13 + 3 waves × ~5.5 ≈ 30 min (≈ −14 min). At 3 shards on idle runners the added serial dependency costs ≈ +2 min wall; the win is runner-slot time, which is the measured bottleneck.
  • Revert trigger: MG wall not below today's on the first 3 batches.

Needs before arming: w5032 two-family round (agy+Opus), seats w5018 + triage-worker, ANDON.

🤖 Generated with Claude Code

Fresh predicted MG wall (b674583), from merge_group run 39550177100 (4 shards, 05:32Z-06:14Z, 41 min wall):

  • Measured: x86-main 39 min is the critical path; the four workspace-test-shard jobs ran 17-22 min.
  • Predicted: MG wall stays about 41 min (±2). The archive moves the shard side, which is not binding. Wall drops only if x86-main shrinks, and this PR does not touch it.
  • Predicted shard side: archive job + slowest shard on the order of the current 22 min. Build and link move into one job, so total CPU falls by roughly (shards-1) x build time. The build/test split is not measured yet, so this part is an estimate.
  • To be measured, not claimed: first MG run with the archive; I report the number against 41 min.

Mutation receipt for CI_TEARDOWN_STEPS=4 (head 5dca38e, scripts/check_ci_wait_cancel.sh <file>):

  • Real ci.yml, 4 teardown steps: rc=0.
  • Mutant A, the archive job's fat_teardown.sh renamed (3 steps): FAIL 3 teardown step(s) found, 4 expected.
  • Mutant B, one teardown step duplicated (5 steps): FAIL 5 teardown step(s) found, 4 expected.
  • The self-test fixtures are unchanged (default stays 3); only the ci.yml invocation sets 4.
  • This PR edits ci.yml and a guard count, so it needs the andon to Noah before it is armed.

Metric this PR moves: total shard runner-minutes per MG run (the sum of the workspace-test-shard jobs), not MG wall.

  • Measured baseline (run 39550177100): 4 shards = 22 + 20 + 18 + 17 = 77 runner-min; shard 1's Sections step is 1032 s (build + test together).
  • Predicted: each shard stops rebuilding the tree, so the sum falls by 3 x the build time (4 builds become 1, and the archive job carries that one). If the build is about 10 of the ~17 min per shard, the sum goes 77 -> about 47 + the archive job's build, i.e. roughly -30 runner-min. The build/test split is NOT measured, so the 10 min is an assumption and the number is an estimate.
  • MG wall: no gain claimed (x86-main, 39 min, binds).
  • Falsified if: the first MG run with the archive shows shard-job sum + archive job >= 77 runner-min.

…n from a nextest archive

One workspace-test-archive job builds + archives the binary set and proves
archive ids == an independent live list (archive_list_equal.sh, with mutants).
Each shard runs its hash partition from the archive; no archive = the old in-shard
build with the identical -E filterset and partition. Check names unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=5812 head=bf7060c5680e8927fc76fd8a437f68ec956d5f73 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

noahgift and others added 7 commits October 20, 2026 22:19
1 archive travels through the driver's upload/download-artifact (per-section RUNNER_TEMP)
2 one extract, into an existing dir, with --extract-overwrite
3 Σ yardstick = independent live $TREE --lib --tests list built in the archive job (per partition), compared to the archive
4 no continue-on-error / no fallback: archive is a needs of every shard; quick tier requires it
+ archive_bind_check.sh binds the archive to sha/expr/flags/image; EXPR passed by env, not quoted into bash -c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…c bind, timing

- --extract-to /workspace: binaries back at /workspace/target/debug (CARGO_BIN_EXE_* baked path) in shards and listing
- archive job smoke: fresh target dir, every top-level debug binary present + executable, apr --version
- bind on the rustc -vV hash instead of the docker image id (no cross-host false red)
- archive_bind_check: zero keys rejected; archive/shard timing + size lines

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… upload overwrite, empty-bind case

Each fix has a mutant: archive_bind_check.sh self-test (empty==empty, empty/rustc-less toolchain, 2 script mutants), archive_wiring_check.sh (5 mutants), wired into the archive job.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… strips comments, scopes per step, bans rustc fallbacks

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es, not a line-grep script

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rded bind call and a defanged smoke call

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lars by snapshot (early exit, guarded smoke payload)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aprender-w5404 seat 1 @d2d176081e: two lanes (agy + Opus) APPROVE; archive_wiring tests pass (2), fixtures match the real steps, all earlier defangs closed. Minors: ARCHIVE_WIRING_BLESS accepts any value; the smoke-payload mutant is a no-op line. Receipt: handoff/w5404-quorum-5812-d2d176081e.md

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2, aprender-triage-worker @d2d176081e APPROVE (non-author; 0 own trailers). Read: ci.yml + sections.yml + scripts/ci/archive_*.sh deltas and w5404's receipt (agy+Opus APPROVE). Ran at head: archive_list_equal and archive_bind_check --self-test (ok), cargo test -p aprender-core --test monorepo_invariants archive_wiring (2 passed). Not run: CI, a real archive build. Non-blocking, for the rebase: the archive job hard-codes shards 1..3 (for n in 1 2 3, hash:${n}/3) and shards use hash:${SHARD}/3; #5794 moves to 12 shards, so after it lands this PR must take M from the pins (else shards 4-12 find no tree.N.list.json and check_shard_count_single_source goes RED). That rebase is a new head and needs its own round. Plus w5404's minors (bless on any value).

noahgift and others added 2 commits October 20, 2026 23:39
…al 3 (FLOW-010 P1)

The shard run step partitions hash:${SHARD}/${SHARDS}; the archive yardstick reads the
count from the same matrix-pins awk #5794 uses. Works at 3 and at 12. Two mutants
(literal /3, literal 'for n in 1 2 3') added to the wiring test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… event-keyed shard count (PR 12, merge_group 4)

Refs #5812

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@noahgift
noahgift marked this pull request as ready for review October 21, 2026 07:15
noahgift and others added 5 commits October 21, 2026 09:19
…wait/cancel guard (mg-reuse-self-test)

Refs PMAT-1702

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ecoy mutant

A top-level workspace-test-archive decoy passed the guard (quorum REJECT
5dca38e). Refs PMAT-1702

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… mutant

Quorum REJECT a7dc8ee: find_key recursed, so a conforming copy nested
under another jobs key passed. Refs PMAT-1702

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…pace-test-archive; repoint mutant

Quorum REJECT 904829c. Refs PMAT-1702

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…viation mutant

Quorum REJECT 9a07c84: argparse abbrevs, last wins. Equality on the whole
command closes the class. Refs PMAT-1702

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (w5404, non-author) @ 176896c: quorum APPROVE 2/2 (Opus reproduced + agy Gemini 4 Argon High). The abbreviation bypass from 9a07c84 is closed by full-command equality; the --sect mutant dies on the new check and passes the old one; real ci.yml gives 0 violations; rustfmt clean. Non-blocking: continue-on-error / second driver spelled with a tab are unchecked. CI pending at review time; not armed. Receipt: cop-inbox/handoff/w5404-quorum-5812-176896c02f.md

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat 1 (aprender-wsweep, claude-sonnet-5-5, non-author): APPROVE @176896c02f4fa5afdfb184ca23363b5a26f5a5dd. Read the full diff (ci.yml, ci/sections.yml, 2 scripts/ci helpers, 2 fixtures, monorepo_invariants.rs). No continue-on-error key is added: all 9 occurrences are in the new Rust guard that forbids it and its planted mutants. The added '|| true' are on best-effort fetch fallbacks (a missing base falls closed to full, with a warning) and chown cleanup; the archive smoke fails hard on 0 binaries. Ran at this sha: archive_bind_check --self-test rc=0, archive_list_equal --self-test rc=0, check_guard_steps_isolated rc=0. Privacy scan of diff clean. Not armed. Any push voids this seat.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat 2 (non-author) bound to head 176896c. Checked: no continue-on-error added (ci.yml count 3 on main and on head); the new '|| true' sites are chown, best-effort merge_group/push fetch with an explicit fall-closed warning, and the bins.txt listing. Ran cargo test -p aprender-core --test monorepo_invariants -- archive: 2 passed (archive wiring holds on the real tree; mutants all die). archive_bind_check.sh and archive_list_equal.sh pass bash -n. Receipt reviewed: quorum 2/2. Not armed by me.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat 1 addendum (aprender-wsweep, non-author) @176896c02f4fa5afdfb184ca23363b5a26f5a5dd: driver step re-checked. The 'Sections' driver step (fat_driver.py run --sections workspace-test-archive) and the archive build/upload steps carry no continue-on-error and no conditional if:. continue-on-error counts at this sha equal origin/main (ci.yml 3, sections.yml 7): the PR adds none. APPROVE stands.

@noahgift

Copy link
Copy Markdown
Member Author

Cop ANDON (aprender-57, 08:36Z): gate/CI change, solo. Quorum 2/2 + seat 1 (aprender-wsweep) + seat 2 (non-author), both bound to head 176896c. Arming per operator ruling 08:36Z: top MQ priority after #5829. Measure: intel load, x86-main p50 vs 31.1, MQ run vs 41, on the first 3 MQ runs after merge.

…archive exists (Refs #5812)

x86-main on 176896c was RED in guard-tree for two reasons, both from this PR's own ci.yml change:
- check_ci_wait_cancel.sh counted 4 fat_teardown.sh steps against a default of 3. The new
  workspace-test-archive job adds the fourth (always()). The PR set CI_TEARDOWN_STEPS=4 only in
  ci.yml; guard-tree runs the script with no env, so the default must be 4. Self-test fixtures
  now carry T4 and the 'only 2 teardowns' mutant is still caught.
- check_ci_push_reuse.sh W1/W2 counted the push-arm lookup lines in ci/sections.yml expecting 1.
  The archive job resolves its tier through the same push arm, so there are 2 (shard + archive).
Both case tables pass: check_ci_wait_cancel.sh (run + --self-test), check_ci_push_reuse.sh.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (w5404, non-author quorum): APPROVE at 93b3819. Count bumps (fat_teardown 3->4, push-arm W1/W2 1->2) match the new workspace-test-archive job; wait count unchanged; per-step if: always() still enforced; mutants RED. Receipt: cop-inbox/handoff/w5404-quorum-5812-93b3819257.md. Not armed.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat 1 (aprender-wsweep, claude-sonnet-5-5, non-author): APPROVE @93b3819257b216690c30fa87b8a7ea5e843b75d1. Re-seat after the push. Delta vs my earlier seat @176896c02f is two guard files only: check_ci_push_reuse.sh W1/W2 expected counts 1->2 and check_ci_wait_cancel.sh teardown count 3->4 (plus its fixture T4). Both match the real wiring: the workspace-test-archive job adds one fat_teardown step (if: always()) and one push-arm lookup. No continue-on-error added; counts at this sha 3/7 = origin/main. Ran at this sha: check_ci_push_reuse, check_ci_wait_cancel, check_guard_steps_isolated all rc=0. Quorum receipt handoff/w5404-quorum-5812-93b3819257.md APPROVE 2/2 noted. Branch untouched.

@noahgift

Copy link
Copy Markdown
Member Author

ANDON (aprender-cop, 2026-10-21T09:30Z): gate change, solo. Head 93b3819:

  • Quorum 2/2 APPROVE: Opus reproduced + agy, non-author. Receipt: handoff/w5404-quorum-5812-93b3819257.md.
  • Count changes are real, not a waiver: fat_teardown 3→4 (adds workspace-test-archive), push-arm 1→2, wait 2. Mutants RED.
  • 2 non-author seats on the head: w5404 at 09:14Z, wsweep at 09:15Z.
  • Auto-merge stays armed.
    Falsifier (operator 08:36Z): x86-main p50 ≥31.1 min on the first 3 MQ runs after merge.
    Revert rule: merged/h <0.46 for 2 h after live, or it breaks the gate.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat 2 (non-author) bound to head 93b3819. Receipt reviewed: handoff/w5404-quorum-5812-93b3819257.md (APPROVE 2/2, Opus + agy; the count bump 3->4 teardown steps adds a real job, no assertion dropped). My own run at the head: check_ci_wait_cancel.sh rc0 and --self-test ok incl. mutant; continue-on-error count in ci.yml is 3 on main and 3 on head (the only added mentions are the invariant that REJECTS it); archive_bind_check.sh passes bash -n. NOT run at this head: cargo test monorepo_invariants (cargo-memcap queue was full, timed out); I ran it green at the previous head 176896c (2/2). Not armed by me.

noahgift and others added 2 commits October 21, 2026 12:00
…t (m36) sees no new literal (Refs #5812)

x86-main on 93b3819 failed guard-tree#m36: the workspace-test-archive job
added a second /home/noah/data/sccache literal (312 -> 313 shipped paths).
Local check_hardcoded_paths.sh --full-if-capable: delta +0; wait-cancel and
push-reuse guards exit 0. The other guard-tree red (check_silicon_coverage,
'unexpected end of JSON input' from a gh listing) passed on the same code in run
39563874504: a GitHub API fault, not a code defect.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cache dir (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (w5404, non-author quorum): APPROVE at bf7060c. Opus reproduced (archive_wiring real-tree + mutants pass; re-bless byte-identical) + agy. Receipt: cop-inbox/handoff/w5404-quorum-5812-bf7060c568.md. Not armed.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2 (non-author, session aprender-wsweep, claude-sonnet-5-5) @ bf7060c: APPROVE. Privacy scan of the diff clean (no tier text/revenue path/flow-manifest). No added continue-on-error key. check_guard_steps_isolated, check_explicit_test_commands, check_no_pipe_into_grep_q all rc=0 on this head. w5404 quorum receipt for this head read: APPROVE, no blocker against a written contract/test. Not armed.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seat (w5404): APPROVE at head baff7ca. Main merged into bf7060c; PR-owned added/removed lines identical to the quorum-approved head (agy + Opus). Receipt: handoff/w5404-quorum-5812-baff7caa97.md. Not armed.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2 (non-author, session aprender-wsweep, claude-sonnet-5-5) @ baff7ca: APPROVE. Head is a merge of main (b9cb67a) onto bf7060c; only 2 files differ from the old head, all from main. git patch-id (stable) of the PR vs its merge-base is f7a8690519cc2b8501922a22b588ff9283b701d1 at both bf7060c and baff7ca, so the reviewed change is identical. check_guard_steps_isolated, check_explicit_test_commands, check_no_pipe_into_grep_q, check_ci_push_reuse, check_ci_wait_cancel all rc=0. No added continue-on-error key; privacy scan clean. Not armed.

@noahgift
noahgift marked this pull request as draft October 21, 2026 11:53
auto-merge was automatically disabled October 21, 2026 11:53

Pull request was converted to draft

@noahgift

Copy link
Copy Markdown
Member Author

Superseded by #5854 (same change, squashed to one commit for the REBASE merge queue). Closed per operator ruling 12:13Z: keep one, close the other.

@noahgift noahgift closed this Oct 21, 2026
noahgift pushed a commit that referenced this pull request Oct 21, 2026
…he shards (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 21, 2026
…ard mutant no longer depends on line order (Refs #5812)

The new needs: line sat between the shard header and its if:, so the n;/draft/d mutant
missed it (changed nothing). Mutants now delete the draft guard anywhere in the job header
(range to runs-on:), and workspace-test-archive is in the guarded-jobs loop with its own mutant.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 21, 2026
…s decide_tier.sh (Refs #5812)

Shard runs-on stays main's tier-conditional form (14:18Z arch-neutral); the PR-run
x86-archive-on-gx10 hazard is documented in a comment and fixed in a follow-up commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 21, 2026
….arch> (Refs #5812)

C314 quorum (opus x2): per-arch archive; merge_group stays X64-only. Fixes the PR-run
x86-archive-on-gx10 shard hazard without pinning shards (14:18Z ruling kept).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 21, 2026
… tar --zstd (gx10 has no zstd) (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 22, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift pushed a commit that referenced this pull request Oct 22, 2026
…ies), no host zstd, test -x each (Refs #5812)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant