Skip to content

EPIC E1 0.71: Fast Train #3998

Open
26 of 64 issues completed
Open

Description

@noahgift

owner: aprender-wmainred (re-owned 2026-10-09 by clerk aprender-3e for cop aprender-13: works children #4687 #4715 #4960 (most child work on 2026-10-09))

Spec: APR-EPIC-001 §2 · Epic: E1 · Train: 0.71 · Successor at cut: S1

Theme

Fast Train

done_when

Operator ruling C334 (2026-10-08 10:45Z), train 0.71:

  1. C332 a, rehearsal. Every night on main's head, the release-day script and policy file run everything except the tag push and the uploads. Green = exit 0, every lane measured on that night's commit, no rerun. The 3 nights before release day are green; a red night resets the count. Receipt required.
  2. C332 b, the script releases. One unattended pass of scripts/release/autopilot.sh from the bump to live: 0 steps typed outside it, 0 emergency scopes, 0 commits on the release branch after the bump, 0 gate-script edits on release day. Receipt required.
  3. C332 c, receipts. Two planted tests pass inside the rehearsal: a GitHub read that fails twice and then answers costs under 2 min and no lane rerun; a notes-only commit is ready to tag in under 10 min on saved results. Receipt required.
  4. C333: the checks that can stop a release number 20 or fewer (merge 10, tag 5, publish 5), listed in code.
  5. Freeze to publish in 4 h or less.

Every build ticket goes to E1 (C334 item 1).

done_when before C334 (kept, not deleted)

freeze→publish ≤ 4 h [O]; vX.Y.Z-rc.N on every green merge to release/*; rc→final assets sha256-equal; nightly publishes every [[bin]] plus a CUDA apr; every fleet host runs the newest verified build, or andon.

Budget

At most 10 open children (docs/roadmaps/epics.yaml). Repo-wide cap 100 open issues; andon at 95.

Build-bar rows (C334 item 3)

Filed as checklist rows, not issues: the ticket tree is over 100 (ISSUE-TREE-001).

  • C332-a: nightly release rehearsal on main's head. Same script and policy file as release day; everything except the tag push and the uploads; green = exit 0, every lane measured on that night's commit, no rerun; 3 green nights before release day, and a red night resets the count (evidence: nightly rehearsal receipts; related PR feat(release): nightly assets rehearsal — build every release asset on a schedule, unable to upload (#4806) #4813)
  • C332-c: receipts with reads first. Planted tests inside the rehearsal: a GitHub read that fails twice and then answers costs under 2 min and no lane rerun; a notes-only commit is ready to tag in under 10 min on saved results. Includes C330 item 6: reads run first, the 10-min re-ask applies to reads only, and the coverage lookup by commit takes the newest run that holds a receipt (evidence: planted-test receipts)

From #4678 (C334 item 2)

#4678's 37 open children, ranked by APR-EPIC-001 rule 8. Plan receipt, with the quorum record: docs/audits/apr-epic-001/plan-C334-E1.json; write log: applied-C334-E1.json.

Absorbs

#4033, #4288, #4232, #3081, #3058

Prior body

Theme: Fast Train (operator decision 2026-09-23: "make .70 fast train")

A release goes from freeze to publish in hours, not a day, and every later release inherits that speed. 0.71's "Don't Leave Behind" (#3994) runs a 4-backend matrix and will need several fix-and-re-sweep cycles, so speed comes first.

Exit bar. Freeze to publish ≤ 4 h, measured on 0.70 itself. The lock's idle-GPU fraction falls by more than half, per paiml/infra#1146's method re-measured.

Must-carry

Area Items
GPU lock limited to GPU work paiml/infra#1146 P1: build outside the lock, only CUDA tests under it; gpu-q refuses a bare cargo test/cargo build (via infra/forjar)
CPU work off the lock paiml/infra#1146 P3: ladder CPU legs and hashing unlocked
Parallel hosts paiml/infra#1146 P5: idle same-arch pre-screen as a standard pre-freeze step; the sweep shards across lambda, gx10, yoga and the Mac host
VRAM-budget sharing (pilot) paiml/infra#1146 P4: correctness jobs only; benchmarks stay exclusive. Separate processes, not MPS (MPS lets one client's fault kill others)
Sweep hygiene one hash pass per sweep; the serve probe keeps its own evidence (#3949); the serve health wait never counts lock-queue time (the gx10 q4k.apr false-RED shape, 2026-09-23)
Dark test targets into CI #3984, #3982
Fail-fast release gates #3892 (publish cascade retry), #3974 (WARN/SKIP exit 0)
0.69.1 known REDs (no defer) #3951, #3987, low-bit admission #3963/#3953/#3960, subject to operator confirmation
Ratchet slice 1 of 5 #3997

Out: new backends and silicon (0.71) and serve/agentic features.

Parked

Closed over budget (APR-EPIC-001 rule 5). Reopen = pull (rule 7).

Superseded epics

Activity

  1. added this to the 0.70.0 milestone on Sep 23, 2026
  2. noahgift commented on Sep 23, 2026

    @noahgift
    ContributorAuthor

    Plan posted: PR #4010

    Rows FT-1 to FT-12 each have a done_when, a measured baseline and a first-green proof. aprender-19 wrote the draft; aprender-cb added measurements and ran the decision quorum. Quorum: 3 lanes, all gemini (the gpt pool is 429 until about 09-27), so the result is advisory. Nothing is applied: no child issues and no milestone moves. The PR is not armed.

  3. noahgift commented on Sep 23, 2026

    @noahgift
    ContributorAuthor

    Milestone triage PROPOSAL: 0.70.0 + unmilestoned (nothing moved)

    Table: PR #4024, docs/roadmaps/milestone-triage-2026-09-23.md. It covers 314 open issues (178 in 0.70.0, 136 with no milestone), each with a proposed milestone and a one-line reason. Moves need the operator's approval; nothing has been moved.

    0.70.0 today (178), proposed destinations:

    Proposed Count
    stays in 0.70 77
    0.71 44
    0.72 15
    0.73 16
    0.74 17
    0.75 5
    close? 4

    So 0.70 would shrink from 178 to 77, and what remains is release/CI/gate/lock/sweep work plus debt-ratchet slice 1 (coverage).

    No milestone today (136), proposed destinations:

    Proposed Count
    0.70 32
    0.71 46
    0.72 10
    0.73 11
    0.74 5
    0.75 1
    0.69.1 in flight (closes at the tag) 13
    verify-close 6
    close? 3
    DECIDE 8
    pinned thread 1

    Decisions for the operator (named by more than one epic):

    Findings:

    Prepared by aprender-36 for aprender-cf (cop).

  4. noahgift commented on Sep 23, 2026

    @noahgift
    ContributorAuthor

    Milestone triage APPLIED (#4024): 202 moved, 35 still pending

    Operator approval 2026-09-23, relayed by aprender-cf. 202 issues were moved to the milestone #4024 proposes (97 out of 0.70.0, 105 from no milestone), each with a comment naming the table and its reason. No issue was closed.

    Open issues per milestone now:

    Milestone Open issues
    0.70.0 126
    0.71.0 106
    0.72.0 55
    0.73.0 32
    0.74.0 23
    0.75.0 7

    0.70.0 now holds 126:

    The milestone API's 140 also counts pull requests.

    Still pending (untouched): 35

    Operator decision (two or more epics name it): 8

    Close candidates (stale or superseded): 7

    Verify and close (the release tree implements it): 6

    0.69.1 in flight (closes at the tag, or carries): 13

    Pinned thread: 1

  5. noahgift commented on Sep 23, 2026

    @noahgift
    ContributorAuthor

    Correction to the applied triage (#4024), after its quorum (Opus 5.5 lane 1, FAIL, findings accepted).

    Three applied moves broke the table's own rule that an issue named by two epics is an operator decision:

    They are reverted to their pre-move state (#3421 and #3428 back to 0.70.0; #3997 back to no milestone), each with a comment. They are now pending decisions:

    #3951 (must-carry in #3998 and #3994) is also a decision now: 0.70 or 0.71. It is likely resolved by #3990, so re-measure it first.

    The two comments that had failed (#3917, #3532) are posted. #3483's 0.71 placement contradicts the table's own rule for FP8/batched paths (0.73); it is flagged for your re-check and was not re-moved.

    Net moves: 199. Still pending: 38 (12 DECIDE, 7 close?, 6 verify-close, 12 in flight, 1 pinned). Nothing was closed. The doc is updated on #4024 @ 5e27b35.

  6. 142 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicEpic — multi-story umbrellaowner:build-kaizenowner session build-kaizen

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions