Skip to content

EPIC: release train 0.69.0 — COMPLETE Qwen usability on pure CUDA (all 0.68.x folded in) #3080

Closed

Description

@noahgift

Release train 0.69.0 — all remaining 0.68.x work folded in. Headline: COMPLETE usability of Qwen models on pure CUDA.

Spec of record: docs/specifications/06x-release-schedule.md §3 (0.69.0); §2 holds each row's acceptance command.
Rebuilt 2026-09-20 on the operator's instruction to merge 0.68.x into 0.69.0 after 0.68.2 shipped. Prior scope preserved below as 69-*; new rows carry issue numbers.
Window: re-dated, not renumbered — due 2026-09-23 (0.70 → 09-26, 0.71 → 09-29, 0.72 → 10-02, 0.73 → 10-05). 0.68.2 consumed the original 09-17→09-18 window as an interrupt release.


Governed by 30-day-plan.md v2 (2026-09-20). v1 was returned do-not-implement-as-written by a 3/3 adversarial quorum; v2's doctrine is what this train runs under. The changes that affect 0.69 are in a comment below.

0.69 purpose: the tools stop lying. Excluded (predicates, labels now exist): label:arch-model-new · label:ont-depth · label:viz · label:perf-claim -label:has-receipt. A move out of this milestone cites a predicate or is logged as an override.

Two gates can refuse T-0: (1) must-carry slice M — #3452 (ruled) → #3420 → #3430 → #3447 → #3446; (2) the tag path under review — #3459/#3461, "nothing beside it" in days 1–3. Fix-or-refuse is how we ship, never how we measure: every goal needs a floor on the fix side plus a positive control.

⚠ Correction (2026-09-20, after review): Band 5 was wrong — slice M is MUST-CARRY

An earlier revision of this epic put PP-QUANT-001 M1 (#3430) in a "root cause, drop first" band. That was an error, and it inverted the critical path.

06x-release-schedule.md §1.1a names PP-QUANT-001 slice M as the 0.69.0 must-carry row, and APR-RELEASE-001 §0 row 1a puts a §1.5 release-critical ticket above every other row — T-0 waits on it, and row 1 will not run the train until "§1.5 scope for that tag is merged". A must-carry row is not droppable by definition; it is the thing the tag is refused for.

Lane A is therefore the critical path and it is idle, blocked on one decision: #3452 (the M1 plan) has been through four quorum rounds — 0/3, 1/3, 1/3, 1/3 — and is awaiting a ruling. The chain is #3452 (ruling) → #3420 → #3430 → #3447 → #3446, with epic #3421. No amount of tidying elsewhere cuts T-0 while that ruling is outstanding.

Process defect this exposes, worth its own row: two consecutive non-passing plan quorums on a must-carry ticket should escalate automatically. #3452 sat through four.


Three co-equal release goals (operator, 2026-09-20)

0.69 has three blocking goals, not one headline with extras. They are stated as equal because the operator stated them as equal.

Goal Issue What it means
A · Qwen3.5 usable on pure CUDA #3519 tool calls, basic fine-tune, automation — a committed receipt per (model, host) on both GPU hosts
A′ · First-party dogfood at a Pareto-optimal size #3558 arbiter, RAH and apex dogfood Qwen3.5 from here on; we publish the selection rule and its evidence — smallest size that measurably works, not the largest that fits. Internal consumers' own details are theirs and are not reproduced in this public repo
B · Ontology to 80%, SHACL first #3559 ≈7 more ONT-001 rows (41% → 80%), every shape armed with a red-turning falsifier. Explicitly equal in priority to A/A′
C · Docs, 1037 examples and the cookbook SHACL-validated and Qwen3.5-current #3560 measured today: 442 book pages, 1037 example targets, 6 artifacts mention Qwen3.5. Downstream of B's extractors

One conflict the team must resolve before B is committed: APR-RELEASE-001 §11 sets ontology cadence at one row per train. Seven rows is seven trains' worth. #3559 states the three options; my recommendation is a dated §11 amendment that keeps the arming discipline (rows may batch; a shape still may not arm without a falsifier), because the cadence exists to stop unfalsifiable gates landing in bulk and the falsifier requirement is what actually controls that risk.


Where we start from: 0.68.2 shipped 2026-09-20

74/74 crates on crates.io at 07:59:49Z, tag v0.68.2 @ 338f1d49b, 16 assets, clean-room green. Qwen3.5 verified on the published artifact (cargo install aprender --version 0.68.2 --features cuda): 0.8B/2B/4B/9B/27B × {CPU, CUDA} on lambda (RTX 4090, sm_89, x86_64) and gx10 (GB10, sm_121, aarch64) — 20/20 cells correct, every GPU arm reporting the real device. Details in #3477, #3038.

So 0.69 is not about making Qwen run. It is about making it usable, and about the fact that four of our own gates were caught lying while we proved the above.

Headline (#3519)

Operator, 2026-09-19: "The goal of this release is to ensure we have COMPLETE usability of: tool calls, fine-tuning (basic), automation, etc of qwen models that work with pure CUDA."

"Usable" is defined as measured, not claimed: a committed receipt showing the capability working on both ladder hosts (lambda sm_89, gx10 sm_121) on the GPU, per (model, host). 0.68.3 is folded here and closed as a milestone — two trains bidding for the same two GPU hosts to build the same value is muda.


Scope, in the order it must be worked

Band 1 — Andon: gates of ours that were measured lying (do these first, they are cheap)

# Item Acceptance
#3543 post-publish dogfood cannot pass — version-unpublished is not phase-aware, banner says "pre-release" in a post-publish run dogfood.sh --phase post-publish GO at a published version and RED at an unpublished one — both directions
#3544 a DEFER nothing discharges — no evidence/dogfood/<v>/<host>.json since 0.67.0, three releases owed. This is row 69-A1 check_multiplatform_dogfood.sh exit 0 against receipts the train itself committed
#2730 cbtop reports PASS / ci_result: green over its own grade F force grade F, assert non-zero exit and status: FAIL
#3170 apr-agent --help created a worktree and a branch — any flag is accepted as a task slug apr-agent --help exits 0, git worktree list and git branch unchanged
#3541 apr run --no-gpu prints no Backend: line, so the CPU arm is intent, not measurement every arm prints Backend: naming the path actually dispatched
#3561 P0 the T-0 preflight returns GO when the dogfood crashes before emitting a row — fails == 0 because there are no rows at all, on the gate that guards publishing point it at a tree where dogfood exits early; the verdict must be NO-GO (today: GO … dogfood_rc=2)
#3545 P0 apr devices reports cuda unavailable reason=NotCompiled on a binary that demonstrably runs CUDA — the first screen a new user sees says our CUDA support does not exist devices --json agrees with what run --gpu actually dispatches, on a -cuda and a -cpu asset, with a RED-turning test
#3542 apr run emits no tok/s — wall time (incl. model load) gets read as a decode rate prints tokens, decode seconds and tok/s, with load time separate

Why first: every receipt 0.69 produces is worth less until these are true, and #3544 is scope (69-A1), not overhead.

Band 2 — Decode correctness (gates the batching bands)

# Item Acceptance
#2753 P0 batched CUDA decode emits garbage for every m>1 greedy parity vs m=1 across m∈{2,4,8} on both GPU hosts
#2770 cuBLAS decode route diverges at m≥4 (coherent but different continuation) same corpus, same seed: identical token stream at m=4 and m=8
#2765 P0 cublas_prefill_fp8_gemm reads 16 rows of FP8 scratch where 1 was written bounds assertion + a test that RED-turns on the padded tail

Rule for the cut: if these are red, 69-F is CUT, not waived. We do not publish a batching band measured on a path that produces garbage.

Band 3 — Headline value: Qwen usability on pure CUDA (#3519)

Row Item Host
69-G1 apr finetune --recipe — resolver, --plan, example recipes, book + contract CPU + gx10
69-G2 7B QLoRA recipe smoke nightly (loss window, merge, run) gx10
#2926 the four WT receipts: Unsloth QLoRA vs apr finetune -m qlora on Qwen2.5-7B-Instruct both
#2924 apr finetune --max-seq-len honoured, or refused in one line with a refusal code both
#2906 honest training banner: -m lora --gpu-backend cuda trains on the GPU or refuses both
#3519 tool calls (Hermes-style <tool_call>) + automation, receipt per (model, host) both

Band 4 — Release-train mechanics

# Item
#3459 P0 move the autopilot's tag path into the repo — the train currently runs from a scratch bundle
#3462 cascade-publish.sh TIERS is not a dependency order (47 non-dev violations). 0.68.2 published from a bundle-local publish-order.txt; the in-tree path still carries the defect
#3446 APR-RELEASE-001 §1.5 must-carry section on main

Band 5 — Root cause (see Band 0: M1 is must-carry, not optional)

# Item
#3430 PP-QUANT-001 M1 — must-carry, gates T-0; see the correction at the top
#3421 epic (phase 0/1)

The rest of PP-QUANT/PP-TENSOR (#3418 #3429 #3431 #3433 #3434 #3428) moved to 0.70: right fix for the recurring quant-dispatch defects, not a three-day train.

Standing

Carried from the original §3 scope

69-F1/F2 (batching bands VALID, CONFORMANT ledger rows) — conditional on Band 2. 69-B1 (kernel ladder nightly), 69-D1 (yoga ≥240 min/day), 69-C1 (gx10 ≥720 min/day × 3), 69-E1 (PR budget guard), 69-R (release).


Index — the 36 open items, by band (2026-09-20)

Band 1 · Andon (8) — #3543 #3544 #3545 #3561 #3554 #2730 #3541 #3542 (#3170 → 0.70)
Band 2 · Decode correctness (3) — #2753 #2770 #2765 · gates 69-F
Band 3 · Headline, Qwen usability + dogfood (6) — #3519 #3558 #3555 #2926 #2924 #2906 · + 69-G1/G2
Band 3b · Ontology / SHACL — co-equal priority (4) — #3567 (blocks both) #3559 #3560 #3269
Band 4 · Train mechanics (5) — #3462 #3446 #3547 #3553 #3562 (#3459/#3461 promoted to Band 0b)
Band 0 · MUST-CARRY, gates T-0 (6) — #3452 (ruled 2026-09-20) → #3420 → #3430 → #3447 → #3446 · epic #3421
Band 0b · TAG PATH, also gates T-0 (2) — #3459 #3461 · days 1–3, nothing beside it; §1.1a lands in this chain
Band 6 · Release assets & installer (2) — #3546 #3540
Band 7 · Downstream consumer (5) — #3548 #3549 #3550 #3551 #3552
Standing (1) — #3513
Epic — #3080

Bands 6 and 7 came from partner sessions after the rebuild: the assets lane (infra-98) measured the published v0.68.2 assets, the consumer lane (apex-c8) pinned aprender-viz 0.68.2 from a clean registry, the fleet lane (infra-45) measured the boxes, and the train lane (aprender-bf) listed what 0.68.2 still owes. Their full reports are in the comments, credited, with their acceptance commands verbatim — #3545 in particular came from the assets lane and is the most consequential thing found today.


Cut from 0.69 (moved, not dropped)

→ 0.70: #3401 #3204 paiml/infra#1144 #3478 #3161 #3466 #3464 #3418 #3428 #3429 #3431 #3433 #3434 #2853 #2507 #2616 #3531 #3522 #3062
→ 0.71: #3512 (Qwen3.5-35B-A3B — operator 2026-09-19: 0.68.2 supports up to 27B, 35B is post-0.70)

Acceptance (the train is done when)

For the team to rule on

  1. Which Qwen model does 0.69 bless? (Name the Qwen model 0.69 blesses and commit its parity receipt at the SERVING shape — our gates are all single-stream #3555) — a downstream reviewer lane's admission depends on the answer, and the receipt for Qwen3.5 already exists at 0.999 while the number that lane holds (Qwen2.5-1.5B, 0.9418) is our threshold's negative control.
  2. Make the host-receipt DEFER fail closed — 0.68.2 discharged it by hand, and the gate's first run fabricated 4 host failures from a missing directory #3544's host matrix — does the train emit and commit evidence/dogfood/<v>/<host>.json, or is the matrix narrowed to the hosts the train actually visits? mini (Apple M4) has never had a receipt and no aprender job targets its labels (#3518, now 0.70), so today the gate cannot go green even if we do our part.
  3. 69-F under Band 2 — accept the "cut, don't waive" rule, or fund the decode fixes as the train's long pole?
  4. Scope size — after the re-cut, ~30 items on a 3-day cadence, three of them co-equal release goals plus a must-carry lane. My earlier suggestion that Band 5 could be dropped first is withdrawn — it is must-carry. If that is too much, Band 5 (PP-QUANT-001 M1: one #[repr(u32)] GgmlType (43 rows: 35 live + 8 Removed) + TRAITS[43] below core/compute/serve — ggml enums 3 → 1 #3430) is the first thing I would drop; Band 1 is the last.

Original scheduled scope, before the 2026-09-20 rebuild (kept for diffing against the spec)

Release train 0.69.0 — declarative fine-tune lands; batching bands turn VALID

Spec of record: docs/specifications/06x-release-schedule.md (PMAT-1097; this epic is §3 "0.69.0" of that document, and §2 holds every row's acceptance command). Contract: contracts/release-schedule-06x-v1.yaml.

Window (§1.1): opens 2026-09-17T06:00Z · must leave by 2026-09-18T06:00Z · planned tag 2026-09-17T18:00Z. T is the previous train's actual publishedAt; the train leaves on time — scope slips forward to the next epic with slipped_from: 0.69.0, dates do not slip. A red release gate holds the train at most 12 h.

First train with priority G on it: apr finetune --recipe (69-G1) and the 7B QLoRA recipe smoke on gx10 (69-G2).

Decisions apply at their §6 defaults unless overridden in a comment here (D-1 FULL tier only nightly + pre-publish dogfood · D-2 NVIDIA Rust default per layer · D-3 yoga advisory→required behind the runner-group restriction · D-4 gx10 takes arch-neutral guards · D-5 trains leave on time · D-6 parity hosts yoga + gx10 · D-7 recipe format · D-8 #480 rewriter fate · D-9 non-Linux binaries out). An override comment is copied into the spec's §11 with decided_by and the comment link.

Scope (spec §3, as of origin/main @ 2c584a168)

Row Item Source Host
69-G1 apr finetune --recipe (resolver, --plan, example recipes, book + contract) §2 G CPU + gx10
69-G2 7B QLoRA recipe smoke nightly on gx10 (loss window, merge, run) §2 G gx10
69-F1 W-E scheduler + W-C prefill; c=4/8 VALID; Target agg(4) ≥ 3 × dec(1) recorded §2 F yoga, gx10
69-F2 first CONFORMANT ledger rows, both comparators, both hosts §2 F yoga, gx10
69-B1 kernel ladder nightly (cutile ports vs production) on both archs; per-kernel promotion rule §2 B gx10, yoga
69-D1 yoga runs the sm_89 ladder leg + parity bands; ≥ 240 min/day §2 D yoga
69-C1 gx10 ≥ 720 min/day for 3 consecutive days §2 C gx10
69-A1 post-publish dogfood installs via the installer on gx10, yoga, intel (C4) §2 A three hosts
69-E1 PR budget guard required; 10 consecutive PRs ≤ 20 min §2 E —
69-R release §1.3 —

Acceptance (the train is done when)

  • every row above is MERGED or moved to the next epic with slipped_from:
  • pre-publish dogfood FULL (scripts/dogfood.sh --phase pre-publish) GO
  • tag v0.69.0 published inside the window (or §11 amendment says why)
  • release carries 4 apr binaries ({cuda,cpu} × {x86_64,aarch64}) + .sha256 + 8 pv assets — checked by command, never by eye
  • post-publish dogfood receipts on gx10 and yoga under evidence/dogfood/0.69.0/
  • closing comment here: planned vs actual instant, receipt paths, next train's T

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZjEDVTFkFv71yCPD1dmUJ

Activity

  1. added this to the 0.69.0 milestone on Sep 10, 2026
  2. noahgift commented on Sep 10, 2026

    @noahgift
    ContributorAuthor

    Spec PR: #3087 (branch PMAT-1097-06x-release-schedule, auto-merge armed). The §3 table in this epic's body is the state at origin/main @ 2c584a168; §10 of the spec records the agy /teamwork-preview review.

  3. noahgift commented on Sep 12, 2026

    @noahgift
    ContributorAuthor

    Release estimate (orchestrator, 2026-09-12 12:20Z): eligible Wed 2026-09-17 ~16:00 UTC, tag Thu 2026-09-17 evening → Fri 2026-09-18 ~16:00 UTC. By this train the P1 fast set (from ≥20 ledger records) should have cut the gate time; the estimate assumes today's durations until that is measured. Slides one takt per skipped predecessor.

    Basis. docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md (PR #3164): a train becomes eligible 48 h after the previous tag on main and must be runnable by 72 h; any RED step skips the train at that HEAD and the next eligibility is 48 h later. Train duration measured on 0.67 today: queue drain of the open PRs (~1 h per FULL-tier entry on intel), pre-publish dogfood (~1 h, 0.66 precedent), the G3.EX examples sweep (981 targets, ~5 h on a loaded box — the long pole, shrinks when P0/P1/P2 land), assets + preflight (~1 h). The crates.io cascade is attended (spec §3.7): the tag and GitHub release are autonomous, the publish waits for the operator, and its minutes are recorded from 0.67 on.

  4. noahgift commented on Sep 17, 2026

    @noahgift
    ContributorAuthor

    0.69.0 cut precondition (#3445, merged in #3455). From main @ d79a6875b, at 2026-09-17T15:01:34Z:

    $ bash scripts/check_milestone_cut.sh 0.69.0   # exit 1
    ADMITTED paiml/aprender#3080 issue [enhancement,epic] EPIC: release train 0.69.0 — 06x-release-schedule.md §3 (0.69.0) -- the release epic of this train, closed at 06x section 4 step 8
    RED   milestone 0.69.0 (#6): 22 open item(s) -- no tag until each is closed or carried
    

    Per 06x §4 steps 1 and 4 and APR-RELEASE-001 T-0/T-3, the tag waits until this exits 0. Each of the 22 open item(s) has to be closed or carried: gh issue edit N --milestone 0.70.0 (or gh pr edit) plus a slipped_from: 0.69.0 comment. The script prints the exact command for each item. This epic (#3080) is the one admitted item. The train's autopilot is an untracked per-train copy and does not call the gate on its own. Add bash scripts/check_milestone_cut.sh 0.69.0 || die … as the first line of its tag block (gap tracked in #3454).

  5. noahgift commented on Sep 20, 2026

    @noahgift
    ContributorAuthor

    0.69.0 triage — 2026-09-20, after 0.68.2 shipped

    0.68.2 is on crates.io (74/74, 07:59:49Z, tag v0.68.2 @ 338f1d49b) and Qwen3.5 is validated on the published artifact: cargo install aprender --version 0.68.2 --features cuda, 0.8B/2B/4B/9B/27B × {CPU, CUDA} on lambda (RTX 4090, sm_89, x86_64) and gx10 (GB10, sm_121, aarch64) — 20/20 cells correct, every GPU arm reporting the real device. #3038 closed on that receipt; 35B stays #3512 (0.71).

    Decision 1 — one train, not two. 0.68.3 is folded into 0.69.

    0.68.3's epic #3519 carries the operator's 2026-09-19 goal ("COMPLETE usability of tool calls, fine-tuning (basic), automation of qwen models on pure CUDA"). 0.69's own scheduled G rows (apr finetune --recipe, 7B QLoRA smoke) are the mechanism that delivers #3519's fine-tuning row. Two trains would bid for the same two GPU hosts to build the same value — muda. #3519 is now the 0.69 headline; its fine-tune receipt rows (#2926, #2924, #2906) come with it. 0.68.3 is empty and should be closed as a milestone.

    Decision 2 — the andon cord comes before the feature.

    Three of our own release gates were measured lying today, during 0.68.2's post-publish run. Until they are fixed, every receipt 0.69 produces is worth less than it looks, so they are first and they are cheap:

    # defect why first
    #3543 post-publish dogfood cannot pass — version-unpublished is not phase-aware, and the banner says "pre-release" in a post-publish run the phase that is supposed to discharge DEFERs can never emit GO
    #3544 a DEFER nothing discharges — no evidence/dogfood/<v>/<host>.json since 0.67.0, three releases owed; the hosts step writes a different file, elsewhere, uncommitted this IS row 69-A1; fixing it is scope, not overhead
    #2730 cbtop reports PASS / ci_result: green over its own grade F a gate that reports green over its own red
    #3170 apr-agent --help created a worktree and a branch any flag is accepted as a task slug
    #3541 #3542 apr run prints no Backend: line on the CPU arm, and no tok/s at all found validating 0.68.2: the CPU arm cannot be proved, and wall time gets misread as a decode rate. Pair with #3543 — same file, same hour

    Decision 3 — the decode-correctness P0s gate the batching bands, so they are scope, not backlog.

    69-F1/F2 ask for c=4/8 VALID and agg(4) ≥ 3 × dec(1). That cannot be honest while:

    These three are the long pole of the train. If they do not land, 69-F is cut — we do not publish a band measured on a path that produces garbage.

    Priority order (Toyota Way: stop the line → make the value → fix the system)

    1. P0 andon — The post-publish dogfood CANNOT pass: version-unpublished is not phase-aware and the banner still says pre-release #3543, Make the host-receipt DEFER fail closed — 0.68.2 discharged it by hand, and the gate's first run fabricated 4 host failures from a missing directory #3544 (=69-A1), cbtop reports status PASS and ci_result green over its own grade F when the profiler returns zero bricks #2730, apr-agent accepts any flag as a task slug — 'apr-agent --help' created a worktree, a branch and a claim lockfile #3170, apr run: the --no-gpu arm prints no Backend: line, so the CPU arm cannot be proved (post-publish 0.68.2) #3541, apr run emits no tok/s — every throughput claim from it is wall time including model load (post-publish 0.68.2) #3542
    2. P0 correctness — P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753, cuBLAS decode route diverges at m>=4: coherent but different continuation, so c=4 fails parity while c=2 passes #2770, cublas_prefill_fp8_gemm reads 16 rows of FP8 scratch where 1 was written: the padded tail is the previous GEMM's activations #2765 (gates 69-F; if red at the cut, 69-F is cut, not waived)
    3. P1 headline — EPIC: release 0.68.3 — COMPLETE usability of Qwen models on pure CUDA: tool calls, basic fine-tuning, automation — every claim a receipt per (model, host) #3519: tool calls · basic fine-tune (T-0: the four WT receipts: Unsloth QLoRA and apr finetune -m qlora on Qwen2.5-7B-Instruct, both hosts, n=5 interleaved, paired at the highest ladder rung apr completes; first two traini #2926 T-2: apr finetune --max-seq-len honoured or refused with one line and the refusal code on every path; the effective value printed and written to the receipt; the wgpu instruct pipeline  #2924 R-3: T-6 honest training banner: -m lora --gpu-backend cuda refuses or trains on the GPU; the cuBLAS-backward line prints only when a cuBLAS backward ran #2906, = 69-G1/G2) · automation, each with a committed receipt per (model, host) on both GPU hosts
    4. P1 train mechanics — release train: move the release autopilot's tag path into the repo — tag step calls scripts/check_milestone_cut.sh and cannot be reached without it #3459 (tag path into the repo), cascade-publish.sh TIERS is not a dependency order — 47 non-dev violations at v0.68.1; it only works through the drain's retries #3462 (TIERS is not a dependency order — 0.68.2 published from a bundle-local publish-order.txt; the in-tree path still carries the defect), docs(release): APR-RELEASE-001 has no must-carry section on main — land §1.5 (must-carry rows; T-0 waits; > 72 h andon) #3446 (§1.5 must-carry on main)
    5. P2 root cause, one milestone only — PP-QUANT-001 M1: one #[repr(u32)] GgmlType (43 rows: 35 live + 8 Removed) + TRAITS[43] below core/compute/serve — ggml enums 3 → 1 #3430 (PP-QUANT-001 M1: one #[repr(u32)] GgmlType). The rest of PP-QUANT/PP-TENSOR (ARCH: quant-type dispatch is duplicated across ~30 files with no single source of truth — propose a dedicated consolidation release #3418 PP-QUANT-001 M3: regression fixtures — #1749 #1789 #2535 #3341 (+ #3091 reopen) on tiny synthetic GGUFs, observed RED→GREEN #3429 PP-QUANT-001 M2: reconciliation gate — GgmlType vs the vendored ggml.h id list at a pinned sha; exhaustive matches, no _ => #3431 PP-TENSOR-001 T1: typed TensorStorage — a dense consumer cannot receive an MoE placeholder #3433 PP-TENSOR-001 T2: sentinel ban — qtype: 0 + byte_size: 0 may not be constructed; 0 is F32 #3434 EPIC: PP-TENSOR-001 — a tensor that has no bytes is a different type from one that does (MoE / tied-embedding placeholders) #3428) moved to 0.70: they are the right fix for the recurring quant-dispatch defects, but they are not a three-day train.
    6. Standing — DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513 (DP4A through the GDN recurrence stays pinned off; the falsifier test is the guard)

    Cut from 0.69 (moved, not dropped)

    → 0.70: #3401 #3204 paiml/infra#1144 #3478 #3161 #3466 #3464 #3418 #3428 #3429 #3431 #3433 #3434 #2853 #2507 #2616 #3531 #3522 #3062

    Schedule note

    0.69's window (§1.1) was 2026-09-17→09-18 and 0.70's opens 09-21; 0.68.2 consumed it as an interrupt release. Either re-date 0.69 in 06x-release-schedule.md or let 0.69 take 0.70's slot — carrying a train with an expired window is how the milestone-cut gate goes vacuous. Recommendation: re-date, don't renumber.

  6. changed the title [-]EPIC: release train 0.69.0 — 06x-release-schedule.md §3 (0.69.0)[/-] [+]EPIC: release train 0.69.0 — COMPLETE Qwen usability on pure CUDA (all 0.68.x folded in)[/+] on Sep 20, 2026
  7. noahgift commented on Sep 20, 2026

    @noahgift
    ContributorAuthor

    Fleet & CI prerequisites — from the fleet lane (infra-45), measured this week

    Added to the epic after the rebuild. These are prerequisites to 0.69's GPU work, not parallel nice-to-haves.

    # Item Acceptance command Host
    NEW (gx10 disk) gx10 at 87% (749G/916G, 121G free) with 220G of CI scratch (targets 113G + cargo-ci 107G); set and enforce a gx10 disk budget before 0.69 GPU work ssh gx10 'df --output=pcent / | tail -1 | tr -cd 0-9' | awk '$1<80' gx10
    NEW (gx10 deploy visibility) bin-aprender on gx10 is a github_release pinned to tag: nightly whose completion check is presence-only — forjar reports converged while serving 0.68.2 forjar drift -f machines/gx10/forjar.yaml --state-dir machines/gx10/state | grep -c 'bin-aprender.*DRIFT' gx10
    #3540 book-contracts.yml: delete the three rust-cache steps — the last aprender site; the shared-cargo-home guard BLOCKs it from today bash machines/clean-room/shared-home-guard/check.sh aprender .github/workflows/book-contracts.yml | grep -c VIOLATION intel
    infra#780 sovereign-ci's per-PR CARGO_TARGET_DIR is written by 4 concurrent jobs on one host; re-key to <run-id>/<job> and teach the reaper the new depth in the same PR bash machines/intel/sovereign-ci/systemd/reaper/test-ci-targets-depth.sh intel+gx10+yoga
    infra#781 ephemeral runners lack capabilities host runners have (no ~/.ssh #776, no Lean #781) — declare per-job capability and ask each host, not the Dockerfile bash machines/clean-room/capability-parity.sh --selftest && bash machines/clean-room/capability-parity.sh intel+gx10+yoga
    NEW (matrix vacuity) no aprender job targets mini's labels, yet the dogfood gate demands a mini receipt bash scripts/check_dogfood_matrix_is_visited.sh mini

    Why gx10 disk is listed above every GPU row: a full disk there does not fail loudly — it fails as corrupted builds and No such file or directory errors that get filed as flakes. 0.69's GPU work is gx10-shaped. This already cost us once: gx10 hit 100% on 2026-09-19 mid-train.

    Why the tag: nightly row matters for a release epic: if "deployed on the fleet" is to mean anything in 0.69, the declaration needs a version the completion check can compare. Today forjar cannot tell you gx10 is on 0.69.

    Ruling on question 1 (#3544's host matrix) — narrow it, and derive the narrowing

    Decided with the fleet lane, and I'm adopting it:

    • Do not emit a mini.json the train never visited. That converts "we do not test on mini" into a green tick — a receipt for a measurement nobody took. The current red is TRUE; a manufactured green is a lie the gate itself produces, which is this fleet's signature defect.
    • Do not hand-maintain a narrowed list either — a constant list drifts from reality and nobody re-reads it.
    • Derive the matrix from the hosts the train actually visits (the labels its jobs select, read from the workflow). A host with no job targeting it is recorded as an explicit NA{reason, decided_by, date} row — declared absence, not silent absence. When paiml/infra#1144 lands and mini gets a job, mini re-enters automatically because the derivation sees the new job, not because someone remembered.
    • The gate fails if the demanded set and the visited set disagree in either direction: demanded-but-not-visited is today's bug, visited-but-not-demanded is tomorrow's.

    And the more serious half, stated plainly: mini is a scoping error. lambda, intel and gx10 having no receipt across two releases is a gate that has not run. Fix that first; mini is then a footnote.

  8. noahgift commented on Sep 20, 2026

    @noahgift
    ContributorAuthor

    Release-assets & installer — from the assets lane (infra-98), measured on the published v0.68.2 assets

    Headline finding, escalated to its own P0: #3545. The published -cuda asset prints backend: cuda unavailable reason=NotCompiled source=not-compiled on a 4090 host. Found by the assets lane; the release lane reproduced it and settled the open branch — the asset does run CUDA (apr run … --gpu → Backend: GPU (CUDA, NVIDIA GeForce RTX 4090) [qwen35 hybrid forward, paiml/aprender#3090], correct output, and the binary carries dp4a_q4k_gemv / tensor_core_q4k_gemm / flash_decoding_chunk which the -cpu asset does not). So devices lies; the build is fine. The first command a new user runs tells them our CUDA support does not exist.

    # Item Acceptance Host
    #3545 apr devices reports NotCompiled on a binary that demonstrably runs CUDA devices --json | jq -e '.entries[]|select(.kind=="cuda")|.source.kind=="compiled-in"' on a -cuda asset, non-zero on -cpu, plus a RED-turning test lambda, gx10
    NEW flavour gate on the artifact alone — no GPU, no model needed strings apr | grep -q libcuda.so.1 PASSes for -cuda, FAILs for -cpu, and devices --json agrees any host, incl. yoga
    #2869 / PMAT-994 install.sh does not exist — /releases/latest/download/install.sh 404s and nothing in the tree provides it, yet T-2 asserts install.sh --version v0.N.0 on intel and gx10, and the 0.68.0 note says "install.sh verified on intel/gx10" curl -fsS .../latest/download/install.sh | head -1, then install.sh --version v0.69.0 on a clean HOME intel, gx10
    #3544 the receipt schema below, written by the autopilot instead of free-text receipts/<host>.txt jq -e over the committed file all
    NEW PP-066 R-6 says the installer's last line is apr devices — on today's assets that line prints a false cuda … not-compiled R-6 acceptance must read the devices JSON, not just exit 0 all
    NEW crates/aprender-serve/models/qwen2-0.5b-q4.gguf is 0 bytes a fixture model is non-empty and loads, or it is deleted —

    Adopted: what a CUDA asset gate asserts when the host has no GGUF — two tiers

    Adopted: the #3544 receipt schema

    {host, arch, version_tested, date, install_rc} plus the fields that let a later reader falsify the claim rather than take it: provenance ∈ {release-asset, crates.io, local-build} and the exact asset filename; sha256_published and sha256_measured as two fields so a match is visible as a match; the apr devices --json entries array verbatim; generate: {model_sha256, backend_line_verbatim, tokens, wall_ms, output_sane} or generate: null with a reason; driver (nvidia-smi + CUDA runtime) and nproc; and a non-empty unmeasured[] naming what this host could not prove.

    "A receipt from a host with no GPU and an empty unmeasured[] is a lie, and that is the only field that makes yoga's receipt worth committing at all." — assets lane

    install.sh is listed as uncovered. Not "covered by a weak check" — there is nothing to cover. Treat R-6 as the work, not the check.


    Outside-in — from the downstream consumer (apex-c8), measured against the published crate

    Validation closed: nothing failed. apex pinned aprender-viz 0.68.2 from crates.io — cargo update --precise clean, cargo check/test --workspace rc 0, cargo deny check rc 0, and apex#99 green including publish-sim (offline --network none). The .crate was verified independently of the index (sha256 cbd7448685d7…, matching Cargo.lock's registry checksum). 0.68.2 also closed a real gap: 0.68.1's .crate had no src/breaks.rs at all, which is why apex was carrying a git patch.

    # Item Acceptance
    NEW aprender-viz packages fixtures/breaks/ (manifest.json + generate.py) into the published .crate — 396 KB of test fixtures shipped to every consumer, and nobody downstream runs breaks_golden.rs tar tzf aprender-viz-<v>.crate | grep -c 'fixtures/breaks/' is 0. Pure packaging, no API change — the cheapest row in the epic
    NEW the crate name and the lib name differ and nothing says so: the package is aprender-viz, but use aprender_viz::… does not compile — it is use trueno_viz::breaks::extended cargo doc -p aprender-viz front page and the crates.io README both state it in the first screen
    NEW a consumer-shaped facade: apex wants to draw a chart and transitively compiles aprender-compute, aprender-quant, aprender-gemm-codegen, aprender-contracts-macros from a scratch bin depending only on the facade, a program calling extended and panels builds, and cargo tree -e normal shows none of those crates
    NEW aprender-viz is f32, consumers compute in f64, and the narrowing is undocumented and unguarded a doc example shows the intended narrowing; DataFrame's docs state the f32 domain and what happens outside it (three review lanes flagged mantissa precision loss, point-order retention, subnormal underflow)
    NEW Facet/Coord have no runnable example — EV-15 (byte-identical SVG across architectures) and EV-16 (unbranded export) have nothing to build against one examples/ program: study data → panels → Coord::apply → SVG, compiling against the published crate only

    "The single worst guess I had to make" — the aprender-viz / trueno_viz name split. "Every new consumer will hit this in their first five minutes."

    Caveat recorded by the lane itself: apex#99 is green but deliberately unmerged this session pending an apex-side defect (an EV-2a probe whose crates.io call has no User-Agent → 403, so the probe can never pass). That is apex-side and unrelated to this train; the build and resolve evidence above stands on its own.

  9. noahgift commented on Sep 20, 2026

    @noahgift
    ContributorAuthor

    0.68.2 train debt — from the train lane (aprender-bf), measured 2026-09-20

    The train that just shipped still owes two receipts. Both are the release lane's to write, today, and both are listed here so the debt is visible in the milestone Noah's team reviews rather than living in a session.

    # Item Acceptance State
    NEW T-5 reconcile receipt for v0.68.2 is not written — docs/build-ledger/ on main has 2026-09-16/17/18 and nothing for 09-19/20 test -f docs/build-ledger/2026-09-20/338f1d49b-reconcile.json and every §6 predicate reads 0 OWED — hard gate (operator 2026-09-13); the train has no DONE line without it
    NEW train ledger record (ledger.py) not written the row committed OWED, same close-out batch
    #3544 host receipts check_multiplatform_dogfood.sh exit 0 against committed receipts the autopilot's hosts step was running at the time of this report; if it discharges, 0.68.2 becomes the first train since 0.67.0 with host receipts, and #3544 changes shape — from "nobody ever ran it" to "make the DEFER fail closed"
    #3462 + #3459 same fix, not two: in-tree cascade-publish.sh TIERS is not a dependency order (47 non-dev violations), which is why 0.68.2 published from a bundle-local publish-order.txt + publish_strict.sh. #3459 (move the tag path in-tree) is the vehicle; #3462 is the defect scripts/cascade-publish.sh --order-check exits 0 against the tag the drain only works through retries — 8 passes today
    #3541 #3542 apr run prints no tok/s and no Backend: line on the CPU arm already filed from the gx10 measurement now docketed

    On #3459, the argument in one line: the autopilot (autopilot.sh, publish_strict.sh, t2_preflight.sh, publish-order.txt) is bundle-local and unversioned, and today a stale STOP-FOR-OPERATOR line in it — contradicting the very spec revision it cited — was corrected by hand at 07:30Z in a file no PR has ever reviewed. A release train whose control flow is not in the repo is not reviewable, and the correction proves it.

    Deferrals audit: the only conscious deferrals from 0.68.2 with no ticket were the two rows above plus #3486 (qwen3-8b's optional rung, already ticketed). Nothing else was deferred silently — which is the answer #3544 exists to make routine.

  10. 19 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical priorityenhancementNew feature or requestepicEpic — multi-story umbrella

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions