Repository navigation
EPIC: release train 0.69.0 — COMPLETE Qwen usability on pure CUDA (all 0.68.x folded in) #3080
Description
Release train 0.69.0 — all remaining 0.68.x work folded in. Headline: COMPLETE usability of Qwen models on pure CUDA.
Spec of record: docs/specifications/06x-release-schedule.md §3 (0.69.0); §2 holds each row's acceptance command.
Rebuilt 2026-09-20 on the operator's instruction to merge 0.68.x into 0.69.0 after 0.68.2 shipped. Prior scope preserved below as 69-*; new rows carry issue numbers.
Window: re-dated, not renumbered — due 2026-09-23 (0.70 → 09-26, 0.71 → 09-29, 0.72 → 10-02, 0.73 → 10-05). 0.68.2 consumed the original 09-17→09-18 window as an interrupt release.
Governed by
30-day-plan.mdv2 (2026-09-20). v1 was returneddo-not-implement-as-writtenby a 3/3 adversarial quorum; v2's doctrine is what this train runs under. The changes that affect 0.69 are in a comment below.0.69 purpose: the tools stop lying. Excluded (predicates, labels now exist):
label:arch-model-new·label:ont-depth·label:viz·label:perf-claim -label:has-receipt. A move out of this milestone cites a predicate or is logged as an override.Two gates can refuse T-0: (1) must-carry slice M — #3452 (ruled) → #3420 → #3430 → #3447 → #3446; (2) the tag path under review — #3459/#3461, "nothing beside it" in days 1–3. Fix-or-refuse is how we ship, never how we measure: every goal needs a floor on the fix side plus a positive control.
⚠ Correction (2026-09-20, after review): Band 5 was wrong — slice M is MUST-CARRY
An earlier revision of this epic put PP-QUANT-001 M1 (#3430) in a "root cause, drop first" band. That was an error, and it inverted the critical path.
06x-release-schedule.md §1.1a names PP-QUANT-001 slice M as the 0.69.0 must-carry row, and APR-RELEASE-001 §0 row 1a puts a §1.5 release-critical ticket above every other row — T-0 waits on it, and row 1 will not run the train until "§1.5 scope for that tag is merged". A must-carry row is not droppable by definition; it is the thing the tag is refused for.
Lane A is therefore the critical path and it is idle, blocked on one decision: #3452 (the M1 plan) has been through four quorum rounds — 0/3, 1/3, 1/3, 1/3 — and is awaiting a ruling. The chain is #3452 (ruling) → #3420 → #3430 → #3447 → #3446, with epic #3421. No amount of tidying elsewhere cuts T-0 while that ruling is outstanding.
Process defect this exposes, worth its own row: two consecutive non-passing plan quorums on a must-carry ticket should escalate automatically. #3452 sat through four.
Three co-equal release goals (operator, 2026-09-20)
0.69 has three blocking goals, not one headline with extras. They are stated as equal because the operator stated them as equal.
| Goal | Issue | What it means |
|---|---|---|
| A · Qwen3.5 usable on pure CUDA | #3519 | tool calls, basic fine-tune, automation — a committed receipt per (model, host) on both GPU hosts |
| A′ · First-party dogfood at a Pareto-optimal size | #3558 | arbiter, RAH and apex dogfood Qwen3.5 from here on; we publish the selection rule and its evidence — smallest size that measurably works, not the largest that fits. Internal consumers' own details are theirs and are not reproduced in this public repo |
| B · Ontology to 80%, SHACL first | #3559 | ≈7 more ONT-001 rows (41% → 80%), every shape armed with a red-turning falsifier. Explicitly equal in priority to A/A′ |
| C · Docs, 1037 examples and the cookbook SHACL-validated and Qwen3.5-current | #3560 | measured today: 442 book pages, 1037 example targets, 6 artifacts mention Qwen3.5. Downstream of B's extractors |
One conflict the team must resolve before B is committed: APR-RELEASE-001 §11 sets ontology cadence at one row per train. Seven rows is seven trains' worth. #3559 states the three options; my recommendation is a dated §11 amendment that keeps the arming discipline (rows may batch; a shape still may not arm without a falsifier), because the cadence exists to stop unfalsifiable gates landing in bulk and the falsifier requirement is what actually controls that risk.
Where we start from: 0.68.2 shipped 2026-09-20
74/74 crates on crates.io at 07:59:49Z, tag v0.68.2 @ 338f1d49b, 16 assets, clean-room green. Qwen3.5 verified on the published artifact (cargo install aprender --version 0.68.2 --features cuda): 0.8B/2B/4B/9B/27B × {CPU, CUDA} on lambda (RTX 4090, sm_89, x86_64) and gx10 (GB10, sm_121, aarch64) — 20/20 cells correct, every GPU arm reporting the real device. Details in #3477, #3038.
So 0.69 is not about making Qwen run. It is about making it usable, and about the fact that four of our own gates were caught lying while we proved the above.
Headline (#3519)
Operator, 2026-09-19: "The goal of this release is to ensure we have COMPLETE usability of: tool calls, fine-tuning (basic), automation, etc of qwen models that work with pure CUDA."
"Usable" is defined as measured, not claimed: a committed receipt showing the capability working on both ladder hosts (lambda sm_89, gx10 sm_121) on the GPU, per (model, host). 0.68.3 is folded here and closed as a milestone — two trains bidding for the same two GPU hosts to build the same value is muda.
Scope, in the order it must be worked
Band 1 — Andon: gates of ours that were measured lying (do these first, they are cheap)
| # | Item | Acceptance |
|---|---|---|
| #3543 | post-publish dogfood cannot pass — version-unpublished is not phase-aware, banner says "pre-release" in a post-publish run |
dogfood.sh --phase post-publish GO at a published version and RED at an unpublished one — both directions |
| #3544 | a DEFER nothing discharges — no evidence/dogfood/<v>/<host>.json since 0.67.0, three releases owed. This is row 69-A1 |
check_multiplatform_dogfood.sh exit 0 against receipts the train itself committed |
| #2730 | cbtop reports PASS / ci_result: green over its own grade F |
force grade F, assert non-zero exit and status: FAIL |
| #3170 | apr-agent --help created a worktree and a branch — any flag is accepted as a task slug |
apr-agent --help exits 0, git worktree list and git branch unchanged |
| #3541 | apr run --no-gpu prints no Backend: line, so the CPU arm is intent, not measurement |
every arm prints Backend: naming the path actually dispatched |
| #3561 | P0 the T-0 preflight returns GO when the dogfood crashes before emitting a row — fails == 0 because there are no rows at all, on the gate that guards publishing |
point it at a tree where dogfood exits early; the verdict must be NO-GO (today: GO … dogfood_rc=2) |
| #3545 | P0 apr devices reports cuda unavailable reason=NotCompiled on a binary that demonstrably runs CUDA — the first screen a new user sees says our CUDA support does not exist |
devices --json agrees with what run --gpu actually dispatches, on a -cuda and a -cpu asset, with a RED-turning test |
| #3542 | apr run emits no tok/s — wall time (incl. model load) gets read as a decode rate |
prints tokens, decode seconds and tok/s, with load time separate |
Why first: every receipt 0.69 produces is worth less until these are true, and #3544 is scope (69-A1), not overhead.
Band 2 — Decode correctness (gates the batching bands)
| # | Item | Acceptance |
|---|---|---|
| #2753 | P0 batched CUDA decode emits garbage for every m>1 | greedy parity vs m=1 across m∈{2,4,8} on both GPU hosts |
| #2770 | cuBLAS decode route diverges at m≥4 (coherent but different continuation) | same corpus, same seed: identical token stream at m=4 and m=8 |
| #2765 | P0 cublas_prefill_fp8_gemm reads 16 rows of FP8 scratch where 1 was written |
bounds assertion + a test that RED-turns on the padded tail |
Rule for the cut: if these are red, 69-F is CUT, not waived. We do not publish a batching band measured on a path that produces garbage.
Band 3 — Headline value: Qwen usability on pure CUDA (#3519)
| Row | Item | Host |
|---|---|---|
| 69-G1 | apr finetune --recipe — resolver, --plan, example recipes, book + contract |
CPU + gx10 |
| 69-G2 | 7B QLoRA recipe smoke nightly (loss window, merge, run) | gx10 |
| #2926 | the four WT receipts: Unsloth QLoRA vs apr finetune -m qlora on Qwen2.5-7B-Instruct |
both |
| #2924 | apr finetune --max-seq-len honoured, or refused in one line with a refusal code |
both |
| #2906 | honest training banner: -m lora --gpu-backend cuda trains on the GPU or refuses |
both |
| #3519 | tool calls (Hermes-style <tool_call>) + automation, receipt per (model, host) |
both |
Band 4 — Release-train mechanics
| # | Item |
|---|---|
| #3459 | P0 move the autopilot's tag path into the repo — the train currently runs from a scratch bundle |
| #3462 | cascade-publish.sh TIERS is not a dependency order (47 non-dev violations). 0.68.2 published from a bundle-local publish-order.txt; the in-tree path still carries the defect |
| #3446 | APR-RELEASE-001 §1.5 must-carry section on main |
Band 5 — Root cause (see Band 0: M1 is must-carry, not optional)
| # | Item |
|---|---|
| #3430 | PP-QUANT-001 M1 — must-carry, gates T-0; see the correction at the top |
| #3421 | epic (phase 0/1) |
The rest of PP-QUANT/PP-TENSOR (#3418 #3429 #3431 #3433 #3434 #3428) moved to 0.70: right fix for the recurring quant-dispatch defects, not a three-day train.
Standing
- DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513 — DP4A Q4K/Q6K GEMV through the GDN recurrence stays pinned off; the falsifier test in
forward_qwen35_cuda_tests.rsis the guard on both hosts.
Carried from the original §3 scope
69-F1/F2 (batching bands VALID, CONFORMANT ledger rows) — conditional on Band 2. 69-B1 (kernel ladder nightly), 69-D1 (yoga ≥240 min/day), 69-C1 (gx10 ≥720 min/day × 3), 69-E1 (PR budget guard), 69-R (release).
Index — the 36 open items, by band (2026-09-20)
Band 1 · Andon (8) — #3543 #3544 #3545 #3561 #3554 #2730 #3541 #3542 (#3170 → 0.70)
Band 2 · Decode correctness (3) — #2753 #2770 #2765 · gates 69-F
Band 3 · Headline, Qwen usability + dogfood (6) — #3519 #3558 #3555 #2926 #2924 #2906 · + 69-G1/G2
Band 3b · Ontology / SHACL — co-equal priority (4) — #3567 (blocks both) #3559 #3560 #3269
Band 4 · Train mechanics (5) — #3462 #3446 #3547 #3553 #3562 (#3459/#3461 promoted to Band 0b)
Band 0 · MUST-CARRY, gates T-0 (6) — #3452 (ruled 2026-09-20) → #3420 → #3430 → #3447 → #3446 · epic #3421
Band 0b · TAG PATH, also gates T-0 (2) — #3459 #3461 · days 1–3, nothing beside it; §1.1a lands in this chain
Band 6 · Release assets & installer (2) — #3546 #3540
Band 7 · Downstream consumer (5) — #3548 #3549 #3550 #3551 #3552
Standing (1) — #3513
Epic — #3080
Bands 6 and 7 came from partner sessions after the rebuild: the assets lane (infra-98) measured the published v0.68.2 assets, the consumer lane (apex-c8) pinned aprender-viz 0.68.2 from a clean registry, the fleet lane (infra-45) measured the boxes, and the train lane (aprender-bf) listed what 0.68.2 still owes. Their full reports are in the comments, credited, with their acceptance commands verbatim — #3545 in particular came from the assets lane and is the most consequential thing found today.
Cut from 0.69 (moved, not dropped)
→ 0.70: #3401 #3204 paiml/infra#1144 #3478 #3161 #3466 #3464 #3418 #3428 #3429 #3431 #3433 #3434 #2853 #2507 #2616 #3531 #3522 #3062
→ 0.71: #3512 (Qwen3.5-35B-A3B — operator 2026-09-19: 0.68.2 supports up to 27B, 35B is post-0.70)
Acceptance (the train is done when)
- every row above is MERGED or moved to the next epic with
slipped_from: - Band 1 closed — and each fix demonstrated in both directions (passes when it should, reds when it should)
- Band 2 closed, or 69-F explicitly cut in a comment here with the measurement that justified it
- pre-publish dogfood FULL GO; post-publish dogfood GO (impossible today — that is The post-publish dogfood CANNOT pass: version-unpublished is not phase-aware and the banner still says pre-release #3543)
- tag
v0.69.0published inside the window (or §11 amendment says why) - release carries 4
aprbinaries ({cuda,cpu} × {x86_64,aarch64}) +.sha256+ 8pvassets — checked by command, never by eye -
evidence/dogfood/0.69.0/<host>.jsoncommitted for every host in the matrix (Make the host-receipt DEFER fail closed — 0.68.2 discharged it by hand, and the gate's first run fabricated 4 host failures from a missing directory #3544) - T-5 reconcile receipt committed and every §6 predicate reads 0
- closing comment: planned vs actual instant, receipt paths, next train's
T
For the team to rule on
- Which Qwen model does 0.69 bless? (Name the Qwen model 0.69 blesses and commit its parity receipt at the SERVING shape — our gates are all single-stream #3555) — a downstream reviewer lane's admission depends on the answer, and the receipt for Qwen3.5 already exists at 0.999 while the number that lane holds (Qwen2.5-1.5B, 0.9418) is our threshold's negative control.
- Make the host-receipt DEFER fail closed — 0.68.2 discharged it by hand, and the gate's first run fabricated 4 host failures from a missing directory #3544's host matrix — does the train emit and commit
evidence/dogfood/<v>/<host>.json, or is the matrix narrowed to the hosts the train actually visits?mini(Apple M4) has never had a receipt and no aprender job targets its labels (#3518, now 0.70), so today the gate cannot go green even if we do our part. - 69-F under Band 2 — accept the "cut, don't waive" rule, or fund the decode fixes as the train's long pole?
- Scope size — after the re-cut, ~30 items on a 3-day cadence, three of them co-equal release goals plus a must-carry lane. My earlier suggestion that Band 5 could be dropped first is withdrawn — it is must-carry. If that is too much, Band 5 (PP-QUANT-001 M1: one
#[repr(u32)] GgmlType(43 rows: 35 live + 8 Removed) +TRAITS[43]below core/compute/serve — ggml enums 3 → 1 #3430) is the first thing I would drop; Band 1 is the last.
Original scheduled scope, before the 2026-09-20 rebuild (kept for diffing against the spec)
Release train 0.69.0 — declarative fine-tune lands; batching bands turn VALID
Spec of record: docs/specifications/06x-release-schedule.md (PMAT-1097; this epic is §3 "0.69.0" of that document, and §2 holds every row's acceptance command). Contract: contracts/release-schedule-06x-v1.yaml.
Window (§1.1): opens 2026-09-17T06:00Z · must leave by 2026-09-18T06:00Z · planned tag 2026-09-17T18:00Z. T is the previous train's actual publishedAt; the train leaves on time — scope slips forward to the next epic with slipped_from: 0.69.0, dates do not slip. A red release gate holds the train at most 12 h.
First train with priority G on it: apr finetune --recipe (69-G1) and the 7B QLoRA recipe smoke on gx10 (69-G2).
Decisions apply at their §6 defaults unless overridden in a comment here (D-1 FULL tier only nightly + pre-publish dogfood · D-2 NVIDIA Rust default per layer · D-3 yoga advisory→required behind the runner-group restriction · D-4 gx10 takes arch-neutral guards · D-5 trains leave on time · D-6 parity hosts yoga + gx10 · D-7 recipe format · D-8 #480 rewriter fate · D-9 non-Linux binaries out). An override comment is copied into the spec's §11 with decided_by and the comment link.
Scope (spec §3, as of origin/main @ 2c584a168)
| Row | Item | Source | Host |
|---|---|---|---|
| 69-G1 | apr finetune --recipe (resolver, --plan, example recipes, book + contract) |
§2 G | CPU + gx10 |
| 69-G2 | 7B QLoRA recipe smoke nightly on gx10 (loss window, merge, run) | §2 G | gx10 |
| 69-F1 | W-E scheduler + W-C prefill; c=4/8 VALID; Target agg(4) ≥ 3 × dec(1) recorded |
§2 F | yoga, gx10 |
| 69-F2 | first CONFORMANT ledger rows, both comparators, both hosts | §2 F | yoga, gx10 |
| 69-B1 | kernel ladder nightly (cutile ports vs production) on both archs; per-kernel promotion rule | §2 B | gx10, yoga |
| 69-D1 | yoga runs the sm_89 ladder leg + parity bands; ≥ 240 min/day | §2 D | yoga |
| 69-C1 | gx10 ≥ 720 min/day for 3 consecutive days | §2 C | gx10 |
| 69-A1 | post-publish dogfood installs via the installer on gx10, yoga, intel (C4) | §2 A | three hosts |
| 69-E1 | PR budget guard required; 10 consecutive PRs ≤ 20 min | §2 E | — |
| 69-R | release | §1.3 | — |
Acceptance (the train is done when)
- every row above is MERGED or moved to the next epic with
slipped_from: - pre-publish dogfood FULL (
scripts/dogfood.sh --phase pre-publish) GO - tag
v0.69.0published inside the window (or §11 amendment says why) - release carries 4
aprbinaries ({cuda,cpu} × {x86_64,aarch64}) +.sha256+ 8pvassets — checked by command, never by eye - post-publish dogfood receipts on gx10 and yoga under
evidence/dogfood/0.69.0/ - closing comment here: planned vs actual instant, receipt paths, next train's
T
🤖 Generated with Claude Code
Activity
noahgift commentedon Sep 10, 2026
Spec PR: #3087 (branch
PMAT-1097-06x-release-schedule, auto-merge armed). The §3 table in this epic's body is the state atorigin/main@2c584a168; §10 of the spec records theagy /teamwork-previewreview.noahgift commentedon Sep 12, 2026
Release estimate (orchestrator, 2026-09-12 12:20Z): eligible Wed 2026-09-17 ~16:00 UTC, tag Thu 2026-09-17 evening → Fri 2026-09-18 ~16:00 UTC. By this train the P1 fast set (from ≥20 ledger records) should have cut the gate time; the estimate assumes today's durations until that is measured. Slides one takt per skipped predecessor.
Basis.
docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md(PR #3164): a train becomes eligible 48 h after the previous tag onmainand must be runnable by 72 h; any RED step skips the train at that HEAD and the next eligibility is 48 h later. Train duration measured on 0.67 today: queue drain of the open PRs (~1 h per FULL-tier entry on intel), pre-publish dogfood (~1 h, 0.66 precedent), the G3.EX examples sweep (981 targets, ~5 h on a loaded box — the long pole, shrinks when P0/P1/P2 land), assets + preflight (~1 h). The crates.io cascade is attended (spec §3.7): the tag and GitHub release are autonomous, the publish waits for the operator, and its minutes are recorded from 0.67 on.release(0.67): train A — 17 row PRs of the 0.67.0 train in one build …
PMAT-3445: admit the train's own release epic — it is open at every c…
noahgift commentedon Sep 17, 2026
0.69.0 cut precondition (#3445, merged in #3455). From
main@d79a6875b, at 2026-09-17T15:01:34Z:Per 06x §4 steps 1 and 4 and APR-RELEASE-001 T-0/T-3, the tag waits until this exits 0. Each of the 22 open item(s) has to be closed or carried:
gh issue edit N --milestone 0.70.0(orgh pr edit) plus aslipped_from: 0.69.0comment. The script prints the exact command for each item. This epic (#3080) is the one admitted item. The train's autopilot is an untracked per-train copy and does not call the gate on its own. Addbash scripts/check_milestone_cut.sh 0.69.0 || die …as the first line of itstagblock (gap tracked in #3454).noahgift commentedon Sep 20, 2026
0.69.0 triage — 2026-09-20, after 0.68.2 shipped
0.68.2 is on crates.io (74/74, 07:59:49Z, tag
v0.68.2@338f1d49b) and Qwen3.5 is validated on the published artifact:cargo install aprender --version 0.68.2 --features cuda, 0.8B/2B/4B/9B/27B × {CPU, CUDA} on lambda (RTX 4090, sm_89, x86_64) and gx10 (GB10, sm_121, aarch64) — 20/20 cells correct, every GPU arm reporting the real device. #3038 closed on that receipt; 35B stays #3512 (0.71).Decision 1 — one train, not two. 0.68.3 is folded into 0.69.
0.68.3's epic #3519 carries the operator's 2026-09-19 goal ("COMPLETE usability of tool calls, fine-tuning (basic), automation of qwen models on pure CUDA"). 0.69's own scheduled G rows (
apr finetune --recipe, 7B QLoRA smoke) are the mechanism that delivers #3519's fine-tuning row. Two trains would bid for the same two GPU hosts to build the same value — muda. #3519 is now the 0.69 headline; its fine-tune receipt rows (#2926, #2924, #2906) come with it. 0.68.3 is empty and should be closed as a milestone.Decision 2 — the andon cord comes before the feature.
Three of our own release gates were measured lying today, during 0.68.2's post-publish run. Until they are fixed, every receipt 0.69 produces is worth less than it looks, so they are first and they are cheap:
version-unpublishedis not phase-aware, and the banner says "pre-release" in a post-publish runevidence/dogfood/<v>/<host>.jsonsince 0.67.0, three releases owed; thehostsstep writes a different file, elsewhere, uncommittedPASS/ci_result: greenover its own grade Fapr-agent --helpcreated a worktree and a branchapr runprints noBackend:line on the CPU arm, and no tok/s at allDecision 3 — the decode-correctness P0s gate the batching bands, so they are scope, not backlog.
69-F1/F2 ask for c=4/8 VALID and
agg(4) ≥ 3 × dec(1). That cannot be honest while:cublas_prefill_fp8_gemmreads 16 rows of FP8 scratch where 1 was written (P0).These three are the long pole of the train. If they do not land, 69-F is cut — we do not publish a band measured on a path that produces garbage.
Priority order (Toyota Way: stop the line → make the value → fix the system)
publish-order.txt; the in-tree path still carries the defect), docs(release): APR-RELEASE-001 has no must-carry section onmain— land §1.5 (must-carry rows; T-0 waits; > 72 h andon) #3446 (§1.5 must-carry on main)#[repr(u32)] GgmlType(43 rows: 35 live + 8 Removed) +TRAITS[43]below core/compute/serve — ggml enums 3 → 1 #3430 (PP-QUANT-001 M1: one#[repr(u32)] GgmlType). The rest of PP-QUANT/PP-TENSOR (ARCH: quant-type dispatch is duplicated across ~30 files with no single source of truth — propose a dedicated consolidation release #3418 PP-QUANT-001 M3: regression fixtures — #1749 #1789 #2535 #3341 (+ #3091 reopen) on tiny synthetic GGUFs, observed RED→GREEN #3429 PP-QUANT-001 M2: reconciliation gate —GgmlTypevs the vendoredggml.hid list at a pinned sha; exhaustive matches, no_ =>#3431 PP-TENSOR-001 T1: typedTensorStorage— a dense consumer cannot receive an MoE placeholder #3433 PP-TENSOR-001 T2: sentinel ban —qtype: 0+byte_size: 0may not be constructed; 0 is F32 #3434 EPIC: PP-TENSOR-001 — a tensor that has no bytes is a different type from one that does (MoE / tied-embedding placeholders) #3428) moved to 0.70: they are the right fix for the recurring quant-dispatch defects, but they are not a three-day train.Cut from 0.69 (moved, not dropped)
→ 0.70: #3401 #3204 paiml/infra#1144 #3478 #3161 #3466 #3464 #3418 #3428 #3429 #3431 #3433 #3434 #2853 #2507 #2616 #3531 #3522 #3062
Schedule note
0.69's window (§1.1) was 2026-09-17→09-18 and 0.70's opens 09-21; 0.68.2 consumed it as an interrupt release. Either re-date 0.69 in
06x-release-schedule.mdor let 0.69 take 0.70's slot — carrying a train with an expired window is how the milestone-cut gate goes vacuous. Recommendation: re-date, don't renumber.[-]EPIC: release train 0.69.0 — 06x-release-schedule.md §3 (0.69.0)[/-][+]EPIC: release train 0.69.0 — COMPLETE Qwen usability on pure CUDA (all 0.68.x folded in)[/+]noahgift commentedon Sep 20, 2026
Fleet & CI prerequisites — from the fleet lane (infra-45), measured this week
Added to the epic after the rebuild. These are prerequisites to 0.69's GPU work, not parallel nice-to-haves.
ssh gx10 'df --output=pcent / | tail -1 | tr -cd 0-9' | awk '$1<80'bin-aprenderon gx10 is agithub_releasepinned totag: nightlywhose completion check is presence-only — forjar reports converged while serving 0.68.2forjar drift -f machines/gx10/forjar.yaml --state-dir machines/gx10/state | grep -c 'bin-aprender.*DRIFT'bash machines/clean-room/shared-home-guard/check.sh aprender .github/workflows/book-contracts.yml | grep -c VIOLATIONCARGO_TARGET_DIRis written by 4 concurrent jobs on one host; re-key to<run-id>/<job>and teach the reaper the new depth in the same PRbash machines/intel/sovereign-ci/systemd/reaper/test-ci-targets-depth.sh~/.ssh#776, no Lean #781) — declare per-job capability and ask each host, not the Dockerfilebash machines/clean-room/capability-parity.sh --selftest && bash machines/clean-room/capability-parity.shbash scripts/check_dogfood_matrix_is_visited.shWhy gx10 disk is listed above every GPU row: a full disk there does not fail loudly — it fails as corrupted builds and
No such file or directoryerrors that get filed as flakes. 0.69's GPU work is gx10-shaped. This already cost us once: gx10 hit 100% on 2026-09-19 mid-train.Why the
tag: nightlyrow matters for a release epic: if "deployed on the fleet" is to mean anything in 0.69, the declaration needs a version the completion check can compare. Today forjar cannot tell you gx10 is on 0.69.Ruling on question 1 (#3544's host matrix) — narrow it, and derive the narrowing
Decided with the fleet lane, and I'm adopting it:
mini.jsonthe train never visited. That converts "we do not test on mini" into a green tick — a receipt for a measurement nobody took. The current red is TRUE; a manufactured green is a lie the gate itself produces, which is this fleet's signature defect.NA{reason, decided_by, date}row — declared absence, not silent absence. When paiml/infra#1144 lands and mini gets a job, mini re-enters automatically because the derivation sees the new job, not because someone remembered.And the more serious half, stated plainly: mini is a scoping error. lambda, intel and gx10 having no receipt across two releases is a gate that has not run. Fix that first; mini is then a footnote.
noahgift commentedon Sep 20, 2026
Release-assets & installer — from the assets lane (infra-98), measured on the published v0.68.2 assets
Headline finding, escalated to its own P0: #3545. The published
-cudaasset printsbackend: cuda unavailable reason=NotCompiled source=not-compiledon a 4090 host. Found by the assets lane; the release lane reproduced it and settled the open branch — the asset does run CUDA (apr run … --gpu→Backend: GPU (CUDA, NVIDIA GeForce RTX 4090) [qwen35 hybrid forward, paiml/aprender#3090], correct output, and the binary carriesdp4a_q4k_gemv/tensor_core_q4k_gemm/flash_decoding_chunkwhich the-cpuasset does not). Sodeviceslies; the build is fine. The first command a new user runs tells them our CUDA support does not exist.apr devicesreportsNotCompiledon a binary that demonstrably runs CUDAdevices --json | jq -e '.entries[]|select(.kind=="cuda")|.source.kind=="compiled-in"'on a-cudaasset, non-zero on-cpu, plus a RED-turning teststrings apr | grep -q libcuda.so.1PASSes for-cuda, FAILs for-cpu, anddevices --jsonagreesinstall.shdoes not exist —/releases/latest/download/install.sh404s and nothing in the tree provides it, yet T-2 assertsinstall.sh --version v0.N.0on intel and gx10, and the 0.68.0 note says "install.sh verified on intel/gx10"curl -fsS .../latest/download/install.sh | head -1, theninstall.sh --version v0.69.0on a clean HOMEreceipts/<host>.txtjq -eover the committed fileapr devices— on today's assets that line prints a falsecuda … not-compiledcrates/aprender-serve/models/qwen2-0.5b-q4.ggufis 0 bytesAdopted: what a CUDA asset gate asserts when the host has no GGUF — two tiers
devices --jsonagrees. Cheap, runs on yoga, and would have caught apr devices reports 'cuda unavailable reason=NotCompiled' on a binary that demonstrably runs CUDA — the first screen a user sees is wrong #3545.unmeasured.Adopted: the #3544 receipt schema
{host, arch, version_tested, date, install_rc}plus the fields that let a later reader falsify the claim rather than take it:provenance ∈ {release-asset, crates.io, local-build}and the exact asset filename;sha256_publishedandsha256_measuredas two fields so a match is visible as a match; theapr devices --jsonentriesarray verbatim;generate: {model_sha256, backend_line_verbatim, tokens, wall_ms, output_sane}orgenerate: nullwith areason;driver(nvidia-smi + CUDA runtime) andnproc; and a non-emptyunmeasured[]naming what this host could not prove.install.sh is listed as uncovered. Not "covered by a weak check" — there is nothing to cover. Treat R-6 as the work, not the check.
Outside-in — from the downstream consumer (apex-c8), measured against the published crate
Validation closed: nothing failed. apex pinned
aprender-viz 0.68.2from crates.io —cargo update --preciseclean,cargo check/test --workspacerc 0,cargo deny checkrc 0, and apex#99 green includingpublish-sim(offline--network none). The.cratewas verified independently of the index (sha256cbd7448685d7…, matching Cargo.lock's registry checksum). 0.68.2 also closed a real gap: 0.68.1's.cratehad nosrc/breaks.rsat all, which is why apex was carrying a git patch.aprender-vizpackagesfixtures/breaks/(manifest.json + generate.py) into the published.crate— 396 KB of test fixtures shipped to every consumer, and nobody downstream runsbreaks_golden.rstar tzf aprender-viz-<v>.crate | grep -c 'fixtures/breaks/'is 0. Pure packaging, no API change — the cheapest row in the epicaprender-viz, butuse aprender_viz::…does not compile — it isuse trueno_viz::breaks::extendedcargo doc -p aprender-vizfront page and the crates.io README both state it in the first screenaprender-compute,aprender-quant,aprender-gemm-codegen,aprender-contracts-macrosextendedandpanelsbuilds, andcargo tree -e normalshows none of those cratesaprender-vizis f32, consumers compute in f64, and the narrowing is undocumented and unguardedDataFrame's docs state the f32 domain and what happens outside it (three review lanes flagged mantissa precision loss, point-order retention, subnormal underflow)Facet/Coordhave no runnable example — EV-15 (byte-identical SVG across architectures) and EV-16 (unbranded export) have nothing to build againstexamples/program: study data →panels→Coord::apply→ SVG, compiling against the published crate onlyCaveat recorded by the lane itself: apex#99 is green but deliberately unmerged this session pending an apex-side defect (an EV-2a probe whose crates.io call has no User-Agent → 403, so the probe can never pass). That is apex-side and unrelated to this train; the build and resolve evidence above stands on its own.
noahgift commentedon Sep 20, 2026
0.68.2 train debt — from the train lane (aprender-bf), measured 2026-09-20
The train that just shipped still owes two receipts. Both are the release lane's to write, today, and both are listed here so the debt is visible in the milestone Noah's team reviews rather than living in a session.
docs/build-ledger/on main has 2026-09-16/17/18 and nothing for 09-19/20test -f docs/build-ledger/2026-09-20/338f1d49b-reconcile.jsonand every §6 predicate reads 0DONEline without itledger.py) not writtencheck_multiplatform_dogfood.shexit 0 against committed receiptshostsstep was running at the time of this report; if it discharges, 0.68.2 becomes the first train since 0.67.0 with host receipts, and #3544 changes shape — from "nobody ever ran it" to "make the DEFER fail closed"cascade-publish.shTIERS is not a dependency order (47 non-dev violations), which is why 0.68.2 published from a bundle-localpublish-order.txt+publish_strict.sh. #3459 (move the tag path in-tree) is the vehicle; #3462 is the defectscripts/cascade-publish.sh --order-checkexits 0 against the tagapr runprints no tok/s and noBackend:line on the CPU armOn #3459, the argument in one line: the autopilot (
autopilot.sh,publish_strict.sh,t2_preflight.sh,publish-order.txt) is bundle-local and unversioned, and today a staleSTOP-FOR-OPERATORline in it — contradicting the very spec revision it cited — was corrected by hand at 07:30Z in a file no PR has ever reviewed. A release train whose control flow is not in the repo is not reviewable, and the correction proves it.Deferrals audit: the only conscious deferrals from 0.68.2 with no ticket were the two rows above plus #3486 (qwen3-8b's optional rung, already ticketed). Nothing else was deferred silently — which is the answer #3544 exists to make routine.
19 remaining items
PMAT-3445: no tag while the milestone being cut holds an open item — …
PP-LLAMA-001 row 2: the perf002 prefill-path probe, with refusals tha…
roadmap: push the five tickets minted today (they existed only in a w…
PMAT-3459 part 2: port the tag path into the repo — autopilot.sh + 2,…
PMAT-3587: fail-fast discards verdicts — the default flips, and skipp…
PMAT-3561: the T-0 preflight returned GO when the dogfood crashed — G…
PMAT-3459: the tag step cannot be reached without a clean milestone —…
PMAT-3567: the shapes gate on the FLEET-PINNED pv, on the runner it e…
paiml#3545: apr devices told every user CUDA was unavailable on a bin…
PMAT-3459 + PMAT-3460: register the in-repo tag-path ticket (P0, 0.69…
release(0.69): batch — 15 receipted PRs + fleet-pv andon fix, one CI …
PMAT-3685: apr parity REFUSES (exit 12) a model the CUDA capability g…
release(0.69.1): batch 1 — 20 receipted rows + 3 combination fixes, o…