EPIC: release train 0.79.0 ANY-MODEL — MoE on CUDA, a kernel registry that refuses, imatrix quantization, new arch as config #6572

Plan doc: docs/roadmaps/releases/0.79.0-any-model.md (PR #6569).

Work ticket: #6567. Categorization: docs/roadmaps/releases/categorization-2026-10-30.md (reviewed with this epic; 3 rounds, fix-forward after round 3 per the 15:01Z review cap).

Goal gap, stated first. The goal is that any model llama.cpp runs, apr runs: correctly, on the GPU, from a quant apr itself produced.

Expected green by 12-18:

• the kernel registry refuses what it does not know (M3)
• the resource planner and the tiny-MoE fixture (M2)

Expected yellow: Qwen3-MoE on CUDA (M1). It is expected to run correctly with no fallback, but below 0.5× llama.cpp's decode speed, which scores YELLOW.

Expected yellow or red:

• imatrix quality (M5), new code with no prior measurement
• the new-architecture line budget (M4)

Not in 0.79, stated now:

• AWQ and GPTQ (B-08, B-09) have algorithm-level classifiers only and no quantization pipeline. They are stretch rows, cut first.
• Speculative decoding (C-09) is stretch.
• Vision input (C-12) moves to 0.80: no Rust code loads a vision projector.

Operator rulings applied (not relitigated):

• 0.79.0 is due 12-18 and date-driven: it ships whatever is green; a red cell ships red with its receipt; nothing is waived.
• NOT_MEASURED counts red.
• 1.5-day andon: a late row is split or dropped to 0.80 the same day.
• ≤5 workers, 1 ticket in progress each; capacity 5/day.
• No new Python.
• The planning worker files.

Release: 0.79.0 (new milestone, due 2026-12-18).

How the cells score. The cells below are a release scorecard, not CI checks.

• Releases are date-driven, so no cell blocks a merge or a release.
• Each cell ships GREEN, YELLOW or RED with its receipt, in the colour scheme of the 0.76 epic (ruling 5): exit 0 = GREEN, 3 = YELLOW (ships yellow, counted not-green), 1 or 2 = RED.
• The falsification and coverage tests the rows add are LAB under C324 until promoted under its item 7 (operator yes, three green nights). None goes on a PR or merge-queue path: they run in a nightly LAB lane and are not added to ci/explicit-test-commands.d/, which runs on every PR (scripts/check_model_tests_wired.sh:92-93). Each is named only in a nightly LAB workflow, which satisfies that wiring guard. A cell's receipt comes from an on-demand run on the pinned release binary, not from a count of nightly runs. A cell colour is a scorecard measurement: a red cell ships as red and blocks no merge or release, so scoring a cell on one of these tests gives the test no GATE power. GPU falsifiers go in gpu-lab-nightly.yml, CPU ones in a nightly LAB workflow named on the row.
• A row that is cut leaves its cell RED (NOT_MEASURED), named in the AM-70 report.

Absorbs (become children of this epic):

Carried prerequisites (10, plus #3714 = 11; the carry script moves an item only one release ahead, so at each earlier cut the planner re-milestones it here from backlog; categorization §1 and §6 step 6):

• 0.73.0: apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602, GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759, resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (#3821, second instance) #3850, P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — GH-480 PTX patch injects .reg inside a global array initializer #4096 (children of CUDA quantized kernels: correct block layouts, cache keys and precision #5323)
• 0.74.0: PP-TENSOR-001 T1: typed TensorStorage — a dense consumer cannot receive an MoE placeholder #3433, DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513 (children of EPIC E7 0.74: Any Model #4001); apr: llama.cpp logit-compare verb as the PRM parity.oracle (PRM-001 v2 §4.1) #4444 (child of Qwen3.5 dogfood lane: blessed model, parity receipts at the serving shape, thinking-on closure #5317), the M1 logit oracle
• 0.73.0: CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739 (child of EPIC E6 0.73: Runs Everywhere #3999), the CRUX dogfood set AM-00 reads
• 0.77.0: CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) #3111, PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen3-MoE forward reuse the previous GEMV's input under DP4A #4378 (children of CUDA quantized kernels: correct block layouts, cache keys and precision #5323)
• plus qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714 (0.74.0), above.

Related: #4444 llama.cpp logit-compare verb (0.74.0), used as the M1 oracle; carried above.

1. Baseline (measured, cited)

CRUX rows in this theme (contracts/crux-<row>-v1.yaml, e.g. contracts/crux-B-07-v1.yaml):

The three B-row tests are wired by ci/explicit-test-commands.d/475-apr-cli-kept-cli-tests.cmd.

2. Exit (done_when)

Measurement:

• Host: lambda 4090. Receipts under evidence/any-model-079/<date>/.
• Exit codes: 0 PASS (GREEN), 1 FAIL, 2 NOT_MEASURED, 3 YELLOW. NOT_MEASURED counts red.
• The binary is pinned with scripts/apr_bin.sh.
• n=5 for every speed figure, reported as a median.

Exit cells:

• M1 MoE on CUDA (qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714): on both qwen3moe Q4_K_M files, on lambda (gx10 is reported, not scored):
  • apr run --gpu runs on CUDA with no fallback, proven by a trace line, not a flag.
  • apr qa capability_match and golden_output PASS, not skipped.
  • GPU-vs-CPU parity at ≥64 decode positions: at every position the top-1 token agrees and the logit cosine is ≥ 0.99.
  • Logit cosine against llama.cpp on the same GGUF is ≥ 0.98 through the apr: llama.cpp logit-compare verb as the PRM parity.oracle (PRM-001 v2 §4.1) #4444 verb.
  • GREEN: all four hold and median decode tok/s is ≥ 0.5× llama.cpp's on the same GGUF. YELLOW: all four hold, speed below 0.5×. RED: any of the four fails.
• M2 Colibri (EPIC E6 0.73: Colibri CRUX — expert streaming, auto-fit planner, tiny-MoE fixtures #6481): GREEN when M2a and M2b both hold; otherwise RED. M2c is stretch and reported with no colour, so cutting it does not change M2.
  • M2a CH-1: apr serve plan chooses quant, context and cache from a RAM/VRAM/disk budget. A ≥3-budget matrix test passes, and a planted over-budget case is refused.
  • M2b CH-2: a tiny-MoE fixture, a CPU oracle, a *_bad negative fixture the oracle rejects, and a Rust replay simulator.
  • M2c CH-3 (stretch), EPIC E6 0.73: Colibri CRUX — expert streaming, auto-fit planner, tiny-MoE fixtures #6481's done_when 1 verbatim: today's mmap path is measured first as the control; then, on a Qwen3-MoE 30B-A3B Q4_K_M GGUF on lambda (cell X1), CPU decode (no GPU offload, so the weights are host-resident) under cgroup memory.max at 60% of the GGUF size, ≥50% of the unlimited CPU mmap baseline's tok/s on the same host, n=5; output token-exact against the in-RAM path on the tiny fixture (X2).
• M3 Kernel registry (KREG-001: SHACL kernel registry — loader refuses any unregistered (qtype, layout, arch, shape, precision) kernel #4539 phase 1, CUDA quantized kernels: correct block layouts, cache keys and precision #5323 exit): GREEN when all four hold; otherwise RED.
  • The registry and its shapes are in-tree, with pv validate green. One already exists, crates/aprender-serve/kernel-registry.json (KREG-001, read by the wgpu adapter); AM-30 extends it to the CUDA kernels and shapes rather than starting a second file.
  • The kernel selector returns an error for an unregistered (qtype, layout, arch, shape) and never falls back silently.
  • KREG-001: SHACL kernel registry — loader refuses any unregistered (qtype, layout, arch, shape, precision) kernel #4539's two falsifiers each exit non-zero on their planted input (col-major Q4K on row-major GGUF; untagged raw_q4k) and zero on the clean input.
  • A test proves every dispatchable kernel has a registry row; the count is derived, not hardcoded.
• M4 New architecture as config (EPIC E7 0.74: Any Model #4001 exit):
  • One architecture apr refuses today, chosen at AM-00 from measured refusals (apr qa / apr run on the CRUX dogfood set, CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739), lands.
  • The boundary: AM-01's contract lists, by path, the config and TRAITS-table files. The delta is the non-test Rust lines added outside those paths, measured by git diff --numstat on the landing PR(s).
  • GREEN: apr qa golden_output PASS on it and a delta ≤300 lines. YELLOW: golden_output PASS and a delta >300 lines. RED: golden_output fails or it does not land.
• M5 imatrix (B-07):
  • apr quantize computes an importance matrix from a fixed calibration text and uses it for Q4_K_M.
  • Base model: Qwen2.5-Coder-1.5B (the 0.76 C1 base).
  • Three GGUFs: apr-imatrix-Q4_K_M, apr-no-imatrix-Q4_K_M, llama.cpp-imatrix-Q4_K_M. Both importance matrices (apr's and llama-imatrix's) come from the same checked-in calibration text, read by its sha256. All three are scored by the same evaluator, llama-perplexity (pinned), on the same held-out text.
  • GREEN: apr-imatrix PPL ≤ llama.cpp-imatrix PPL × 1.01, and apr-imatrix PPL < apr-no-imatrix PPL.
  • YELLOW: apr-imatrix beats apr-no-imatrix but misses the 1.01 bound. RED otherwise.
  • The calibration and held-out texts are checked-in files with their sha256.

3. Cells

4. Children

New rows are AM-nn, each ≤1 day, filed just in time (categorization §5). #6481's CH-2 (M) and CH-3 (L) are split here to fit ≤1 day.

W0 Measure first:

• AM-00 refusal census on the CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739 set; picks X4.
• AM-01 contract contracts/any-model-079-v1.yaml (M1–M5, including the M4 boundary path list); pv validate green.
• AM-02 X1 baseline: today's rc and refusals, plus llama.cpp tok/s.

W1 MoE CUDA (M1):

• AM-10 qwen3moe expert dispatch on CUDA.
• AM-11 router top-k on GPU.
• AM-12 GPU-vs-CPU parity at ≥64 positions.
• AM-13 apr: llama.cpp logit-compare verb as the PRM parity.oracle (PRM-001 v2 §4.1) #4444 logit-cosine receipt.
• AM-14 apr qa gates present for qwen3moe.
• qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714 closes on AM-10..14.

W2 Colibri (M2):

• AM-20 CH-1 plan auto-fit (choose quant).
• AM-21 CH-1 budget matrix + over-budget falsifier.
• AM-22 CH-2 tiny-MoE fixture + _bad.
• AM-23 CH-2 route-trace format + replay simulator.
• Stretch: AM-24 CH-3 bounded expert cache, AM-25 CH-3 cgroup receipt (with the mmap control).

W3 Registry (M3):

• AM-30 registry + shapes.
• AM-31 selector check.
• AM-32 the two falsifiers.
• AM-33 coverage test (derived count).
• Depends on whichever of the 10 carried prerequisites are still open.

W4 Arch + imatrix (M4, M5):

• AM-40 X4 config + delta.
• AM-41 X4 golden output.
• AM-50 importance-matrix collection in apr quantize.
• AM-51 imatrix-weighted Q4_K_M.
• AM-52 PPL receipt for the three GGUFs.

W5 Report:

• AM-70 docs/BEATS.md and contract discharge_status updated from receipts only; names every cut row.
• AM-71 opens the 0.80 residual tickets (B-08, B-09, C-09, C-12, and whatever M1 speed gap remains).

Stretch (cut first):

• AM-60 GPTQ pre-quantized checkpoint import (B-09).
• AM-61 AWQ pre-quantized import (B-08).
• AM-62 speculative decoding wired into apr run (C-09).

Flow lane: ≤5 tickets taken at filing time from #6520 and #5923, both standing epics that the categorization moves from 0.77.0 to 0.80.0.

Cut order (first to 0.80):

1. AM-60, AM-61, AM-62
2. AM-24, AM-25
3. the flow lane
4. M4 (AM-40, AM-41), last, and only when a row passes 1.5 days (the split-or-drop ruling)

M1, M3, M2a/b and M5 base rows are never cut by this capacity order. The 1.5-day andon still applies to them: a late row is split or dropped to 0.80 the same day, and its cell ships red with its receipt.

5. Schedule

• Capacity: weekdays only. 12-11…12-17 holds 5 weekdays (12-11, 12-14, 12-15, 12-16, 12-17; 12-12/13 are a weekend) × 5 workers = 25 ticket-days.
• Load: 23 base rows (AM-00..02, 10..14, 20..23, 30..33, 40, 41, 50..52, 70, 71), 5 stretch rows, ≤5 flow, plus any carried prerequisites. Base alone nearly fills 25, so the stretch rows and the flow lane are expected to be cut.
• Critical path: AM-02 → AM-10 → AM-12/13. AM-10's code starts 12-14 beside AM-02 on a non-lambda CUDA host; only its receipt waits for AM-02.
• Long pole: W1, a new CUDA MoE forward.
• Lambda is serial across releases. 0.77 holds it to 12-03, 0.78 to 12-10, 0.79 from Monday 12-14. One release's slip moves the next one's lambda days, and that is reported.

6. Risks

• MoE CUDA forward is larger than 5 rows. AM-10 is the riskiest row. The 1.5-day andon splits it on day 2; M1 then ships red with its receipt.
• 24 GB fit. 30B-A3B Q4_K_M at ~18 GB plus KV and workspace. If it does not fit at the receipt context, the receipt states the context used; reducing it is allowed, offloading to CPU is not (it would be a fallback).
• imatrix new code. No prior measurement. A missed 1.01 bound ships YELLOW. The evaluator is llama.cpp's for all three GGUFs, so the comparison is not apr grading itself.
• Prerequisite carry. Ten earlier-release prerequisites plus qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714 (11 in all); each costs a ticket-day; beyond what the cut order frees, a cell ships red under the andon and is named in the cut report. M3 is not cut.
• Lambda contention with 0.78. 0.78's final receipts run 12-10; AM-02 runs Monday 12-14.

7. Owners

• ≤5 workers, 1 ticket in progress each; the cop assigns at filing.
• Suggested split: W0+W5+flow; W1; W2 (aprender-w5032, EPIC E6 0.73: Colibri CRUX — expert streaming, auto-fit planner, tiny-MoE fixtures #6481's owner); W3; W4.

Review debt (Haiku, round 3: FAIL; shipped fix-forward under the 15:01Z review cap, not a blocker)

• (fixed forward 10-30, needs re-review) line 54: issue 4444 (M1 oracle, AM-13) is OPEN at milestone 0.74.0 under parent issue 5317, but it is not in the carried-prerequisites list (lines 48-52). At the 0.74 cut it goes to backlog and nothing carries it, so the logit-cosine criterion (line 94) and AM-13 (line 139) have no oracle.
• (fixed forward 10-30, needs re-review) line 3: Header cites the categorization as plan2/cat-final-307b8bc968a6.md. That file exists only under plan2/old/. The current categorization is plan2/cat-final-a0101943a8f8.md.
• (fixed forward 10-30, needs re-review) line 181: "M1, M3, M2a/b and M5 base rows are never cut" disables the andon drop for those rows. The ruling (line 25) says a late row is split or dropped to 0.80 the same day, with no exemption.
• (fixed forward 10-30, needs re-review) line 35: The LAB claim says the new falsification tests stay out of ci/explicit-test-commands.d/. That is not sufficient: scripts/check_model_tests_wired.sh (lines 2-3, 77-79, 92-94) fails any model-tests-gated integration target that is not named in a workflow or .cmd fragment. A new falsifier written with #![cfg(feature="model-tests")] (AM-21, AM-22, AM-32) would turn every PR red.
• (fixed forward 10-30, needs re-review) line 100: Baseline (§1) and M3 ignore the in-tree kernel registry crates/aprender-serve/kernel-registry.json (KREG-001, read at runtime by realizar::kernel_registry, row_major only). M3's first criterion ("the registry and its shapes are in-tree") may already be partly met, and AM-30 "registry + shapes" may duplicate it.
• (open) line 194: Filing 28 new rows at 12-11 (line 161 of the categorization, "0.79's at 0.78 close") breaks the ≤100 open-ticket cap unless the open count is already under 72 on that date. The categorization's own §4 gives a net 206 open after the plan. The epic never states the expected open count or the cap check at filing.
• (fixed forward 10-30, needs re-review) line 106: AM-00 takes its input from the CRUX dogfood set issue 3739, which is OPEN at milestone 0.73.0 under issue 3999 (not 0.80). It is not carried, so it goes to backlog at the 0.73 cut while the epic depends on it.
• (fixed forward 10-30, needs re-review) line 112: M5 compares apr-imatrix with llama.cpp-imatrix, but the epic does not say llama.cpp's importance matrix is computed from the same checked-in calibration text. The comparison is then confounded.
• (fixed forward 10-30, needs re-review) line 99: M2c's tok/s device is not specified (CPU mmap or GPU), so "≥50% of the unlimited mmap baseline" is ill-defined.
• (open) line 97: M2a and M2b, and M3 (lines 102-104), score planted-failure falsifiers that are LAB (line 35), and the cells do not say which lane run supplies the 12-17 receipt. A falsifier added on 12-14 has at most 3 nightly runs before 12-17, and the 7-day stop rule is not addressed.
• (fixed forward 10-30, needs re-review) line 173: "issue 6520 and issue 5923, both standing epics that stay in 0.80.0". Both are 0.77.0 today (open-issues.tsv:182, :120); the categorization moves them to 0.80.0. "Stay" is wrong.
• (open) line 34: Unverified in this lane: the "ruling 5" citation to the 0.76 epic, owner aprender-w5032 for issue 6481 (no owner in the snapshot), and issue 6567 (not in the 10-30 open-issues snapshot). The GitHub reads needed to confirm them were not possible here.

Folded in by the 0.73–0.76 triage (2026-10-31)

These were open tickets; to keep the repo at 100 open or fewer they now live here as checkboxes. Reopen a ticket when this epic schedules it.

• issue 3714 (was 0.74.0): qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --