EPIC E3 0.72: Don't Leave Behind #3994

owner: aprender-w5032 (re-owned 2026-10-19 by aprender-triage-worker at the cop's word: previous owner session absent; picks up children only as batch constituents after its current timed task)

Spec: APR-EPIC-001 §2 · Epic: E3 · Train: 0.71 · Successor at cut: E7

Theme

Don't Leave Behind

done_when

every (Qwen × quant × verb × backend) cell is Pass or Refused{removed_by}; the universe is read from tensor headers; 0 silent CPU fallbacks under forced accelerator.

Budget

At most 9 open children (docs/roadmaps/epics.yaml). Repo-wide cap 100 open issues; andon at 95.

Absorbs

#3062, #3597

Prior body

Theme: Don't Leave Behind (operator decision, 2026-09-23)

The operator picked scope A: "lets put all into A".

Bar. Every Qwen model that fits on a fleet host works, in every format and quant, through every verb (run / chat / serve / code / qa), on every backend and silicon class. Zero known REDs, no deferred cells, no RED-model exceptions.

Out of scope. qwen35moe 35B-A3B (#3977, a new SSM+MoE architecture) goes to a later release (pending operator confirmation). Also out: performance and new features.

Must-carry (0.69.1's known REDs + today's early-ladder findings)

• Low-bit quant admission + CUDA correctness: IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker #3963, IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker #3953, Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960
• IQ4_XS thinking never closes: Qwen3.5-0.8B-IQ4_XS never closes its think block while Q4_K_M does: CUDA IQ4 GEMV or real quant damage? #3951
• qwen3moe chat/serve/code: qwen3moe CUDA reaches run+qa only: chat rc 8, serve 501/500, code rc 1 — serve loads no mapped model and would route MoE to the CPU path #3987
• GPU correctness underneath: F2 GPU validation fails OPEN: returns true ('assume GPU is fine') when the CPU reference or probe is unavailable — 'F2 validation PASSED' can mean nothing was validated #3973 (F2 fails open), Q4_K GEMV kernels: FusedKVHwDp4aQ4KGemv generate_ptx returns "" but is launched (q4k_mwv_gemv.rs:602); Dp4aSIMDQ4KGemv entry-name mismatch #3976 (Q4_K GEMV empty PTX), GPU/CPU f32 APR forward diverges at layer-0 QKV (std 4.44 vs 0.30); gpu_cpu_trace_compare fails 91.9% L2 once it compiles #3975 (GPU/CPU f32 APR divergence)
• Verb surface: apr code hardcodes 'apr serve --gpu' (no CPU lane, no --max-tokens/thinking flag) and picks a colliding port 19384+pid%1000 #3978 (apr code hardcodes --gpu), apr serve: APR-CPU fallback and safetensors routers have no GET / route index and SSE ends without finish_reason/[DONE] — every .apr/.safetensors serve cell is RED #3979 (.apr serve routes)

Work to size before committing a date

1. Probe (first, after the 0.69.1 freeze): the 8 contract rungs through --features wgpu on lambda and on a Mac. Report the red count. This sizes the unmeasured half.
2. Ladder: a CPU leg for every inventory model, a wgpu backend leg, and a mac host in contracts/model-capability-ladder-v1.yaml.
3. Release time: CPU legs across the inventory roughly double the sweep, which is 2.5–3 h CUDA-only today. The GPU-lock work (#3986 P1/P3/P5) matters here.
4. Re-bucket the 187 open issues in this milestone into 0.70 / 0.71 / 0.72 / backlog against this bar (operator approval before any move).

Open decisions

• Which Mac is the ladder host?
• 0.70.0: qwen35moe (Qwen3.5-35B-A3B hybrid SSM MoE) CUDA forward #3977 is out of 0.70.0: confirm.
• The milestone is due 2026-09-26. Set a real date once the probe reports.

Members not listed above (132, every other open 0.71.0 issue)

Rule (operator 2026-09-24, applied by the cop aprender-cf): a milestone is an epic only if this issue lists its members, and tickets without an epic go to backlog. This list was derived from the milestone's open issues on 2026-09-24 by aprender-dd (#4159).

• No exceptions: remove the registry: true provability exemption — 500 exempt, but 434 already prove something #2557
• block v0.1.6 will be rejected by a future rustc and sits under wgpu->metal, the only macOS GPU backend #2572
• dispatch.rs tells users only the root crate can enable CUDA; the nightly builds it via -p apr-cli #2586
• compute_class() cannot return 'metal' that the matrix demands of mini, and apr-cli's wgpu feature reports a GPU backend it does not enable (I-2) #2779
• apr prefills at decode rate on CPU: no batched prefill outside src/cuda/, 6.5x slower than llama.cpp on the same box #2787
• apr bench's compute_class() shells to nvidia-smi -L and reports cuda if a GPU is PRESENT — 'hardware present' is what I-2 forbids by name #2792
• wgpu decode on Metal is 0.123 tok/s (8.14 s/token) — mini's W1 cell cannot carry a meaningful metal ratio #2826
• R-0: [premises discharged: S0-7 CONFIRMED, S0-14 CONFIRMED] BackendRegistry: enumerate cpu/cuda/wgpu at startup from the machine (dlopen libcuda.so.1, wgpu adapters, cpu always); one pr #2904
• R-2: [premises discharged: S0-14 CONFIRMED, S0-15 measured] W-I: cuda joins default features iff S0-14 passed on all four hosts (D-9) and only behind R-0; apr serve run --gpu refuses vi #2905
• R-5: apr-* assets for the 5 nightly.yml targets attached to every tagged release, built per D-10; sha256 + minisign-signed manifest (S0-19: public-key scheme); release cut as prerelease #2908
• R-6: one-line installer served at /releases/latest/download/install.sh: detect OS/arch, download the matching asset from the latest non-prerelease tag, verify .sha256 and the minisign-s #2909
• R-7: README leads with the installer one-liner; cargo install aprender second; the backend line generated from the registry kinds #2910
• B-A1: hosts.intel.wgpu cell in perf-matrix.yaml: compute_class wgpu, status UNMEASURED with owner, expiry anchored on D-3 (S0-3 found two AMD W5700X + llvmpipe on intel) #2928
• B-G1: check_backend_firstclass.sh: clauses 1-7 per backend computed from the tree (workflows, BACKEND_VALUES, perf-matrix.yaml, receipts, dogfood, formats, pins); prints N/7; FAIL if any #2929
• B-W0: quantized-weight upload path in serve: Q4K_GEMV_SHADER (already consumed by compute wgsl_forward.rs:424) wired into OwnedQuantizedModelWgpu; chunked to maxStorageBufferBindingSize; #2951
• B-W1: wgpu lane on intel: the job, adapter selected by name/type and printed (two AMD W5700X, RADV; llvmpipe present so the rasteriser falsifier is live — S0-3/S0-4) #2952
• B-W2: Vulkan comparator pin (llama.cpp Vulkan build) in llama_pin.toml for the intel lane #2953
• B-W3: PP-26 witnesses on the wgpu lane (m=1 vs m=c) #2954
• B-W4: W1 receipt on the wgpu lane; "wgpu loses to CPU" is a recorded outcome #2955
• B-W5: clauses 5-6 for wgpu in check_backend_firstclass.sh (dogfood + formats) #2956
• B-M1: metal as a Ready registry entry on mini via the wgpu-Metal adapter (no MSL dispatcher exists — S0-12); NA -> UNMEASURED only when the registry prints Ready on mini (#2841 by constr #2957
• B-M2: llama.cpp-Metal + Ollama comparator pins on mini #2958
• B-M3: PP-26 witnesses and the W1 receipt on mini (#2826: the f32 wgpu path is 0.123 tok/s; needs B-W0) #2959
• B-M4: clauses 5-6 for metal in check_backend_firstclass.sh #2972
• B-S1: 0.5B GGUF+APR decode under wasmtime (wasm bound computed from bytes per receipt; W1 excluded by address space) #2973
• B-S2: browser harness for the wasm32 build #2974
• B-S3: wasm size budget + self-ratchet #2975
• B-S4: clauses 5-6 for wasm32 in check_backend_firstclass.sh (clause 4 N/A at m=1, printed) #2976
• B-A2: vendor-neutral Vulkan lane receipt on AMD silicon (intel W5700X); ROCm/HIP row with its blocker #2977
• R-0b: backend resolution reads the registry — accel.rs/serve/dispatch stop consulting cfg!(feature=…); --backend/--gpu/--device refused by enumeration with the registry's reason; BACKEND_VALUES widened to the REG-11 kinds with runtime Unavailable(NotCompiled) (split of R-0 per the design quorum 2026-09-06) #3002
• PP-066 G-10b: analyser pin guard — every pmat reference under scripts/ and workflows resolves through scripts/pmat_bin.sh (baseline 281, shrink-only) #3013
• PP-066 R-8: nightly-latest-dogfood — install from latest on four hosts, apr devices --json, C14; drift files an issue #3019
• R-0b follow-up: a forced accelerator whose runtime attempt fails must refuse BEFORE generation in realizar (decompose try_wgpu_generate cog 49 / try_apr_wgpu_inference cog 69) #3042
• KEY: keys/apr-release-minisign.pub on main — the release-asset signing key R-5 signs with and R-6 pins (R-5 blocked_by KEY) #3045
• aprender-gpu: driver, kernels, memory AND ptx are all #[cfg(feature = "cuda")] — 2,160+ tests incl. the GH-480 rewriter's and the PTX emitter's run ONLY in the gx10 nightly's perf053 filter #3067
• cuda-unit on yoga (RTX 4060): test_cublas_gemm_f16_training_shape asserts a 4090 floor — 'must exceed 50 TFLOP/s, got 23.6' is the only red in 2604 cuda tests #3135
• CUDA Q6K GEMV panics on an uninitialised workspace buffer: two examples die in library code on a working 4090 #3178
• aprender-zram-core's cuda feature has never compiled: unresolved trueno_gpu imports in library code, dark since before v0.66.0 #3179
• cuda-unit on yoga (RTX 4060 8 GB): cta64_vs_cta32_vs_cublas_fp16 dies with CUDA_ERROR_OUT_OF_MEMORY — the advisory job has been red on every PR today #3184
• gpu-quick: the CUDA test binary SIGSEGVs at teardown on gx10-eph after every test passes (1 in 5 runs) #3226
• cuda-unit OOM-kills rustc on yoga: no CARGO_BUILD_JOBS cap on a 22-core / 30 GB box #3234
• apr-format golden_v2 F32 writer emits 516 bytes against a 1092-byte golden — and the target is dark in CI #3235
• cbtop renders 0 bytes of memory on macOS: read_meminfo's ENOENT is unwrap_or_default()'d into zeros #3236
• 1,424 of 1,469 integration targets run only when a PR touches their own crate — four of tonight's defects trace to this #3239
• A dev-dependency's feature arms a sibling's runner-gated test: the BSE-17 quick tier ran three browser falsifiers on a browser-less runner #3242
• T-3 promotes on asset PRESENCE, not on asset IDENTITY — nothing binds a released binary to a passing parity test #3283
• aprender-gpu: every cuda test passes and then the test binary SIGSEGVs at teardown — exit 101 reads as a failing assertion #3287
• FP8 batched prefill fails CPU parity at the post-prompt decode step — catastrophic on per-head-QK-norm models (Qwen3-8B cos −0.10), marginal on Qwen2.5 (0.92) #3483
• VERIFY-001: run the 132 Kani harnesses in CI (proof credit from runs, not declarations) and pilot Verus on one dequant/parser function #3495
• Qwen3.5-35B-A3B (hybrid Gated DeltaNet + routed MoE): CPU reference forward + CUDA path + ladder rung on both hosts (UD-IQ4_XS 17.5 GB) #3512
• qwen3-8b Q4_K_M on CUDA sm_89 (24 GB): 23.1 GB peak VRAM for a 5 GB model, apr qa golden_output Empty output — optional ladder rung red since 0.68.1 #3532
• Name the Qwen model 0.69 blesses and commit its parity receipt at the SERVING shape — our gates are all single-stream #3555
• First-party dogfood: Qwen3.5 at a Pareto-optimal size — publish the selection rule and the evidence a consumer re-derives it from #3558
• PP-LLAMA-001 §12: a substring test cannot tell a citation from a claim — five live rows discharged themselves by quoting one #3573
• No committed parity receipt names an external comparator — the 0.98 threshold's own basis is apr-vs-apr self-consistency, not just the Qwen 3.5 rows #3576
• apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595
• apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602
• 51 contract test files say 'DO NOT EDIT — regenerate with pv probar --binding' and pv probar cannot produce any of them — obeying the header replaces real tests with ignored stubs #3630
• pv extract emits a PARTIAL graph on an unparseable binding.yaml (one dropped quote → 356 triples silently gone, --check agrees with itself) — it must refuse, naming the line #3640
• The repo-graph freshness gate compares committed == fresh and passes when both are 356 triples short — assert a floor from binding.yaml entries / main's count, not self-consistency #3641
• pv shapes: the LONE-contract venue declines with no report — by_shape and declines exist only on the directory path #3648
• apr convert --quantize q4k fails on a plain Q4_K_M GGUF: the Q4K passthrough's detector admits Q5_K, its writer refuses it, and the refusal propagates instead of falling through #3663
• Thread the DECLARED unknown token (tokenizer.ggml.unknown_token_id / tokenizer.json unk_token) through every tokenizer loader — #3609 uses a by-name interim #3675
• BPETokenizer greedy encode matches Latin-1 characters as byte-level GLYPH tokens: "é" encodes as byte 0xE9 and decodes to U+FFFD (every byte-level GGUF served via AppState) #3677
• Qwen3.5-9B Q4_K_M on CUDA at ~12k-token context corrupts identifiers it quotes from the prompt (as_millis→as_mi, Backend→Backen) and runs at 1.0 tok/s; parity passes only at 79 positions — model weakness or long-context CUDA divergence is UNMEASURED #3693
• aprender-gpu PTX builder: shfl_xor_f32 emits 'shflbfly' — ptxas rejects it; no emitter arm for PtxOp::ShflBfly #3746
• Every CPU-reference validator compares an FP32-activation GPU against a Q8_K-activation CPU — on Qwen3 outlier tokens it rejected an exact GPU (cos 0.873 vs 1.000000 against FP32); measure CPU Q8_K vs FP32 across the fleet #3751
• #3745 S3: hand-list guard — refuse literal apr verb/flag enumerations in release surfaces (vocabulary from apr surface --json) + migrate the ladder/B1 cells to derived cells #3752
• Parity lane refuses on intel and mini: the pinned llama.cpp (d1d3c3396) has no llama-bench/llama-server there — REPORT for 0.69.1 (first-green of #3731) #3756
• GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759
• apr inspect --json 'quantization' is wrong: Q6_K for a Q4_K_M GGUF, None for every .apr, and inspect FAILS on IQ4_XS ('Unsupported dtype 23') #3762
• #3712 row A2: the Q4_K universe is read from each file's tensor header (apr tensors --json), never a *Q4_K* filename glob; the judge recomputes membership #3763
• cargo check -p apr-cli --features wgpu fails: finetune.rs imports entrenar's WgpuInstructPipeline / wgpu_training items that don't exist (5 errors) — a declared feature that cannot build #3779
• C14 parity at long context: apr parity CPU vs CUDA at ≥8k and 20k positions on lambda and gx10 (night lane), and a long-context row in the C14 corpus (split from #3693 done_when 2) #3780
• #3712 row B2: the cells producer — a resident runner over S2's derived cells (cell_id rows, output_sha256, wall_ms load vs measured), ≤ 8 h/host, a cell tests its own command shape #3783
• validate.rs extract_model_paths is a hand-typed list of model-loading commands — derive it from the S1 ModelPath/ModelRef markers (#3745 follow-up) #3799
• The CPU default (Q8_K activations) changes the greedy answer on 4 pairs on lambda and 7 on gx10, different sets — the FP32 default is blocked on aarch64 having no NEON FP32 Q4_K dot #3811
• transformer_layer_indexed hardcodes the Q4K kernel for Q/K/attn_output under a comment claiming exhaustive dispatch (silent corruption, currently masked) #3822
• q8_gemv_tests.rs: 15 tests, 4 asserts — dtype.rs whitelists Q8_0 onto the GPU as a 'verified GPU GEMV kernel' that had no numerical test at any shape #3823
• gpu_state_isolation: fails on coder-0.5b, and its own message truncates both outputs to an identical prefix so it never shows the difference #3824
• try_wgpu_generate's doc claims "Proven: cosine=0.999863 on Blackwell sm_121" — #3757 measured 0.955046 on gx10, which IS Blackwell #3827
• THE UNIVERSE IS A GLOB: 10 of 28 held models are never measured, because inventory.patterns only matches q4_k/q4k — incl. a Q4_0 MoE and an unmeasured 35B Qwen3.5 #3846
• resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (#3821, second instance) #3850
• Qwen3.5-4B-Q4_K_M: deterministic GPU/CPU divergence at exactly one interior position (21) on the hybrid forward #3851
• owned_fused_matmul refuses IQ4_NL while its own message claims IQ* is supported — the refusal overclaims its coverage #3852
• falsification_tests[].test does not bind at all — a prose test name is decoration, and prose is the majority shape in contracts/ #3858
• 477 YAMLs live in crate-local contracts/ dirs that pv lint has never read — and a sample says some would fail if it did #3860
• contracts/apr-cli-commands-v1.yaml's top-level command list is compared to nothing — the de-facto registry is a hand-coded Rust vec #3862
• check_readme_claims.sh --self-test: a must-RED row returns GREEN, because two counters measure different populations and the census carries no sha to date itself #3866
• README.md is not ungated — it is gated in a REGION, and the boundary is invisible in the rendered file: enumerate every numeric claim, each gated or exempted with a reason #3867
• tensor-layout-v1.yaml asserts a compile-time guarantee that is FALSE — and the copy that ships to crates.io carries it #3868
• IQ4_NL (ggml type 20) has no dequant path in realizar — two Qwen2.5 inventory models cannot generate on either backend #3869
• Re-apply #3784's binary-identity fix as a design: the ladder receipt names the CHECKOUT HEAD, not the binary that ran #3871
• the ladder's log truncates diagnostics mid-number ([:70]), producing a complete-looking WRONG value — plus one genuine rc misclassification #3872
• the golden-output gate reports a GPU failure while asserting a CPU pass it never measured — it turned a model that fails everywhere into a CUDA blocker #3873
• ladder receipt: bytes describes the symlink, sha256 describes the model — one row, two objects #3876
• ladder receipt: a dangling symlink records "sha256": "" — an identity claim of nothing #3878
• golden gate: the thinking-ON leg sends a prompt in NEITHER mode the model declares — it deletes the think block where the model opens it #3882
• apr-cli ships 9 INVALID contracts that nothing reads, and no gate looks at them — pv lint never leaves the root contracts/ dir #3888
• apr kernel-explain reports ProofLevel::Proven from a SUBSTRING — nothing runs kani, and fixing the shipped contracts would make six kernels falsely claim Proven to users #3890
• two surfaces report quant=Q4_K for .apr models that contain no Q4_K tensors (f16 and BF16 measured) #3893
• gpu_cpu_trace_compare compiles again and fails: GPU L2 diverged 91.92% from CPU on a synthetic fixture (cause undetermined) #3895
• a ladder row red only on serve prints reason 'unknown' — #3886 moved the verdict and left the explanation behind #3901
• Every BF16 SafeTensors import produces a .apr that cannot run on the GPU: wgpu has no type-30 dequant and the CUDA path is never selected #3908
• An .apr records nothing about what built it: 'stale artifact or live defect' is unanswerable from the file, and the provenance fields exist but are never populated #3910
• apr chat never consults the tokenizer the .apr carries — four filesystem locations, none of them the model #3911
• apr diff measured a 197-of-338-tensor corruption and exited 0: --values cannot fail, --threshold is silently ignored there, and --quant-roundtrip cannot read a quantized model #3913
• The thinking-ON leg exists twice and has now drifted twice: the hybrid copy takes neither #3907's budget resolver nor #3914's template referee #3919
• #3911's embedded tokenizer splits every ChatML control token into six pieces — 32 ids where the reference gives 18 #3920
• RULE A VIOLATION: a Q4_K .apr produces garbage on CUDA on BOTH hosts while answering correctly on CPU — rc=0, no fallback, recorded GREEN #3922
• AprV2ModelCuda is Q4K-only by construction and wrong for Q4K — no configuration does useful GPU work #3923
• A GPU/CPU correctness gate exists and only the infer module uses it — enumerate every entry point #3924
• the ladder judges the terminal, not the model: apr chat's own separator reads as gibberish and reddens every rung #3925
• apr run is the last verb whose generated text nothing judges — and qa's golden leg runs once per rung, not once per backend #3928
• golden_output's GPU leg judges GGUF only, so every .apr rung gets a CPU-only verdict that reports as passed #3931
• the ladder's serve /health window is a fixed 90s, so the 27B is red for load time while passing capability_match and golden_output #3936
• A CUDA chat cell can say rc=0 for a chat that ran on CPU: chat cannot refuse a forced accelerator, and the producer discards the fell_back it does report #3937
• check_model_ladder.sh binds the model file's sha and the release version, but never the binary that produced the receipt #3940
• PROVABILITY-001 requires every kernel contract to declare kani_harnesses, and nothing in this repo runs kani — the schema mandates the declaration #3890 had to neutralise #3941
• Decide: should golden_output judge .apr on the GPU, or formally delegate that to chat/serve? (the gap #3931 made visible) #3942
• aprender-core has no IQ dequantizer, so tensor_contract cannot inspect three release-blocking models — port from the validated aprender-serve decoders #3947
• golden thinking-ON leg discards its backend (_used_gpu) and passes an empty <think></think> as thinking #3961
• GPU whitelist admits a quant TYPE on one model's shapes — admission must key on per-shape device conformance (census-derived, #3945) #3968
• HF cache blob for Qwen2.5-Coder-1.5B-Instruct is a rewritten float32 file under upstream's sha256 name — hf+vLLM answer garbage, judge scored it GREEN #3971
• 17 more files build the dense GGUF CUDA model with no qwen3moe route — audit each (derived from #3987's guard) #3992
• Process rule: llama.cpp fit (llama-fit-params) is the required placement gate for every model certification/test until apr has a parity fit tool #4016
• apr run -v PANICS on a non-ASCII prompt: formatted_prompt log slices a str at byte 200 (not a char boundary) #4018
• apr thinking-ON closes </think> far less often than llama.cpp on the official template (0.8B Q4_K_M: 1/13 vs 5/13) — possible long-generation divergence #4019
• 0.71: qwen35-0.8b-q4km golden thinking-ON never closes on apr (greedy near-tie path, no localized defect) — shipped as known RED in 0.69.1 #4030
• binary-debt #5: one apr entry point (#2585) + fold aprender-ptx-debug, aprender-explain, apr-qa, apr-qa-readme-sync, apr-corpus-ingest into apr subcommands #4062
• clippy debt ratchet 2/5 (0.71): qa-runner, first half (~560) #4154
• unwrap sweep: 2137 banned .unwrap() sites across 11 workspace crates (qa-runner 1038, test-cli 444, graph 235, apr-cli 143, …) — take clippy_member_baseline's disallowed_methods keys to zero #4161

Parked

Closed over budget (APR-EPIC-001 rule 5). Reopen = pull (rule 7).

• F16/BF16 GGUF CPU matmul has no SIMD path — unquantized models appear to hang #3076 F16/BF16 GGUF CPU matmul has no SIMD path — unquantized models appear to hang — over E3 budget (rank 11/79)
• [~] CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) #3111 CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs — over E3 budget (rank 10/79) (reopened 2026-09-25, retro quorum split)
• CUDA Q6K GEMV panics on an uninitialised workspace buffer: two examples die in library code on a working 4090 #3178 CUDA Q6K GEMV panics on an uninitialised workspace buffer: two examples die in l — over E3 budget (rank 21/79)
• llama2-train panics on a config file the repo ships: the path is relative to the crate, cargo run is not #3180 llama2-train panics on a config file the repo ships: the path is relative to the — over E3 budget (rank 23/79)
• apr-format golden_v2 F32 writer emits 516 bytes against a 1092-byte golden — and the target is dark in CI #3235 apr-format golden_v2 F32 writer emits 516 bytes against a 1092-byte golden — and — over E3 budget (rank 76/79)
• cbtop renders 0 bytes of memory on macOS: read_meminfo's ENOENT is unwrap_or_default()'d into zeros #3236 cbtop renders 0 bytes of memory on macOS: read_meminfo's ENOENT is unwrap_or_def — over E3 budget (rank 77/79)
• resolve_base: grep -q SIGPIPEs the git rev-list feeding it, so under pipefail the behind-the-tip arm is unreachable #3353 resolve_base: grep -q SIGPIPEs the git rev-list feeding it, so under pipefail th — over E3 budget (rank 78/79)
• aprender-orchestrate: test_find_apr_binary asserts the HOST has apr on PATH — an environment assertion in a --lib test #3464 aprender-orchestrate: test_find_apr_binary asserts the HOST has apr on PATH — — over E3 budget (rank 79/79)
• qwen3-8b Q4_K_M on CUDA sm_89 (24 GB): 23.1 GB peak VRAM for a 5 GB model, apr qa golden_output Empty output — optional ladder rung red since 0.68.1 #3532 qwen3-8b Q4_K_M on CUDA sm_89 (24 GB): 23.1 GB peak VRAM for a 5 GB model, apr q — over E3 budget (rank 14/79)
• apr run emits no tok/s — every throughput claim from it is wall time including model load (post-publish 0.68.2) #3542 apr run emits no tok/s — every throughput claim from it is wall time including m — over E3 budget (rank 15/79)
• apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595 apr chat hard-skips CUDA for every qwen35 model and prints a banner citing GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 — over E3 budget (rank 13/79)
• apr inspect --json drops every qwen35.* metadata key (context_length 262144, block_count, full_attention_interval…): 46 header KVs, 24 shown #3733 apr inspect --json drops every qwen35.* metadata key (context_length 262144, blo — over E3 budget (rank 35/79)
• tokenizer parity: the APR/HF encode paths are sibling copies #3726 did not touch — aprender-core BpeTokenizer::pre_tokenize is a whitespace splitter, realizar apr/tokenizer::bpe_encode merges with no pre-tokenizer #3742 tokenizer parity: the APR/HF encode paths are sibling copies apr's byte-level BPE tokenizer is non-canonical: every non-ASCII byte → id 0, ASCII segmented unlike llama.cpp (2 of 10 ids agree on a path) — corrupts copy tasks and every non-ASCII user; apr parity is blind (shared tokenizer) #3726 did not touch — over E3 budget (rank 36/79)
• apr inspect --json 'quantization' is wrong: Q6_K for a Q4_K_M GGUF, None for every .apr, and inspect FAILS on IQ4_XS ('Unsupported dtype 23') #3762 apr inspect --json 'quantization' is wrong: Q6_K for a Q4_K_M GGUF, None for eve — over E3 budget (rank 37/79)
• entrenar's .apr export embeds a tokenizer that drops array-form merges and the added tokens (two hand-rolled tokenizer.json readers in cuda_trainer.rs) #3803 entrenar's .apr export embeds a tokenizer that drops array-form merges and the a — over E3 budget (rank 22/79)
• The CPU default (Q8_K activations) changes the greedy answer on 4 pairs on lambda and 7 on gx10, different sets — the FP32 default is blocked on aarch64 having no NEON FP32 Q4_K dot #3811 The CPU default (Q8_K activations) changes the greedy answer on 4 pairs on lambd — over E3 budget (rank 16/79)
• transformer_layer_indexed hardcodes the Q4K kernel for Q/K/attn_output under a comment claiming exhaustive dispatch (silent corruption, currently masked) #3822 transformer_layer_indexed hardcodes the Q4K kernel for Q/K/attn_output under a c — over E3 budget (rank 52/79)
• gpu_state_isolation: fails on coder-0.5b, and its own message truncates both outputs to an identical prefix so it never shows the difference #3824 gpu_state_isolation: fails on coder-0.5b, and its own message truncates both out — over E3 budget (rank 53/79)
• apr run --chat is INERT: the chat template is applied whether or not the flag is passed, and there is no way to send a prompt raw #3848 apr run --chat is INERT: the chat template is applied whether or not the flag is — over E3 budget (rank 17/79)
• Qwen3.5-4B-Q4_K_M: deterministic GPU/CPU divergence at exactly one interior position (21) on the hybrid forward #3851 Qwen3.5-4B-Q4_K_M: deterministic GPU/CPU divergence at exactly one interior posi — over E3 budget (rank 18/79)
• owned_fused_matmul refuses IQ4_NL while its own message claims IQ* is supported — the refusal overclaims its coverage #3852 owned_fused_matmul refuses IQ4_NL while its own message claims IQ* is supported — over E3 budget (rank 12/79)
• IQ4_NL (ggml type 20) has no dequant path in realizar — two Qwen2.5 inventory models cannot generate on either backend #3869 IQ4_NL (ggml type 20) has no dequant path in realizar — two Qwen2.5 inventory mo — over E3 budget (rank 19/79)
• golden gate: the thinking-ON leg sends a prompt in NEITHER mode the model declares — it deletes the think block where the model opens it #3882 golden gate: the thinking-ON leg sends a prompt in NEITHER mode the model declar — over E3 budget (rank 54/79)
• two surfaces report quant=Q4_K for .apr models that contain no Q4_K tensors (f16 and BF16 measured) #3893 two surfaces report quant=Q4_K for .apr models that contain no Q4_K tensors (f16 — over E3 budget (rank 55/79)
• gpu_cpu_trace_compare compiles again and fails: GPU L2 diverged 91.92% from CPU on a synthetic fixture (cause undetermined) #3895 gpu_cpu_trace_compare compiles again and fails: GPU L2 diverged 91.92% from CPU — over E3 budget (rank 56/79)
• Two f32_matmul functions with swapped argument orders and identical types — a wrong import is a silent transpose the compiler cannot see #3896 Two f32_matmul functions with swapped argument orders and identical types — a wr — over E3 budget (rank 20/79)
• THINKING_ON_BUDGET is one 8B model's measurement applied to every thinking model, including a 0.8B — and the truncation guard's fixture claims to mirror production while building its own string #3907 THINKING_ON_BUDGET is one 8B model's measurement applied to every thinking model — over E3 budget (rank 38/79)
• Every BF16 SafeTensors import produces a .apr that cannot run on the GPU: wgpu has no type-30 dequant and the CUDA path is never selected #3908 Every BF16 SafeTensors import produces a .apr that cannot run on the GPU: wgpu h — over E3 budget (rank 39/79)
• An .apr records nothing about what built it: 'stale artifact or live defect' is unanswerable from the file, and the provenance fields exist but are never populated #3910 An .apr records nothing about what built it: 'stale artifact or live defect' is — over E3 budget (rank 57/79)
• apr chat never consults the tokenizer the .apr carries — four filesystem locations, none of them the model #3911 apr chat never consults the tokenizer the .apr carries — four filesystem locatio — over E3 budget (rank 40/79)
• detect_format_from_name's ordered substring table has produced three defects — one a correct fix that never executed; replace with exact keys #3918 detect_format_from_name's ordered substring table has produced three defects — o — over E3 budget (rank 58/79)
• The thinking-ON leg exists twice and has now drifted twice: the hybrid copy takes neither #3907's budget resolver nor #3914's template referee #3919 The thinking-ON leg exists twice and has now drifted twice: the hybrid copy take — over E3 budget (rank 59/79)
• #3911's embedded tokenizer splits every ChatML control token into six pieces — 32 ids where the reference gives 18 #3920 apr chat never consults the tokenizer the .apr carries — four filesystem locations, none of them the model #3911's embedded tokenizer splits every ChatML control token into six pieces — 3 — over E3 budget (rank 30/79)
• RULE A VIOLATION: a Q4_K .apr produces garbage on CUDA on BOTH hosts while answering correctly on CPU — rc=0, no fallback, recorded GREEN #3922 RULE A VIOLATION: a Q4_K .apr produces garbage on CUDA on BOTH hosts while answe — over E3 budget (rank 41/79)
• AprV2ModelCuda is Q4K-only by construction and wrong for Q4K — no configuration does useful GPU work #3923 AprV2ModelCuda is Q4K-only by construction and wrong for Q4K — no configuration — over E3 budget (rank 60/79)
• apr chat --json emits the backend, not the reply — the transcript is the only place the model's text exists #3927 apr chat --json emits the backend, not the reply — the transcript is the only pl — over E3 budget (rank 61/79)
• apr run is the last verb whose generated text nothing judges — and qa's golden leg runs once per rung, not once per backend #3928 apr run is the last verb whose generated text nothing judges — and qa's golden l — over E3 budget (rank 42/79)
• Qwen3NoThink prefills <think>\n</think>\n where the model's own template uses <think>\n\n</think>\n\n — the model spends two tokens re-closing it #3930 Qwen3NoThink prefills \n\n where the model's own template uses <t — over E3 budget (rank 62/79)
• golden_output's GPU leg judges GGUF only, so every .apr rung gets a CPU-only verdict that reports as passed #3931 golden_output's GPU leg judges GGUF only, so every .apr rung gets a CPU-only ver — over E3 budget (rank 43/79)
• aprender-core has no IQ dequantizer, so tensor_contract cannot inspect three release-blocking models — port from the validated aprender-serve decoders #3947 aprender-core has no IQ dequantizer, so tensor_contract cannot inspect three rel — over E3 budget (rank 34/79)
• Qwen3.5-0.8B-IQ4_XS never closes its think block while Q4_K_M does: CUDA IQ4 GEMV or real quant damage? #3951 Qwen3.5-0.8B-IQ4_XS never closes its think block while Q4_K_M does: CUDA IQ4 GEM — over E3 budget (rank 31/79)
• IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker #3953 IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker — over E3 budget (rank 24/79)
• golden thinking-ON leg discards its backend (_used_gpu) and passes an empty <think></think> as thinking #3961 golden thinking-ON leg discards its backend (_used_gpu) and passes an empty <thi — over E3 budget (rank 63/79)
• GPU whitelist admits a quant TYPE on one model's shapes — admission must key on per-shape device conformance (census-derived, #3945) #3968 GPU whitelist admits a quant TYPE on one model's shapes — admission must key on — over E3 budget (rank 44/79)
• GPU/CPU f32 APR forward diverges at layer-0 QKV (std 4.44 vs 0.30); gpu_cpu_trace_compare fails 91.9% L2 once it compiles #3975 GPU/CPU f32 APR forward diverges at layer-0 QKV (std 4.44 vs 0.30); gpu_cpu_trac — over E3 budget (rank 25/79)
• 0.70.0: qwen35moe (Qwen3.5-35B-A3B hybrid SSM MoE) CUDA forward #3977 0.70.0: qwen35moe (Qwen3.5-35B-A3B hybrid SSM MoE) CUDA forward — over E3 budget (rank 64/79)
• apr code hardcodes 'apr serve --gpu' (no CPU lane, no --max-tokens/thinking flag) and picks a colliding port 19384+pid%1000 #3978 apr code hardcodes 'apr serve --gpu' (no CPU lane, no --max-tokens/thinking flag — over E3 budget (rank 45/79)
• qwen3moe CUDA reaches run+qa only: chat rc 8, serve 501/500, code rc 1 — serve loads no mapped model and would route MoE to the CPU path #3987 qwen3moe CUDA reaches run+qa only: chat rc 8, serve 501/500, code rc 1 — serve l — over E3 budget (rank 46/79)
• apr serve advertises dead GGUF generation routes: /v1/batch/completions and /stream|/realize/generate 503; /generate, /batch/generate and /realize/batch echo the prompt #3991 apr serve advertises dead GGUF generation routes: /v1/batch/completions and /str — over E3 budget (rank 47/79)
• 17 more files build the dense GGUF CUDA model with no qwen3moe route — audit each (derived from #3987's guard) #3992 17 more files build the dense GGUF CUDA model with no qwen3moe route — audit eac — over E3 budget (rank 48/79)
• Qwen3.5-0.8B-UD-IQ2_XXS fails '2+2' on BOTH apr and llama.cpp (official template) — RED-MODEL candidate pending the step-3 logit margin #4004 Qwen3.5-0.8B-UD-IQ2_XXS fails '2+2' on BOTH apr and llama.cpp (official template — over E3 budget (rank 65/79)
• 0.69.1: prove non-ASCII tokenization equals llama-tokenize on every tokenizer path apr ships (#3726 re-verify, twin of #3992) #4005 0.69.1: prove non-ASCII tokenization equals llama-tokenize on every tokenizer pa — over E3 budget (rank 49/79)
• apr run -v PANICS on a non-ASCII prompt: formatted_prompt log slices a str at byte 200 (not a char boundary) #4018 apr run -v PANICS on a non-ASCII prompt: formatted_prompt log slices a str at by — over E3 budget (rank 50/79)
• apr thinking-ON closes </think> far less often than llama.cpp on the official template (0.8B Q4_K_M: 1/13 vs 5/13) — possible long-generation divergence #4019 apr thinking-ON closes far less often than llama.cpp on the official te — over E3 budget (rank 66/79)
• CRUX judge: thinking-ON cells must not be judged on </think> closure alone — use answer-when-both-close, else teacher-forced agreement (from #4019) #4021 CRUX judge: thinking-ON cells must not be judged on closure alone — use — over E3 budget (rank 67/79)
• 0.71: qwen35-0.8b-q4km golden thinking-ON never closes on apr (greedy near-tie path, no localized defect) — shipped as known RED in 0.69.1 #4030 0.71: qwen35-0.8b-q4km golden thinking-ON never closes on apr (greedy near-tie p — over E3 budget (rank 68/79)
• test_forward_batched_graphed_replay passes when graph capture FAILS (falls back silently); parallel graph capture → CUDA_ERROR_UNKNOWN 901 + segfault (found on #3810) #4043 test_forward_batched_graphed_replay passes when graph capture FAILS (falls back — over E3 budget (rank 69/79)
• BUG: apr serve run with no backend flag resolves to CPU on a cuda build while apr run uses CUDA (qwen3.5-4b, yoga) #4089 BUG: apr serve run with no backend flag resolves to CPU on a cuda build while ap — over E3 budget (rank 51/79)
• apr run --benchmark --json prints a non-JSON block first, and --profile recommends '--gpu' / a CPU bottleneck on a run it reports as GPU #4211 apr run --benchmark --json prints a non-JSON block first, and --profile recommen — over E3 budget (rank 26/79)
• qwen35 MTP speculative decoding with GDN state rollback #4245 qwen35 MTP speculative decoding with GDN state rollback — over E3 budget (rank 70/79)
• Qwen3.5 has no wgpu path: apr serve --backend wgpu refuses at load ('0 transformer layers'), apr run --backend wgpu refuses too — #4252 intel tier RED #4271 Qwen3.5 has no wgpu path: apr serve --backend wgpu refuses at load ('0 transform — over E3 budget (rank 71/79)
• perf(decode): gdn_decode_attention takes 3.72 ms/tok on 4B decode #4298 perf(decode): gdn_decode_attention takes 3.72 ms/tok on 4B decode — over E3 budget (rank 72/79)
• perf(decode): unexplained per-token H2D copy costs 2.13 ms/tok #4299 perf(decode): unexplained per-token H2D copy costs 2.13 ms/tok — over E3 budget (rank 73/79)
• perf(decode): Q4K/Q5K/Q6K GEMVs take ~7 ms/tok on 4B decode #4300 perf(decode): Q4K/Q5K/Q6K GEMVs take ~7 ms/tok on 4B decode — over E3 budget (rank 74/79)
• apr serve: /v1/chat/completions stream leaks <|im_end|> into delta content (1/14 CRUX prompts) #4340 apr serve: /v1/chat/completions stream leaks <|im_end|> into delta content (1/14 — over E3 budget (rank 27/79)
• apr serve native routes leak <|im_end|> into text: /generate, /batch/generate, /stream/generate, /realize/generate, /realize/batch #4344 apr serve native routes leak <|im_end|> into text: /generate, /batch/generate, / — over E3 budget (rank 28/79)
• apr serve: GPU/CUDA /v1/completions paths stop on a single EOS or none (follow-up to #4339) #4345 apr serve: GPU/CUDA /v1/completions paths stop on a single EOS or none (follow-u — over E3 budget (rank 29/79)
• effective-config reports Qwen3.5 CUDA residency as 'cpu residency' and drops the F2 parity verdict (v0.69.3) #4374 effective-config reports Qwen3.5 CUDA residency as 'cpu residency' and drops the — over E3 budget (rank 32/79)
• PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen3-MoE forward reuse the previous GEMV's input under DP4A #4378 PMAT-027 stale Q8_1 activation cache: batched fallback, batched decode, and Qwen — over E3 budget (rank 33/79)
• Plausible-value defaults on file-decoded arch/tokenizer/quant: unknown arch exported as Qwen2, APR cache reads an untyped weight as Q4_K (#3850 audit) #4419 Plausible-value defaults on file-decoded arch/tokenizer/quant: unknown arch expo — over E3 budget (rank 75/79)
• C14 parity at long context: apr parity CPU vs CUDA at ≥8k and 20k positions on lambda and gx10 (night lane), and a long-context row in the C14 corpus (split from #3693 done_when 2) #3780 C14 parity at long context: apr parity CPU vs CUDA at ≥8k and 20k positions on l — floor orphan, not linked over budget (rule 21 wins over rule 6, cop 18:08Z)

Superseded epics

• EPIC: nvidia-cuda-rust-library-integration.md — stabilize GPU quality with NVIDIA's CUDA Rust libraries (0.67) #3062 EPIC: nvidia-cuda-rust-library-integration.md — stabilize GPU quality with NVIDI — superseded (A4); owner lines: none

0.7x schedule (release-schedule-07x, 2026-10-08)

Train 0.71.0: plan cut 2026-10-16, tag 2026-10-17 (plan dates, not measured). Report: docs/reports/0.7x-epic-reports.md.

Open children kept in 0.71.0 (owners are proposals; the cop assigns):

• apr run --gpu is 2-3x SLOWER than --no-gpu on the dense path: the GPU result fails correctness (cosine 0.4153) and falls back to CPU after the user has paid for it #3602 · P0 · ticket · size M · proposed owner la-0.71
• GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759 · P0 · ticket · size M · proposed owner la-0.71
• resolve_qtype silently decodes an UNKNOWN ggml quant as Q4_K, and the whitelist that would refuse it is not on the apr qa GPU path (#3821, second instance) #3850 · P0 · ticket · size M · proposed owner la-0.71
• apr code on Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs through apr serve --gpu (#3571) #3719 · P0 · ticket · size M · no proposed owner yet
• IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker #3953 · P1 · ticket · size M · proposed owner la-0.71
• qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714 · P0 · ticket · size M · proposed owner la-0.71
• P0: IQ3_S/IQ2_S GEMV kernels fail to load on cc>=12 (GB10/Blackwell) — GH-480 PTX patch injects .reg inside a global array initializer #4096 · P0 · ticket · size M · proposed owner la-0.71
• perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) #4486 · P0 · ticket · size M · proposed owner la-0.71
• KREG-001: SHACL kernel registry — loader refuses any unregistered (qtype, layout, arch, shape, precision) kernel #4539 · P1 · ticket · size M · no proposed owner yet
• qwen35 session: #4274 headroom doubles a 32k turn to 65536 positions — 27B OOMs and silently falls back to CPU #4443 · P1 · ticket · size M · proposed owner la-0.71
• F2 receipt key uses CARGO_PKG_VERSION, so rc.1/rc.2/final (and every dev build) of one version share receipts: a forward-path change skips the CPU reference check #4290 · P0 · ticket · size M · proposed owner la-0.71
• C14: qwen3-8b GPU decode — stale position buffers after failed graph capture (land fix/c14-graph-fail-stale-position) #4582 · P0 · ticket · size M · proposed owner la-0.71
• gdn_ops split-decode: format! kernel keys per call → 12 host allocs/token (qwen35 decode steady-state RED) #4508 · P0 · ticket · size M · no proposed owner yet
• 0.70.1 P0: Qwen3 on GB10 (sm_121) is correct but ~30x slow — ship sm_121 SASS, re-test batched prefill #4590 · P1 · ticket · size M · proposed owner la-0.71
• qwen35 CUDA-vs-CPU e2e logits exceed LOGITS_BUDGET 7e-2 on GB10 sm_121 (7.105e-2, 2/2 deterministic; budget measured on 4090 only) #4523 · P1 · ticket · size M · no proposed owner yet
• qwen35 --gpu tok/s counts the CUDA build + F2 guard as generation (inference_ms not generation-only) — sibling of #3714 #4342 · P0 · ticket · size M · proposed owner la-0.71
• DP4A Q4K/Q6K GEMV profile is catastrophic through the Gated DeltaNet recurrence — Qwen3.5 on sm_89 needs the float MWV variant (PMAT-806 class) #3513 · P1 · ticket · size M · proposed owner la-0.71

Moving to another milestone per the gate-graph walk (triage applies each move):

• CUDA Q5_K gemv/gemm read a non-ggml block layout (qh bit order; gemm also qh/qs offsets and scale unpacking) #3111 → milestone 0.72.0
• apr mis-tokenizes <|im_end|> after sentence punctuation: 4 junk tokens are fed to the model instead of the turn delimiter #2815 → milestone 0.72.0
• P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753 → milestone 0.73.0
• FP8 prefill GEMM overflows its FP16 output: D = true × 448/act_absmax exceeds 65,504 on qwen2.5-coder-7b (cosine 0.5668, lambda) #3728 → milestone 0.73.0
• FP8 batched GEMM multiplies a STALE activation on mixed-quant models: the PMAT-084 cache key (ptr, count) aliases across layers — #3602's cosine 0.4153 #3727 → milestone 0.73.0