EPIC E2 0.73: Verbs Are Fast #3598

owner: la-0.71 (re-owned 2026-10-09 by clerk aprender-3e for cop aprender-13: works child #4925 (open PR #4923))

Spec: APR-EPIC-001 §2 · Epic: E2 · Train: 0.71 · Successor at cut: E6

Theme

Verbs Are Fast

done_when

Qwen3.5 TTFT ≤ 2× llama.cpp d1d3c3396 on the same GGUF [O 2026-09-20]; apr serve resident with continuous batching; tok/s is generation-only; decode at 32k context ≥ [U]× llama.cpp (instrument first, then ratchet).

Budget

At most 7 open children (docs/roadmaps/epics.yaml). Repo-wide cap 100 open issues; andon at 95.

Absorbs

#4249

Prior body

Three rows, one lane: make apr run say where the time went, make the load cost payable once, and stop apr devices lying about the backend. Operator ruling 2026-09-20 — 0.69 measures performance, 0.70 improves it.

Row 1 — apr run --json gains load_ms · h2d_ms · prefill_ms · decode_ms · tokens_out (closes #3542)

This is the row every other performance fix depends on, and the evidence is that wall-clock cannot substitute for it. Measured on noah-Lambda-Vector (RTX 4090 sm_89), Qwen3.5-4B-Q4_K_M.gguf, published 0.68.2, all layers on GPU:

From that data the cop named the mechanism twice and was wrong twice inside fifteen minutes:

• "The 14 s is load" — killed by a 2-char prompt costing 6.8 s, so ~7 s is prompt-dependent.
• "Prefill runs at ~4.7 tok/s, 170× slower" — killed by 144→288 words costing only ~2 s; the marginal rate at the long end is ~24× better than that two-point slope implied.

Scaling, --max-tokens 1: Hi 6.77/6.80 · 24w 12.76 · 72w 17.69 · 144w 17.17/17.66/16.71 · 288w 19.01/19.18. Not linear. Consistent with a large one-time cost a trivial prompt skips plus a low marginal cost — chunked prefill with an expensive first chunk, a one-time graph capture or JIT, or KV/buffer allocation rounded to a context size. Three candidates, and wall clock admits all three. Run-to-run variance is ±2 s on identical work (token counts were identical across trials), which is larger than the signal being chased.

done_when

1. apr run --json emits load_ms, h2d_ms, prefill_ms, decode_ms, tokens_out.
2. Falsifier: the fields sum to within a stated tolerance of wall clock, and a planted delay in one stage moves that field and only that field. A timing row that cannot localise a planted delay is decoration.
3. The JSON goes to one declared stream. Today Backend: goes to stderr and Completed in … to stdout; a consumer cannot guess.
4. The (cached) suffix on Completed in N s states what was cached or is removed — it currently prints on runs that demonstrably executed.
5. Slow is allowed in 0.69; invisible is not. Landing this makes the 14 s honest.

Row 2 — apr serve resident (#3571)

For every consumer that keeps a process up — arbiter, apex, chat — the fixed cost is paid once, not per call. That converts a ~5× user-visible gap into a one-time cost today, before the root cause is known, which is why it is 0.69 and the fix is not.

done_when: model resident across requests; m=1; OpenAI-shape endpoint; health; a parity receipt at the serving shape (not the CLI shape — a receipt for a different shape is not a receipt for this one).

Row 3 — apr devices must not report NotCompiled on a CUDA build (#3545)

A verb that lies about the backend is the 0.69 headline defect in miniature. done_when: on a CUDA build it reports CUDA; the falsifier is a non-CUDA build still reporting honestly, so the fix is not "always say yes".

Constraints

• Open as draft; the cop grants the CI slot. Do not undraft.
• Nothing touches census.json or lint-baseline.json outside the labelled path — one contract file already obliges three serialized derivatives (Decision 8: census.json becomes derived (CI computes, guard refuses PR diffs); lint-baseline counts become a base-vs-head never-down check #3569), each masking the next.
• 0.70 follow-on, plan-and-draft only, admissible because its plan is ruled before 09-23: Schema-constrained decoding in apr (--json-schema or grammar) — three customers: arbiter's quorum lane, tool calling in verb set V, and the ruchy language #3568 --json-schema / grammar.

Refs #3542, #3571, #3545, #3596, #3569, #3080.

Parked

Closed over budget (APR-EPIC-001 rule 5). Reopen = pull (rule 7).

• cbtop --simulated --ci: the verdict is a coin flip (jittered brick timings, CV 88%) — M008 was dark for five months because of it #3117 cbtop --simulated --ci: the verdict is a coin flip (jittered brick timings, CV 8 — over E2 budget (rank 21/23)
• A dev-dependency's feature arms a sibling's runner-gated test: the BSE-17 quick tier ran three browser falsifiers on a browser-less runner #3242 A dev-dependency's feature arms a sibling's runner-gated test: the BSE-17 quick — over E2 budget (rank 22/23)
• guard-tree is fail-fast: one red guard darkens 48 of its 59 steps, including the §6 R-2 PR-body gate #3282 guard-tree is fail-fast: one red guard darkens 48 of its 59 steps, including the — over E2 budget (rank 23/23)
• Qwen3.5 TTFT has a large term OUTSIDE prefill that grows with prompt length (13.6 s at 20k → 168 s at 262k; 211 s before a pre-load refusal at 263k) — tokenize/template/parse, unmeasured and unprofiled #3787 Qwen3.5 TTFT has a large term OUTSIDE prefill that grows with prompt length (13. — over E2 budget (rank 13/23)
• apr run --format json tok_per_sec counts model load + F2 validation as inference (0.2 tok/s reported vs 38.6 measured decode) #3981 apr run --format json tok_per_sec counts model load + F2 validation as inference — over E2 budget (rank 8/23)
• aprender-serve lib tests accumulate memory across tests in one process: 30 GB peak single-threaded, 45 GB at 22 threads, though each module alone is small #4028 aprender-serve lib tests accumulate memory across tests in one process: 30 GB pe — over E2 budget (rank 9/23)
• apr serve (0.69.1) has no prompt-prefix reuse on the qwen35 CUDA path: an identical ~850-token prompt costs 11.0 s on both runs (ollama: 8.2k tok/s uncached, 37k cached) #4214 apr serve (0.69.1) has no prompt-prefix reuse on the qwen35 CUDA path: an identi — over E2 budget (rank 16/23)
• apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215 apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama — over E2 budget (rank 10/23)
• CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4220 CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure — over E2 budget (rank 15/23)
• perf(cpu): multi-row K-quant×Q8_K GEMM + batched qwen35 CPU prefill (CRUX #4223 T1, llamafile iqk parity) #4228 perf(cpu): multi-row K-quant×Q8_K GEMM + batched qwen35 CPU prefill (CRUX CRUX parity spike: apr serve vs llamafile on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4223 — over E2 budget (rank 11/23)
• perf(qwen35-cuda): capture the Qwen3.5 decode step as a CUDA graph (vLLM O2 FULL_AND_PIECEWISE; CRUX #4222 T1) #4233 perf(qwen35-cuda): capture the Qwen3.5 decode step as a CUDA graph (vLLM O2 FULL — over E2 budget (rank 12/23)
• perf(serve): continuous batching for Qwen3.5 — apr serve holds one Mutex<Qwen35Session>, so 16 streams run serially (vLLM max_num_seqs 128; CRUX #4222 T2) #4234 perf(serve): continuous batching for Qwen3.5 — apr serve holds one Mutex<Qwen35S — over E2 budget (rank 14/23)
• measure(qwen35-cuda): #3596's chunk-resident delta-rule scan vs the WY-chunked (tensor-core) form — prefill share at 4k/32k (CRUX #4222 T3) #4235 measure(qwen35-cuda): Time-to-first-token on Qwen3.5-4B-Q4_K_M is 14-16s vs llama.cpp's 1.8-2.9s on the same box (7.7x); apr's decode rate is unmeasurable with shipped tooling #3596's chunk-resident delta-rule scan vs the WY-chunked ( — over E2 budget (rank 18/23)
• measure(qwen35-cuda): can the prefill projections leave f32 SGEMM for BF16 tensor cores within a recurrence tolerance? (CRUX #4222 T4) #4236 measure(qwen35-cuda): can the prefill projections leave f32 SGEMM for BF16 tenso — over E2 budget (rank 19/23)
• Qwen3.5 batched prefill re-dequantizes every weight to f32 per chunk and runs FP32 SGEMM (no tensor cores, no MMQ) — est. 0.3–0.5x llama.cpp prefill #4260 Qwen3.5 batched prefill re-dequantizes every weight to f32 per chunk and runs FP — over E2 budget (rank 17/23)
• qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442 qwen35 batched prefill: flash attention for the full-attention layers — 32k pref — over E2 budget (rank 20/23)
• apr qa tensor_contract holds the whole GGUF on the heap: 19.36 GB RssAnon on Qwen3-30B-A3B (rosetta validate → GgufReader::from_file) — read tensor data from a map #3790 apr qa tensor_contract holds the whole GGUF on the heap: 19.36 GB RssAnon on Qwe — swapped out of E2 for apr bench (qwen35 --fast) refuses every run with warmup+iterations > 1: one session reuses its own prefix cache #4445 (one-in-one-out, cop ruling)

0.7x schedule (release-schedule-07x, 2026-10-08)

Train 0.71.0: plan cut 2026-10-16, tag 2026-10-17 (plan dates, not measured). Report: docs/reports/0.7x-epic-reports.md.

Open children kept in 0.71.0 (owners are proposals; the cop assigns):

• FP8 batched prefill fails CPU parity at the post-prompt decode step — catastrophic on per-head-QK-norm models (Qwen3-8B cos −0.10), marginal on Qwen2.5 (0.92) #3483 · P1 · ticket · size M · proposed owner la-0.71
• Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725 · P0 · ticket · size M · proposed owner la-0.71
• install.sh does not exist: /releases/latest/download/install.sh 404s, nothing in the tree provides it — yet T-2 asserts it ran on intel and gx10 #3546 · P1 · ticket · size M · proposed owner la-0.71
• CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4220 · P1 · ticket · size M · proposed owner la-0.71
• apr: llama.cpp logit-compare verb as the PRM parity.oracle (PRM-001 v2 §4.1) #4444 · P0 · ticket · size M · proposed owner la-0.71
• perf(serve): Qwen3.5 tensor-core prefill projections (RCA-SRV-001 fix 1) #4483 · P1 · ticket · size M · no proposed owner yet
• perf(serve): decode GEMV precision on Qwen3.5 (RCA-SRV-001 fix 3) #4485 · P1 · ticket · size M · proposed owner la-0.71
• APR-OBS-001: dogfood observability (OBS-00 + row checklist) #4487 · P1 · ticket · size M · proposed owner aprender-cd
• PVL-13: Aeneas binding spike — model_of from extracted Rust (scalar references) #4555 · P1 · ticket · size M · no proposed owner yet
• TR-01: BrickTracer honesty (TRACE-001) #4556 · P1 · ticket · size M · proposed owner la-0.71
• TR-02: cbtop escalation (TRACE-001) #4557 · P1 · ticket · size M · proposed owner la-0.71
• TR-03: CPU serve trace payload (TRACE-001) #4558 · P1 · ticket · size M · proposed owner la-0.71
• TR-04: mock rename (TRACE-001) #4559 · P1 · ticket · size M · proposed owner la-0.71
• TR-05: tool resolution (TRACE-001) #4560 · P1 · ticket · size M · proposed owner la-0.71
• TR-10: CRUX bind (TRACE-001) #4565 · P1 · ticket · size M · proposed owner la-0.71
• perf(session): poll the cancel token before the decode forward, not after — one wasted forward per cancelled request (0.69.4) #4325 · P1 · ticket · size M · proposed owner la-0.71
• serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions #4274 · P1 · ticket · size M · proposed owner la-0.71
• perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) #4313 · P1 · ticket · size M · proposed owner la-0.71
• PRM C6+C11: rex pool-admit backfill + FALSIFY-TAS-009 (land rex/001 @292c01d75c) #4524 · P1 · ticket · size M · no proposed owner yet
• MEAS-001: granular resource receipts (RSS/VRAM/CPU/IO/faults/energy) for bench, CRUX, dogfood, CI ledger #4522 · P1 · ticket · size M · proposed owner la-0.71
• perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450 · P1 · ticket · size M · proposed owner la-0.71
• PRA-001 T12: agent-trace terms/provider tagging + provider/channel-excluding pool rebuild (P-TERMS) #4446 · P1 · ticket · size M · no proposed owner yet
• qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442 · P1 · ticket · size M · proposed owner la-0.71
• PROMETHEUS (PRM-001 v3) epic: Qwen 3.5 4B local lane — fast enough and good enough to break quorum ties #4354 · P0 · ticket · size M · no proposed owner yet
• P0: measured token usage for the fleet -- Claude Code + agy, by project / model / lane / hour, with the top levers #4162 · P0 · ticket · size M · proposed owner la-0.71

Moving to another milestone per the gate-graph walk (triage applies each move):

• #3750 PR B: the 17 header-only whole-file model reads outside apr qa — magic/header/metadata readers in their format crates #3761 → milestone backlog
• RandomForestRegressor/DecisionTreeRegressor carry the identical O(n²) split search — #3815's fix does not reach them #3859 → milestone 0.76.0
• Class: the shipped CPU path runs a narrower kernel than the host supports — dispatch returns before the wider ISA, kernels absent, module DCE'd, prefill run as decode #2880 → milestone 0.73.0