Repository navigation
apr serve (0.69.1) has no prompt-prefix reuse on the qwen35 CUDA path: an identical ~850-token prompt costs 11.0 s on both runs (ollama: 8.2k tok/s uncached, 37k cached) #4214
Description
Measured 2026-09-24 on lambda-labs (RTX 4090) by the local-Qwen prototype (infra PMAT-1009). Receipt: infra branch feat/local-qwen35-recon, docs/audits/LQW-001/2026-09-24-qwen35-4b-apr-vs-ollama.md.
- apr v0.69.1 CUDA release asset,
apr serve run --gpu, unsloth/Qwen3.5-4B-GGUF Q4_K_M (sha256 00fe7986…11a4), /v1/chat/completions, temp 0. - The same ~850-token prompt sent twice took 11.0 s to first token both times. The second request reused nothing.
- Reference: ollama 0.34.2 with the same base weights gives 37108 tok/s prefill on the identical repeat.
Related: #3553 (the prefix-cache counter is exposed nowhere, so this can't be read from /metrics), #3596 (prefill one token at a time).
Acceptance: a repeated identical prompt on the qwen35 path skips the prefill of its shared prefix, and the test shows the cached TTFT is a fraction of the uncached one. The test must go red with the fix reverted.
Activity
noahgift commentedon Sep 24, 2026
RC1
7ff50ec2aevidence (infra-76 receipt/mnt/nvme-raid0/tmp/rc1-llama/RC1-RECEIPT.md): turn 2 logs[qwen35] batched prefill: … from position 0with 1076, 4215 and 31909 tokens, so the whole conversation is re-prefilled every turn. llama-server on the same prompt reuses the prefix and prefills 22 tokens. At 32k that is ~14–15 s of TTFT per turn that llama.cpp does not pay.test(#4214): a prompt prefills in two batched spans, split at its gen…
WIP perf(serve): resume a diverging turn from a DeltaNet checkpoint; …
test(#4214): RED — a repeated or re-rendered prompt resumes from the …
fix(session): a repeated or re-rendered prompt resumes from a recurre…
test(#4214): a prompt prefills in two batched spans, split at its gen…
test(#4274): a re-rendered chat turn resumes from turn 1's checkpoint…
fix(test): #4250's serve prefill count follows #4214's two-span split…
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over E2 budget (rank 16/23). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).
test(#4214): RED — a repeated or re-rendered prompt resumes from the …
fix(session): a repeated or re-rendered prompt resumes from a recurre…
test(#4214): a prompt prefills in two batched spans, split at its gen…
test(#4274): a re-rendered chat turn resumes from turn 1's checkpoint…
fix(test): #4250's serve prefill count follows #4214's two-span split…
fix(bench): qwen35 bench forgets the session prefix before every turn (…
noahgift commentedon Oct 15, 2026
Reopened (pull): 0.73 llama.cpp parity row under #3598, owner aprender-df, fresh branch from main 8f601ca. Done-when: a repeated-prefix shot shows timings.prompt_n < prompt_tokens on apr, measured the way #5305's pp/tg harness measures it.