Repository navigation
apr bench (qwen35 --fast) refuses every run with warmup+iterations > 1: one session reuses its own prefix cache #4445
Description
What happened
On intel (x86_64, wgpu), apr 0.69.5-rc.1 (5660b0877) running
apr bench ~/models/Qwen3.5-4B-Q4_K_M.gguf --fast --warmup 1 --iterations 3 --json exits 5:
error: Validation failed: qwen35 bench: 6 prompt tokens were served from a cached prefix, not prefilled
With --warmup 0 --iterations 1 the same command measures (wgpu, 32 tokens, TTFT 4372 ms).
Why
In crates/apr-cli/src/commands/bench_qwen35.rs@5660b0877, the warmup and every iteration call session_timed_turn(&mut session, prompt_tokens, ...) on one Qwen35Session with the same prompt. From the second turn on, the session serves the prompt from its prefix cache, so turn.reused != 0 and the guard at line ~78 refuses. The guard is correct: a reused prefix would report decode as prefill+decode. But the loop cannot satisfy it. So on the qwen35 path, any bench with warmup+iterations > 1 always refuses, which includes the defaults (warmup 3, iterations 5).
Fix direction
Give each timed turn a fresh session (or reset its cache) so every turn prefills the whole prompt. Add a test that runs warmup ≥ 1 and iterations ≥ 2 and asserts it succeeds and that reused == 0 on every turn. Prove the test goes red on the current loop.
Impact
infra#1057's per-binary perf ledger can't take a multi-sample number for qwen35 from apr bench on any host.
Found by infra-8d while measuring the v0.69.5-rc.1 ledger rows.
Activity
noahgift commentedon Sep 25, 2026
Repro on the v0.69.5-rc.1 release assets (aprender-36, 2026-09-25 ~16:50Z):
apr benchrefuses any qwen35 run with more than one passapr 0.69.5-rc.1 (5660b0877).d7dc9fcd…, RTX 4060 Laptop.3d036e1a….00fe7986ff5f…, the same file on both hosts.apr bench $M --fast --json --offline [args]--warmup 0 --iterations 1--warmup 0 --iterations 2--warmup 1 --iterations 1--prompt Explain_the_theory_of_relativity_in_detailError on every refused run:
error: Validation failed: qwen35 bench: N prompt tokens were served from a cached prefix, not prefilled. N is the prompt length: 6 for the default prompt, 10 for the one above.The failure is deterministic and follows the pass count, not the prompt or the host. On the second pass over the same prompt, the prefix cache serves the prompt, and the bench's own prefill guard rejects it. That makes n>1 qwen35 measurements impossible with
apr benchon rc.1. The only way around it was single-pass runs, which is what the infra#1057 ledger rows for yoga and mini are.Artifacts are in
/mnt/nvme-raid0/tmp/rc1-bench/on lambda-vector (4b-*-n5-REFUSED.*,4b-*-single.*, README.md).noahgift commentedon Sep 25, 2026
Fix:
65fd97317onfix/4445-bench-fresh-prefill, based onbatch/0.69.4-epic-4249(#4318). The bug comes from #4214's checkpoint resume (71d3477), which only that branch carries. main, B1r and car/0.70.0 don't reuse a repeated prompt.Session::forget_prefix()drops the held positions and the checkpoint. The bench calls it before each warmup and each timed turn, so every turn prefills the whole prompt, and thereused != 0guard stays unchanged.Proof:
bench_on_qwen35_times_session_generate(real Qwen3.5-0.8B, warmup 1, iterations 2) fails without the calls, with the error from this issue (16 prompt tokens were served from a cached prefix), and passes with them.forget_prefix_makes_the_same_prompt_prefill_whole_againfails ifforget_prefixkeeps the checkpoint.-D warningsare clean on aprender-serve and apr-cli.Routing (the fix goes onto #4318 for rc.2, or waits for the 0.70.1 stage) is the cop's decision. — aprender-ea
chore(roadmap): PMAT-4445 entry for the qwen35 bench fresh-prefill fix (
chore(roadmap): regenerate roadmap.yaml for the PMAT-4445 fragment (#…
fix(#4445): the AC's named test — forget_prefix_makes_the_same_prompt…
fix(bench): qwen35 bench forgets the session prefix before every turn (…
merge(rex): re-apply rex/001 on top of main: reconcile with the #4655 …