Repository navigation
serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions #4274
Description
Measured by aprender-f5 on 0.69.3 RC1 7ff50ec (receipt /mnt/nvme-raid0/agent-wt/f5-g1/receipt-rc1-7ff50ec2a.md). Setup: Qwen3.5-4B Q4_K_M, RTX 4090.
What happens: on turn 2, serve logs [qwen35] batched prefill: 1076 tokens … from position 0. The 1076 tokens are the whole conversation, not just the new turn, so the KV cache from turn 1 is not reused.
Likely cause, not yet verified: the rendered chat history does not match the token ids generated in turn 1, so the common-prefix check fails.
Output is correct: the batched and one-token-at-a-time paths produce the same text on both turns. Only time is lost. At 32k context, turn 2's time to first token is 15.5 s, against 13.9 s for turn 1.
Done when: turn 2 extends from kv_len > 0, shown by the trace line, and the text is identical to a full re-prefill.
Found by aprender-f5; filed by aprender-36.
Activity
noahgift commentedon Sep 24, 2026
Claimed by aprender-60 (#4252 dogfood owner). It will land in the next serve like-batch, not as its own PR (10-PR cap).
fix(session): a repeated or re-rendered prompt resumes from a recurre…
test(#4274): a re-rendered chat turn resumes from turn 1's checkpoint…
fix(serve): a turn past MIN_CAPACITY re-prefilled the next turn from 0 (
noahgift commentedon Sep 25, 2026
Root cause + fix on batch/0.69.4-epic-4249 @90f8ec7a5 (aprender-ont).
Root cause:
ensure_capacitysized the first state to exactly the turn whenever that turn was bigger than MIN_CAPACITY (4096). At p4k, turn 1 needed 4197 positions, so it got capacity 4197. Turn 2 needed 4279, so it reallocated.Session::reservedrops the checkpoint on any reallocation, so turn 2 re-prefilled from position 0. p850 fit inside 4096, which is why it extended.Fix: pure
grown_capacity(). It doubles MIN_CAPACITY until the turn fits, then clamps to the context. One-call states (min_capacity == 0) stay exact.a_state_that_outgrows_min_capacity_leaves_headroom_for_the_next_turn: 9 rows plus the e2e shape. Mutant (old formula restored) → REDleft: 4197 right: 8192; source restored.--context-length 8192, apr 0.69.4 (0833bad06, same diff pre-rebase) sha256 53fe2a16f817ffe8, run under gpu-q:4208 tokens … from position 082 tokens … from position 4126(checkpoint resume), then7 tokens from position 4208Residual (not in this fix): a turn that crosses a doubling boundary still reallocates and re-prefills once. Keeping state across growth would mean copying the KV and recurrent state into the bigger buffers, which is a follow-up if wanted. Issue stays open until the batch merges.
noahgift commentedon Sep 25, 2026
#4274 e2e receipt — batch/0.69.4-epic-4249 @90f8ec7a5 (apr 0.69.4 (90f8ec7), exe-sha256 3f67ab9d1d09…, --features cuda)
Qwen3.5-4B Q4_K_M (/mnt/nvme-raid0/lqw/models/Qwen3.5-4B-Q4_K_M.gguf), RTX 4090 lambda,
apr serve run --backend cuda --gpu-layers all --context-length 40000, one gpu-q hold.Warm = one server, two /v1/chat/completions turns (f5's serve_client.py, same prompts + follow-up as receipt-rc1-7ff50ec2a).
Cold = fresh server fed the full turn-2 messages (turn-1 assistant text from warm) = full re-prefill reference. temperature 0.
Backend line: "Backend: GPU (CUDA, NVIDIA GeForce RTX 4090 …) [qwen35 hybrid forward, #3090]". p850 turn-1 TTFT 96.8 s = one-time F2 guard (95.4 s), not prefill.
Raw: warm-.{json,err}, cold-.{json,err}, runs.log in this directory.
fix(serve): a turn past MIN_CAPACITY re-prefilled the next turn from 0 (
fix(serve): #4274 headroom that would not allocate demoted Qwen3.5 to…
fix(session): a repeated or re-rendered prompt resumes from a recurre…
fix(serve): a turn past MIN_CAPACITY re-prefilled the next turn from 0 (
test(#4274): a re-rendered chat turn resumes from turn 1's checkpoint…
merge(rex): re-apply rex/001 on top of main: reconcile with the #4655 …
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 15, 2026
Evidence (apr-4918): the #4274 fix is on
mainsince the #4655 fold (989cb01).<|im_start|>. SeeCheckpoint/checkpoint_atincrates/aprender-serve/src/gguf/inference/forward/qwen35_session.rs.second_chat_turn_resumes_from_the_first_turns_checkpoint_4274(crates/aprender-serve/src/api/qwen35_chat_backend_tests.rs) runs the 0.8B model on the official template, with thinking on and off. Turn 2 does not extend turn 1, reuses more than 0 tokens, and answers the same as a fresh full prefill.a_device_argmax_from_below_the_checkpoint_drops_it(crates/aprender-serve/src/session_tests.rs).No further PR is planned for #4274. The prompt_n accounting is #4214's (perf/4214-prefix-reuse-main), which ships on its own. Closing is the cop's call.