apr serve: Qwen3.5 session reuses state only when a prompt extends the previous one; a shared system+tools prefix is lost whenever the tail differs #5476

Summary

Qwen35Session keeps one decode state. Per its module doc: "a prompt that EXTENDS the tokens the state holds
prefills only the new suffix; any other prompt resets the state in place and prefills whole". That is the right
behaviour for one chat. For an agent endpoint serving several conversations (or several agents) that share the same
system prompt and tool schemas, it means the shared prefix is re-prefilled every time the conversation changes.

There is no flag or request field to control it (apr serve exposes no pin / prefix-cache option). The CUDA path has a
separate PrefixCache (gguf/batch_scheduler_prefix_cache.rs, feature = "gpu") that keys on a hash of the whole
prompt and needs entry.tokens == tokens, so it does not do longest-prefix either, and the CPU session does not use it.

Measurement

apr 0.71.4 (5027da624), Qwen3.5-0.8B-Q4_K_M.gguf, Lambda arm64, warm, max_tokens 1, prompt ~420 tokens
(system + 2 tool schemas + user line). Prefill is slow here (see the batched-prefill issue), which makes reuse
easy to see:

Raw numbers are in the details block at the end. A tool-call turn 2 that extends turn 1 was measured separately at
4.6-8.1 s, consistent with the extension rule.

Why it is not just "keep more of the KV cache"

Qwen35State is conv_states + ssm_states (DeltaNet recurrent state) + kv_cache. The recurrent part cannot be
rewound to an earlier position the way a KV cache can be truncated, so reusing a shared prefix means snapshotting
the whole state at the prefix boundary, not remembering a longer cache.

Proposal (for discussion, not a patch)

• Snapshot the state at the end of the system message (the template boundary the server already knows) into a small
  LRU of N slots keyed by the token ids of that prefix; on a request whose prompt starts with a stored prefix,
  restore the snapshot and prefill only the rest.
• Automatic, no new API required. If an explicit control is wanted, an opt-in request field beats a flag, because the
  caller is the one who knows which prefix is shared.
• Memory: each slot costs the recurrent state plus the KV for the prefix; N should be bounded and reported.

Questions

1. Is a snapshot at the system-message boundary acceptable, or should the boundary be caller-chosen?
2. Should the CUDA PrefixCache and this CPU mechanism share an interface so both servers behave the same?
3. Is there a reason the single-slot rule is intentional for apr serve (memory, determinism)?

Not in scope

Faster prefill itself (separate issue): reuse only helps the second conversation onward and does nothing for a cold
start or a recycled environment.

Related

• apr serve (0.69.1) has no prompt-prefix reuse on the qwen35 CUDA path: an identical ~850-token prompt costs 11.0 s on both runs (ollama: 8.2k tok/s uncached, 37k cached) #4214 (no prefix reuse on the qwen35 CUDA path, identical prompt twice) and serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions #4274 (turn 2 re-prefills from position 0
  via /v1/chat/completions, 0.69.3). Those are the GPU path. On the CPU path at 0.71.4 the extension rule does
  engage (identical repeat 1.8 s; tool-call turn 2 4.6-8.1 s), so this issue is only about what the CPU rule cannot do:
  a shared prefix with a different tail, and more than one conversation.
• Qwen3.5 speed: prefill and decode parity with llama.cpp, and the harness that measures it #5305 lists "session reuse so turn 2 does not re-prefill" as a done-when; this is the multi-conversation half of it.

{
  "date": "2026-10-16",
  "function": "aprender-llm-endpoint-dev",
  "shape": "tools2 (421 prompt tokens), max_tokens=1, temperature 0",
  "mechanism": "crates/aprender-serve/src/gguf/inference/forward/qwen35_session.rs header: a prompt that EXTENDS the held tokens prefills only the suffix; any other resets and prefills whole",
  "run1": [
    {
      "label": "A_first",
      "duration_ms": 101861,
      "init_ms": 9069
    },
    {
      "label": "B_same_prefix_different_tail",
      "duration_ms": 101945
    },
    {
      "label": "C_new_prompt",
      "duration_ms": 101693
    },
    {
      "label": "D_identical_to_C",
      "duration_ms": 1808
    }
  ],
  "run2": [
    {
      "label": "X_first",
      "duration_ms": 101794
    },
    {
      "label": "X_again",
      "duration_ms": 1805
    },
    {
      "label": "Y_other",
      "duration_ms": 101356
    },
    {
      "label": "X_after_Y",
      "duration_ms": 101492
    }
  ],
  "not_measured": [
    "cpu_part per call (all earlier runs were Graviton3)",
    "a real tool-call turn 2 in this run (earlier measured 4.6-8.1 s)",
    "longest-common-prefix reuse across conversations (absent by design)"
  ]
}