apr serve / apr run: Qwen3.5 on CPU prefills one token at a time (~4 tok/s Q4_K_M on Lambda arm64); the batched prefill is GPU-only #5475

Summary

On the CPU backend the resident Qwen3.5 session prefills the prompt by calling forward_single_qwen35 once per
token (crates/aprender-serve/src/gguf/inference/forward/qwen35_session.rs, forward_one, Backend::Cpu arm).
The batched prefill that landed in 0.69.3 (try_batched_prefill) is #[cfg(feature = "cuda")] and returns None
unless Backend::Gpu. A CPU-only deployment therefore pays prefill linearly in prompt tokens at the single-token
rate, and an agent prompt (system + tool schemas) is mostly prefill.

Measurement

apr 0.71.4 (5027da624), Qwen3.5-0.8B-Q4_K_M.gguf, AWS Lambda arm64 (Graviton3, cpu_part=0xd40), 10,240 MB,
warm environment, temperature 0, max_tokens 1, via /v1/chat/completions:

That is ~4.1 tok/s for prefill. Raw numbers are in the details block at the end. These runs generate one token, so the time is prefill. An earlier two-tool turn with a 64-token reply took 71.5-72.3 s on the same function; the difference is not explained here (prompt text and CPU sample differ), so treat the rate as ~4-6 tok/s.

For an OpenAI-compatible agent endpoint this makes a first turn with a realistic tool list exceed a 120 s client
timeout, and it is the dominant term in per-turn cost.

What we already know about a fix

Our spike 020 built a batched CPU prefill for the hybrid (projections as one trueno::blis GEMM over the row; the
DeltaNet recurrence per head in parallel; conv/gates/norm over the row), ~280 lines on top of upstream's kernels:

• Parity on Kev-0.8B (F32 GGUF, 12 rows): worst |dp| vs torch fp32 1.3e-6, argmax 12/12; vs the token-at-a-time
  path max |dh| 1.5e-4 (GEMM summation order).
• M4 Pro, 6 threads: 87 tokens 6.6 s -> 0.36 s; 915 tokens 73 s -> 3.9 s.
• Limit: F32 weights only. Q4_K_M, Q8_0, BF16 and F16 fall back to the per-row matvec (fused_matmul), so the
  win does not transfer to the quantized model this issue measures until a batched quantized matmul exists.
  We have not measured a Q4_K_M batched prefill and are not claiming a number for it.

Ask

1. Is a CPU batched prefill for Qwen3.5 something you would take? If yes, the open design question is the
   quantized kernel: dequantize a tile to F32 and call the GEMM, or a row-batched fused Q4_K x Q8_K kernel.
2. Whatever lands, the session already logs which prefill it took on GPU; the CPU path should log the same line so
   a per-token regression is visible in a serve log.

Not in scope here

Cross-request reuse of the decode state (separate issue). GPU behaviour (we did not measure it).

Related

• Qwen3.5 speed: prefill and decode parity with llama.cpp, and the harness that measures it #5305 (Qwen3.5 prefill and decode parity) and its sub-tickets perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450 / perf(serve): Qwen3.5 tensor-core prefill projections (RCA-SRV-001 fix 1) #4483 / perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) #4313: all of that work is on the CUDA path.
  This issue is the CPU-only deployment (no GPU, e.g. Lambda / Fargate arm64), which none of them cover.
• Time-to-first-token on Qwen3.5-4B-Q4_K_M is 14-16s vs llama.cpp's 1.8-2.9s on the same box (7.7x); apr's decode rate is unmeasurable with shipped tooling #3596 (closed): the same one-token-at-a-time symptom on GPU, fixed by the 0.69.3 batched prefill.
• EPIC E2 0.73: Verbs Are Fast #3598 (E2 epic).

{
  "date": "2026-10-16",
  "function": "aprender-llm-endpoint-dev",
  "shape": "tools2 (421 prompt tokens), max_tokens=1, temperature 0",
  "mechanism": "crates/aprender-serve/src/gguf/inference/forward/qwen35_session.rs header: a prompt that EXTENDS the held tokens prefills only the suffix; any other resets and prefills whole",
  "run1": [
    {
      "label": "A_first",
      "duration_ms": 101861,
      "init_ms": 9069
    },
    {
      "label": "B_same_prefix_different_tail",
      "duration_ms": 101945
    },
    {
      "label": "C_new_prompt",
      "duration_ms": 101693
    },
    {
      "label": "D_identical_to_C",
      "duration_ms": 1808
    }
  ],
  "run2": [
    {
      "label": "X_first",
      "duration_ms": 101794
    },
    {
      "label": "X_again",
      "duration_ms": 1805
    },
    {
      "label": "Y_other",
      "duration_ms": 101356
    },
    {
      "label": "X_after_Y",
      "duration_ms": 101492
    }
  ],
  "not_measured": [
    "cpu_part per call (all earlier runs were Graviton3)",
    "a real tool-call turn 2 in this run (earlier measured 4.6-8.1 s)",
    "longest-common-prefix reuse across conversations (absent by design)"
  ]
}