Qwen3.5 speed: prefill and decode parity with llama.cpp, and the harness that measures it #5305

Scope: Prefill (prompt processing) for Qwen3.5 in apr run/serve: int8 MMQ and tensor-core projections, flash attention on full-attention layers, no one-token-at-a-time fallback, chunked-prefill token budget, and session reuse so turn 2 does not re-prefill from position 0.

Why it matters: Prefill sets time-to-first-token. Today pp is about 0.2x llama.cpp, so every long prompt and every chat turn waits several times longer than on the competitor.

Done when: llama-bench style pp512 on the same GGUF is at least 0.8x llama.cpp on the lambda GPU host, with a committed receipt, and a second chat turn reuses the session (no re-prefill from 0).

Epic: #3598. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.

Sub-tickets

• perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) #4313 perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) (milestone 0.71.0); branch 79/4313-on-4502
  • accept: the outcome in the title, with a receipt
• 0.70.x: prefill kernel parity — llama-bench pp ≥0.8x llama.cpp (today 0.19–0.20x) #4376 0.70.x: prefill kernel parity — llama-bench pp ≥0.8x llama.cpp (today 0.19–0.20x) (milestone backlog); branch 79/4376-mmq
  • accept: the outcome in the title, with a receipt
• qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442 qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) (milestone 0.71.0); branch fix/4442-qwen35-flash-prefill-4b
  • accept: the outcome in the title, with a receipt
• perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450 perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4220 child) (milestone 0.71.0); branch fb/4450-prefill-ladder
  • accept: the outcome in the title, with a receipt
• perf(serve): Qwen3.5 tensor-core prefill projections (RCA-SRV-001 fix 1) #4483 perf(serve): Qwen3.5 tensor-core prefill projections (RCA-SRV-001 fix 1) (milestone 0.71.0); branch wprod/4483-tc-prefill
  • accept: the outcome in the title, with a receipt
• realizar: chunked prefill has no token-budget term, so continuous-batching-v1 chunked_prefill has no binding #5007 realizar: chunked prefill has no token-budget term, so continuous-batching-v1 chunked_prefill has no binding (milestone none)
  • accept: Done when:
• serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions #4274 serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions (milestone 0.71.0); branch fix/4274-on-0.69.3
  • accept: Done when: turn 2 extends from kv_len > 0, shown by the trace line, and the text is identical to a full re-prefill.

Merged in (TRIAGE-100, 2026-10-14)

• Qwen3.5 CUDA decode speed: close the GEMV and DeltaNet gaps to llama.cpp #5320 — Qwen3.5 CUDA decode speed: close the GEMV and DeltaNet gaps to llama.cpp

  • done when: tg128 on the same GGUF is at least 0.9x llama.cpp on the lambda GPU host, and 16k-context decode is not more than 2x slower than 512-context, each with a committed receipt.

• Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) #3725 Qwen3.5 CUDA long-context DECODE is slow: the decode attention kernel runs 16 blocks — split-K needed for consumer use (RAH ~10 tok/s at 16k) (milestone 0.72.0); branch PMAT-3725-splitk-on-main

  • accept: the outcome in the title, with a receipt

• perf(serve): decode GEMV precision on Qwen3.5 (RCA-SRV-001 fix 3) #4485 perf(serve): decode GEMV precision on Qwen3.5 (RCA-SRV-001 fix 3) (milestone 0.71.0); branch perf/4485-decode-gemv-precision

  • accept: the outcome in the title, with a receipt

• perf(qwen35-cuda): decode GEMV at 223–504 GB/s vs llama.cpp MMVQ 776–894 — Q5_K 3.5×, Q4_K/Q6_K 1.75× slower (79% of the 4B decode gap) #5221 perf(qwen35-cuda): decode GEMV at 223–504 GB/s vs llama.cpp MMVQ 776–894 — Q5_K 3.5×, Q4_K/Q6_K 1.75× slower (79% of the 4B decode gap) (milestone 0.71.1); branch perf/5221-qwen35-decode-gemv-bw

  • accept: Acceptance:

• perf(qwen35-cuda): gdn_delta_rule_recurrence 44.7 µs/call vs llama.cpp 2.5 µs (17.5×) — ~20% of the 4B decode gap #5222 perf(qwen35-cuda): gdn_delta_rule_recurrence 44.7 µs/call vs llama.cpp 2.5 µs (17.5×) — ~20% of the 4B decode gap (milestone 0.71.1); branch perf/5222-gdn-recurrence

  • accept: Acceptance:

• Speed measurement harness: resource receipts, observability and the ollama/llama.cpp ratio #5322 — Speed measurement harness: resource receipts, observability and the ollama/llama.cpp ratio

  • done when: One command produces a receipt with resource numbers and the apr/competitor ratio on the same GGUF and client, and that receipt is what release notes cite.

• P0: measured token usage for the fleet -- Claude Code + agy, by project / model / lane / hour, with the top levers #4162 P0: measured token usage for the fleet -- Claude Code + agy, by project / model / lane / hour, with the top levers (milestone 0.71.0); branch a01/4162-on-ont-10

  • accept: the outcome in the title, with a receipt

• Release gate: apr serve throughput vs llama.cpp on the SAME GGUF (Qwen3.5 0.8B/2B/4B/9B/27B) — a regression ratchet that blocks the tag, plus a published parity target #4218 Release gate: apr serve throughput vs llama.cpp on the SAME GGUF (Qwen3.5 0.8B/2B/4B/9B/27B) — a regression ratchet that blocks the tag, plus a published parity target (milestone backlog)

  • accept: Done when: the 0.69.4 tag can't be cut without a committed scoreboard receipt whose ratchet verdict is PASS, and reverting Time-to-first-token on Qwen3.5-4B-Q4_K_M is 14-16s vs llama.cpp's 1.8-2.9s on the same box (7.7x); apr's decode rate is unmeasurable with shipped tooling #3596 makes the gate RED.

• APR-OBS-001: dogfood observability (OBS-00 + row checklist) #4487 APR-OBS-001: dogfood observability (OBS-00 + row checklist) (milestone 0.71.0); branch fold/4487-obs-00

  • accept: the outcome in the title, with a receipt

• MEAS-001: granular resource receipts (RSS/VRAM/CPU/IO/faults/energy) for bench, CRUX, dogfood, CI ledger #4522 MEAS-001: granular resource receipts (RSS/VRAM/CPU/IO/faults/energy) for bench, CRUX, dogfood, CI ledger (milestone 0.71.0); branch 52/4522-r1-resources

  • accept: Done when a real receipt from nightly and from an rc shows every field non-null (or null with a stated reason), and one gate has been proven RED by mutation.

• client-driven Ollama arm (apr test llm bench + client sha256) before any ratio vs Ollama #5033 client-driven Ollama arm (apr test llm bench + client sha256) before any ratio vs Ollama (milestone 0.72.0)

  • accept: the outcome in the title, with a receipt