apr serve request path: BPE tokenization, graceful drain, oversize prompts and correct CUDA request batching #5329

Scope: The serve request path: GGUF prompts tokenized by greedy longest-match instead of BPE merges, mis-tokenized end-of-turn tokens, no drain on SIGTERM, and 500s instead of a named refusal for prompts over the KV budget.

Why it matters: These change answers between apr run and apr serve, and break clients during restarts and on long prompts.

Done when: apr serve and apr run produce the same token ids for the same prompt, SIGTERM drains in-flight requests, and over-budget prompts get a 4xx with a reason, each covered by a serve test.

Epic: #3994. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.

Sub-tickets

• apr mis-tokenizes <|im_end|> after sentence punctuation: 4 junk tokens are fed to the model instead of the turn delimiter #2815 apr mis-tokenizes <|im_end|> after sentence punctuation: 4 junk tokens are fed to the model instead of the turn delimiter (milestone 0.72.0); branch fix/2815-im-end-after-punct
  • accept: the outcome in the title, with a receipt
• serve tokenizes GGUF prompts by greedy longest-match, not BPE merges: serve/code answers differ from run #4979 serve tokenizes GGUF prompts by greedy longest-match, not BPE merges: serve/code answers differ from run (milestone none); branch fix/4979-serve-gguf-byte-level-bpe
  • accept: the outcome in the title, with a receipt
• apr serve: drain on SIGTERM/SIGINT — keep listening, 503 + Retry-After for new requests, finish in-flight; today SIGTERM cuts requests to EOF #4449 apr serve: drain on SIGTERM/SIGINT — keep listening, 503 + Retry-After for new requests, finish in-flight; today SIGTERM cuts requests to EOF (milestone 0.71.0); branch fix/4449-serve-drain
  • accept: the outcome in the title, with a receipt
• D5 follow-up: /generate and /batch/generate still return 500 for over-KV CUDA prompts #4619 D5 follow-up: /generate and /batch/generate still return 500 for over-KV CUDA prompts (milestone 0.71.0)
  • accept: the outcome in the title, with a receipt
• D5 follow-up: APR Q4K CUDA chat/completions backends have no serving-context pre-flight (scheduler errors map to 500) #4625 D5 follow-up: APR Q4K CUDA chat/completions backends have no serving-context pre-flight (scheduler errors map to 500) (milestone 0.71.0)
  • accept: the outcome in the title, with a receipt

Merged in (TRIAGE-100, 2026-10-14)

• apr serve on CUDA: batch concurrent Qwen3.5 requests correctly #5312 — apr serve on CUDA: batch concurrent Qwen3.5 requests correctly — reopened 2026-10-14 as a 0.72.0 floor row; tracked on its own ticket, not here
  • done when: At c=4/8/16 on Qwen3.5-4B, apr serve forms batches (m_formed>0), every reply equals the single-request reply token for token, and aggregate tok/s rises with c, shown by a perf041 witness on lambda.
• serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) (milestone none)
  • accept: the outcome in the title, with a receipt
• serve: CUDA batch scheduler over a BatchEngine trait (#5279 P0) #5280 serve: CUDA batch scheduler over a BatchEngine trait (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P0) (milestone none)
  • accept: Done when:
• serve: Qwen3.5 CUDA multi-slot greedy step, looped (#5279 P1) #5289 serve: Qwen3.5 CUDA multi-slot greedy step, looped (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P1) (milestone none)
  • accept: Done when.
• serve: Qwen3.5 CUDA batch engine and serve wiring (#5279 P2) #5293 serve: Qwen3.5 CUDA batch engine and serve wiring (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P2) (milestone none)
  • accept: Done when.
• realizar: batched GEMV row-equal to the single-row kernels for Qwen3.5 (#5279 P3) #5303 realizar: batched GEMV row-equal to the single-row kernels for Qwen3.5 (serve: Qwen3.5 on CUDA never batches concurrent requests (m_formed=0 at c=4/8/16) #5279 P3) (milestone none)
  • accept: Done when.
• V3 bar vs code gap: apr serve runs Qwen3.5 one request at a time, so S3 cannot pass #5046 V3 bar vs code gap: apr serve runs Qwen3.5 one request at a time, so S3 cannot pass (milestone 0.71.1)
  • accept: the outcome in the title, with a receipt
• V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} #4971 V3 serving-shape parity: lambda perf041 witness for Qwen3.5-4B at c in {1,4,8,16} (milestone 0.71.1); branch la-711a/4971-v3a-margin-m3
  • accept: Done when: one lambda perf041 witness for Qwen3.5-4B (sha256 recorded) measures c in {1,4,8,16} with m_formed == c, divergence_at >= declared_min vs m=1 or top2
• P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753 P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens (milestone 0.73.0)
  • accept: the outcome in the title, with a receipt
• perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) #4486 perf(serve): CUDA-graph decode step + alloc-free token loop (RCA-SRV-001 fix 4) (milestone 0.71.0); branch perf/4486-qwen35-decode-graph
  • accept: the outcome in the title, with a receipt