Repository navigation
D5 follow-up: APR Q4K CUDA chat/completions backends have no serving-context pre-flight (scheduler errors map to 500) #4625
Description
Split out of PMAT-4616 / #4614 (D5), which covers the dense GGUF CUDA serve paths.
try_apr_q4k_chat_backend (crates/aprender-serve/src/api/cuda_chat_backend.rs ~795-889) and its mirror in gpu_completions_handler.rs (~295-334) send to the APR Q4K scheduler with no serving-context pre-flight, and map every scheduler error (a String) to 500. The scheduler bounds at metadata().max_position_embeddings (apr_q4k_scheduler.rs:130), which the with_apr_q4k_and_vocab* constructors never see, so AppState::serving_context() is None on those states.
Fix: pass the Q4K context into the state at construction (set cached_serving_context), so the existing preflight_serving_context / serve_context_refusal 400 applies before the send. Found by the quorum (sonnet lane) on #4614 @1531ead607.
Agent: aprender-2e-serve
🤖 Generated with Claude Code
Activity
docs(PMAT-4616): scope criterion 3 to dense GGUF CUDA; APR Q4K split to
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5329 (apr serve request path: BPE tokenization, graceful drain, no 500 on oversize prompts), split out of #5312 by theme.