Repository navigation
D5 follow-up: /generate and /batch/generate still return 500 for over-KV CUDA prompts #4619
Description
Follow-up to #4616 / PR #4614 (OPERATOR D5: an over-length request must get a clear 4xx, never a 500).
The non-author pr-review receipt for #4614 (sonnet-5; agy gemini corroborated it) measured that try_cuda_generate (crates/aprender-serve/src/api/batch_processing.rs:492) and try_cuda_batch_generate (batch.rs:515) map every CUDA error to 500. So /generate and /batch/generate still turn an over-KV prompt into a 500. The CUDA chat, completions and stream handlers do not; #4614 fixed those.
Fix: the same serve_context_refusal pre-flight, plus generation_error_status, on both paths. It was kept out of #4614 so that PR's signed receipt stays valid.
Gate question for the cop: does any gate-named serve cell hit /generate? If one does, this joins the rc.2 set.
Activity
noahgift commentedon Sep 28, 2026
Scope note (2e, read-only): the ladder calls /v1/chat/completions, /v1/completions and /api/chat, never /generate (scripts/model_ladder.sh:778, check_ladder_serve_verdict.sh), so this is off the gate path. Separate observation: the Ollama /api/chat and /api/generate reshape (apr-cli serve/ollama.rs:350, openai_response_to_parts:304) drops the inner status and returns 200 with the error text as content. An over-length refusal (400) there becomes a 200 whose body is the error. That is by design (PMAT-923: 'ALWAYS yields parts'), but a gate reading only HTTP status on /api/chat would count it as a pass. The cop should decide whether that needs a ticket.
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
apr codeon Qwen3.5 Q4_K_M (4B, 9B) on CUDA: does it complete a scripted edit-and-verify task on lambda and gx10? UNMEASURED, and its driver runs throughapr serve --gpu(#3571) #3719noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5329 (apr serve request path: BPE tokenization, graceful drain, no 500 on oversize prompts), split out of #5312 by theme.