P0: streaming /v1/chat/completions never emits tool_calls — tool calls arrive as raw <tool_call> text in delta.content, finish_reason "stop" #4918

Severity: P0

Tool calling over /v1/chat/completions works only when "stream": false. With "stream": true, apr sends the model's tool call to the client as ordinary text: the <tool_call>…</tool_call> markup arrives in delta.content and the stream ends with finish_reason: "stop". Agent clients (OpenCode, and in practice any agent harness) always stream, so they show the raw call to the user and never run it.

Reproduction (apr 0.70.2 89e261cda, CUDA build, RTX 4090)

apr serve run ~/Downloads/Qwen3.5-4B-Q4_K_M.gguf --gpu    # chat template: Qwen3NoThink

{"model":"qwen3-coder",
 "messages":[{"role":"user","content":"what are the directories in ~/tmp/"}],
 "tools":[{"type":"function","function":{"name":"bash","description":"Run a shell command",
   "parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}}],
 "max_tokens":200,"temperature":0}

"stream": false: correct.

"message": {"role":"assistant","content":"",
  "tool_calls":[{"id":"call_0","type":"function",
    "function":{"name":"bash","arguments":"{\"command\":\"ls -la ~/tmp/\"}"}}]},
"finish_reason": "tool_calls"

"stream": true: broken. The same generation (30 completion tokens) comes back as content deltas:

data: {..."delta":{"role":"assistant"}...,"stream_mode":"live"}
data: {..."delta":{"content":"<tool_call>"}...}
data: {..."delta":{"content":"\n"}...}
data: {..."delta":{"content":"<function"}...}
data: {..."delta":{"content":"=b"}...}
data: {..."delta":{"content":"ash"}...}
   … <parameter=command>\nls -la ~/tmp/\n</parameter>\n</function> …
data: {..."delta":{"content":"\n</tool_call>"}...}
data: {..."delta":{},"finish_reason":"stop"}
data: [DONE]

In OpenCode the assistant turn renders as the literal text <tool_call>\n<function=bash>\n<parameter=command>…, and no tool runs.

Root cause (traced in code on main @ 828ccb845)

1. Tool-call parsing runs only on the non-streaming path. build_chat_response (crates/aprender-serve/src/api/openai_handlers.rs:698) calls build_tool_calling_message (:621) when request.tools is set. PMAT-801 (feat(serve): wire tool/function-calling into OpenAI /v1/chat/completions (non-streaming) — PMAT-801 #2086) was explicitly scoped "(non-streaming)", and P0: apr serve run never passes tools into the Qwen3.5 chat template — tool calling silently dead for GGUF hybrid sessions #4650 / fix(serve): request tools reach the chat template; Qwen3.5 XML tool calls parse (#4650) #4866 added Qwen3.5 <function=…> XML parsing to that same non-streaming path only.
2. Neither SSE builder receives request.tools. true_streaming_sse_response (openai_handlers.rs:959; used by the qwen35, CUDA, CPU-quantized and MoE arms) and pregenerated_sse_response (:842; used by the GPU, cached, APR, registry and MoE-GPU arms) write every decoded delta as content. They pick the terminal finish_reason with FinishReason::from_generation, which can only return stop or length.
3. The streaming wire type has nowhere to put a call. ChatDelta (crates/aprender-serve/src/api/mod_create_demo.rs:801) has only role and content, with no tool_calls field.
4. For the reported model, the request goes through try_qwen35_backend (crates/aprender-serve/src/api/qwen35_chat_backend.rs:85). It sends the streaming branch to true_streaming_sse_response (:163) and only the non-streaming branch to build_chat_response (:221).

Impact

• P0: apr serve run never passes tools into the Qwen3.5 chat template — tool calling silently dead for GGUF hybrid sessions #4650 / fix(serve): request tools reach the chat template; Qwen3.5 XML tool calls parse (#4650) #4866 made the model emit correct tool calls (tools now reach the Qwen3.5 template). Streaming clients still can't receive them, so for OpenCode-style agents, tool calling through apr serve doesn't work end to end.
• It fails silently: the server returns HTTP 200 with well-formed SSE, and the defect only shows up as wrong client behaviour.
• The path is shared, so this affects every backend arm that streams, not just Qwen3.5.

Fix shape

• Apply tool-call detection to both SSE builders, after the existing ChatStopFilter, and only when the request declares tools and tool_choice is not "none".
• Stream text live, but hold back any tail that could still grow into a <tool_call> or <function= marker. Once a marker appears, capture the rest of the output.
• At end of stream, parse the captured text with the same build_tool_calling_message the non-streaming path uses, so the two paths can't disagree about what counts as a call.
• Emit the calls as an OpenAI streaming delta.tool_calls chunk (index, id, type: "function", function.name, function.arguments as a JSON string), and set the terminal chunk's finish_reason to "tool_calls".
• If the captured text parses to no call (an undeclared tool, or a mere mention of the tag), flush it as content so nothing is lost.
• Without tools, or with tool_choice: "none", the stream must stay byte-identical to today's.

Acceptance / falsifiers

• A test replays the exact delta sequence above (the markers split across tokens) through true_streaming_sse_response. It asserts that no content delta carries any part of the call, that exactly one delta.tool_calls[0] arrives with name == "bash" and arguments {"command":"ls -la ~/tmp/"}, and that the terminal finish_reason == "tool_calls". Making the detector a pass-through must turn this test RED.
• With tools declared, a plain-text answer still streams token by token and finishes "stop".
• Live check: a streaming /v1/chat/completions request with tools, against a Qwen3.5 GGUF on CUDA, returns delta.tool_calls and finish_reason: "tool_calls".

Out of scope

The non-standard /v1/chat/completions/stream endpoint (chat_completions_stream.rs) and the Ollama-compat /api/chat stream (see #4182) have the same gap. Either track them here as follow-ups or split them into their own issues.

Why P0

OpenCode and other agentic clients default to streaming against OpenAI-compatible endpoints. #4650 was P0 because tool calling was silently dead for Qwen3.5 GGUF sessions; this is the other half of that defect, and until it's fixed the #4650 fix has no effect for every streaming client.

Refs #4650, #4866, #2086 (PMAT-801).