Repository navigation
fix(serve): Ollama shim reports done_reason "length" when num_predict is hit (Refs #5357) - #5414
Conversation
…num_predict (#5357) The apr-cli /api/chat and /api/generate shim hardcoded done_reason stop on streamed replies and sent none on coalesced ones. Coalesced and batch-NDJSON bodies now read the upstream finish_reason; the live token stream reports length once it has emitted num_predict tokens. Refs #5357 Agent: la-0.71.1a Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stop" (Refs #5357) Review of the done_reason change found two gaps. A stream that errored after reaching num_predict reported "length"; it now reports "stop", since only a clean close can mean the cap was hit. The reshape tests only covered finish_reason "length", so a shim that always said "length" would have passed; a "stop" body case pins the other side. apr-cli serve lib tests: 405 passed. Agent: la-0.71.1a Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift
left a comment
There was a problem hiding this comment.
seat 1, aprender-seat-o10 @c0e73ccbed APPROVE
Opus seat. be492a5 and c0e73cc are Sonnet 5.5 (la-0.71.1a). The middle commit 5e20d9b is a main-merge with no conflict hunks (git diff-tree --cc is empty). This session authored no commit on the PR.
Read and checked at the head:
- Coalesced chat/generate:
openai_done_reasonreturns "length" only on a 2xx whosechoices[0].finish_reason == "length". Anything else is "stop": non-2xx, unparseable body, missing field. That is the conservative default; it never reports a cap hit it cannot see. - Batch NDJSON reshape: it maps the upstream reply to
StreamEnd::Length/Stop, which is consistent with the coalesced path. - Live APR-CPU stream: I checked
spawn_cpu_token_text_streamat :181. It sends one channel message per token id via thegeneratecallback, so the stream'scountis a token count andLengthAt(max_tokens)compares like with like.max_tokenscomes fromollama_sampling:num_predict(u32) or a default of 32, capped at 4096. So a turn that hits the default cap also correctly reads "length". - Backend errors: an
Erron the channel is forced to "stop" (ended @ (Some(Err(_)) | None)), and the rowlive_stream_error_at_the_cap_reports_stop_not_lengthpins it. Agenerate()failure only reaches eprintln and then closes the channel, so it can read "length" only if the cap had already been reached. - Other callers keep their old behavior:
ollama_ndjson_streamnow delegates withStreamEnd::Stop, so its other callers (ollama_stream_error, the existing tests) still answer "stop".ollama_stream_finalhas exactly one non-test caller, and it was updated. - Tests: they cover both reasons on every path: the coalesced chat/generate and the batch reshape for length and stop, the live stream at cap-1, at the cap and on error, and the
StreamEndboundaries at 2/3. Each assertion would fail if the old hardcoded "stop" came back. The PR body reports 405 passed forcommands::serve; I did not re-run it.
Not blocking, and already stated in the body:
LengthAtis approximate: a natural EOS on exactly the last allowed token reads "length". If thegeneratecallback also fires for the EOS id, a reply of cap−1 content tokens plus EOS reads "length" too. Worth one row on #5357 once the SafeTensorsfinish_reasonfollow-up lands.num_predict: 0givesLengthAt(0)→ "length" on an empty turn. Arguably correct, since the cap was hit.
CI on this head was PENDING at read (authorize/x86-main/shards queued or running). This APPROVE is bound to c0e73cc; the PR is READY only once the rollup reads green on it.
noahgift
left a comment
There was a problem hiding this comment.
seat 2, aprender-seat-o9 @c0e73ccbed -- APPROVE (no blocking finding)
Reviewed the full diff (be492a5, the main merge 5e20d9b, c0e73cc): ollama.rs and handler_apr_cpu_completion.rs. It is Sonnet-authored, and this seat runs Opus.
- Coalesced chat/generate:
openai_done_reasonreadschoices[0].finish_reason. It returns "length" only for a 2xx withfinish_reason == "length". Non-2xx, bad JSON or a missing field return "stop", which is the old behaviour, so the change never invents a "length". - Batch NDJSON: the upstream "length" maps to
StreamEnd::Length. Anything else maps toStop. - Live APR-CPU stream:
StreamEnd::LengthAt(max_tokens)gives "length" whencount >= cap.spawn_cpu_token_text_streamsends oneOkper generated token, socountis a token count. The exact-cap natural-stop case the PR body states is the only false "length". ASome(Err(_))terminal is forced to "stop" (c0e73cc), and a test covers it. ollama_ndjson_streamkeeps its signature through aStopwrapper, so other callers are unchanged.ollama_stream_finalhas one non-test caller, and it was updated.- Tests discriminate: cap 2 vs 3 over 2 tokens, length vs stop for each reshaped route, error at the cap, and the four
openai_done_reasoncases.
Measured locally on a worktree at c0e73cc: cargo test -p apr-cli --lib -- commands::serve::ollama gave 25 passed and 0 failed, rc=0.
Non-blocking: as the body says, the SafeTensors upstream still hardcodes finish_reason: "stop", so those routes stay "stop". keep-open: #5357 tracks it.
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…n change (Refs #5357) check_dogfood_coverage G2.1: ollama.rs changed. The /api/pull and /api/delete rows keep their meaning; the done_reason change added 76 lines above them, so their cited lines move 650 -> 726 and 654 -> 730. Agent: la-0.71.1a Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
noahgift
left a comment
There was a problem hiding this comment.
seat 1, aprender-seat-o10 @95dfdbd8ad APPROVE
Re-seat. The only change from c0e73cc (my seat-1 APPROVE above, whose reasoning carries over unchanged) is 95dfdbd. It is a Sonnet 5.5 commit that touches only docs/audits/surface_audit.csv, 2 lines. No code changed.
The two apr-cli .../serve/ollama.rs refs move from :650/:654 to :726/:730. That is a uniform +76 shift, which matches the 76 lines this PR adds above the route table (ollama_pull_body moves 622 → 698).
Not blocking. The bad refs come from main, and this commit faithfully carries them forward: both refs were already off on main and are still off here.
- On main, :650 is
};and :654 is"/api/show". The real lines were/api/pull:659 and/api/delete:664. - At this head, :726 is
};and :730 is"/api/show". The correct refs are POST /api/pull → :735 and DELETE /api/delete → :740.
Worth fixing in this PR if it is cheap. Otherwise it belongs on the stale-ref ticket I recommended on #5393: whatever G2.1 checks, it does not check that a ref lands on the route it names.
CI on this head was PENDING at read: x86-main and shards 1-3 in progress. This APPROVE is bound to 95dfdbd; READY only once the rollup is green on it.
noahgift
left a comment
There was a problem hiding this comment.
seat 2, aprender-seat-o9 @95dfdbd8ad -- APPROVE (no blocking finding)
Delta reviewed: c0e73cc..95dfdbd, one commit. It changes docs/audits/surface_audit.csv only (2+/2-). git diff c0e73ccbed 95dfdbd8ad -- crates is empty, so my code review at c0e73cc carries over unchanged.
The renumber is consistent. At the merge base, ollama.rs has vec![ at 651, "/api/pull" at 659 and "/api/delete" at 664, and the ledger anchors are 650 and 654. At 95dfdbd those lines are 727, 735 and 740. The new anchors 726 and 730 keep the same offsets (+76, matching the PR's inserted lines). No ledger row is added, dropped or re-graded.
The apr-cli Ollama shim always ended a turn with
done_reason: "stop". It now reports"length"when the turn hitnum_predict:finish_reason.StreamEnd::LengthAt(max_tokens). This is approximate: a natural stop on exactly the last allowed token also reads "length". A backend error reports "stop".Not covered: the SafeTensors upstream still hardcodes
finish_reason"stop", so its Ollama routes cannot report "length" yet.cargo test -p apr-cli --lib -- commands::serve: 405 passed.Refs #5357
keep-open: #5357 tracks the remaining shim gaps (SafeTensors finish_reason, the compat_ollama copy, json_schema refusal text).
ont-delta: none
🤖 Generated with Claude Code