Skip to content

fix(serve): Ollama shim reports done_reason "length" when num_predict is hit (Refs #5357) - #5414

Merged
noahgift merged 4 commits into
mainfrom
la-711a/5357-shim-done-reason
Oct 16, 2026
Merged

noahgift merged 4 commits into
mainfrom
la-711a/5357-shim-done-reason

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

The apr-cli Ollama shim always ended a turn with done_reason: "stop". It now reports "length" when the turn hit num_predict:

  • Coalesced chat/generate and batch-NDJSON replies read the upstream OpenAI finish_reason.
  • The live APR-CPU stream ends with StreamEnd::LengthAt(max_tokens). This is approximate: a natural stop on exactly the last allowed token also reads "length". A backend error reports "stop".

Not covered: the SafeTensors upstream still hardcodes finish_reason "stop", so its Ollama routes cannot report "length" yet.

cargo test -p apr-cli --lib -- commands::serve: 405 passed.

Refs #5357
keep-open: #5357 tracks the remaining shim gaps (SafeTensors finish_reason, the compat_ollama copy, json_schema refusal text).
ont-delta: none

🤖 Generated with Claude Code

noahgift and others added 3 commits October 16, 2026 01:17
…num_predict (#5357)

The apr-cli /api/chat and /api/generate shim hardcoded done_reason stop on
streamed replies and sent none on coalesced ones. Coalesced and batch-NDJSON
bodies now read the upstream finish_reason; the live token stream reports
length once it has emitted num_predict tokens.

Refs #5357

Agent: la-0.71.1a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stop" (Refs #5357)

Review of the done_reason change found two gaps. A stream that errored after
reaching num_predict reported "length"; it now reports "stop", since only a
clean close can mean the cap was hit. The reshape tests only covered
finish_reason "length", so a shim that always said "length" would have passed;
a "stop" body case pins the other side.
apr-cli serve lib tests: 405 passed.

Agent: la-0.71.1a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 1, aprender-seat-o10 @c0e73ccbed APPROVE

Opus seat. be492a5 and c0e73cc are Sonnet 5.5 (la-0.71.1a). The middle commit 5e20d9b is a main-merge with no conflict hunks (git diff-tree --cc is empty). This session authored no commit on the PR.

Read and checked at the head:

  • Coalesced chat/generate: openai_done_reason returns "length" only on a 2xx whose choices[0].finish_reason == "length". Anything else is "stop": non-2xx, unparseable body, missing field. That is the conservative default; it never reports a cap hit it cannot see.
  • Batch NDJSON reshape: it maps the upstream reply to StreamEnd::Length/Stop, which is consistent with the coalesced path.
  • Live APR-CPU stream: I checked spawn_cpu_token_text_stream at :181. It sends one channel message per token id via the generate callback, so the stream's count is a token count and LengthAt(max_tokens) compares like with like. max_tokens comes from ollama_sampling: num_predict (u32) or a default of 32, capped at 4096. So a turn that hits the default cap also correctly reads "length".
  • Backend errors: an Err on the channel is forced to "stop" (ended @ (Some(Err(_)) | None)), and the row live_stream_error_at_the_cap_reports_stop_not_length pins it. A generate() failure only reaches eprintln and then closes the channel, so it can read "length" only if the cap had already been reached.
  • Other callers keep their old behavior: ollama_ndjson_stream now delegates with StreamEnd::Stop, so its other callers (ollama_stream_error, the existing tests) still answer "stop". ollama_stream_final has exactly one non-test caller, and it was updated.
  • Tests: they cover both reasons on every path: the coalesced chat/generate and the batch reshape for length and stop, the live stream at cap-1, at the cap and on error, and the StreamEnd boundaries at 2/3. Each assertion would fail if the old hardcoded "stop" came back. The PR body reports 405 passed for commands::serve; I did not re-run it.

Not blocking, and already stated in the body:

  1. LengthAt is approximate: a natural EOS on exactly the last allowed token reads "length". If the generate callback also fires for the EOS id, a reply of cap−1 content tokens plus EOS reads "length" too. Worth one row on #5357 once the SafeTensors finish_reason follow-up lands.
  2. num_predict: 0 gives LengthAt(0) → "length" on an empty turn. Arguably correct, since the cap was hit.

CI on this head was PENDING at read (authorize/x86-main/shards queued or running). This APPROVE is bound to c0e73cc; the PR is READY only once the rollup reads green on it.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2, aprender-seat-o9 @c0e73ccbed -- APPROVE (no blocking finding)

Reviewed the full diff (be492a5, the main merge 5e20d9b, c0e73cc): ollama.rs and handler_apr_cpu_completion.rs. It is Sonnet-authored, and this seat runs Opus.

  • Coalesced chat/generate: openai_done_reason reads choices[0].finish_reason. It returns "length" only for a 2xx with finish_reason == "length". Non-2xx, bad JSON or a missing field return "stop", which is the old behaviour, so the change never invents a "length".
  • Batch NDJSON: the upstream "length" maps to StreamEnd::Length. Anything else maps to Stop.
  • Live APR-CPU stream: StreamEnd::LengthAt(max_tokens) gives "length" when count >= cap. spawn_cpu_token_text_stream sends one Ok per generated token, so count is a token count. The exact-cap natural-stop case the PR body states is the only false "length". A Some(Err(_)) terminal is forced to "stop" (c0e73cc), and a test covers it.
  • ollama_ndjson_stream keeps its signature through a Stop wrapper, so other callers are unchanged. ollama_stream_final has one non-test caller, and it was updated.
  • Tests discriminate: cap 2 vs 3 over 2 tokens, length vs stop for each reshaped route, error at the cap, and the four openai_done_reason cases.

Measured locally on a worktree at c0e73cc: cargo test -p apr-cli --lib -- commands::serve::ollama gave 25 passed and 0 failed, rc=0.

Non-blocking: as the body says, the SafeTensors upstream still hardcodes finish_reason: "stop", so those routes stay "stop". keep-open: #5357 tracks it.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=5414 head=95dfdbd8adb3658858d972417fae293aa3f3266c verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…n change (Refs #5357)

check_dogfood_coverage G2.1: ollama.rs changed. The /api/pull and /api/delete
rows keep their meaning; the done_reason change added 76 lines above them, so
their cited lines move 650 -> 726 and 654 -> 730.

Agent: la-0.71.1a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@noahgift noahgift left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 1, aprender-seat-o10 @95dfdbd8ad APPROVE

Re-seat. The only change from c0e73cc (my seat-1 APPROVE above, whose reasoning carries over unchanged) is 95dfdbd. It is a Sonnet 5.5 commit that touches only docs/audits/surface_audit.csv, 2 lines. No code changed.

The two apr-cli .../serve/ollama.rs refs move from :650/:654 to :726/:730. That is a uniform +76 shift, which matches the 76 lines this PR adds above the route table (ollama_pull_body moves 622 → 698).

Not blocking. The bad refs come from main, and this commit faithfully carries them forward: both refs were already off on main and are still off here.

  • On main, :650 is }; and :654 is "/api/show". The real lines were /api/pull :659 and /api/delete :664.
  • At this head, :726 is }; and :730 is "/api/show". The correct refs are POST /api/pull → :735 and DELETE /api/delete → :740.

Worth fixing in this PR if it is cheap. Otherwise it belongs on the stale-ref ticket I recommended on #5393: whatever G2.1 checks, it does not check that a ref lands on the route it names.

CI on this head was PENDING at read: x86-main and shards 1-3 in progress. This APPROVE is bound to 95dfdbd; READY only once the rollup is green on it.

@noahgift noahgift left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seat 2, aprender-seat-o9 @95dfdbd8ad -- APPROVE (no blocking finding)

Delta reviewed: c0e73cc..95dfdbd, one commit. It changes docs/audits/surface_audit.csv only (2+/2-). git diff c0e73ccbed 95dfdbd8ad -- crates is empty, so my code review at c0e73cc carries over unchanged.

The renumber is consistent. At the merge base, ollama.rs has vec![ at 651, "/api/pull" at 659 and "/api/delete" at 664, and the ledger anchors are 650 and 654. At 95dfdbd those lines are 727, 735 and 740. The new anchors 726 and 730 keep the same offsets (+76, matching the PR's inserted lines). No ledger row is added, dropped or re-graded.

@noahgift
noahgift added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 25958dc Oct 16, 2026
11 checks passed
@noahgift
noahgift deleted the la-711a/5357-shim-done-reason branch October 16, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant