Skip to content

Server: one engine flag for models whose replies carry no reasoning block - #1933

Merged
ashhart merged 3 commits into
ashhart:mainfrom
laudney:pr-ocr-reasoning-info
Nov 2, 2026
Merged

ashhart merged 3 commits into
ashhart:mainfrom
laudney:pr-ocr-reasoning-info

Conversation

@laudney

@laudney laudney commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Part of #1927.

Some checkpoints never emit a reasoning block, but their chat templates still read enable_thinking. The OCR models in #1927 are like this. Today, the server splits their replies as if they could think: with thinking on, which is the default, the whole OCR text lands in reasoning_content, and content stays empty. This PR adds one engine flag for such models. A family that sets it gets the template's own prompt rendering, and its reply is always answer content.

Before (main) After (this PR)
Reply of a no-reasoning model with thinking on (the default) all text in reasoning_content, content empty all text in content
Prompt rendering under enable_thinking follows the template unchanged: follows the template
Harmony channel markers in such a reply parsed into channels kept as answer text, streamed and whole
Existing families (Info.reasoning defaults to true) — unchanged on both reply paths
How a native HIP family opts out — pub const reasoning = false;
Places that combine thinking with this capability — one, chat.prepare (Prepared.reasoning)

Changes

  • engine_api.Info.reasoning: bool = true.
  • chat.prepare keeps thinking for prompt rendering. It computes reasoning = thinking and srv.info.reasoning once and stores it on Prepared. Each reply-side use reads it:
    • feature checks
    • the think budget and reserve
    • logprobs admission
    • the think-end marker
    • the reasoning split and its streamed counterpart
    • the length warning for an empty reply
  • With reasoning = false, the Harmony fallbacks (parseHarmony in finish and streamingVisible in the stream path) do not run.
  • native/hip.zig: a family's pub const reasoning = false; sets the info flag. A family without that declaration keeps true.

Gates

  • hf_eos_http_test.zig, through the real HfText and the OpenAI handler, both streamed and whole:
    • reasoning = false under default, explicit-on and explicit-off thinking: the prompt follows the template, the reply is all content, and reasoning_tokens is 0.
    • The default true under the same three settings: the split works exactly as before.
    • A Harmony reply (<|channel|>analysis<|message|>plan<|channel|>final<|message|>OK) with reasoning = false stays complete answer content. This test fails if the stream-path guard is removed.
  • native/hip.zig: the registry test shows that a declared family gets false and every other family keeps true.
  • zig build test: 465/465 steps. zig build test -Dhip=true: 492/492 steps. test-cuda-host and the native-hip build pass.
  • Lean checker: zero new problems over the 425 baseline. chat.zig stays at 600 lines.

@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @laudney, reviewed: this lands. Info.reasoning defaults to true, so every path on main stays exactly as it was (reasoning equals thinking, and the Harmony parse and the think-close id are unchanged). No registered family sets it false yet, as your test checks, so it only takes effect once the #1927 OCR families declare it. Its red zig and zig-windows checks aren't from this PR: main has failed qwen4_cuda_attention_host.cpp (fn_attn_merge_body) since #1739, and #1935 fixes that. Once #1935 is in, we'll rebase this onto main, let its CI run, and land it. You don't need to push anything.

ashhart added a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
@ashhart
ashhart force-pushed the pr-ocr-reasoning-info branch from c9aaa6d to 51e7a75 Compare November 2, 2026 14:16
laudney added a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name dots.mocr. The dots refusal test now
asserts that the registered dots family declares no reasoning block,
which keeps its OCR text in message content. Only the dots assertion
is added, so later OCR families do not conflict with it.
laudney added a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
laudney added a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name Nemotron Parse. A new registry test
asserts that the registered Parse family declares no reasoning block,
which keeps its OCR text in message content. Only this family's
assertion is added, so the other OCR families do not conflict with it.
laudney and others added 3 commits November 2, 2026 17:08
Some families, such as OCR models, never emit a reasoning block, yet
share chat templates whose enable_thinking switch the server honours.
Info gains a reasoning flag, default true; when false, the prompt
still renders with the requested enable_thinking, but the reply is
all answer content. chat.prepare computes the reply-side switch once
and keeps it on Prepared.reasoning, which feature checks, the think
budget and reserve, logprobs, the think-end marker, the reasoning
split and the empty-content warning now read; the startup thinking
line follows it too. A native HIP family opts out with a `reasoning`
decl, and HTTP tests cover both settings in both response modes.
The registry test required every registered HIP family to keep
reasoning on, so each OCR family that declares replies without
reasoning would have had to edit the shared test. The two inline
families already prove that the declaration turns it off and that its
absence keeps the default, so the registry loop is dropped.
@ashhart
ashhart force-pushed the pr-ocr-reasoning-info branch from 51e7a75 to 6c38647 Compare November 2, 2026 17:08
@ashhart
ashhart merged commit 933b969 into ashhart:main Nov 2, 2026
1 of 11 checks passed
@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thank you @laudney. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

@laudney
laudney deleted the pr-ocr-reasoning-info branch November 2, 2026 21:03
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 2, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
ashhart pushed a commit that referenced this pull request Nov 3, 2026
The registry test from #1933 checks the hook on two anonymous
namespaces, so it does not name GLM-OCR. The GLM-OCR flag test now
asserts that the registered family declares no reasoning block, which
keeps its OCR text in message content. Only this family's assertion
is added, so the other OCR families do not conflict with it.
laudney added a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name dots.mocr. The dots refusal test now
asserts that the registered dots family declares no reasoning block,
which keeps its OCR text in message content. Only the dots assertion
is added, so later OCR families do not conflict with it.
laudney added a commit to laudney/TensorFold that referenced this pull request Nov 3, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name Nemotron Parse. A new registry test
asserts that the registered Parse family declares no reasoning block,
which keeps its OCR text in message content. Only this family's
assertion is added, so the other OCR families do not conflict with it.
@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @laudney! This is in TensorFold 1.2.2, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.2

ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 4, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name dots.mocr. The dots refusal test now
asserts that the registered dots family declares no reasoning block,
which keeps its OCR text in message content. Only the dots assertion
is added, so later OCR families do not conflict with it.
ashhart pushed a commit to laudney/TensorFold that referenced this pull request Nov 4, 2026
The registry test from ashhart#1933 checks the hook on two anonymous
namespaces, so it does not name dots.mocr. The dots refusal test now
asserts that the registered dots family declares no reasoning block,
which keeps its OCR text in message content. Only the dots assertion
is added, so later OCR families do not conflict with it.
ashhart pushed a commit that referenced this pull request Nov 4, 2026
The registry test from #1933 checks the hook on two anonymous
namespaces, so it does not name dots.mocr. The dots refusal test now
asserts that the registered dots family declares no reasoning block,
which keeps its OCR text in message content. Only the dots assertion
is added, so later OCR families do not conflict with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants