Repository navigation
docs: what a no-thinking GLM-5.3 prompt is on each engine (#1911) - #2297
Conversation
|
Thanks @devopsfortunato. This matches what we measured on #1911, and what |
dd28d33 to
3648d03
Compare
|
Thanks @devopsfortunato. The push still says what we measured on #1911 and what The quality question you left open on #1911 (whether the kept head changes greedy no-thinking output) stays there: we'll keep #1911 open for it and note that the docs half is done. |
|
Done — the body now names head |
The Reasoning section said a no-thinking GLM-5.3 prompt has no reasoning-effort line. Native renders the checkpoint template unchanged and appends </think> once to close its dangling think block, so the template's Reasoning Effort head stays and the prompt is the render plus one token. The released 0.6 Python engine's thinking_off instead strips that head and closes the block inline, which is why its no-thinking prompts report 6-7 fewer prompt_tokens than native's on the same request. State both engines' behavior and what follows for prompt counts, cache keys and kept states.
3648d03 to
e0a230b
Compare
|
Thank you @devopsfortunato. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
What this changes
docs/api.md's Reasoning section said a no-thinking GLM-5.3 prompt has "no reasoning-effort line". Native renders the checkpoint's template unchanged and appends</think>once to close its dangling think block (zig/src/server/prompt.zig), so the template'sReasoning Efforthead stays and the prompt is the render plus one token. The released Python 0.6 engine instead rewrites GLM's render (thinking_offstrips the head and closes the block inline), which is what its smallerusage.prompt_tokensreports. The section now states both engines' behavior and what follows: a prompt count, a cache key or a kept state from one engine does not match the other's forenable_thinking: false. The CHANGELOG carries one bullet.--no-thinkingprompt 6-7 tokens shorter than the checkpoint template; native renders it verbatim #1911 — the issue asked for a docs line if the engines' prompts differ deliberately; @ashhart's measurement there (5 Nov) settled native's shape (the render unchanged plus one appended</think>, id 154842, the 22-token vector pinned in the host suite) and confirmed the Python lane's 6-7-token-shorter prompts. This PR records that in the docs; the issue's quality question (whether the kept head degrades greedy no-thinking output) stays open there, and nothing in this change touches code.8993a3cca.atc overlap --json: run after the final rebase on8993a3cca. 384 entries;completeisfalsebecause ATC reportspullFilesLimit, so 374 entries areunknownfor lack of those pull requests' file lists. Every open pull request overlaps onlyCHANGELOG.md; five reportconflicts(glm cuda: TF_GLM_GRAPHS=1 replays verify windows as CUDA graphs (#1490) #2279, contract: an XPU device runner (gates 1 to 4 on a B70, gate 5 live, planted controls) #2269, server: an explicit thinking off that a template can't honour is refused #2170, CUDA: #1764's BF16 prompt tile for Qwen ModelOpt, Nemotron and Dory, plus two exact Qwen prompt kernels (#1762) #2008, qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011) and each is that one file — every one prepends a bullet under## Unreleased, resolved at landing by keeping all bullets.docs/api.mdoverlaps with no open pull request; one local worktree holding this pull request's previous head shares both files and conflicts onCHANGELOG.mdalone.Checks on this head
3648d035b(rebased on8993a3cca; the approved head wasdd28d3360on2e81d5df1). The rebase changed no content:docs/api.mdis byte-identical to that approved head, and the only conflict was theCHANGELOG.mdtop bullet, resolved by putting this change's bullet at the top of## Unreleased.python3 tools/zig/lean_check.py: base8993a3cca425 problems, this head 425 problems;lean_new.pyreports425 problems, 425 on the base, 0 new. The nonzero count is inherited from main.zig fmt --check build.zig zigexit 0 andgit diff --checkexit 0.zig15m03s,zig-windows22m52s,zig-macos-dev-tools9m19s,release-paths1m17s,fmt13s,benchmark-clients13s and 22s.zig build testand the other suites locally — the diff is two Markdown files (docs/api.md,CHANGELOG.md); the build graph is unchanged, and CI builds and runs the unchanged tree on this head.Receipt
n/a — documentation only. The change cannot reach inference: no source, build-graph or configuration file is touched, so prompts, tokens and caches are byte for byte as main's.
Checklist