[laidick]

Part of #2396: the shared store (TF_KV_SHARE) is refused with this switch, not covered. Stacked on #2176: the first three commits are #2176 cherry-picked onto main (c70d67bb2). The last two commits are new here: 0a14c8813 (the change) and c2a2479f3 (the review changes).

What it does

TF_GLM_REQUEST_STARTS=1 on rank 0 (opt-in, off by default, like TF_QWEN_REQUEST_STARTS) asks the server for message starts on GLM's 2,048-row grid. A chat's next turn then resumes where its previous turn's history ends, instead of at the last 2,048-row point.

• native/cuda_snaps.zig: requestStep(F) reads each family's own switch (F.request_starts_switch). Qwen names TF_QWEN_REQUEST_STARTS, GLM names TF_GLM_REQUEST_STARTS. One family's switch does not opt the other in.
• GLM cuts (cuda_prompt_cuts.zig): a pass with cuts chunks to min(len, start + 2048, next start) (nextCut). A kept state resumes only where a cold pass would cut (coldCut). set stores a sequence's starts, and stepRows sizes a step, refusing a solo pass when there are cuts. With no cuts this is today's grid exactly, so the switch off changes nothing. cuda_engine.zig calls these and is 599 lines.
• Ranks (cuda_follow.zig): a new Kind.cuts step sends the pass's starts to the other ranks before its first prompt step. That covers both the whole-pass and the stepped follower paths.
• Lanes (cuda_lanes.zig): resumeFrom uses coldCut instead of % prefill_rows. A stream with starts takes no batched pass (canBatch) and no solo pass (cuda_prompt_solo.zig, and the engine refuses one).
• Startup: TF_GLM_REQUEST_STARTS=1 is refused by name before any rank opens, in two cases. With TF_KV_SHARE above 0 it gives GlmRequestStartsWithSharedKv, because there is no shared store. With --learn-conversations it gives GlmRequestStartsWithLearnConversations, until the disk image takes off-grid rows.
• Build: the copies module in glm_family_host.zig gets its core import. Without it, test-glm-snapshots-host does not compile once a GLM requestStep is reached. On main it already fails to compile on macOS for the same reason.

Receipts: 2× GB10 at TP=2

Setup:

• Checkpoint: EXL3 TR3 Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw@9eaebb7c. Drafter: DFlash2 incoai/GLM-5.3-Flash-DFlash2@bf582e4e, unless the run is marked MTP.
• Serve flags: --context 131072 --parallel 2, RoCE.
• Builds: main = c70d67bb2, branch = 0279d373c for the receipts below (before the review). The two-chat and picture runs use c2a2479f3. All builds use the same fatbins (nvcc 13.0.2, sm_121), with host binaries cross-built.
• Requests: greedy, thinking off, max_tokens 120, one request at a time.

Chats:

• short: 6 turns, under 256 tokens.
• mid: 8 turns, 818 to 1,436 tokens.
• long: 4 turns of about 58.7K tokens.
• cold: one 37,526-token prompt of 48 messages.

1. Same tokens as a fresh run; cached_tokens = "resumed at".

• Each of the 19 turns has one reply sha256 across every run that served it. There are 12 runs: main, switch off, switch on, and switch on with --prompt-cache-gib 0, with DFlash2 and also with MTP alone (long and short chats). The mid chat ran on its own in four runs: main, off, on, and on with cache 0.
• In every run, each turn's usage.prompt_tokens_details.cached_tokens equals rank 0's prompt cache: … resumed at N.

2. Both ranks cut at the same rows. The TF_GLM_PROF=1 run logs glm prompt chunk rank=… start=… rows=… on both ranks. For the cold prompt, rank 0 and rank 1 log the same 26 (start, rows) pairs: (0,783) (783,1546) (2329,1580) … (36707,816) (37523,3). With the switch off, both log the grid (0,2048) … (36864,662). A resume sends the same from to every rank in the chunk step; a rank that does not cut there refuses with BadResume.

3. The re-prefill is gone. "Resumed at" values and time to first token (TTFT):

chat, turn	prompt	main resumed at	switch on resumed at	main TTFT s	switch on TTFT s
mid 2	937	0	805	0.93	0.34
mid 4	1,162	0	805	1.32	0.63
mid 5	1,257	0	1,159	1.39	0.29
mid 8	1,436	0	1,415	1.54	0.16
long 2	58,770	57,344	58,735	1.58	0.25
long 3	58,893	57,344	58,735	1.71	0.45
long 4	58,932	57,344	58,735	1.75	0.48

Notes:

• The "short" chat (prompts under 256 tokens) resumes at 0 in both builds. That is the server's min_chunk of 256: no start is planned there.
• Turns that resume at the same row again (mid 3 to 4, long 3 to 4) have a newer assistant start within 256 tokens of the last cut. The server merges it, as for Qwen.

4. Switch off = main. The off runs (branch binary, no switch) match main in every reply hash, cached_tokens and "resumed at", across short, mid, cold and long.

5. Cost of off-grid cuts. Cold 48-message prompt, TF_GLM_PROF=1, rank 0. Rank 1 is within 0.01 s.

	chunks	sum of chunk walls	ms per row (full chunks, median)	KDA	indexer score+select
switch off (2,048 grid)	19	42.08 s	1.113	2.25 s	0.46 s
switch on (cut at every message)	25	41.16 s	1.084	2.20 s	0.47 s

• No kernel needs 2,048 alignment. The off-grid chunks cost nothing measurable in KDA or the indexer.
• Shorter chunks move time from MLA (8.03 → 5.05 s) to experts and FFN (+1.4 s each). The net is 2% faster in this one run.
• Unprofiled cold TTFT: 41.3 s (on) vs 42.0 s (off). Run-to-run spread between server starts was larger (36 to 42 s), so read this as no regression, not a gain.

6. Two chats at once, and a picture (review). Branch c2a2479f3, switch on, --parallel 2 --vision, with the 1104 recipe's vision template (--chat-template; TR3's own template writes no image placeholder). Chats:

• A and B: six turns each, 799 to 1,266 tokens, each with its own system notes.
• C: system notes, then a 224×224 PNG with a question, a text follow-up, then a second PNG.

Three runs:

• alone: A, then B, then C, one request at a time.
• interleaved: A's and B's turn k sent at the same moment (two streams live), then C.
• fresh: alone with --prompt-cache-gib 0.

Results:

• All 15 turns have one reply sha256 across the three runs.
• In alone and interleaved, each turn's cached_tokens equals rank 0's "resumed at" for that prompt length, and the two runs resume at the same rows.
• The picture replies are right (4 bands red/green/blue/yellow; then 2 bands black/white).

turn	prompt	resumed at (alone = interleaved)	TTFT alone	TTFT interleaved	TTFT fresh
A2	917	786	0.34	0.34	1.14
A4	1,128	786	0.62	1.34	1.41
A5	1,167	1,125	0.21	0.55	1.53
B2	926	797	0.34	0.78	1.15
B5	1,210	1,152	0.25	0.25	1.58
B6	1,266	1,152	0.33	0.72	1.66
C2 (text after a picture)	939	801	0.39	0.38	1.20
C3 (second picture)	1,041	801	0.49	0.49	1.30

• Interleaved TTFT is higher on the turns where the other stream was in its prompt pass at the same time. The resume rows and tokens are unchanged.
• C resumes at 801, the start of the user message that carries the first picture. C1's assistant start is 87 tokens past that cut, inside the server's 256-token minimum chunk, so it is merged and the picture is prefilled again on C2 and C3. A picture in a longer conversation resumes past it like any other row.
• --learn-conversations with the switch on: at main the server already refuses --learn-conversations for GLM before the family opens ("keeps conversation states on disk for Nemotron, the Qwen hybrids and Flash Next only on CUDA"), so both ranks stop there on the pair. The new GLM refusal sits in open() for when GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411 lets GLM take the flag. A host test checks it through open() and explain.

Host tests

• zig build test-glm-request-starts-host test-request-starts-host test-qwen35-request-starts-host test-glm-snapshots-host test-glm-admission-host test-glm-logprobs-host pass (55/55), plus compile-glm-native-open, compile-glm-prompt-gate-host and compile-glm-fp8-gate-host.
• Unfiltered module tests (macOS, through a throwaway build step): the GLM CUDA module (addTest(.{ .root_module = glm })) gives 197 pass, 1 skip (main: 186 pass, 1 skip); native gives 155 pass, 2 skip; qwen35 passes.
• 49 of the 52 test-glm*/exl3*/lane*/request* host steps pass on macOS. The other 3 (test-glm-sampling, test-lanes-see, test-exl3-prompt-host) also fail on main here.
• python3 tools/zig/lean_check.py: 155 problems on the branch, the same 155 as main c70d67bb2, so none are added. Every changed file is at or under 600 lines (cuda_engine.zig 599, cuda_lanes.zig 600).
• zig fmt --check is clean.

Shared with #2215 and #2358

• core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 (one walk for kept states): no file overlap. A state kept at an off-grid row ends mid-page; its walk already splits kept pages at each page's end, so it copies such a state unchanged.
• glm cuda: the startup plan counts TF_KV_SHARE's kept states, a slot each beside the pool's pages #2358 (TF_KV_SHARE startup sizing): both touch glm/cuda_native.zig open() (this PR adds requestStartsAlone() as its first line), the GLM recipe's request list, and the changelog. This switch refuses TF_KV_SHARE, so the two never run together; whichever lands second rebases a few lines.
• qwen3_5 cuda: a chat's next turn can resume at the previous turn's history (opt-in) #2176: this PR builds on its server plumbing and its per-family opt-in. If qwen3_5 cuda: a chat's next turn can resume at the previous turn's history (opt-in) #2176 changes, the last commit rebases onto it.

Receipts and scripts (driver, rank launcher, window runner) are from laidick's two Sparks. I can attach the logs if useful.

[ashhart]

Thanks @laidick, this is a careful PR, and it meets every point we asked for on #2396:

• the cold pass cuts at the same starts, so resumed turns equal fresh ones, with DFlash2 and with MTP;
• both ranks get the cuts in their own step and log the same rows;
• TF_KV_SHARE is refused by name before any rank opens;
• with the switch off, every reply, cached_tokens and resume row matches main.

The off-grid cost numbers are useful too. A few things before it can land:

1. The 600-line limit: families/glm/cuda_engine.zig goes from 595 to 611 lines. Could setCuts and the cut-aware stepping move into cuda_prompt_cuts.zig, or somewhere else that brings it back under 600?
2. Two chats at once: every run sends one request at a time at --parallel 2. Since cuts live on each sequence and are sent at each bind, please post one run with two of your chats interleaved, switch on, with reply hashes, cached_tokens and "resumed at" per turn, against the same chats run alone. A request with an image part would be welcome too, if you serve vision.
3. The body: "Part of GLM CUDA: kept states only on the 2,048-row prompt grid, so every resumed turn re-prefills up to 2,047 rows (GLM counterpart of #1334) #2396" rather than "Fixes", since the shared store (TF_KV_SHARE) is refused rather than covered.
4. CHANGELOG: please end the line "(glm cuda: a chat's next turn can resume at the previous turn's history (opt-in, #2396) #2404, part of GLM CUDA: kept states only on the 2,048-row prompt grid, so every resumed turn re-prefills up to 2,047 rows (GLM counterpart of #1334) #2396). Thanks to @laidick."
5. With --learn-conversations: GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411 (disk-kept GLM states) keeps rows on a 4-row pool, and a message-start resume can land off it. Until the disk image takes such rows, please refuse TF_GLM_REQUEST_STARTS=1 together with --learn-conversations, by name, at startup.

On the order: this sits on #2176, so it rebases once #2176 lands. Your last commit makes requestStep read each family's own switch, which we'd like to keep, and we'll say on #2176 how we take it. zig/build/glm_family_host.zig's core import is also in #2403, so whichever lands second drops that line.

[laidick]

Thank you for the careful review, @ashhart. I've made all five changes. They are in c2a2479f3, which sits on top of 0a14c8813 and the #2176 commits. The PR body is updated.

1. cuda_engine.zig under 600 lines. I moved the cut stepping into cuda_prompt_cuts.zig, as you suggested. set stores a sequence's starts and refuses bad ones with BadCuts. stepRows sizes a prompt step: it refuses a step off the cut grid with BadChunk, and it refuses a solo step when there are cuts. Engine.setCuts and the nextCut/coldCut re-exports are gone, so cuda_follow.zig and cuda_lanes.zig import the cuts file directly. Line counts are now 599 for cuda_engine.zig and 600 for cuda_lanes.zig. lean_check.py reports 155 problems on the branch, the same as main c70d67bb2.
2. Two chats at once, and a picture. These ran on the pair at --parallel 2 --vision with the switch on (receipt 6 in the body). Chats A and B had six turns each. Picture chat C had a PNG, a text follow-up, then a second PNG. Each chat ran alone, then with A and B interleaved (both streams live at once), then fresh with the cache off:
   • All 15 turns gave the same reply hash across the three runs.
   • Interleaved and alone resumed at the same rows, and cached_tokens equals rank 0's "resumed at" for every turn.
   • Interleaving only added TTFT on the turns where the other stream was in its prompt pass.
   • The picture answers are correct. C resumes at the start of the message that carries the picture: its assistant start falls within the 256-token minimum chunk, so the picture is prefilled again.
   • Note: TR3's own chat template writes no image placeholder, so I served it with the 1104 recipe's vision template.
3. "Part of GLM CUDA: kept states only on the 2,048-row prompt grid, so every resumed turn re-prefills up to 2,047 rows (GLM counterpart of #1334) #2396". Changed. The body says the shared store is refused, not covered.
4. CHANGELOG. The line now ends "(glm cuda: a chat's next turn can resume at the previous turn's history (opt-in, #2396) #2404, part of GLM CUDA: kept states only on the 2,048-row prompt grid, so every resumed turn re-prefills up to 2,047 rows (GLM counterpart of #1334) #2396). Thanks to @laidick." The recipe doc names both refusals.
5. --learn-conversations refused by name. GLM's Options now carries learn_conversations, forwarded by cuda_options.set. requestStartsAlone checks it first in open(), before any rank opens, the same way as the TF_KV_SHARE refusal. The message reads: "TF_GLM_REQUEST_STARTS=1 cannot be used with --learn-conversations: unset request starts or serve without it". A host test covers it through open() and explain. On the pair, today's main already stops --learn-conversations for GLM in the server, before the family opens. So this refusal takes over once GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411 lets GLM take the flag.

On the other points:

• I'll rebase once qwen3_5 cuda: a chat's next turn can resume at the previous turn's history (opt-in) #2176 lands.
• The per-family requestStep switch stays as it is.
• The core import in glm_family_host.zig is also in glm cuda: test-glm-snapshots-host's copies module imports core, which cuda_snaps.zig reads for TF_GLM_PROMPT_TAIL #2403. Whichever PR lands second drops that line, and I'll do it here if this one is second.

Host checks are all green:

• The targeted steps pass (55/55).
• The GLM module gives 197 pass, 1 skip, and native gives 155 pass, 2 skip.
• zig fmt is clean.

The logs and the driver for receipt 6 are on my side if you'd like them attached.

[ashhart]

Thanks @laidick, all five are in:

• cuda_engine.zig is 599 with the stepping in cuda_prompt_cuts.zig;
• the interleaved and picture runs give one reply hash a turn, with matching resume rows and cached_tokens;
• the body says "Part of GLM CUDA: kept states only on the 2,048-row prompt grid, so every resumed turn re-prefills up to 2,047 rows (GLM counterpart of #1334) #2396";
• the CHANGELOG line has its ending;
• --learn-conversations is refused by name before any rank opens.

Thanks also for the note on TR3's template. One thing for your rebase: #1423 is queued ahead of this and takes cuda_engine.zig to 600 on its own. If it lands first, this PR would bring the file to about 604, so a few more lines will need to move out then, for example one more helper into cuda_prompt_cuts.zig. This still rebases once #2176 lands.

[ashhart]

@laidick, a heads-up for your rebase: #1423 landed this morning, and families/glm/cuda_engine.zig is now at exactly 600 lines on main. Whatever this PR adds there needs to move out as it rebases, for example into cuda_prompt_cuts.zig beside the stepping. #2176, which this follows, is in our landing queue.