[jschmied]

What this changes

A cleanup, no gain: Qwen 3.5, Gemma 4, Nemotron and Flash Next on CUDA move their kept-state copies and spills onto #2215's core/state_walk.zig. Each family's own loop that split live pieces at page ends is gone. No kernel, flag or allocation changes, and the copies are the same copies.

• Issue: Refs Metal: KV pages from the core pool #1293 (step 4).
• Stacked on: core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215, which stays a draft, so this does too. The first three commits are core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215's. Review the five after them: one a family, then the CHANGELOG line. Base: main 5a45539af. This head doesn't apply to main past that: 733e4144b (Qwen pageCopies, a 2D copy a page) and 4f6017ced (Flash Next .rows) landed in the same loops, and need a port and new evidence.
• Overlap: from each open PR's file list. atc overlap was cut short by its limits and marked every pair unknown.
  • flashnext/cuda_snap.zig: EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441-EXL3 Flash Next: KV pages from the core pool (4/8) #1444, EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522, flashnext cuda: prompt cache marks inside a chunk are kept on the way instead of cutting the pass #1866, core: a bounded ladder of intermediate prompt-cache checkpoints #1953.
  • qwen3_5/cuda_reuse.zig: qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011, qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model at 2 and 3 stages (RFC #2005 step 2, stacked on #2241) #2115 (both ours).
  • nemotron/cuda_reuse.zig: Living Weights: slide-forget counting only with --slide-forget, Metal's mask shift for wide pockets (#678) #2026.
  • gemma4/cuda_snap.zig: gemma4 cuda: TF_GEMMA4_CHUNK_SOLO, on a solo-chunk rule shared in lanes (#1554 option C) #2391 (ours).

What each family moves

• Qwen 3.5 (cuda_reuse.zig): Runs is gone. Each Layout piece is a walk part, and on pages span is the piece's length in a page and page the page's bytes. One hook, KeptCopy, has both of main's arms at the same offsets: device-to-device for a state in device memory (Copy.dev, qwen CUDA: an integrated GPU keeps prompt states in device memory, without pinning a copy each save (#2043) #2113's default on an integrated GPU), device-to-host and host-to-device otherwise. Pieces can number in the thousands, so it's one walk call a piece, with no parts array.
• Gemma 4 (cuda_snap.zig): each builds parts from the ring runs and the paged global rows. Kept bytes are a buffer, or offsets the hook hands to kv_prompts.Blob.
• Nemotron (cuda_reuse.zig): rowSpan and deviceRows are gone. Rows are parts on the bound pages, and Mamba parts are whole. It walks one part at a time because rows and Mamba go to two kept buffers. error.PastPages stays as a check before the part's first copy. The staging copy in hostMove is download-only (.into = true), as on main.
• Flash Next (cuda_snap.zig): move builds parts from the whole pieces and the paged attention rows, with one device-to-device hook.

Checks on this head

• Head commit: eda4fd4b7
• Lean checker on main and on this head: 155 and 155.
• Tests, each rc=0: zig build test, test-cuda-host, test-kernel-contract, test-golden, test-server-security, test-pull-security, fuzz-cortheon, test-gemma4-cuda-host, test-glm-expiry, test-learn-recovery, check-release-paths, a plain zig build. zig fmt and git diff --check are clean.
• Host test expectations: unchanged. Every test block main has in the touched files is byte-equal here (25 blocks); core/root.zig's test block only gains _ = state_walk;.
• test-glm-snapshots-host doesn't build on main 5a45539af: module copies lacks core, which native/cuda_snaps.zig:11 needs since fc8bd41ed. With that one import added locally, not in this PR: 21/21 on main, on core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 and on this head.
• Walk against each family's old loop, 20,000 random plans each, same copies in the same order:
  • Qwen 3.5, rerun against today's main with both arms: 125,873,879 copies, hash 30f293bf3b163bb0, across device and host, paged and not, save and restore.
  • Gemma 4: 1,179,384 copies, hash 4cd007311d2a0cf.
  • Flash Next: 4,092,710 copies, hash 771ef7c7710a4c56. On a refused plan (5,059, SnapShape alike) main queues the copies before the refused piece, and the walk refuses before any.
  • Nemotron: 336,324 copies, device moves both ways and host downloads. Its hash includes the host buffer's address, so it varies by run.
  • A planted swap in the device arm turns the Qwen oracle red.
• CI: not run yet.
• Not run, and why:
  • Metal: no Metal file is touched beyond core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215's.
  • The GPU kernel contract: no kernel changes here. Its last run was on this PR's earlier tree, with the same PASS and FAIL names as main.

Receipt

• Environment: GB10 sm_121, clock capped 2300, Zig 0.17.0, page cache dropped before each server start, one fresh server per arm. Main binary md5 afe1d25e, this head's 913d5f80. Two cold builds of one commit differ here, so these md5s name these two builds, not the sources.
• Requests: 3-turn chat, then turn 2's prompt again; greedy, ignore_eos, 48 tokens, return_token_ids, --context 16384. A fresh run replays the same prompts with --prompt-cache-gib 0.

Exactness. In every row, main and this branch match on tokens and cached_tokens on all four turns, both match a fresh run, and every resumed turn has cached_tokens > 0.

Family	Arm	How it's reached	How we know it was	cached_tokens
Qwen 3.5	device keep	default	startup: "prompt cache in device memory"	0, 8192, 8192, 8192
Qwen 3.5	host keep	--prompt-cache-disk	startup: "in host memory"; states written to disk	same
Qwen 3.5	paged, device	TF_QWEN_KV_PAGES=256, _ALL=1, --parallel 2	4,158 copies against 189	same
Qwen 3.5	paged, host	the same with --prompt-cache-disk	4,158 copies against 189	same
Gemma 4	paged	default	pool lane closes in the log	0, 9306, 9383, 9383
Nemotron	window	default	no pool line	0, 9436, 9517, 9517
Nemotron	paged	TF_KV_PAGES=all, --parallel 2	pool lane closes in the log	same
Nemotron	disk tier, window and paged	with --prompt-cache-disk	states written to disk	same
Flash Next	window	default	573 copies	0, 9734, 9816, 9816
Flash Next	paged	TF_FLASHNEXT_KV_PAGES=256, _ALL=1, --parallel 2	4,526 copies against 573	same
Flash Next	disk tier	--prompt-cache-disk	states written to disk	same

The page switches alone don't page the first stream, so at --parallel 1 these arms would sit on the window. That's why they run with _ALL or all, and each one's arm is checked above.

Real copies under nsys, one resumed turn, main against this branch, kind and bytes of each copy in order: identical in all six cells (Qwen 3.5 device, host and both paged; Flash Next window and paged). Same bytes in the paged cells as in their window controls, in many more copies.

A disk-kept state saved by main and reused here. Main writes with --prompt-cache-disk and stops. Main and this branch each read their own copy of the store after a restart:

Family	resume_from	disk_tokens	Turn 3 cached	Turn 2 cached	Log
Qwen 3.5	disk	8,192	8,192	8,192	read 8,192 tokens from disk
Flash Next	disk	9,890	9,890	9,816	read 9,890 and 9,816
Nemotron	disk	9,590	9,590	9,517	read 9,590 and 9,517

cached_tokens and tokens equal main's, the writer's and a fresh run's.

Limit. Nemotron keeps its states on the device on a GB10, with the disk tier on too, so hostMove isn't reached here. Its download arm is in the oracle and the host tests. Its upload isn't covered by either: it uses the same parts and walk, and the hook picks the direction.

Resumed time to first token. A-B-B-A, median of 12 warm turns, and mean cold, in seconds:

Cell	Warm main	Warm branch		Cold main	Cold branch
Qwen 3.5, device	1.3680	1.3679	-0.01 %	7.298	7.298
Qwen 3.5, paged	1.4034	1.4007	-0.19 %	7.398	7.398
Gemma 4	0.3208	0.3209	+0.01 %	11.260	11.252
Nemotron	0.0616	0.0616	+0.01 %	1.274	1.277
Nemotron, paged	0.0628	0.0622	-0.94 %	1.272	1.277
Flash Next	0.1911	0.1907	-0.22 %	4.870	4.869
Flash Next, paged	0.1900	0.1892	-0.42 %	5.348	5.347

• Decode speed: not measured. The change is on kept-state copies, not the decode path.
• Licence of new weights or drafter: n/a.

Checklist

• Comments are one line.
• No personal data, machine names or local paths. No AI attribution lines.
• CHANGELOG line under ## Unreleased, citing (cuda: Qwen 3.5, Gemma 4, Nemotron and Flash Next keep states through core's walk, no gain, a cleanup (#1293, on #2215) #2392, part of Metal: KV pages from the core pool #1293).

[ashhart]

Thanks @jschmied. I read this as a kept-state change on top of #2215. By reading, Flash Next, Gemma 4 and Nemotron copy the same bytes, in the same order, at the same kept offsets as main. A few things before it can follow #2215:

1. A rebase on today's main. Two files no longer apply:
   • glm/snapshot.zig: main reworded the doc comments on copy and spill, and core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 needs the same fix.
   • qwen3_5/cuda_reuse.zig: since qwen CUDA: an integrated GPU keeps prompt states in device memory, without pinning a copy each save (#2043) #2113, an integrated GPU keeps Qwen 3.5 states in device memory with a device-to-device copy, which is the default on a GB10. walkHost only has the host arm, so the hook needs both, at the same offsets.
2. A body saying it's stacked on core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215, and what each family moves.
3. Your evidence per family, on a GB10, main against this branch, greedy with return_token_ids:
   • a resumed turn's tokens and cached_tokens equal to main's and to a fresh run, for Qwen 3.5, Gemma 4, Nemotron and Flash Next;
   • the paged arm of each: Gemma 4 by default, TF_FLASHNEXT_KV_PAGES, TF_KV_PAGES for Nemotron, TF_QWEN_KV_PAGES;
   • both of Qwen 3.5's keep arms: the default device arm, and the host arm;
   • a disk-kept state saved by main and reused by the branch, for Qwen 3.5 and Flash Next;
   • the host tests, with their expectations unchanged.
4. A CHANGELOG line under ## Unreleased, even though nothing a user sees changes, citing "(cuda: Qwen 3.5, Gemma 4, Nemotron and Flash Next keep states through core's walk, no gain, a cleanup (#1293, on #2215) #2392, part of Metal: KV pages from the core pool #1293)".

A small one: main's Nemotron staging copy in hostMove always downloads, while the walk's HostCopy would upload when into is false. Only into = true reaches it today, so a download-only hook or an assert would keep that from changing quietly later.

[jschmied]

Thanks @ashhart. All four are done, on eda4fd4b7 (main 5a45539af, on #2215's 183fca37a), but see the first note below: it conflicts with main again.

1. Rebase. glm/snapshot.zig takes main's new doc comments, in core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 too. qwen3_5/cuda_reuse.zig's hook now has both arms, device-to-device for a state in device memory and host otherwise, at main's offsets.
2. Body. It says it's stacked on core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215, and what each family moves.
3. Evidence. It's in the body, one row an arm, and every row matches main and a fresh run, with cached_tokens > 0 on each resumed turn. The disk-kept state saved by main is reused by this branch for Qwen 3.5 and Flash Next, and Nemotron too. Host test expectations are unchanged: 25 test blocks byte-equal.
4. CHANGELOG. The line is in, citing "(cuda: Qwen 3.5, Gemma 4, Nemotron and Flash Next keep states through core's walk, no gain, a cleanup (#1293, on #2215) #2392, part of Metal: KV pages from the core pool #1293)".

The staging copy in hostMove is now download-only, .into = true whatever the caller's into.

Four things you should know:

• It conflicts again. After I pinned 5a45539af, main landed 733e4144b (Qwen's paged BF16 saves and restores as pageCopies, one 2D copy a page) and 4f6017ced (Flash Next's move gains .rows). Both sit in the loops this PR replaces, so a rebase is a port, and the evidence here doesn't carry over to it. Do you want the 2D copy in core/state_walk.zig (a new kind of part), or Qwen's paged BF16 left outside the walk? core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 still merges cleanly.
• A withdrawn row. The "Qwen 3.5, paged" row in my earlier body wasn't paged. At --parallel 1, TF_QWEN_KV_PAGES leaves the engine's own stream on its window, and its log says so. The same goes for TF_FLASHNEXT_KV_PAGES and TF_KV_PAGES. The paged cells now run with _ALL or all, and each row says how we know the arm was reached. The earlier Qwen timing row also predates qwen CUDA: an integrated GPU keeps prompt states in device memory, without pinning a copy each save (#2043) #2113. The timing table is new.
• Main's test-glm-snapshots-host doesn't build. Module copies in zig/build/glm_family_host.zig lacks core, which native/cuda_snaps.zig:11 needs since fc8bd41ed. With the import added locally it's 21/21 on main, core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215 and this branch. I left it out of both PRs. Do you want it as a separate small PR?
• One gap. Nemotron keeps states on the device on a GB10, even with the disk tier on, so hostMove isn't reached here. Its upload stays covered by neither the oracle nor a served run; the body says so.