GLM-5.3-Flash on CUDA cannot keep conversation states on disk: an 18.7k-token conversation evicted by one 60k filler recomputes its next turn in 15.6 s (0.49 s resumed), and 15.4 s after a restart #1312

Version: native main 674fb9e (the code below is unchanged on 67cc8fb). Model: GLM-5.3-Flash
(glm5_next, ModelOpt NVFP4). GPU/OS: two NVIDIA DGX Spark (GB10), TP=2, --parallel 8, 131,072-token context,
Ubuntu 24.04. Error: --learn-conversations keeps conversation states on disk for Nemotron only on CUDA: serve this model without it (zig/src/native/cuda_refusal.zig:47).

What happens

GLM on CUDA keeps prompt states in RAM only: each is a copy of the committed prefix in leased snapshot pages on every
rank (families/glm/cuda_snapshots.zig; sized by position since #1163 and #1211). Once a state is evicted, or the
server restarts, the next turn prefills the whole conversation again. The rows store that keeps states on disk
(core/rows_store.zig, rows.Image) serves Nemotron on CUDA and four families on Metal; GLM on CUDA is refused at
startup.

Measured on the pair, main 674fb9e (same request each time, cached_tokens from the reply):

For scale, not a measurement of this proposal: my spill tier on a downstream engine restored such states from disk in
0.85-0.89 s, after eviction and after a restart.

Earlier commitments

On #427 you wrote that keeping GLM-5.3-Flash's prompt states on local disk across turns and restarts is the right fix,
with its on-disk rules (every rank writes its own part; a state is read back only when the new request extends the same
tokens) going into that design. On #373: "kept prompt states go to disk through the shared --learn store; the native
GLM CUDA port will use that store." This issue proposes that port.

Proposed approach

Implement rows.Image for GLM on CUDA, as Nemotron's CUDA family does (native/cuda_disk.zig):

• The image of a state at at (a chunk start): 48 bytes of words (format, at, the head's rows, both ring bases,
  the KDA copy bits), the KDA prefix (conv, recurrent states, a window's saved rows), then per DSA layer the latent rows,
  fp8 scales and whole pools of pooled keys as rows by position (shared blocks), and as the state's own: the two
  index rings (base-relative), the pool in progress, the MTP head's caches at min(capacity, at + 8) rows (zeros past
  its rows) and the tail. The image is the snapshot plan's bytes in its order, so a read restores exactly what a RAM
  restore does. A host-tested draft of this layout (sizes against Plan, rows by position, a lease round trip on a
  host-memory driver) is ready on a branch; nothing is wired yet.
• What crosses ranks. The latent rows, index keys and pooled keys are the same bytes on every rank (full-width
  projections of a hidden row every rank sums in rank order); only the KDA prefix (163.1 MB at TP=2) is each rank's
  own. So the shared rows are written once.
• Ranks, through the existing follow link from rank 0's engine thread: blank (every rank leases a state under
  rank 0's id), put and take (bytes between rank 0's host and a rank's lease, in pieces through the link's buffer,
  no new device allocation) and seal (the words). A failed read drops the id on every rank through the existing drop
  path, and the turn prefills.
• Outside the family: the adapter and identity in native/ beside Nemotron's, the two F == nemotron.native
  checks in native/cuda.zig and the refusal turned into a family declaration, docs/api.md and the changelog.

Open questions

1. Where does rank 1's part live? (a) Rank 0's disk holds every rank's image: no core change, one trailer and hash
   cover every rank's bytes, so a read is all-or-nothing before any rank resumes; rank 1's prefix (163 MB a state)
   crosses the link on each write and read. (b) Each rank writes its own part, your CUDA spill tier: GLM-5.3-Flash kept prompt states on local disk (--spill-gib) #427 rule: the store tells the other
   ranks a state's name on write, remove and read (an Image addition like prompt_imprint's peer_*), every rank
   answers before a resume, and each rank's disk has its own cap and floor. I propose (a) first, (b) if you prefer it.
2. The follow protocol. Four step kinds in cuda_follow.zig, which its header calls TP over processes "until Zig CUDA family for full GLM-5.3 (glm_moe_dsa) at TP4: plan and questions #552".
   Add them there now, or wait for Zig CUDA family for full GLM-5.3 (glm_moe_dsa) at TP4: plan and questions #552?
3. TP=1 first? It needs no protocol (Metal shipped one Mac first), but the NVFP4 checkpoint does not fit one GB10,
   so its receipt would need the EXL3 checkpoint. Or land TP=2 directly with the pair's receipt?
4. The identity: checkpoint, GLM's kernel images, device, driver, TP world, kv8, head and drafter on or off,
   min(capacity, 4096) (the rings' rows), and a probe's state hash (a 2,048 + 40 token prompt through the lane
   backend). Anything else you want in it, such as the agreed rank settings text?
5. Both KDA copies or the current one? The RAM snapshot copies both recurrent copies; the image could keep only the
   current (71 MB a rank less a state) at the cost of a layout that differs from the RAM copy.

A pull request follows once 1 and 2 are settled, with host tests and a pair receipt: evict and restore against cold
(token_sha), a restart restore, both ranks' bytes, and a damaged file refused.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)