[jschmied]

Part of #840, the "Spill of evicted states" row (#68, @gilby). Split from #1359 at review: this PR only turns on CUDA spill. Stacked on #1359 (Flash Next's disk image) and #1358; only the last commit is new here (82a693e).

• The flag helpers live in a backend-neutral module, native/disk_spill.zig: slots(), refusal(o, where) and flag(). metal_spill.zig delegates to it (refusal(o, " on Macs"), so the Mac wording is unchanged), and CUDA's cuda_disk.open and cuda_refusal.backend call it. No wording is copied any more.
• gilby's flashnext: shared sessions spill evicted states to disk (--spill-gib) #1349 edits metal_spill.zig too: it adds writesBehind() beside these helpers. git merge-tree of this branch with flashnext: shared sessions spill evicted states to disk (--spill-gib) #1349's head (90b2dc2) is clean. If flashnext: shared sessions spill evicted states to disk (--spill-gib) #1349 lands first, writesBehind can move into disk_spill.zig with the others in a follow-up.

Against #1411 (GLM-5.3-Flash): the spill refusal is a line of its own in cuda_refusal.family, ahead of the drafter block, and its test is at the end of the file. Neither touches #1411's refusal edits; with both in, GLM serves --learn-conversations and refuses --spill-gib until it declares kept_disk. disk_spill.zig is new, and #1411 does not touch it.

How Nemotron spill is gated on CUDA:

1. The CUDA backend now lists --spill-gib and --checkpoint-slots in serves, so the server no longer refuses them for CUDA as a whole.
2. cuda_refusal.backend refuses --checkpoint-slots without the disk store (no --spill-gib, no --learn-conversations), with the shared wording.
3. cuda_refusal.family refuses --spill-gib (spill == .evicted) for every family except Nemotron (nemotron.native, whose image is native/cuda_disk.zig's own) and those that declare kept_disk (qwen3_5 and Flash Next). It does so on its own line, ahead of the drafter and --learn-conversations refusals: --spill-gib keeps evicted states on disk for Nemotron, the Qwen hybrids and Flash Next only on CUDA.
4. At open, native/cuda.zig makes a prompt store only for a family with snap_save and a nonzero cache. Then, with --learn-conversations (which --spill-gib sets, together with spill = .evicted) and a learn root, it calls cuda_disk.attach. That goes to open() → rows.attach(..., .spill = o.spill, .slots = disk_spill.slots(o)).

docs/api.md now names the three CUDA families that take both flags, and the Open.spill doc comment no longer says Macs only.

So Nemotron spills exactly when the store exists and the flag reached it. A host test in cuda_refusal.zig covers the families that open with the flags and the ones refused by name.

zig build test passes on this branch, and lean shows 0 new problems against main.

[ashhart]

Thanks for splitting the spill switch out of #1359 and for moving the shared wording into disk_spill.zig. We checked the PR's own commit on its head: zig fmt, the lean check (no new findings), test-cuda-host and the Linux cross-compile pass, and the refusals read right.

Three asks:

1. Please restack on cuda flashnext: --learn-conversations keeps Flash Next's states on disk (#840) #1359's current head (e018f176e), which is on main with cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk (#840) #1358's rebase: there startsBelow already sits in rows_write_sync.queued, which main's rows_write.queued now calls, so the Qwen hybrids' grid states keep their chunk starts. Your commit's docs/api.md text will conflict with main's new spill description; see the next point.
2. With main's writer, CUDA spill stages a copy at eviction on the engine thread, writes it on the I/O thread, saves RAM states at a stop and reads queued requests' states ahead. docs/api.md:519-523 here says nothing is written while idle and the write happens before the request goes on. Please describe main's behaviour for the three CUDA families, including the 1 GiB staging limit, past which a state is not saved.
3. No test checks that CUDA's open hands --spill-gib's mode and --checkpoint-slots to the store (zig/src/native/cuda_disk.zig:184), which is the change itself. A host test with a fake disk module, like the attach test below it, that fails on main would cover it.

Once it is rebased we'll run Nemotron, a Qwen3.5 hybrid and Flash Next on a GB10: evict, restart and read back against fresh prompts, concurrent against solo, and no change without the flags.

[jschmied]

Thanks. Restacked on #1359's e018f176e and force-pushed: aeb3cdca4 (the switch) and bd8ab9b5e (the test).

1. Restack. The switch commit cherry-picks onto e018f176e. The only conflict was docs/api.md, resolved as in point 2. Code is unchanged from 82a693e70.

2. docs/api.md. Main's Mac paragraph stays as main has it. The only change there: --checkpoint-slots "needs --spill-gib or --learn-conversations" no longer says "on a Mac", and the three CUDA families are named next to the four Mac ones. A new paragraph describes main's writer on CUDA for Nemotron, the Qwen3.5 hybrids and Flash Next:

   • At an eviction, the engine thread stages a host copy of the state (waiting for its GPU producers), releases the RAM entry and goes on. The I/O thread writes and fsyncs the copy.
   • Staging holds at most 1 GiB and eight jobs. A state whose new bytes (rows not already on disk) do not fit what is free is not saved, so one past 1 GiB never is. A later request for it prefills.
   • When the engine closes, at SIGTERM/SIGINT or an idle unload, it saves the states still in RAM, newest first, within the disk cap and the 20-second stop budget.
   • While other streams run, queued requests' states are read ahead on an I/O thread (the existing paragraph below).

   I checked each point against the code: rows_write.evicted/stage, rows_io.Worker (max_jobs = 8, staging_bytes = 1 << 30, and the reserve fails when the new bytes exceed what is free), lane_host_rows.flush → rows.write.drain on close, and lane_host_admit.pick → fetch.queue. All three families serve through native/cuda.zig's LaneHost.

3. Test. cuda_disk.zig gains "CUDA disk open: --spill-gib's mode and --checkpoint-slots reach the store". A fake disk module declares attach and opens the store through cuda_disk.open under a fixed identity, with no device. It runs for --spill-gib alone with --checkpoint-slots 3, and with .finish and no slots, and checks store.disk.spill and store.most.

   No step on Linux ran native/cuda.zig's tests unfiltered, so cuda_disk.zig had no runner at all. native/cuda.zig's test block now references cuda_disk, and a filtered step test-cuda-disk-open-host is part of test-cuda-host. With main's open swapped in (Rows.open, no mode or slots), the test fails with expected .evicted, found .off. With the PR, it passes.

Checks: zig fmt --check zig clean; lean 0 new; zig build test -Dnvcc=... RC 0; ATC overlap flags #728, #1411, #1439 and #1446. #1411 shares docs/api.md and the cuda_disk/cuda_refusal files, and #728 and #1439 share zig/build/cuda.zig, where the new test step lands; #1446 shares no file with this branch. No GB10 runs on this head yet.

[ashhart]

Merged, thank you @jschmied, and @gilby for your part. It ships in 1.1.8, with the changelog entry in 141cf65. As with #1359, we rebased your branch onto main to land it as a merge: your two commits as they were, plus our docs commit, which credits you.

Our device run on a GB10, with --checkpoint-slots 1 --prompt-cache-gib 1 --spill-gib 8: conversation A, then B, which sends A to disk, then A's next turn. Each family read A back from disk:

Model	Next turn, cached	First token	Fresh
Nemotron 3.5 Lightning	16,061 of 16,133	0.89 s	2.52 s
Qwen3.8-27B	12,288 of 16,028	6.99 s	16.83 s
Qwen3.8-Flash-Next	15,939 of 16,004	1.97 s	9.70 s

Every reply equals a fresh server's, at --parallel 1 and 2. Without the flags, main and the PR give the same replies on all three. The full test suite, the Linux CUDA cross-compiles and the release path check pass on the merged tree.

[ashhart]

This is in TensorFold 1.1.8, released today. Thank you, @jschmied and @gilby!