[jschmied]

Closes #2043.

On an integrated GPU (CU_DEVICE_ATTRIBUTE_INTEGRATED), Qwen3.5's kept prompt states now live in a cuMemAlloc buffer and are saved and restored with device-to-device copies. Before, every save pinned a fresh host buffer. On a GB10 a 32k save pinned 655 MB with cuMemHostAlloc (167–190 ms) ahead of an 11 ms copy, and dropping the state unpinned it again (about 75 ms). The CPU and GPU share that memory, so nothing needs pinning.

A discrete card keeps pinned host copies. So does --learn-conversations, because its disk tier reads the bytes on the CPU. The store still counts the states against the host budget, which on a GB10 is the same physical memory.

Micro (655 MB, 10 repeats, beside a 60 GB held allocation)

	before	after
allocate	cuMemHostAlloc 167–190 ms	cuMemAlloc 3.8–4.0 ms (14.6 ms for the first, beside the held 60 GB)
copy	GPU to host 12.2 ms	GPU to GPU 5.5–5.7 ms
free	cuMemFreeHost 72–78 ms	cuMemFree 0.23–1.4 ms

Serving (GB10, SM clock locked at 2300 MHz, page cache dropped before every server start)

speed2.py, 8 rounds × 3 repeats, ABBA, with two runs of main as the A/A control (A/A = main against main, head = this PR against main). Negative is faster. Texts and first-row logits are equal in every arm, round and repeat.

cell	35B-A3B A/A	35B-A3B head	27B A/A	27B head
8k prompt	−0.07 %	−2.11 % (−31 ms)	−0.00 %	−1.63 % (−98 ms)
32k prompt	+0.06 %	−1.99 % (−131 ms)	+0.01 %	−1.67 % (−429 ms)
warm turn	+0.25 %	−0.03 %	−0.06 %	−0.08 %

Head is faster in 8 of 8 rounds in both prompt cells on both models. The 2k prompt, decode and 4-stream cells are within the A/A on both models. The warm turn doesn't move: restoring its 141 MB state took 1–2 ms either way.

Tests

A new host test with a counting driver checks two things: a device save allocates no pinned memory, and dropping the state frees its device buffer. A planted failure confirmed the test catches it. zig build test, CI and diff-check pass, and lean reports 0 new problems.

Questions

1. On an integrated GPU, should Qwen's kept states count against the device budget instead of the host's, like Nemotron's host_snaps = false? That would shrink the KV room planned at startup, so I left it on the host.
2. The startup line still says "prompt cache in host memory" on a GB10. Should it name device memory there?
3. The paged path (opt-in on main) still copies once per page. Should the gather into device memory go with qwen CUDA: streams on pages by default (256 positions, the engine's own too), with the paged prompt and decode at the window path's speed #2011, or into this PR?

Overlaps: cuda_reuse.zig is also touched by #1335 and #1879, which ATC reports as conflicting, and by #2020, which is clean. I'll rebase whichever order you prefer.

[ashhart]

Thanks @jschmied, this is part 1 as we hoped: no pinning on a GB10, replies and first-row logits equal, and 8k and 32k prompts 1.6 to 2.1% sooner in 8 of 8 rounds against a clean A/A. Your questions: keep counting the states on the host budget, since on a GB10 it's the same memory and moving it would shrink everyone's KV room; yes, please name device memory in the startup line on an integrated GPU, in this PR; and leave the paged path's gather for the paged follow-up. With the startup line and a CHANGELOG line, we'll repeat your cold 32k cell on our GB10 and land it.

[jschmied]

Thanks @ashhart. Both are in 2f1fbc5, a new commit on top. On an integrated GPU the startup line now names device memory, and the budget stays on the host. Here's the 35B on our GB10:

prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in device memory, budgeted as host memory; own glue kernels

With --learn-conversations the states stay on the host, so the line keeps its host wording there, and every other family's line is unchanged. A kv_pool test covers both cases. The CHANGELOG line is under Unreleased. zig build test, the no-GPU CI set and the lean check all pass, with 0 new lean problems.

[ashhart]

Thanks @jschmied, that's everything. We'll repeat your cold 32k cell on our GB10 and land it.

[ashhart]

@jschmied, our GB10 run is in and it's green: the cold 32K prompt, your head against its base, alternating builds. Tokens are identical in all 4 rows. Prompt speed is 1,278.9 -> 1,282.1 tok/s cold (+0.3%) and 1,262.4 -> 1,266.9 repeated (+0.4%), and decode is level. That's a smaller gain than your 1.6-2.1%, on our box, with no loss. Queuing it to land. Thanks.