[Bizuayeu]

On two GB10s at TP=2 (--context 262144 --parallel 4), TF_KV_SHARE=16's kept states now take slots in the startup
plan, where main allocated them at load outside it, and the startup line counts them. At the default prompt cache 13 of
the 16 fit, and the line and a warning name the --prompt-cache-gib that holds all 16; at that size all 16 fit. Either
way the rest of the cache becomes pages, where main leaves it aside:

main                       ... taking 2052 pages of 512 rows (12.77 GiB) ..., the prompt cache over the pool's pages
head                       ... taking 2071 pages of 512 rows (12.89 GiB) ..., 13 of TF_KV_SHARE=16 kept states' slots (2.3 GiB; --prompt-cache-gib 3 holds all 16) and the prompt cache over the pool's pages
main --prompt-cache-gib 3  ... taking 1954 pages of 512 rows (12.16 GiB) ..., the prompt cache over the pool's pages
head --prompt-cache-gib 3  ... taking 1989 pages of 512 rows (12.38 GiB) ..., 16 kept states' slots (2.8 GiB) and the prompt cache over the pool's pages

Every reply's tokens are main's, cold, resumed and two at once, and with TF_KV_SHARE unset the head's startup line is
main's.

What this changes

With TF_KV_SHARE=n, GLM keeps a prompt's rows as the pool's pages and copies the rest of each kept state into a
snapshot pool that share() allocated at load: n states at the window's last mark, outside the startup plan, while the
plan set the whole prompt cache aside as copied rows that no store uses under share. Past the cache's size, that pool
came out of the reserve.

Under TF_KV_SHARE, GLM's layout now declares that state's bytes as its slot (cuda_pages.shareSlot). The plan keeps
min(TF_KV_SHARE, cache / slot) slots and makes the rest of the cache the pool's pages (kv_pool.Budget.slots_only),
so nothing past the slots is set aside. The family readies the plan's count (share_states, called from
cuda_kv_pool.attach), and the store keeps that many. The startup line names the slots, and when the cache holds
fewer than TF_KV_SHARE, how many fit of how many were asked and the --prompt-cache-gib that holds them all. A value
past the 64 slots a plan holds (kv_pool.most_snaps) is refused by name before any rank opens. The warning under 4
states a stream counts the states the cache keeps rather than the ones asked, says how many, and names 4 a stream
and the --prompt-cache-gib their slots take, rounded as the startup line rounds (kv_pool.holdingGib).
TF_GLM_HOST_CACHE keeps the states in host memory, outside the device plan, as before. With TF_KV_SHARE unset, the
layout declares no slot and the plan is main's. The GLM recipe's line says so.

• Issue: Fixes GLM CUDA: kept states under TF_KV_SHARE sit outside the startup plan #2307 (option 1, the first reading: GLM CUDA: kept states under TF_KV_SHARE sit outside the startup plan #2307 (comment)).
• Stacked on: none; on main 691fb4334, and it merges into main 6433d511a without conflicts.
• atc overlap --json: on this head against main ebb67e67e: 10 clean and 21 unknown, incomplete
  (otherPullRequestsLimit, pullFilesUnavailable, timeLimit). Test-merged here, glm cuda: TF_GLM_GRAPHS=1 replays verify windows as CUDA graphs (#1490) #2279 conflicts with this branch
  alone, in docs/recipes/glm-5.3-flash-cuda.md, where both add a bullet at the same place. glm cuda: windows to 32 rows, priced by rounds timed at load #2257
  (zig/src/families/glm/cuda_native.zig) and lfm3: serve LFM3-1.7B on CUDA: NVFP4 and FP8 weight-only, MLX 4-bit (stacked on #1513, #1492 and #1503) #1508, lfm3 CUDA: serve the block FP8 export weight-only and the MLX 4-bit export as stored (stacked on #1508) #1634, lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503, Muse Spark Open on CUDA: EXL3, DFlash2, two GPUs #1295 and lfm3-vl: chat-template thinking suffix, fitGrid geometry, and deployment recipe #2045 (zig/src/native/cuda.zig) conflict
  in this branch's files as they do with main itself. prompt cache: a state past the whole budget names TF_KV_SHARE where the family on pages could keep it #2333 and this branch merge without conflicts.

Checks on this head

• Head commit: 8a2beb792, on main 691fb4334.
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 252 and 252, 0 new.
• Tests run, with the command and pass, fail and skip counts (Zig 0.17.0, Linux x86_64 under WSL):
  • zig build test: 569/569 steps; 7,346 passed, 0 failed, 61 skipped.
  • zig build test-cuda-host (318/318 steps), test-golden (5/5), test-cache-capacity-host (9/9 tests),
    test-kv-safety-host (26/26 tests), test-kv-merge-host (25/25 tests), test-glm-pages-host (4/4 steps),
    test-glm-startup-host (3/3), test-glm-image-cache-host (4/4), test-gemma4-shared-host (4/4),
    compile-cuda-tests (10/10, native and -Dtarget=aarch64-linux-gnu) and compile-config-server (7/7): pass.
  • zig fmt --check build.zig zig: clean.
  • With kv_pool.plan's slots_only branch taken out, zig build test fails the new test and only it, in both test
    binaries that hold it (7,344 passed, 2 failed, 61 skipped).
• CI: green on 8a2beb792, 7 of 7 (zig, zig-windows, zig-macos-dev-tools, fmt, release-paths and
  benchmark-clients on Ubuntu and Windows).
• Not run, and why: TF_GLM_HOST_CACHE with TF_KV_SHARE on a device (its states stay in host memory, outside the
  device plan, as before); HIP and Metal, whose plans never set slots_only and stay main's.

Receipt

• Environment: main 691fb4334 and this commit, built with Zig 0.17.0 and nvcc 13.3.73 for sm_121 (binaries
  95e71cc28c5e860a and 6666c9b694adbd72); two GB10s (128 GB unified memory each, Ubuntu 24.04.5, kernel
  7.0.0-1019-nvidia, driver 580.178.04) at TP=2 over RoCE; Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f91;
  --context 262144 --parallel 4, on KV pages (the default at TP=2).
• Paths and platforms the change reaches: GLM's CUDA startup plan with TF_KV_SHARE at the default cache, at
  --prompt-cache-gib 3 and at 2, the refusal past 64, and the plan with it unset, which stays main's. I ran all of
  them. Not run: TF_GLM_HOST_CACHE with TF_KV_SHARE, HIP and Metal.
• Exactness: the server's token hash of each 64-token reply is dcfb6c2929b3 for all 72 requests, main and head, with
  and without TF_KV_SHARE, cold, resumed and two at once. The replies are all thinking at 64 tokens, so their text is
  empty and not compared.
• Decode speed before and after: n/a, no compute path changes. One stream decodes its 64 tokens in 1.56 to 1.65 s on
  every arm.
• Prompt speed before and after: n/a for the same reason. One stream's first token comes in 10.22 to 10.29 s cold on
  every arm but m1a, the first launch of the second run (10.69 s), and in 1.05 to 1.08 s resumed from 14,336 tokens. Two
  at once vary between runs on main as on the head (the table below).
• Licence of any new weights or drafter: n/a.

Each arm is a fresh launch: m for main and h for the head; 0 with TF_KV_SHARE unset (m0 came up cold, and m2 is main
again after a warm launch), 1 with TF_KV_SHARE=16, 4 with 16 and --prompt-cache-gib 3, 2 with 16 and
--prompt-cache-gib 2, and 3 with 65. a and b are a second run that alternates main and the head. Every arm but h2 and
h3 then takes a 3,200-line chat sent fresh, again, and twice at once (64 tokens); a and b take it twice. From rank 0's
log:

m0   [tensorfold] CUDA sm_121 ... taking 2052 pages of 512 rows (12.77 GiB) ..., of 16.0 GiB left ...; ..., a 2.2 GiB prompt cache
h0   [tensorfold] CUDA sm_121 ... taking 2052 pages of 512 rows (12.77 GiB) ..., of 16.1 GiB left ...; ..., a 2.4 GiB prompt cache
m2   [tensorfold] CUDA sm_121 ... taking 2052 pages of 512 rows (12.77 GiB) ..., of 16.1 GiB left ...; ..., a 2.4 GiB prompt cache
m1b  [tensorfold] CUDA sm_121 ... taking 2052 pages of 512 rows (12.77 GiB) ..., of 16.2 GiB left ...; ..., the prompt cache over the pool's pages
h1b  [tensorfold] CUDA sm_121 ... taking 2071 pages of 512 rows (12.89 GiB) ..., of 16.2 GiB left ...; ..., 13 of TF_KV_SHARE=16 kept states' slots (2.3 GiB; --prompt-cache-gib 3 holds all 16) and the prompt cache over the pool's pages
h1b  warning: TF_KV_SHARE=16 keeps 13 states in the prompt cache's slots, under 4 a GLM stream at --parallel 4: live streams evict each other's before a resume; 16 take TF_KV_SHARE=16 and --prompt-cache-gib 3
m4a  [tensorfold] CUDA sm_121 ... taking 1954 pages of 512 rows (12.16 GiB) ..., of 16.2 GiB left ...; ..., the prompt cache over the pool's pages
h4a  [tensorfold] CUDA sm_121 ... taking 1989 pages of 512 rows (12.38 GiB) ..., of 16.2 GiB left ...; ..., 16 kept states' slots (2.8 GiB) and the prompt cache over the pool's pages
h2   [tensorfold] CUDA sm_121 ... taking 2062 pages of 512 rows (12.84 GiB) ..., of 16.1 GiB left ...; ..., 11 of TF_KV_SHARE=16 kept states' slots (1.9 GiB; --prompt-cache-gib 3 holds all 16) and the prompt cache over the pool's pages
h2   warning: TF_KV_SHARE=16 keeps 11 states in the prompt cache's slots, under 4 a GLM stream at --parallel 4: live streams evict each other's before a resume; 16 take TF_KV_SHARE=16 and --prompt-cache-gib 3
h3   error: TF_KV_SHARE=65 asks more kept states than the 64 a startup plan holds: set 64 or fewer

What is left after the weights moves by a step between launches (16.0 to 16.2 GiB above), and the cache and the pages
that take the rest move with it; m1b and h1b, and m4a and h4a, share 16.2 GiB. At --prompt-cache-gib 3 the pages hold
3 full windows instead of 4 (enough for 3 at a 262144-token window), on main (m4a) as on the head (h4a): that setting
sets the cache aside first.

Two at once, the pair's later reply in seconds:

	TF_KV_SHARE unset	TF_KV_SHARE=16
main	4.35 (m0), 4.40 (m2)	4.37 (m1); 4.44, 4.44 (m1a); 4.79, 4.50 (m1b); 4.38, 4.56 (m4a)
head	4.40 (h0)	4.63 (h1); 4.36, 4.64 (h1a); 4.40, 4.66 (h1b); 5.15 (h4); 4.37, 4.60 (h4a)

With TF_KV_SHARE=16, main's 7 pairs average 4.50 s and the head's 8 pairs 4.60 s, 4.52 without its slowest. m4a and
h4a hold the same 16 states at the same cache and differ in the pool alone (35 more pages on the head): 4.38 and 4.56
against 4.37 and 4.60.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency (with kv_pool.plan's
  slots_only branch taken out, zig build test fails the new test and only it; they have no dependency to skip on).
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution
  lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off (the plan's
  bytes move between slots and pages; no compute path changes).
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (the
  receipt's 72 replies share one token hash, cold, resumed from 14,336 tokens and two at once; nothing here
  touches tokens).
• No precision traded for speed. If the bits change, the description says which sums change and why (no arithmetic
  changes).
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no
  slower than without it, greedy and sampled (no compute path changes; one stream measures as main, and two at once vary
  between runs on main as on the head, see Receipt).

[ashhart]

Thanks @Bizuayeu. This applies cleanly to today's main. By reading, the slot it counts is the same size main's
share() readies for each state (stateBytes at the window's last mark, with share_pages on so the store's prefix
pages stay out), and with TF_KV_SHARE unset the plan, the admission and the startup line are main's. It touches no
file #2215 does, and #2215 leaves Plan.size and rounded as they are, so neither has to wait on the other.
A few things:

1. A CHANGELOG line under ## Unreleased, ending "(glm cuda: the startup plan counts TF_KV_SHARE's kept states, a slot each beside the pool's pages #2358). Thanks to @Bizuayeu." Please say that with TF_KV_SHARE
   the cache keeps as many states as its slots hold, that a value over 64 now stops the server, and that a cache too
   small for one slot (for example --prompt-cache-gib 0) keeps none, where main kept them outside the plan.
2. When no slot fits, share_states skips readySnapshots, so the followers never get share_pages and the store
   stays closed, but rank 0 still has share_pages = true from finishRank. That relaxes rank 0's idle line
   (cuda_pages.zig:245). Could share_pages be set only when there are slots (or under TF_GLM_HOST_CACHE)?
3. Under TF_GLM_HOST_CACHE, the attach error prints the plan's snaps (0) while share_states readies
   TF_KV_SHARE states, and the over-64 refusal's text says "a slot each in the startup plan", which isn't where
   those states live. Wording only.
4. Optional: a host test that shareSlot in finishRank equals stateBytes at the last mark after Pages.init
   (the size main readied), and one that count passes slots only with share on, a layout slot and the cache on
   the device.

If you have them, main-against-branch numbers with TF_GLM_HOST_CACHE=1 and TF_KV_SHARE=16 (startup line, and a
resumed reply against a fresh one) would cover the one path your receipt leaves out. native/cuda.zig is now
exactly 600 lines, so nothing more can go there.