[Bizuayeu]

With the engine's own rows on pages, the prompt cache gets what the windows the room holds leave, so a room just past a
window's threshold leaves it next to nothing. By the planner's arithmetic on the new test's layout, at TP=3 with a 1M
window and four streams (291 MiB fixed each):

Room	Unasked (16 GiB), main and this head	--prompt-cache-gib 4 on main	--prompt-cache-gib 4 on this head
38.8 GiB	12.44 GiB cache, 4,098 pages (2 windows)	4.00 GiB, 5,452 pages (2.66)	4.00 GiB, 5,452 pages (2.66)
39.6 GiB	0.48 GiB cache, 6,147 pages (3 windows)	0.48 GiB, 6,147 pages (3)	4.00 GiB, 5,581 pages (2.72)

On two GB10s at TP=2 with a 262,144-token window and four streams, the prompt cache off the pool's pages so the startup
line gives its size (the room each start found is in the first column):

Start	Pages	Startup line's windows	Prompt cache	token_sha256 (128 greedy tokens)
main, unasked (16.0 GiB)	2,052	"enough for one"	2.3 GiB	44be1c0c6b3d3911
main, --prompt-cache-gib 4 (16.1 GiB)	2,052	"enough for one"	2.3 GiB	44be1c0c6b3d3911
this head, unasked (16.1 GiB)	2,052	"enough for 4"	2.3 GiB	44be1c0c6b3d3911
this head, --prompt-cache-gib 4 (16.1 GiB)	1,785	"enough for 3"	4.0 GiB	44be1c0c6b3d3911

Unasked, the four windows still come first and the cache gets what they leave; asked, main still gave it what the
windows left, and this head gives it the 4 GiB and the pool the rest.

The startup line counts the whole windows the pool's pages hold: a 262,144-token window and a decode window's 8 rows
take 513 pages of 512 rows, so 2,052 pages are 4 windows and 1,785 are 3.48, shown as 3. All four streams run either
way, each taking pages as it needs them; the count is how many can be at a whole window at once.

What this changes

553c8bb set as many windows as the room holds aside before the prompt cache, so the pool never holds fewer rows
than windows in the same room, and left --prompt-cache-gib after them too. This keeps that order when the cache is not
asked for, and lets an operator who names a size trade the extra windows for it: a cache --prompt-cache-gib names
(Budget.asked) comes after the engine's own window and a fixed part and page for each other stream, before any other
window, and the pool takes the rest, so the streams still take pages as they need them. holds() counts the same way,
and Plan.taken is what the windows took from the cache.

The one place this changes 553c8bb's order is an asked cache, and so the one test it changes is its edge in
cuda_kv_pool.zig: the unasked 16 GiB edge keeps the four windows first, and the asked 4 GiB edge is the engine's
window and a fixed part and page each, as before 553c8bb. kv_pool_test.zig's tests of 553c8bb pass unchanged.

The warning that a cache cannot keep one state said --prompt-cache-gib only caps the budget; on pages whose windows
took the cache first, it now names what they took and that --prompt-cache-gib N keeps N GiB before them. A second
commit makes the startup line count the whole windows the pages hold, where it said "enough for one" whatever they held.

@MiaAI-Lab found the drop on their GLM-5.3-Flash recipe
(MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104: no prompt cache
at a 19.8 GiB room, 3.6 GiB at 18.9) and traced it to held(), with a cache set aside before the extra windows as the
fix.

• Issue: n/a (no issue was filed; the recipe above describes the drop).
• Stacked on: main (c085aab04, where the receipt ran; it merges cleanly with main at 8abb30f6f), two commits
  (3a52172fb, 4b59f59cb).
• atc overlap --json (2026-11-05 03:10 UTC, base fb0895099; complete: false, as atc could not read every open pull
  request's files): 31 open pull requests share files with this one. atc test-merged ten: nine merge cleanly and glm: TF_GLM_KV_LIVE_HALF=1 keeps the live half of the KDA state alone in a kept prompt state #2181
  conflicts in docs/recipes/glm-5.3-flash-cuda.md, where each adds a bullet at the same place, so both stay. I
  test-merged the other 21 with git merge-tree: sixteen merge cleanly in this branch's files, and Muse Spark Open on CUDA: EXL3, DFlash2, two GPUs #1295, lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503, lfm3: serve LFM3-1.7B on CUDA: NVFP4 and FP8 weight-only, MLX 4-bit (stacked on #1513, #1492 and #1503) #1508,
  lfm3 CUDA: serve the block FP8 export weight-only and the MLX 4-bit export as stored (stacked on #1508) #1634 and lfm3-vl: chat-template thinking suffix, fitGrid geometry, and deployment recipe #2045 conflict in native/cuda.zig as they already do with main (one conflict block each, with this branch
  or without it; this branch's one line there merges).

Checks on this head

• Head commit: 4b59f59cb
• Lean checker on the base and on this head: 425 and 425 problems, 0 new (the 425 are inherited).
• Tests run (Zig 0.17.0, WSL, x86_64): zig build test-kv-pool 38/38 (37/37 on the base; the new test is in this
  step), test-cuda-host 4,795 of 4,818 (23 skipped), test 2,215 of 2,251 (36 skipped; Zig's cache kept the steps
  another run had built from the same inputs), test-glm-prompts-host 7/7, test-glm-prompt-settings-host 6/6,
  test-glm-tp3-host 8/8, test-glm-prompt-arena-host 1/1, test-glm-startup-host, test-glm-pages-host,
  test-golden, compile-cuda-tests (also aarch64), compile-config-server: no failures. zig fmt --check build.zig zig and git diff --check: clean.
• CI: green on this head (zig, zig-windows, zig-macos-dev-tools, fmt, benchmark-clients on Ubuntu and Windows,
  release-paths).
• Not run, and why: TP=3 and a room just past a 1M window's threshold on the device (the first table is the planner's
  arithmetic there; the device run shows the same order at a 256K window); the other families whose engine's own rows
  are pages, whose checkpoints this pair does not hold.

Receipt

• Environment: base c085aab04 and this head, each built with Zig 0.17.0 and nvcc 13.3.73 for sm_121; two GB10s (128 GB
  unified memory each, Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 580.178.04) at TP=2 over RoCE;
  Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f91; --context 262144 --parallel 4, TF_KV_PAGES unset (pages),
  TF_GLM_KV_SHARE unset.
• Paths and platforms the change reaches: CUDA's startup planning when --prompt-cache-gib is given, for the families
  whose engine's own rows are pages (GLM at two or more ranks, Gemma 4, Kolibri-1, Dory, Spark2.5, and Qwen3.5 with
  TF_QWEN_KV_PAGES_ALL=1); Metal's pool plans (metal_admit.zig, qwen35.zig) pass no cache and are unchanged. Run:
  GLM at TP=2 on two GB10s (sm_121), unasked and asked. Not run: the other families, other SMs, TP=3.
• Exactness: the same greedy reply on all four starts (token_sha256 44be1c0c6b3d3911). On this head with
  --prompt-cache-gib 4: bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial
  found every reply at one and four streams equal to its request alone (code and chat, 3 and 12 of 3 and 12) and the
  alone runs equal to "draft": false; chat prompts of about 7K, 15K and 30K tokens and the decode check's count prompt
  (256 greedy tokens) gave the same token_sha256 drafted and with "draft": false, the second resuming the first's
  kept prompt (6,144 to 28,672 tokens cached). bench_cache.py's raw-token completions found no kept state at these
  settings on main or on this head (their repeats ran cold, with the first request's tokens), so the resumed check is the
  chat one.
• Decode speed before and after: n/a (the plan sizes memory only; no kernel or round changes).
• Prompt speed before and after: n/a (as above).
• Licence of any new weights or drafter: n/a.

Raw results

Startup lines (rank 0), TP=2, --context 262144 --parallel 4, TF_GLM_KV_SHARE unset:

main, unasked: [tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 90.46 GiB; 4 streams at once, the engine's own and 3 more taking 2052 pages of 512 rows (12.77 GiB) as they need them (enough for one at a 262144-token window), 340 MiB fixed each, of 16.0 GiB left after a 9.0 GiB reserve; prompts in passes of up to 2048 rows, a 2.3 GiB prompt cache; own glue kernels
main, --prompt-cache-gib 4: [tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 90.46 GiB; 4 streams at once, the engine's own and 3 more taking 2052 pages of 512 rows (12.77 GiB) as they need them (enough for one at a 262144-token window), 340 MiB fixed each, of 16.1 GiB left after a 9.0 GiB reserve; prompts in passes of up to 2048 rows, a 2.3 GiB prompt cache; own glue kernels
this head, unasked: [tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 90.46 GiB; 4 streams at once, the engine's own and 3 more taking 2052 pages of 512 rows (12.77 GiB) as they need them (enough for 4 at a 262144-token window), 340 MiB fixed each, of 16.1 GiB left after a 9.0 GiB reserve; prompts in passes of up to 2048 rows, a 2.3 GiB prompt cache; own glue kernels
this head, --prompt-cache-gib 4: [tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 90.46 GiB; 4 streams at once, the engine's own and 3 more taking 1785 pages of 512 rows (11.11 GiB) as they need them (enough for 3 at a 262144-token window), 340 MiB fixed each, of 16.1 GiB left after a 9.0 GiB reserve; prompts in passes of up to 2048 rows, a 4.0 GiB prompt cache; own glue kernels

bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, this head with --prompt-cache-gib 4:

{"prompt": "code", "temperature": 0.0, "streams": 1, "failed": 0, "aggregate_tps": 44.1, "per_stream_tps": 44.1, "alone": {"equal": 3, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "code", "temperature": 0.0, "streams": 4, "failed": 0, "aggregate_tps": 118.2, "per_stream_tps": 29.8, "alone": {"equal": 12, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "chat", "temperature": 0.0, "streams": 1, "failed": 0, "aggregate_tps": 36.1, "per_stream_tps": 36.1, "alone": {"equal": 3, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "chat", "temperature": 0.0, "streams": 4, "failed": 0, "aggregate_tps": 93.4, "per_stream_tps": 24.1, "alone": {"equal": 12, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}

bench_cache.py --require-cold --require-reuse as each server's first requests, --prompt-cache-gib 4: no kept state for these raw-token completions on main or on this head:

{"run": "main", "tokens": 8192, "qualified": false, "rows": [["first", "cold", 0, null], ["repeated", "cold", 0, true], ["repeated", "cold", 0, true], ["concurrent", "cold", 0, true], ["concurrent", "cold", 0, true]]}
{"run": "main", "tokens": 32768, "qualified": false, "rows": [["first", "cold", 0, null], ["repeated", "cold", 0, true], ["repeated", "cold", 0, true], ["concurrent", "cold", 0, true], ["concurrent", "cold", 0, true]]}
{"run": "this head", "tokens": 8192, "qualified": false, "rows": [["first", "cold", 0, null], ["repeated", "cold", 0, true], ["repeated", "cold", 0, true], ["concurrent", "cold", 0, true], ["concurrent", "cold", 0, true]]}
{"run": "this head", "tokens": 32768, "qualified": false, "rows": [["first", "cold", 0, null], ["repeated", "cold", 0, true], ["repeated", "cold", 0, true], ["concurrent", "cold", 0, true], ["concurrent", "cold", 0, true]]}

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency (this head's
  kv_pool_test.zig and cuda_cache_capacity_test.zig on the base fail to build: kv_pool.Plan has no taken there
  and warnCapacity takes one argument fewer; they have no dependency to skip on).
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (Exactness
  above).
• No precision traded for speed. If the bits change, the description says which sums change and why (no arithmetic
  on rows changes).
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter,
  no slower than without it, greedy and sampled (the plan sizes memory only: no kernel, round or drafter path changes).

[ashhart]

Thanks @Bizuayeu, and @MiaAI-Lab for finding it. Unset, the order is main's, and a named cache now gets what it names. It's in the queue; we'll add the changelog line, which says that a command already passing --prompt-cache-gib N splits memory differently after an upgrade.

[ashhart]

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @Bizuayeu and @MiaAI-Lab! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3