[wojo]

What this changes

GLM-5.3-Flash on CUDA at TP=2 now serves several streams that share a long window. Before, --parallel 8 at a 1M,
512k or 256k window was refused at start. Two commits:

1. 75218c46 core kv_pool: when the engine's own sequence is on pages (Budget.engine_paged, GLM's), plan sets
   aside, before the prompt cache, the pages of as many windows as the room would hold as a window a stream:
   min(lanes, 1 + (room - window pages) / (window pages + fixed)), and at least the one window glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211 already sets
   aside. The cache gets what those leave, so the pool never holds fewer rows than windows would in the same room.
   holds counts the same way. Nemotron (engine_paged false) is unchanged.
2. dbd073dc glm/cuda: TF_KV_PAGES=auto pages from two ranks (world >= 2), not three. One GPU and
   --kv-cache fp8 keep their windows, as do off and 0.

Refs #1313 (step 1 of its proposal; #1163's question 1, open since #1211).

What is wrong on main (67cc8fb)

• auto keeps a window per stream at one and two ranks (zig/src/families/glm/cuda_start.zig:22). At TP=2 with an
  EXL3 checkpoint, a 1M window is 12.93 GiB a stream, so the startup refuses --parallel 8 (verbatim in GLM-5.3-Flash CUDA at TP=2: --parallel 8 at a 1M window is refused (90.48 GiB of windows, 2.33 GiB left), though KV pages would serve it #1313:
  "needs 90.48 GiB for its streams …, and the CUDA memory budget leaves 2.33 GiB").
• TF_KV_PAGES=on would serve it, but auto could not simply page at two ranks. kv_pool.plan funds the prompt
  cache before every page past one window and a page a stream (kv_pool.zig:79), and the default cache is 16 GiB.
  At TP=2's usual --context 131072 --parallel 4, with glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211's 9.5 GiB of room, that is 260 pages (133,120 tokens)
  for four streams, where windows hold four of 131,072. That is the reason glm/cuda: KV pages from the core pool, kept states sized by their position (#1142) #1163 gave for keeping windows there.

The change

held(l, b, fit): the pages the cache leaves the other streams, the larger of a page and a run for each stream (as
before) and (windows - 1) windows' pages. windows(l, b) is 1 unless the engine's own sequence pages. Then plan
gives the cache min(cache, left -| held); holds checks left -| (fit x fixed + held) >= cache. The page count and
its bound (most) are as before. With auto at two ranks, by the plan's arithmetic (computed; the receipt has the
measured lines):

TP, window, --parallel	main, auto	this branch, auto
2, 131072, 4	4 windows, cache ~2.4 GiB	1,028 pages (4 windows' rows, shared), cache ~2.6 GiB
2, 262144, 8	refused (--parallel auto: 4 windows)	~2,052 pages (4 windows' rows) for 8 streams, cache ~1.2 GiB
2, 1048576, 8	refused	~2,056 pages (one window and a page each) for 8 streams, cache ~1.2 GiB
3, 1048576, 8	3,355 pages, cache 16 GiB	4,098 pages (2 windows' rows), cache ~11.4 GiB

The last row is the one default this moves at three ranks (question 1 on #1313). The engine's-own test in
kv_pool_test.zig kept its checks but at a room under two windows (24 GiB and an 8 GiB cache instead of 40 and 16).
At 40 GiB the plan now sets two more windows aside before the cache, which the new test checks. The same goes for
cuda_kv_pool.zig's fits threshold: it is now four windows' pages, which its 8,192-token room holds.

Tests

• kv_pool_test "the engine's own sequence on pages: the windows the room would hold come before the cache" covers
  three cases. TP=2 at 128k with four streams in a 9.5 GiB room gives 4 x 257 pages, and the cache gets the rest,
  whole for holds and not one byte more. TP=2 at 1M with eight streams in a room of one window is unchanged: the
  window, a page each, the cache. TP=3 at 1M with eight streams in 38 GiB gives 2 x 2,049 pages before the cache.
• cuda_native "TF_KV_PAGES: auto pages from two ranks, …": auto and unset page at 2, 3 and 16 ranks and keep
  windows at one. The FP8 test is unchanged (windows at every rank count, on and all refused).

On main (the tests alone, zig build test-kv-pool / test-cuda-host; trimmed, paths relative):

error: 'kv_pool_test.test.the engine's own sequence on pages: the windows the room would hold come before the cache' failed:
       expected 1028, found 260
       zig/src/core/kv_pool_test.zig:113:5: in test.the engine's own sequence on pages: the windows the room would hold come before the cache (test)
error: 'cuda_native.test.TF_KV_PAGES: auto pages from two ranks, off and 0 give windows, all and other values page like on' failed:
       zig/src/families/glm/cuda_native.zig:440:9: in test.TF_KV_PAGES: auto pages from two ranks, …
               try std.testing.expect(try paging(null, w, false));

Both pass on this branch.

Receipt

• Host, base 67cc8fbce and this branch (dbd073dc), fresh caches each, Zig 0.17.0, Apple M4 Mac, macOS 27:
  test-kv-pool 21/21 on base and 22/22 on the branch (the new test), test-glm-startup-host 5/5 on both,
  test-kv-safety-host 21/21 on both, test-kv-merge-host 19/19 on both, test-cuda-host
  3,246/3,262 on base (14 skipped, 2 failed) and 3,249/3,263 on the branch (14 skipped, 0 failed). The base's two
  failures are timing tests outside this change (cortheon fuzz and html "linear time"); they passed in the
  branch's run. zig fmt --check is clean on the changed files. lean_check.py: 434
  problems on base and on the branch, none new.

• GPU: base 67cc8fbce and this branch (dbd073dc2), both tensorfold-native built alike (Zig 0.17.0,
  aarch64-linux-gnu, -Doptimize=fast) with the same 97 embedded CUDA kernel images (one fatbin set, nvcc 13.0
  V13.0.88, sm_121). Two NVIDIA DGX Spark (GB10, 121 GiB unified memory) at TP=2 over a direct RoCE link, driver
  580.178.04 (CUDA 13.0), cuBLASLt 13.1.1.3, NCCL 2.30.7, glibc 2.39, Ubuntu 24.04.5 LTS, kernel 6.17.0-1032-nvidia,
  aarch64; one model process on each. Checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold@078455ff, MTP drafts.
  Every serve: serve <checkpoint> --backend cuda --parallel P --context C --no-update-check with
  TENSORFOLD_CUDA_KERNELS=native TF_HEAT_HIGH=92 TF_HEAT_LOW=88, and the env named per row.

• Starts. Each line is rank 0's start-up line, cut to the part that differs (all print of X GiB left after a 12.2 GiB reserve and prompts in passes of up to 2048 rows):

  	tree, TF_KV_PAGES	--context, --parallel	start-up line	left	prompt cache
  1a	main, unset	1048576, 8	refused: --parallel 8 needs 90.48 GiB for its streams at a 1048576-token window, and the CUDA memory budget leaves 3.89 GiB
  1b	main, on	1048576, 8	the engine's own and 7 more taking 2056 pages of 512 rows (12.80 GiB) as they need them (enough for one at a 1048576-token window), 180 MiB fixed each	15.6 GiB	1.6 GiB
  1c	branch, unset	1048576, 8	the same as 1b: 2056 pages of 512 rows (12.80 GiB), 180 MiB fixed each	15.8 GiB	1.8 GiB
  1d	main, on	262144, 8	taking 520 pages of 512 rows (3.24 GiB) ... (enough for one at a 262144-token window)	16.2 GiB	11.8 GiB
  1e	branch, unset	262144, 8	taking 2052 pages of 512 rows (12.77 GiB): 1,050,624 rows, four windows'	16.1 GiB	2.1 GiB
  1f	main, unset	131072, 4	4 streams at once, 1.77 GiB each at a 131072-token window: 524,288 rows	14.5 GiB	9.2 GiB
  1g	branch, unset, TF_KV_POISON=1	131072, 4	the engine's own and 3 more taking 1028 pages of 512 rows (6.40 GiB): 526,336 rows	16.2 GiB	9.3 GiB
  1h	branch, off	131072, 4	1f's line: 4 streams at once, 1.77 GiB each at a 131072-token window	14.7 GiB	9.4 GiB

  1c equals 1b and 1h equals 1f apart from 0.2 GiB of left between starts. The room was 14.5 to 16.2 GiB, more than
  the 9.5 GiB the table in "The change" assumes, so the measured caches are larger than that table's. Main's refusal
  here leaves 3.89 GiB, where GLM-5.3-Flash CUDA at TP=2: --parallel 8 at a 1M window is refused (90.48 GiB of windows, 2.33 GiB left), though KV pages would serve it #1313 quotes 2.33 GiB from an earlier start. After every branch serve with traffic the
  idle census is even (kv pool idle: promised 0, free N + store 0 == total N, N = 2056 on 1c, 1028 on 1g and the
  speed serves; no UNEVEN).

• Exactness at 131072, --parallel 4, default prompt cache: pages = 1g (TF_KV_POISON=1), windows = 1h, main = 1f.

  • The recipe's tools/exact.py, recorded on pages, then on windows and on main: compare pages against windows and
    pages against main, IDENTICAL (0 problem(s), 31 cases) both.
  • bench_concurrent.py --levels 1,2,4,8 --alone --serial on pages: alone 60/60 equal and serial 72/72 equal
    (0 unequal, 0 failed).
  • reuse_served.py record --planned --min-prompt 2048 on pages: 44 requests, 0 not cached as planned; its 20
    drafted / "draft": false pairs give equal tokens. replay on a main serve (--prompt-cache-gib 0) and
    compare: 43/43 replies equal and as cached as planned.

• Speed at 131072, --parallel 4, --prompt-cache-gib 0, no poison, serves alternated main, branch, main, branch.
  Decode: bench_openai.py --tokens 256 --reps 3 --temperatures 0, median. Prefill: reuse_served.py cold and
  bench_cache.py --require-cold (the 32k chat), seconds. The last column is the branch's better serve over main's
  worse one (time, so under 1 is faster):

  measure	main 1	branch 1	main 2	branch 2	branch best / main worst
  decode fibonacci-raw, tok/s	48.87	48.88	48.96	48.86	0.9999
  decode gpu-chat-no-think, tok/s	27.40	27.34	27.38	27.27	1.0015
  cold 32k chat, s	2.055	2.054	2.054	2.074	0.9993
  cold 1k, s	0.877	0.875	0.895	0.891	0.9779
  cold 2k, s	1.780	1.783	1.776	1.786	1.0019
  cold 8k, s	7.611	7.592	7.602	7.645	0.9975
  cold 16k, s	15.18	15.10	15.09	15.19	0.9943
  cold 32k, s	30.29	30.20	30.16	30.36	0.9968
  cold 64k, s	60.82	60.88	60.63	61.05	1.0010

  Every serve gave the same tokens for each decode prompt and for the 32k chat. No measure is over 1.002. The second
  branch serve is the slowest on 2k-64k by up to 0.7% against main's best; main's own two serves differ by up to 2%
  (1k).

• A ~1M prompt with seven short ones beside it, on the branch at 1048576 x 8, unset (1c). The prompt:
  tf-receipt-long-v1/chat/seed=1/words=854334/turns=8 (long_prompt.py in the recipe repo; 18 messages, body
  sha256 b849e8e45a16ddb1aeb6a91525b5e100239cd870e5cbabb58fe9dc6edf445c2a), 1,020,560 prompt tokens, 16 output
  tokens: cached 0, prefill 1,276.2 s, time to first token 1,276.5 s, wall 1,277.4 s, token_sha 9e0c38a5d727.
  Seven short prompts (1,177 to 1,406 tokens, 64 output tokens each) sent 30 s into its fill were all admitted beside
  it: time to first token 2.45 to 11.47 s (alone afterwards on the same serve: 1.27 to 1.42 s), none waited for the
  long one to end, and each equals its alone run. Rank 0's device_memory_peak_bytes 101.1 GiB; nvidia-smi
  103,898 MiB on each rank at ready; the lowest MemAvailable sampled during the serve 11.7 GiB (rank 0) and 12.3 GiB
  (rank 1). The idle census after it: free 2056 + store 0 == total 2056.

• Not run: TP=3 or more (no ring of three here; the table's TP=3 row is the plan's arithmetic, checked by the host
  test). --kv-cache fp8 on a device (its windows are checked by the host test). Exactness at --parallel 8 and a
  windows serve in the speed alternation. Also not run: one GPU, Metal and MLX (unchanged paths).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  (On pages: 20/20 drafted / "draft": false pairs, 43/43 resumed replies equal on main, bench_concurrent alone
  60/60 and serial 72/72.)
• No precision traded for speed. No kernel or arithmetic changes: only how many pages the pool gets, and the
  default that picks pages.
• A new or changed GPU kernel is registered with the kernel contract. (None changed.)
• Prompt processing is no slower than the last release. (Compared with main, alternated, not with a release:
  cold 1k-64k and the 32k chat within 0.2%, best against worst.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (Host tests on an
  Apple M4 without the Metal toolchain; served CUDA at TP=2 on two DGX Spark (GB10). No Metal device, no TP=3.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Both fail on
  main and pass here; host tests with nothing to skip on.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)

[ashhart]

Landed on main (3f1d669..0d24ecb), with both commits under your authorship. Thank you! On two GB10s at TP=2 with the NVFP4 checkpoint, against main in alternating runs (base, new, new, base):

• tokens: 52 of 52 equal;
• prefill: +17.4% at 2k, flat at 8k, +5.5% at 32k;
• two streams: +14.4% for chat and +14.2% for code;
• one stream: +7.7% for chat, flat for code;
• resumed chats qualified with thinking on and off.

The startup line confirms pages from the core pool at two ranks. This closes #1313.

- Synthema, Ash's AI Assistant.