GLM-5.3-Flash CUDA at TP=2: --parallel 8 at a 1M window is refused (90.48 GiB of windows, 2.33 GiB left), though KV pages would serve it #1313

Seen on native main 674fb9e (unchanged on 67cc8fb), GLM-5.3-Flash on CUDA at TP=2: two GB10s over a direct RoCE
link, an EXL3 checkpoint, MTP drafts, --parallel 8, TF_KV_PAGES unset, the default prompt cache. Every context I
tried past 131072 is refused at start, the same way at each:

--parallel 8 needs 90.48 GiB for its streams at a 1048576-token window, and the CUDA memory budget leaves 2.33 GiB:
serve --parallel 1, or lower --context

(the same at --context 524288 and 262144, with the numbers for those windows). The pair serves one conversation at
a 1M window, or eight short ones, but not eight that share a long window.

Why

• TF_KV_PAGES=auto pages from three ranks only (zig/src/families/glm/cuda_start.zig:22,
  return world >= 3 and !kv8;; the test cuda_native.zig:436 and cuda_ranks.zig:307 say the same). At one and two
  ranks every stream past the engine's own takes a whole window, 12.93 GiB a rank at 1M, so seven more need 90.48 GiB.
• glm/cuda: KV pages from the core pool, kept states sized by their position (#1142) #1163's question 1 gave the reason, and glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211 answered half of it. kv_pool.plan funds a page a stream, then the
  prompt cache, then the other pages, and at TP=2 the room after the weights is smaller than the default 16 GiB
  cache. Before glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211 that left the pool a few pages. Since glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211 the engine's own sequence is on pages and the plan
  sets one window of them aside first (kv_pool.zig:98), so on always serves one conversation at --context. glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211
  left the other half open: whether auto should page at one and two ranks.
• With pages, TP=2 has the room. By kv_pool.plan's own arithmetic (computed, not measured) the same pair with
  TF_KV_PAGES=on at --context 1048576 --parallel 8 would get about 15.2 GiB of room (the 2.33 GiB above plus the
  engine's own window, which is now pages), take 2,049 pages for the window, 180 MiB fixed and a page for each of the
  seven other streams, and give the cache the last ~1.2 GiB: 2,056 pages, 1,052,672 tokens that the eight streams
  share. That is the same arithmetic that predicts TP=3's 3,357 pages against the 3,355 in glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211's receipt.

What stops auto from paging at two ranks today

The plan's order, not exactness or speed: #1163's and #1211's receipts already show TP=2 on pages equal to windows
token for token and within noise of main's speed. What would change is the default memory split. At TP=2's usual
--context 131072 --parallel 4 (#1211's receipt: 9.5 GiB of room) with the default 16 GiB cache, the plan as it is
gives (computed):

So turning auto on at two ranks with the plan as it is would queue four concurrent 64k conversations that run
together today. The same thing happens at a 256k window with --parallel 8: 520 pages (266,240 tokens) for eight
streams, where windows hold four streams of 262,144 tokens each.

Proposal, in two steps

Step 1: pages by default from two ranks, never holding fewer rows than windows would.

1. kv_pool.plan with engine_paged: before the prompt cache, set aside the pages of as many windows as the room
   would hold as a window a stream (n = min(lanes, 1 + (room - window pages) / (window pages + fixed)), at least
   the one window glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211 sets aside). Then the cache gets what those leave. Nemotron (engine_paged == false) does not
   change. The pool then holds at least the rows windows would have held in the same room, whatever the rank count.
   The cache gets what windows would have left it, less the fixed parts of the streams windows could not have served.
   • TP=2, 128k, --parallel 4: the third row of the table above.
   • TP=2, 1M, --parallel 8: unchanged from on today (n = 1).
   • TP=3, 1M, --parallel 8, as in glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211's receipt: 4,098 pages for the streams (the bound of two windows) and an
     11.4 GiB cache, where today there are 3,355 pages and a 16 GiB cache. Windows there serve only two streams and
     leave a 12.0 GiB cache. This is the one default that moves at three ranks (question 1).
2. TF_KV_PAGES=auto pages from two ranks (world >= 2), still keeping windows with --kv-cache fp8. One GPU stays
   on windows: I have no one-GPU GLM serve to qualify, and question 2 asks about it.
3. Optional, separately: FP8 rows on pages, as glm/cuda: KV pages from the core pool, kept states sized by their position (#1142) #1163's question 3 sketched. Each 512-row page keeps its rows' FP32
   scales after its rows (516 bytes a row and cache instead of 1,024). Windows would keep the same blocks, so a window
   is pages in order and one address formula serves both. tf_glm_lat_write_fp8 and tf_glm_chunks_{16,32}_fp8 would
   take the table as their BF16 twins do, and tf_glm_dsa_tokens already writes a sparse token as a page row. This
   changes a kept FP8 state's byte order (the snapshot copies blocks, not one run of rows and then one of scales), so it
   would need the kernel contract's registrations and its own receipt. I can write it as its own pull request after
   step 1.

Step 2: the latent split across ranks on those pages (design; any TP > 1).

TP does not split the latent today: every rank keeps every row (#1142: 13,056 bytes a token on each rank, 12,288 of
them latent). The DeepSeek precedent, kv/split.zig on sessions/pool.zig, stripes a split family's pages over the
ranks by residue (localPage: page p is rank p mod W's) and all-gathers each selection's owned rows. For GLM,
moving rows costs more than moving queries: a sparse row reads up to 2,048 latent rows of 1 KiB a layer, where its 32
heads' absorbed queries are 32 KiB. So the design keeps DeepSeek's page book and owner rule but moves queries:

• Owner. A conversation's latent pages all sit on one rank, its owner, chosen at admission by the most free
  latent pages, alike on every rank. Pooled index keys, rings and KDA state stay replicated, so the token selection
  runs on every rank as now. The latent part of a page sits only in its owner's arena: the pool keeps one id book a
  rank (DeepSeek's residue book; a conversation leases ids of its owner's residue), and begin carries the owner.
  Per rank, a token costs 768 + 12,288 / W bytes: 6,912 at TP=2 (1.89x the rows), 4,864 at TP=3, 3,840 at TP=4.
• Decode: the owner computes every head. Each rank sends the absorbed queries of the rows other ranks own to their
  owners (R x heads/W x 512 bf16). The owner runs the same chunk and merge launches over its pages for the other
  ranks' heads, and sends each rank its heads' outputs back. Each head of each row is computed by the same launches
  over the same bytes in the same order as today, so every rank's output equals the replicated engine's bit for bit;
  a rank that owns no row of a window runs empty passes. Cost per layer and rank at TP=2: two exchanges of R x 32 KiB.
• Prompt chunks. Borrowing first: a fresh prompt leases temporary latent pages on every other rank for the whole
  fill, runs unsharded (every rank writes and reads its own copy), and gives them back before its first decode round.
  When the other ranks have no free pages, the chunk fetches its history from the owner. That is the union of the
  rows its selection names, which both ranks compute from the replicated selection, so no count crosses the link,
  compacted, sent in pieces and renumbered. That is DeepSeek's union path with GLM's 528 or 1,024-byte rows.
• Kept states and resumes. A kept state holds the latent rows on its owner only. A resume goes to its state's
  owner when that rank has the pages, and otherwise carries the rows over (2,048 tokens or more; shorter ones
  recompute).
• What it is not. It is not the residue striping of one conversation's pages: the sparse pass reads its token
  list in 512-token chunks, and splitting a chunk by owner changes which keys meet in a chunk, so striping is not
  bit-exact for sparse rows without moving the rows themselves.

A working version of this design runs on a two-GB10 fork of the Zig engine (owner decode, history fetch,
borrowing, carries). On the same pair it serves a 1,039,996-token prompt with eight streams sharing 4,157,440 tokens
of pool, against 2,316,288 unsharded, every reply equal to the unsharded engine's. It is TP=2 only and keeps its own
pool. Porting it means putting it on kv_pool's pages, at any W, and that port is what step 2 would be.

Questions

1. The plan's order (glm/cuda: KV pages from the core pool, kept states sized by their position (#1142) #1163's question 1, open since glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211): is "pages before the cache, as many windows as the room
   would hold" the default you want? It moves TP=3's 1M default from a 16 GiB cache to an 11.4 GiB one with 22% more
   pages. The alternative keeps TP=3 as it is: hold every stream's window only when the room holds them all, and
   otherwise glm/cuda: the engine's own sequence on KV pages too, one window of them planned first (stacked on #1163) #1211's order. But then TP=2 at 256k and --parallel 8 holds fewer rows than four windows.
2. auto at one GPU: windows, as now, or pages like two and three ranks?
3. FP8 on pages: is the per-page scale block above the layout you want, windows included?
4. Step 2: an owner a conversation on the core pool's pages, moving queries rather than rows, at any TP. Would you take
   it in pieces (the per-rank latent book and owner, then owner decode, then the history fetch, then borrowing and
   carries), each one off by default until the last?

A pull request for step 1 (the plan and auto, with host tests that fail on main) follows.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)