[russellmm]

What this changes

On CUDA, GLM-5.3-Flash keeps its prompt states in one device arena that gets only the memory the KV pool leaves over.
On discrete cards that is often a fraction of a GiB, so a long conversation's state is refused and every turn
prefills the whole conversation again (#1510).

With TF_GLM_HOST_CACHE=1 on every rank:

• The snapshot arena is pinned host memory mapped into the device's address space (HostBuffer.allocMapped), so the
  existing cuMemcpyDtoDAsync keep/restore copies move a state across PCIe.
• GLM reports host_snaps, so the server uses its existing host-cache path: --prompt-cache-gib sizes the cache in
  host memory, and the KV pool plan sets no device cache aside.
• Unset, nothing changes.

Fixes #1735. Refs #1510: this is a host-memory route to that issue, complementing #1732 (prompts as pool pages) rather
than replacing it; the two combine (see "Not run"). No kernel, scheduler or arithmetic change.

Overlap with #1732: both add a field at the end of GLM's Loaded struct in cuda_native.zig, a one-line textual
conflict. Nothing else overlaps (atc overlap: #1411 and #1655 merge cleanly). Happy to rebase whichever lands second.

Why: the first step toward GLM-5.3-Flash on two 96 GB cards with system memory

The goal. Serve GLM-5.3-Flash (Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on two RTX PRO 6000 Blackwell cards
at TP=2, using the host's system memory to win back the KV room the weights take, without a major speed loss.

Why VRAM is the limit.

• The checkpoint's weights take 86.2 GiB of each 96 GB card.
• What's left for KV and kept prompt states is about 4-5 GiB a rank. BF16 latent KV costs about 12.8 GiB per 1M tokens.
• So with ≥1.5 GB left free on each card, the window tops out near 320-384K, against the model's 1M. The prompt cache
  gets what the KV pool leaves: under 1 GiB, enough to resume conversations up to ~30K tokens.
• These cards have ~57 GB/s each way to pinned host memory over PCIe Gen5 x16, and the host has ~100 GB free.

The plan, as separate PRs. Each one is one change with its own receipt. Later steps are listed for context, not
for review here.

1. This PR: kept prompt states in host memory.
   • Long conversations resume from host memory instead of prefilling again.
   • The device cache's share goes back to the KV pool.
   • Decode and prefill are unchanged (receipt below).
2. GLM's vision tower in host memory while not encoding, as --vision-offload does for other families. It frees
   ~1.05 GiB on rank 0, the rank that bounds both ranks' room.
3. Live KV in host memory for long windows.
   • GLM-5.3-Flash suits it: 34 of its 45 layers are KDA with a fixed-size state, and the 11 DSA layers read only the
     indexer's top 2,048 rows per query.
   • Keeping the latent rows of long contexts in pinned host memory, with the pooled index keys on the card, moves
     roughly 25 MB a decode token per rank across PCIe.
   • That makes a 512K-1M window plausible on these cards without a major decode loss.
   • We'd first measure the actual rows read per step and prototype it. We would build on the host tier in
     KV pool: host tier for the core pool (5/8) #1501/EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522 rather than add a second one, and we'd like maintainer guidance on that direction before starting.

Receipt

• Environment.

  • TensorFold main 94b90c9 with and without this commit, built with Zig 0.17.0 and nvcc 13.4.92 (-Dsm=120).
  • Two RTX PRO 6000 Blackwell Workstation cards (96 GB each, PCIe Gen5 x16, P2P between them), TP=2 on one host,
    NCCL 2.32.3, cuBLASLt 13.4, driver 615.71.09, Linux 7.3.
  • Host: Threadripper 9970X, 128 GB DDR5.
  • Checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold @ 76c0b517.
  • Serve flags on both ranks: --context 327680 --kv-cache bf16 --parallel 2 --vision,
    TENSORFOLD_MEMORY_RESERVE_GIB=2.6. Candidate: TF_GLM_HOST_CACHE=1 --prompt-cache-gib 24. GPU cap 400 W for
    every speed number.

• Exactness.

  • tools/bench_concurrent.py --levels 1,2 --tokens 256 --temperatures 0 --alone --serial, code and chat. Baseline
    and candidate: alone 3/3 and 6/6 equal, serial 1/1 equal in every cell; 0 unequal, 0 failed.
  • Resumed equals fresh: one fixed 59,208-token conversation, 4 turns, greedy 200-token replies. The baseline
    re-prefills every turn (cached_tokens 0). The candidate restores turns 2-4 from host memory (cached_tokens
    57,344). All 4 replies are identical to the baseline's, byte for byte.
  • Cache-off control: TF_GLM_HOST_CACHE=1 --prompt-cache-gib 0. The startup line reads "no prompt cache", no host
    memory is pinned, and cached_tokens is 0 on every turn.

• Decode speed (tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0), median tok/s; identical
  token_sha in every cell.

  prompt	baseline	candidate
  fibonacci-raw	262.2	261.7
  gpu-chat-no-think	145.3	144.9

  bench_concurrent.py aggregate at 2 streams: code 382.2 vs 381.1, chat 243.6 vs 242.8.

• Prompt speed (tools/bench_cache.py --require-cold --tokens 64), fixed token arrays from the repository's own
  docs, fresh server each run, alternating baseline/candidate. Cold first-request ttft_s:

  prompt	baseline (2 runs)	candidate (2 runs)
  32,768 tokens	7.85 / 7.88 s	7.88 / 7.88 s
  65,536 tokens	16.27 / 16.31 s	16.30 / 16.34 s

  Same token_sha, cached_tokens 0 on every first request.

• What it buys. Same server, same settings:

  	baseline	candidate
  device prompt cache	0.5 GiB	none; 24 GiB per rank in host memory
  KV pool	642 pages (4.00 GiB)	723 pages (4.50 GiB)
  rank 0 device memory free while serving	2.75 GiB	2.71 GiB
  next turn of a 104K / 250K conversation	prefills again (34 s / 83 s)	0.7 / 0.8 s
  10 interleaved conversations, 40K-200K	0% of follow-up tokens resumed	74%: 6 of 10 follow-ups in 0.2-0.6 s; the 4 oldest evicted when the 24 GiB filled

  Pinned host memory measured ~57 GB/s each way on this machine.

• Tests run.

  • zig build test --test-timeout 10m: 5,752 passed, 27 skipped, 0 failed.
  • test-kernel-contract: 141/141. test-golden: pass. test-server-security test-pull-security: 26/26.
  • fuzz-cortheon: 17/17. test-glm-snapshots-host: 20/20, with the new host-arena round trip in
    zig/glm_snapshot_host_ram.zig, which fails before this change. test-glm-pages-host: pass.
  • Plain zig build: pass. build_options_test.py: pass. git diff --check: clean.
  • lean_new.py against main: 424 problems on the base, 424 on the head, 0 new.

• Not run, and why.

  • Metal, and GB10 or other unified-memory hosts: not available here. On unified memory the arena is the same memory
    either way, so the setting gains nothing there.
  • TP=3, the NVFP4 and FP4-Local checkpoints, and sm_80-sm_89 cards: not available.
  • --learn-conversations: disk images go through the same snapshot plan, but were not exercised with the host arena.
  • glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732 (TF_GLM_KV_SHARE) combined with this setting: checked on glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732's head with this commit on top.
    • Its pool-page states' fixed parts move to host memory: rank 0 had 2.76 GiB free instead of 1.86 GiB with
      TF_GLM_KV_SHARE=8.
    • Resumed replies were identical.
    • With sharing on, conversations still keep their pages in the pool, so the interleaved test resumes no more than
      glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732 alone.

Checklist

• Every token still goes through the lane rounds. Only the snapshot arena's memory changes.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. The bits don't change: a state is copied byte for byte.
• No new or changed GPU kernel.
• Prompt processing is no slower than main (cold 32K/64K above).
• CUDA TP=2 on sm_120 only; Metal and the other CUDA classes are not run (above).
• The new test fails before the change and passes after it; it needs no device.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.