[gilby]

What this changes

The Metal lane target (prompts, windows of several slots in one launch, keeps), the prompt cache with cached_tokens, and the native server over Metal (zig build dsv41-metal-serve); text only.

Part 16 of 18 of the stack for #1676; stacks on dsv41-metal-pr/15-serve-lanes (part 15, #1719), on main with parts 1 to 5 (#1705 to #1709) merged.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold 26f52c1 on main ad98b60 (native Zig 0.17.0; no MLX or PyTorch); Apple M3 Ultra (60-core GPU), 96 GB, macOS, for the kernel and device tests; Apple M3 Ultra (80-core GPU), 512 GB for full-model numbers; checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: the tiny model served through LaneHost in test-dsv41-metal-serve: concurrent == solo == serial, drafted == "draft": false, prompt-cache hits == fresh runs, seeded replies repeat, a cancelled prompt leaves the next request valid.
• Full model (Apple M3 Ultra, 80-core GPU, 512 GB; deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights; serve --parallel 4 --context 1048576 --prompt-cache-gib 8 --vision, DSpark drafts on; measured at the stack's top before the rebase onto main 3ce4a04, 60d51fd, the same 13 commits on main 9e8c368 (git range-diff shows each patch unchanged except part 10's one planted-control line, TF_CTL == 4 only, which the served kernel compiles out), whose served path for text is this part's): tools/bench_concurrent.py --tokens 128 --temperatures 0,1 --levels 1,4 --alone --serial: 8 cells, 0 failed, 0 unequal (alone 1/1 or 4/4 and serial 1/1 or 4/4 equal in every cell). The memory gate, 4 cold 912K-token prompts at once in 4 slots of 1,048,576 positions (all four recall their needle, about 333 GB of GPU memory at peak), is from the 29 Oct stack's top (5db73126) and was not re-run on this head.
• Real release weights (Apple M3 Ultra, 80-core GPU, 512 GB; deepseek-ai/DeepSeek-V4.1-Flash-1017 release shards; run at the stack's top, a5e2ec9, whose Metal package and decoder layer are this part's): TF_DSV41_REAL=<checkpoint> zig build test-dsv41-metal-real on layer 2 (compress ratio 2) and layer 21 (ratio 1): exit 0, 5/5 steps, 4/4 tests each; every stage at 1 and 4 rows within its float64 bound, the closest margins at 1.000 of the bound (layer 2: shared SwiGLU 6.103e-5 vs 6.104e-5, routed + shared join 2.441e-4 vs 2.441e-4, hc post 3.906e-3 vs 3.907e-3; layer 21: shared SwiGLU 1.221e-4 vs 1.221e-4, join 4.883e-4 vs 4.883e-4, hc post 1.953e-3 vs 1.953e-3); mapped weights give the same output as copied weights, bit for bit, at 4 rows.
• Decode speed before and after, from tools/bench_openai.py: no Metal DeepSeek-V4.1 backend before this stack. After (60d51fd, the same patches before this rebase, --tokens 256 --reps 3 --temperatures 0,1, median): drafts on, fibonacci-raw 53.8 tok/s at T=0 and 50.5 at T=1, gpu-chat-no-think 43.8 at T=0 and 44.2 at T=1; drafts off (serve --no-drafts, one token a round), fibonacci-raw 35.4 at T=0 and 34.8 at T=1, gpu-chat-no-think 35.4 at T=0 and 34.6 at T=1. bench_concurrent aggregate at 1 / 4 streams: code 46.9 / 78.5 at T=0 and 52.1 / 61.4 at T=1; chat 43.2 / 76.1 at T=0 and 47.5 / 62.2 at T=1.
• Prompt speed before and after: no Metal DeepSeek-V4.1 backend before this stack. After (60d51fd, the same patches before this rebase): tools/bench_cache.py --prompt-file <32,768 token ids> --expected-prompt-tokens 32768 --require-cold --require-reuse --tokens 64 --repeats 2: cold first request ttft_s 42.76 (about 766 tok/s), cached_tokens 0; repeats 0.028 and 0.029 s and two concurrent 0.029 and 0.056 s with cached_tokens 32,768, the same tokens in every row; qualified.
• Tests run, with pass, fail and skip counts: zig build test tf-kernel-contract -Dcpu=apple_m1: exit 0, 627/627 steps succeeded; kernel contract, tf-kernel-contract --only dsv41 (Metal): SUMMARY 89 kernels: 89 pass gates 1-4, 0 fail, 0 known findings; 186 controls: 186 red as planted, 0 missed; 0 limit problems; gates 119142 ms, limits 161 ms; zig build test-dsv41-metal-all -Dcpu=apple_m1 (every DeepSeek Metal device step): exit 0, 9/9 steps succeeded; 37/37 tests passed; python3 tools/zig/lean_check.py: 5144 files, 424 problems, the same count as main (5027 files, 424). Zig's summary counts only the test binaries it ran in that invocation, so the step count is the comparable figure.
• Not run, and why: CUDA: no NVIDIA device here; Metal on M1, M2, M4 and M5: one M3 Ultra only; time-dsv41-metal-real (timing only) was not part of this receipt.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt.
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[gilby]

The full-model receipt in #1720 and #1722 is now from the stack's final head, 60d51fd7 on main 9e8c368b (Apple M3 Ultra, 80-core GPU, 512 GB; deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights; serve --parallel 4 --context 1048576 --prompt-cache-gib 8 --vision), replacing the numbers from the same stack before its rebase. The bodies carry the full lines; in short:

• tools/bench_concurrent.py --tokens 128 --temperatures 0,1 --levels 1,4 --alone --serial: 8 cells, 0 failed, 0 unequal.
• tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0,1, medians: drafts on, fibonacci-raw 53.8 / 50.5 tok/s and gpu-chat-no-think 43.8 / 44.2 (T=0 / T=1); --no-drafts 35.4 / 34.8 and 35.4 / 34.6.
• tools/bench_cache.py --require-cold --require-reuse, 32,768 tokens: cold ttft_s 42.76 (about 766 tok/s), reuse 0.028 to 0.056 s with cached_tokens 32,768; qualified.
• Release weights at the same head: layers 2 and 21 against float64 4/4 each with mapped == copied bit for bit, and the vision tower 14/14.

Each number is within 1% of the pre-rebase run. The 4 x 912K memory gate is still the 29 Oct run, as the bodies say.

[ashhart]

Thanks @gilby. The code holds: no new kernel, the target refuses wrong slots, bad tokens, position and row mismatches, tree windows and non-prefix keeps, the lanes keep DeepSeek's opt-outs, the prompt cache is an exact token-prefix store under a byte cap, and CUDA is unchanged. Exactness is shown (concurrent = solo = serial, drafted = draft:false, cache hits = fresh; full model 0 unequal of 8 cells), and drafts are faster in all four single-stream cells (53.8/50.5/43.8/44.2 against 35.4/34.8/35.4/34.6 tok/s). One ask, a receipt, so never-slower can be checked: bench_concurrent at 1 and 4 streams with --no-drafts, and a 32,768-token cold first request with and without drafts. Small: adding logits to an existing entry can briefly exceed the cache cap, and the 4x912K memory gate isn't re-run on this head.

[ashhart]

Re-reviewed after the 05:25 push: the code is as we reviewed, rebased. The one open item is the never-slower receipt, which is listed with #1722's on #1722.

[ashhart]

Thanks @gilby. We compared this push with the version we reviewed. The only change is one build line, the lane test now importing core so it compiles on current main. The code stands as reviewed. The one open item is still the receipt, and we've listed it once on #1722 so it covers both. Two small points stay open, neither blocking: adding logits to an existing cache entry can briefly go over the cap, and the 4 × 912K memory gate is still the 29 Oct run.