[jschmied]

What this changes

A new page, docs/optimization-ledger.md: which speed and memory optimization applies to which model family, and where each one stands, so a change proven on one family gets ported to the others instead of re-invented.

• One table of 27 optimizations (rows) by 21 native families (columns). Each cell is yes (applies, not done), no (with a short reason), done, failed (tried, no gain or rejected) or open, with the backend where it matters and a link to the PR or issue. Four cells are ? (add-plus-norm on deepseek_v41, lfm3 and kimi_k3; shared draft rounds on deepseek_v41), for whoever knows. To stay within 120 columns, the table is split into five column groups with the same rows.
• After the table, a numbered list: how each optimization works and what it saves, with per-family gain tables where PRs measured them (fused projections, copy drafts, shared draft rounds, solo chunks, read-ahead, direct load, L2 prefetch), and the prompt tail's per-family estimates from qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229's survey, marked as estimates.
• A rough-gain table: each optimization's estimated effect on prefill, decode and memory (low under 3 %, medium 3–15 %, big over 15 %, or what lets a case fit), for planning a port. Measured numbers stay in the list.
• The intro points at the shared helpers a port builds on. Two are not on main yet and say so: core/prompt_tail.zig (qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229) and core/state_walk.zig (core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215).

The page asks that a PR changing a cell updates it in the same PR.

• Issue: n/a, documentation only.
• Stacked on: main 01e13a04a, one commit.
• atc overlap --json: no conflicts.

When a cited PR lands

These cells say open and flip to done when their PR lands:

• qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229: row 1 (q3.5).
• CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033: row 21 (nemo, and the kol1, spk25, mm3 and dsv41 cells).
• core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215: row 5. qwen CUDA: an integrated GPU keeps prompt states in device memory, without pinning a copy each save (#2043) #2113: row 4. qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model at 2 and 3 stages (RFC #2005 step 2, stacked on #2241) #2115: row 26. GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411: row 8 (glm, CUDA). KV pool: host tier for the core pool (5/8) #1501: row 7. metal nemotron: TF_KV_SHARE keeps prompt states as core pool pages (stacked on #1837) #1850: row 3 (nemo, Metal).
• glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #2162: row 24 (glm). In the later parts: minimax_m3 (CUDA): draft several streams' DSpark blocks in one pass #1426 (row 19) and DeepSeek-V4.1: read the pack's weight spans past the page cache (core/direct_io) #1924 (row 22) in the second, lfm2: DSpark drafting with owned context and confidence scheduling #1394 (row 17) and lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503 (rows 2 and 3) in the third, qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 (row 25) in the fourth, kimi: K3 and K3.1 through the native server on Metal #2221 (row 17) in the fifth.

Checks on this head

• Head commit: 93c8a2be8.
• Lean checker: 425 problems on this head, 0 new. git diff --check clean. No line over 120 columns.
• Every link (117) was checked against the repository's PR and issue lists; every path in the intro exists on main except the two marked as not yet there.
• Not run, and why: no build or tests; the page cannot reach inference.

Receipt

n/a: documentation only. Corrections welcome, cell by cell.

[ashhart]

Thanks @jschmied, a page that says which family has which optimization, with every cell linked, is something we've needed. The links and switch names check out against main. A few corrections:

1. The prompt_tail.zig bullet names your branch; please drop the parenthesis, so it ends "it is on [qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229]".
2. Row 8, glm: main's Metal server already keeps GLM-5.3-Flash's conversation states on disk (native/metal.zig lists it with Qwen3.8-27B, Nemotron and Flash Next), so that cell is done M¹ open C[#1411]. Nemotron's cell should gain M¹ for the same reason.
3. Rows 18, 23 and 27 cite Flash Next on CUDA: copy drafts where a reply repeats earlier text #468, Flash Next on CUDA: drop the checkpoint from the page cache after load (fixes #1868) #1880, CUDA: Qwen4 and tensor-parallel loaders drop the checkpoint's pages after every rank is ready #1887 and fix(profile): report Nemotron CUDA paging overrides #2061 as done, but none of them has a CHANGELOG line, and none is open. Could you check each one landed? If not, done¹ or the PR that did it.
4. GLM CUDA: opt-in context copies priced by the lane host #2146 has landed, and CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033 and qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229 are queued, so rows 1, 18 and 21 will flip soon. A note in the PR helps whoever lands them update the page.

[jschmied]

Thanks for checking the cells. All four are in 93c8a2be8:

1. The prompt_tail.zig bullet now ends "it is on [qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229]".
2. Row 8: glm is done M¹ open C[#1411], and nemotron gained M¹.
3. All four landed, so the cells stay done with their PRs: Flash Next on CUDA: copy drafts where a reply repeats earlier text #468 (10-07), Flash Next on CUDA: drop the checkpoint from the page cache after load (fixes #1868) #1880 (11-01), CUDA: Qwen4 and tensor-parallel loaders drop the checkpoint's pages after every rank is ready #1887 (11-04) and fix(profile): report Nemotron CUDA paging overrides #2061 (11-04). None of them added a CHANGELOG line.
4. GLM CUDA: opt-in context copies priced by the lane host #2146 has landed, so row 18's glm cell is done [#2146]. The PR description now lists every open PR the page cites, with its rows, so whoever lands one knows which cells to flip.

To keep the page current, I'd propose one pre-flight line in docs/pr/SKILL.md: "If this PR adds, finishes or drops an optimization for a family, update that family's cell in docs/optimization-ledger.md." Should I add it in this PR, or in its own? Until then I'll flip the cells myself as their PRs land.

[ashhart]

Thanks @jschmied, all four are in, and listing every open PR the page cites with its rows is a good way to keep it current. One small thing: "[#2211] moves GLM onto the shared store" reads as done; please mark it "(open)" like the others. On the docs/pr/SKILL.md line: yes please, but in its own small PR, since it changes contributor instructions rather than the ledger. With that, this is ready to land.

[jschmied]

Done in 5e8895e26: #2211 now reads "(open)". The docs/pr/SKILL.md line is its own PR, #2243. It links the ledger page, so it should land after this one.

[ashhart]

Thanks @jschmied. Queued for after the 1.2.3 cut, with #2243 after it.

[jschmied]

One small correction before it lands, in 2bd212bab: row 3's Flash Next cell now reads yes (host store), and the prompt-store paragraph says why. Flash Next already resumes a prefix from the host store's whole-state copies, so the shared store would save memory (about 0.9 GiB of attention rows per kept 32k state, 27 KiB a token per cuda_snap.zig's test), not add resuming. Docs only; nothing else changed.