[mrv777]

What this changes

Before: every host's rows store stages disk writes under a fixed 1 GiB permit, so a Qwen3.8-27B save on a Mac with more than about 13K new tokens (about 65 KB a token) is skipped with StagingFull. A long conversation that has no shorter state on disk is then never written. After: the store sizes the permit to one whole state at the host's longest context, with none of its rows on disk yet, computed from the family's own image layout. The permit never drops below the 1 GiB floor and never exceeds the RAM prompt cache's budget. The 27B host passes its --context; other hosts keep 1 GiB.

• Issue: Fixes Qwen3.8-27B on Metal: kept-state saves with more than ~13K new tokens are skipped with StagingFull (fixed 1 GiB staging permit) #2382
• Stacked on: main (c3f64ca8a); one commit.
• atc overlap --json: run at 2026-11-06T18:41Z against main c3f64ca8a. 5 clean, 5 conflicts, 84 unknown (the report hit its pull request and file limits). All five conflicts (qwen3_5 cuda: readout streams fill in pieces of final-norm rows (#2324 step 2) #2384, qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383, CUDA: #1764's BF16 prompt tile for Qwen ModelOpt, Nemotron and Dory, plus two exact Qwen prompt kernels (#1762) #2008, fix(hip): zero masked V per row in the wide prompt tile #2357, moss image: serve pictures on CUDA (encoder, decoder and host for #1573) #2281) are inherited: git merge-tree shows each one conflicts with main in the same files without this branch, mostly CHANGELOG.md, and fix(hip): zero masked V per row in the wide prompt tile #2357 also in hip_records_families.zig. cuda disk: a family's host staging permit for kept states #2337 (the CUDA family permit) merges cleanly with this head and touches different hosts.

How: rows.Options.longest (default 0, so nothing else changes). write.coldCharge(r, at) bounds a save of at tokens: every whole block, the state's own bytes, metadata, trailers and the worker's bookkeeping. It uses the same charged helper that stageJob uses. attach sets staging_bytes = max(staging_bytes, min(coldCharge(longest), store.budget)) and falls back to the floor if the layout can't be priced, so startup cannot fail. Saves are still charged only their new bytes; the permit is only the ceiling.

Checks on this head

• Head commit: 5c87ad4d4
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 210 on base c3f64ca8a, 210 on head, 0 new.
• Tests run, with the command and pass, fail and skip counts:
  • zig build test test-golden -Dcpu=apple_m1 (Metal device checks on): 717/720 steps, 5054/5098 tests passed, 43 skipped, 1 failed (Zig's step cache skips unchanged test binaries; the uncached run on 8163ff926 was 6587/6657, 69 skipped, the same 1 failed). Goldens: 322/335 and 55/55 equal to server goldens, 0 unexpected differences. The one failure is exl3_load.test.a scan that fails an allocation frees every group prefix it held in test-cuda-host. It fails only in this checkout, which lives under a .claude/worktrees/ path. A copy of the same head in another directory passes test-cuda-host, and so does the base. The test doesn't reach the rows store.
  • zig build test-rows-host: pass. The new test is rows stage: a permit sized to the longest state writes a cold save the fixed permit skips. With the sizing reverted (planted control) it fails with expected 55060, found 4096; restored, it passes. The test also asserts the real staged charge stays within coldCharge.
  • Before the rebase, on 8163ff926 over b3933ecd3: test-rows-prefetch-host, test-learn-recovery, test-learn-warning, test-lane-api-host, dev-tools and zig build native -Dcpu=apple_m1 all passed. None of the 51 commits main gained since touch these files.
  • zig fmt --check build.zig zig and git diff --check: clean.
• CI: pending on this head.
• Not run, and why: CUDA and HIP hosts (no hardware here; test-cuda-host runs as part of zig build test above); Metal chips other than M5 Ultra.

Receipt

• Environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: base b3933ecd3, head 8163ff926 (the same change before the rebase; 5c87ad4d4 differs only in a CHANGELOG merge). Zig 0.17.0, Apple metal 32023.922, Apple M5 Ultra, 96 GB, macOS 27.2. Checkpoint: a local Qwen3.8-27B MLX 4-bit (config.json sha256 f238d3a11e8d96b4…, model.safetensors.index.json 13b840162b4cb35c…). Drafter: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c4cde6860d4eee73e2547cd786fe8e8a4. Server flags: --drafter … --parallel 3 --context 81920 --compact-at 0.9 --max-tokens 16384 --prompt-cache-gib 12 --prompt-cache-disk DIR --loop-guard --temperature 0.6.

• Paths and platforms the change reaches, which I ran and which I could not: only the Metal Qwen3.8-27B disk tier (qwen27_disk.zig is the one caller that passes longest). Every other rows host keeps longest = 0 and the 1 GiB floor. Ran: Metal, M5 Ultra. Not run: other Metal chips.

• The fix, on a cold 49,160-token prompt built from this repo's docs/*.md (fresh scratch disk dir, same flags):

  • base: imprint: saving 47104 tokens skipped (StagingFull), and the same for the 49153- and 49163-token states.
  • head: imprint: 47104 tokens on disk: 192 new blocks and its own 146 MiB (3191 MiB this state …) in 2290 ms. The 49153 and 49163 states follow, charged only their 15 and 0 new blocks.
  • head, server restarted on the same disk dir with the same prompt sent: imprint: read 49153 tokens from disk (3319 MiB in 639 ms), and the request took 0.87 s instead of 39.8 s cold.
  • Peak phys_footprint (from footprint -f bytes) over the cold run: base 49.80 GB, head 49.97 GB (+0.16 GiB). At --context 81920 the permit is about 5.4 GiB, but the staged copy lands under the prefill's own peak.

• Exactness, from tools/bench_concurrent.py --alone --serial: --tokens 256 --reps 3 --temperatures 0 --levels 1,3 --alone --serial (3 is the widest admitted at --parallel 3). Base and head: every cell alone equal 3/3 or 9/9 and serial equal 1/1, 0 unequal, 0 failed. The token_sha256 sets match between base and head.

• Main against this branch, same machine and checkpoint, posted before asking for a run on our machines (or why you can't): above, base b3933ecd3 against head 8163ff926, alternating base and head servers on the same machine.

• Decode speed before and after, from tools/bench_openai.py: --tokens 256 --reps 3 --temperatures 0, two alternating rounds. Medians, tok/s:

  prompt	base round 1	base round 2	head round 1	head round 2
  fibonacci-raw	227.7	227.7	226.6	228.1
  gpu-chat-no-think	161.1	160.8	160.3	161.2

  At 3 streams (bench_concurrent), aggregate tok/s: code 337.5 base vs 336.0 head; chat 289.9 base vs 288.4 head. All within run-to-run noise; the change touches no decode path.

• Prompt speed before and after: n/a for bench_cache.py, because staging runs after the reply and doesn't reach prompt processing. The cold 49,160-token request above took 39.34 s on base and 39.67 s on head (end-to-end, including 3 reply tokens).

• Licence of any new weights or drafter, or "n/a": n/a

Checklist

• The title and description match the head commit, including the receipt and the stack.

• New tests fail before the change, pass after it, and skip cleanly without their dependency.

• Comments and docstrings are one line. No measurements or history in the source.

• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.

• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Drafted equals "draft": false and concurrent equals solo: checked above. Not checked: a token comparison of the disk-resumed reply against a fresh one. The change only decides whether a save is staged, not what is written or read.)

• No precision traded for speed. If the bits change, the description says which sums change and why.

• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled (Checked: greedy, with --drafter, at 1 and 3 streams. Not run: sampled, and without --drafter.)

[ashhart]

Thanks @mrv777, this covers what we asked on #2382:

• One shared mechanism. rows.Options.longest lives in the core rows store, so cuda disk: a family's host staging permit for kept states #2337's CUDA permit can take the same field when one of the two rebases.
• The bound. max(the 1 GiB floor, min(one cold state, the RAM store's budget)). Every host that passes no longest keeps exactly 1 GiB, and only qwen27_disk.zig passes one.
• Memory. Peak footprint is up 0.16 GiB on the cold run, since the staged copy lands under the prefill's own peak.
• The evidence. Saves that main skips are written (47,104 tokens in 2.3 s), a restart restores 49,153 tokens in 0.64 s (0.87 s for the request, against 39.8 s cold), --alone --serial is equal, and decode is level.
• The planted control that reverts the sizing fails as it should.

It's in the landing queue. Your line is in its own hunk, so we'll keep it there.

[ashhart]

Thank you @mrv777. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thank you @mrv777. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

@mrv777, this was riding tonight's landing train and passed its CI, but GitHub won't merge it while it's marked as a draft. If the draft is on purpose, no problem: we'll leave it until you mark it ready. Since today's release freezes at 08:00 UTC, it would then be in the next one.