[jrb-ai]

What this changes

On CUDA, the Qwen3.5 family keeps a verified window with tf_q35k_apply, which reads and writes every linear layer's recurrent state once a stream every round. This change leaves the kept rows pending and lets the next round's slot scan, which loads that state anyway, apply them. Every output keeps its bits. TF_QWEN_KEEP_LATE=0 keeps today's keep.

• keep() copies each kept row's DeltaNet heads (decay, write, key) from the window's log into the cache's own pending buffer, in path order, so a tree's path replays as a chain. The conv tails and the attention compaction run as before.
• Every recorded slot scan replays the pending rows into the state it loads, with tf_q35k_apply's fp32 steps in the same order, and writes the state back only at a one-row window or once 24 rows are pending.
• A prompt pass, a prefix-cache save or put, and a kept copy (saveCache) land the pending rows first with one new kernel, tf_q35k_apply_pend (tf_q35k_apply's loop over the pending buffer). A restore (restoreCache) drops the rows pending in the state it replaces.
• The buffer belongs to the cache (48 rows, 66.5 MiB a stream for the 27B, counted in streamBytes), so pending rows never wait in shared recurrent scratch between rounds.
• Seg gains the pending count and buffer (pn, plog); the four contract tests that mirror Seg follow.

This is the Qwen3.5-family counterpart of #1763, which keeps Flash Next's pending DeltaNet rows in a per-stream log.

Receipt

• Environment: TensorFold main 72cc3ea89 and this change (one commit; since rebased onto main a8c02eef, whose 34 commits touch none of its files); DGX Spark (GB10, 128 GB), Ubuntu 24.04 aarch64, kernel 7.0.0-1019-nvidia, driver 580.178.04; both built in nvcr.io/nvidia/pytorch:26.07-py3 (nvcc 13.3.73) with Zig 0.17.0, -Doptimize=fast; Vontra/Qwen3.8-27B-MLX-4bit at 70ae7fac with z-lab/Qwen3.8-27B-DFlash2 at 50307d4c (default drafter bits); serve --context 16384 --parallel 16 --no-thinking, prompt cache at its default; a fresh server for every run, in the order main, this change, this change, main. The receipts marked "after review" ran on this branch's head (a8c02eef and four commits), built the same way.

• Exactness: bench_concurrent --levels 1,4,16 --alone --serial, greedy and seeded sampled, code and chat: every reply equals its solo run and "draft": false in all four runs (1,008 solo and 408 serial comparisons, none unequal). Every completion of every benchmark here (342 a run) has the same token_sha on main and this change. The served receipts ran this change before its contract records and the four test mirrors were added; those leave every CUDA image byte-identical and no engine source changed. bench_cache --require-cold on the 3,001- and 12,001-token prompts of zig/tests/cuda/flashnext/lr7: repeated and concurrent requests equal the first on both.

• Resumed prompts (after review): this family keeps a prompt state only at a multiple of 4,096 rows inside a prompt, and the host store plans none for a raw /v1/completions prompt, so the 33- to 3,001-token prompts tried first never resumed. A first chat turn of 9,452 tokens and a second of 9,540 (the first, its 64-token reply, a follow-up), each sent twice, greedy and then seeded sampled, ran on six servers: the host store and the Prefix store (TF_QWEN_KV_PAGES=256 TF_QWEN_KV_SHARE=64), each with TF_QWEN_KEEP_LATE unset, =0, and no prompt cache (--prompt-cache-gib 0). Where a store was on, every send after the first resumed at 8,192 tokens; each request (two turns, greedy and sampled) has one token_sha256 across all six servers and both of its sends. On the Prefix store, bench_cache --require-cold --require-reuse on the 12,001-token prompt qualifies with the switch unset and =0 (repeated and concurrent requests resume at 8,192), and its replies' hash equals a server's without a store.

• Rows pending at a kept copy and a restore (after review): served traffic has none at either, because a copy is taken at a mark inside a prompt pass, which lands them first, and a restore starts a request on a cache just reset. So tf-qwen35-cuda tree takes TF_Q35_RESUME_TREE=J: a kept copy after window J, restored at the end into the same cache with rows pending again, and the windows after J run again. With J = 3, 16-row tree windows and 160 tokens, the copy lands 24 pending rows and the 136 rows after it repeat their logits byte for byte, with the switch unset and =0; all four runs' logits (with and without the copy, both settings) are equal. With this change's two cuda_reuse.zig lines reverted it fails (ResumeMismatch).

• Contract case (after review): tf-cuda-test qwen-pending (zig/tests/cuda/contract_qwen_pending.zig, also run by kernel-contract) on GB10. Each slot scan's replay, written back and not, and tf_q35k_apply_pend, bit-equal to tf_q35k_apply and then the scan: 30 cases (a decode row, a 4-row chain, two 16-row trees and a 64-row chain at 1, 7, 8, 9, 24 and 48 pending rows, on every scan its slot peak allows) and the six counts of tf_q35k_apply_pend, ending PASS qwen35 pending: controls RED as planted (a dropped pending row, two swapped) on every scan and tf_q35k_apply_pend.

• Decode speed, main -> this change, tok/s, mean of the two alternated rounds:

  	greedy	seeded sampled
  bench_openai code / chat (256 tokens, 3 reps)	68.1 -> 69.7 (+2.3 %) / 51.5 -> 52.4 (+1.8 %)
  bench_concurrent code x1 / x4 / x16	69.0 -> 69.7 (+1.1 %) / 186.6 -> 193.2 (+3.5 %) / 303.8 -> 307.6 (+1.3 %)	77.3 -> 78.0 (+0.9 %) / 199.2 -> 203.9 (+2.4 %) / 260.5 -> 276.2 (+6.0 %)
  bench_concurrent chat x1 / x4 / x16	51.8 -> 52.4 (+1.1 %) / 143.6 -> 146.6 (+2.1 %) / 221.2 -> 238.4 (+7.8 %)	50.0 -> 50.5 (+1.1 %) / 116.4 -> 119.9 (+3.0 %) / 216.3 -> 231.5 (+7.0 %)

  The round gap (p50, chat greedy) at x1 / x4 / x16: 81.5 -> 80.6 / 111.0 -> 107.2 / 171.6 -> 156.2 ms. The two rounds of each binary agree within 0.5 % at x1.

• Prompt speed: the cold first request's ttft_s (bench_cache --require-cold) at 3,001 / 12,001 tokens: main 2.27, 2.24 / 9.79, 9.02 s; this change 2.21, 2.23 / 9.01, 9.02 s. A prompt pass does more work only when rows are pending (a later turn in the same stream), and then lands them in one launch.

• Memory: 1.90 -> 1.96 GiB a stream at a 16,384-token window (the pending buffer); 16 streams admitted either way.

• Tests: zig build test-cuda-host on main: 4,174 of 4,186 pass, 10 skipped, 2 failed (both drama_voice_tool cleanup tests, directory permissions in this container); on this change the same two fail and nothing else (pending_132_test's fixture now reads the one-slot scan's new 13.0 register count). test-kernel-contract and test-delta-slots-contract-host: 21 / 21 steps, 126 / 126 tests. zig fmt --check clean on every changed file. tools/zig/lean_check.py: 424 problems, the same as main. After review, on this branch's head: 4,120 of 4,132 pass, 10 skipped, and the same two drama_voice_tool tests fail; zig fmt --check clean; lean_check.py 424, none in a changed file.

• Kernel contract (GB10, nvcc 13.3.73), zig build kernel-contract on this change: SUMMARY 527 kernels: 527 pass gates 1-4, 0 fail, 6 known findings; 344 controls: 344 red as planted, 0 missed; 229 limit problems. The four contract tests that mirror Seg take its two new fields; without them the paged-attention runs lost the context. That run predates the review changes; the new case ran on its own (above), and no kernel has changed since.

• Kernel limits: tf_q35k_apply_pend registers as a pending lane launch beside tf_q35k_apply; the five tf_q35_gdn_slots_* take a few more registers (64 -> 77 at sm_80 for one slot, for example). kernel-contract-limits on five-SM fatbins (sm_80, 86, 89, 120, 121) with nvcc 13.3.73: every changed launch's record current and the same 274 inherited problems as main, launch for launch (the records of tf_q35_attn_partial8, _paged and _legacy, whose registers moved with Seg, refreshed). With nvcc 13.0.88: the new launch's and the slot scans' base records current, and 197 problems on both main and this change, the same launches (the base records of those three attention launches were already stale on main; their counts move by a few registers and are left for a refresh). Not re-derived: the 13.2 records of the slot scans (measurements/qwen_lanes_132.zig), with no 13.2 compiler here; tf_q35k_apply_pend stays pending under 13.2.

• Not run: Metal and HIP (CUDA-only change; their DeltaNet keeps are untouched); CUDA chips other than GB10 (launch records only); tensor parallelism (one GB10); the family's other checkpoints and MoE members (NVFP4 27B, 35B-A3B), which take the same keep.

Credit

This change carries code, not only the approach, from fkiene's submission to Layr Labs' cuda.fast Qwen3.8-27B challenge (Layr-Labs/cudafast-qwen38-27b-tf-engine d7c540f, the challenge's validation commit, co-authored by fkiene), which deferred a kept chain's replay into the next scan through a second log half. From it, nearly as written: Seg's pn and plog, the scan's before write after the replay and the write-back block at its end, the flush and fold call sites in the forward, and the flush before a reuse save; rewritten on the same shape: Cache.pend with flush and fold, and the scan's replay loop. That commit changes TensorFold files vendored in that repository; its LICENSE puts the harness, benchmarks, tests and docs under MIT and leaves the vendored engine under its own licence, which THIRD_PARTY_NOTICES names as Apache-2.0. This change keeps the rows in the cache's own buffer instead, so a tree's path folds the same way and pending rows outlive the recurrent scratch lease, and adds the replay without write-back, the batched loads, tf_q35k_apply_pend, the prefix-cache landings and TF_QWEN_KEEP_LATE. docs/credits.md names it.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. Every pending row takes tf_q35k_apply's fp32 steps in the same order; no sum changes.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (CUDA only.)
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (contract_qwen_pending.zig needs this change's kernel and replay; TF_Q35_RESUME_TREE fails with this change's two cuda_reuse.zig lines reverted. Both are GPU runs of tf-cuda-test and tf-qwen35-cuda, which have no skip path.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

If this lands, a credit to @JRB_AI on X would be appreciated: https://x.com/JRB_AI

Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.

[ashhart]

Thanks @jrb-ai. Folding the kept rows into the scan that loads the state anyway is a neat saving, and +7% to +8% at 16
streams with every token_sha unchanged is a good result. The replay matches tf_q35k_apply's steps and order. Since
this is on by default, we'd like the paths that read the state outside the scan covered before it lands:

1. A resumed prompt with rows pending. The new flushes before a prefix save or put and a reuse save or restore
   didn't run in your receipts, because no cached tokens showed. A multi-turn chat where the next turn resumes from
   a kept state, plus a reuse save and restore, with tokens equal between TF_QWEN_KEEP_LATE=0 and unset, would
   settle it. If the 27B keeps no states at those settings, tell us which settings you tried and we'll find one
   that does.
2. A contract case for the replay: the slot scan with pending rows, both when it writes them back and when it
   doesn't, and tf_q35k_apply_pend, each bit-equal to tf_q35k_apply followed by the scan, with a planted control
   (a dropped or swapped pending row) that fails as planted.
3. The credit: could you say whether code came from the cuda.fast repository or only the approach, and under
   which licence? If it's code, a line in docs/credits.md.

We'll run it on our GB10s with the NVFP4 27B and the 35B-A3B too, and refresh the nvcc 13.2 records ourselves.

[jrb-ai]

Hey Ash, yep, totally reasonable. Here are the details you requested.

1. A resumed prompt with rows pending. Our earlier prompts were too short to resume. This family keeps a prompt state only at a multiple of 4,096 rows inside the prompt, and the host store plans none for a raw /v1/completions prompt. We had sent 33-, 148- and 3,001-token prompts, at default settings and with TF_QWEN_KV_PAGES=32 (and =256) TF_QWEN_KV_PAGES_ALL=1 TF_QWEN_KV_SHARE=64.

The test now uses two chat turns:

• Turn 1: one user message of 9,452 tokens (the repository's docs and a question).
• Turn 2: turn 1, its 64-token reply and a follow-up (9,540 tokens).

Each turn is sent twice, greedy and then seeded sampled, drafts on, on six servers of this branch:

	host store (default)	Prefix store (TF_QWEN_KV_PAGES=256 TF_QWEN_KV_SHARE=64)
TF_QWEN_KEEP_LATE unset	every send after the first resumes at 8,192	the same
TF_QWEN_KEEP_LATE=0	the same	the same
--prompt-cache-gib 0	every send cold	the same

Each request has one token_sha256 across all six servers and both of its sends: greedy turn 1 0be1aa219acb… and turn 2 314e95871d14…; sampled 019a9ce685a6… and a36a9be8da16…. On the Prefix store, bench_cache --require-cold --require-reuse on the 12,001-token prompt also qualifies with the switch unset and =0. Its repeated and concurrent requests resume at 8,192, with one output hash, the same as on a server without a store.

Rows cannot be pending at a served save or restore, though:

• A copy is only taken at a mark inside a prompt pass, and every pass that is not a recorded slot scan lands pending rows first (cuda_engine.zig).
• A restore only starts a request, on a cache that take() has just reset or created (cuda_prefill.zig opening).
• The engine keeps no state at the end of a reply.

So the turns above show that resumes stay exact with the late keep on, but they cannot put rows pending at those points. To cover that directly, tf-qwen35-cuda tree gets TF_Q35_RESUME_TREE=J. It takes a kept copy after window J, restores the copy at the end into the same cache with rows pending again, and reruns the windows after J.

With J = 3, 16-row tree windows and 160 tokens of the 3,001-token lr7 prompt, the copy lands 24 pending rows. The 136 rows after it repeat their logits byte for byte, with the switch unset and =0. All four runs' logits are equal (with and without the copy, both settings).

That check found one gap. restoreCache, the restore the served path uses, kept the pending count of the state it replaced; only the vtable restore, which this family never calls, flushed. Now saveCache lands pending rows itself and restoreCache drops them, so the two vtable wrappers lose their flushes. With those two lines reverted, the check fails (ResumeMismatch).

2. A contract case for the replay. zig/tests/cuda/contract_qwen_pending.zig runs in kernel-contract and alone as tf-cuda-test qwen-pending.

• Fixture: the 27B's linear widths, two layers, and two streams in one launch.
• Scans: all five slot scans, on a decode row, a 4-row chain, two 16-row trees and a 64-row chain.
• Pending counts: 1, 7, 8, 9, 24 and 48 rows (around the 8-row load batches, pend_commit and pend_cap), written back and not.
• Each case is bit-equal to tf_q35k_apply and then the scan: rows out, the window's log, the pre-round state, and the state (the reference's after a write-back, unchanged otherwise).
• tf_q35k_apply_pend is bit-equal to tf_q35k_apply on the same rows.
• Five repeats each, with inputs and guards checked.
• Controls: the last pending row dropped, and the last two swapped.

On GB10 it gives 37 PASS lines in about two minutes and ends:

PASS qwen35 pending: controls RED as planted (a dropped pending row, two swapped) on every scan and tf_q35k_apply_pend

It follows contract_qwen_pages.zig's form, so its controls are its own and the SUMMARY counts don't move.

3. The credit. It is code, not only the approach. From fkiene's d7c540f:

• Nearly as written: Seg's pn and plog; the scan's before write moved after the replay, and the write-back block at its end; the flush and fold call sites in the forward; the flush before a reuse save.
• Rewritten on the same shape: Cache.pend with flush and fold (here over the cache's own buffer), and the replay loop in the scan (here batched, with a mode that does not write back).

That commit changes TensorFold files vendored in that repository. Its LICENSE puts the harness, benchmarks, tests and docs under MIT and says the vendored engine keeps its own licence; THIRD_PARTY_NOTICES names that licence as Apache-2.0. The commit itself is the challenge's validation commit, co-authored by fkiene.

Ours: the cache-owned pending buffer (so a tree's path waits, not only a chain prefix), the replay without write-back, the batched loads, tf_q35k_apply_pend, the prefix-cache landings and TF_QWEN_KEEP_LATE. There's now a line in docs/credits.md, and the PR body's credit section says the same.

Thanks for running it on your GB10s with the NVFP4 27B and Qwen3.6-35B-A3B, and for refreshing the 13.2 records.

[ashhart]

Thank you @jrb-ai. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[jrb-ai]

Thank you @jrb-ai. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

Thanks Ash! Appreciate your review and support.

[ashhart]

@jrb-ai, a heads-up on #1824, which landed this morning. On a GB10, Qwen3.8-27B MLX 4-bit served chats fail on main: 8 of 64 greedy chat requests each time, in two separate runs, at prompts of 524 to 32,780 tokens. Each fails mid-reply with cuMemcpy2DAsync: CUDA_ERROR_INVALID_VALUE, then CudaFailed and HTTP 500. That copy is pendRows's in qwen3_5/cuda_cache.zig. With TF_QWEN_KEEP_LATE=0 nothing fails, and neither does v1.2.1. So #1958 makes the deferred keeps opt-in (=1) until the copy is fixed. Replies are the same either way. If you can find what makes that copy invalid (a pitch, width or height for some path shape, or a source past the log), we'd gladly turn it back on with a served chat run on our GB10s. The quality plan that reproduces it is ours, and we can share the request bodies.

[jrb-ai]

The cuMemcpy2DAsync: CUDA_ERROR_INVALID_VALUE in pendRows on the 27B MLX 4-bit came from kept paths longer than pend_cap. A verify window keeps up to 64 rows, but the pending buffer holds 48. After the flush, a path of 49 to 64 rows ran past pbuf. #1982 defers a path only when it fits, and lands a longer one at once. It also checks each pending copy against the log and the buffer before the driver sees it.

Served on a GB10 with Vontra/Qwen3.8-27B-MLX-4bit @ 70ae7fac and the DFlash2 drafter: 16 copy-heavy greedy chats at 504 to 33,597 prompt tokens, all at once.

Run	Failed	Exactness
main 5fe74ebb9 (deferred keeps on)	2 of 16, INVALID_VALUE, then CudaFailed	the 14 that completed equal #1982's replies
#1982	0 of 16	16 of 16 equal "draft": false; concurrent equals alone
#1982 with TF_QWEN_KEEP_LATE=0	0 of 16	16 of 16 equal #1982's replies

nvidia/Qwen3.8-27B-NVFP4 shows the same pattern. The full receipt is in #1982. With #1958 merged, deferred keeps stay opt-in; #1982 makes TF_QWEN_KEEP_LATE=1 safe to turn back on after your served run.

Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.

[ashhart]

Thanks @jrb-ai! This is in TensorFold 1.2.2, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.2