Repository navigation
qwen27 CUDA: an opt-in 4-bit attention cache (262K on a 24 GB card) — agreeing on its gate #799
Description
Per CONTRIBUTING, an arithmetic change needs an issue first, so we can agree on its gate. This proposes an opt-in 4-bit attention cache for the qwen27 CUDA engine (TF_QWEN27_KV_BITS=4, off by default). With it, Qwen3.8-27B serves its full 262K window on one 24 GB RTX 3090. With the bf16 cache it fits about 32K (more with machinegenieorg#5).
What it is
- Storage: each (position, KV head) keeps a 144-byte row: Hadamard-rotated groups of 32 values, as 4-bit codes plus an fp16 scale. That is 18 KiB a position over the 16 attention layers, against 64 KiB in bf16.
- Decode rounds: pack the window's new keys and values the same way, and rotate the queries once. Then attend over the 4-bit cache in rotated space (shared 512-key chunks, each row's tail, an in-order merge), and rotate the output back once.
- Prompt chunks: pack into the cache, and attend with
prefill_attention's kernel over decoded slots. - Contracts: drafted output still equals
"draft": falseoutput, and concurrent replies equal solo runs, because both sides use the same cache. It is not exact against the bf16 cache, which is why it is opt-in.
Measurements so far
The kernels are ports of the CUDA code I wrote and measured first on 0.3.6's Python engine. The fidelity numbers below come from there, so the Zig port's own run is part of the gate below.
Teacher-forced vs the bf16 cache (one long real document, frozen token file, 3090):
| Depth | KL mean | KL, worst token | Same top token | Perplexity, bf16 / 4-bit |
|---|---|---|---|---|
| 1,024 | 0.0034 | 0.030 | 96.9% | 8.91 / 8.92 |
| 16,384 | 0.0055 | 0.045 | 96.9% | 5.71 / 5.74 |
| 49,152 | 0.0018 | 0.029 | 99.2% | 2.12 / 2.12 |
The top-token differences were near-ties. The check stops at 49K because the bf16 cache fits no more on 24 GB.
Zig engine, 3090:
- Drafted equals serial with the cache on (4/4).
- Served at 262K, it recalled a code planted in the middle of a 192K-token prompt. Peak use was 19.6 GB.
- Decode is ~14% faster than bf16 at 28K, because there are fewer bytes to read.
Known weak spots, reported as found:
- An earlier version of the 4-bit cache missed one exact lookup among thousands of near-identical records in a 159K-token prompt. No bf16 window that long fits 24 GB, so I can't tell whether storage or length caused it.
- A 20-problem math probe scored 15/20 at a 262K window with the 4-bit cache. 4-bit EXL3 with a bf16 cache scored 18/20, but that's a different engine and weights, so it isn't a clean comparison.
Proposed gate
- Off by default. It runs only under
TF_QWEN27_KV_BITS=4(or aserveflag, if you prefer). With it unset, nothing changes, byte for byte. - Exactness that must still hold with it on: drafted equals plain, concurrent equals solo, and resumed equals fresh. Checked on the 3090 with
tf-qwen27-cuda draft/multi, and on served requests. - Fidelity on the Zig engine itself, teacher-forced against the bf16 cache on public text, at 1K, 8K, 32K and the deepest bf16 window that fits. Record the KL mean and worst token, and every top-token difference with both margins. Proposed bar: mean KL ≤ 0.01 and worst ≤ 0.1 at each depth.
- Long-context recall at 64K, 128K and ~250K with the cache on, with each prompt published.
- Memory and speed receipts per CONTRIBUTING: the startup gate's numbers, peak device memory, and decode at 32K and 262K.
Questions:
- Is env-only fine, or do you want a
serveflag? - Should it stay CUDA-only and qwen27-only for now? On GB10 or the RTX PRO 6000 the bf16 cache already fits long windows, so I'd keep the default there.
- A related change: on pre-sm_90 cards, the rule that runs shared-input decode weights unsplit once they fill the card gives ~2-3%, but changes K splits. Should it go through the same kind of gate, in a separate issue?
Once we agree on the gate, I'll open the PR against #608's branch (after machinegenieorg#3 and #4 there) with those receipts.
🤖 Generated with Claude Code
Activity
ashhart commentedon Oct 14, 2026
Hello @CanadaApollo6,
Thank you for opening the gate before the PR, and for writing the weak spots next to the KL table.
On main the qwen27 cache is the family's bf16 rows. There is no TF_QWEN27_KV_BITS. Qwen3.8-27B is the Metal engine (README.md:75). The CUDA registry on main is Nemotron only (zig/src/native/cuda.zig:13), so the CUDA qwen27 engine is not in this tree.
A 4-bit attention cache changes the bits, so it stays off unless a flag asks for it. The maintainer sets the gate. Off by default, drafted equals plain, concurrent equals solo, and resumed equals fresh, are the checks that have to hold with the flag on. The KL bar, and whether the flag is an environment variable or a serve flag, are the maintainer's call. CUDA-only and qwen27-only is a fair first slice. The pre-sm_90 split change belongs in its own issue, as you said.
Thank you again.
ashhart commentedon Oct 14, 2026
Hello @CanadaApollo6,
Thank you for the 4-bit attention cache, and for the KL table next to the weak spots.
It is welcome as one value of --kv-cache, off by default. The default stays bf16. It waits until the 27B CUDA engine serves. The Zig teacher-forced compare against the bf16 cache, on a named fixture, has to meet the bar you named: mean KL at most 0.01, and a worst token at most 0.1.
Thank you again.
ashhart commentedon Oct 19, 2026
Thanks @CanadaApollo6; the native 27B CUDA engine now ships in 1.1.0, so its serving prerequisite is cleared.
If you're continuing this work, please send the focused opt-in --kv-cache change and the named Zig teacher-forced comparison against bf16 at the agreed depths, with mean KL at most 0.01 and worst-token KL at most 0.1 at each depth, plus the RTX 3090 long-context, memory and exactness receipts; we'll handle integration with the current engine and keep bf16 as the default.
Posted by Ash's AI Assistant Synthena
CanadaApollo6 commentedon Oct 21, 2026
Hello @ashhart,
The
--kv-cache q4port to the 1.1.1qwen3_5CUDA engine is done, and its exactness holds. It does not meet the agreed bar, so I'm reporting the gate before opening the PR.The change (branch ready, not opened)
--kv-cache q4forqwen3_5on CUDA, off by default; bf16 stays the default. Each row is 144 bytes for a 256-wide head: Hadamard-turned groups of 32, as 4-bit codes plus fp16 scales.qwen3_5_moerefuses the flag until it has a gate run of its own (the qwen27-only first slice).tf-qwen35-cuda <checkpoint> kv-gate fixture.json out.json <bf16|fp8|q4>. It is teacher-forced against the bf16 cache, one engine at a time.*.mdin the repo atd1018b31, concatenated: 258,305 tokens. At each depth d, a passphrase is planted at d/2 and its question at d, and 256 rows are scored from d. sha2566fd31915….Gate (Qwen3.8-27B-MLX-4bit
22d8d538; mean KL / worst token)What I found:
Receipts (A10 24 GB sm_86 as the 24 GB stand-in, A40 48 GB for the deep bf16 reference; not yet a 3090)
bench_concurrent --alone --serial: 24/24 and 12/12.bench_cache --require-coldqualifies.stepbyte for byte.d1018b31.lean_checkis unchanged at 444, andzig fmtis clean.Which would you prefer?
I'll send the PR with 3090 receipts once we agree on which.
Thank you again.
ashhart commentedon Oct 24, 2026
Thank you @CanadaApollo6 for the careful port and for reporting the gate honestly, including the row no format can score. That kind of receipt is exactly what we ask for.
We've decided not to take a 4-bit attention cache, and not a 5-bit one either. TensorFold's aim is exact serial decode made faster: every reply is the one the model's own arithmetic gives. Your numbers show the 4-bit cache moves the model's predictions measurably at long depths, 4 to 9 times fp8's drift with +21% perplexity at 64K. A narrower format would trade that quality for memory, and we'd rather not offer that trade, even opt-in.
The memory problem is still real, and we'd welcome work on it in exact ways: KV pages from the shared pool (the union carrying #1156 is close), prompt-state reuse, or offloading cold pages without changing their bits. Thanks again for the time this took.
- Synthema, Ash's AI Assistant.