Repository navigation
TurboQuant: head_dim as a multiple of 128, three fixes, test coverage, fused kernel - #2
Merged
Pascal-SAPUI5 merged 18 commits intoAug 10, 2026
Merged
Conversation
github-actions
Bot
added
documentation
TurboQuant rejected any model whose head_dim was not exactly 128, which excluded Qwen3.5 (head_dim 256). The restriction was only ever in the guard. The whole data path already works in fixed 128-element chunks over a flat buffer and never looks at head boundaries: ggml-quants.c:612 for (offset = 0; offset < k; offset += TURBO_HEAD_DIM) set-rows.cu:444 GGML_ASSERT(ne00 % TURBO_HEAD_DIM_SR == 0) convert.cu:731 num_chunks = k / TURBO_HEAD_DIM_GPU A 256-dim head is therefore quantized as two independently normalized halves. Each normalized half is again a unit vector in R^128 — exactly the Beta((d-1)/2, (d-1)/2) distribution the Lloyd-Max codebooks were fitted for — so the existing codebooks stay optimal and need no change. The norm is already replicated per 32-element block, so the second half-norm costs no extra bytes: bit rate stays at 3.5 / 4.5 bpw. Chunk boundaries never straddle two heads because heads are contiguous in the row and head_dim is a multiple of the chunk size. Reconstruction quality is unchanged (10000 vectors, CPU reference): head_dim turbo3 MSE*d turbo4 MSE*d 128 0.0341 0.0093 256 0.0342 0.0094 384 0.0341 0.0094 Tests parametrized over head_dim 128/256/384, plus a chunk-independence test asserting that one 256-element pass packs byte-identically to two separate 128-element passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llama-bench on RX 9070 XT shows turbo lifting pp512 by ~53% over f16/f16 (3933 vs 2565 t/s) with token generation unchanged — the opposite of the ~6% seen at head_dim 128. Any turbo operand triggers it, so a different flash-attention kernel is likely being selected; not verified. KV cache at 131072 context measures 896 MiB with turbo3 across the 8 attention layers, against 4096 MiB computed for f16. Perplexity is recorded for f16 only (1.5258). The turbo runs hit an out-of-memory abort because another llama-server held the GPU; the table marks them as not yet measured rather than leaving them blank. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reruns with the GPU idle. Quality loss at head_dim 256 is smaller than at 128: turbo4 +0.002 and turbo3 +0.009 PPL, against +0.010 and +0.051 for Qwen3-14B. Two independently normalized halves per head track variance differences more closely than a single norm, and cost nothing extra because the norm is replicated per block either way. KV cache at 131072 context: f16 4096 MiB (14.81 GB total VRAM), turbo4 1152 MiB (11.75 GB), turbo3 896 MiB (11.47 GB). f16 leaves almost no room for the vision encoder on a 16 GB card. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TurboQuant had no automated GPU test at all. Three things were missing: 1. ggml_quantize_chunk had no TURBO case, so the default branch asserted and test-backend-ops could not even allocate a turbo tensor. Added quantize_turbo3_0/quantize_turbo4_0 row wrappers, which assert n_per_row % 128 == 0 — a row shorter than one FWHT chunk would let a chunk straddle two rows. 2. init_tensor_uniform quantized with n_per_row = blck_size = 32, splitting the 128-element chunk the FWHT operates on. Turbo now uses 128 there. 3. The CPU had no turbo flash-attention path: type_traits_cpu had no entry, so ggml_compute_forward_flash_attn_ext called a null vec_dot pointer and segfaulted while computing the reference. ggml_vec_dot_turbo3_0/4_0 already existed in ggml-quants.c and are correct (they do the inverse FWHT internally), they were just never registered. vec_dot_type is F32; no from_float, because quantize_row_turbo*_ref asserts on the chunk size that the generic per-block callers do not honor. With turbo3/turbo4 added to the FLASH_ATTN_EXT matrix (head_dim a multiple of 128, not restricted to 64/72 like the other quantized KV types), all 24 cases now fail with ERR ~1.0 and a GPU page fault: Memory access fault by GPU node-1 ... Reason: Page not present That is the reproducible symptom of turbo_pre_dequantize reading past its pool buffer on views that are not contiguously allocated — which is what the test graphs build. Fix follows separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three related defects, all in the CUDA/HIP path.
B1 — turbo_pre_dequantize dequantized the whole KV view linearly from
src->data while rescaling nb[3] proportionally. For a view that is not
contiguously allocated (nb[3] > nb[2]*ne[2], which holds whenever
n_kv < kv_size) that reads past the pool buffer. Symptom in
test-backend-ops: 24/24 failures plus 'Memory access fault by GPU node-1'.
It stayed invisible in practice only because the default n_parallel = 1
makes n_stream = 1, and then the view happens to be gap-free.
Removed the function entirely rather than adding a guard. The tile, MMA and
WMMA kernels all pass need_f16_K/V = true to launch_fattn, which already
converts K/V correctly via ggml_get_to_fp16_cuda (contiguous) or
ggml_get_to_fp16_nc_cuda (strided). Dropping the pre-pass also removes the
temporary mutation of dst->src[1]/[2] and lets get_best_fattn_kernel see the
real tensor types instead of substituted f16 ones.
B2 — ggml_get_to_{fp16,bf16,fp32}_nc_cuda mapped turbo to
dequantize_block_cuda<..., dequantize_turbo3_0>, which only does
codebook[idx] * norm: no inverse FWHT, no 1/sqrt(128). Since removing
turbo_pre_dequantize makes that path live, it now needs to be right. Added
dequantize_block_turbo_nc_kernel, which computes the source block index from
the row/slice strides and otherwise decodes exactly like the bulk kernel.
The shared decode is factored into turbo_decode_chunk_smem so the contiguous
and strided paths cannot drift apart — they must stay bit-identical for the
fused kernel work to be verifiable against them.
B3 — the vector kernel does not convert its inputs (need_f16_K is
type_K == F16), so it must never receive turbo. get_vec_dot_KQ mapped turbo
to vec_dot_fattn_vec_KQ_f16, reinterpreting 14-byte blocks as half2, and
dequantize_V_turbo3_0 dropped the FWHT. Unreachable so far, but any dispatch
slip would have produced silently wrong numbers. Removed both, along with
the turbo vector template instances (which this build config compiled), and
excluded turbo from can_use_vector_kernel — every VEC return depends on it.
A dispatch mistake is now a link error.
Verified on RX 9070 XT (gfx1201):
test-turboquant 23/23
test-backend-ops FLASH_ATTN_EXT type_KV=turbo3 640 ok, 0 fail
test-backend-ops FLASH_ATTN_EXT type_KV=turbo4 640 ok, 0 fail
test-backend-ops FLASH_ATTN_EXT permuted ok
test-backend-ops SET_ROWS ok
llama-server -np 4 --no-kv-unified: four concurrent requests, all coherent
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
L2 baseline after the non-contiguous fix. Perplexity is unchanged by that rework (f16 1.5258, turbo4 1.5281, turbo3 1.5343 — identical to before), confirming it was numerically neutral. The depth sweep exposes what the aggregate numbers hid: token generation degrades badly with context depth. At d65536 turbo3 does 31.5 t/s against f16's 46.9, and turbo3/turbo4 together drop to 24.5. The penalty scales with the number of converted caches, not which one is converted — the signature of a per-cache bulk conversion that runs once per FA call and therefore costs O(n_kv) per generated token. Prompt processing is unaffected (turbo is faster there) because one conversion amortizes over the whole batch. Groundwork for the fused kernel that removes this: turbo.cuh collects the shared TurboQuant primitives, and convert.cu now uses them instead of its own copies. New is turbo_decode_chunk_warp, which decodes a 128-element chunk in one warp — 4 registers per lane, two register-local butterfly stages and five cross-lane shuffles — with no shared memory and no block synchronization. Stage order and operand order match turbo_fwht_smem_128 exactly, so results must be bit-identical rather than merely close. A standalone HIP harness confirms that for turbo3 and turbo4 across 1/2/7/64 chunks, with reconstruction NMSE landing on the paper values (0.033-0.036 and 0.0086-0.0099). That equivalence is what makes the fused kernel verifiable against the established path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decodes turbo blocks inside flash attention instead of expanding the whole KV
cache to f16 per FA call. Covers the decode path (ncols <= 2), head_dim 128
and 256; everything else keeps using tile/MMA with launch_fattn conversion.
The naive approach — decode each KV token — would need O(n_kv) inverse FWHTs
and lose to the bulk conversion it replaces. Two properties of the Hadamard
matrix (symmetric, orthogonal) reduce that to O(1). With x = norm * s * H c:
K: <x_j, q> = norm_j * s * <c_j, H q> -> transform Q once at kernel start
V: sum p_j x_j = H(sum p_j norm_j s c_j) -> accumulate in the Hadamard
domain, transform once at the end
Valid for V because online-softmax rescaling is a scalar and H is linear.
Verified against a double-precision reference attention over explicitly
decoded K/V: NMSE ~1e-14 across n_kv 1/4/37/256.
Written as a separate kernel rather than a branch in fattn-vec.cuh. The
accumulator has to be float — codebook values are small and the final
transform amplifies rounding by sqrt(128) — and VKQ's half2 type is wired
through eight shared code sites there. A dedicated kernel cannot regress
f16/bf16/q4_0/q8_0, and drops the type branches turbo never needs.
Lane l owns elements [4l, 4l+4) of a chunk, which is exactly the layout
turbo_fwht_warp_128 expects, so both transforms run in registers with no
shared memory.
Three defects found by the test matrix during bring-up, all now fixed:
mask indexing used the wrong stride and double-counted the loop offset;
FATTN_KQ_MAX_OFFSET was missing, so a fully masked tile produced NaN; and the
dispatch did not require K->ne[1] % FATTN_KQ_STRIDE == 0, so kv=113 read past
the end of the cache.
test-backend-ops FLASH_ATTN_EXT: turbo3 640/640, turbo4 640/640.
GGML_CUDA_TURBO_FUSED_FA_OFF=1 restores the previous path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A/B in the same build, Qwen3.5-9B Q8_0 on gfx1201, tg128:
depth fused bulk f16
0 52.5 55.0 56.8
16384 35.7 42.5 54.1
65536 16.6 23.6 46.9
The kernel is correct — 640/640 on the FLASH_ATTN_EXT matrix for both turbo3
and turbo4, on either path — but it loses to the bulk conversion it was meant
to replace, by ~30% at depth 65536.
It trades memory bandwidth for per-token work: unpacking 3-bit indices across
byte boundaries plus four codebook lookups per lane per KV token per chunk. On
AMD a __constant__ array is an ordinary vector load rather than a broadcast
from a constant bank, so those lookups are real memory traffic. That likely
outweighs the bandwidth saved, but I have not verified it at ISA level.
Rather than tune on a hunch, the kernel is now opt-in via
GGML_CUDA_TURBO_FUSED_FA=1 and the measurement is written down. Two concrete
leads if this gets picked up again: hold the codebook lane-resident and read it
with __shfl_sync instead of from memory, and check whether the XOR-16 shuffle
stage lowers to ds_bpermute — it crosses the DPP16 row boundary on RDNA 4.
The default path is unchanged and still measures exactly as in the L2 baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first version reduced over all 32 lanes for every KV token — five shuffle
stages each, 160 per warp iteration. Splitting the two lane layouts fixes that:
the FWHT still needs 4 consecutive elements per lane, but the KQ dot does not.
Q is transformed in the 4-per-lane layout, then redistributed through shared
memory into 16 consecutive elements per lane, so 8 lanes cover one token, four
tokens are dotted at once and a reduction is 3 stages. At E=16 the turbo3 3-bit
indices also land on byte boundaries (16*3 = 6 bytes).
tg128, Qwen3.5-9B Q8_0, gfx1201:
depth KQ=32 KQ=8 bulk f16
0 52.5 53.2 55.0 56.8
16384 35.7 39.2 42.5 54.1
65536 16.6 20.4 23.6 46.9
23% better at depth 65536, still behind. The rest is structural, not a tuning
problem: for Q->ne[1] == 1 with gqa_opt_applies, f16 does not use the vector
kernel at all — it goes to the tile kernel, which handles all gqa_ratio query
heads per KV read. This kernel handles one head per block and reads K/V four
times as often on Qwen3.5. Per element and query head that is 0.4375 bytes
against tile+f16's 0.50, so the 4.57x compression is worth 12% of bandwidth,
which unpacking and codebook lookups more than spend.
Left disabled by default, with the analysis written down. The next step that
could actually win is a fused tile kernel (fattn-tile.cuh:663 already holds a
complete contiguous head_dim vector per V-tile row in shared memory), not
further micro-optimization here.
test-backend-ops FLASH_ATTN_EXT turbo3: 640/640 on both paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preparation for fusing TurboQuant into the tile kernel. The existing decoders assume either 128 threads (one per chunk element) or a full 32-lane warp with 4 registers each. The tile kernel has neither: at head_dim 256 on RDNA it runs with 64 threads. turbo_decode_chunk_smem_n strides over the chunk instead of fixing one element per thread, so it works for any thread count that divides or exceeds 128. The butterfly stages keep the same order and operand order as turbo_fwht_smem_128, which the test confirms: 128-thread, 32-lane-warp and 64/32-thread-strided decode all produce bit-identical output for turbo3 and turbo4 across 1/2/7/64 chunks. That equivalence matters because the tile kernel's V path will decode into the same shared-memory tile the f16 path fills, and any divergence between decoders would show up as a silent accuracy difference rather than a test failure. No behavior change: nothing calls the new function yet. test-backend-ops FLASH_ATTN_EXT turbo3 still 640/640. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second attempt at avoiding the bulk KV conversion, this time in the tile kernel rather than the vector kernel — the tile path is what f16 actually uses for decode with GQA, and its V tile already holds complete contiguous head_dim rows. type_V is threaded through flash_attn_tile -> _iter -> load_tile with a default of F16, so existing instantiations are untouched; four turbo instances are added for head_dim 128 and 256. flash_attn_tile_load_tile_turbo decodes a chunk per warp straight into the tile via turbo_decode_chunk_warp — no scratch shared memory, no extra barrier — and need_f16_V is set false so launch_fattn skips the conversion. Correct on the first try: test-backend-ops FLASH_ATTN_EXT 640/640 for turbo3 and turbo4. But it does not help throughput, measured at depth 65536: f16/turbo3 31.46 (bulk) -> 31.08 (fused V) turbo3/turbo3 23.58 (bulk) -> 23.54 (fused V) Traffic does drop from 2 to 0.4375 bytes per element. The kernel is not waiting on that: the original tile load is a plain copy, fully pipelined across all 64 threads, while decoding walks chunk by chunk through an FWHT whose five shuffle stages form a dependency chain. It is latency-bound, not bandwidth-bound. Two independent attempts now agree — the fused vector kernel and this — that TurboQuant decoding is too expensive to do inside attention. The bulk conversion amortizes it across the whole cache in a way an attention kernel cannot. Kept anyway: it removes the f16 staging buffer for V (~128 MB at 65536 context) at no throughput cost, and it is the groundwork any future attempt would need. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both attempts measured and written down: the fused vector kernel (20.4 t/s after optimization, against 23.6 for bulk conversion) and the fused tile V path (23.54 against 23.58). Neither beats expanding the cache to f16 once per FA call. The vector kernel loses on GQA amortization — it reads K/V once per query head where the tile kernel reads once per four. The tile V path removes that objection entirely and still gains nothing, which locates the cost in the decode itself: an FWHT with five dependent shuffle stages per chunk, walked chunk by chunk, against a plain fully-pipelined copy. Documented so the next person does not spend the same two attempts finding it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
KVarN (arXiv 2606.03458, llama.cpp #24139) argues that KV-cache quantization
error compounds during autoregressive decoding, which would make the +0.009
figure in this document — measured at context 2048 — an understatement.
Measured at 2048 and 8192 on Qwen3.5-9B Q8_0, the gap to f16 does not widen:
context f16 turbo4 delta turbo3 delta
2048 1.5258 1.5276 +0.0018 1.5338 +0.0080
8192 1.4107 1.4124 +0.0017 1.4172 +0.0065
For turbo3 it narrows slightly, absolutely and relatively (0.52% -> 0.46%).
The explanation is in the format: the per-chunk L2 norm is stored in fp16 and
never quantized, so only the direction of the normalized vector is approximated.
KVarN attributes the accumulating error to incorrect per-token magnitude
scaling, which cannot happen here. The per-chunk normalization that makes
TurboQuant expensive at runtime is what makes it robust over long contexts.
The 16384 row is incomplete — f16 finished at 1.3431, the turbo runs hit the
job's time limit. Two depths are a thin basis for a trend; they are enough to
rule out the failure mode, not to characterize the curve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
turbo3 at context 16384 measures 1.3497 against f16's 1.3431, a delta of +0.0066. With the earlier points that gives: ctx f16 turbo3 delta relative 2048 1.5258 1.5338 +0.0080 0.52% 8192 1.4107 1.4172 +0.0065 0.46% 16384 1.3431 1.3497 +0.0066 0.49% Flat after the first step, across an 8x range of context lengths. The previous commit called two depths a thin basis for the claim; three across that range make it solid. KVarN's accumulation failure mode does not apply to TurboQuant. turbo4 at 16384 was not measured — the job was stopped — but its delta sits at ~0.0017 at both measured depths, where the question barely matters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First version of the fused tile V path decoded each chunk as it was loaded, which meant one inverse FWHT per KV token — the same O(n_kv) cost the bulk conversion has, just relocated. It measured exactly as fast as what it replaced. The two good ideas were split across two kernels: the vector kernel accumulated in the Hadamard domain (O(1) transforms) but lost on GQA; the tile kernel had GQA but decoded eagerly. This combines them. The V tile is now filled with raw codebook values scaled by norm/sqrt(128), no rotation. Because H is linear and every scaling along the way is a scalar, sum_j p_j H c_j = H (sum_j p_j c_j): the accumulator stays in the Hadamard domain and one inverse FWHT per 128-element chunk at the end of the kernel brings it back. The transform is a dependency chain, the accumulation is not — which is why moving it out of the inner loop is what matters. The final transform goes through the KV_tmp scratch buffer rather than a third register-layout FWHT variant: VKQ is interleaved (lane x holds elements 2x, 2x+1, 64+2x, ...), and it runs once per launch, so the shared-memory round trip is irrelevant. tg128 at depth 65536, Qwen3.5-9B Q8_0 on gfx1201: config bulk eager decode Hadamard accumulation f16 / turbo3 31.46 31.08 37.07 (+18%) turbo3 / turbo3 23.58 23.54 26.62 (+13%) First measured gain from fusing anything into attention. The remaining gap to f16's 46.9 is the K side, which still goes through the bulk conversion. test-backend-ops FLASH_ATTN_EXT: turbo3 640/640, turbo4 640/640. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for fusing the K path, the remaining half of the bulk conversion. flash_attn_tile_load_tile_turbo_k fills a K tile with raw codebook values scaled by norm/sqrt(128). Unlike the V tile this loads a slice of each row — nbatch_K is 64 at the head sizes turbo supports, half a chunk. That is fine because the KQ dot is element-wise and never needs a whole chunk at once; it only requires Q to be in the Hadamard domain first, since <H c, q> = <c, H q>. A slice never straddles a chunk boundary, so the norm is one load per row. Threads iterate over half2 pairs rather than single elements: having two threads write the two halves of one slot would be a read-modify-write race. Not called yet — type_K still has to be threaded through flash_attn_tile_iter_KQ and the Q load has to apply the FWHT. Committed so the work is not lost; behavior is unchanged and test-backend-ops FLASH_ATTN_EXT turbo3 is still 640/640. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
type_K is now threaded through flash_attn_tile / _iter / _iter_KQ, the K tile can be filled with raw codebook values, and Q is transformed into the Hadamard domain once after loading (in float via the KV_tmp scratch — Q_tmp may be half, and rotating in half would cost accuracy). Correct for small batches, wrong for large ones: nb=1 160/160 ok nb=3 160/160 ok nb=32 132/160 ok nb=75 24/160 ok ERR ~0.042, not noise The failures track the batch size, i.e. the configurations with many Q columns per block, where cpw > 1 and nbatch_K varies (64, 128 and 256 all occur at these head sizes). Something in the Q transform or the slice indexing does not hold there; the per-element chunk derivation added for nbatch_K = 256 is the most recent suspect and has not been isolated yet. Left disabled rather than debugged further in this pass: the dispatch and the instances select <F16, TURBO*>, so K goes through launch_fattn's conversion as before and the V-side gain (+18% isolated, +13% combined) is kept. All 640 cases green again for both turbo types. The K-side code is retained because it is most of the work and the remaining bug is bounded — it only shows up in a known subset of configurations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the fused tile path. K now uses the same Hadamard-domain trick as V: Q is transformed once at kernel start and the KQ dot runs against raw codebook values, since <H c, q> = <c, H q> and H is symmetric. tg128 at depth 65536, Qwen3.5-9B Q8_0: bulk conversion 23.58 0.50x of f16 V fused 26.62 0.57x K and V fused 29.59 0.63x f16 46.90 1.00x +25% over the baseline. 0.63x is what llama.cpp discussion #20969 reports for quantized KV at 110K context, so this is the expected ceiling rather than a shortfall — quantized KV is slower than f16 at decode across methods and backends. Two bugs, both surfaced by the test matrix: The K tile loader assumed a slice narrower than one chunk. That holds for nbatch_K = 64 but not for 128 or 256, which the config table also selects at these head sizes; the chunk and its norm are now derived per element. Staging all of Q for the transform needed ncols*DKQ floats in KV_tmp, which is dimensioned by nbatch_fa*nbatch_K — and those shrink as ncols grows. At ncols=64 the buffer was short by 1.8x, which is why nb=1 passed and nb=75 failed. Q is now transformed one chunk at a time, needing 128 floats regardless of ncols. test-backend-ops FLASH_ATTN_EXT: turbo3 640/640, turbo4 640/640. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pascal-SAPUI5
force-pushed
the
feat/turboquant-head-dim-multiples-of-128
branch
from
August 10, 2026 18:43
c0e80c1 to
7e40072
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started from wanting to run Qwen3.5-9B (head_dim 256) with TurboQuant, which was rejected outright. Grew from there.
head_dim support
TurboQuant required head_dim to be exactly 128, which excluded Qwen3.5. The restriction lived only in the guard — the whole encode/decode path already works in fixed 128-element chunks over a flat buffer and never looks at head boundaries. A 256-dim head is now quantized as two independently normalized halves; each half is again a unit vector in R^128, exactly the distribution the Lloyd-Max codebooks were fitted for, so they stay optimal unchanged.
Quality is better at 256 than at 128 (+0.009 vs +0.051 PPL for turbo3), because two half-norms track variance differences within a head more closely than one — at no extra cost, since the norm is replicated per block either way.
Three fixes
turbo_pre_dequantizewalked the KV view linearly while scalingnb[3]proportionally, reading past the pool buffer for views that are not contiguously allocated. Invisible in practice only because the defaultn_parallel = 1makes the view gap-free;-np 4 --no-kv-unifiedwould have produced silent corruption. Removed the pre-pass entirely — tile/MMA/WMMA all request f16 andlaunch_fattnalready converts correctly.ggml_get_to_{fp16,bf16,fp32}_nc_cudamapped turbo to a decoder without the rotation or the 1/sqrt(128) factor. Unreachable before, live after removing the pre-pass, so it needed a real strided FWHT kernel.get_vec_dot_KQmapped turbo to the f16 dot, reinterpreting 14-byte blocks ashalf2. Unreachable, but any dispatch slip would have produced silently wrong numbers rather than a crash. Removed along with the dead turbo vector instances; turbo is now excluded fromcan_use_vector_kernel, so a mistake is a link error.Test coverage
The GPU side of TurboQuant had no automated test at all — three things blocked it simultaneously:
ggml_quantize_chunkhad no turbo case,init_tensor_uniformquantized with 32-element blocks (splitting the 128-element FWHT chunk), and the CPU had no turbo flash-attention entry, so the reference computation dereferenced a nullvec_dot.Now 640 cases per turbo type run through
FLASH_ATTN_EXT, plus the write path. That matrix found four bugs during the kernel work that would otherwise have surfaced in production.Fused kernel
Decodes turbo blocks inside flash attention instead of expanding the cache to f16 once per FA call. The naive form — decode each chunk as loaded — costs O(n_kv) transforms and measures exactly as fast as what it replaces. Working in the Hadamard domain reduces that to O(1):
tg128 at depth 65536, Qwen3.5-9B Q8_0, RX 9070 XT:
0.63x matches what llama.cpp discussion #20969 reports for quantized KV at 110K context — quantized KV is slower than f16 at decode across methods and backends, so this is the ceiling rather than a shortfall.
An earlier fused vector kernel is kept but disabled (
GGML_CUDA_TURBO_FUSED_FA=1): it loses on GQA amortization, reading K/V once per query head where the tile kernel reads once per four.Quality over context depth
KVarN (arXiv 2606.03458) argues KV quantization error compounds during decoding, which would make a short-context PPL number an understatement. Measured at three depths, the gap does not widen:
TurboQuant stores the per-chunk L2 norm in fp16 and never quantizes it, so the per-token magnitude error that KVarN describes cannot occur.
Note on history
This branch was built on a local
masterthat predates this fork's currentmaster; the two share the TurboQuant commits under different hashes. The diff should be clean, but the commit list contains two duplicates. A rebase ontopascal/masterwould remove them if a linear history is preferred.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q5jMcvdevae8j4T36C9h2z