TurboQuant: head_dim as a multiple of 128, three fixes, test coverage, fused kernel

docs/superpowers/specs/2026-08-10-turboquant-head-dim-256-design.md

@@ -0,0 +1,109 @@
+# TurboQuant für head_dim als Vielfaches von 128
+
+**Datum:** 2026-08-10
+**Status:** Design abgenommen
+**Ziel:** Qwen3.5-9B (head_dim 256) mit `--cache-type-k turbo3/turbo4` betreiben
+
+## Problem
+
+`llama-kv-cache.cpp` bricht die Kontext-Initialisierung ab, sobald ein Modell
+head_dim ≠ 128 hat:
+
+```
+llama_kv_cache: TurboQuant requires head_dim=128, got 256 (layer 3)
+llama_init_from_model: failed to initialize the context: turbo types require head_dim=128
+```
+
+Qwen3.5 (`Qwen3_5ForConditionalGeneration`) verwendet head_dim 256 und fällt
+damit heraus. Betroffen sind die 8 `full_attention`-Layer; die übrigen 24
+`linear_attention`-Layer halten einen rekurrenten State in F32 und berühren den
+KV-Cache nicht.
+
+## Befund: der Datenpfad ist bereits chunk-basiert
+
+Die Beschränkung liegt ausschließlich im Guard, nicht in der Verarbeitung.
+Alle drei Ebenen arbeiten in 128-Element-Chunks über einen flachen Speicher-
+bereich und kennen den Begriff „Head" gar nicht:
+
+| Ebene | Datei | Verhalten |
+|---|---|---|
+| CPU-Referenz | `ggml/src/ggml-quants.c:612` | `for (offset = 0; offset < k; offset += TURBO_HEAD_DIM)` |
+| GPU-Write | `ggml/src/ggml-cuda/set-rows.cu:444` | `GGML_ASSERT(ne00 % TURBO_HEAD_DIM_SR == 0)`, ein CUDA-Block je Chunk |
+| GPU-Read | `ggml/src/ggml-cuda/convert.cu:731` | `num_chunks = k / TURBO_HEAD_DIM_GPU`, ein CUDA-Block je Chunk |
+
+Alle drei prüfen bereits auf Teilbarkeit, nicht auf Gleichheit. Ein Modell mit
+head_dim 256 durchläuft sie unverändert korrekt — es entstehen zwei Chunks pro
+Head statt einem.
+
+## Ansatz: 2×128-Chunking
+
+Ein 256-dimensionaler Head wird als zwei unabhängige 128er-Hälften behandelt.
+Jede Hälfte wird separat L2-normalisiert, per FWHT rotiert und gegen das
+bestehende Codebook quantisiert.
+
+```
+Head (256 dim)
+ ├─ Hälfte A (128) → norm_A → FWHT128 → Codebook → 4 Blöcke
+ └─ Hälfte B (128) → norm_B → FWHT128 → Codebook → 4 Blöcke
+```
+
+### Warum die Codebooks gültig bleiben
+
+Die Zentroide sind Lloyd-Max-optimal für die Beta((d−1)/2, (d−1)/2)-Verteilung,
+die entsteht, wenn ein **Einheitsvektor in R^128** per FWHT rotiert wird. Nach
+der separaten Normalisierung ist jede Hälfte genau das: ein Einheitsvektor in
+R^128. Die Verteilungsannahme gilt also exakt weiter — im Gegensatz zu einer
+FWHT über 256 Dimensionen, die Beta(127,5, 127,5) erzeugen würde und neue
+Zentroide (um Faktor 1/√2 enger) erfordert hätte.
+
+### Warum kein zusätzlicher Speicher entsteht
+
+Die Norm wird ohnehin redundant in jedem 32er-Block abgelegt
+(`ggml-quants.c:636`, „Store same norm in every block of this chunk"). Zwei
+Halb-Normen pro 256er-Head belegen dieselben Bytes wie eine Norm pro 128er-Head.
+Die Bitrate bleibt unverändert bei 3,5 bpw (turbo3) bzw. 4,5 bpw (turbo4).
+
+### Warum Chunks nie Head-Grenzen überspannen
+
+Heads liegen kontiguierlich in der Row (`ne00 = n_head_kv × head_dim`). Da
+head_dim ein Vielfaches der Chunk-Größe 128 ist, fällt jede Chunk-Grenze mit
+einer Head-Grenze oder einer Head-Mitte zusammen, nie über einen Head hinaus.
+
+## Änderungen
+
+| Datei | Änderung |
+|---|---|
+| `src/llama-kv-cache.cpp:136-149` | Guard `!= 128` → `% 128 != 0`, Fehlertext anpassen |
+| `tests/test-turboquant.cpp` | head_dim parametrisieren, Fälle 256 und 384 ergänzen |
+| `docs/turboquant.md` | Requirements/Limitations korrigieren, Benchmark-Zeilen für Qwen3.5 |
+
+`ggml-quants.c`, `set-rows.cu` und `convert.cu` bleiben unverändert.
+
+## Nicht im Umfang
+
+- head_dim 80/96/112 (kein Vielfaches von 128) — der Guard lehnt weiterhin ab
+- Neu berechnete Codebooks für echte FWHT über 256
+- Fusionierter Flash-Attention-Kernel (die Pre-Dequantize-Strategie bleibt)
+
+## Verifikation
+
+1. **Unit-Tests** — `test-turboquant` mit head_dim 128, 256, 384: FWHT-Selbst-
+   inversion, Roundtrip-MSE·d gegen die Paper-Bereiche, Bitpack-Determinismus.
+   Erwartung: MSE·d bei 256 im selben Bereich wie bei 128.
+2. **Modell lädt** — Qwen3.5-9B-Q8_0 mit `-ctk turbo3 -ctv turbo3` startet und
+   erzeugt kohärenten Text.
+3. **Perplexität** — Qwen3.5-9B: f16 vs. turbo4 vs. turbo3. Erwartung:
+   Delta in der Größenordnung der dokumentierten Werte (turbo4 ≈ +0,01,
+   turbo3 ≈ +0,05).
+4. **Durchsatz** — `llama-bench` pp512/tg128 gegen f16-Baseline.
+5. **VRAM** — gemessener KV-Verbrauch gegen die Rechnung
+   (turbo3 bei head_dim 256: 7,3 KiB/Token über 8 Attention-Layer).
+6. **Vision** — mmproj-Pfad funktioniert weiterhin mit aktivem turbo3.
+
+## Risiko und Rückfallweg
+
+Die zwei Halb-Normen pro Head sind ein realer Unterschied zum d=128-Fall. Sie
+erfassen Varianzunterschiede zwischen Head-Hälften feiner, was die Qualität eher
+verbessert; belegt ist das erst durch Schritt 3. Fällt die Perplexität deutlich
+schlechter aus als erwartet, ist der Rückfallweg eine echte FWHT über 256 mit
+neu berechneten Lloyd-Max-Codebooks für Beta(127,5, 127,5).

docs/turboquant.md

@@ -35,6 +35,133 @@ All benchmarks: Qwen3-14B Q4_K_M, AMD RX 9070 XT (16 GB VRAM, gfx1201), ROCm 6.1
 | **turbo4/turbo4** | **1812** | **1816** | **1321** | **926** | **550** | **460** | **49.3** | **4.5** |
 | **turbo3/turbo3** | **1836** | **1816** | **1319** | **924** | **548** | — | **49.6** | **3.5** |
 
+### Throughput — Qwen3.5-9B Q8_0 (head_dim 256)
+
+RX 9070 XT, `-fa 1 -ngl 99`, 2 runs each. Note the reversed picture compared to
+head_dim 128: turbo is substantially *faster* at prompt processing here.
+
+| type_k | type_v | pp512 | tg128 |
+|--------|--------|-------|-------|
+| f16 | f16 | 2565 ± 175 | 56.4 |
+| f16 | turbo4 | 4084 ± 118 | 55.8 |
+| turbo4 | turbo4 | 3929 ± 34 | 55.3 |
+| turbo4 | f16 | 3981 ± 28 | 55.7 |
+| turbo3 | turbo3 | 3933 ± 25 | 55.2 |
+| turbo3 | f16 | 3966 ± 17 | 55.9 |
+
+Any turbo operand lifts pp512 by ~53% over f16/f16 while token generation stays
+flat. The likely cause is a different flash-attention kernel being selected once
+an operand is a turbo type — unverified, worth a look with `GGML_CUDA_DEBUG`.
+
+KV cache at 131072 context, 8 attention layers (Qwen3.5 is hybrid — the other
+24 layers are linear attention with a constant-size recurrent state):
+
+| KV Type | KV cache | Total VRAM | Savings |
+|---------|----------|-----------|---------|
+| f16 | 4096 MiB | 14.81 GB | — |
+| turbo4 | 1152 MiB | 11.75 GB | 72% |
+| turbo3 | 896 MiB | 11.47 GB | 78% |
+
+At 131072 context f16 needs 14.81 GB of the card's 16.3 GB, leaving almost no
+headroom for the vision encoder. turbo3 brings that down to 11.47 GB.
+
+### Token generation degrades with context depth
+
+The pre-dequantize strategy converts the whole KV cache to f16 once per FA call.
+That cost is O(n_kv) per generated token, so it grows with context depth while the
+attention work per token stays flat. Measured on Qwen3.5-9B Q8_0, RX 9070 XT:
+
+| type_k | type_v | tg128 @ d0 | tg128 @ d16384 | tg128 @ d65536 |
+|--------|--------|-----------|---------------|---------------|
+| f16 | f16 | 56.8 | 54.1 | **46.9** |
+| f16 | turbo4 | 55.9 | 48.0 | 32.3 |
+| f16 | turbo3 | 56.1 | 47.3 | 31.5 |
+| turbo4 | f16 | 55.7 | 48.0 | 32.5 |
+| turbo3 | f16 | 56.4 | 48.2 | 31.9 |
+| turbo3 | turbo4 | 55.9 | 43.1 | **24.5** |
+
+One quantized cache costs ~32 t/s at depth 65536, two cost ~24.5, f16 gets 46.9.
+The penalty tracks the *number* of converted caches, not which one — exactly what
+a per-cache bulk conversion predicts.
+
+Prompt processing is unaffected (turbo is in fact faster there, 1012 vs 730 t/s at
+d65536) because the single conversion amortizes over the whole batch.
+
+### Fused kernel: implemented, correct, currently slower
+
+A fused vector kernel that decodes turbo blocks inside flash attention exists
+(`fattn-vec-turbo.cuh`). It removes the bulk conversion entirely by working in
+the Hadamard domain — Q is transformed once at kernel start, V is accumulated
+untransformed and transformed once at the end, so two transforms per launch
+replace two per KV token.
+
+It is correct (test-backend-ops FLASH_ATTN_EXT: turbo3 640/640, turbo4 640/640)
+but **slower than the path it replaces**, measured A/B in the same build:
+
+| depth | fused (KQ=32) | fused (KQ=8) | bulk conversion | f16 |
+|-------|--------------|-------------|-----------------|-----|
+| 0 | 52.5 | 53.2 | 55.0 | 56.8 |
+| 16384 | 35.7 | 39.2 | 42.5 | 54.1 |
+| 65536 | 16.6 | **20.4** | **23.6** | **46.9** |
+
+Narrowing the KQ reduction from 32 lanes per token to 8 (one warp reduction is
+then 3 stages instead of 5, and 4 tokens are dotted at once) bought 23% at depth
+65536. It does not close the gap, because the remaining cause is structural:
+
+**GQA amortization.** For `Q->ne[1] == 1` with `gqa_opt_applies`, f16 does not
+use the vector kernel at all — `fattn.cu` routes it to the tile kernel, which
+processes all `gqa_ratio` query heads per KV read. The fused vector kernel
+handles one query head per block and therefore reads K and V `gqa_ratio` times
+as often. Per element and query head, with Qwen3.5's gqa_ratio = 4:
+
+    tile + f16:     2 bytes / 4 heads = 0.50 bytes
+    fused + turbo3: 0.4375 bytes x 1  = 0.4375 bytes
+
+The 4.57x compression collapses to a 12% bandwidth edge, which the unpacking and
+codebook lookups more than consume. No amount of tuning inside the vector kernel
+recovers this; a fused kernel has to support GQA to beat the tile path.
+
+Therefore **disabled by default**. Enable with `GGML_CUDA_TURBO_FUSED_FA=1`.
+
+### Fused tile path: K and V decoded inside attention
+
+The tile kernel is what f16 uses for decode with GQA, so decoding there avoids
+the vector kernel's GQA penalty. Both sides are now fused (`type_K`/`type_V`
+threaded through `flash_attn_tile` / `_iter` / `_iter_KQ`, `need_f16_K/V = false`).
+
+The decisive detail is *where* the Hadamard transform happens. A first version
+decoded each chunk as it was loaded — one inverse FWHT per KV token, the same
+O(n_kv) cost as the bulk conversion, just relocated. It measured exactly as fast
+as what it replaced. Working in the Hadamard domain instead reduces that to O(1):
+
+  V:  sum_j p_j H c_j = H (sum_j p_j c_j)   accumulate raw, transform once at the end
+  K:  <H c, q> = <c, H q>                   transform Q once at kernel start
+
+tg128 at depth 65536, Qwen3.5-9B Q8_0 on gfx1201:
+
+| stage | t/s | of f16 |
+|-------|-----|--------|
+| bulk conversion (baseline) | 23.58 | 0.50x |
+| V fused | 26.62 | 0.57x |
+| K and V fused | **29.59** | **0.63x** |
+| f16 reference | 46.90 | 1.00x |
+
+0.63x matches what llama.cpp discussion #20969 reports for quantized KV at 110K
+context. Quantized KV is slower than f16 at decode across methods and backends —
+that is a property of the approach, not of this implementation. What was
+recoverable has been recovered.
+
+Two bugs worth recording, both found by the test matrix rather than by reading:
+
+- The K tile loader assumed a slice narrower than one 128-element chunk. True for
+  `nbatch_K = 64`, false for 128 and 256, which the config table also selects.
+  The chunk (and its norm) is now derived per element.
+- Staging all of Q for the transform needed `ncols*DKQ` floats in `KV_tmp`, which
+  is sized by `nbatch_fa*nbatch_K` — and those shrink as `ncols` grows. At
+  `ncols=64` the buffer was short by a factor of 1.8. Transforming one chunk at a
+  time needs 128 floats regardless of `ncols`. This is why only large batches
+  failed while `nb=1` stayed green.
+
 ### Perplexity (lower is better)
 
 Measured on ~960KB C++ source code corpus, context=2048.
@@ -46,6 +173,45 @@ Measured on ~960KB C++ source code corpus, context=2048.
 | turbo4 | 1.7134 | +0.010 |
 | turbo3 | 1.7544 | +0.051 |
 
+Qwen3.5-9B Q8_0 (head_dim 256), 600KB C++ corpus, context=2048:
+
+| KV Type | PPL | Delta vs f16 |
+|---------|-----|-------------|
+| f16 | 1.5258 ± 0.0071 | — |
+| turbo4 | 1.5281 ± 0.0071 | +0.002 |
+| turbo3 | 1.5343 ± 0.0072 | +0.009 |
+
+Both deltas are markedly smaller than at head_dim 128 (+0.010 / +0.051). Splitting
+a 256-dim head into two independently normalized halves stores two norms instead
+of one, which tracks variance differences between the halves more closely — at no
+extra cost, since the norm is replicated per block either way.
+
+#### The error does not accumulate with context depth
+
+KVarN (arXiv 2606.03458, llama.cpp #24139) argues that KV-cache quantization
+error compounds during autoregressive decoding, so a short-context perplexity
+number can understate the real cost. Measured against that concern:
+
+| context | f16 | turbo4 | delta | turbo3 | delta |
+|---------|-----|--------|-------|--------|-------|
+| 2048 | 1.5258 | 1.5276 | +0.0018 | 1.5338 | +0.0080 |
+| 8192 | 1.4107 | 1.4124 | +0.0017 | 1.4172 | +0.0065 |
+| 16384 | 1.3431 | not measured | — | 1.3497 | +0.0066 |
+
+The gap does not widen. For turbo3 it drops from +0.0080 at 2048 to +0.0065 at
+8192 and then holds at +0.0066 through 16384 — 0.52%, 0.46%, 0.49% in relative
+terms. Three depths across an 8x range, flat after the first step.
+
+(turbo4 at 16384 is missing; its delta is ~0.0017 at both measured depths, small
+enough that the trend question hardly applies.)
+
+That is consistent with how TurboQuant stores data: the per-chunk L2 norm is kept
+in fp16 and never quantized, so only the *direction* of the normalized vector is
+approximated. KVarN attributes the accumulating error specifically to incorrect
+per-token magnitude scaling — the mechanism does not exist here. The per-chunk
+normalization that makes TurboQuant expensive at runtime is the same property
+that makes it robust over long contexts.
+
 ### KV Cache Memory
 
 | KV Type | Bytes per element | Savings vs f16 |
@@ -67,19 +233,29 @@ where the Lloyd-Max codebook is optimal. The codebook centroids are precomputed
 for the Beta((d-1)/2, (d-1)/2) distribution that arises after FWHT of unit vectors
 in d=128 dimensions.
 
+The transform is always applied in fixed 128-element chunks, walking the row.
+A head larger than 128 is therefore split into independently normalized chunks:
+a 256-dim head (Qwen3.5) becomes two halves, each with its own L2 norm and its
+own FWHT. Because every normalized half is again a unit vector in R^128, the
+same codebooks stay optimal — no per-head-size codebook is needed. The stored
+norm is already replicated per 32-element block, so the extra half-norm costs
+no additional bytes and the bit rate is unchanged.
+
 Current implementation uses a pre-dequantize strategy: turbo KV data is bulk-converted
 to f16 before standard Flash Attention runs. This adds minimal overhead (~3% pp, ~8% tg)
 while avoiding the complexity of a fused FA kernel.
 
 ## Requirements
 
 - Flash Attention enabled (`-fa 1`)
-- head_dim = 128 (covers Llama, Qwen, Mistral, Gemma, and most current models)
+- head_dim must be a multiple of 128 — 128 covers Llama, Qwen3, Mistral and
+  Gemma; 256 covers Qwen3.5
 - AMD ROCm (gfx1201 tested) or CPU
 
 ## Limitations
 
-- head_dim must be exactly 128 (FWHT requirement)
+- head_dim must be a multiple of 128 (FWHT chunk size). Models with head_dim
+  80, 96 or 112 are not supported.
 - Mixed turbo/quantized combinations (e.g. turbo4/q8_0) are not supported — use turbo/turbo or turbo/f16
 - No CUDA support yet (HIP/ROCm only for GPU path)
 - Pre-dequantize strategy means full KV cache is converted to f16 per FA call (a fused kernel would eliminate this)

ggml/src/ggml-cpu/ggml-cpu.c

@@ -393,6 +393,24 @@ static const struct ggml_type_traits_cpu type_traits_cpu[GGML_TYPE_COUNT] = {
     [GGML_TYPE_I32] = {
         .from_float               = (ggml_from_float_t) ggml_cpu_fp32_to_i32,
     },
+    // TurboQuant: KV-cache only. vec_dot takes f32 on the right-hand side and
+    // dequantizes the left-hand side internally (inverse FWHT + norm), so
+    // vec_dot_type is F32. Without these entries the CPU flash-attention path
+    // calls a null vec_dot pointer, which is what test-backend-ops hits when it
+    // computes its CPU reference.
+    // No .from_float: quantize_row_turbo*_ref asserts k % 128 == 0, which the
+    // generic per-block callers do not honor. Quantization goes through
+    // ggml_quantize_chunk instead.
+    [GGML_TYPE_TURBO3_0] = {
+        .vec_dot                  = ggml_vec_dot_turbo3_0,
+        .vec_dot_type             = GGML_TYPE_F32,
+        .nrows                    = 1,
+    },
+    [GGML_TYPE_TURBO4_0] = {
+        .vec_dot                  = ggml_vec_dot_turbo4_0,
+        .vec_dot_type             = GGML_TYPE_F32,
+        .nrows                    = 1,
+    },
 };
 
 const struct ggml_type_traits_cpu * ggml_get_type_traits_cpu(enum ggml_type type) {

ggml/src/ggml-cpu/quants.h

@@ -42,6 +42,11 @@ void ggml_vec_dot_q5_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const voi
 void ggml_vec_dot_q5_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
 void ggml_vec_dot_q8_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
 
+// TurboQuant. Defined in ggml-quants.c (not arch-specific): dequantizes the
+// left-hand side internally, so the right-hand side is plain f32.
+void ggml_vec_dot_turbo3_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_turbo4_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
 void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
 void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
 

ggml/src/ggml-cuda/CMakeLists.txt

@@ -120,9 +120,7 @@ if (CUDAToolkit_FOUND)
             template-instances/fattn-vec-instance-f16-f16.cu
             template-instances/fattn-vec-instance-q4_0-q4_0.cu
             template-instances/fattn-vec-instance-q8_0-q8_0.cu
-            template-instances/fattn-vec-instance-bf16-bf16.cu
-            template-instances/fattn-vec-instance-turbo3_0-turbo3_0.cu
-            template-instances/fattn-vec-instance-turbo4_0-turbo4_0.cu)
+            template-instances/fattn-vec-instance-bf16-bf16.cu)
     endif()
 
     ggml_add_backend_library(ggml-cuda