Repository navigation
IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker #3953
Description
capability_match refuses IQ2_S (ggml_type 22) on CUDA: gpu_supported: false. One of four types blocking inv:Qwen3.5-0.8B-UD-IQ2_XXS.gguf, measured against the whitelist at the release head from the file itself:
| type | ggml | tensors | ticket |
|---|---|---|---|
| IQ2_XXS | 16 | 95 | #3950 (aprender-6c) |
| IQ3_XXS | 18 | 24 | offered to aprender-6c |
| IQ2_S | 22 | 5 | this |
| Q2_K | 10 | 3 | unowned |
The cop ruled all four are rows now (operator doctrine: no defer). This kernel alone does not turn the row green.
Format (82 bytes / 256 elements, 2.5 bpw)
d f16 at +0 · qs[32] at +2 (index low 8 bits) · signs[32] at +34 · qh[8] at +66 (index high 2 bits) · scales[8] at +74. Per (sub-block ib, group l): idx = qs[4ib+l] | ((qh[ib] >> 2l) & 3) << 8 into the 1024-entry IQ2S_GRID; scale = scales[ib] low nibble for l<2, high nibble for l>=2; db = d·(0.5+nib)·0.25.
Plan
- Pin geometry and prove the A/B oracle non-degenerate, before any kernel.
- PTX kernel, grid table generated from
IQ2S_GRIDrather than hand-copied (the IQ3_S PTX hand-copies 512 numbers, and nothing compared them until a follow-up test). - Device A/B vs
crates/aprender-serve/src/quantize/iq2_s.rs, with at least three planted faults proven RED first, each on a different mechanism: sign bits ignored · qh high bits dropped (10-bit index read as 8) · scale nibble selected byl&1instead ofl>>1. - Whitelist + guard flip in one commit, after rebasing onto IQ2_XXS (ggml 16) has no CUDA GEMV — 0.69.1 release blocker #3950 — the two touch the same whitelist sites and must not race.
Template: #3884 (IQ3_S), whose layout (separate qs/qh/signs/scales byte arrays) is closer to IQ2_S than IQ2_XXS's packed u32s are.
Collapsed rows (APR-EPIC-001 rule 20)
One issue per PR: tick a line when its PR merges; the PR closing the last line says Closes #3953, the others Refs #3953 row <id>.
- Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960: Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker (Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960, last comment Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960 (comment))
- IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker #3963: IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker (IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker #3963, last comment IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker #3963 (comment))
Activity
feat(gpu): prove a GPU receipt had the card to itself, or refuse it (#…
test(#3953): IQ2_S decoder proven against gguf-py on every real type-…
noahgift commentedon Sep 24, 2026
triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.73.0 — unmilestoned → theme 0.73.0. It was (no milestone). Open PR / live branch: PR#4024, br:feat/3953-iq2s-gpu-gemv.
feat(gpu): prove a GPU receipt had the card to itself, or refuse it (#…
noahgift commentedon Sep 25, 2026
APR-EPIC-001 A2 (rule 9): stays OPEN. Legs (a)+(b) hold:
5fa13fe8c6d8on origin/main names this issue ("* feat(gguf): admit IQ2_S (22), IQ3_XXS (18) and Q2_K (10) to the GPU whitelist in ONE edit, each on its measured kernel (#3953 #3963 #3960)"). Missing leg: c-needs-repro. Readers: Strong row; whitelist landed but device A/B tests and planted faults not confirmed, and done_when never rerun. / Strong row requires done_when to be reproduced green at HEAD, which has not been done.. Applied by aprender-d1 under APR-EPIC-001 A2, quorum receipt handoff/a2-quorum.json (claude-sonnet-5 + agy gemini-3.1-pro-high; apr shadow lane unreachable, zero weight).noahgift commentedon Sep 26, 2026
Leg (c) reproduced at stage/0.70.1 @ec6574f00 on lambda RTX 4090 (aprender-57, 2026-09-26):
--features cuda --libfilteriq2_s: 17 passed, 0 failed (device A/B vs CPU decoder, every real model shape, every real type-22 tensor of Qwen3.5-0.8B-UD-IQ2_XXS; whitelist test).iq3_xxs+q2_k32 passed, 0 failed (IQ3_XXS (ggml 18) has no CUDA GEMV — 0.69.1 release blocker #3963, Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960 device A/B).generate_iq2_s_gemv_ptx: control rc=0; signs ignored rc=101; qh high bits dropped rc=101; scale selectorl&1rc=101.Finding row:
docs/findings/aprender-57.jsonlon branch fix/3953-repro-at-stage @42f978441 (to stage/0.70.1 via 59).release: merge back v0.69.3 into main (paiml#4338)
3 remaining items
noahgift commentedon Oct 9, 2026
Branch
wprod/3953-iq2s-repro@86cef5d2b6, pushed. There is no PR until a slot frees.The title is stale on one point. IQ2_S (type 22) already has a CUDA GEMV on main. What refused was the Qwen3.5 batched prefill: it dequantizes every projection weight to f32 (
qwen35_dequant_f32), and type 22 had no dequant. Soapr run --gpuon Qwen3.5-0.8B-UD-IQ2_XXS stopped at the model's five type-22ffn_downweights.The branch adds a loop-free IQ2_S → f32 dequant kernel. Measured on an RTX 4090 (sm_89), it is bit-exact against the CPU oracle on synthetic shapes and on all 5 real type-22 tensors (5 × 3,670,016 values). 9 of 9 planted faults go RED. The model row stays red until the other four types also have dequants (#4952, stacked on this branch).
Agent: aprender-wprodmodels
fix(cuda): Qwen3.5 batched prefill f32 dequants for IQ2_S, IQ2_XXS, I…
noahgift commentedon Oct 12, 2026
evidence for close: 5fa13fe (#4338 merge-back, on main) + #5131 @ 5f8f904, checked by reading main:
crates/apr-cli/contracts/apr-model-capability-v1.yamllists IQ2_S ggml_type 22gpu_supported: truewith the device A/B and planted-fault record;generate_iq2_s_gemv_ptxis incrates/aprender-serve/src/cuda/kernels_generate_gemm_cuda.rs; #5131 added the IQ2_S → f32 dequant for the Qwen3.5 batched prefill with its tests. The leg (c) repro above (17 passed, 3 planted faults RED) was the open item.noahgift commentedon Oct 12, 2026
Cop batch close (FLOW-004 drain, ≤43/d): fix verified on main — merged #4338 #5131 ; ancestor of origin/main: 5f8f904 5fa13fe . Evidence: #3953 (comment)