Repository navigation
ARCH: quant-type dispatch is duplicated across ~30 files with no single source of truth — propose a dedicated consolidation release #3418
Description
tl;dr
Aprender has no equivalent of llama.cpp's ggml_type_traits[] — the single
table that tells the entire codebase what a quant type is (block size,
byte size, dequant fn) once, so every operation and every model architecture
gets it for free. Instead, ~30 files in crates/aprender-serve/src/ each
independently match on quant type. The result is that "does architecture X
work" is not actually one question — it's architectures × quant_types × backends × call_sites, and today most of that matrix is untested. Five
closed tickets (#1749, #1789, #2535, #3341, #3091) are the same defect class
recurring in different cells of that matrix, each discovered by a user
crash rather than by a gate that could have covered all of them at once.
This is not a "fix Qwen" ticket — #3090/#3091/#3413 stay where they are for
the specific model-family defects. This is the structural ticket: the
dispatch architecture itself is the reason those keep happening, and it
will keep happening for every future model (Qwen4 preview is already sitting
in a Downloads folder untested) unless the dispatch is consolidated once.
Evidence
1. The size gap between "new architecture" costs, aprender vs llama.cpp
llama.cpp's entire Qwen3-MoE architecture — tensor shapes, attention, RoPE,
MoE routing — is 179 lines (src/models/qwen3moe.cpp in a local llama.cpp
checkout, current as of 3173a5647 2026-08-29). It's short because it's
almost entirely calls into shared, quant-agnostic building blocks
(build_qkv, build_attn, build_moe_ffn, ggml_rope_ext) that every one
of llama.cpp's ~50 architectures reuses unchanged.
Aprender's Qwen3-MoE-specific code alone
(crates/aprender-serve/src/gguf/qwen3_moe_load.rs +
crates/aprender-serve/src/infer/qwen3_moe_generate.rs) is 1,678 lines —
before counting anything it depends on.
2. Quant-type dispatch is scattered, not centralized
$ grep -rln "GGUF_TYPE_Q4_K\|GGUF_TYPE_Q4_0\|match qtype\|match.*\.qtype" \
crates/aprender-serve/src/ --include="*.rs" | grep -v test | wc -l
30
30 files, each with its own quant-type match arms:
gguf/transformer.rs, gguf/inference/fused_matmul_into.rs,
gguf/inference/matmul_fused.rs, gguf/qwen3_moe_load.rs,
gguf/metadata.rs, cuda/executor/layers/gemv_dispatch.rs,
gpu/adapters/wgpu_adapter.rs, and 23 more (full list in the investigation
that produced this ticket — available on request / in session transcript).
llama.cpp's equivalent is ggml_type_traits[]
(ggml/src/ggml.c:~750-900) — one array, one entry per type
(type_size, blck_size, to_float, from_float), read generically by
every operation in the compute graph.
Aprender has a partial version of this idea —
QuantBlockFormat (crates/aprender-serve/src/quantize/format_trait.rs) —
but only 5 of the ~30 call sites are known to route through it consistently,
and the trait itself only models affine quantization (x = d*s*q - dmin*m: Q4_0/Q8_0/Q4_K/Q5_K/Q6_K). It cannot represent the IQ family
(IQ2_XXS/XS, IQ3_XXS/S, IQ4_NL/XS), which are codebook/lattice
quantization — each block indexes into a fixed grid table, not a per-block
affine scale. That's a second trait shape needed, not an extension of the
first.
3. The universe aprender tracks is smaller than the universe it needs to track
$ grep -rohE "GGUF_TYPE_[A-Z0-9_]+" crates/aprender-serve/src/ | sort -u
GGUF_TYPE_ARRAY GGUF_TYPE_BF16 GGUF_TYPE_F16 GGUF_TYPE_F32
GGUF_TYPE_INT32 GGUF_TYPE_Q2_K GGUF_TYPE_Q3_K GGUF_TYPE_Q4_0
GGUF_TYPE_Q4_1 GGUF_TYPE_Q4_K GGUF_TYPE_Q5_0 GGUF_TYPE_Q5_1
GGUF_TYPE_Q5_K GGUF_TYPE_Q6_K GGUF_TYPE_Q8_0 GGUF_TYPE_STRING
GGUF_TYPE_UINT32
16 constants named anywhere in the crate. The GGUF spec defines 30+
(missing here: the entire IQ1/IQ2/IQ3/IQ4 family, Q8_1, Q8_K, TQ1_0/TQ2_0).
Every one of those missing types is a RealizarError::UnsupportedOperation
waiting to be hit by the next real-world download, in whichever of the 30
files happens to load the tensor first.
4. The ticket pattern — same defect, different cells of the matrix
| Issue | Architecture | Broke because |
|---|---|---|
| #1749 | MoE (generic) | matmul_fused.rs:211 index OOB on MoE GGUF |
| #1789 | Qwen3-Coder-30B MoE | Same line, F32 weight this time |
| #2535 | MoE (generic) | Misdiagnosed as "truncated/corrupt" — the real cause was an unhandled tensor shape/qtype combination |
| #3341 | Qwen3-Coder-30B MoE | Q4_0 experts refused — the MoE loader's qtype list only had Q4_K/Q6_K |
| #3091 | Qwen3.5 (qwen35) |
IQ2_XS/IQ4_XS refused in tensor_byte_size — that function's qtype list only had 9 of the ~30 real types |
Five tickets, five separate fixes, one root cause each time: "this
particular file's quant-type list was incomplete." None of these fixes
closed the class — each is scoped to the one file that crashed. The next
new model (Qwen4-preview architecture is already sitting untested in
~/Downloads/Qwen3.8-Flash-Next-UD-Q2_K_XL-...gguf, per #3413's sibling
investigation) will find the next incomplete list, in some 31st file.
Proposed consolidation
- A
QuantCodebookFormattrait alongside the existing
QuantBlockFormat, covering lattice/codebook quant types. Grid tables
are portable data — llama.cpp'sggml-quants.ciq2xs_grid/iq3xs_grid
etc. are the reference, not code to be creatively reinvented. - A single dispatch entry point analogous to
ggml_type_traits[]:
given aqtype: u32, return{byte_size, dequant_to_f32, family}in one
place. Every one of the ~30 call sites is refactored to call through it
instead of maintaining its own arm list. - A completeness gate: a test (or
pvcontract) that enumerates every
GGUF_TYPE_*the format spec defines and asserts the dispatch table has
an entry for each — so a missing type is a CI failure at merge time, not
a user's crash report three months later. This is the mechanical
backstop; CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091's own history shows the honor-system approach (each file
remembering to update its own list) has failed five times already. - Per-backend fan-out is then additive, not multiplicative: CPU
SIMD/CUDA/wgpu each implement the trait once per type; the ~30
call-site refactor happens once regardless of how many backends exist.
Why this should be its own release, not squeezed into the current cadence
Per docs/specifications/06x-release-schedule.md, this project runs a
2-3 day release cadence, 24/7 (v0.66.0 → v0.70.0 in the current
train). That cadence is built for landing scoped, independently-shippable
fixes — exactly the shape of #3091, #3341, #3413. It is the wrong cadence
for this ticket, for three reasons:
- Blast radius. The refactor touches ~30 files across the CPU, CUDA,
and wgpu backends simultaneously — nearly every file in
crates/aprender-serve/src/gguf/,quantize/,cuda/, andgpu/.
Landing it incrementally alongside unrelated model-fix PRs on a 2-3 day
cadence means every one of those PRs is rebasing against a moving
dispatch layer, and every dequant/matmul kernel needs re-verification
against the same parity bar this project already holds itself to (per
CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091's own history: cosine ≥0.98 vs llama.cpp, per quant type, per
backend). That verification matrix (≥9 missing quant types × 3 backends)
is itself real work, not a rename. - Regression risk to the one thing that currently works. Qwen2.5-Coder
(README's demonstrated, BEATS.md-measured, GPU-parity model) runs through
several of these same 30 files today. A consolidation that isn't done in
one dedicated, fully-gated pass risks the exact kind of silent regression
this ticket exists to prevent — on the one model line that currently has
real proof behind it. - It competes for the same attention the model-specific tickets need.
GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 (GPU), P0: dense Qwen3 (Qwen3ForCausalLM) has no current proof it runs — last verification is 7 months stale and CI never covers it #3413 (dense Qwen3 proof), and whatever Qwen4-preview turns
into all sit downstream of this dispatch layer. Doing them first means
redoing parts of them once this lands; doing this first means every
future model ticket gets cheaper, not just the current ones.
Ask: reserve one full release cycle (e.g. a dedicated milestone, not
squeezed as a rider into 0.69/0.70/0.71) where this consolidation is the
only architectural change landing, with model-specific bug fixes limited
to true P0 hotfixes during that window. The exit criterion is the
completeness gate in (3) above going green, plus a full parity re-run of
the existing Qwen2.5-Coder GPU-beat measurement to prove no regression.
Related
- CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091, GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090, feat: Qwen3.5 (qwen35) SSM/Gated Delta Net inference support #3038 — Qwen3.5 CPU/GPU (the immediate trigger for this
investigation) - P0: dense Qwen3 (Qwen3ForCausalLM) has no current proof it runs — last verification is 7 months stale and CI never covers it #3413 — dense Qwen3 proof-of-life gap (downstream of the same dispatch
layer) - defect:
apr benchpanics on MoE GGUF models (matmul_fused.rs:211 index OOB) #1749, apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight #1789, MoE GGUF misdiagnosed as 'truncated/corrupt … file is incomplete' — the file is provably complete, and apr validate agrees #2535, CPU+GPU: Qwen3-Coder-30B MoE (Q4_0) fails with empty-buffer error on blk.0.ffn_gate.weight — possible #1789 regression #3341 — prior instances of the same defect class - Docs: publish a GPU-vs-CPU model architecture support table (no user-facing doc exists today) #3077 — GPU/CPU model support table (would become truthful, not
aspirational, once (3)'s completeness gate exists)
Activity
docs(spec): PP-QUANT-001 — fix non-object-safe dispatch design (#3418)
noahgift commentedon Sep 17, 2026
PP-QUANT-001 — Phase 0/1 ground truth (research run, 2026-09-17). Verdict:
STOPPED(framing-falsified)Measured on
origin/main@c79a53223, v0.68.0 tagged 2026-09-17T06:35Z; llama.cpp @3173a5647. Marks:[V]verified by the command shown ·[C]computed from a cited artifact ·[U]unverified. Original figures are struck, never silently replaced. No code, spec, or issue body was edited.1. The ticket's numbers, re-derived
30GGUF_TYPE_*— gguf/ 11, gguf/inference 10, gguf/cuda 4, convert 1, gpu/adapters 1[V]git grep -l -E 'GGUF_TYPE_[A-Z0-9_]+' origin/main -- 'crates/aprender-serve/src/*.rs'16UINT32 STRING INT32 ARRAY) — two different id spaces counted as one[V]"30+"GGML_TYPE_COUNT = 43: 35 live, 8 removed (4, 5, 31–33, 36–38). New since January: NVFP4=40, Q1_0=41, Q2_0=42[V]git show 3173a5647:ggml/include/ggml.hQuantBlockFormatquantize/— zero users outside it[V][V]— but see §32. The universe is larger than the ticket says — three enums, not one crate
[V]GgmlQuantType(gguf/types.rs:101)GgmlType(format/gguf/types.rs:55, the import path)GgmlType(inference/gguf.rs:27)Plus 14 further
Quant*/Quantization*enums across cbtop, core, cuda-edge, qa-runner, registry, train. 19 of 35 live spec ids are unnamed even in serve. A single table insideaprender-serve/src/quantize/cannot be the source of truth while the importer (core) and the kernels (compute) keep their own; it has to sit below all three. Also: bare-integer dispatch (12 =>,qtype == 2) exists beside the constants — a regex upper bound of 558 non-test lines across 10+ crates[U](noisy; needs a case table) — so a ban gate keyed onGGUF_TYPE_*alone would not see it.3. Five-whys on the cited instances: 1 of 5 is "local qtype list incomplete"
[C](from each fix PR's own root-cause text; diffs not re-read)matvec_for_qtypefallback arm on Q4_0 experts; fixed by single-sourceSUPPORTED_EXPERT_QTYPES+ drift test (#3405)qwen35architecture refused; fix is a new forward pass (#3114)Four of five share a different class: MoE placeholder tensors (
byte_size=0,qtype=0) reaching dense-path consumers. A quant traits table would not have prevented #1749, #1789, #2535 or #3091. The duplication in §2 is real and worth fixing, but the defect history does not support it as the cause of the user-found crashes, and it does not by itself justify a must-carry slot. The run's own rule for this outcome is: report, do not plan a release around it.4. The dedicated-release ask
Not needed either way. CI on
main: p50 31 min, p95 62 min (n=31 successful of the last 50 push runs)[V]→ ~69 serial slots per 72 h window, and the merge queue builds 3 in parallel. A table → ban gate → per-directory migration ordering handles rebase churn without stopping the train.5. Proposed re-scope (for #3421 / PR #3420 — not applied; both are human-authored)
apr benchpanics on MoE GGUF models (matmul_fused.rs:211 index OOB) #1749/apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight #1789/MoE GGUF misdiagnosed as 'truncated/corrupt … file is incomplete' — the file is provably complete, and apr validate agrees #2535; PP-QUANT-001 keeps CPU+GPU: Qwen3-Coder-30B MoE (Q4_0) fails with empty-buffer error on blk.0.ffn_gate.weight — possible #1789 regression #3341 only.Not done: per-call-site table, spec revision, quorum, child tickets. Also noted: #3091 is still open in the
0.68.0milestone though 0.68.0 has shipped;APR-RELEASE-001onmainhas no §1.5 must-carry table for a 0.69 row to land in.spec(PP-QUANT-001): Phase 0 inventory as a pv contract + the Rust des…
docs(spec): PP-QUANT-001 — cross-link downstream PP-ARCH-001, close o…
alfredodeza commentedon Sep 17, 2026
This ticket is the umbrella problem statement for "new architectures are too
expensive to add" — two sub-problems have since been evidenced and are tracked
under it, each with its own spec and epic (design docs stay separate; this issue
is the one narrative entry point):
docs/specifications/PP-QUANT-001-MASTER.md, epic EPIC: PP-QUANT-001 — quant-type dispatch, Phase 0/1 (0.69 train) #3421, milestone0.69.0for Phase 0/1 (done: PR spec(PP-QUANT-001): quant-type dispatch consolidation — design, contract, Phase 0 gate #3420), milestoneInference dispatch & architecture consolidationfor Phase 2/3 (the 30-file migration + parity re-proof — the large-blast-radius part, not squeezed into a train).attention.rs,ffn_block.rs) already existing and sitting at 0% adoption.docs/specifications/PP-ARCH-001-MASTER.md, epic EPIC: Inference dispatch & architecture consolidation (dedicated cycle) #3423, milestoneInference dispatch & architecture consolidation.Both land out of the same dedicated milestone once past their low-risk Phase 0/1
work, tracked jointly by epic #3423. #3422 is not independent scope — it's the
other half of what this ticket asked for ("easier to add new model support").
docs(spec): PP-QUANT-001 — Phase 2/3 gets its real milestone, not "ma…
docs(spec): PP-ARCH-001 — reflect ticket/epic consolidation (#3422)
noahgift commentedon Sep 17, 2026
Follow-up (2026-09-17, PMAT-3427) — corrections to my 09:20Z comment, and where this now lives
Corrections to my own numbers. (a) I classed #3091 as "arch", giving a 4/1 split from PR bodies. Read from the fix diffs, plus today's reopen, it is 3 placeholder / 2 qtype-list — so "framing falsified" was too strong: the qtype-list class is real and recurring, it is just the smaller of two. (b) Spec ids: 35 live + 8 removed, not 34 + 9. (c) Live ids named by no aprender enum: 13; unnamed in serve alone: 19.
New since then
[V]unsloth/Qwen3.5-0.8B-GGUF,UD-IQ2_XXS→ exit 8, type 16;UD-Q4_K_XL→ type 23).tensor_byte_sizehas 11 arms and lacks Q3_K and Q5_1, which the crate defines; there are three such functions.dense_ffn_placeholderisbyte_size: 0, qtype: 0, and 0 is F32. feat(apr-cli): bench MoE dispatch — routes Qwen3-MoE GGUFs through forward_qwen3_moe (#1749) #1751, fix(serve): #1789 matmul defensive guard against empty / undersized weights #1790 and fix(serve): a complete MoE model was rejected as "truncated/corrupt" (#2535) #2541 each guard or re-route one consumer; none removes it. 63 non-test files read the dense FFN fields; 14 carry no guard token, including the CUDA and wgpu consumers.Where it lives
GgmlType+TRAITS[43]→ #3431 reconciliation gate → #3432 byte size + single refusalmain06x-release-schedule.md§1.1a; receipt with Tables G and S)The shared-block / LOC-gap half is Alfredo's #3422 (PP-ARCH-001); my duplicate #3444 is closed in its favour.
Still true: no dedicated release is needed for throughput reasons (CI p95 62 min, ~69 slots per 72 h, queue 3 wide). Still not done: no independent quorum has reviewed the design positions — everything above is one reviewer's measurement, and PR #3447 stays draft until that happens.
PMAT-3427: roadmap rows arrive as fragments; an owed contract is name…
noahgift commentedon Sep 23, 2026
Filed #3945 as the test half of this ticket: an enumerated
(qtype × op × backend × shape)conformance table derived from the dispatch registry, so admitting a quant type automatically creates its proof obligations.Your framing is the specification and I am citing it verbatim there — "'does architecture X work' is not actually one question — it's
architectures × quant_types × backends × call_sites, and today most of that matrix is untested." What was missing is the thing that enumerates it; llama.cpp pairsggml_type_traits[]withtest-backend-ops, which auto-enumerates (op × type × shape) and compares backend against CPU by NMSE. You asked for the first half; #3945 is the second.The 0.69.1 release measured the cost this week: the GPU-eligibility whitelist turned out to be recorded in nine places (the commit that warned about six undercounted), the ninth was invisible to three audits because it names a type nothing had changed, and
dtype.rs— which calls itself the single source of truth — contradicted its own expression in both directions. Also noted on #3421: its Phase 0 checkbox is marked complete while PR #3420 is still open and the contract is not onmain.27 remaining items
refactor(quant): QuantType bits per weight read the one ggml TRAITS t…
refactor(gguf): get_tensor_f32 reads block sizes and decoders from th…
refactor(gguf): the Qwen3.5 embedding row decode reads an in-place to…
refactor(gguf): the fused matmul sends Q4_0 and Q8_0 through the one …
fix(gguf): padded weight rows decode row by row through the one ggml …
fix(gguf): the APR token embedding decodes padded rows row by row thr…
fix(apr_transformer): the Q4 SIMD transformer runs each weight's own …
fix(apr_transformer): the quantized APR transformer's LM head decodes…
fix(gpu): the GPU Q4 adapter types every upload by the tensor's own q…
refactor(serve): the MoE expert matvec runs the one table's row for i…
refactor(serve): fused_matmul keeps no list of fused types; the table…
noahgift commentedon Oct 14, 2026
merged into #5315 (master ticket for epic #4001; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.