CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists

Summary

Unlike most GPU-ineligible architectures (e.g. Phi's LayerNorm gap, #3075), which fall
back to CPU inference with a warning, Qwen3.5/Qwen3.8-architecture GGUFs (hybrid Gated
DeltaNet / SSM layers, general.architecture = qwen35) are refused completely —
there is no CPU inference path at all today. Confirmed with two different downloads:

apr chat ~/Downloads/Qwen3.8-27B-UD-IQ2_XXS.gguf
apr chat ~/Downloads/Qwen3.5-4B-UD-Q4_K_XL.gguf

Both fail identically, even without --gpu:

[GGUF model parse failed: Format error: Architecture 'qwen35' uses SSM/Gated Delta Net
layers which are not yet supported for inference. Use a standard transformer model
(e.g., Qwen2.5, LLaMA, Mistral) or wait for SSM support in a future release., will use CPU]
You: hey
Assistant: [Error: Failed to create GGUF model: Format error: Architecture 'qwen35' ...]

The chat loop even prints "will use CPU" in the CUDA-init warning line, but then the
CPU load of the same GGUF fails with the exact same error — because the refusal is in
the shared GGUF transformer constructor (QuantizedGGUFTransformer::from_gguf), which
both the CPU and GPU code paths call. There is no degraded-but-working CPU mode; the
model simply cannot be loaded by this binary in any form.

Root cause

crates/aprender-serve/src/gguf/transformer.rs::unsupported_architecture_reason is
called unconditionally inside QuantizedGGUFTransformer::from_gguf (GH-704), before
backend selection, and returns Err(FormatError) whenever any tensor name contains
ssm_/ssm.. This is upstream of the CPU/GPU fork — it blocks both.

Ask

Implement a scalar/SIMD CPU Gated DeltaNet (linear attention) forward pass for
GGUF-loaded hybrid layers — recurrent state + short causal conv1d + gated delta rule,
per Qwen3.5/Qwen3-Next's architecture — analogous to what
crates/aprender-serve/src/gpu/scheduler/linear_attn.rs already sketches for the GPU
scheduler (SafeTensors path), but for the GGUF QuantizedGGUFTransformer CPU path.
Until CPU inference exists, consider narrowing the current blanket refusal message so
it doesn't imply CPU already works ("will use CPU" in the CLI's CUDA-fallback log line
is actively misleading for this architecture — see below) — or explicitly state in
the error that neither backend supports it yet, to save users a round trip.
Once a real CPU path lands, extend apr qa/apr validate to report this
architecture's status accurately (load-time capability, not just a runtime crash).

Secondary bug: misleading CLI message

The chat CLI's own log line claims a fallback that doesn't exist:

[GGUF model parse failed: ... will use CPU]

printed by crates/apr-cli/src/commands/chat_load_tokenizers.rs:166
(format!("[GGUF CUDA init failed: {}, will use CPU]", e)). That code path assumes any
GGUF parse failure on the GPU attempt is recoverable by retrying on CPU — true for the
Phi LayerNorm case (#3075), false here, since the same parse failure happens on the
CPU retry too. Worth a small fix regardless of when the real SSM CPU path lands: don't
print "will use CPU" when the failure reason is architecture-unsupported rather than
GPU-capability-specific, since the retry is guaranteed to fail with the identical message.

Evidence

crates/aprender-serve/src/gguf/transformer.rs:73-86,158-167 (refusal call site,
upstream of backend selection)
crates/apr-cli/src/commands/chat_load_tokenizers.rs:166 (misleading "will use CPU" log line)
Repro commands above, no --gpu flag, same host

GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 (GPU support for the same architecture — companion ticket)
Phi-2/Phi-3 GGUF never GPU-eligible: LayerNorm kernel exists but isn't wired into forward_gpu_resident #3075 (Phi LayerNorm — contrast case where CPU fallback genuinely works)
Docs: publish a GPU-vs-CPU model architecture support table (no user-facing doc exists today) #3077 (GPU/CPU model support table doc) — needs a row/footnote for qwen35

Triage (0.67.0 train, PMAT-1098 — decided plan, not a question):

Confirmed on lambda with a local Qwen3.5-0.8B-Q4_K_M.gguf (arch qwen35, 24 layers = 18 ssm_* Gated-DeltaNet + 6 gated full-attention): the published 0.66.0 apr run refuses it from unsupported_architecture_reason before backend selection, exactly as described. Qwen3.8-* files in the wild carry general.architecture=qwen35; this repo treats them as one architecture.

Split into three deliverables:

Ships in 0.67.0 — the honest refusal: the message says neither CPU nor GPU implements Gated DeltaNet yet and names GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090/CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091; the CLI stops printing "will use CPU" for a format-level refusal and exits non-zero on the first attempt; the qwen3_5 row in arch_constraints_fallback.rs describes the real hybrid; apr qa/apr validate report it as a structured load-time capability. In progress on branch PMAT-1098-qwen35-honest-refusal.
CPU Gated DeltaNet in the GGUF path (CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091 ask 1) — in progress on PMAT-1098-qwen35-cpu-deltanet with greedy-token parity against llama.cpp master on the 0.8B fixture as the gate. It ships in 0.67.0 only if parity is green before the 2026-09-12T18:00Z cut; otherwise it moves to 0.68.0 with the measured gap recorded here.
GPU path (GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090) — after (2) exists, because the parity oracle for the GPU scheduler is the CPU implementation; scheduled for 0.68.0 unless (2) lands early.

Dissent recorded: adding two P1s two days before a train's cut is exactly what the schedule's "slip first" rule is for; the honest refusal is the part that cannot slip because today's message actively misleads.

Ask 2 (honest refusal, no phantom CPU fallback) is in PR #3099 with the measured acceptance on the 0.8B fixture — the message now says neither backend implements Gated DeltaNet yet, the CLI exits non-zero on the first attempt, and apr qa reports it as a structured capability failure. Ask 1 (CPU Gated DeltaNet) is in progress on branch PMAT-1098-qwen35-cpu-deltanet with llama.cpp-master greedy parity as the gate; ships in 0.67.0 only if green before the 2026-09-12T18:00Z cut.

Measured status of ask 1 (CPU Gated DeltaNet), 2026-09-10 16:42Z on lambda: the first implementation pass did not reach a working path. The tree it left builds and its one new qwen35 unit test passes, but apr run ~/models/Qwen3.5-0.8B-Q4_K_M.gguf --prompt "The capital of France is" --max-tokens 16 still exits 8 with the new honest refusal ("NEITHER the CPU nor the GPU backend implements Gated DeltaNet/SSM layers yet"), i.e. the loader never routes qwen35 layers to a DeltaNet forward. llama.cpp master runs the same file, so the oracle for the parity gate is in place. The honest refusal (ask 2) is PR #3099 and ships in 0.67.0 either way; the CPU forward path gets a second pass now and ships in 0.67.0 only if greedy-token parity with llama.cpp is green before the 2026-09-12T18:00Z cut, otherwise in 0.68.0 — the measurement will be posted here.

If anyone needs a working CPU implementation of the Qwen3.5/3.8 Gated DeltaNet/SSM forward pass as a reference for the Rust port, I recently implemented this in pure C99 for Project Zero (we parse the qwen35 hybrid layers natively from GGUF).

The scalar and SIMD state-gating math might be useful to look at when sketching out the QuantizedGGUFTransformer CPU path. Happy to share notes on the short causal conv1d handling if it helps get this unblocked!

@shifulegend

@shifulegend if you have anything that you have that could help here that would be incredible. Thank you so much for being aware on the progress we are making!

Ask 1 status, measured 2026-09-11 ~05:10Z on lambda. The CPU path works end to end in draft #3114 (PMAT-1098). Stacked on #3099 (refusal) and #3110 (Q5_K).

apr run ~/models/Qwen3.5-0.8B-Q4_K_M.gguf --prompt "What is 2+2? Answer in one word." → Four.. For "The capital of France is" it gives The capital of France is **Paris**.. apr chat answers coherently. The dense Qwen2.5 path is unchanged.
Root causes. The failed first pass had nine defects, all located by comparing each op against llama-eval-callback's per-node sums:
- DeltaNet: q/k L2 norm was global and applied twice, instead of per head.
- DeltaNet: RoPE was applied, but llama.cpp applies none there.
- DeltaNet: a stray L2 norm on the output.
- DeltaNet: beta went through sigmoid twice.
- DeltaNet: conv output went through SiLU twice.
- Attention: full-width GPT-J RoPE instead of partial NEOX (n_rot 64 of 256).
- Attention: the output gate was never applied.
- Attention: head width taken as 128 instead of 256.
- Q5_K weights decoded with an invented layout (attn_qkv and ssm_out are Q5_K). Fixed in fix(quantize): Q5_K readers decoded an invented layout — every Q5_K tensor on the CPU path and in apr import was wrong (PMAT-1101) #3110.
Unit tests. New tests cover per-head L2, the sigmoid gate and partial NEOX RoPE. Reverting either fix turns them red.

The 0.67 gate (greedy 32-token parity with llama.cpp master): NOT green as written. From examples/qwen35_parity.rs, recorded in evidence/parity/qwen35/lambda/parity.json:

prompt	free-running greedy	teacher-forced	disagreement
0 (5 tokens)	diverges at step 0	27/28	llama.cpp's token is realizar's rank 2, gap 0.145 logits
1 (10 tokens)	identical, 23/23	23/23	none
2 (16 tokens)	diverges at step 13	16/17	rank 2, gap 0.185

Both divergences are near-ties. As a check, I made realizar emulate llama.cpp's CPU precision: Q8_K/Q8_0 activation rounding before each matmul, plus f16 K, V and q. That does not remove the near-ties; it moves them. Teacher-forced agreement drops to 63/68, with new rank-2 misses at other steps (gaps 0.016 to 0.24). They are noise-level near-ties, not a defect (the same pattern as #2359).

Consequence under the decided plan. At the 0.67 freeze this moves to 0.68.0, with the gap above recorded, unless one of two things happens by then:

strict greedy goes green, or
the operator accepts the margin-aware gate the example implements, which PASSES: llama.cpp's token must be realizar's rank ≤ 2 within 0.25 logits.

Recommendation: accept the margin-aware gate. Exact greedy equality across two backends with different accumulation precision is a coin flip at a near-tie. The teacher-forced table shows realizar tracks llama.cpp everywhere else.

Dissent recorded: the plan fixed the gate in advance precisely so it could not be relaxed after the fact.

slipped_from: 0.67.0 → 0.68.0 (freeze 2026-09-12T06:00Z, 06x-release-schedule §4.1).

Gate of record was "ships in 0.67 only if greedy 32-token parity with llama.cpp is green". Measured on draft #3114 (branch PMAT-1098-qwen35-cpu, fixture Qwen3.5-0.8B-Q4_K_M.gguf): strict greedy diverges on 2 of 3 prompts at near-ties — llama.cpp's token is realizar's rank 2 with logit gaps 0.145 / 0.185; teacher-forced agreement 66/68; emulating llama.cpp's activation precision moved the near-ties (63/68) rather than removing them. The margin-aware gate in examples/qwen35_parity.rs (rank ≤ 2, gap ≤ 0.25) passes. apr run and apr chat work on the CPU path.

Not shipped in 0.67.0 because the strict gate is red and no one has accepted the margin-aware gate as the contract. If the margin gate is accepted (comment here), #3114 is the PR to un-draft for 0.68.0; the GPU path is #3090.

release(0.67): train A — 17 row PRs of the 0.67.0 train in one build …

merge origin/main — the Q5_K conflicts are main's own renames (FALSIF…

Reprioritized to P0 / milestone 0.68.2 (interrupt release, EPIC #3477), pulled forward from 0.70.0. This is one of the two Qwen3.5 tickets the operator named explicitly as must-happen for the next release, along with GPU (#3090). Its blocker (#3432, the tensor_byte_size/TRAITS fix) is being pulled forward to 0.68.2 alongside it so the dependency doesn't strand this ticket again the way the 0.68.0 no-slip commitment did.

Measured 2026-09-18 on two hosts (main `4a538dd` + published 0.68.1, both `--features cuda`)

apr run and apr chat on Qwen3.5-0.8B-Q4_K_M: correct output on CPU, lambda-vector and gx10. --gpu prints [qwen35: Gated DeltaNet runs on the CPU … (#3090)] and is correct.
Still refused by the tooling: apr qa Capability Match FAIL with the pre-feat(qwen35): Qwen3.5 / Qwen3.8 hybrid GGUFs run on the CPU — apr run + apr chat (#3091) #3114 text ("NEITHER the CPU nor the GPU backend implements…"), 8 of 12 gates SKIP, exit 5 — on both hosts, both binaries. Same predicate is called from crates/apr-cli/src/commands/qa_capability.rs:98, parity_refusal.rs:157 and ptx_map.rs:441; only the loader (gguf/transformer.rs:163) was fixed. That is the second refusal site PP-QUANT-001 M4: tensor_byte_size and ONE refusal site read TRAITS — closes the #3091-reopen / #3341 class for all 35 ids at once #3432 must collapse.
The UD-IQ2_XXS / UD-IQ4_XS crash in the reopen is not exercised here — neither host holds such a quant. Do not close on Q4_K_M evidence alone; the M3 fixture (PP-QUANT-001 M3: regression fixtures — #1749 #1789 #2535 #3341 (+ #3091 reopen) on tiny synthetic GGUFs, observed RED→GREEN #3429) is the right RED→GREEN for it.
Contract/SHACL requirements for this ticket are on EPIC: release train 0.68.2 (interrupt) — Qwen3 CPU+GPU verification, P0 #3477 (contract rows 5–6, shape 2).

Verified on the real files, lambda-vector, branch PMAT-3477-qwen3-cuda-qk-norm (commits 6181a30 loader table, 530b6d4 IQ dequant, 56a347c apr qa), pinned ./target/release/apr:

apr run Qwen3.5-0.8B-IQ4_XS.gguf --no-gpu --prompt "The capital of France is" --max-tokens 8 → The capital of France is **Paris**. rc=0
apr run Qwen3.5-0.8B-UD-IQ2_XXS.gguf --no-gpu … → The user wants to know about the capital rc=0 (coherent, 2-bit)
apr qa Qwen3.5-0.8B-Q4_K_M.gguf → rc=0, 13 PASS; Capability Match PASS as CPU-only ("Gated DeltaNet runs on the CPU (CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091); the GPU backend has no SSM kernels (GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090), GPU gates skipped"); Golden Output PASS on the CPU forward; no "NEITHER".
Loader: tensor_byte_size reads one 31-row GGML type table (gguf/ggml_type_table.rs, IQ* included); CPU dequant for IQ2_XXS/IQ4_XS (+IQ2_S/IQ3_S/IQ3_XXS) with grid/sign-table invariant tests. PR follows on EPIC: release train 0.68.2 (interrupt) — Qwen3 CPU+GPU verification, P0 #3477.

ci: 355 contract tests were dark, and the target that ran them was br…

contracts(0.68): 14 obligations had no id, and 4 proofs pointed at no…

docs(run): §7 report — clean-room root-caused, 11 PRs unstranded (pai…

feat(qwen35): Qwen3.5 / Qwen3.8 hybrid GGUFs run on the CPU — apr run…

PMAT-3445: no tag while the milestone being cut holds an open item — …

feat(dogfood): model capability ladder — T-2 proves every (architectu…

PMAT-3477: 0.68.2 — Qwen3 CUDA QK-norm restored in the graph and the …

fix(guard): R-2 close-issue guard now fails closed on the paiml#3400 …

paiml#3571 layer 1: apr serve routes the Qwen3.5 hybrid to Qwen35Mode…

release(0.69): batch — 15 receipted PRs + fleet-pv andon fix, one CI …

PMAT-3685: apr parity REFUSES (exit 12) a model the CUDA capability g…

Navigation Menu

Repository navigation

CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091

Summary

Root cause

Ask

Secondary bug: misleading CLI message

Evidence

Related

25 remaining items

Measured 2026-09-18 on two hosts (main `4a538dd` + published 0.68.1, both `--features cuda`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091

Description

Summary

Root cause

Ask

Secondary bug: misleading CLI message

Evidence

Related

Activity

noahgift commented on Sep 10, 2026

noahgift commented on Sep 10, 2026

noahgift commented on Sep 10, 2026

shifulegend commented on Sep 10, 2026

alfredodeza commented on Sep 10, 2026

noahgift commented on Sep 11, 2026

noahgift commented on Sep 12, 2026

25 remaining items

alfredodeza commented on Sep 18, 2026

noahgift commented on Sep 18, 2026

Measured 2026-09-18 on two hosts (main 4a538dd + published 0.68.1, both --features cuda)

noahgift commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured 2026-09-18 on two hosts (main `4a538dd` + published 0.68.1, both `--features cuda`)