Repository navigation
feat: Qwen3.5 (qwen35) SSM/Gated Delta Net inference support #3038
Description
Summary
apr run --gpu (and presumably CPU) on a Qwen3.5-family GGUF refuses inference with a clear, intentional error rather than a crash — but the underlying capability (SSM / Gated Delta Net hybrid-attention inference) is still unimplemented. Filing this as a tracked low-priority feature request for actual support, since the current behavior is "fail fast with a good message," not "works."
Repro
apr run --gpu ~/Downloads/Qwen3.8-27B-UD-IQ2_XXS.gguf
=== APR Run ===
Source: /home/alfredo/Downloads/Qwen3.8-27B-UD-IQ2_XXS.gguf
error: Inference failed: Format error: Architecture 'qwen35' uses SSM/Gated Delta Net layers which are not yet supported for inference. Use a standard transformer model (e.g., Qwen2.5, LLaMA, Mistral) or wait for SSM support in a future release.
Note on the filename: the GGUF is named Qwen3.8-27B-UD-IQ2_XXS.gguf (Unsloth "UD" dynamic-quant naming, IQ2_XXS), which doesn't match any published Qwen version number I'm aware of (3, 3.5, 3-Next...). Regardless of what the download is labeled, apr's own GGUF metadata read reports general.architecture = "qwen35", so the routing decision is based on the file's actual metadata, not the filename — worth double-checking against the specific file if anyone picks this up, but I don't think the architecture string itself is wrong.
Is the error message correct?
As far as I can verify from the source, yes — this looks intentional and accurate, not a bug in the message itself:
crates/apr-cli/src/commands/inference_output.rshas a test (tests_2403::inference_context_appears_exactly_once) that pins this exact string:"Format error: Architecture 'qwen35' uses SSM/Gated Delta Net layers ...". That test was added for MCP apr.run / apr.bench: wrong-typed optional arguments silently dropped; doubled error prefix #2403, which fixed a doubled"Inference failed:"prefix — the message content itself predates and survives that fix.- This traces back to apr run: Qwen3.5 (qwen35) inference fails — attn_output.weight not found #704 ("apr run: Qwen3.5 (qwen35) inference fails — attn_output.weight not found"), which originally hit a cryptic
Tensor 'blk.0.attn_output.weight' not foundpanic-ish error. apr run: Qwen3.5 (qwen35) inference fails — attn_output.weight not found #704's "Expected Behavior" section offered two options: (1) actually support Qwen3.5 inference with correct tensor mapping, or (2) surface a clear "unsupported architecture" error instead of the cryptic tensor error. apr run: Qwen3.5 (qwen35) inference fails — attn_output.weight not found #704 was closed by delivering option 2 only — the clear error we see today — not option 1. - So: the message is correct and working as designed; the underlying limitation (no SSM/Gated Delta Net inference path) is the real gap, and appears to still be open.
What already exists (partial groundwork)
Import/convert-side plumbing for Qwen3.5 already exists, which suggests inference is the last missing piece rather than a from-scratch effort:
Architecture::Qwen3_5variant + parsing of"qwen3_5"/"qwen3.5"incrates/apr-cli/src/commands/import.rsTransformerConfig::qwen3_5_9b()incrates/apr-cli/src/commands/model_config.rskernel_explainfamily registration forqwen3_5(crates/apr-cli/src/commands/kernel_explain/mod.rs)- Provable contracts already on file:
contracts/qwen35-shapes-v1.yaml,contracts/qwen35-hybrid-forward-v1.yaml,contracts/qwen35-e2e-verification-v1.yaml(see also closed spec issues Contract: Hybrid layer dispatch + linear attention (Qwen3.5) #294 "Hybrid layer dispatch + linear attention (Qwen3.5)" and Contract: Qwen3.5 shape instantiation + RoPE frequency scaling #296 "Qwen3.5 shape instantiation + RoPE frequency scaling") - Related/adjacent unsupported-architecture work: Architecture coverage gaps: Mamba (SSM) and OpenELM (variable-width) unsupported #266 ("Mamba (SSM) and OpenELM unsupported"), feat: Falcon-H1 hybrid transformer+SSM architecture support #269 ("Falcon-H1 hybrid transformer+SSM architecture support")
Ask
Track actual inference support for Qwen3.5's hybrid SSM / Gated Delta Net layers in realizar (per CLAUDE.md's realizar-first architecture — this is inference, not training). Given the existing contracts/shape work, the main lift is presumably the hybrid layer dispatch + linear-attention forward pass itself.
Marking low priority — this is a niche/newer architecture, current behavior fails safely with a good error, and there's no indication of user-facing urgency beyond "would be nice to run this model."
Environment
- Reported error via
apr run --gpu - Model: locally downloaded GGUF,
Qwen3.8-27B-UD-IQ2_XXS.gguf(filename per Unsloth dynamic-quant convention; GGUF metadata reports architectureqwen35)
Activity
noahgift commentedon Sep 7, 2026
Thanks! Will address in upcoming release shortly. Keep them coming.
docs(PP-066): session 2026-09-08 — tickets minted, PMAT-1062 complete…
noahgift commentedon Sep 8, 2026
Corroborated independently, from a different direction — worth recording on the ticket.
While measuring the L0-1b parity fix we ran the m=1 greedy stream across five local models before/after the change, to prove the fix touched nothing else. Qwen3.5-0.8B refused on both sides, identically: SSM layers unsupported, rc 8. So the refusal you hit on a 27B
qwen35is not size- or quantisation-specific — it is the architecture, it is deterministic, and it is stable across a kernel change that moved every other model's numbers.That is the good half of your report: the refusal is a real, consistent gate rather than a crash or, worse, a plausible-looking wrong answer. It also makes this ticket cheap to verify when it is picked up — the before/after check already has a Qwen3.5 row that must flip from
rc 8to a stream.Staying P3 and unscheduled: 0.66.0 was rescoped today (D-14) to the CUDA parity fix and nothing else, and this is a new capability rather than a defect in a shipped one.
noahgift commentedon Sep 12, 2026
slipped_from: 0.67.0 → 0.68.0 (freeze 2026-09-12T06:00Z). Same feature as #3091 — the CPU forward lives in draft #3114 and ships with the parity gate decision recorded there.
docs(PP-066): session 2026-09-06 — SPEC-2.0 (v2.0, 18-row scope, clai…
noahgift commentedon Sep 15, 2026
Operator ruling 2026-09-15 — this is the 0.68 Qwen 3.5 EPIC
One capability was tracked three times at inconsistent priorities. Consolidated:
Scope (operator): CPU is in 0.68. GPU is in if it lands before the cut.
The gate is the EXISTING parity stack, not a new one
evidence/parity/thresholds.yamlalready carriesmin_cosine: 0.98on logits, onmain, with a measured basis (n=5 over two known-good and two known-bad pairs). That isthe gate, applied to the CPU path against llama.cpp CPU as the
[X]reference.Greedy-token equality is NOT the gate — it fails on legitimate quantization numerics.
And "rank ≤ 2, gap ≤ 0.25" is not a gate either: it has no basis, and §3.3 forbids a
threshold without one. Both are recorded as report-only ledger fields — greedy
agreement and margin, per prompt. A threshold waits for 20 records.
The CPU parity reference must not come from the PR it measures
PR #3114 currently introduces
crates/aprender-serve/examples/qwen35_parity.rsandevidence/parity/qwen35/lambda/parity.json, and both are absent frommain. A numberthat cites only the change under test is not evidence. Build the CPU reference from the
existing GPU methodology — same
thresholds.yamlshape, a CPU host, llama.cpp as[X]—and land it independently of the PR being judged.
Note the gap this fills: the parity stack on
main(
docs/specifications/0.66-performance-parity-report.md,thresholds.yaml,evidence/parity/LEDGER.md,scripts/perf_gate.sh) is GPU-hosted and dense-7B-scoped.There is no CPU-inference parity reference on
mainat all. 68-F1 is GPU-only(yoga + gx10).
Contracts are the definition of support, not a follow-up
#3114 adds 1,133 lines of new forward-pass code and zero contract rows, while three
qwen35 contracts sit un-discharged on
main:contracts/qwen35-hybrid-forward-v1.yaml,contracts/gated-delta-net-v1.yaml,contracts/qwen35-e2e-verification-v1.yaml.Ruling: discharge them with RED→GREEN evidence — as commits on #3114 or as a PR ahead
of it. If the code cannot discharge them, the code is not done. CLAUDE.md Rule 7:
coverage without contracts is rejected.
#3114 stays DRAFT until it carries discharged contract rows. Green required checks do
not override Rule 7 — and its
ciandworkspace-testwere green; what was red wasfleet-wide pin drift (#3291), fixed by #3301 on 2026-09-15T11:09:40Z.
Correction to the Lane B brief (operator)
aprender-canonical-benchmark-rfc.mdwas a Project-upload filename, never a repo path.It resolves to paiml/infra#259 (CLOSED, a different repo, self-declaring "zero
authoritative numerics"). The real stack is the four files named above.
alfredodeza commentedon Sep 18, 2026
Milestone moved to 0.68.2 (interrupt release, EPIC #3477) to track alongside its concrete sub-work #3090 (GPU, now P0) and #3091 (CPU, now P0). Left at P1 here since this issue itself is the umbrella/closure ticket, not new engineering work — closes when #3090 and #3091 do.
noahgift commentedon Sep 18, 2026
Status 2026-09-18, measured on lambda-vector (4090) and gx10 (GB10) with main 4a538dd and the published 0.68.1: CPU inference for
qwen35works (apr run/apr chat, Q4_K_M) — the refusal in this ticket's body is gone from the loader. Remaining: GPU GDN path (#3090), the second refusal site inapr qa/apr check(#3432, still prints the old message), UD-IQ quants (#3091). Full matrix and the contract/SHACL requirements on the epic #3477.noahgift commentedon Sep 18, 2026
slipped_from: 0.68.2 — umbrella for #3090/#3091. #3091 (CPU) ships in 0.68.2 (real UD-IQ2_XXS/IQ4_XS files run, apr qa certifies the CPU rung); #3090 (GPU) is a dated non-goal for 0.68.2 and moves to 0.69.0, so this umbrella moves with it.
noahgift commentedon Sep 20, 2026
Closing: shipped in 0.68.2 (published to crates.io 2026-09-20T07:59:49Z, tag v0.68.2 @ 338f1d4).
The refusal this issue was opened against is gone on both backends, and the capability is measured on the PUBLISHED artifact rather than a working tree —
cargo install aprender --version 0.68.2 --features cuda, private roots, two hosts, 20 cells:GPU (CUDA, NVIDIA GeForce RTX 4090, 24035 MB VRAM) [qwen35 hybrid forward, #3090]Implementation landed across #3090 (GDN device kernels), #3517, #3527 (block_count loader + GQA tiled head mapping), #3432/#3091 (real-file loads), #3413 (dense Qwen3 QK-norm on the fast paths). Held by the per-layer and end-to-end parity contracts (QHF-GPU-008/009, QE2E-DEV-008) and by the 0.68.2 model-capability-ladder receipts on both hosts.
Scope not covered here, tracked separately: 35B-A3B (hybrid GDN + routed MoE) is #3512 in 0.71.0, per the operator's 2026-09-19 ruling that 0.68.2 supports up to 27B and 35B is post-0.70.
Two observability gaps found while validating, both in 0.69: #3541 (the
--no-gpuarm prints noBackend:line) and #3542 (apr runemits no tok/s).