apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers

apr chat hard-skips CUDA for every qwen35 model and tells the user the GPU backend does not exist — citing the issue that closed implementing it, one day earlier. The corrected replacement is written, tested, and called by nothing.

Reported 2026-09-20 by Alfredo, running Qwen3.5-4B-UD-Q4_K_XL.gguf:

[qwen35: Gated DeltaNet runs on the CPU; the GPU backend does not implement it yet (#3090)]
You: hey
warning: GPU (CUDA) qwen35 path rejected, falling back to CPU: the CUDA model would not
build: Operation 'qwen35_cuda_upload' not supported: 'qwen35.blk.0.ssm_alpha.weight':
GGML type 1 has no verified GPU GEMV kernel

Two messages, two different reasons, opposite remedies. The first says the feature does not exist — nothing to do but wait. The second says it exists and this file is ineligible — get a different quant. A user reading both reasonably concludes the product is broken.

1. The banner is false, and was known to be false

crates/apr-cli/src/commands/chat_load_tokenizers.rs:140-146:

// #3091: Qwen3.5/Qwen3.8 hybrids have a CPU forward but no GPU one yet (#3090). Skip the
// CUDA attempt instead of failing it, so chat does not report a load error for a model
// it can run.
if mapped.model.architecture() == Some("qwen35") {
    eprintln!("[qwen35: Gated DeltaNet runs on the CPU; the GPU backend does not implement it yet (#3090)]");
    return Ok((None, false));
}

#3090 closed 2026-09-19T11:37:37Z; #3517 ("Qwen3.5 (Gated DeltaNet hybrid) runs on CUDA") merged the same minute. Qwen35CudaModel is referenced in four files on main. The banner cites, as evidence that the feature is missing, the ticket that shipped it.

2. It is not only a stale string — the early return disables the GPU

That return Ok((None, false)) is unconditional for every qwen35 model: no probe, no capability check, no dependence on the file's quantization or on the host. On this path Qwen3.5 — the flagship model of the 0.68.2 release, validated 20/20 on CPU+CUDA — never attempts the GPU at all. The eligibility question Alfredo's second message asks is never reached here; something else in his run reached it, which is why he saw two disagreeing explanations.

3. The fix exists, is tested, and has no callers

crates/aprender-serve/src/gguf/inference/forward/forward_qwen35.rs:1334-1348 defines qwen35_route_notice, whose own doc comment says:

#3477: the GPU case used to print [qwen35: Gated DeltaNet runs on the CPU; the GPU backend does not implement it yet (#3090)] — which is now false: the hybrid runs on the GPU.

It returns the right thing in all three cases: NoCudaBackend → a notice naming the binary, Gpu and Requested → None.

$ git grep -c qwen35_route_notice origin/main -- crates/
origin/main:crates/aprender-serve/src/gguf/inference/forward/forward_qwen35.rs:6

Six references, all inside its own file. Zero callers. Somebody diagnosed this correctly, wrote the replacement, wrote down that the live string is false — and did not change the line that prints it. A facility with a self-test and no caller, and the partial-repair class: the correct diagnosis frozen in a comment three files from the code still emitting the falsehood. Third instance of that class today (cf. #3583's ggml size table, #3573's spec_conformance).

4. What the user actually needed to be told

GGML type 1 is F16 (tensors.rs::ggml_dtype_name NAMES[1]). Unsloth's UD-Q4_K_XL is a dynamic quant: it keeps sensitive tensors — here ssm_alpha.weight — at F16. The GPU GEMV whitelist (gguf/dtype.rs:46) is exactly F32, Q4_0, Q4_1, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K; F16 is outside it, so the upload correctly refuses. The file is fine and the refusal is correct — the message just never says the one useful thing: a plain Q4_K_M build runs on the GPU. Qwen3.5-4B-Q4_K_M.gguf (sha256 00fe7986…) is a validated ladder rung.

done_when

chat_load_tokenizers.rs calls qwen35_route_notice and attempts CUDA for qwen35 like any other architecture; the unconditional early return goes.
A test asserts the two cannot drift again: a qwen35 model on a CUDA-capable binary must reach the upload path. The current shape passes every test precisely because nothing asserts the GPU is tried.
One message per outcome, naming the actionable half — which tensor, which dtype, and that a Q4_K_M build is GPU-eligible. Not two messages with different reasons.
A guard for the class: a pub fn whose doc comment states that live behaviour elsewhere is wrong, with zero callers outside its own module, is a repair that did not land. Cheap to detect, and this is the third one today.

P0 for adoption: the release's flagship model is CPU-only on the chat path, and the product says so in words that read as permanent.

Refs #3090, #3517, #3091, #3077, #3080.

Correction to this issue, measured on the published binary — section 2 was wrong

I claimed "on this path Qwen3.5 never attempts the GPU at all." That is false. Measured on noah-Lambda-Vector (RTX 4090, sm_89) with the crates.io binary apr 0.68.2, Qwen3.5-4B-Q4_K_M.gguf:

$ printf 'Name the capital of Peru in one word.\n/exit\n' | apr chat Qwen3.5-4B-Q4_K_M.gguf --gpu
[qwen35: Gated DeltaNet runs on the CPU; the GPU backend does not implement it yet (#3090)]
You: Backend: GPU (CUDA, NVIDIA GeForce RTX 4090, 24035 MB VRAM) [qwen35 hybrid forward, #3090]

The early return at chat_load_tokenizers.rs:143-145 does skip that CUDA init, but a later path reaches the GPU anyway and the hybrid forward runs there. I read the early return and asserted the consequence without tracing to the end — the same error I corrected in #3594 four hours ago, in the same shape: a true mechanism, a consequence that does not follow.

The defect is smaller in blast radius and sharper in kind than I wrote. It is not "the flagship model is CPU-only on the chat path." It is:

The product prints that the GPU backend does not implement Gated DeltaNet, and then one line later prints that it is running Gated DeltaNet on the GPU. Both lines cite #3090.

A user cannot reconcile those, and the reasonable reading is the first one, because it is printed first and sounds authoritative. That is what happened to the reporter: he concluded his models were wrong and stopped.

Unchanged by this correction:

GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 closed 2026-09-19; the banner cites it as unimplemented. Still false.
qwen35_route_notice still has zero callers outside its own file, and its doc comment still says the live string "is now false". The repair was written and not landed.
The unconditional return Ok((None, false)) for every qwen35 is still wrong as code even though something downstream recovers — a recovery that nobody asserted is a recovery nobody is protecting.

Revised severity: P1, not P0 — nothing is unusable; a correct capability is being advertised as absent. Relabelled accordingly.

done_when item 2 is strengthened by the correction: the test must assert the printed banner against the actual route, not merely that the GPU is reached. Today's behaviour reaches the GPU and prints that it cannot, so any test asserting only "the GPU was used" passes on the broken state. That is the same negative-control failure named on #3594: the arm fires, the verdict is right, the reason is wrong.

Gap at 0.69.0, measured on both GPU hosts (aprender-c7, before any code)

Binary: apr 0.69.0 (225b2a9ab), /mnt/nvme-raid0/targets/ladder-069/release/apr, on lambda (RTX 4090, sm_89) and gx10 (GB10, sm_121). Two-turn chat at greedy:

printf 'My name is Zorblat. What is the capital of Peru? Answer in one word.\nWhat is my name? Answer in one word.\n/exit\n' \
  | apr chat Qwen3.5-<size>-Q4_K_M.gguf --gpu --temperature 0 --max-tokens <200|1024>

size	host	turn 1	turn 2	`Backend: GPU` lines (one per turn)	fallback
0.8B	lambda, gx10	Santo Domingo ✗	Zořblat ✗	2	none
2B	lambda, gx10	Lima	Zorblat	2	none
4B	lambda, gx10	Lima (with 1024 tokens; at 200 the `<think>` block used the whole budget)	Zorblat	2	none
9B	lambda	Lima	Zorblat	1	turn 2: `warning: GPU (CUDA) qwen35 path rejected, falling back to CPU: … CUDA_ERROR_OUT_OF_MEMORY`, 99.7 s
9B	gx10	Lima	Zorblat	2	none
27B	lambda	Lima	Zorblat	2	none

The 0.8B miss is the model, not the backend: --no-gpu gives the same Santo Domingo / Zořblat. 27B on gx10 is not measured yet, because it waits on the host's GPU lock (unified memory).

1. The banner is still false at 0.69.0: every size, both hosts

Every run prints [qwen35: Gated DeltaNet runs on the CPU; the GPU backend does not implement it yet (#3090)] and then Backend: GPU (CUDA, …) [qwen35 hybrid forward, #3090]. That's the correction comment above, now confirmed on sm_121 too.

2. NEW, and this is the root: the Qwen3.5 CUDA model is rebuilt on EVERY turn

The GH-224 session cache (cached_gguf_cuda: Option<OwnedQuantizedModelCuda>) only holds the dense model. try_init_gguf_cuda returns (None, false) for qwen35, so every turn falls through to generate_gguf_cpu_tokens (chat_generate_session_02.rs:302-318). That runs Qwen35Model::create_base_model and then run_qwen35_generate_dispatch, which, per message, does four things:

Qwen35CudaModel::with_max_seq_len: uploads every weight again.
The F2 guard: a sha256 of the whole file (27B: sha256 8328 ms on turn 1, 7562 ms on turn 2).
A token-by-token re-prefill of the entire conversation (qwen35_gpu_decode).
On gx10, the full PTX JIT set again ([GH-480] Patched … sm_121 JIT workaround ×22 per turn).

GPU memory on lambda, 9B, sampled every 500 ms: 1187 → 6772 → 6936 → 1576 → 6772 → 6936 → 1576 MiB. The model is allocated and freed each turn. This is also why the 9B turn-2 OOM → CPU fallback happened: the rebuild re-allocates ~5.4 GB mid-conversation, and anything else on the device at that moment wins. A resident model allocates once, at load.

Timings: the first runs were contended (other apr jobs on the same GPUs), so per-turn seconds are withheld. The re-run is queued under flock /tmp/apr-gpu.lock on both hosts, and before/after per-turn time and memory will be posted with the fix.

What the fix will do (mapped to done_when)

try_init_gguf_cuda stops returning early for qwen35. The chat session holds a resident Qwen3.5 CUDA model plus its decode state, built once. F2 runs once per session. The route line comes from qwen35_route_notice (done_when 1).
A turn whose prompt extends the tokens already processed prefills only the new suffix. Otherwise the state resets, and nothing is truncated silently.
A test asserts the printed banner against the route actually taken, as the correction comment asks (done_when 2).
done_when 3 (one message naming the F16 tensor and pointing at a Q4_K_M build) and 4 (a zero-caller-repair guard) will be scoped in the fragment.

Hosts, binary and raw logs: aprender-c7 scratch; lambda …/scratchpad/m/, gx10 ~/c7-measure/.

PMAT-3595 done_when 3: the upload refusal names the tensor, the dtype…

PMAT-3595: a Qwen3.5 session binds its CUDA context to whatever threa…

PMAT-3571 (2): the generator decodes the context-bounded budget, and …

evidence(PMAT-3595): per-turn time and memory before and after, both …

fold #3791 (+ #3571 unit 1, #3595, #3571 unit 2): APR Q4K GPU serve n…

feat(#3686): a gate for "the corrected replacement has zero callers" …

milestone → 0.72.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: serve/agent/telemetry surface: Agent Ready (#4000)

docs(roadmap): 10 placement corrections pending approval + 6 duplicat…

milestone 0.72.0 → 0.71.0: a CORRECTION per docs/roadmaps/milestone-triage-2026-09-23.md, section Corrections (#4024); operator 2026-09-23 via cf. Reason: apr chat hard-skips CUDA for every qwen35 model: a verb missing on a backend (#3994's bar), not the serve surface

Parked by APR-EPIC-001: over E3 budget (rank 13/79). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).

Navigation Menu

Repository navigation

apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595

1. The banner is false, and was known to be false

2. It is not only a stale string — the early return disables the GPU

3. The fix exists, is tested, and has no callers

4. What the user actually needed to be told

done_when

Correction to this issue, measured on the published binary — section 2 was wrong

Gap at 0.69.0, measured on both GPU hosts (aprender-c7, before any code)

1. The banner is still false at 0.69.0: every size, both hosts

2. NEW, and this is the root: the Qwen3.5 CUDA model is rebuilt on EVERY turn

What the fix will do (mapped to done_when)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595

Description

1. The banner is false, and was known to be false

2. It is not only a stale string — the early return disables the GPU

3. The fix exists, is tested, and has no callers

4. What the user actually needed to be told

done_when

Activity

noahgift commented on Sep 20, 2026

Correction to this issue, measured on the published binary — section 2 was wrong

noahgift commented on Sep 21, 2026

Gap at 0.69.0, measured on both GPU hosts (aprender-c7, before any code)

1. The banner is still false at 0.69.0: every size, both hosts

2. NEW, and this is the root: the Qwen3.5 CUDA model is rebuilt on EVERY turn

What the fix will do (mapped to done_when)

noahgift commented on Sep 23, 2026

noahgift commented on Sep 23, 2026

noahgift commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions