qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule

Measured 2026-09-21 on lambda (RTX 4090), `apr 0.69.0 (225b2a9ab)` --features cuda

Q4_K sweep row: Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf: apr qa rc 5 with no gates in the JSON (capability_match and golden_output both absent); apr run --gpu gives rc 14, "gpu was forced and selected, but its runtime attempt on this model failed and the generation ran on CPU". -v shows Architecture: Transformer [GGUF: qwen3moe] (48 layers) and no CUDA backend line before the refusal. The same file family is on gx10 (Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf on both hosts); gx10's sweep hasn't reached it yet. The CPU MoE path exists (#3367, closed 0.69.0).

Operator, verbatim: "you must ensure all models Q4_K CUDA work; the end. on gx10 and lambda-labs".

done_when

apr run --gpu on both qwen3moe Q4_K_M files runs on CUDA with no fallback on lambda AND gx10, and apr qa capability_match plus golden_output PASS (not skipped). The 30B-A3B at Q4_K_M (~18 GB) must fit the 4090's 24 GB with KV, or the refusal names the memory arithmetic, and the operator rules.
GPU vs CPU parity on the qwen3moe forward (apr parity must no longer refuse it as UNMEASURED-TOOL, apr chat (CPU) is architecture-blind: never enters the MoE path for any qwen3_moe GGUF, and returns rc=0 on the failure #3367's residual) at ≥ 64 positions
apr qa on qwen3moe emits its gates, never an empty report (an empty gate list is a FAIL of qa itself)
both files added as REQUIRED ladder rungs (P0 release-process: the train can publish with "most models working" — the model gate is a hand-picked ladder with OPTIONAL rungs, a skip reads as pass, and the dogfood can be inherited. Every Q4_K model × {lambda, gx10} × CUDA must be green or NOTHING ships #3712)

First step: measure what's missing (routed-expert GEMV on CUDA? gating/top-k? the expert-weight layout?) with apr trace/-v and pmat query "moe cuda forward", and post the gap list here before writing code.

#3714 gap list: measured before any code (aprender-eb)

Binaries. lambda apr 0.69.0 (225b2a9ab), built --features cuda from my worktree (dep-info points at agent-wt/eb-3714, pinned via APR_BIN). gx10 apr 0.69.0 (225b2a9ab), built --features cuda in ~/src/aprender-eb-3714 and run under flock /tmp/apr-gpu.lock choom -n 1000.

host	file	`apr run --gpu -v`	`apr qa --json`	`apr parity`
lambda (4090, sm_89)	Qwen3-30B-A3B-Instruct-2507-Q4_K_M	rc 14; the text ("Paris") is produced on CPU	rc 5, 0-byte JSON	rc 12 REFUSED `architecture=qwen3_moe`
lambda	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	rc 14, same	(same mechanism)	(same)
gx10 (GB10, sm_121)	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	rc 14, same	(same mechanism)	(same)

File facts (gguf-py, both files). 48 layers · hidden 2048 · 32 q heads / 4 kv heads · head_dim 128 · 128 experts, top-8 · expert FFN 768 · rope base 1e7 · router ffn_gate_inp F32 [128×2048] · ffn_gate_exps/ffn_up_exps Q4_K · ffn_down_exps Q4_K ×24 + Q6_K ×24 · attn_v Q4_K ×24 + Q6_K ×24 · output Q6_K · 18.55 GB tensors, of which 17.55 GB are experts. Each expert is contiguous inside the 3-D tensor: gate/up are 884,736 B/expert (Q4_K), down is 1,290,240 B (Q6_K) or 884,736 B (Q4_K). A per-expert device pointer is therefore base + e·stride, with no repacking.

Gaps, each with its mechanism

G1. No GPU attempt is ever made, so rc 14's wording is wrong. crates/aprender-serve/src/infer/inference_result.rs:237-243 sends qwen3_moe to the CPU run_qwen3_moe_generate and hard-codes (tokens, false) // CPU-only path; GPU MoE wiring is M32d follow-up. -v prints no backend reason: there isn't one, because CUDA is never tried. That makes "its runtime attempt on this model failed" false for this arch. Behind it, run_gguf_generate refuses qwen3_moe outright (infer/gguf_gpu_generate.rs:343), and api/cuda_chat_backend.rs:1054,1081 (chat/serve --gpu) also route qwen3moe to the CPU chain.

G2. The CUDA MoE code on main is unreachable and not production-shaped. OwnedQuantizedModelCuda::forward_qwen3_moe_cuda (gguf/cuda/forward_qwen3_moe_cuda.rs:98, M-GPU-MOE-1.x) has no generation caller: only apr bench-moe (commands/bench_moe.rs:144,171) and tests/qwen3_moe_gpu_parity.rs reach it, and no run, chat, serve or qa path does. It runs embed, attention, norms, router, top-k and lm_head on the CPU, and has no KV cache: every token recomputes the whole prefix. Its expert GEMVs go through CudaExecutor::q4k_matvec/q6k_gemv, which call GpuBuffer::from_host(weights) on every call (cuda/executor/execute.rs:288). That is 48×8×3 host→device uploads per token, about 1 GB per token. M-GPU-MOE-3 (#1583) closed with a layer-47 routing divergence parked (47/48 layers ≥ 0.99 cosine). Verdict: use it as a reference, not a foundation.

G3. No device-resident qwen3moe model. The template is Qwen35CudaModel (gguf/cuda/forward_qwen35_cuda.rs, #3090): weights live in the executor's cache, the hidden state stays on the device, it keeps a per-layer KV cache, and there are no syncs inside a layer. The Qwen3 attention qwen3moe needs is a strict subset of its attention layer (drop the q|gate split and the sigmoid gate). The MoE FFN maps onto existing executor primitives:

router: f32_gemv_into (exists)
experts: gemv_dispatch(Q4K|Q6K, base + e·stride, …) (exists; the qtype is per tensor, so the mixed Q4_K/Q6_K ffn_down_exps is handled per layer)
fused_swiglu_into and residual_add_into (exist)
weighted sum Σ w_e·y_e: the aprender-gpu ScaleKernel exists but has no executor wrapper. Scale followed by residual_add is 2 launches per expert, so there's no new kernel for correctness.
softmax, top-8, renorm: host-side in v1, via one 512-byte D2H per layer (48 syncs per token).

→ A correct v1 needs zero new kernels. Where the oxide-first ruling applies is v2 (see G8).

G4. apr qa emits an empty report: a gate error aborts the run. qa_gguf.rs::golden_gate_for sends only cpu_only || hybrid archs to the runtime golden path (run_golden_output_gate_runtime → realizar::run_inference, the MoE-aware one). qwen3moe falls to run_golden_output_gate → golden_output_gguf_cpu → the dense OwnedQuantizedModel::generate_with_cache, which reads the MoE placeholder (EMPTY data buffer (in_dim=2048, out_dim=16384, qtype=0); this is #3367's defect, still present in qa). That error propagates with ? out of run_qa (qa.rs dispatch_gate → runner()?), so no report is serialized: 0-byte JSON, rc 5. Two fixes: (a) route qwen3moe through the runtime golden path, which is a predicate change in qa_gguf.rs, not golden_output.rs/output_verification.rs (0e/fd own those); (b) a gate that errors becomes a FAILED gate row, and the report is always written.

G5. apr parity refuses (commands/parity_refusal.rs:162). It needs the new GPU forward against the CPU forward_qwen3_moe at ≥ 64 positions, plus --per-op rows (aprender-37's harness, #3715 receipt).

G6. VRAM arithmetic. 18.55 GB = 17,691 MiB of tensors. The 4090 has 24,564 MiB, and a KV cache (f32, 2048 positions, 48 layers × 4 kv heads × 128) is 384 MiB. It fits only with the card otherwise empty: during this measurement other sessions held 2.3–16.6 GB on lambda's 4090. The lambda runs need the GPU lock and an idle card. A refusal must name this arithmetic, never OOM. gx10 (GB10, 119 GB unified) has no fit problem but must never run concurrently (rule rev 2).

G7. Not in scope, found on the way: both files get the chat template twice (<|im_start|>user\n<|im_start|>user …). That's #3672, filename-driven, and hits dense *instruct* files too (reproduced on qwen2.5-1.5b-instruct). Not MoE-specific; it's owned by #3672.

G8. v2 performance, where oxide-first applies. The host top-k sync blocks CUDA-graph capture. The oxide candidates (f32/int, not DP4A, so the GO class per experiments/cuda-oxide/README.md) are:

K1 moe_router_topk: softmax, top-8 and renorm on device
K2 moe_scaled_accumulate: fused acc += w[slot]·y, with the weight read from device
K3: expert-pointer gather so the GEMV reads the expert id from device memory

The expert GEMVs stay on the existing DP4A kernels: an oxide Q4K GEMV measured about 4× slower than HwDp4a (same README).

Plan (one PR per row, each receipted before the next)

R1 · CUDA path, correctness first. Qwen3MoeCudaModel built on the GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 pattern, with resident experts and host routing, wired at G1's three sites (run, chat/serve) plus G4(a)/(b). The CPU forward_qwen3_moe is the spec. A differential test runs the GPU forward against the CPU forward at ≥ 64 positions on both hosts. Done when apr run --gpu runs on CUDA on both hosts and qa capability_match and golden_output PASS.
R2 · apr parity for qwen3moe (G5), and the per-op rows for pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715.
R3 · v2 kernels, oxide-first (G8). Each gets a CPU twin and a differential receipt on sm_89 and sm_121.
R4 · ladder rungs (P0 release-process: the train can publish with "most models working" — the model gate is a hand-picked ladder with OPTIONAL rungs, a skip reads as pass, and the dogfood can be inherited. Every Q4_K model × {lambda, gx10} × CUDA must be green or NOTHING ships #3712, with aprender-62).

Risk to track: GPU and CPU accumulation order can flip a near-tie top-8 selection (the M-GPU-MOE-3 layer-47 divergence). The parity metric is logits/argmax at ≥ 64 positions, not routing identity. When routing differs, the receipt names the layer and the CPU logit gap between expert 8 and expert 9.

fix(capacity): on unified memory (GB10) plan against MemAvailable les…

fix(capacity): co-tenant refusal text — no run of spaces; on unified …

#3714 R1: receipt (aprender-eb). Local branch `PMAT-3714-qwen3moe-cuda`, code at `5615e7afe`, not pushed and awaiting its quorum (cop slot 4)

done_when 1 and 3, measured at apr 0.69.0 (5615e7afe) on both hosts, under the GPU lock:

host	file	`apr run --gpu`	`apr qa`
lambda (RTX 4090)	Qwen3-30B-A3B-Instruct-2507-Q4_K_M	rc 0, CUDA, F2 1.0000 on 50 positions, "The capital of France is Paris.", 87 tok/s	rc 0, capability_match PASS, golden_output PASS with 3/3 golden runs on the GPU
lambda	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	rc 0, CUDA, "Paris", 88 tok/s	rc 0, PASS / PASS, 3/3 on the GPU
gx10 (GB10)	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	rc 0, CUDA, "Paris", 36.5 tok/s	rc 0, PASS / PASS, 3/3 on the GPU
gx10	Qwen3-30B-A3B-Instruct-2507	not measured: the file is not on gx10	same

GPU forward vs the CPU specification, with FP32 activations, 64 positions + 1 decode: cosine 1.000000 at every position with 0 argmax mismatches, in 5/5 cells: lambda Coder plain and chat-start, lambda Instruct-2507 chat-start, gx10 Coder plain and chat-start.

The oracle bites. Routing faults are injected in test builds only. After a clean control, every fault is REJECTED by both the exact check and the runtime F2, on both hosts: WrongExpert −0.158, DropTopExpert −0.223, UniformWeights 0.487.

The Q8_K point, now measured on Instruct-2507: the GPU that the Q8_K-judged F2 had rejected at 0.8725 matches the FP32-activation CPU at 1.000000 on the same file. That's the direct measurement #3751 lacked.

Receipt, with every command and number: docs/audits/impl-PMAT-3714-receipt.md on the branch. Open: the Instruct-2507 cell on gx10 (copy the 18.5 GB file, or the operator rules), R2 (apr parity, done_when 2), R4 (ladder rungs, done_when 4), and chat/serve --gpu (a persistent resident model).

PMAT-3714 R1: evidence — Qwen3-30B-A3B-Instruct-2507 on gx10 (copied,…

PMAT-3714 R1: qa's MoE predicate streams the GGUF header instead of p…

PMAT-3714 R1: receipt — the streamed header read and its peak-RSS row…

R2 (done_when 2): `apr parity` measures qwen3moe — lambda verified, gx10 queue-blocked (aprender-eb)

Branch PMAT-3714-r2-on-v2 @ a376d80a7 (rebuilt on R1-v2 01989323d). Receipt: docs/audits/impl-PMAT-3714-r2-receipt.md.

apr parity --assert, 92-token prompt, lambda RTX 4090: Qwen3-Coder-30B-A3B 92/92 pass, min cosine 1.000000, 0 argmax mismatches, 17 s; Qwen3-30B-A3B-Instruct-2507 identical. Evidence evidence/3714/r2-apr-parity-64-lambda.txt.

Two fixes en route:

the real-file test's prompt tokenized short of 64 and failed its own precondition before measuring (f6c84afd7);
apr parity --per-op now refuses the hybrid and MoE arms by name instead of reaching the dense loop's 0-byte MoE placeholders — the pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715 "absence read as conformance" class (341c958af, falsifier per_op_refuses_the_arms_it_has_no_tap_for).

gx10: built (apr 0.69.0 (f6c84afd7)), run queued since 00:47Z. Measured at 01:20Z: two prio-1 gpu-q jobs from another session (serve_e2e.sh) have waited on the flock for 4h18m, so the gx10 lock is backed up independent of this branch. The gx10 row folds in when it lands. R3/R4 unstarted.

PMAT-3714 R2: gx10 measured — both qwen3moe files 92/92 at cosine 1.0…

R2 done_when 2 is met on BOTH hosts (aprender-eb)

The gx10 gpu-q backlog cleared; the run completed 01:33Z. Branch PMAT-3714-r2-on-v2 @ 6f6fa9a83.

host	file	positions	passed	min cosine	wall
lambda RTX 4090	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	92	92	1.000000	17 s
lambda	Qwen3-30B-A3B-Instruct-2507-Q4_K_M	92	92	1.000000	17 s
gx10 GB10	Qwen3-Coder-30B-A3B-Instruct-Q4_K_M	92	92	1.000000	82 s
gx10	Qwen3-30B-A3B-Instruct-2507-Q4_K_M	92	92	1.000000	49 s

apr parity --assert, 92-token prompt, apr 0.69.0 (b4007590f) on lambda / (f6c84afd7) on gx10, both under gpu-q --prio 1. Evidence: evidence/3714/r2-apr-parity-64-{lambda,gx10}.txt. Receipt: docs/audits/impl-PMAT-3714-r2-receipt.md.

Per the cop's ruling (aprender-3e, 2026-09-22), the gx10 row was REQUIRED even though done_when 2 names no host, since the operator's standing rule is that every Q4_K model works on CUDA on gx10 AND lambda. Both are now on the receipt.

Also in this row: the real-file test's prompt tokenized short of 64 and failed its own precondition before measuring (f6c84afd7), and apr parity --per-op now refuses the hybrid and MoE arms by name instead of reaching the dense loop's 0-byte MoE placeholders (341c958af, falsifier per_op_refuses_the_arms_it_has_no_tap_for) — the #3715 "absence read as conformance" class.

Quorum: 0/3, blocked on the pool. R2 stacks on R1 (PMAT-3714-r1-on-3750 @ 01989323d), which stacks on #3711 in batch-1.

Gap list, measured on `apr 0.70.3 (95f64ba14)`, release binary, RTX 4090 and GB10

Each GPU run took the GPU lock. A cell counts as on CUDA only if its Backend: line says GPU (CUDA) and the F2 guard line is present.

done_when	RTX 4090	GB10
`apr run --gpu` on CUDA, no fallback, both Q4_K_M files	PASS. `Backend: GPU (CUDA …) [qwen3moe routed-expert forward]`. F2 guard: 21 positions, min cosine 1.0000. "4" for 2+2	PASS, same lines
`apr qa`: capability_match + golden_output PASS	PASS on both files	PASS on both files
`apr qa` overall	PASS on both files	Coder PASS at 10.1 tok/s. Instruct-2507 rc 5: throughput 5.5 tok/s < 10
`apr parity` at ≥ 64 positions	85/85 on both files, min cosine 1.0, argmax equal	85/85 on both files, min cosine 1.0
REQUIRED ladder rungs (#3712)	Both files match the `q4_k.gguf` inventory, and `deferred: {}`. The long-rung representative is still withdrawn (`cells.withdrawn.representatives.qwen3moe`), and its reason is now false ("this build has NO CUDA forward")	same

Gaps

The GB10 qa throughput is red on the qa side, not the model side. apr run decodes at 39 tok/s on GB10, but qa's runtime gate counts the MoE CUDA build and the F2 guard (~7 s + ~5 s on GB10) as generation time. 6d8d5b1 (2026-09-24, on PMAT-3714-moe-on-070) diagnosed and fixed exactly this, and it never reached main. Next: port it on its own branch, re-measure on GB10.
Prefill is token-by-token. 1,685 prompt tokens take 21.9 s on the 4090 (77.5 tok/s). On GB10 the rate is 39 tok/s, so a 60k rung is ~26 min and consumer-max (148k) is ~62 min per cell. That is the same shape as the 0.70.1 P0: Qwen3 on GB10 (sm_121) is correct but ~30x slow — ship sm_121 SASS, re-test batched prefill #4590 dense de-claim. Fix: a batched prefill() on Qwen3MoeCudaModel from the existing qwen35 prefill pieces (qwen35_project_rows, qwen35_rope_rows, batched_rmsnorm_into, cuBLAS qwen35_prefill_attention, since head_dim 128 has no flash-256 kernel), plus expert grouping with host top-k, gather, per-expert dp4a Q4_K GEMM and weighted scatter-add. No new GEMM kernel.
The KV is f32 (f16_kv_decode_available: false), at 196,608 B/token. On the 4090 that leaves room for ~28k tokens after 17.7 GB of weights, so the 60k and 148k rungs are honest pre-load refusals there, and GB10 owes them.
Restore the long-rung representative only after 1 and 2. Restoring it before 2 owes cells that run for hours.

Refs E1 #3998

Navigation Menu

Repository navigation

qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714

Measured 2026-09-21 on lambda (RTX 4090), `apr 0.69.0 (225b2a9ab)` --features cuda

done_when

#3714 gap list: measured before any code (aprender-eb)

Gaps, each with its mechanism

Plan (one PR per row, each receipted before the next)

#3714 R1: receipt (aprender-eb). Local branch `PMAT-3714-qwen3moe-cuda`, code at `5615e7afe`, not pushed and awaiting its quorum (cop slot 4)

R2 (done_when 2): `apr parity` measures qwen3moe — lambda verified, gx10 queue-blocked (aprender-eb)

R2 done_when 2 is met on BOTH hosts (aprender-eb)

46 remaining items

Gap list, measured on `apr 0.70.3 (95f64ba14)`, release binary, RTX 4090 and GB10

Gaps

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

qwen3moe (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Q4_K_M) has NO working CUDA path: apr run --gpu falls back to CPU (rc 14), apr qa emits no gates — blocks 0.69.1 under the all-Q4_K-CUDA rule #3714

Description

Measured 2026-09-21 on lambda (RTX 4090), apr 0.69.0 (225b2a9ab) --features cuda

done_when

Activity

noahgift commented on Sep 21, 2026

#3714 gap list: measured before any code (aprender-eb)

Gaps, each with its mechanism

Plan (one PR per row, each receipted before the next)

noahgift commented on Sep 21, 2026

#3714 R1: receipt (aprender-eb). Local branch PMAT-3714-qwen3moe-cuda, code at 5615e7afe, not pushed and awaiting its quorum (cop slot 4)

noahgift commented on Sep 22, 2026

R2 (done_when 2): apr parity measures qwen3moe — lambda verified, gx10 queue-blocked (aprender-eb)

noahgift commented on Sep 22, 2026

R2 done_when 2 is met on BOTH hosts (aprender-eb)

46 remaining items

noahgift commented on Oct 9, 2026

Gap list, measured on apr 0.70.3 (95f64ba14), release binary, RTX 4090 and GB10

Gaps

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured 2026-09-21 on lambda (RTX 4090), `apr 0.69.0 (225b2a9ab)` --features cuda

#3714 R1: receipt (aprender-eb). Local branch `PMAT-3714-qwen3moe-cuda`, code at `5615e7afe`, not pushed and awaiting its quorum (cop slot 4)

R2 (done_when 2): `apr parity` measures qwen3moe — lambda verified, gx10 queue-blocked (aprender-eb)

Gap list, measured on `apr 0.70.3 (95f64ba14)`, release binary, RTX 4090 and GB10