GPU inference paths: missing kernels, NaN, slow arch and host safety in nightly runs #5327

Scope

GPU inference defects found in nightly and qualification runs: missing kernels for some quant types, NaN outputs, slow paths on one architecture, and runs that are unsafe for the host.

Why it matters

These rows are wrong or missing output on real models, or a run that can hurt a shared host. They are user-visible.

Done when

Every row below is fixed with a test or a nightly receipt, or closed with evidence that it no longer reproduces.

Parent epic: #3998.

Rows

• qwen-story-daily cargo-installs apr into the gx10 host's ~/.cargo/bin, overwriting the fleet-bins release binary every beat #4346 — qwen-story-daily cargo-installs apr into the gx10 host's ~/.cargo/bin, overwriting the fleet-bins release binary every beat (branch: fix/4346-qwen-story-no-host-install)
• LADDER-HOST-SAFE: model ladder must never take out its host — resource box + PSI brake, load-once cells, huge CPU cells nightly on gx10, byte budgets #4520 — LADDER-HOST-SAFE: model ladder must never take out its host — resource box + PSI brake, load-once cells, huge CPU cells nightly on gx10, byte budgets (branch: 98/4520-ladder-box)
• 0.70.2 P0: Qwen3.5-35B-A3B-UD-IQ4_XS.gguf capability_match: no CUDA forward for qwen35moe (known failure in 0.70.1) #4665 — 0.70.2 P0: Qwen3.5-35B-A3B-UD-IQ4_XS.gguf capability_match: no CUDA forward for qwen35moe (known failure in 0.70.1) — done when: the ladder row is OK on lambda and gx10. (branch: fix/4665-qwen35moe-forward)
• examples-nightly #4756 v2: one final-tree receipt, cut performance_parity GPU-path work in debug, re-measure the debug-scope change #4765 — examples-nightly examples-nightly: 29 examples fail and 7 time out on main (run 37169058144) #4756 v2: one final-tree receipt, cut performance_parity GPU-path work in debug, re-measure the debug-scope change — done when: one scripts/dogfood_examples.sh --timeout-secs 180 run on the final tree reports the 36 rows
• performance_parity GPU-010 asserts a llama.cpp reference it never measures #5072 — performance_parity GPU-010 asserts a llama.cpp reference it never measures — done when: GPU-010 either measures its comparator on the same config and model, or reports only its own figure with no parity claim; PERF-003's M7 row says which. (branch: fix/5072-gpu010-no-asserted-parity)
• Q2_K prefill dequant PTX gives NaN on a non-finite d/dmin where the CPU oracle gives 0 #5119 — Q2_K prefill dequant PTX gives NaN on a non-finite d/dmin where the CPU oracle gives 0 (branch: fix/5119-q2k-prefill-nan)
• performance_parity GPU-020: generate_optimized reads 0.60-0.85x of generate_with_cache in release (LAB) #5219 — performance_parity GPU-020: generate_optimized reads 0.60-0.85x of generate_with_cache in release (LAB) — done when: in a release run the "optimized" path does not clone weights per token, and GPU-020 reads ≥ 0.90x in 3 of 3 runs. Otherwise GPU-020 reports only its own figure, (branch: fix/5219-optimized-gen-no-weight-clones)
• perf(serve): qwen35 memory parity on 8 GB unified memory: embedding copy, fp16 prewarm, serve RSS (Refs #5254) #5277 — perf(serve): qwen35 memory parity on 8 GB unified memory: embedding copy, fp16 prewarm, serve RSS (Refs ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254)

Each row was a separate ticket, folded here under the 100-open-ticket cap (TRIAGE-100, 2026-10-14) and closed as not planned. Reopen a row's ticket when its work starts; an open PR for it reopens it.

Agent: aprender-clerk

Folded P0 and operator-worded rows (TRIAGE-100, 2026-10-14)

These tickets were folded here by the cop ruling of 2026-10-14 (sub-rows of this master). Each keeps its priority and its owner. The operator text each carried is quoted below exactly as written; if an owner objects, the ticket is reopened.

• 0.70.1 P0: Qwen3 on GB10 (sm_121) is correct but ~30x slow — ship sm_121 SASS, re-test batched prefill #4590 — 0.70.1 P0: Qwen3 on GB10 (sm_121) is correct but ~30x slow — ship sm_121 SASS, re-test batched prefill (P1 owner:la-0.71)

  Operator ruling (2026-09-28): 0.70 de-claims Qwen3 on GB10. apr warns, the release notes list it as a known issue, and only the Qwen3×GB10 cells leave pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715's required set.