Repository navigation
perf(qwen35-cuda): DP4A requant staleness (#4258) + argmax first-of-ties (#4259) + allocation-free decode plumbing (#4215) — replaces #4322 #4332 - #4439
Merged
noahgift merged 17 commits intoSep 26, 2026
Conversation
…change (Closes #4258) The DP4A Q4_K/Q6_K GEMVs cache their Q8_1 activation behind `q8_activation_valid`. Nothing on the Qwen3.5 decode path cleared it, so every DP4A GEMV after the first reused the first quantized activation across projections, layers and tokens. The cache is now keyed by its source: `ensure_q8_activation` quantizes unless the flag is valid AND the input is the same (pointer, length) that was last quantized. The writers that rewrite a GEMV input in place on the qwen35 path (rmsnorm_into, per_head_rmsnorm_into, residual_add_into, fused_swiglu_into and every gdn_* launch) invalidate it when their output is the cached source. tests_q8_activation_staleness.rs: a DP4A GEMV on a second buffer, and after each of the five writers rewrites its input, must land within Q8_1 rounding of the float GEMV of the new contents, for HwDp4a and MwvDp4a. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 1f14aea) (cherry picked from commit 263958f)
…atastrophic' reading was #4258 The stale-Q8_1 fix (263958f) makes qwen35_cuda_dp4a_gemv_is_catastrophic_through_the_recurrence fail as intended: DP4A now holds the parity contract on 0.8B. Assert that instead, so the stale-activation reuse cannot come back silently. The float pin stays until 2B/4B re-measure (#4030). Refs #4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 555a677)
…max it replaces returns the FIRST (Closes #4259) The block and final tree reductions kept `other` only on strict `>`. After the strided load, thread 0 holds index 768 and thread 1 holds index 1, so on an exact tie the reduction keeps 768; in the final pass a subtree that has absorbed block 128 beats block 1 the same way. `gguf::ops::argmax` (strict `>` in index order) returns the first maximum, so a temp-0 decode moved onto the GPU argmax (#4215 (b)) would emit a different token on every exact tie. `other_wins` = greater OR (equal AND lower index), applied at all three reduction sites. Both comparisons are ordered, so NaN still never wins. New builder ops: `setp_eq_f32`, `or_pred`. Top-k is untouched. tests/argmax_falsification_f114/tie_break.rs: within-block tie, cross-block tie, all-equal, NaN-at-low-index, and a no-tie regression at Qwen3.5's 248320 vocab, each asserted against the CPU first-max contract. APR_REQUIRE_CUDA=1 turns a missing device into a failure instead of a skip. Closes #4259. Cherry-picked from perf/4215 (c8ce988c3). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit e4df9c2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 7e08584)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f08fce5)
…ce and downloads 4 bytes (Refs #4215) Remaining host allocations were the format! module keys in q5k_gemv_into (two DeltaNet projections per layer, 36/token), q4_1_gemv_into and q8_quantize_{async,into}; all now use module_key!. Measured on the 4090: host allocs 1713 -> 0, device allocs 1 -> 0, D2H 993,280 -> 4 bytes/token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f80b3f8)
… main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t — the complexity ratchet read cognitive 40 -> 41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se lookup, not main's #3759 test-build key proof On main, ensure_kernel_module proves every cache hit in test builds, and the proof format!s the KernelType on each launch. Cherry-picked onto main, the #4215 steady-state test counted those allocations and failed its host == 0 assert. Device allocs (0) and D2H (4 B per token) already held. The warmup still runs the proof over every key the token uses. Only the measured window suspends it, through a cfg(test) switch on the ledger. Release builds have no ledger. Also drops a comment that knocked rustfmt's field alignment out of place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
enabled auto-merge
September 25, 2026 15:33
This was referenced Sep 25, 2026
Closed
…easons stated) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…mment had shifted it by 2 check_no_claim_literals keys its baseline by line; the moved line read as a NEW claim. Comment-only (3 lines → 1). Delta quorum 3/3 PASS (sonnet-5 ×2, haiku-4-5; degraded: same-family). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
disabled auto-merge
September 25, 2026 21:56
…al guard ran inside the pool guards serial_alone() only flagged events BETWEEN start/end of l_serial. On a loaded runner the order was a|b|serial-start|serial-end|a-end|b-end: both pool guards wrapped the serial one, the window was empty, and the mutant looked serial (#4439 guard-tree job 108232939859). Count guards still open at the serial start. Case table: new 6/6, old misses both wrap-around orders. The same fix removes a false PASS from row 53. Refs #4439 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Delta quorum @ c3546d3 — 3/3 PASS, The delta is test-only:
The two reds the cop reported:
|
noahgift
enabled auto-merge
September 25, 2026 22:47
noahgift
deleted the
perf/qwen35-dp4a-argmax-decode-replaces-4322-4332
branch
September 26, 2026 00:42
noahgift
added a commit
that referenced
this pull request
Sep 26, 2026
… fat jobs (#4441) Cop ruling: ci.yml uses main's 5-fat-job shape with the car deltas re-applied. - ci.yml = main's; x86-main --sections adds changes + provable-ladder; gate keeps the #3668 docs-only rule, reading changes.outputs.docs_only from x86-main. - ci/sections.yml carries the car's job-body deltas (infra#1059 admit, #3668 changes job, guard-tree moves, guard-cargo docs-only skip, provable-ladder advisory). determinism always runs: its raster upload is if-no-files-found: error. - fat_driver.py exports a section's job outputs in the results JSON. - forward_qwen35_cuda.rs: #4233 decode graph + #4439 argmax/alloc-free hidden_buf; forward_single_greedy stays eager. - roadmap.yaml: union (main's PMAT-4215/4258/4259 added). - check_tree_reader_tests.sh / ci_test_tier.sh read ci/sections.yml. No runs-on / secrets / permissions drift. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #4322 and #4332 (operator ruling via cop: no main-merges into aged PRs; one fresh branch off main, cherry-picked).
Carried (cherry-picked with
-x, not on main or on batch/0.69.4):q8_activation_srckey plusq8_activation_writtenon every writer kernel), with its staleness tests and the DP4A parity guard. Closes Qwen3.5 CUDA: q8_activation_valid is never cleared on the qwen35 path, so DP4A reuses a stale Q8_1 activation — #3513's 'DP4A catastrophic' measured this bug #4258.module_key!pre-formattedArc<str>module keys, stack-array GDN args (gdn_pack_args), device-alloc/D2H counters, andQwen35CudaModel::forward_single_greedy(device argmax, 4-byte download). Refs apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215.Dropped: #4322's #4316 persistent residual. It is superseded by batch/0.69.4 commit 64b20f8.
Reconciliation with main (the judgement calls):
ensure_kernel_module(GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759 key ledger) is kept at every site.module_key!only builds the key.module_key!(…, "…_b{:08x}", …, x.to_bits()), which formats the same key string as main'sf32_bits_tag.fused_kv_hw_dp4a_q4k_gemv_intostays deleted, as it is on main.qwen35_gpu_decodeno longer exists on main (generation goes throughQwen35Session), so the apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215 greedy driverqwen35_gpu_decode_greedyis not carried. Wiringforward_single_greedyintoQwen35Sessionis a follow-up on apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215, which is why this PR only Refs it.CudaExecutor::newinitializes the two new fields withDefault::default(), so the complexity ratchet stays at cognitive 40.Local checks:
cargo check -p aprender-serve --features cuda --lib --profile test: pass.cargo check -p aprender-gpu --lib --profile testand the non-cuda aprender-serve lib check: pass.cargo fmt --all --check: pass.check_complexity_ratchet.sh: pass.make roadmap-aggregate-check: pass.keep-open: #4215 stays open until Qwen35Session greedy decode calls forward_single_greedy (follow-up); #3759 is cited for the key ledger this PR keeps, not changed.
🤖 Generated with Claude Code