Skip to content

perf(qwen35-cuda): DP4A requant staleness (#4258) + argmax first-of-ties (#4259) + allocation-free decode plumbing (#4215) — replaces #4322 #4332 - #4439

Merged
noahgift merged 17 commits into
mainfrom
perf/qwen35-dp4a-argmax-decode-replaces-4322-4332
Sep 26, 2026
Merged

noahgift merged 17 commits into
mainfrom
perf/qwen35-dp4a-argmax-decode-replaces-4322-4332

Conversation

@noahgift

@noahgift noahgift commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Replaces #4322 and #4332 (operator ruling via cop: no main-merges into aged PRs; one fresh branch off main, cherry-picked).

Carried (cherry-picked with -x, not on main or on batch/0.69.4):

Dropped: #4322's #4316 persistent residual. It is superseded by batch/0.69.4 commit 64b20f8.

Reconciliation with main (the judgement calls):

Local checks:

  • cargo check -p aprender-serve --features cuda --lib --profile test: pass.
  • cargo check -p aprender-gpu --lib --profile test and the non-cuda aprender-serve lib check: pass.
  • cargo fmt --all --check: pass.
  • check_complexity_ratchet.sh: pass.
  • make roadmap-aggregate-check: pass.
  • GPU tests have not run yet: the GPU was held by a foreign process. They run under the GPU lock before merge.

keep-open: #4215 stays open until Qwen35Session greedy decode calls forward_single_greedy (follow-up); #3759 is cited for the key ledger this PR keeps, not changed.

🤖 Generated with Claude Code

noahgift and others added 13 commits September 25, 2026 16:15
…change (Closes #4258)

The DP4A Q4_K/Q6_K GEMVs cache their Q8_1 activation behind
`q8_activation_valid`. Nothing on the Qwen3.5 decode path cleared it, so
every DP4A GEMV after the first reused the first quantized activation
across projections, layers and tokens.

The cache is now keyed by its source: `ensure_q8_activation` quantizes
unless the flag is valid AND the input is the same (pointer, length) that
was last quantized. The writers that rewrite a GEMV input in place on the
qwen35 path (rmsnorm_into, per_head_rmsnorm_into, residual_add_into,
fused_swiglu_into and every gdn_* launch) invalidate it when their output
is the cached source.

tests_q8_activation_staleness.rs: a DP4A GEMV on a second buffer, and after
each of the five writers rewrites its input, must land within Q8_1
rounding of the float GEMV of the new contents, for HwDp4a and MwvDp4a.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1f14aea)
(cherry picked from commit 263958f)
…sed address; document the (ptr,len) key's limit (Refs #4258)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e9d074b)
(cherry picked from commit 56c9f91)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4482ab9)
(cherry picked from commit 4d4a784)
…atastrophic' reading was #4258

The stale-Q8_1 fix (263958f) makes qwen35_cuda_dp4a_gemv_is_catastrophic_through_the_recurrence
fail as intended: DP4A now holds the parity contract on 0.8B. Assert that instead, so the
stale-activation reuse cannot come back silently. The float pin stays until 2B/4B re-measure (#4030).

Refs #4258

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 555a677)
Refs #4258

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 36f1ea7)
…n parity thresholds (comment)

Refs #4258

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 3b23f45)
…max it replaces returns the FIRST (Closes #4259)

The block and final tree reductions kept `other` only on strict `>`. After
the strided load, thread 0 holds index 768 and thread 1 holds index 1, so on
an exact tie the reduction keeps 768; in the final pass a subtree that has
absorbed block 128 beats block 1 the same way. `gguf::ops::argmax` (strict
`>` in index order) returns the first maximum, so a temp-0 decode moved onto
the GPU argmax (#4215 (b)) would emit a different token on every exact tie.

`other_wins` = greater OR (equal AND lower index), applied at all three
reduction sites. Both comparisons are ordered, so NaN still never wins. New
builder ops: `setp_eq_f32`, `or_pred`. Top-k is untouched.

tests/argmax_falsification_f114/tie_break.rs: within-block tie, cross-block
tie, all-equal, NaN-at-low-index, and a no-tie regression at Qwen3.5's
248320 vocab, each asserted against the CPU first-max contract.
APR_REQUIRE_CUDA=1 turns a missing device into a failure instead of a skip.

Closes #4259. Cherry-picked from perf/4215 (c8ce988c3).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e4df9c2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 7e08584)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f08fce5)
…ce and downloads 4 bytes (Refs #4215)

Remaining host allocations were the format! module keys in q5k_gemv_into
(two DeltaNet projections per layer, 36/token), q4_1_gemv_into and
q8_quantize_{async,into}; all now use module_key!. Measured on the 4090:
host allocs 1713 -> 0, device allocs 1 -> 0, D2H 993,280 -> 4 bytes/token.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f80b3f8)
… main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t — the complexity ratchet read cognitive 40 -> 41

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se lookup, not main's #3759 test-build key proof

On main, ensure_kernel_module proves every cache hit in test builds, and the
proof format!s the KernelType on each launch. Cherry-picked onto main, the
#4215 steady-state test counted those allocations and failed its host == 0
assert. Device allocs (0) and D2H (4 B per token) already held.

The warmup still runs the proof over every key the token uses. Only the
measured window suspends it, through a cfg(test) switch on the ledger.
Release builds have no ledger. Also drops a comment that knocked rustfmt's
field alignment out of place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…easons stated)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=4439 head=c3546d373fb6a9fe4499493cccb0646c4cd70c4f verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

noahgift and others added 2 commits September 25, 2026 20:51
…mment had shifted it by 2

check_no_claim_literals keys its baseline by line; the moved line read as a
NEW claim. Comment-only (3 lines → 1). Delta quorum 3/3 PASS (sonnet-5 ×2,
haiku-4-5; degraded: same-family).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift
noahgift disabled auto-merge September 25, 2026 21:56
…al guard ran inside the pool guards

serial_alone() only flagged events BETWEEN start/end of l_serial. On a loaded
runner the order was a|b|serial-start|serial-end|a-end|b-end: both pool guards
wrapped the serial one, the window was empty, and the mutant looked serial
(#4439 guard-tree job 108232939859). Count guards still open at the serial
start. Case table: new 6/6, old misses both wrap-around orders. The same fix
removes a false PASS from row 53.

Refs #4439

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

Delta quorum @ c3546d3 — 3/3 PASS, degraded: same-family (apr-m0694). Auto-merge was DISABLED at 0f3b6a2 before this push.

The delta is test-only: scripts/tests/guard_tree_test.sh serial_alone().

lane model verdict
1 claude-sonnet-5 PASS
2 claude-sonnet-5 PASS
3 claude-haiku-4-5-20251001 FAIL on the first run → PASS on the rerun
  • Lane 3 first run: its own awk trace returns bad=1 on the CI sequence. It then read ci-failure.txt as output of the NEW code; it was the pre-fix failure. My prompt did not state that provenance. The rerun added only the provenance line, and both outputs are kept.
  • Lanes: claude -p --permission-mode plan --allowedTools Read,Grep,Glob over a chmod a-w snapshot (/mnt/nvme-raid0/tmp/m0694-q4439-c354). Outputs are in /mnt/nvme-raid0/tmp/m0694-q4439-out/.

The two reds the cop reported:

  1. guard-tree row 55 (job 108232939859) was a real test defect, fixed here. The events were a|b|serial-start|serial-end|a-end|b-end: both pool guards wrapped the serial one, and the old check only looked inside the serial window. Case table: the new check gets 6/6 interleavings right; the old one misses both wrap-around orders. The local run gives 35 checks, 0 failed.
  2. cuda-unit (job 108241779302) is not a code failure. The module_key_guard.rs:88 panic is the expected panic of a test that printed ok. The binary actually died silently mid-test at 20:51:06Z. yoga's journal shows earlyoom SIGTERMing concurrent rustc/cargo processes and this test binary (realizar-52dd89, RSS 1.6 GB) at 20:51:05–06Z. The same test passes locally on the 4090 (1 passed, 12.5 s). Car release(0.70.0): sealed car — REPLACES #4372 #4429's cuda-unit (job 108271525807) died the same way at a different test, killed by the kernel OOM killer at 22:31:19Z.

@noahgift
noahgift enabled auto-merge September 25, 2026 22:47
@noahgift
noahgift added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit cbf623e Sep 26, 2026
25 of 26 checks passed
@noahgift
noahgift deleted the perf/qwen35-dp4a-argmax-decode-replaces-4322-4332 branch September 26, 2026 00:42
noahgift added a commit that referenced this pull request Sep 26, 2026
… fat jobs (#4441)

Cop ruling: ci.yml uses main's 5-fat-job shape with the car deltas re-applied.
- ci.yml = main's; x86-main --sections adds changes + provable-ladder; gate
  keeps the #3668 docs-only rule, reading changes.outputs.docs_only from x86-main.
- ci/sections.yml carries the car's job-body deltas (infra#1059 admit, #3668
  changes job, guard-tree moves, guard-cargo docs-only skip, provable-ladder
  advisory). determinism always runs: its raster upload is if-no-files-found: error.
- fat_driver.py exports a section's job outputs in the results JSON.
- forward_qwen35_cuda.rs: #4233 decode graph + #4439 argmax/alloc-free hidden_buf;
  forward_single_greedy stays eager.
- roadmap.yaml: union (main's PMAT-4215/4258/4259 added).
- check_tree_reader_tests.sh / ci_test_tier.sh read ci/sections.yml.
No runs-on / secrets / permissions drift.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant