Repository navigation
perf(0.69.4): batch: qwen35 persistent embed buffer (#4316) + hot-path allocs + stale q8 (#4258) - #4322
Closed
noahgift wants to merge 11 commits into
Closed
perf(0.69.4): batch: qwen35 persistent embed buffer (#4316) + hot-path allocs + stale q8 (#4258)#4322noahgift wants to merge 11 commits into
noahgift wants to merge 11 commits into
Conversation
…er — no per-token GpuBuffer::from_host (#4316) forward_single allocated a fresh [hidden_dim] device buffer every token (cuMemAlloc + blocking HtoD + cuMemFree) and got a new device pointer each time, which no captured CUDA graph can replay (#4215). The residual is now allocated once in the constructor. Each token uploads its embedding row into it with an in-stream cuMemcpyHtoDAsync (CudaExecutor::upload_into_async), so the copy is ordered after the previous token's kernels and before this token's. The buffer leaves `self` for the length of the token (the layer calls take &mut self) and is restored on every path. Test: qwen35_cuda_the_residual_is_one_buffer_across_tokens pins the device pointer across six tokens and a refused one. Parity stays with qwen35_cuda_forward_single_matches_cpu_logits_end_to_end. Refs #4316, #4249 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4316 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4316 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4316 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4316 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…change (Closes #4258) The DP4A Q4_K/Q6_K GEMVs cache their Q8_1 activation behind `q8_activation_valid`. Nothing on the Qwen3.5 decode path cleared it, so every DP4A GEMV after the first reused the first quantized activation across projections, layers and tokens. The cache is now keyed by its source: `ensure_q8_activation` quantizes unless the flag is valid AND the input is the same (pointer, length) that was last quantized. The writers that rewrite a GEMV input in place on the qwen35 path (rmsnorm_into, per_head_rmsnorm_into, residual_add_into, fused_swiglu_into and every gdn_* launch) invalidate it when their output is the cached source. tests_q8_activation_staleness.rs: a DP4A GEMV on a second buffer, and after each of the five writers rewrites its input, must land within Q8_1 rounding of the float GEMV of the new contents, for HwDp4a and MwvDp4a. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 1f14aea)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 4482ab9)
…atastrophic' reading was #4258 The stale-Q8_1 fix (263958f) makes qwen35_cuda_dp4a_gemv_is_catastrophic_through_the_recurrence fail as intended: DP4A now holds the parity contract on 0.8B. Assert that instead, so the stale-activation reuse cannot come back silently. The float pin stays until 2B/4B re-measure (#4030). Refs #4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n parity thresholds (comment) Refs #4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
Member
Author
guyernest
pushed a commit
to guyernest/aprender
that referenced
this pull request
Sep 29, 2026
…-of-ties (paiml#4259) + allocation-free decode plumbing (paiml#4215) — replaces paiml#4322 paiml#4332 (paiml#4439) * fix(cuda): a DP4A GEMV requantizes when its input buffer or contents change (Closes paiml#4258) The DP4A Q4_K/Q6_K GEMVs cache their Q8_1 activation behind `q8_activation_valid`. Nothing on the Qwen3.5 decode path cleared it, so every DP4A GEMV after the first reused the first quantized activation across projections, layers and tokens. The cache is now keyed by its source: `ensure_q8_activation` quantizes unless the flag is valid AND the input is the same (pointer, length) that was last quantized. The writers that rewrite a GEMV input in place on the qwen35 path (rmsnorm_into, per_head_rmsnorm_into, residual_add_into, fused_swiglu_into and every gdn_* launch) invalidate it when their output is the cached source. tests_q8_activation_staleness.rs: a DP4A GEMV on a second buffer, and after each of the five writers rewrites its input, must land within Q8_1 rounding of the float GEMV of the new contents, for HwDp4a and MwvDp4a. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 1f14aea) (cherry picked from commit 263958f) * test(cuda): pre-allocate every input so no staleness case rides a reused address; document the (ptr,len) key's limit (Refs paiml#4258) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit e9d074b) (cherry picked from commit 56c9f91) * docs(roadmap): register PMAT-4258 (Refs paiml#4258) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 4482ab9) (cherry picked from commit 4d4a784) * test(qwen35-cuda): the DP4A falsifier becomes a parity guard — its 'catastrophic' reading was paiml#4258 The stale-Q8_1 fix (263958f) makes qwen35_cuda_dp4a_gemv_is_catastrophic_through_the_recurrence fail as intended: DP4A now holds the parity contract on 0.8B. Assert that instead, so the stale-activation reuse cannot come back silently. The float pin stays until 2B/4B re-measure (paiml#4030). Refs paiml#4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 555a677) * PMAT-4258: impl receipt — carried from paiml#4302, GPU green on 555a677 Refs paiml#4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 36f1ea7) * PMAT-4258: quorum AGREED 3/3 PASS on 36f1ea7 + stale falsifier name in parity thresholds (comment) Refs paiml#4258 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 3b23f45) * GPU argmax returned the LAST of two exactly-equal maxima; the CPU argmax it replaces returns the FIRST (Closes paiml#4259) The block and final tree reductions kept `other` only on strict `>`. After the strided load, thread 0 holds index 768 and thread 1 holds index 1, so on an exact tie the reduction keeps 768; in the final pass a subtree that has absorbed block 128 beats block 1 the same way. `gguf::ops::argmax` (strict `>` in index order) returns the first maximum, so a temp-0 decode moved onto the GPU argmax (paiml#4215 (b)) would emit a different token on every exact tie. `other_wins` = greater OR (equal AND lower index), applied at all three reduction sites. Both comparisons are ordered, so NaN still never wins. New builder ops: `setp_eq_f32`, `or_pred`. Top-k is untouched. tests/argmax_falsification_f114/tie_break.rs: within-block tie, cross-block tie, all-equal, NaN-at-low-index, and a no-tie regression at Qwen3.5's 248320 vocab, each asserted against the CPU first-max contract. APR_REQUIRE_CUDA=1 turns a missing device into a failure instead of a skip. Closes paiml#4259. Cherry-picked from perf/4215 (c8ce988c3). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit e4df9c2) * [WIP] perf(qwen35): allocation-free greedy decode token (Refs paiml#4215) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 7e08584) * test(qwen35): decode tok/s at 850 bench, ignored (Refs paiml#4215) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f08fce5) * perf(qwen35): a greedy decode token allocates nothing on host or device and downloads 4 bytes (Refs paiml#4215) Remaining host allocations were the format! module keys in q5k_gemv_into (two DeltaNet projections per layer, 36/token), q4_1_gemv_into and q8_quantize_{async,into}; all now use module_key!. Measured on the 4090: host allocs 1713 -> 0, device allocs 1 -> 0, D2H 993,280 -> 4 bytes/token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f80b3f8) * style: rustfmt the paiml#4215 driver re-exports after the cherry-pick onto main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cuda): CudaExecutor::new inits the paiml#4215/paiml#4258 fields with Default — the complexity ratchet read cognitive 40 -> 41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(qwen35-cuda): the paiml#4215 zero-host-alloc token measures the release lookup, not main's paiml#3759 test-build key proof On main, ensure_kernel_module proves every cache hit in test builds, and the proof format!s the KernelType on each launch. Cherry-picked onto main, the paiml#4215 steady-state test counted those allocations and failed its host == 0 assert. Device allocs (0) and D2H (4 B per token) already held. The warmup still runs the proof over every key the token uses. Only the measured window suspends it, through a cfg(test) switch on the ledger. Release builds have no ledger. Also drops a comment that knocked rustfmt's field alignment out of place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: re-run §6 R-2 against the keep-open body (paiml#3759 paiml#4215 are Refs, reasons stated) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(guard): keep par-062's baselined claim line at 257 — the paiml#4215 comment had shifted it by 2 check_no_claim_literals keys its baseline by line; the moved line read as a NEW claim. Comment-only (3 lines → 1). Delta quorum 3/3 PASS (sonnet-5 ×2, haiku-4-5; degraded: same-family). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(guard-tree test): row 55 read an overlap as 'alone' when the serial guard ran inside the pool guards serial_alone() only flagged events BETWEEN start/end of l_serial. On a loaded runner the order was a|b|serial-start|serial-end|a-end|b-end: both pool guards wrapped the serial one, the window was empty, and the mutant looked serial (paiml#4439 guard-tree job 108232939859). Count guards still open at the serial start. Case table: new 6/6, old misses both wrap-around orders. The same fix removes a false PASS from row 53. Refs paiml#4439 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.69.4 perf batch: the ONE open PR for the 0.69.4 perf queue (operator: batch like PRs; cap 10). Workers push commits onto
perf/0694-qwen35-persistent-embed. Do not force-push, and do not open new PRs.qwen35_cudatests queued in gpu-q, not run yetThis PR stays a draft and is not armed until every IN row has its GPU run green and one batch quorum has passed on the final head. It lands after v0.69.3 is tagged.
🤖 Generated with Claude Code