Repository navigation
performance_parity GPU-020: generate_optimized reads 0.60-0.85x of generate_with_cache in release (LAB) #5219
Description
crates/aprender-serve/examples/performance_parity.rs GPU-020 (bench_optimized_generation, M17) asserts that GpuModel::generate_optimized is no slower than generate_with_cache: speedup ≥ 0.90x. In a release build on a GPU adapter it reads red on every run.
State: LAB (C324). Owner: aprender-wprodmodels. This check cannot block a merge or a release:
- The only CI job that runs the example is examples-nightly. That job builds in debug, and a debug build runs GPU-001 alone, so GPU-020 never executes there. Run 38192640422 (main @cc71bde029, 2026-10-12) shows
pass aprender-serve::performance_parity. - A red row does not change the example's exit code: the release run below exited 0 with GPU-020 red.
Measured (local --release runs of the example on lambda, code identical to main for GPU-020):
| run | GPU-020 speedup |
|---|---|
| 1 | 0.60x |
| 2 | 0.84x |
| 3 | 0.80x |
| 4 | 0.85x |
The target is 0.90x. In the same runs GPU-019 read 1.89–2.20x and GPU-021 1.28–1.36x, both green.
Lead (not yet a cause). forward_block_incremental_optimized (gpu/scheduler/model_forward_block_gpu.rs) clones every weight matrix (qkv, out, fc1, gate, fc2) and the cached K/V (to_vec) on every token of every layer, to get around &mut self in do_matmul. forward_gpu_incremental_optimized does the same for the LM head. The baseline generate_with_cache path borrows them. The split-borrow route this needs already exists in matmul.rs (IMP-1007).
Done when: in a release run the "optimized" path does not clone weights per token, and GPU-020 reads ≥ 0.90x in 3 of 3 runs. Otherwise GPU-020 reports only its own figure, with no pass verdict.
Activity
fix(gpu): optimized generation lends block weights instead of cloning…
noahgift commentedon Oct 12, 2026
Evidence, branch
fix/5219-optimized-gen-no-weight-clones@ fd4314e (branch only, no PR; slot is a3's call).State: LAB. GPU-020 lives in
performance_parity, which runs only in examples-nightly (debug build, GPU-001 only, a red row never changes the exit code). It cannot block a merge or a release. Owner: aprender-wprodmodels.Cause.
forward_block_incremental_optimizedcloned the qkv/out/fc1/gate/fc2 weights and copied the whole K/V cache (to_vec) on every token for every layer;forward_gpu_incremental_optimizedcloned the LM head per token. The "optimized" path paid more copying than the KV-cached baseline it is compared against.Fix. Weights are taken out with
mem::take, lent todo_matmuland put back (matmul_block_weight); K/V is appended and read in place viaget_valid; same for the LM head. The PMAT-216 transpose check moved verbatim intocheck_lm_head_transposebecause the pre-commit complexity gate refused the touched file on a pre-existing violation (from_apr_weightscognitive 27, unchanged from main).Measured (release example, same box):
Run 3 was under host load (other rows fell too: GPU-010 10 tok/s, GPU-003 0.24 GB/s), so 0.96x is the low reading, not the typical one.
Tests. New
test_generate_optimized_matches_cached_and_returns_weights: on seeded weights with a SwiGLU gate,generate_optimizedproduces the same tokens asgenerate_with_cacheand hands every weight back. Mutation check: dropping the put-back line turns it red (panic on an empty weight).cargo test -p aprender-serve --lib -- gpu: 1698 passed, 0 failed, 8 ignored. fmt andclippy -p aprender-serve --lib --tests -D warningsclean.Done-when per the ticket: no per-token weight clones (met on this branch), and GPU-020 >= 0.90x in 3 of 3 release runs (met here). The ticket stays open until the branch lands.