P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens

Batched CUDA decode produces garbage output for every m > 1. The slots served from a batch emit a constant token to the cap, never emit a stop token, and therefore always run to max_tokens.

Found while root-causing F-BATCH-001's RED (#2747). The RED was real; the cause was not serialization.

Evidence

One concurrent round, with [PMAT-044] Batch m=3 done in the server log proving the batched path engaged:

slot	path	output	finish	tokens
A	m=1 fast path	coherent English	`stop`	319
B, C, D	served from the batch	`!!!!…` (token id 0 to cap)	`length`	400

Same binary, same model, same prompt, same second — only the path differs. At m=4 the garbage takes other forms (strarstrarstrar…, 001.CompilerServicesétique…). Reproduced on qwen2.5-coder-0.5b with m=3 slots garbage and m=1 slots fine in the same run, so it is not model-specific.

What this invalidates

PERF-001's credited "3.32x aggregate at N=8, ~1100 tok/s" is throughput of garbage tokens. Its completed status in docs/roadmaps/roadmap.yaml needs revisiting on those grounds, not merely on F-BATCH-001 being RED.

It also explains the shape that made the serialization reading look plausible. The index is flat from c=2 (2.45 / 2.29 / 2.39) because every batched band hits the same max_tokens cap regardless of c. The recorded numbers refuted serialization before any new run: with wall(1)=1.20s the band walls are 1.20 / 2.94 / 2.75 / 2.87, so wall(2) > wall(4). Adding a client cannot make a serialized server faster; the quantity charged at c=2 is a constant. Two independent recordings agree on that direction.

The constant has two parts. The correctness defect above contributes 400/319 = 1.25x. A real per-token path penalty contributes the rest: c=1 replays a captured CUDA decode graph while c>1 launches eagerly, and the batched loop retakes model.write(), rescans for recycle/join, and embeds and try_sends per slot per token — measured at 2.93 ms/tok fast versus 5.01 ms/tok batched, 1.71x token-normalised. Neither part scales with m, which is the entire flat shape.

The honest restatement: aggregate throughput at c=2 is 0.82x of a single client. c=4 and c=8 "pass" only because the threshold rises with c while a constant does not.

Why nothing caught it

contracts/continuous-batching-v1.yaml already forbids this exactly. CB-008 requires "All M slots produce distinct tokens per decode step (not constant)" — a verbatim description of the observed signature. CB-006's test: field read "probador correctness at c=1 and c=4, compare token-level output": a sentence, not a command. Nothing ever ran it.

Two more reasons it stayed invisible:

F-BATCH-002's filter --lib batch_admission selected 0 tests. Both schedulers are #[cfg(feature="cuda")] at their mod line, so the admission predicate was a dark target. It has been moved out and now selects 4 tests in every build.
The harness F-BATCH-001 names cannot have produced its recorded numbers. At both ce712eae0 and 50d2bc2bb, scripts/perf000_serialization_probe.sh runs the 7B model at max_tokens=64, one replicate, no warmup or quiesce — against a receipt recording 1.5B / 400 / N=3. It never counts a token, so it could not have seen the 319-vs-400 divergence that dominates the ratio it reported.

Landed on `feat/v1-batchred` (`087e7ca89`)

CB-006 made runnable as scripts/perf041_batched_parity_probe.{sh,py}, classifying 0 PASS / 1 code defect / 2 UNMEASURABLE. It returns 2 when no m>1 batch formed (two staggered requests make two m=1 batches) rather than passing vacuously. Plus a §4.4-conformant harness that counts tokens and prints the token-normalised index beside the wall-clock one, and APR_FORCE_BATCHED_PATH as a real knob.

Mutations, all observed: four on the admission predicate, each RED then GREEN on revert. On the probe, ok = got == ref_prefix → ok = True gives a GREEN exit 0 on identical garbage, reverting gives RED; and PREFIX_CHARS = 40 → 0 exits 2, not a false pass, because the positive control rejects an empty reference.

scripts/perf-matrix.yaml untouched. Binary attribution: apr 0.64.0 (3eb2dae6a) == HEAD at measurement time.

Not fixed here

This ticket diagnosed and made the defect detectable. The garbage itself is unfixed — the batched CUDA decode kernel needs its own repair, and that is an architectural change rather than a guard.

Root cause found and fixed. And the framing of this issue — mine — was wrong.

This was never an `m > 1` bug

With APR_FORCE_BATCHED_PATH=1 and a single request, no batch at all, output is !!!! (token id 0) to the cap. The m=1 fast path is healthy only because it runs a different attention kernel (paged/incremental.rs), which computes new_max into a fresh register.

That single observation excludes every candidate mechanism I listed when dispatching this: per-slot KV offset, batch-dimension stride, logits row selection, RoPE index, slot-to-request mapping. The correlation with m > 1 was real and entirely incidental — batching is simply what routes you onto the broken kernel.

The mechanism

BatchedIncrementalAttentionKernel::build_ptx (crates/aprender-gpu/src/kernels/attention/paged/batched.rs) wrote let old_max = max_score;. In a PTX builder that binds the same VirtualReg, so the in-place max clobbered it. Emitted PTX, verbatim:

max.f32 %f8, %f8, %f23;     // running max, in place
sub.f32 %f27, %f8, %f8;     // "old_max - new_max" — a register minus ITSELF
ex2.approx.f32 %f29, %f28;  // correction == exp2(0) == 1.0, every position

The online-softmax correction was a compile-time 1.0, so sum_exp and the output accumulators were never rescaled onto the new max. Nothing overflows and nothing NaNs — the running max and exp_score are both still correct — which is exactly why it survived. Only the rescale is missing, so every term accumulated before a max increase keeps a weight too large by an unbounded factor.

The already-repaired sibling flash_decoding/chunk_kernel.rs carries a comment describing this precise hazard. paged/batched.rs was the one sibling that still had it.

The mutation, and the mechanism as a number

Reverting the one-line fix:

batched_incremental_attention_rescales_online_softmax ... FAILED
  online-softmax rescale computes `%f8 - %f8` ... left: "%f8" right: "%f8"
batched_attention_matches_cpu_softmax_when_the_running_max_grows ... FAILED
  slot 0 head 0 dim 0 = 4.5, CPU softmax reference = 7.5247297
flash_decoding_chunk_kernel_stays_green ... ok

4.5 is the bug expressed as a number. With V[p] = p+1 over eight positions, 4.5 is their unweighted mean: pinning the correction to 1.0 collapses the weights toward uniform, so the kernel returns an average instead of a softmax.

CB-008 — which described this signature all along and was never run — is now discharged by two layers: a PTX-codegen test needing no GPU, and a GPU numerical test whose K is constructed so the running max grows at every position. A flat score sequence passes with the bug in place, so the input is part of the assertion rather than incidental to it.

Still RED: CB-006

The fix changed the output — different wrong tokens, so the mechanism engaged — but did not repair it. Re-run at apr 0.64.0 (e4e2e3da9), RTX 4090:

batched path PROVEN engaged: max batch m=2 / m=4
FALSIFY-CB-006 RED: 6 batched output(s) diverge from the m=1 reference

Constraints on the residue, all measured rather than reasoned:

not m-dependent — one request through the batched path is still garbage
not prefill — the multi-prompt prefill and the sequential per-slot prefill are different implementations writing the batched KV cache differently, and at m=4 they produce token-for-token identical decode output (3156, then 46539). Two unrelated writers agreeing exactly locates the fault in the reader.
not the GEMM dispatch alone — BATCHED_DP4A=0 routes decode through cuBLAS and it stays wrong, at a different wrong token, so the swap engaged

Two further defects surfaced while bisecting and are filed separately.

On #2711

I warned this lane that #2711 might be a partial cause or partial fix. It is neither. #2711 is purely additive — one crate::andon::record_admission(...) call at the top of each spawn_* — and touches neither process_cuda_batch, per-slot handling, nor the decode loop. Zero file overlap. No sequencing needed.

Fixed on feat/v9-batchfix (bac214f87).

PERF-050: widen the CB-008 GPU fixture to production shape — the old …

PERF-050: the decode-step attention WIRING is correct too — a second …

PERF-050: FALSIFY-CB-009 executed — the prefill→decode KV handoff is …

PERF-050: record round 2 — four exclusions, one tension, and the allo…

Round 2 — second defect not found; the residue is now bounded by proofs rather than by suspicion

Four components proven correct, each with an observed mutation rather than an inspection:

Component	Mutation → RED
Attention kernel at production shape	`9` vs reference `15.11563`
Decode-step attention wiring	lengths `pos+1 → pos`; `k_ptrs` slot stride
Prefill→decode KV handoff	slot offset → `0`
Prefill itself	two implementations agree token-for-token (round 1)

The round-1 fixture could not have caught what it claimed to

Worth recording before the rest. The GPU fixture written in round 1 used 2 heads / 1 KV head with identical Q per head and identical K/V per slot — so a GQA head-mapping fault or a slot-stride fault would have passed it silently. It has been rebuilt at the shape this issue actually reproduces at (12 heads over 2 KV heads, head_dim 128, 2048 cache, M=3, per-slot seq_lens [17, 11, 5]) with asymmetric Q and K/V, and it now checks its own fixture is non-inert before asserting.

The wiring test deliberately runs KV_SLOTS=8 > M=3 so the padded pointer arrays are under test. The handoff test uses TARGET_SLOT=2 across 3 layers, compares by to_bits(), and separately asserts peer slots are untouched — the failure mode that matters at c>1 and that an equality-only test cannot see. The cb009 assertion was refactored for the complexity gate and the mutation was re-run in the refactored scope rather than carried over.

A contradiction, stated rather than smoothed over

prefill_eager — the fast path's prefill, which produces coherent English — calls the same two per-layer functions batched decode calls: batched_qkv_rope_phase and batched_attn_ffn_phase. At m=4 with BATCHED_DP4A=0 the GEMM routing matches as well (use_cublas fires at m≥4 in decode, m=16 in prefill). The only remaining difference is the attention branch — and that branch is now proven correct in isolation.

One of those statements must be wrong. Recorded in the contract as a tension to resolve, not as an unfinished search.

The lead: output depends on an allocation size it does not index

Sweep at m=1 through the batched path, apr 0.64.0 (82c50cfe5):

default                                    '!!!!...'  (token id 0)
CUBLAS_GEMM_THRESHOLD=1                    'ICENSE.0:num==...'
CUDA_MAX_BATCH=1                           '!!!!...'
CUBLAS_GEMM_THRESHOLD=1 CUDA_MAX_BATCH=1   '!!!!...'

Rows 2 and 4 have identical GEMM routing and differ only in how many KV slots were allocated — 32 versus 1, 3584 MB versus 112 MB — and they produce different output.

Decode output must not depend on the size of an allocation it does not index. That is the signature of a read of memory that was never written, and it fits everything else: a constant token id 0 is what an all-zero logits buffer argmaxes to, and the garbage turns into varied text exactly when a different allocation size changes what happens to sit at those addresses.

The instrument this needs

An in-process A/B at m=1 comparing forward_batched_to_logits against forward_gpu_resident(last_token, cache, position) on the same post-prefill state. forward_gpu_resident is the M=1 path that produces coherent output, so it is a true oracle for the same arithmetic on the same KV: a logits comparison localises the divergence to the forward, and a per-layer hidden_buf2 dump bisects it to the layer. It needs batched_kv_stride temporarily zeroed around the call, exactly as add_slot_to_batch already does.

That is the method that took the attention kernel from suspicion to a one-line fix, applied one level up.

feat/v9-batchfix (5d7b293dc), 10 commits. 2575 aprender-gpu lib tests pass; both crates still cargo check without --features cuda.

PERF-050: the in-process A/B — batched decode vs the M=1 oracle on on…

PERF-050: make the parity probe order-independent (its own NaN was a …

PERF-050: per-layer hidden-state trace — the layer bisect, and an unc…

PERF-050: WITHDRAW the round-3 oracle number — the probe was fed a ze…

PERF-050: add a SENSITIVITY control — does the batched forward depend…

PERF-050: record round 3 — the fault is in the batched forward, and i…

Round 3 — the fault is in the batched forward, and it is not a function of its inputs

The headline

CUBLAS_GEMM_THRESHOLD=1   SELF control  cosine 0.104724   <- FAILS

The SELF control runs the batched forward twice, on the same input, on the same state, and requires identical logits. It returns completely different ones.

A forward pass that is not a function of its inputs is reading state that changes between invocations. That single fact explains why all four components proven in round 2 still pass: each of those tests runs one kernel once, with explicit synchronization — precisely the condition under which this defect is invisible. It also subsumes round 2's allocation-size lead, where identical GEMM routing differing only in KV-cache VRAM produced different output.

The measurement, all controls passing

SELF      argmax 15 vs 15      cosine 1.000000   (identical, as required)
SENSITIVE argmax 15 vs 151935  (diverges — the forward does read its input)
ORACLE    argmax 15 vs 13552   cosine 0.719427   both finite
PERTURBED argmax 15 vs 87075   cosine 0.762145   (diverges, as required)

The M=1 oracle returns 13552 on every request and every configuration tried. Prompt, KV cache and prefill are fixed points; only the batched forward moves. CB-006's residue is in the batched forward, downstream of everything round 2 proved correct.

Two more facts of the same class, controls passing each time: all-NaN on the second batched request from the identical input and KV state that gave finite logits on the first, while the oracle answers identically both times; and all-NaN for a sign-flipped embedding of the same magnitude.

A withdrawn result, and the control that was missing

The probe's first run gave ORACLE cosine 0.803758 and all-NaN on request 2. Both were nearly reported. The per-layer trace then showed NaN at layer 0, which cannot coexist with the finite logits the same probe had just printed — and that contradiction sent the investigation back to the call site. The probe ran at the top of batched_decode_step, where embed_buf is still the all-zero buffer; rsqrt(0) is inf and 0 * inf is NaN.

Neither SELF nor PERTURBED could catch that. Both pass whether or not the input means anything. That is a third kind of control — the input is meaningful — and it was absent. The probe now refuses to report on an all-zero embedding and says why.

The NaN was also first mis-attributed to a stale batched_done_mask. Fixing the mask and re-running rather than trusting the explanation is what proved that wrong.

One sub-hypothesis was refuted by reading rather than measurement: cuBLAS is bound to the executor's own stream (ensure_cublas calls handle.set_stream(&self.stream)), so this is not a naive stream mismatch of the CORRECTNESS-012b kind.

What settles it next

The question is now narrow and mechanical: which buffer does the batched forward read that differs between two identical invocations? The probe already runs the forward twice in-process, so the bisect is to dump each stage's inputs on both runs and find the first stage whose input differs. The candidates that exist only on the second call are the lazily-populated FP8/FP16/interleaved conversion caches and the cuBLAS workspace.

feat/v9-batchfix (68d812e1a), 16 commits.

PERF-050 round 4: the stack is deterministic, the output stage is not…

PERF-050: record round 4 in the contract — 28 layers excluded, one hy…

milestone → 0.73.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: batched CUDA decode correctness: must be fixed before 0.73 claims any batched throughput (#3999)

fix(#3418): the CUDA layer graph carries each weight's own ggml id, n…

P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.

merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Navigation Menu

Repository navigation

P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753

Evidence

What this invalidates

Why nothing caught it

Landed on `feat/v1-batchred` (`087e7ca89`)

Not fixed here

Root cause found and fixed. And the framing of this issue — mine — was wrong.

This was never an `m > 1` bug

The mechanism

The mutation, and the mechanism as a number

Still RED: CB-006

On #2711

Round 2 — second defect not found; the residue is now bounded by proofs rather than by suspicion

The round-1 fixture could not have caught what it claimed to

A contradiction, stated rather than smoothed over

The lead: output depends on an allocation size it does not index

The instrument this needs

Round 3 — the fault is in the batched forward, and it is not a function of its inputs

The headline

The measurement, all controls passing

A withdrawn result, and the control that was missing

What settles it next

42 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

P0: batched CUDA decode emits garbage for every m>1 — PERF-001's 3.32x aggregate is throughput of garbage tokens #2753

Description

Evidence

What this invalidates

Why nothing caught it

Landed on feat/v1-batchred (087e7ca89)

Not fixed here

Activity

noahgift commented on Aug 29, 2026

Root cause found and fixed. And the framing of this issue — mine — was wrong.

This was never an m > 1 bug

The mechanism

The mutation, and the mechanism as a number

Still RED: CB-006

On #2711

noahgift commented on Aug 29, 2026

Round 2 — second defect not found; the residue is now bounded by proofs rather than by suspicion

The round-1 fixture could not have caught what it claimed to

A contradiction, stated rather than smoothed over

The lead: output depends on an allocation size it does not index

The instrument this needs

noahgift commented on Aug 29, 2026

Round 3 — the fault is in the batched forward, and it is not a function of its inputs

The headline

The measurement, all controls passing

A withdrawn result, and the control that was missing

What settles it next

42 remaining items

noahgift commented on Sep 23, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Landed on `feat/v1-batchred` (`087e7ca89`)

This was never an `m > 1` bug