Repository navigation
perf(session): poll the cancel token before the decode forward, not after — one wasted forward per cancelled request (0.69.4) #4325
Description
Found by the gemini lane in the #4309 batch quorum (M2b, 15c3032).
In realizar::session::Session::generate, a mid-decode cancel costs one decode forward pass whose logits are thrown away. With CancelToken::with_budget(8), the session emits 8 tokens and runs 8 decode passes (FALSIFY-SERVE-CANCEL-009, crates/aprender-serve/src/api/tests/apr_q4k_cancel_2465.rs). Before M2b, q4k_decode polled at the top of the loop body: its 8 passes produced 9 tokens and none was wasted. The old test documented this as the reason to poll at the top: "the latter costs one wasted forward pass per cancelled request, and on a 7B Q4K model that is not free".
The same change applies to every arch that goes through Session, not just Q4K.
Done when:
- A cancelled run performs exactly
emitted - 1decode passes (the first token comes from prefill). CANCEL-009's prediction is updated to match. - A mutant that moves the poll after the forward turns the test RED.
- The uncancelled token identity is unchanged, covered by the G1 cells.
Not a blocker for 0.69.3: the cost is at most one forward per cancelled request, and it is declared in contracts/apr-serve-cancellation-v1.yaml.
Activity
noahgift commentedon Sep 24, 2026
Fix on branch
PMAT-4325-cancel-poll-before-forward(db61dd8, based on rc/0.69.3-rc.2; Session is not on main/#4318 yet).Session::generatenow polls the cancel token before each decode forward, so a cancelled turn computes no discarded token. FALSIFY-SERVE-CANCEL-009 now requires emitted-1 forwards (7 for 8 emitted). Reverting the fix turns it RED (tested). Cancel and session lib tests green; clippy -D warnings clean.perf(session): poll the cancel token before the decode forward, not a…
noahgift commentedon Sep 25, 2026
Verified on batch/0.69.4-epic-4249 (e573ed6 and later; aprender-ont): the cancel poll sits before the decode forward in
Session::generate. Cancel tests 9/9 pass. Moving the poll after the forward turns FALSIFY-SERVE-CANCEL-009 RED (8 tokens need 7 decode passes); source restored and porcelain clean. Not independently measured: the G1 uncancelled-output identity. Issue stays open until the batch merges.fix(#4318): three code reds at 90f8ec7 — FUSION pins, generate comple…
perf(session): poll the cancel token before the decode forward, not a…
fix(#4318): three code reds at 90f8ec7 — FUSION pins, generate comple…
merge(rex): re-apply rex/001 on top of main: reconcile with the #4655 …
noahgift commentedon Oct 12, 2026
evidence for close: #4655 / 989cb01 (on main), how checked:
crates/aprender-serve/src/session.rspolls the cancel token at the top of the decode loop, before the forward (the comment cites perf(session): poll the cancel token before the decode forward, not after — one wasted forward per cancelled request (0.69.4) #4325).crates/aprender-serve/src/api/tests/apr_q4k_cancel_2465.rs(FALSIFY-SERVE-CANCEL-009) now assertsemitted - 1decode forwards for a cancelled run. The move-the-poll mutant was not re-run here.noahgift commentedon Oct 12, 2026
Cop batch close (FLOW-004 drain, ≤43/d): fix verified on main — merged #4655 ; ancestor of origin/main: 989cb01 . Evidence: #4325 (comment)