Commit accepted speculative prefixes in fixed state pools - #2455
Closed
Tianlei Wu (tianleiwu) wants to merge 7 commits into
Closed
Commit accepted speculative prefixes in fixed state pools#2455Tianlei Wu (tianleiwu) wants to merge 7 commits into
Tianlei Wu (tianleiwu) wants to merge 7 commits into
Conversation
Contributor
Author
This was referenced Aug 23, 2026
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/20260823/gdn-paged-checkpoints
branch
2 times, most recently
from
August 28, 2026 04:09
43b6b78 to
9bc0bbc
Compare
Replace dense recurrent checkpoints with packed convolution values and GDN capsules that replay accepted prefixes into fixed-state banks. Integrate fixed state with paged dynamic batching through explicit manifest bindings and a validated two-phase cache commit.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit c8648296919d85a9fb65949a8953920be3971273)
The packed varlen operators already produce the per-token state series a speculative decoder needs to roll back a partially accepted draft (VarlenCausalConvWithState.prefix_states, GatedDeltaNet.checkpoints), but the builder hardcoded both windows to zero and rejected state_window outright on the paged path. Emit them as a separate third output per operator instead of widening the committed state the way the dense export does, so the Engine's fixed-state bank contract is unchanged and a model exported with a window still runs unmodified when the outputs are not fetched. genai_config carries the window as checkpoint_count plus a checkpoint_alignment, because the conv writes slot j after local token j while GatedDeltaNet writes slot W-1 after the last token.
A speculative step runs more tokens than it ends up keeping. The operators already publish the state after each of those tokens, so rolling back a rejected draft only needs the pool to stage a different source row: FixedStateReservation now binds the checkpoint series when the step asks for it, and CommitPrefix selects the slot for the accepted prefix instead of the step's final state. The slot depends on the group's alignment, which is why the manifest carries it. Capture is opt-in per reservation, so a non-speculative step allocates exactly the staging it did before and every existing caller is unchanged.
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/20260823/gdn-paged-checkpoints
branch
from
August 28, 2026 23:50
c8e0d4a to
0830990
Compare
RequestStepPlan carries a draft count, the composite planner turns any nonzero count into a checkpoint-capturing fixed reservation (and sizes its staging for it), and the hybrid decoder binds the checkpoints output only on such a step, so the operators skip writing it entirely otherwise. Nothing sets a draft count yet, so every step still plans and reserves exactly what it did before.
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/20260823/gdn-paged-prefix-state
branch
from
August 29, 2026 00:02
d4abc79 to
492022d
Compare
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/20260823/gdn-paged-checkpoints
branch
3 times, most recently
from
August 29, 2026 08:27
4631a59 to
1fd351c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack
PR 2/6. Depends on
tlwu/20260823/gdn-paged-checkpoints.Validation
engine_unit_tests322/322 passed