Repository navigation
core: a bounded ladder of intermediate prompt-cache checkpoints - #1953
Draft
javierbits wants to merge 1 commit into
Draft
javierbits wants to merge 1 commit into
javierbits wants to merge 1 commit into
Conversation
A long prompt keeps only its history, shared and rewind marks, so a later request that preserves a large prefix after compaction has no state inside that prefix and prefills from the last shared cut. Keep up to `intermediate_most` checkpoints at power-of-two multiples of the ladder's spacing under the history mark. They are admitted after the endpoint and rewind, inside the store's budget and slots, and are exempt from superseded eviction, so they stay across append turns while remaining evictable as ordinary oldest states. Flash Next on CUDA opts in at 8,192-token spacing with four checkpoints, behind `TF_PROMPT_INTERMEDIATE=1` until a served A/B shows the extra marks pay. Host tests cover the ladder plan, a compacted 160K-token prompt resuming at 64K against the 4K shared cut without it, an append turn, admission beside the endpoint, and eviction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1909.
What this changes
A long prompt kept only its history, shared and rewind marks. After a conversation was compacted to a shorter retained prefix, a later request had no saved state inside that prefix and prefilled from the last shared cut.
core/prompt_rules.zigaddsintermediate(spacing),intermediate_most(count) andintermediate_min(length gate).core/prompt_intermediate.zigplans up tointermediate_mostcheckpoints at power-of-two multiples of the spacing under the history mark, away from the resume point and existing marks. Admission is after the endpoint and active rewind, inside the store's budget and slots; no checkpoint is planned when the endpoint itself does not fit.Entry.intermediatemarks a checkpoint position.prompt_cache_policy.victimButno longer chooses such an entry as a superseded victim, so it survives later append turns, while it stays eligible as the ordinary oldest victim.TF_PROMPT_INTERMEDIATE=1until a served A/B shows the extra marks pay.Receipt
Host, this machine (aarch64, Zig 0.17.0):
zig build test: 475/475 steps, 3805 tests passed, 7 skipped, 0 failed.tools/zig/lean_check.py: 424 problems, the same asmain.zig build native -Doptimize=fast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121 -j2: built.The new tests in
zig/src/core/prompt_cache_intermediate_test.zig:Not run: GPU. The machine is shared and another model server holds 95 GiB of 121 GiB, while a Flash Next checkpoint needs about 74 GiB, so no exactness, cold-TTFT, snapshot I/O, append-reuse or compacted-prefix receipt is claimed. The opt-in default stays off until those numbers exist.
atc overlapreports conflicts with #1335, #1394, #1651, #1780 and #1784, and each has the same conflict set againstmainalone (includingCHANGELOG.mdand, for #1335,zig/src/native/cuda_snaps.zig), so this change adds none.