Skip to content

core: a bounded ladder of intermediate prompt-cache checkpoints - #1953

Draft
javierbits wants to merge 1 commit into
ashhart:mainfrom
javierbits:prompt-intermediate-1909
Draft

javierbits wants to merge 1 commit into
ashhart:mainfrom
javierbits:prompt-intermediate-1909

Conversation

@javierbits

Copy link
Copy Markdown
Contributor

Fixes #1909.

What this changes

A long prompt kept only its history, shared and rewind marks. After a conversation was compacted to a shorter retained prefix, a later request had no saved state inside that prefix and prefilled from the last shared cut.

  • core/prompt_rules.zig adds intermediate (spacing), intermediate_most (count) and intermediate_min (length gate).
  • core/prompt_intermediate.zig plans up to intermediate_most checkpoints at power-of-two multiples of the spacing under the history mark, away from the resume point and existing marks. Admission is after the endpoint and active rewind, inside the store's budget and slots; no checkpoint is planned when the endpoint itself does not fit.
  • Entry.intermediate marks a checkpoint position. prompt_cache_policy.victimBut no longer chooses such an entry as a superseded victim, so it survives later append turns, while it stays eligible as the ordinary oldest victim.
  • Flash Next on CUDA opts in at 8,192-token spacing with four checkpoints, behind TF_PROMPT_INTERMEDIATE=1 until a served A/B shows the extra marks pay.

Receipt

Host, this machine (aarch64, Zig 0.17.0):

  • zig build test: 475/475 steps, 3805 tests passed, 7 skipped, 0 failed.
  • tools/zig/lean_check.py: 424 problems, the same as main.
  • zig build native -Doptimize=fast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121 -j2: built.

The new tests in zig/src/core/prompt_cache_intermediate_test.zig:

  • A 160K-token prompt with a shared cut at 4K keeps 32K/64K/128K checkpoints. A 120K-token request that preserves its first 100K tokens resumes at 64K and equals a fresh pass. The same request without the ladder resumes at 4K, which is the missing intermediate prefix from the issue.
  • An append turn leaves the checkpoint in place and a later compacted request still resumes it.
  • Admission keeps the endpoint and rewind ahead of checkpoints under every budget and slot count tried, and respects both.
  • Prompts below 32K and rules without the ladder plan no new mark.

Not run: GPU. The machine is shared and another model server holds 95 GiB of 121 GiB, while a Flash Next checkpoint needs about 74 GiB, so no exactness, cold-TTFT, snapshot I/O, append-reuse or compacted-prefix receipt is claimed. The opt-in default stays off until those numbers exist.

atc overlap reports conflicts with #1335, #1394, #1651, #1780 and #1784, and each has the same conflict set against main alone (including CHANGELOG.md and, for #1335, zig/src/native/cuda_snaps.zig), so this change adds none.

A long prompt keeps only its history, shared and rewind marks, so a
later request that preserves a large prefix after compaction has no
state inside that prefix and prefills from the last shared cut.

Keep up to `intermediate_most` checkpoints at power-of-two multiples
of the ladder's spacing under the history mark. They are admitted
after the endpoint and rewind, inside the store's budget and slots,
and are exempt from superseded eviction, so they stay across append
turns while remaining evictable as ordinary oldest states.

Flash Next on CUDA opts in at 8,192-token spacing with four
checkpoints, behind `TF_PROMPT_INTERMEDIATE=1` until a served A/B
shows the extra marks pay. Host tests cover the ladder plan, a
compacted 160K-token prompt resuming at 64K against the 4K shared cut
without it, an append turn, admission beside the endpoint, and
eviction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prompt cache: bounded intermediate checkpoints for long histories that are later compacted

1 participant