Repository navigation
feat: Replay Signed Reasoning in Restored History - #619
Open
Shabonasar wants to merge 7 commits into
Open
Shabonasar wants to merge 7 commits into
Shabonasar wants to merge 7 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
A restored turn has to serialize like the live requests of that turn, or the next turn stops being a prefix continuation at the first difference. Four things drifted: - Sequential tool rounds of one turn were merged into one parallel batch when restored. Rounds now carry a round id and restore as separate assistant messages; text emitted before a call stays with that call. - A Responses tool round restored from stored parts lost the provider items the live request carried (function_call ids and status, reasoning items, message ids) and gained an empty assistant message. After a response with tool calls the graph slices the output per call and hands the slices to the new ON_PROVIDER_OUTPUT_ITEMS event; the content aggregator stores them on the tool_call parts and formatAgentMessages puts them back. They are replayed only to the model and endpoint that produced them, never under zdrEnabled, only when every item is a reasoning, function_call, or message item under a size cap, and reasoning references older than the retention of stored responses are left out. The replay gate takes the current time as an argument. Hosts must route the event to aggregateContent and persist the parts unchanged. - An in-process tool result made of content blocks was stored as JSON and restored as one string. The completion now marks it with outputFormat 'blocks' and restore parses it back into the same blocks. - Anthropic sent a tool_result as a text block only while the tail cache marker sat on it, and OpenRouter with prompt caching did the same for tool messages. A single plain text tool_result is now always sent as its string (Anthropic), and every non-empty tool message as one text block (OpenRouter). Tests: a generic live-versus-restored suite runs real turns against scripted streams in each wire protocol and requires the first request of turn N+1 to start byte for byte with the last request of turn N; Responses specs cover the reasoning variants and the gate branches.
Rebased onto main and LibreChat-AI#615. The OpenAI Responses reasoning carrier (provider_replay.openai_responses) is not ported: on this base Responses reasoning is carried by providerItems (LibreChat-AI#615). The gpt-5.6 prefixed-model regex fix is kept. Co-authored-by: shars <32410570+Shabonasar@users.noreply.github.com>
Rebased onto main and LibreChat-AI#615. OpenAI Responses parts (snapshot dedup, Responses filtering) are not ported: Responses reasoning travels as providerItems on this base. Co-authored-by: shars <32410570+Shabonasar@users.noreply.github.com>
…tion Adapts the persisted provider reasoning from the previous two commits to the tool-round restore of LibreChat-AI#615. - Opt-in on both sides: AgentInputs.providerReasoningReplay turns capture on, formatAgentMessages({ replayProviderReasoning: { provider, model } }) turns restore on. Without them provider_replay is neither produced nor read. - Restore puts the signed blocks at the slot of their THINK part inside the round they belong to (before the text, after the text, between rounds) instead of at the start or end of the message; reasoning that resumes after text gets its own step so the aggregator keeps that position. - Capture records the model that served the call: the attempt stamp of the primary or fallback, then the model the graph resolved for the call (override model, modelName alias), then client options. - Recounts the stored token weight of messages that carry replay when the option is set, since the blocks are not part of it. - Anthropic requests with replay active carry the thinking binding beta and block_binding.prefix_mismatch_behavior=drop_block, and surface the top-level input_transformations of the response in response_metadata; nothing is sent otherwise. - Harness: live and restored requests keep earlier turns' thinking byte for byte (signed, redacted, empty text, interleaved, Anthropic and Bedrock), the replayed history is append-only, and the same scenarios without replay break the turn boundary. Model change and fallback scenarios are covered. - Only the contiguous tail after the last hole is replayed: a turn whose blocks are missing ends the chain, since the provider binds each block to the whole prefix before it. A summary resets the chain; the message that carries the summary keeps the blocks it produced after it. - Blocks whose prefix was rewritten inside a run (compaction, the retained tail after a summary, fading of old tool results) are dropped from the request and withdrawn from the host, so the next turn restores what was actually sent. The summarizer never receives signed reasoning. - Without the opt-in nothing changes: no thinking is injected, and the fallback and summarization filters stay off. Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
Author
|
Merged current Checked: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Claude models with thinking (Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable) keep earlier turns' thinking blocks in the prompt. A restored history has no signed blocks, so the next turn differs from the live request at the first thinking block of the previous turn, and the provider cache re-writes everything after it. This is what @rmesser measured in LibreChat-AI/LibreChat#17102. This PR captures the signed blocks and replays them in the restored history, at their original position, so turn N+1 starts with the bytes of turn N. Result on the real Anthropic API through LibreChat: tool turns stop missing the cache (3 misses -> 0) and a conversation costs 22-24% less.
Builds on the work of @upman in #371 (first two commits, his authorship). Related to LibreChat-AI/LibreChat#17102. Stacked on #615. Opt-in: without
providerReasoningReplay/replayProviderReasoningnothing changes.How it works
thinking-binding-controls-2026-08-01beta andblock_binding.prefix_mismatch_behavior: drop_block; Bedrock retries once without the blocks on a signature error.Measurements
Real Anthropic API through LibreChat (
/api/agents/chat), 8 turns, 3 turns with MCP tool calls, LibreChat process restart after turn 4. "dev" is upstream, "stack" the series withstableContext, "replay" adds this PR and its LibreChat counterpart.Cached share of prompt tokens; a miss is a request that reads less than 98% of what the previous request cached (here 5-10k tokens re-written after each tool turn). With replay the history carries 8-12 signed blocks by turn 8 and survives the restart;
input_transformationsstays empty.Without tool turns the blocks of the final answer are not in the cache yet, so plain chat loses nothing; the gain is on turns where the model thought before calling a tool. G2 (web search and three MCP servers toggled, Haiku 5.5): stack 82.3% with 2 misses, replay 86.3% with 0.
Haiku 5.5 runs used LibreChat-AI/LibreChat#17062; LibreChat
devrecognizes Haiku 5.5 since LibreChat-AI/LibreChat#17201. Method and scripts: LibreChat-AI/LibreChat#17102.Testing
liveRestore.test.ts: for models that keep earlier thinking, turn boundaries are compared with reasoning included (fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615 set it aside); the same scenarios without replay break the boundary. Removing replay fails 12 tests.reasoningReplay.test.ts,rewrittenPrefix.test.ts: model change, fallback, a mid-run rewrite of the prefix and what the host stores.formatAgentMessages.anthropicReplay.test.ts,formatAgentMessages.bedrockReplay.test.ts: position of blocks, holes, summary carrier, withdrawn parts.node.test.ts,ensureThinkingBlock.test.ts,reasoningTypes.test.ts: retained tail after a summary, tool round of the current run after compaction, summarizer input.tsc --noEmitand eslint clean; 1119 tests in the changed files pass.Type of change
Checklist