Activity

1. codegraph-librechat

   added

   🗺️ Backend Infracodegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9)codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9)

   on Oct 14, 2026
2. Shabonasar commented on Oct 15, 2026

   

   Shabonasar

   on Oct 15, 2026

   Author

   More actions

   Status: the series is in final preparation. The PRs will be opened in the order of the plan in this issue over the next hours, starting with the three @librechat/agents PRs; the table in the description will show each one as planned, in progress, open or merged, together with the live cache measurements.
3. Shabonasar commented on Oct 15, 2026

   

   Shabonasar

   on Oct 15, 2026

   Last edited by Shabonasar

   Author

   More actions

   The whole series is open, on current dev (all branches merge cleanly):

   • agents: fix: Keep Per-Conversation Context and the Summary at a Fixed System Position agents#613 -> feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools agents#614, fix: Restore Tool Rounds Exactly as the Live Request Sent Them agents#615
   • LibreChat: fix: Keep the Prompt Prefix Stable Across Turns and Conversations #16369 -> refactor: Assemble the System Prompt from Structured, Kind-Classified Parts #17145 -> feat: Plan the Turn Context at Send Time and Freeze the System Prompt per Segment #17146 -> feat: Keep the Tool Set Fixed per Segment with Tool Groups and the SDK Dispatcher #17147 -> feat: Route Built-in Capabilities Through Tool Groups and Declare Prefix Resets #17148 -> feat: Store Provider Output Items and Pass the Stable Context as the SDK Conversation Block #17149

   Each LibreChat PR is stacked on the previous one, so its diff on GitHub includes the earlier PRs until they merge; the PR's own changes are its non-merge commit(s). #17147 needs LibreChat-AI/agents#614, #17149 needs LibreChat-AI/agents#613 and LibreChat-AI/agents#615; without the SDK release the setting stays off with one warning.

   Cached share of prompt tokens with the whole stack, stableContext on vs dev (Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through LiteLLM), OpenAI chat / Anthropic Messages format on LiteLLM endpoints): plain chat 0% -> 56.5% / 54.9%, MCP and tools toggled mid-chat 17.1% / 28.6% -> 74.1% / 73.7%, compaction 2.2% / 6.1% -> 20.1% / 30.2%. The backend caches in 1600-token blocks. Details in the issue body.
4. rmesser commented on Oct 16, 2026

   

   rmesser

   on Oct 16, 2026

   More actions

   We hit this issue in our use of LibreChat with Claude Opus 5.5, and tested the PRs to see if it resolved our use case. This is the conclusion of that testing:

   The PRs in this stack don't restore the prompt cache across turns for Claude with thinking on. We measured this against the real Anthropic API (LibreChat v0.8.8, @librechat/agents 4.0.1, Claude Opus 5.5 with adaptive thinking, 1-hour prompt caching) by logging the requests between LibreChat and api.anthropic.com.

   Opus 5.5 keeps earlier turns' thinking blocks in the prompt. formatAgentMessages doesn't replay them: it skips think parts, and the stored parts have no signature. So every turn that follows one where the model thought breaks the prefix at the start of that turn's reply. agents#615 leaves this as is on purpose; its liveRestore.test.ts says:

   Restored history does not replay reasoning of earlier turns (unchanged SDK behavior), so the turn boundary is compared with those blocks set aside.

   That comparison passes in the harness, but against the real API those set-aside blocks are where the cache breaks. We rebuilt one turn's follow-up request exactly from the live requests and sent it twice:

   Next-turn request	Cache read	Cache write
   Earlier thinking blocks kept	85,621	525
   Earlier thinking blocks removed (what restore sends)	73,601	11,920
   What LibreChat actually sent	73,601	11,952

   The read stops at the previous turn's first thinking block.

   We also saw the two differences agents#615 fixes: sequential tool rounds merged into one message, and text before a tool call moved into the final answer. On turns where the model thought, fixing those alone doesn't change the result, because the request already differs at the first thinking block, before either of them.

   For Claude models that keep earlier thinking, restore needs to replay the signed thinking and redacted_thinking blocks from earlier turns, as #14547 / LibreChat-AI/agents#371 propose. The harness should also compare turn boundaries with reasoning included for those models. Across our deployments, these re-writes at turn boundaries account for about a quarter of model spend.

   👍React with 👍1Reacted by shars
5. Shabonasar commented on Oct 16, 2026

   

   Shabonasar

   on Oct 16, 2026

   Author

   More actions

   @danny-avila when you have time, could you take a look at this series?

   Provider prompt caches stop at the first changed byte. Today the per-turn timestamp, tools toggled mid-chat and the compaction summary rendered in two places break the prefix early. With endpoints.agents.promptCache.stableContext: true the cached share of prompt tokens goes, dev -> on: plain chat 0% -> 55-57%, MCP/tools toggled mid-chat 17-29% -> 74%, compaction 2-6% -> 20-30% (self-hosted vLLM through LiteLLM; the real provider APIs were not measured). The setting is off by default, and then requests are unchanged.

   I opened the stack without claiming it first, sorry about that. I'm happy to split it, cut it down or close the parts you don't want; #17145-#17149 are in draft until the first two are reviewed. Suggested order: LibreChat-AI/agents#613, then #16369, then up the stack.

   Would you be open to reviewing it, and could this issue be assigned to me?
6. Shabonasar commented on Oct 16, 2026

   

   Shabonasar

   on Oct 16, 2026

   Author

   More actions

   @rmesser thank you for testing the stack against the real API and for the exact request comparison. That is the measurement my setup could not give: my runs were on a self-hosted model through LiteLLM, without Anthropic thinking.

   You're right: with models that keep earlier turns' thinking, restored history breaks the prefix at the previous turn's first thinking block, and the liveRestore comparison hides it by setting those blocks aside. I'm taking this on.

   @upman already did the core of it in #14547 / LibreChat-AI/agents#371 (signed or encrypted reasoning persisted and replayed only for the originating provider, model and API mode). Rather than writing a second version, I'll integrate that work into this series, keeping your authorship, and reconcile it with LibreChat-AI/agents#615 and #17149, which store Responses provider items on the same path. Then:

   • the harness compares turn boundaries with reasoning included for models that keep earlier thinking, so this case fails instead of passing;
   • I measure it on the real Anthropic API and post the numbers here;
   • until then it is listed as a known limitation in this issue.
7. Shabonasar commented on Oct 16, 2026

   

   Shabonasar

   on Oct 16, 2026

   Author

   More actions

   Update: the issue body now has results on real provider APIs, and the series has two new PRs.

   Added:

   • feat: Replay Signed Reasoning in Restored History agents#619 and feat: Store and Replay Signed Reasoning for Claude Agents with Stable Context #17189 replay Claude's signed thinking blocks in the restored history. This closes the last known gap, the 5-10k tokens re-written after every tool turn that @rmesser measured. They build on feat(agents): restore provider reasoning across runs #14547 and feat(messages): persist provider reasoning across runs agents#371 by @upman, and his commits keep his authorship.
   • New "Scenarios" section: G1-G3 plus tools with thinking.
   • Results on the Anthropic API (Claude Haiku 5.5, Sonnet 5.5, Opus 5.5), OpenAI Responses (gpt-5.4-nano, gpt-5.4-mini) and the Kimi and Z.ai coding plans (kimi-for-coding, GLM-5.2).
   • A kit to repeat the runs: https://github.com/Shabonasar/LibreChat/tree/tools/prompt-cache-measurements-delete_in_2027_q3/prompt-cache-measurements

   All PRs of the series, including the two new ones, are ready for review; merge order and dependencies are in the PR table of the issue body.

   Savings, dev -> series, in uncached prompt tokens over G1-G3:

   • Claude Haiku 5.5: -56%, -58% with replay
   • gpt-5.4-nano: -61%
   • kimi-for-coding: -45%
   • GLM-5.2: -39%

   @rmesser since you hit this with Claude Opus 5.5: our Opus 5.5 run (effort low, MCP tool calls, restart in the middle) goes from 2 misses to 0 with #17189 and LibreChat-AI/agents#619. A re-run on your setup would be very welcome.

   With thinking and tool calls, conversations on Claude Haiku, Sonnet and Opus are 22-24% cheaper and have no cache misses. Claude Haiku 5.5 needs #17062.
8. Shabonasar commented on Oct 18, 2026

   

   Shabonasar

   on Oct 18, 2026

   Author

   More actions

   Update: all PRs of the series are updated to current dev / main and mergeable again; only conflict adaptations, plus one small fix in LibreChat-AI/agents#619 for the new non-streaming Bedrock path (with a test). Claude Haiku 5.5 is recognized by dev since #17201, so #17062 is no longer needed; a short re-run on plain dev + the series with Haiku 5.5 gives the same numbers (tools 87.4% from cache, 0 misses; G2 85.3%, 0 misses).
9. SwDevOp72 commented on Oct 19, 2026

   

   SwDevOp72

   on Oct 19, 2026

   More actions

   Sonnet 5.5 / Haiku 5.5: append-only becomes a correctness requirement, not only a cache optimization

   Thanks for this series. promptMessageTimestamps in particular is exactly what we need: the send time derived from the stored createdAt gives the model the current time for questions like "which meetings do I still have today?" without touching the system prompt.

   One aspect I could not find in the PRs: on Claude Sonnet 5.5 (and, per its docs, Haiku 5.5), thinking blocks are bound to the prefix that produced them. The API checks whether system, tools or any earlier message changed since a block was produced. On accounts created on or after 2026-08-31 this is enforced by default and returns a 400. With the thinking-binding-controls-2026-08-01 beta and prefix_mismatch_behavior: "drop_block" (which LibreChat already sends for models where bindsThinkingBlocks is true), the affected blocks are dropped instead. Anthropic recommends keeping conversations append-only and describes drop_block as a stopgap.

   For prefix caching, a part that moves every turn only costs the tail. For these models it costs more: every thinking block after the changed position is dropped, so the model loses its earlier reasoning on each turn, which Anthropic notes can raise cost per task.

   Two places where this still seems to happen, if I read the descriptions correctly:

   1. fix: Keep the Prompt Prefix Stable Across Turns and Conversations #16369: memory, RAG and one-turn queued file context stay in additional_instructions, "where the SDK moves them next to the newest user message". Whenever one of these parts is present, each turn removes it from its previous position.
   2. feat: Plan the Turn Context at Send Time and Freeze the System Prompt per Segment #17146 stores retrieved context with its user message, which looks right. The target layout in Tracking: stable prompt prefix cache across turns, conversations and providers #17102, however, shows [D3] u3 on turn 3 and u3 without D3 on turn 4. Is D rendered again at its original position on later turns, or only on its own turn?

   Suggestion: for models where bindsThinkingBlocks is true and stableContext is on, treat the invariant strictly: each request is a byte-for-byte extension of the previous one up to the last assistant message, and nothing disappears from an earlier position. Per-turn parts would stay stored on their message and render at the same place on every later turn, like contextUpdates. Logging the input_transformations that the API returns with the beta header would also let operators see when blocks get dropped.

   This is based on the Anthropic docs and the PR descriptions, not on a measurement against the real API.

   Docs:

   • https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5
   • https://platform.claude.com/docs/en/build-with-claude/preserved-thinking
   • https://platform.claude.com/docs/en/models/haiku-5-5/whats-new-haiku-5-5