Repository navigation
feat: Keep the Tool Set Fixed per Segment with Tool Groups and the SDK Dispatcher - #17147
Open
Shabonasar wants to merge 30 commits into
Open
Shabonasar wants to merge 30 commits into
Shabonasar wants to merge 30 commits into
Conversation
The per-request web search context (current date/time with milliseconds) was joined before additional_instructions, which carry mostly static text such as the artifacts prompt. Provider-side prefix caches stopped at the timestamp, so the static text after it was recomputed on every request. Move the dynamic context to the end via buildAdditionalInstructions and cover the ordering with unit tests.
…che-order # Conflicts: # packages/api/src/agents/initialize.ts
…e date changes in history The web search runtime context and temporal special variables were anchored to the turn start with millisecond precision. The system message precedes the whole history, so every turn invalidated the cached history and new conversations could not reuse the system prompt (LibreChat-AI#16877). - Web search runtime context states the conversation start in the user's timezone with minute precision; temporal variables resolve against the same persisted anchor. - A user message sent on a later local date than the model last saw gets a <context_update> block in its prompt copy only, derived from stored createdAt values so it renders identically on later turns; compaction boundaries follow the agents SDK formatter. - Optional endpoints.agents.promptMessageTimestamps adds the send time to every user message; the turn's user message is stored with createdAt = turnStartedAt on every save path. - Provider payload stability tests through createRun for OpenAI, an OpenAI-compatible endpoint, Anthropic, OpenRouter, Bedrock and Google.
… routes The browser resolved promptPrefix with its own clock on every send, so turn 1 of a spec-backed conversation carried the raw template while turn 2 carried resolved text in a different system-prompt position. Only Assistants routes resolve on the client now; the server resolves against the conversation start.
… of a conversation
Adds `endpoints.agents.promptCache.stableContext` (default off). Off, the prompt is
built as before. On:
- prompt time (`{{current_datetime}}` and friends, the web search runtime context) is
anchored to the conversation start, and a date change is announced by a
`<context_update>` block at the start of the first user message sent after it;
- the conversation-stable part of `additional_instructions` is merged into the system
instructions for Anthropic and OpenRouter with prompt caching, while memory, RAG and
one-turn queued file context stay in `additional_instructions`, where the SDK moves
them next to the newest user message;
- dynamic tool context goes after the additional instructions;
- the user message is inserted with the turn start as `createdAt`, so later turns render
the same stamp.
`promptMessageTimestamps` (default off) additionally prefixes each user message with the
time it was sent.
The history notes, the instruction split and the lever live in `packages/api`; the
controller only calls them. `applyContextToAgent` accepts the run context as parts and
records which are volatile, so no text is split by matching.
…assified parts Producers (project, memory guard, artifacts, skills catalog, RAG, memory, agent context files, always-apply skills of isolated subagents) contribute keyed parts through contributeContext instead of concatenating strings into additional_instructions. One registry fixes the canonical order and classifies every part, tool context map entries included, as static, conversation or turn. A runtime key the registry does not know counts as turn, so a new producer cannot break the stable prefix by default. assembleSystemPrompt builds the strings handed to the SDK. Once parts exist they are the only source of the tail; a direct write to additional_instructions is ignored with a warning. Without promptCache.stableContext it reproduces the previous request; with it, createRun moves only the turn parts with the prompt-cache tail and merges the rest into the instructions. The assembled text is byte-identical to string concatenation for both values of the lever, guarded by a snapshot recorded on the previous implementation. Isolated subagents with an always-apply skill place the skill ahead of the tool runtime context when stableContext is on.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
…d the SDK dispatcher Behind `endpoints.agents.promptCache.stableContext`, which also stays off, with one warning, while the installed @librechat/agents has no tool dispatcher. Providers serialize `tools` ahead of the system prompt, so switching an MCP server, an action or an agent tool mid-conversation reset the prompt cache from position 0. - Tool groups (`tool:mcp:<server>`, `tool:action:<domain>`, `tool:agent:<toolkit>`) are a turn-context source. The segment anchor stores each group's schemas and MCP instructions; `tools` and the MCP instructions are rendered from it until the next compaction. - The initial set orders MCP servers and their tools by name, other groups as the agent lists them, after the ungrouped tools. A compaction keeps the anchored order, drops groups that are off and appends the new ones. - A group enabled mid-segment is deferred and announced once in the user message with its schemas; the model calls it through the SDK dispatcher (`toolDispatch`), which resolves the call before hooks, approval and remembered approvals, so they see the real tool. An anchored group that is off keeps its schemas with `disabled`; an unreachable one is `unavailable` without an update. - A schema a server changes mid-segment is announced once, in the message sent with the change: the changed parameters, tools that are new (reached through the dispatcher) and tools that are gone. `tools` itself changes at the next anchor. - Tools found by `tool_search` stay deferred and are called through the dispatcher; the host search returns their schemas. - The formatter tool set keeps past dispatcher calls; isolated subagent copies bind the tools as loaded. - MCP servers and their tools load in name order, and MCP instructions are listed by server name; both only with the lever. The instructions of the named servers are read in packages/api, so MCPManager keeps its API. - UI: `resolveDisplayToolCall` routes a dispatcher call to the card and the group header of the tool it ran; the stream reducer keeps `dispatched`. The package depends on the tool dispatcher of @librechat/agents, which is not in a release yet; `package.json` moves to the release that has it.
This was referenced Oct 15, 2026
codegraph-librechat
Bot
added
🗺️ Backend Infra
Shabonasar
marked this pull request as draft
October 16, 2026 06:21
…assified parts Producers (project, memory guard, artifacts, skills catalog, RAG, memory, agent context files, always-apply skills of isolated subagents) contribute keyed parts through contributeContext instead of concatenating strings into additional_instructions. One registry fixes the canonical order and classifies every part, tool context map entries included, as static, conversation or turn. A runtime key the registry does not know counts as turn, so a new producer cannot break the stable prefix by default. assembleSystemPrompt builds the strings handed to the SDK. Once parts exist they are the only source of the tail; a direct write to additional_instructions is ignored with a warning. Without promptCache.stableContext it reproduces the previous request; with it, createRun moves only the turn parts with the prompt-cache tail and merges the rest into the instructions. The assembled text is byte-identical to string concatenation for both values of the lever, guarded by a snapshot recorded on the previous implementation. Isolated subagents with an always-apply skill place the skill ahead of the tool runtime context when stableContext is on.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
…d the SDK dispatcher Behind `endpoints.agents.promptCache.stableContext`, which also stays off, with one warning, while the installed @librechat/agents has no tool dispatcher. Providers serialize `tools` ahead of the system prompt, so switching an MCP server, an action or an agent tool mid-conversation reset the prompt cache from position 0. - Tool groups (`tool:mcp:<server>`, `tool:action:<domain>`, `tool:agent:<toolkit>`) are a turn-context source. The segment anchor stores each group's schemas and MCP instructions; `tools` and the MCP instructions are rendered from it until the next compaction. - The initial set orders MCP servers and their tools by name, other groups as the agent lists them, after the ungrouped tools. A compaction keeps the anchored order, drops groups that are off and appends the new ones. - A group enabled mid-segment is deferred and announced once in the user message with its schemas; the model calls it through the SDK dispatcher (`toolDispatch`), which resolves the call before hooks, approval and remembered approvals, so they see the real tool. An anchored group that is off keeps its schemas with `disabled`; an unreachable one is `unavailable` without an update. - A schema a server changes mid-segment is announced once, in the message sent with the change: the changed parameters, tools that are new (reached through the dispatcher) and tools that are gone. `tools` itself changes at the next anchor. - Tools found by `tool_search` stay deferred and are called through the dispatcher; the host search returns their schemas. - The formatter tool set keeps past dispatcher calls; isolated subagent copies bind the tools as loaded. - MCP servers and their tools load in name order, and MCP instructions are listed by server name; both only with the lever. The instructions of the named servers are read in packages/api, so MCPManager keeps its API. - UI: `resolveDisplayToolCall` routes a dispatcher call to the card and the group header of the tool it ran; the stream reducer keeps `dispatched`. The package depends on the tool dispatcher of @librechat/agents, which is not in a release yet; `package.json` moves to the release that has it.
Shabonasar
marked this pull request as ready for review
October 16, 2026 17:20
The per-request web search context (current date/time with milliseconds) was joined before additional_instructions, which carry mostly static text such as the artifacts prompt. Provider-side prefix caches stopped at the timestamp, so the static text after it was recomputed on every request. Move the dynamic context to the end via buildAdditionalInstructions and cover the ordering with unit tests.
…e date changes in history The web search runtime context and temporal special variables were anchored to the turn start with millisecond precision. The system message precedes the whole history, so every turn invalidated the cached history and new conversations could not reuse the system prompt (LibreChat-AI#16877). - Web search runtime context states the conversation start in the user's timezone with minute precision; temporal variables resolve against the same persisted anchor. - A user message sent on a later local date than the model last saw gets a <context_update> block in its prompt copy only, derived from stored createdAt values so it renders identically on later turns; compaction boundaries follow the agents SDK formatter. - Optional endpoints.agents.promptMessageTimestamps adds the send time to every user message; the turn's user message is stored with createdAt = turnStartedAt on every save path. - Provider payload stability tests through createRun for OpenAI, an OpenAI-compatible endpoint, Anthropic, OpenRouter, Bedrock and Google.
… routes The browser resolved promptPrefix with its own clock on every send, so turn 1 of a spec-backed conversation carried the raw template while turn 2 carried resolved text in a different system-prompt position. Only Assistants routes resolve on the client now; the server resolves against the conversation start.
… of a conversation
Adds `endpoints.agents.promptCache.stableContext` (default off). Off, the prompt is
built as before. On:
- prompt time (`{{current_datetime}}` and friends, the web search runtime context) is
anchored to the conversation start, and a date change is announced by a
`<context_update>` block at the start of the first user message sent after it;
- the conversation-stable part of `additional_instructions` is merged into the system
instructions for Anthropic and OpenRouter with prompt caching, while memory, RAG and
one-turn queued file context stay in `additional_instructions`, where the SDK moves
them next to the newest user message;
- dynamic tool context goes after the additional instructions;
- the user message is inserted with the turn start as `createdAt`, so later turns render
the same stamp.
`promptMessageTimestamps` (default off) additionally prefixes each user message with the
time it was sent.
The history notes, the instruction split and the lever live in `packages/api`; the
controller only calls them. `applyContextToAgent` accepts the run context as parts and
records which are volatile, so no text is split by matching.
…assified parts Producers (project, memory guard, artifacts, skills catalog, RAG, memory, agent context files, always-apply skills of isolated subagents) contribute keyed parts through contributeContext instead of concatenating strings into additional_instructions. One registry fixes the canonical order and classifies every part, tool context map entries included, as static, conversation or turn. A runtime key the registry does not know counts as turn, so a new producer cannot break the stable prefix by default. assembleSystemPrompt builds the strings handed to the SDK. Once parts exist they are the only source of the tail; a direct write to additional_instructions is ignored with a warning. Without promptCache.stableContext it reproduces the previous request; with it, createRun moves only the turn parts with the prompt-cache tail and merges the rest into the instructions. The assembled text is byte-identical to string concatenation for both values of the lever, guarded by a snapshot recorded on the previous implementation. Isolated subagents with an always-apply skill place the skill ahead of the tool runtime context when stableContext is on.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
…d the SDK dispatcher Behind `endpoints.agents.promptCache.stableContext`, which also stays off, with one warning, while the installed @librechat/agents has no tool dispatcher. Providers serialize `tools` ahead of the system prompt, so switching an MCP server, an action or an agent tool mid-conversation reset the prompt cache from position 0. - Tool groups (`tool:mcp:<server>`, `tool:action:<domain>`, `tool:agent:<toolkit>`) are a turn-context source. The segment anchor stores each group's schemas and MCP instructions; `tools` and the MCP instructions are rendered from it until the next compaction. - The initial set orders MCP servers and their tools by name, other groups as the agent lists them, after the ungrouped tools. A compaction keeps the anchored order, drops groups that are off and appends the new ones. - A group enabled mid-segment is deferred and announced once in the user message with its schemas; the model calls it through the SDK dispatcher (`toolDispatch`), which resolves the call before hooks, approval and remembered approvals, so they see the real tool. An anchored group that is off keeps its schemas with `disabled`; an unreachable one is `unavailable` without an update. - A schema a server changes mid-segment is announced once, in the message sent with the change: the changed parameters, tools that are new (reached through the dispatcher) and tools that are gone. `tools` itself changes at the next anchor. - Tools found by `tool_search` stay deferred and are called through the dispatcher; the host search returns their schemas. - The formatter tool set keeps past dispatcher calls; isolated subagent copies bind the tools as loaded. - MCP servers and their tools load in name order, and MCP instructions are listed by server name; both only with the lever. The instructions of the named servers are read in packages/api, so MCPManager keeps its API. - UI: `resolveDisplayToolCall` routes a dispatcher call to the card and the group header of the tool it ran; the stream reducer keeps `dispatched`. The package depends on the tool dispatcher of @librechat/agents, which is not in a release yet; `package.json` moves to the release that has it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Providers serialize
toolsahead of the system prompt, so switching an MCP server, an action or an agent tool in the middle of a conversation resets the prompt cache from the first byte. Withendpoints.agents.promptCache.stableContext(defaultfalse) MCP servers, actions and agent tools become tool groups stored in the segment anchor;toolsand MCP instructions are rendered from it until the next compaction. A group enabled mid-chat is announced once in the user message with its schemas and is called through the SDK dispatchercall_tool, sotoolsnever changes; a group switched off stays bound and is markeddisabled.Related to #17102. Stacked on #17146. Depends on LibreChat-AI/agents#614 (tool dispatcher);
@librechat/agentsmoves to the release that has it, and until then the lever stays off with one warning.How it works
toolsitself changes at the next anchor.tool_searchstay deferred and are called through the dispatcher. MCP instructions are read inpackages/api, soMCPManagerkeeps its API; name sorting applies only with the lever.resolveDisplayToolCall); the stream reducer keepsdispatched.Measurements
Run: group G2 measured on Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through LiteLLM), OpenAI chat and Anthropic Messages, dev / lever off / lever on; the same self-hosted model answers both formats, the real OpenAI and Anthropic APIs were not measured; full method in #17102. With the lever on there is one
toolssha and onesystemsha for all 13 requests and every call runs ascall_tool; the same set of servers chosen in three different orders gives identicaltoolsbytes (in dev and with the lever off they differ).Note: the backend caches in 1600-token blocks, so a request reuses cache only in whole blocks and the first turns of a short conversation (prompt under two blocks) cannot hit; shares are therefore lower on short chats than on long ones.
Type of change
Testing
Enable the lever, start a conversation, switch an MCP server on after the second message, and ask for a tool of that server. The request
toolsstay the same, the announcement is in the user message, and the call shows as the real tool in the UI.Automated tests: the test files this PR adds or changes, run per workspace with
npx jest <files>(mongod-backed suites against a real mongod): packages/api 453, packages/data-provider 4, packages/chat 49, api 486, client 51 tests passing. Files:client.test.js,ToolService.spec.js,Part.registry.test.tsx,dispatched.test.ts,load.spec.ts,toolGroupStability.test.ts,load.spec.ts,promptCache.spec.ts,toolDispatchName.spec.ts,tools.spec.ts,utils.test.ts,instructions.spec.ts,definitions.spec.ts,Part.spec.tsx,steps.spec.ts,toolDispatch.spec.ts.Static checks:
tsc --noEmitin packages/api, packages/data-provider, packages/chat, client; eslint and prettier on the touched files.Screenshots / recordings
The tool card of a dispatched call looks the same as a direct call (a tool enabled mid-chat, called through
call_tool, shown asRan fetch_report in bigtool); screenshot below.Risk / compatibility
Default off is unchanged. With the lever on, the tool order of a new conversation is sorted by name (MCP servers and their tools) instead of the loading order. Requires the SDK release with the dispatcher.
Checklist