Skip to content

Breaking: list each Google ADK Go MCP toolset once per model step - #306

Open
campoy wants to merge 10 commits into
mainfrom
fix/google-adk-go-mcp-histories-fail-replay-on-adk-2-6
Open

campoy wants to merge 10 commits into
mainfrom
fix/google-adk-go-mcp-histories-fail-replay-on-adk-2-6

Conversation

@campoy

@campoy campoy commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

An agent's MCP toolset (NewMCPToolset) now lists its tools once per model step, at the same point in the step, declares only the current list, and leads ADK to generate the same IDs, on every ADK release. A History recorded with ADK v2.5.0 or earlier now replays with v2.6.0, and the other way round. This includes agents whose instructions or ToolFilter issue Workflow commands, agents run concurrently, and tools the MCP server stops advertising, including runs that are later approved through a signal.

  • ADK v2.5.0 and earlier call the toolset's Tools only in the first model step of an agent run, before the request processors (instructions and the others), and keep that list for the whole run. v2.6.0 calls Tools there in every step and keeps each result (fix(llminternal): re-evaluate toolsets on every step in toolProcessor google/adk-go#785). On every release, ADK calls the toolset's ProcessRequest after the request processors in every step.

  • An agent run is one run of one agent: ADK creates a new invocation for it each time the agent runs. In each agent run, mcpToolset lists in Tools until its ProcessRequest has run, and ProcessRequest reuses that list. From then on, ProcessRequest lists and Tools returns without an Activity. So the first step of every turn lists through Tools, and every later step lists after the request processors. The order is the same on every release, even when an InstructionProvider starts a timer.

  • ToolFilter runs in the first Tools call of each agent run and in each ProcessRequest, not when the list is taken. A filter that issues Workflow commands therefore issues them at the same points on every release. ADK resumes a confirmed tool call from the first Tools list, before any ProcessRequest, so a tool the filter excluded while its approval was pending does not run.

  • The later Tools calls that only v2.6.0 makes return the list the first Tools call of the agent run returned, which is the list v2.5.0 and earlier keep. ADK draws a UUID from the workflow's random stream for each tool it keeps before every model step. So the IDs it generates, including tool confirmation IDs, are the same on every release even when the MCP server or the filter changes the list during the run. An approval that carries a recorded confirmation ID, for example from a signal, matches on replay with either release.

  • The toolset tells agent runs apart by the invocation ADK creates for each run, not by invocation ID, agent name or agent.RunConfig:

    • A resumed run keeps the paused run's invocation ID, and its caller may pass the same message object as an earlier attempt or iterate the same runner.Run iterator again.
    • ADK reuses one RunConfig when an iterator whose root is not an LLM agent runs again.
    • Concurrent runs of one agent can share an invocation ID and agent name, for example a custom agent that runs one helper on several branches through platform.RunTasks.

    The toolset reads that invocation from the context ADK passes to Tools and from the tool context it passes to the ProcessRequest of the toolset and its tools, so all three find the caller's own state. A resume, a retry after a failed resume, or another pass over a kept iterator lists and filters again before ADK dispatches the confirmed call. If a context does not expose its invocation, the toolset lists and filters on every call, as before this change.

  • This state is kept per toolset and agent run, on a small store that NewContext puts on the context. So concurrent runs of one agent each use their own state, whether they are separate invocations (parallel agenttool calls) or share one. Another agent with the same name, run by another runner on the same context, does not inherit the first agent's state.

  • The store holds each agent run's invocation through a weak pointer. When a new agent run starts, it drops the state of runs whose invocation the garbage collector has reclaimed. So a long conversation on one NewContext does not keep finished turns' invocations, session events or tool lists.

  • Once the toolset's ProcessRequest has run in an agent run, its proxy tools no longer pack themselves into the request. The request then declares and dispatches only the current list. Before, ADK v2.5.0 and earlier kept the first step's proxies and still declared and dispatched a tool the server had dropped, while v2.6.0 did not.

  • A turn with one tool call schedules 2 ListMcpTools Activities with every ADK release, instead of 3 with v2.5.0 and 4 with v2.6.0. Each step still lists again, so a change in the MCP server's tools still reaches the next model call.

  • Tests: TestMcpListToolsCarriesParameters now checks one listing per model step (2). New tests check:

    • the order of listings and timers over two turns, with and without a ToolFilter that starts a timer (TestMcpListsAtTheSamePointInEveryStep);
    • a dropped tool, which is neither declared nor called, and the same number of generated UUIDs whether the catalog is unchanged, the server drops a tool, or ToolFilter drops it (TestMcpDroppedToolIsNeitherDeclaredNorCalled);
    • parallel and sequential helper runs, as separate agenttool invocations and as runs of one helper on two branches of one invocation (TestMcpParallelInvocationsKeepTheirOwnListings);
    • a tool excluded by ToolFilter while its confirmation is pending (TestMCPToolConfirmationResumeHonorsCurrentToolFilter). Cases: a new message, the original message overwritten, a resume retried with the same message after another toolset failed, and, under a sequential root agent, a new iterator or the paused run's iterator iterated again;
    • a toolset wrapped in tool.FilterToolset or tool.WithConfirmation (TestMcpWrappedToolsetListsOnEveryToolsCall);
    • one toolset used directly by one agent and through tool.FilterToolset by another agent with the same name or a different name (TestMcpSameNamedAgentsKeepTheirOwnListings);
    • 20 turns on one NewContext leave at most one store entry and no reachable finished run after garbage collection (TestMcpListingsReleaseFinishedRuns).

    The approval test also runs with the overwritten message, with a retried resume, and with the paused run's iterator under a sequential root.

No public name or option changes.

Breaking change: no version gate. We are not adding version gates to these fixes: breaking old histories is accepted with the plugins' 1.0 releases. Every run with an MCP toolset now records fewer ListMcpTools Activities: the first model step of each agent run lists once instead of twice, and with ADK v2.6.0 every later step does too. Later steps also list after the request processors instead of before them. So:

  • Runs that used NewMCPToolset recorded before this change, open or closed, fail replay with [TMPRL1100]. An open run stops at its next Workflow Task. This holds for turns without a tool call too. The exception is a toolset used only inside ADK's tool.FilterToolset or tool.WithConfirmation, which lists as before (see "Not covered").
  • Runs without an MCP toolset replay and continue.
  • Rolling back has the same effect: MCP runs that took a turn on this version fail on the earlier version with [TMPRL1100].

Upgrade: let runs that use NewMCPToolset finish on the earlier version before you upgrade, or keep them there with Worker Versioning. Replays of closed MCP runs (queries, resets, replay tests) need the earlier version too.

Why?

See #305. The MCP proxy runs a ListMcpTools Activity each time it lists, and the order and number of those Activities depended on the ADK release. After go get -u moved an application to ADK v2.6.0, open MCP runs failed every Workflow Task with a nondeterminism error. Matching only the number of listings was not enough. The listing and the ToolFilter calls must also sit at the same point relative to other preprocessing commands, stay separate per agent run, and declare the same tools. ADK must also keep the same tools between steps, because it draws one generated ID per kept tool.

Checklist

  1. Closes [Bug] googleadk (Go): MCP toolset histories fail replay after upgrading ADK to v2.6.0 #305

  2. How was this tested:

    • In Docker with Go 1.27.2, in go/googleadk: make sync, make lint, make test, make build and make test TEST_ARGS=-race pass with the committed go.mod (ADK v2.3.0). With ADK v2.6.0 in a separate module file (as the nightly run does), go vet ./... and go test ./... pass, and the MCP tests pass with -race. scripts/ci/check_conventions.py passes. CI's Go 1.26.6, macOS and Windows cells were not run.
    • The generated-ID case of TestMcpDroppedToolIsNeitherDeclaredNorCalled draws 19 UUIDs in every case on both releases. With the previous revision of this PR, it fails on v2.6.0 (18 after a dropped tool, 19 with an unchanged catalog).
    • The shared-invocation cases of TestMcpParallelInvocationsKeepTheirOwnListings pass on both releases. With an earlier revision, the parallel case fails on both: 10 listings instead of 8.
    • TestMcpListingsReleaseFinishedRuns passes on both releases. With the revision before the weak store it fails on both: after 20 turns the store held 20 entries, all with reachable invocation contexts.
    • Earlier revisions of this PR failed other cases that this change passes, on both releases:
      • listing order: listings before the instruction's timer, and 11 or 14 listings instead of 8;
      • a dropped tool called again;
      • the excluded tool run once, with an overwritten message, a retried resume or a saved iterator under a sequential root;
      • one listing through a wrapper on v2.6.0;
      • the second same-named agent's request declaring no tools.
    • Replay checks on a dev server (Temporal CLI), with WorkflowReplayer:
      • With this revision: an agent with two MCP toolsets calls search, then fetch, then a guarded tool. The server stops advertising search after it runs, and the workflow receives the approval as a signal carrying the confirmation ID it exposed through a query. Recorded with ADK v2.3.0 and v2.6.0, with and without the catalog change, every history replays on both builds, regenerates the recorded confirmation ID and runs the approved tool once. With the previous revision, both cross-version replays of the catalog-change runs regenerate a different confirmation ID and do not run the approved tool.
      • With the previous revision: a custom root running one helper (two MCP toolsets) on two branches of one invocation, in parallel and in sequence; two parallel agenttool helper calls; two turns with an InstructionProvider timer; two turns with a ToolFilter timer. Recorded with ADK v2.3.0 and v2.6.0, every history replays on both builds, and each helper scenario records 8 ListMcpTools on both releases. This revision changes only what the later Tools calls of v2.6.0 return, which issue no commands. These scenarios were not re-recorded; their test-env assertions pass on both releases.
      • On earlier revisions: 5 confirmation scenarios (exclude with a saved iterator under a sequential root, exclude with a new iterator, approve with a saved iterator, exclude with a retried resume, plain approve), each recorded with both releases, replayed on both builds. Before that: a plain one-call turn, a tool dropped after its first call, confirmation approve, deny and exclude with new, overwritten and retried messages, and the same-name and different-name shared-toolset cases. Both releases recorded the same commands. These were not rerun on the dev server after the later changes; their test-env assertions pass on both releases.
    • The wrapper cases were checked with TestMcpWrappedToolsetListsOnEveryToolsCall on both releases, not on the dev server.
  3. Any docs updates needed?
    No README change. Release notes should say which runs break (see "Breaking change").

Not covered:

  • An MCP toolset wrapped in ADK's tool.FilterToolset or tool.WithConfirmation. The wrappers have no ProcessRequest, so the toolset lists and runs ToolFilter each time ADK calls Tools, as before this change: once per agent run with v2.5.0 and once per step with v2.6.0. Histories of such runs replay only with the same ADK release. MCPToolsetOptions.ToolFilter avoids the filter wrapper.
  • go.temporal.io/sdk/contrib/googleadk v0.3.0, the plugin's earlier home, has the same defect. No backport here.

Open questions: none.

Draft while the breaking change is open.

ADK v2.5.0 and earlier ask a toolset for its tools once per agent run,
and the MCP proxy's ProcessRequest lists again before each model call.
ADK v2.6.0 also asks before each model call. The proxy runs a
ListMcpTools Activity for each request, so a one-call turn scheduled 3
listings with ADK v2.5.0 and 4 with v2.6.0, and MCP Histories failed
replay after an ADK upgrade or rollback.

ProcessRequest now reuses the list that Tools took earlier in the same
step and lists only when there is none. Each model step lists once on
every ADK release.

Breaking for replay: open runs that used NewMCPToolset fail replay on
the new version, because they recorded more ListMcpTools Activities.
Let those runs finish or keep them on the old version with Worker
Versioning before upgrading.
@campoy
campoy marked this pull request as ready for review October 31, 2026 17:25
@campoy
campoy requested a review from a team as a code owner October 31, 2026 17:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] googleadk (Go): MCP toolset histories fail replay after upgrading ADK to v2.6.0

1 participant