Repository navigation
Breaking: end Google ADK TypeScript runs CANCELLED when ADK absorbs the cancel in a tool or model callback - #322
Conversation
…ancel in a tool or model callback (breaking) ADK hands a tool's error and a model step's error, model callbacks included, to its plugin manager and then goes on. ADK before 2.3 drops a long-running tool's error, and a model callback's error becomes an error event that ends the turn. So a Workflow cancelled while a LongRunningFunctionTool waits for a Signal or Update, or while a beforeModelCallback awaits an Activity, ended COMPLETED. The plugin now also records a cancellation at those two plugin-manager hand-offs, so the existing re-raise ends the run CANCELLED, as it already does for a cancelled model call. A plugin hook's error reaches ADK wrapped in a plain Error, so a cancelled scope counts too.
DABH
left a comment
There was a problem hiding this comment.
Review at head 32f2165 (one commit over db14ec3). On a worktree at the head with pnpm 10.27.0: make lint clean (one pre-existing deprecation warning), make test 234 of 234, and the 19 absorbed-failure tests also pass with REUSE_V8_CONTEXT=false. Both new tests fail with src/absorbed-failure.ts reverted to db14ec3 (the Workflow returns '' instead of failing) and pass at the head. The findings come from probe Workflows run on the local test server against both builds (the same test code, only src/absorbed-failure.ts swapped) and, for replay, Worker.runReplayHistories in both directions; ADK's calls into the two wrapped methods and its plugin-hook wrapping were read in the published 2.0.0, 2.1.0, 2.2.0, 2.2.1, 2.3.0 and 2.4.0 builds. Confirmed as the PR body states: a main-function long-running tool turn now ends CANCELLED (COMPLETED on the base), and the two closed histories fail replay across builds with TMPRL1100 in both directions; a CancellationScope.withTimeout around a tool wait or a plugin callback Activity still completes; the wrap goes in once per Workflow (ADK loads into the per-Workflow module cache of the reusable context) and never outside one. Refuted: that a Signal-driven run whose main function schedules cleanup after the handler raised could get stuck (the cancel still lands last in history, with no Workflow Task failure, even 50,000 microtasks later). Cross-language: Python and Go ADK re-raise or yield unhandled model and callback errors (Python google-adk 2.12.0 flows/llm_flows/core/_finalizer.py:397 and tools/_caller.py:585; Go adk v2.3.0 callLLM), so this TypeScript-only change brings the plugin in line with its siblings; not probed end to end. Not verified: the disclosed open-run Signal-handler failure, and ADK releases other than the locked 2.2.0 at runtime.
Severity: P1 = wrong result or stuck run in realistic use; P2 = correctness or robustness in an edge case; P3 = docs, tests, cleanup.
Where the problems cluster: the wrap itself is sound: it sees exactly the errors ADK is about to absorb in every release the peer range allows, and the outcome logic it feeds is unchanged. What remains is where a recorded cancellation surfaces and how the change is described: in a Signal handler it ends the run before the main function's cleanup (the question you left open); an inner scope's cancel recorded on the new paths can decide a later cancel that the main function handled; and the breaking-change notes miss two classes of closed runs whose outcome and replay change. Commenting rather than approving: there is no P1, but the Signal-handler rule becomes part of the 1.0 contract and should be settled and documented, with the breaking-change notes completed, before this merges.
PR title and body (P3): release notes come from the subject, and 'when a tool or model callback absorbs the cancel' inverts what the change handles. A tool or callback that catches the cancel itself and returns (a beforeModelCallback that returns a response, say) reaches no hand-off, and the run still completes: callLlmAsync yields the callback's response and never calls the model. The branch's commit subject, 'when ADK absorbs the cancel in a tool or model callback', is accurate; the title should say the same. The body still ends with 'Draft while the breaking change is open', but the PR is not a draft, and the open question it refers to is the Signal-handler item above.
| * `Error` (with no `cause` before ADK 2.2.1), so a cancelled scope also counts. | ||
| */ | ||
| function recordAbsorbedCancellation(err: unknown): void { | ||
| const frame = openFrame(); |
There was a problem hiding this comment.
P2: a cancel absorbed in a Signal handler now ends the Workflow as the handler returns, so the main function's non-cancellable cleanup never runs, where the base already ended the same run CANCELLED after that cleanup. Probe on both builds: a Signal handler runs the LongRunningFunctionTool turn while the main function parks in condition(() => false) with finally { await CancellationScope.nonCancellable(() => cleanup()) }, and the Workflow is cancelled during the tool wait. Base: CANCELLED, and the cleanup Activity ran in all four variants (scheduled at once, after 2,000 or 50,000 microtasks, or after a final agent turn). Head: CANCELLED in all four, and the cleanup never ran: its ActivityTaskScheduled (in the agent-turn variant, the turn's model Activity) is recorded in the same Workflow Task completion as WorkflowExecutionCanceled. The base already behaves this way for a model call cancelled in a Signal handler (probe: cleanup never ran) and, on ADK 2.3, for a long-running tool through the model call that follows, so this carries an existing rule to tool waits on ADK 2.0 to 2.2 and to model callbacks on every release. The README never says so: the Model calls section names only 'the Workflow (or the Update handler that ran the turn)', and the new sentence at README line 592 gives no timing. On your open question, I would leave it to the main function. A main function that awaits anything cancellable sees the execution's cancel itself, so the handler's raise adds nothing there and only preempts the cleanup. Recording a Signal frame's cancellation on main (while main has not surfaced) still ends the run CANCELLED when the main function returns, after its cleanup, and leaves the Update rejection as it is. It needs the frame to know it belongs to handleSignal (line 380), plus the same routing in recordAbsorbedFailure's cancellation branch to keep model calls consistent, which changes the base's model-call rule in the same breaking release (closed runs that ended at a Signal handler's return then fail replay like the cases you list). If you keep the current rule, document it in the README next to line 592 and pin it with a test; nothing runs a Signal-handler turn through the new paths.
There was a problem hiding this comment.
Changed as you suggested, in 1fb0119. When a Signal handler returns, the cancellation it recorded moves to the main function's frame (deferToMain), and the main function raises it when it returns, so its cleanup runs first. A model call's cancellation recorded in a Signal handler moves the same way, so the base's model-call rule changes in this release too. The PR body now lists those closed runs under "Breaking change". The cancel outranks the model failures the handler absorbed, so those move with it and are dropped. An Update handler still raises its own. A Signal handler that throws, or that returns after the main function did, raises as before.
The README says so next to line 592. The new test cancellingASignalDrivenTurnLetsTheMainFunctionCleanUpFirst pins it: a Signal handler runs the tool turn while the main function waits, shielded, for it, then runs cleanup Activity one, waits for a Signal, and runs two. Cancelled during the tool wait, the run must still be RUNNING at that wait, schedule both cleanups, then end CANCELLED. A second case runs a failing model turn in the handler first.
| // Only a cancel a model call absorbed lands here, and it decides the outcome only if | ||
| // Only a cancel ADK absorbed lands here, and it decides the outcome only if | ||
| // the execution itself was cancelled. An inner scope's cancel | ||
| // (`CancellationScope.withTimeout`) is the consumer's own doing and is not raised at all. |
There was a problem hiding this comment.
P2: an inner scope's cancel recorded on the new paths is raised later if the execution is cancelled and the main function handles that cancel and returns, so 'not raised at all' does not hold and a handled cancel ends CANCELLED. Probe on both builds: the main function runs the long-running tool turn under CancellationScope.withTimeout('1 second', ...) (the turn falls back, as the PR body says), then parks in try { await condition(() => false) } catch (err) { if (err.name === 'CancelledFailure') return 'stopped' }, and the Workflow is cancelled while parked. Base: COMPLETED with the 'stopped' result. Head: CANCELLED. ADK never saw the execution's cancel; the recording the timeout left at line 181 decided it at line 269. The base does the same for a model call cut short by the timeout (probe: CANCELLED), so the mechanism predates this PR, but withTimeout is the natural way to bound an approval wait, which puts tool waits on it now. Fix: capture the root scope in execute (CancellationScope.current() is the root there) and keep a recording as frame.cancellation only when the root scope is cancel-requested at the moment it is recorded, in both record functions; keep the current-scope check on the synthetic branch as well, so a turn in a nonCancellable cleanup scope is still not recorded. Otherwise narrow this comment and the PR body's withTimeout sentence to runs whose execution is never cancelled afterwards.
There was a problem hiding this comment.
Fixed in 1fb0119, with a stricter check than the root-scope flag. Checking that the root is cancel-requested when the cancel is recorded still misses one order: the Workflow catches its cancellation first, then runs a turn in nonCancellable(() => withTimeout(...)). The root is already cancelled there, so the timeout's cancel would still be recorded. The same goes for a tool or callback that shields its own timed-out wait while the cancel arrives.
So now only the execution's own cancellation is recorded, for the hand-offs and model calls alike. That means the reason the root scope was cancelled with (the SDK passes that same object to every scope it cancels), or the failure of an Activity, local Activity, child Workflow or Nexus operation whose scope that reason cancelled first. A new outbound interceptor marks those failures. At the hand-off the plugin follows the error's cause chain, and what a tool's execute or a plugin hook threw before ADK replaced it with a plain Error. The synthetic CancelledFailure branch is gone, and raiseAbsorbed no longer checks the current scope. A turn in a non-cancellable cleanup scope is still not recorded, because nothing there is cancelled by the root's reason.
Your probe is the tool case of agentRunnerTimedOutTurnThenCaughtCancel in the new test aTimedOutTurnLeavesALaterCaughtCancelCompleted; it now ends COMPLETED with 'caught'. The test also covers the model call (CANCELLED on the base, as you found), the cancel-first order, a tool or callback that shields its own timed-out wait, and an ordinary callback error after a caught cancel: 19 cases, all COMPLETED. The PR body's withTimeout sentence is replaced.
| and reports the node as failed (`NodeReportedError`); the plugin ends the | ||
| execution CANCELLED with the model Activity's own cancellation instead. A | ||
| cancelled Activity node ends it CANCELLED the same way, inside a dynamic run too. | ||
| So does a cancel that ADK absorbs in a tool or a model callback, such as a |
There was a problem hiding this comment.
P2: the breaking-change notes sort affected closed runs into COMPLETED and CANCELLED, but graph runs that ended FAILED and runs that continued as new change to CANCELLED too, and their closed histories fail replay. (a) A graph, new Workflow({ edges: [['START', agent]] }), whose agent node's beforeModelCallback awaits a cancellable Activity, cancelled during it: base FAILED with ApplicationFailure type GoogleAdkNodeReportedError ("Node 'assistant' failed: Activity cancelled"); head CANCELLED, because toWorkflowFailure now finds the recording (absorbed-failure.ts:330). The base history fails replay on the head with [TMPRL1100] Nondeterminism error: Cancel workflow machine does not handle this event, and the head history fails on the base ("Fail workflow machine"). (b) A main-function tool turn followed by continueAsNew, the long-conversation shape this README describes: on the base the first run CONTINUED_AS_NEW and the next run COMPLETED, so the cancel was lost across the chain; on the head the run ends CANCELLED with no new run (raiseAbsorbed in the ContinueAsNew catch, absorbed-failure.ts:357-361), and the base history fails replay on the head with TMPRL1100. Both are real fixes, (b) arguably the most important one here, but the body says closed runs that already ended CANCELLED replay and only COMPLETED ones fail, and the upgrade advice and release notes inherit that. Add both classes to 'Breaking change' and to the release-notes sentence, and say here that a graph run whose agent node absorbed the cancel now ends CANCELLED rather than failing with GoogleAdkNodeReportedError.
There was a problem hiding this comment.
Added both to "Breaking change" in the PR body and to the release-notes sentence. In 6e83a65 the README paragraph also says that an agent node that absorbed the cancel in a tool or model callback ends the execution CANCELLED rather than failing it with GoogleAdkNodeReportedError.
- (a) Graph runs: the new
graph.test.tstest'swaitingcases check the outcome. They endCANCELLEDwith this change, and the callback cases endFAILEDon the base. The replay result in the body is yours. - (b) Continue-as-new: I reran it at 6e83a65 with ADK 2.2.0, a main-function tool turn followed by
continueAsNew, cancelled during the tool wait. This change ends the runCANCELLEDwith no new run. The base'sabsorbed-failure.tsends itCONTINUED_AS_NEW, and the next run goes on.
I also added the Signal-handler model-call runs from your first comment, which the hand-off to the main function changes on every ADK version.
| const handle = await env.client.workflow.start(agentRunnerLongRunningToolWait, { taskQueue, workflowId }); | ||
| // ADK before 2.3 drops a long-running tool's error, so no model call follows to fail in | ||
| // the cancelled scope, and the turn ends normally. | ||
| while (!(await handle.query(adkToolWaitingQuery))) { |
There was a problem hiding this comment.
P3 (tests): this poll has no deadline, so a regression where the tool never starts waiting hangs until ava's two-minute timeout instead of failing with a message, and on ADK 2.3 the case passes without the wrap. waitForScheduledActivities bounds the same kind of poll at 10 s (helpers.ts:446); a deadline and a throw like that one would do. On 2.3 a long-running tool's error goes to the model as a function response (agents/functions.js:412 in 2.3.0), TemporalModel records the model call that follows in the cancelled scope, and the issue's own 2.3.0 output shows the base ending this case CANCELLED. Nightly sync-latest resolves 2.3.x inside <2.4.0, so there nothing guards the runOnToolErrorCallback wrap. A tool case that ends the turn on the error's function response would keep it guarded on every release (for example a tool that sets toolContext.actions.skipSummarization before it waits, which isFinalResponse treats as final); at least say in the comment above that the case only discriminates before 2.3.
There was a problem hiding this comment.
Did both.
- The poll now goes through
waitForStage, which throws after 10 s (1fb0119). - The test tool sets
toolContext.actions.skipSummarizationbefore it waits (6e83a65). So on ADK 2.3 the turn ends on the error's function response, and no model call follows.
Runs at 6e83a65:
- With ADK 2.3.0, without the
runOnToolErrorCallbackwrap, this test now fails at its tool case (the Workflow returns''). Before the change, it passed without the wrap, as you said. - With the wrap, all 20
absorbed-failure.test.tstests pass on 2.3.0. - On 2.2.0 the tool case still fails without the wrap.
…l wait for the main function A cancel absorbed in a Signal handler ended the Workflow when the handler returned, so the main function's non-cancellable cleanup never ran. When a Signal handler returns, the cancellation it recorded now moves to the main function's frame, which raises it when the main function returns. It outranks the model failures the handler absorbed, so those move with it and are dropped. An Update handler still raises its own. A cancel that only an inner scope caused (CancellationScope.withTimeout) was recorded, and decided a later cancellation that the Workflow caught and finished on. Now only the execution's own cancellation is recorded, at the hand-offs and for model calls alike. The root scope's reason reaches every scope it cancels as the same object, and a new outbound interceptor marks the failure of an Activity, local Activity, child Workflow or Nexus operation (its start included) whose scope that reason cancelled first. At the hand-off the plugin follows the error's causes, and what a tool's execute or a plugin hook threw before ADK replaced it with a plain Error. An ordinary error is not recorded either, even after the Workflow caught its cancellation. A cancellation absorbed in a node attempt that ADK's node timeout abandoned is left out: ADK had moved past that work. The plugin runs each BaseNode.run in an async-local attempt and marks it abandoned when the node's timeout, not the invocation's abort, aborts its signal. The tests poll with a 10-second deadline, cover a Signal-driven turn's cleanup, a timed-out turn followed by a caught cancel, Nexus waits, and a node timeout's abandoned work.
… name the graph outcome From ADK 2.3 a long-running tool's error goes to the model as a function response, and the model call that follows ends the run CANCELLED without the runOnToolErrorCallback wrap, so the tool cases passed without it. The test tool now sets skipSummarization before it waits, which ends the turn on that response, so no model call follows on any ADK release. The README now says that an agent node that absorbed the cancel in a tool or model callback ends the execution CANCELLED rather than failing it with GoogleAdkNodeReportedError.
|
Thanks for the thorough review. This round is two new commits on top of 32f2165: 1fb0119 and 6e83a65. I've replied in each thread. On the summary's P3: the title now says "when ADK absorbs the cancel in a tool or model callback". The body drops the "Draft while…" line and the open question. That question is settled the way you suggested: a Signal handler's cancellation goes to the main function. 1fb0119 is bigger than your four comments alone would need. While you reviewed 32f2165, the same two problems (the Signal handler and the inner-scope cancel) went through more review rounds on our side. Those rounds found gaps in the simpler fixes:
Each part has tests: three cases in Checks at 6e83a65 (Node 24.22.0, pnpm 10.27.0, |
A Query to a Worker with no Workflow cache sometimes takes about 10 s, the Workflow Task timeout. main's CI shows it too, in the HITL tests. The new stage polls allowed 10 s (absorbed-failure.test.ts) and 20 s (graph.test.ts), so one or two such Queries failed them on Ubuntu, macOS and Windows. They now allow 60 s, still below ava's 2-minute timeout, so a run that never reaches the stage still fails with a message.
Querying a Workflow on a Worker with no Workflow cache (the test default) slows the Workflow itself. In a plain Workflow running 100 short Activities, four clients querying it made each run take 10 to 21 s instead of 4 to 6 s, with queries up to 20 s. On CI that let ADK's 1-second node timeout fire on the retry in graph.test.ts, so the graph failed with NodeTimeoutError and never reached its stage, on macOS and Windows even with the 60 s deadline of 626c9f7. The test Workflows now record their stage with upsertMemo, and the tests read it with describe(), which the server answers without the Worker. The deadlines go back to 10 s and 20 s.
What was changed
A Workflow cancelled while ADK absorbs the cancel in a tool or a model callback now ends
CANCELLED,as it already does when a model call absorbs it.
PluginManager.runOnToolErrorCallback, and a model step's error, model callbacks included,through
runOnModelErrorCallback. In Workflow code the plugin now wraps these two methods once andrecords a cancellation it sees there, in the same place
TemporalModelrecords one. The existinginterceptor then raises it when the main function or handler returns. What ADK does with the error
is unchanged, and nothing changes outside a Workflow.
decides that from the operation that was cancelled, not from the scope where ADK hands the error
on. The SDK cancels the root scope for the Workflow's cancellation request with a
CancelledFailureand passes that same object to every scope the request reaches, so a timer or
conditionfailswith it. A new outbound interceptor marks the failure of an Activity, local Activity, child Workflow
or Nexus operation when the first cancellation to reach the scope it started in was that object,
whether the operation's start or its result fails (a Nexus operation cancelled before it starts
fails its start).
At the hand-off the plugin follows the error's
causechain, and what a tool'sexecuteor a pluginhook threw: ADK replaces those with a plain
Errorwith nocause(FunctionTool.runAsyncinevery release,
PluginManager.runCallbacksbefore 2.2.1), so the plugin wraps those two methods tokeep the original for ADK's error. To see what
executethrows, aFunctionToolrun in a Workflowgets an
executethat calls the original. A cancel that only an inner scope caused(
CancellationScope.withTimeout) is not recorded, whichever comes first and wherever the Workflowshields its waits (around the turn, or inside the tool or callback), and neither is an ordinary
error, even one thrown after the Workflow caught its cancellation. Before, a model call that a
timeout scope cancelled was recorded, and a cancellation request that the Workflow caught and
finished on ended the run
CANCELLED.timeoutabandons the attempt without cancelling it, so its model call, or a tool or model callback it
waits on, keeps running while the retry answers, and the execution's cancellation may cancel that
work after the graph returned. ADK's node runner calls
BaseNode.runonce per attempt and, for anode with a
timeout, aborts the attempt's own signal when the deadline fires. The plugin nowwraps
BaseNode.prototype.runso each attempt's steps run in an async-local attempt record. Therecord is marked abandoned when that signal aborts while the invocation's own signal has not. A
cancellation absorbed in an abandoned attempt, or in one nested in it, is not recorded, whatever
the retry answered with (a model call or a
beforeModelCallback). A retry, and each item of aParallelWorker, is an attempt of its own, so a live wait is still recorded after anotherattempt or item answered. Ordinary model failures keep
TemporalModel's existing rule. A plainagent turn outside a graph runs in no node attempt and is never left out.
and is raised when the main function returns, so the main function's cleanup runs first. Raised from
the Signal handler, it would end the Workflow at once. It outranks the model failures the handler
absorbed, so those are dropped with the move rather than raised from the handler in its place. An
Update handler still raises its own.
absorbed-failure.test.tsand one ingraph.test.ts. ALongRunningFunctionToolwaiting for anapproval, and a
beforeModelCallback(the agent's and an ADK plugin's) awaiting an Activity, eachcancelled while it waits, end
CANCELLED, and so do the same three waiting on a Nexus operationthat is cancelled before it starts. The test tool sets
skipSummarizationbefore it waits, so withADK 2.3, which hands the tool's error to the model, the turn still ends with no model call after
it, and the tool case still needs the
runOnToolErrorCallbackwrap. A Signal-driven turn cancelled while its tool waits letsthe main function run both cleanup Activities before the run ends
CANCELLED, also when the handlerabsorbed a failing model call before. A turn that its timeout scope cancels (tool, agent callback,
plugin callback, model call) ends
COMPLETEDwhen the Workflow catches its cancellation request,after the turn or before it with the turn in a non-cancellable scope; so does a tool or callback
that shields its own timed-out wait, with the request during the wait or during the cleanup after
it, also when it waits on a Nexus operation the timeout cancels before it starts, and a callback that throws an ordinary error after the Workflow caught its cancellation.
An agent node whose first model call ADK's node
timeoutleft running while the retry answeredends
COMPLETEDwith the retry's answer when the Workflow then catches its cancellation, whichalso cancels that first call; so does one whose first
LongRunningFunctionToolrun orbeforeModelCallback(the agent's or a plugin's) the timeout left awaiting an Activity, with thesame answer as a run that left nothing running, also when the retry answers from a
beforeModelCallbackwith no model call. Without a node timeout, the same waits cancelled whilethey run end
CANCELLED, and so does one in aParallelWorkeritem whose sibling item, the sameagent, has answered.
cancel ends the execution
CANCELLEDrather than failing it withGoogleAdkNodeReportedError),and Agent node timeouts now covers a tool or model callback the timeout leaves running.
No public name or option changes.
Breaking change: no version gate. We are not adding version gates to these fixes: breaking old
histories is accepted with the plugins' 1.0 releases. Only runs where ADK absorbed a cancel change:
CANCELLEDinstead of
COMPLETED, after the main function'sfinallyruns. A closed run of this kind recordedbefore this change fails replay with
[TMPRL1100]at its last event. An open run that already passedthe turn (for example, waiting in a non-cancellable cleanup) replays and ends
CANCELLED. With ADK2.3, a long-running tool's cancel already ended the run
CANCELLEDthrough the model call afterit, so those runs don't change, unless the turn ended on the tool's response (for example, the tool
set
skipSummarization).failed, and the run ended
FAILEDwithGoogleAdkNodeReportedError. It now endsCANCELLED.Closed runs fail replay with
[TMPRL1100], in both directions. (A long-running tool in an agentnode ended
COMPLETEDbefore, as in the first item.)CONTINUED_AS_NEWand the next run went on, so the cancel was lost across the chain. The run now ends
CANCELLEDand starts no new run. Closed runs recorded before this change fail replay with
[TMPRL1100].turn without being cancellable): the run ends
CANCELLEDinstead ofCOMPLETED, after the mainfunction's cleanup; closed runs fail replay with
[TMPRL1100].a long-running tool's turn, through the model call after the tool): the run used to end as soon as
the handler returned, before the main function's cleanup. It now waits for the main function, so
the cleanup runs first, and closed runs of that kind fail replay with
[TMPRL1100].CancelledFailureinstead of completing, soa caller that awaits its result gets an error. The main function is unchanged, and old histories
replay.
catches, on any ADK version: one requested after the turn, or before a turn the Workflow runs in a
non-cancellable scope. With ADK 2.3 a long-running tool's turn under a timeout scope also reaches
such a model call. Those runs now end
COMPLETEDinstead ofCANCELLED, and closed ones fail replaywith
[TMPRL1100](replayed for the ADK 2.3 tool case after the turn).version with
[TMPRL1100]. Open runs carried either way complete as the new Worker's build behaves.Upgrade: replays of closed runs that hit the bug (queries, resets, replay tests) need the earlier
version, or Worker Versioning. Check Update callers that await such an Update's result.
Why?
See #321. ADK absorbs a cancellation in a tool or a model callback, so the run finished as if
nothing happened, and a parent Workflow that cancelled it took its success path. The plugin already
re-raises cancellations that ADK absorbs from model calls; this covers the other two places ADK
absorbs them, without ending a Signal-driven run before its cleanup or letting an inner scope's cancel
decide a cancellation the Workflow handled.
Checklist
Closes [Bug] google-adk (TypeScript): a cancelled Workflow ends COMPLETED when ADK absorbs the cancel in a tool or model callback #321
How was this tested:
typescript/google-adkwith the committed lock(
@google/adk2.2.0):make lint(no errors; one existing deprecation warning),make test(236 passed, run with
TEST_ARGS="--concurrency=1 --timeout=10m"because the shared machine wasslow enough to hit ava's 2-minute idle timeout),
make buildandscripts/ci/check_conventions.pypass. CI's Node 22.17, macOS and Windows cells were not run.
@google/adk2.3.0: all 20absorbed-failure.test.tstests pass. Without therunOnToolErrorCallbackwrap,cancellingAToolOrModelCallbackWaitEndsCancelledfails at its toolcase (the Workflow returns
'') on 2.2.0 and 2.3.0; on 2.3.0 it passed without the wrap before thetest tool set
skipSummarization.continueAsNew, cancelled during the tool wait(
@google/adk2.2.0): this change ends the runCANCELLEDwith no new run;main'sabsorbed-failure.tsends itCONTINUED_AS_NEW, and the next run goes on.absorbed-failure.test.tstests fail onmain: the tool and callback waits andthe Signal turn complete normally, and the timed-out model call turns end
CANCELLED. They passwith this change. The new
graph.test.tstest passes with this change, with@google/adk2.2.0and 2.3.0. On
mainits graph waits cancelled while they run, alone or beside an answeredParallelWorkeritem, endCOMPLETED(tool) orFAILED(callbacks). The abandoned cases passthere, and they guard the hand-offs, which recorded an abandoned call's, tool's or callback's
cancellation in earlier drafts.
@temporalio/google-adk0.1.0 (@google/adk2.0.0),mainandthis change (
@google/adk2.2.0 and 2.3.0): every case endsCANCELLEDwith this change; theFunctionToolandbeforeAgentCallbackcontrols endCANCELLEDeverywhere. The same five caseswaiting on a Nexus operation (
TRY_CANCEL), cancelled before it starts (unpolled endpoint) andafter it started (a Workflow-run operation), all end
CANCELLEDwith this change on both ADKversions.
Worker.runReplayHistories, onmainand this change with@google/adk2.2.0 and 2.3.0. Elevenscenarios, each recorded open (held in non-cancellable cleanup) and finished: the turn in the main
function cancelled during a long-running tool, an agent
beforeModelCallbackActivity or a pluginbeforeModelCallbackActivity; the tool turn in a Signal handler (main function parked orwaiting shielded for the turn) or an Update handler, cancelled; the tool, agent callback or plugin
callback turn under a timeout scope, then a caught cancellation; and two runs off the changed path.
With this change both Signal runs schedule both cleanups before ending
CANCELLED, and thetimeout-scope runs end
COMPLETED. Each build replays its own histories, and the off-pathhistories replay on both. Across builds, only the runs listed under "Breaking change" fail, each
with
DeterminismViolationError[TMPRL1100]; a renamed Activity fails every history the sameway. Open runs moved to a new Worker process, including Signal-handler runs from
mainto thischange and back, complete as the new Worker's build behaves.
Any docs updates needed?
The README changes above. Release notes should say which runs break (see "Breaking change"):
main-function and Signal-handler turns, graph runs that failed with
GoogleAdkNodeReportedError,runs that continued as new, Update callers, and timed-out model calls whose cancellation the
Workflow caught.
Not covered:
(fix(core): normalize tool responses before afterToolCallback runs (#1035) google/adk-js#1061), and the plugin already ended that case
CANCELLEDthrough the model callafter it. This change still covers ADK 2.0 to 2.2, which the plugin supports, a turn that ends on
the tool's response on every version, and model callbacks on every version.
separately.