Skip to content

Breaking: end Google ADK TypeScript runs CANCELLED when ADK absorbs the cancel in a tool or model callback - #322

Open
campoy wants to merge 5 commits into
mainfrom
fix/google-adk-typescript-long-running-tool-wait-swallows-cancel
Open

campoy wants to merge 5 commits into
mainfrom
fix/google-adk-typescript-long-running-tool-wait-swallows-cancel

Conversation

@campoy

@campoy campoy commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

A Workflow cancelled while ADK absorbs the cancel in a tool or a model callback now ends CANCELLED,
as it already does when a model call absorbs it.

  • ADK hands an error to its plugin manager before it absorbs it: a tool's error through
    PluginManager.runOnToolErrorCallback, and a model step's error, model callbacks included,
    through runOnModelErrorCallback. In Workflow code the plugin now wraps these two methods once and
    records a cancellation it sees there, in the same place TemporalModel records one. The existing
    interceptor then raises it when the main function or handler returns. What ADK does with the error
    is unchanged, and nothing changes outside a Workflow.
  • Only the execution's own cancellation is recorded, here and for model calls alike, and the plugin
    decides that from the operation that was cancelled, not from the scope where ADK hands the error
    on. The SDK cancels the root scope for the Workflow's cancellation request with a CancelledFailure
    and passes that same object to every scope the request reaches, so a timer or condition fails
    with it. A new outbound interceptor marks the failure of an Activity, local Activity, child Workflow
    or Nexus operation when the first cancellation to reach the scope it started in was that object,
    whether the operation's start or its result fails (a Nexus operation cancelled before it starts
    fails its start).
    At the hand-off the plugin follows the error's cause chain, and what a tool's execute or a plugin
    hook threw: ADK replaces those with a plain Error with no cause (FunctionTool.runAsync in
    every release, PluginManager.runCallbacks before 2.2.1), so the plugin wraps those two methods to
    keep the original for ADK's error. To see what execute throws, a FunctionTool run in a Workflow
    gets an execute that calls the original. A cancel that only an inner scope caused
    (CancellationScope.withTimeout) is not recorded, whichever comes first and wherever the Workflow
    shields its waits (around the turn, or inside the tool or callback), and neither is an ordinary
    error, even one thrown after the Workflow caught its cancellation. Before, a model call that a
    timeout scope cancelled was recorded, and a cancellation request that the Workflow caught and
    finished on ended the run CANCELLED.
  • A cancellation absorbed in work ADK had moved past is left out. An agent node's timeout
    abandons the attempt without cancelling it, so its model call, or a tool or model callback it
    waits on, keeps running while the retry answers, and the execution's cancellation may cancel that
    work after the graph returned. ADK's node runner calls BaseNode.run once per attempt and, for a
    node with a timeout, aborts the attempt's own signal when the deadline fires. The plugin now
    wraps BaseNode.prototype.run so each attempt's steps run in an async-local attempt record. The
    record is marked abandoned when that signal aborts while the invocation's own signal has not. A
    cancellation absorbed in an abandoned attempt, or in one nested in it, is not recorded, whatever
    the retry answered with (a model call or a beforeModelCallback). A retry, and each item of a
    ParallelWorker, is an attempt of its own, so a live wait is still recorded after another
    attempt or item answered. Ordinary model failures keep TemporalModel's existing rule. A plain
    agent turn outside a graph runs in no node attempt and is never left out.
  • A cancellation recorded in a Signal handler moves to the main function when the handler returns,
    and is raised when the main function returns, so the main function's cleanup runs first. Raised from
    the Signal handler, it would end the Workflow at once. It outranks the model failures the handler
    absorbed, so those are dropped with the move rather than raised from the handler in its place. An
    Update handler still raises its own.
  • Tests: three cases in absorbed-failure.test.ts and one in graph.test.ts. A LongRunningFunctionTool waiting for an
    approval, and a beforeModelCallback (the agent's and an ADK plugin's) awaiting an Activity, each
    cancelled while it waits, end CANCELLED, and so do the same three waiting on a Nexus operation
    that is cancelled before it starts. The test tool sets skipSummarization before it waits, so with
    ADK 2.3, which hands the tool's error to the model, the turn still ends with no model call after
    it, and the tool case still needs the runOnToolErrorCallback wrap. A Signal-driven turn cancelled while its tool waits lets
    the main function run both cleanup Activities before the run ends CANCELLED, also when the handler
    absorbed a failing model call before. A turn that its timeout scope cancels (tool, agent callback,
    plugin callback, model call) ends COMPLETED when the Workflow catches its cancellation request,
    after the turn or before it with the turn in a non-cancellable scope; so does a tool or callback
    that shields its own timed-out wait, with the request during the wait or during the cleanup after
    it, also when it waits on a Nexus operation the timeout cancels before it starts, and a callback that throws an ordinary error after the Workflow caught its cancellation.
    An agent node whose first model call ADK's node timeout left running while the retry answered
    ends COMPLETED with the retry's answer when the Workflow then catches its cancellation, which
    also cancels that first call; so does one whose first LongRunningFunctionTool run or
    beforeModelCallback (the agent's or a plugin's) the timeout left awaiting an Activity, with the
    same answer as a run that left nothing running, also when the retry answers from a
    beforeModelCallback with no model call. Without a node timeout, the same waits cancelled while
    they run end CANCELLED, and so does one in a ParallelWorker item whose sibling item, the same
    agent, has answered.
  • README: four sentences in the cancellation part of Failures (an agent node that absorbed the
    cancel ends the execution CANCELLED rather than failing it with GoogleAdkNodeReportedError),
    and Agent node timeouts now covers a tool or model callback the timeout leaves running.

No public name or option changes.

Breaking change: no version gate. We are not adding version gates to these fixes: breaking old
histories is accepted with the plugins' 1.0 releases. Only runs where ADK absorbed a cancel change:

  • Turn in the main function, cancel absorbed in a tool or a model callback: the run ends CANCELLED
    instead of COMPLETED, after the main function's finally runs. A closed run of this kind recorded
    before this change fails replay with [TMPRL1100] at its last event. An open run that already passed
    the turn (for example, waiting in a non-cancellable cleanup) replays and ends CANCELLED. With ADK
    2.3, a long-running tool's cancel already ended the run CANCELLED through the model call after
    it, so those runs don't change, unless the turn ended on the tool's response (for example, the tool
    set skipSummarization).
  • Graph run whose agent node absorbed the cancel in a model callback: ADK reported the node as
    failed, and the run ended FAILED with GoogleAdkNodeReportedError. It now ends CANCELLED.
    Closed runs fail replay with [TMPRL1100], in both directions. (A long-running tool in an agent
    node ended COMPLETED before, as in the first item.)
  • Run that continued as new after a turn that absorbed the cancel: the run ended CONTINUED_AS_NEW
    and the next run went on, so the cancel was lost across the chain. The run now ends CANCELLED
    and starts no new run. Closed runs recorded before this change fail replay with [TMPRL1100].
  • Turn in a Signal handler whose main function then returns normally (for example, it waits for the
    turn without being cancellable): the run ends CANCELLED instead of COMPLETED, after the main
    function's cleanup; closed runs fail replay with [TMPRL1100].
  • Turn in a Signal handler whose model call absorbed the cancel, on any ADK version (with ADK 2.3, also
    a long-running tool's turn, through the model call after the tool): the run used to end as soon as
    the handler returned, before the main function's cleanup. It now waits for the main function, so
    the cleanup runs first, and closed runs of that kind fail replay with [TMPRL1100].
  • Turn in an Update handler: the Update is rejected with CancelledFailure instead of completing, so
    a caller that awaits its result gets an error. The main function is unchanged, and old histories
    replay.
  • A model call that a timeout scope cancelled no longer decides a cancellation the Workflow
    catches, on any ADK version: one requested after the turn, or before a turn the Workflow runs in a
    non-cancellable scope. With ADK 2.3 a long-running tool's turn under a timeout scope also reaches
    such a model call. Those runs now end COMPLETED instead of CANCELLED, and closed ones fail replay
    with [TMPRL1100] (replayed for the ADK 2.3 tool case after the turn).
  • Rolling back has the same effect the other way: closed runs that changed fail replay on the earlier
    version with [TMPRL1100]. Open runs carried either way complete as the new Worker's build behaves.

Upgrade: replays of closed runs that hit the bug (queries, resets, replay tests) need the earlier
version, or Worker Versioning. Check Update callers that await such an Update's result.

Why?

See #321. ADK absorbs a cancellation in a tool or a model callback, so the run finished as if
nothing happened, and a parent Workflow that cancelled it took its success path. The plugin already
re-raises cancellations that ADK absorbs from model calls; this covers the other two places ADK
absorbs them, without ending a Signal-driven run before its cleanup or letting an inner scope's cancel
decide a cancellation the Workflow handled.

Checklist

  1. Closes [Bug] google-adk (TypeScript): a cancelled Workflow ends COMPLETED when ADK absorbs the cancel in a tool or model callback #321

  2. How was this tested:

    • With Node 24.22.0 and pnpm 10.27.0, in typescript/google-adk with the committed lock
      (@google/adk 2.2.0): make lint (no errors; one existing deprecation warning), make test
      (236 passed, run with TEST_ARGS="--concurrency=1 --timeout=10m" because the shared machine was
      slow enough to hit ava's 2-minute idle timeout), make build and scripts/ci/check_conventions.py
      pass. CI's Node 22.17, macOS and Windows cells were not run.
    • With @google/adk 2.3.0: all 20 absorbed-failure.test.ts tests pass. Without the
      runOnToolErrorCallback wrap, cancellingAToolOrModelCallbackWaitEndsCancelled fails at its tool
      case (the Workflow returns '') on 2.2.0 and 2.3.0; on 2.3.0 it passed without the wrap before the
      test tool set skipSummarization.
    • A main-function tool turn followed by continueAsNew, cancelled during the tool wait
      (@google/adk 2.2.0): this change ends the run CANCELLED with no new run; main's
      absorbed-failure.ts ends it CONTINUED_AS_NEW, and the next run goes on.
    • The three new absorbed-failure.test.ts tests fail on main: the tool and callback waits and
      the Signal turn complete normally, and the timed-out model call turns end CANCELLED. They pass
      with this change. The new graph.test.ts test passes with this change, with @google/adk 2.2.0
      and 2.3.0. On main its graph waits cancelled while they run, alone or beside an answered
      ParallelWorker item, end COMPLETED (tool) or FAILED (callbacks). The abandoned cases pass
      there, and they guard the hand-offs, which recorded an abandoned call's, tool's or callback's
      cancellation in earlier drafts.
    • The reproduction in [Bug] google-adk (TypeScript): a cancelled Workflow ends COMPLETED when ADK absorbs the cancel in a tool or model callback #321, with @temporalio/google-adk 0.1.0 (@google/adk 2.0.0), main and
      this change (@google/adk 2.2.0 and 2.3.0): every case ends CANCELLED with this change; the
      FunctionTool and beforeAgentCallback controls end CANCELLED everywhere. The same five cases
      waiting on a Nexus operation (TRY_CANCEL), cancelled before it starts (unpolled endpoint) and
      after it started (a Workflow-run operation), all end CANCELLED with this change on both ADK
      versions.
    • Replay check on a dev server (Temporal CLI 1.9.1), with Workers that cache no Workflows and
      Worker.runReplayHistories, on main and this change with @google/adk 2.2.0 and 2.3.0. Eleven
      scenarios, each recorded open (held in non-cancellable cleanup) and finished: the turn in the main
      function cancelled during a long-running tool, an agent beforeModelCallback Activity or a plugin
      beforeModelCallback Activity; the tool turn in a Signal handler (main function parked or
      waiting shielded for the turn) or an Update handler, cancelled; the tool, agent callback or plugin
      callback turn under a timeout scope, then a caught cancellation; and two runs off the changed path.
      With this change both Signal runs schedule both cleanups before ending CANCELLED, and the
      timeout-scope runs end COMPLETED. Each build replays its own histories, and the off-path
      histories replay on both. Across builds, only the runs listed under "Breaking change" fail, each
      with DeterminismViolationError [TMPRL1100]; a renamed Activity fails every history the same
      way. Open runs moved to a new Worker process, including Signal-handler runs from main to this
      change and back, complete as the new Worker's build behaves.
  3. Any docs updates needed?
    The README changes above. Release notes should say which runs break (see "Breaking change"):
    main-function and Signal-handler turns, graph runs that failed with GoogleAdkNodeReportedError,
    runs that continued as new, Update callers, and timed-out model calls whose cancellation the
    Workflow caught.

Not covered:

  • With ADK 2.3 and later, ADK hands a long-running tool's error to the model as a function response
    (fix(core): normalize tool responses before afterToolCallback runs (#1035) google/adk-js#1061), and the plugin already ended that case CANCELLED through the model call
    after it. This change still covers ADK 2.0 to 2.2, which the plugin supports, a turn that ends on
    the tool's response on every version, and model callbacks on every version.
  • Python is not changed. A related Python cancel bug has a different cause and will be filed
    separately.

…ancel in a tool or model callback (breaking)

ADK hands a tool's error and a model step's error, model callbacks included, to its
plugin manager and then goes on. ADK before 2.3 drops a long-running tool's error, and a
model callback's error becomes an error event that ends the turn. So a Workflow cancelled
while a LongRunningFunctionTool waits for a Signal or Update, or while a
beforeModelCallback awaits an Activity, ended COMPLETED.

The plugin now also records a cancellation at those two plugin-manager hand-offs, so the
existing re-raise ends the run CANCELLED, as it already does for a cancelled model call.
A plugin hook's error reaches ADK wrapped in a plain Error, so a cancelled scope counts
too.
@campoy
campoy marked this pull request as ready for review October 31, 2026 01:35
@campoy
campoy requested a review from a team as a code owner October 31, 2026 01:35

@DABH DABH left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at head 32f2165 (one commit over db14ec3). On a worktree at the head with pnpm 10.27.0: make lint clean (one pre-existing deprecation warning), make test 234 of 234, and the 19 absorbed-failure tests also pass with REUSE_V8_CONTEXT=false. Both new tests fail with src/absorbed-failure.ts reverted to db14ec3 (the Workflow returns '' instead of failing) and pass at the head. The findings come from probe Workflows run on the local test server against both builds (the same test code, only src/absorbed-failure.ts swapped) and, for replay, Worker.runReplayHistories in both directions; ADK's calls into the two wrapped methods and its plugin-hook wrapping were read in the published 2.0.0, 2.1.0, 2.2.0, 2.2.1, 2.3.0 and 2.4.0 builds. Confirmed as the PR body states: a main-function long-running tool turn now ends CANCELLED (COMPLETED on the base), and the two closed histories fail replay across builds with TMPRL1100 in both directions; a CancellationScope.withTimeout around a tool wait or a plugin callback Activity still completes; the wrap goes in once per Workflow (ADK loads into the per-Workflow module cache of the reusable context) and never outside one. Refuted: that a Signal-driven run whose main function schedules cleanup after the handler raised could get stuck (the cancel still lands last in history, with no Workflow Task failure, even 50,000 microtasks later). Cross-language: Python and Go ADK re-raise or yield unhandled model and callback errors (Python google-adk 2.12.0 flows/llm_flows/core/_finalizer.py:397 and tools/_caller.py:585; Go adk v2.3.0 callLLM), so this TypeScript-only change brings the plugin in line with its siblings; not probed end to end. Not verified: the disclosed open-run Signal-handler failure, and ADK releases other than the locked 2.2.0 at runtime.

Severity: P1 = wrong result or stuck run in realistic use; P2 = correctness or robustness in an edge case; P3 = docs, tests, cleanup.

Where the problems cluster: the wrap itself is sound: it sees exactly the errors ADK is about to absorb in every release the peer range allows, and the outcome logic it feeds is unchanged. What remains is where a recorded cancellation surfaces and how the change is described: in a Signal handler it ends the run before the main function's cleanup (the question you left open); an inner scope's cancel recorded on the new paths can decide a later cancel that the main function handled; and the breaking-change notes miss two classes of closed runs whose outcome and replay change. Commenting rather than approving: there is no P1, but the Signal-handler rule becomes part of the 1.0 contract and should be settled and documented, with the breaking-change notes completed, before this merges.

PR title and body (P3): release notes come from the subject, and 'when a tool or model callback absorbs the cancel' inverts what the change handles. A tool or callback that catches the cancel itself and returns (a beforeModelCallback that returns a response, say) reaches no hand-off, and the run still completes: callLlmAsync yields the callback's response and never calls the model. The branch's commit subject, 'when ADK absorbs the cancel in a tool or model callback', is accurate; the title should say the same. The body still ends with 'Draft while the breaking change is open', but the PR is not a draft, and the open question it refers to is the Signal-handler item above.

* `Error` (with no `cause` before ADK 2.2.1), so a cancelled scope also counts.
*/
function recordAbsorbedCancellation(err: unknown): void {
const frame = openFrame();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: a cancel absorbed in a Signal handler now ends the Workflow as the handler returns, so the main function's non-cancellable cleanup never runs, where the base already ended the same run CANCELLED after that cleanup. Probe on both builds: a Signal handler runs the LongRunningFunctionTool turn while the main function parks in condition(() => false) with finally { await CancellationScope.nonCancellable(() => cleanup()) }, and the Workflow is cancelled during the tool wait. Base: CANCELLED, and the cleanup Activity ran in all four variants (scheduled at once, after 2,000 or 50,000 microtasks, or after a final agent turn). Head: CANCELLED in all four, and the cleanup never ran: its ActivityTaskScheduled (in the agent-turn variant, the turn's model Activity) is recorded in the same Workflow Task completion as WorkflowExecutionCanceled. The base already behaves this way for a model call cancelled in a Signal handler (probe: cleanup never ran) and, on ADK 2.3, for a long-running tool through the model call that follows, so this carries an existing rule to tool waits on ADK 2.0 to 2.2 and to model callbacks on every release. The README never says so: the Model calls section names only 'the Workflow (or the Update handler that ran the turn)', and the new sentence at README line 592 gives no timing. On your open question, I would leave it to the main function. A main function that awaits anything cancellable sees the execution's cancel itself, so the handler's raise adds nothing there and only preempts the cleanup. Recording a Signal frame's cancellation on main (while main has not surfaced) still ends the run CANCELLED when the main function returns, after its cleanup, and leaves the Update rejection as it is. It needs the frame to know it belongs to handleSignal (line 380), plus the same routing in recordAbsorbedFailure's cancellation branch to keep model calls consistent, which changes the base's model-call rule in the same breaking release (closed runs that ended at a Signal handler's return then fail replay like the cases you list). If you keep the current rule, document it in the README next to line 592 and pin it with a test; nothing runs a Signal-handler turn through the new paths.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed as you suggested, in 1fb0119. When a Signal handler returns, the cancellation it recorded moves to the main function's frame (deferToMain), and the main function raises it when it returns, so its cleanup runs first. A model call's cancellation recorded in a Signal handler moves the same way, so the base's model-call rule changes in this release too. The PR body now lists those closed runs under "Breaking change". The cancel outranks the model failures the handler absorbed, so those move with it and are dropped. An Update handler still raises its own. A Signal handler that throws, or that returns after the main function did, raises as before.

The README says so next to line 592. The new test cancellingASignalDrivenTurnLetsTheMainFunctionCleanUpFirst pins it: a Signal handler runs the tool turn while the main function waits, shielded, for it, then runs cleanup Activity one, waits for a Signal, and runs two. Cancelled during the tool wait, the run must still be RUNNING at that wait, schedule both cleanups, then end CANCELLED. A second case runs a failing model turn in the handler first.

// Only a cancel a model call absorbed lands here, and it decides the outcome only if
// Only a cancel ADK absorbed lands here, and it decides the outcome only if
// the execution itself was cancelled. An inner scope's cancel
// (`CancellationScope.withTimeout`) is the consumer's own doing and is not raised at all.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: an inner scope's cancel recorded on the new paths is raised later if the execution is cancelled and the main function handles that cancel and returns, so 'not raised at all' does not hold and a handled cancel ends CANCELLED. Probe on both builds: the main function runs the long-running tool turn under CancellationScope.withTimeout('1 second', ...) (the turn falls back, as the PR body says), then parks in try { await condition(() => false) } catch (err) { if (err.name === 'CancelledFailure') return 'stopped' }, and the Workflow is cancelled while parked. Base: COMPLETED with the 'stopped' result. Head: CANCELLED. ADK never saw the execution's cancel; the recording the timeout left at line 181 decided it at line 269. The base does the same for a model call cut short by the timeout (probe: CANCELLED), so the mechanism predates this PR, but withTimeout is the natural way to bound an approval wait, which puts tool waits on it now. Fix: capture the root scope in execute (CancellationScope.current() is the root there) and keep a recording as frame.cancellation only when the root scope is cancel-requested at the moment it is recorded, in both record functions; keep the current-scope check on the synthetic branch as well, so a turn in a nonCancellable cleanup scope is still not recorded. Otherwise narrow this comment and the PR body's withTimeout sentence to runs whose execution is never cancelled afterwards.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1fb0119, with a stricter check than the root-scope flag. Checking that the root is cancel-requested when the cancel is recorded still misses one order: the Workflow catches its cancellation first, then runs a turn in nonCancellable(() => withTimeout(...)). The root is already cancelled there, so the timeout's cancel would still be recorded. The same goes for a tool or callback that shields its own timed-out wait while the cancel arrives.

So now only the execution's own cancellation is recorded, for the hand-offs and model calls alike. That means the reason the root scope was cancelled with (the SDK passes that same object to every scope it cancels), or the failure of an Activity, local Activity, child Workflow or Nexus operation whose scope that reason cancelled first. A new outbound interceptor marks those failures. At the hand-off the plugin follows the error's cause chain, and what a tool's execute or a plugin hook threw before ADK replaced it with a plain Error. The synthetic CancelledFailure branch is gone, and raiseAbsorbed no longer checks the current scope. A turn in a non-cancellable cleanup scope is still not recorded, because nothing there is cancelled by the root's reason.

Your probe is the tool case of agentRunnerTimedOutTurnThenCaughtCancel in the new test aTimedOutTurnLeavesALaterCaughtCancelCompleted; it now ends COMPLETED with 'caught'. The test also covers the model call (CANCELLED on the base, as you found), the cancel-first order, a tool or callback that shields its own timed-out wait, and an ordinary callback error after a caught cancel: 19 cases, all COMPLETED. The PR body's withTimeout sentence is replaced.

and reports the node as failed (`NodeReportedError`); the plugin ends the
execution CANCELLED with the model Activity's own cancellation instead. A
cancelled Activity node ends it CANCELLED the same way, inside a dynamic run too.
So does a cancel that ADK absorbs in a tool or a model callback, such as a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: the breaking-change notes sort affected closed runs into COMPLETED and CANCELLED, but graph runs that ended FAILED and runs that continued as new change to CANCELLED too, and their closed histories fail replay. (a) A graph, new Workflow({ edges: [['START', agent]] }), whose agent node's beforeModelCallback awaits a cancellable Activity, cancelled during it: base FAILED with ApplicationFailure type GoogleAdkNodeReportedError ("Node 'assistant' failed: Activity cancelled"); head CANCELLED, because toWorkflowFailure now finds the recording (absorbed-failure.ts:330). The base history fails replay on the head with [TMPRL1100] Nondeterminism error: Cancel workflow machine does not handle this event, and the head history fails on the base ("Fail workflow machine"). (b) A main-function tool turn followed by continueAsNew, the long-conversation shape this README describes: on the base the first run CONTINUED_AS_NEW and the next run COMPLETED, so the cancel was lost across the chain; on the head the run ends CANCELLED with no new run (raiseAbsorbed in the ContinueAsNew catch, absorbed-failure.ts:357-361), and the base history fails replay on the head with TMPRL1100. Both are real fixes, (b) arguably the most important one here, but the body says closed runs that already ended CANCELLED replay and only COMPLETED ones fail, and the upgrade advice and release notes inherit that. Add both classes to 'Breaking change' and to the release-notes sentence, and say here that a graph run whose agent node absorbed the cancel now ends CANCELLED rather than failing with GoogleAdkNodeReportedError.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added both to "Breaking change" in the PR body and to the release-notes sentence. In 6e83a65 the README paragraph also says that an agent node that absorbed the cancel in a tool or model callback ends the execution CANCELLED rather than failing it with GoogleAdkNodeReportedError.

  • (a) Graph runs: the new graph.test.ts test's waiting cases check the outcome. They end CANCELLED with this change, and the callback cases end FAILED on the base. The replay result in the body is yours.
  • (b) Continue-as-new: I reran it at 6e83a65 with ADK 2.2.0, a main-function tool turn followed by continueAsNew, cancelled during the tool wait. This change ends the run CANCELLED with no new run. The base's absorbed-failure.ts ends it CONTINUED_AS_NEW, and the next run goes on.

I also added the Signal-handler model-call runs from your first comment, which the hand-off to the main function changes on every ADK version.

const handle = await env.client.workflow.start(agentRunnerLongRunningToolWait, { taskQueue, workflowId });
// ADK before 2.3 drops a long-running tool's error, so no model call follows to fail in
// the cancelled scope, and the turn ends normally.
while (!(await handle.query(adkToolWaitingQuery))) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 (tests): this poll has no deadline, so a regression where the tool never starts waiting hangs until ava's two-minute timeout instead of failing with a message, and on ADK 2.3 the case passes without the wrap. waitForScheduledActivities bounds the same kind of poll at 10 s (helpers.ts:446); a deadline and a throw like that one would do. On 2.3 a long-running tool's error goes to the model as a function response (agents/functions.js:412 in 2.3.0), TemporalModel records the model call that follows in the cancelled scope, and the issue's own 2.3.0 output shows the base ending this case CANCELLED. Nightly sync-latest resolves 2.3.x inside <2.4.0, so there nothing guards the runOnToolErrorCallback wrap. A tool case that ends the turn on the error's function response would keep it guarded on every release (for example a tool that sets toolContext.actions.skipSummarization before it waits, which isFinalResponse treats as final); at least say in the comment above that the case only discriminates before 2.3.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did both.

  • The poll now goes through waitForStage, which throws after 10 s (1fb0119).
  • The test tool sets toolContext.actions.skipSummarization before it waits (6e83a65). So on ADK 2.3 the turn ends on the error's function response, and no model call follows.

Runs at 6e83a65:

  • With ADK 2.3.0, without the runOnToolErrorCallback wrap, this test now fails at its tool case (the Workflow returns ''). Before the change, it passed without the wrap, as you said.
  • With the wrap, all 20 absorbed-failure.test.ts tests pass on 2.3.0.
  • On 2.2.0 the tool case still fails without the wrap.

…l wait for the main function

A cancel absorbed in a Signal handler ended the Workflow when the handler
returned, so the main function's non-cancellable cleanup never ran. When
a Signal handler returns, the cancellation it recorded now moves to the
main function's frame, which raises it when the main function returns.
It outranks the model failures the handler absorbed, so those move with
it and are dropped. An Update handler still raises its own.

A cancel that only an inner scope caused (CancellationScope.withTimeout)
was recorded, and decided a later cancellation that the Workflow caught
and finished on. Now only the execution's own cancellation is recorded,
at the hand-offs and for model calls alike. The root scope's reason
reaches every scope it cancels as the same object, and a new outbound
interceptor marks the failure of an Activity, local Activity, child
Workflow or Nexus operation (its start included) whose scope that reason
cancelled first. At the hand-off the plugin follows the error's causes,
and what a tool's execute or a plugin hook threw before ADK replaced it
with a plain Error. An ordinary error is not recorded either, even after
the Workflow caught its cancellation.

A cancellation absorbed in a node attempt that ADK's node timeout
abandoned is left out: ADK had moved past that work. The plugin runs
each BaseNode.run in an async-local attempt and marks it abandoned when
the node's timeout, not the invocation's abort, aborts its signal.

The tests poll with a 10-second deadline, cover a Signal-driven turn's
cleanup, a timed-out turn followed by a caught cancel, Nexus waits, and
a node timeout's abandoned work.
… name the graph outcome

From ADK 2.3 a long-running tool's error goes to the model as a function
response, and the model call that follows ends the run CANCELLED without
the runOnToolErrorCallback wrap, so the tool cases passed without it.
The test tool now sets skipSummarization before it waits, which ends the
turn on that response, so no model call follows on any ADK release.

The README now says that an agent node that absorbed the cancel in a
tool or model callback ends the execution CANCELLED rather than failing
it with GoogleAdkNodeReportedError.
@campoy campoy changed the title Breaking: end Google ADK TypeScript runs CANCELLED when a tool or model callback absorbs the cancel Breaking: end Google ADK TypeScript runs CANCELLED when ADK absorbs the cancel in a tool or model callback Oct 31, 2026
@campoy

campoy commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. This round is two new commits on top of 32f2165: 1fb0119 and 6e83a65. I've replied in each thread.

On the summary's P3: the title now says "when ADK absorbs the cancel in a tool or model callback". The body drops the "Draft while…" line and the open question. That question is settled the way you suggested: a Signal handler's cancellation goes to the main function.

1fb0119 is bigger than your four comments alone would need. While you reviewed 32f2165, the same two problems (the Signal handler and the inner-scope cancel) went through more review rounds on our side. Those rounds found gaps in the simpler fixes:

  • A root-scope check misses two cases: a turn in nonCancellable(() => withTimeout(...)) after the Workflow caught its cancellation, and a tool or callback that shields its own timed-out wait. So the plugin now follows the root's reason object from the operation it cancelled. A new outbound interceptor marks Activity, local Activity, child Workflow and Nexus failures; a Nexus operation cancelled before it starts fails its start, not its result. Wraps on FunctionTool.runAsync and PluginManager.runCallbacks keep what a tool or hook threw before ADK replaced it with a plain Error.
  • An ADK node timeout abandons its attempt without cancelling it. So the execution's cancellation could cancel that leftover work after the graph returned, and end CANCELLED a run that caught its cancellation and finished. The plugin now wraps BaseNode.prototype.run to track each attempt, and drops a cancellation absorbed in an attempt that the timeout abandoned. A retry, or a ParallelWorker item, is an attempt of its own.

Each part has tests: three cases in absorbed-failure.test.ts and one in graph.test.ts (the PR body lists them).

Checks at 6e83a65 (Node 24.22.0, pnpm 10.27.0, @google/adk 2.2.0 from the lock): make lint (no errors; the existing deprecation warning), make test 236 of 236, make build and scripts/ci/check_conventions.py pass. With ADK 2.3.0 all 20 absorbed-failure.test.ts tests pass.

A Query to a Worker with no Workflow cache sometimes takes about 10 s,
the Workflow Task timeout. main's CI shows it too, in the HITL tests.
The new stage polls allowed 10 s (absorbed-failure.test.ts) and 20 s
(graph.test.ts), so one or two such Queries failed them on Ubuntu,
macOS and Windows. They now allow 60 s, still below ava's 2-minute
timeout, so a run that never reaches the stage still fails with a
message.
Querying a Workflow on a Worker with no Workflow cache (the test default)
slows the Workflow itself. In a plain Workflow running 100 short
Activities, four clients querying it made each run take 10 to 21 s
instead of 4 to 6 s, with queries up to 20 s. On CI that let ADK's
1-second node timeout fire on the retry in graph.test.ts, so the graph
failed with NodeTimeoutError and never reached its stage, on macOS and
Windows even with the 60 s deadline of 626c9f7.

The test Workflows now record their stage with upsertMemo, and the tests
read it with describe(), which the server answers without the Worker.
The deadlines go back to 10 s and 20 s.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] google-adk (TypeScript): a cancelled Workflow ends COMPLETED when ADK absorbs the cancel in a tool or model callback

2 participants