Repository navigation
feat(server-utils): Emit execute_tool spans for Anthropic's beta.messages.toolRunner() - #25320
RulaKhaled wants to merge 7 commits into
Conversation
…essages.toolRunner()` Wraps each runnable tool's `run()` in place with `fill()`, covering tools passed to `toolRunner()`, `addTools()` and `setMessagesParams()`, on both the orchestrion and `instrumentAnthropicAiClient` paths. A tool that throws gets an error span and a captured event, since the runner hides the error from the app. Fixes #25288 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… setting them to undefined Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…echanism type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
size-limit report 📦
|
|
bugbot run |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
isaacs
left a comment
isaacs
left a comment
There was a problem hiding this comment.
Wrapipng tool objects in place rather than wrapping the runner or proxying is the right choice for this SDK: the runner uses private class fields, the client code already avoids Proxies for that reason, and tool.run is the one call point that is stable from 0.63 to 0.133.
Capturing the swallowed error is the useful part of the change, since otherwise those errors disappear.
Open questions are what counts as "handled", and keeping the run contract narrow enough that new kinds of tools entries, like toolsets, don't get wrapped by accident. Otherwise LGTM!
…rors as handled Toolsets are nameless `tools` entries whose `run` takes `(ctx, toolUse)`, so only entries with a string `name` are wrapped now. Tool errors the runner hands back to the model are captured with `handled: true`, as Flue and Mastra do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5e0cfc3. Configure here.
…on older SDKs SDK 0.85 to 0.96 pass only `toolUseBlock` to a tool's `run()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
isaacs
left a comment
isaacs
left a comment
There was a problem hiding this comment.
This looks great!
Anthropic SDK 0.133 adds client.beta.sessions.events.toolRunner(sessionID, { tools }) (lib/tools/SessionToolRunner.ts), which also runs BetaRunnableTools. Those tools get no spans unless the same objects were also given to a beta.messages.toolRunner() (in which case they get spans with the wrong framing, see 1). I think that's out of scope for this PR, but worth a follow-up issue so the PR body saying "the tool runner" isn't read as full coverage.
| })), | ||
| ...['resources/beta/messages/messages.js', 'resources/beta/messages/messages.mjs'].map(filePath => ({ | ||
| channelName: 'tool-runner', | ||
| module: { name: '@anthropic-ai/sdk', versionRange: '>=0.19.2 <1', filePath }, |
There was a problem hiding this comment.
toolRunner only exists from 0.63, so we can tighten this up a little bit :)
| module: { name: '@anthropic-ai/sdk', versionRange: '>=0.19.2 <1', filePath }, | |
| module: { name: '@anthropic-ai/sdk', versionRange: '>=0.63.0 <1', filePath }, |
| // Handled: the runner turns the throw into an error result for the model, so the app never sees it. | ||
| // A `ToolError` is the SDK's way to send that result on purpose. | ||
| if ((error as Error | undefined)?.name !== 'ToolError') { | ||
| captureException(error, { mechanism: { handled: true, type: ANTHROPIC_AI_ORIGIN } }); |
There was a problem hiding this comment.
So, fill replaces run on the user's own tool object for the life of the process. Tool objects are often module-level constants, reused in other places: a direct call in app code or tests, an MCP server, another agent framework, or client.beta.sessions.events.toolRunner().
Every such call now gets an execute_tool span with origin: auto.ai.anthropic, and any throw gets captured with mechanism: { handled: true, type: 'auto.ai.anthropic' }. In those contexts the caller does see the error, so "handled by the Anthropic runner" is wrong. The __sentry_captured__ guard stops a second capture when the app also reports it, but the first (wrong) mechanism wins.
There is no clean way to tell a runner call from a direct call on 0.63 to 0.80 (no context arg). On 0.85+ the runner always passes a context object with toolUseBlock.
I don't think we really need to change the code here, but we might want to document this in the PR body and/or a code comment, since it's a deliberate tradeoff. Maybe we could only capture when context?.toolUse ?? context?.toolUseBlock is present, which is true for every runner call on 0.85+ and false for most direct calls. That would keep the span everywhere but limit the "handled" capture to calls we know came from the runner. Older SDKs would then lose capture, which may be acceptable, or keep capture there unconditionally.
This branch has not been deployed
3 participants
The tool runner runs the user's tools, but Sentry only recorded its chat calls. Tool calls got no span, and when a tool threw, the runner turned the error into an
is_errorresult for the model, so nobody saw it.This wraps each tool's
run()in place withfill(), which the runner calls on every version since 0.63. When a tool throws, its span is marked as an error and the exception is captured. IntentionalToolErrorresults and aborted calls are marked but not captured.Fixes #25288
🤖 Generated with Claude Code