From d1b17965b139ccd1735ebe8debf839d98227bb4e Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:01:36 +0000 Subject: [PATCH 1/2] docs(mastra): Document classifier evaluation tracing Co-Authored-By: Andrei Borza --- .../configuration/integrations/typesafe.mdx | 2 +- .../javascript/guides/mastra/index.mdx | 38 +++++++++++++++++++ includes/mastra/what-gets-captured.mdx | 21 +++++----- 3 files changed, 50 insertions(+), 11 deletions(-) diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index e496bc386773a..30ffe1cb98a80 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -21,7 +21,7 @@ This integration is enabled by default when tracing The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers. -If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). +If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra](/platforms/javascript/guides/mastra/#trace-classifier-evaluations). ```javascript Sentry.init({ diff --git a/docs/platforms/javascript/guides/mastra/index.mdx b/docs/platforms/javascript/guides/mastra/index.mdx index 43ecc77e79488..98f03ac1e6fb9 100644 --- a/docs/platforms/javascript/guides/mastra/index.mdx +++ b/docs/platforms/javascript/guides/mastra/index.mdx @@ -143,6 +143,44 @@ Preload the instrument file when you start Mastra by passing it through `--custo +## Trace Classifier Evaluations + + + +Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. No additional integration or setup is needed. Classifiers require `@mastra/core` version `1.69.0` or later. + +```javascript {filename:src/mastra/classifiers/ticket-classifier.ts} +import { Classifier } from "@mastra/core/classifier"; +import { typeSafeAi } from "@ai-sdk/typesafe-ai"; + +const classifier = new Classifier({ + id: "ticket-classifier", + model: typeSafeAi.evaluationModel("jev-latest"), + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +const result = await classifier.evaluate({ + state: { message: "I was charged twice. Please refund the extra charge." }, +}); + +console.log(result.answers.department.choice); +``` + +Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. + +Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. + +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration. + ## Supported Versions - `@sentry/node`: `>=11.0.0` diff --git a/includes/mastra/what-gets-captured.mdx b/includes/mastra/what-gets-captured.mdx index 4e1440f1d54d9..00ec9f6d2208a 100644 --- a/includes/mastra/what-gets-captured.mdx +++ b/includes/mastra/what-gets-captured.mdx @@ -2,16 +2,17 @@ Sentry's Mastra integration maps Mastra's span types to Sentry operations for the Agents dashboards: -| Mastra Span Type | Sentry Operation | -| -------------------- | --------------------- | -| `AGENT_RUN` | `gen_ai.invoke_agent` | -| `WORKFLOW_RUN` | `gen_ai.invoke_agent` | -| `MODEL_GENERATION` | `gen_ai.chat` | -| `TOOL_CALL` | `gen_ai.execute_tool` | -| `MCP_TOOL_CALL` | `gen_ai.execute_tool` | -| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` | -| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` | -| `RAG_EMBEDDING` | `gen_ai.embeddings` | +| Mastra Span Type | Sentry Operation | +| ----------------------- | --------------------- | +| `AGENT_RUN` | `gen_ai.invoke_agent` | +| `WORKFLOW_RUN` | `gen_ai.invoke_agent` | +| `MODEL_GENERATION` | `gen_ai.chat` | +| `TOOL_CALL` | `gen_ai.execute_tool` | +| `MCP_TOOL_CALL` | `gen_ai.execute_tool` | +| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` | +| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` | +| `RAG_EMBEDDING` | `gen_ai.embeddings` | +| `CLASSIFIER_EVALUATION` | `gen_ai.evaluate` | Spans are tagged with `sentry.origin: auto.ai.mastra`. From 169dcc4ef58e08261f22581a4221a1bb31bb8e39 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:59:37 +0000 Subject: [PATCH 2/2] docs(mastra): Move classifier evaluations to a dedicated page Keep the Quick Start focused on setup and document classifier evaluation tracing under the Mastra Agent Tracing section, like Workers AI for Cloudflare. Co-Authored-By: Andrei Borza --- .../configuration/integrations/typesafe.mdx | 2 +- .../agent-tracing/classifier-evaluations.mdx | 52 +++++++++++++++++++ .../javascript/guides/mastra/index.mdx | 38 -------------- 3 files changed, 53 insertions(+), 39 deletions(-) create mode 100644 docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index 30ffe1cb98a80..08274927ea61d 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -21,7 +21,7 @@ This integration is enabled by default when tracing The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers. -If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra](/platforms/javascript/guides/mastra/#trace-classifier-evaluations). +If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/). ```javascript Sentry.init({ diff --git a/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx new file mode 100644 index 0000000000000..f1a543e4be709 --- /dev/null +++ b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx @@ -0,0 +1,52 @@ +--- +title: Mastra Classifier Evaluations +sidebar_title: Classifier Evaluations +description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.evaluate spans." +--- + + + +Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the [Mastra Quick Start](/platforms/javascript/guides/mastra/), no additional integration or setup is needed. + +```javascript {filename:src/mastra/classifiers/ticket-classifier.ts} +import { Classifier } from "@mastra/core/classifier"; +import { typeSafeAi } from "@ai-sdk/typesafe-ai"; + +const classifier = new Classifier({ + id: "ticket-classifier", + model: typeSafeAi.evaluationModel("jev-latest"), + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +const result = await classifier.evaluate({ + state: { message: "I was charged twice. Please refund the extra charge." }, +}); + +console.log(result.answers.department.choice); +``` + +## What Gets Captured + +Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. + +## Control Recorded Data + +Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. + +## Supported Versions + +- `@sentry/node`: `>=11.6.0` +- `@mastra/core`: `>=1.69.0` + +## Related Integrations + +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration. diff --git a/docs/platforms/javascript/guides/mastra/index.mdx b/docs/platforms/javascript/guides/mastra/index.mdx index 98f03ac1e6fb9..43ecc77e79488 100644 --- a/docs/platforms/javascript/guides/mastra/index.mdx +++ b/docs/platforms/javascript/guides/mastra/index.mdx @@ -143,44 +143,6 @@ Preload the instrument file when you start Mastra by passing it through `--custo -## Trace Classifier Evaluations - - - -Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. No additional integration or setup is needed. Classifiers require `@mastra/core` version `1.69.0` or later. - -```javascript {filename:src/mastra/classifiers/ticket-classifier.ts} -import { Classifier } from "@mastra/core/classifier"; -import { typeSafeAi } from "@ai-sdk/typesafe-ai"; - -const classifier = new Classifier({ - id: "ticket-classifier", - model: typeSafeAi.evaluationModel("jev-latest"), - questions: { - department: { - type: "choice", - instructions: "Which team should handle this?", - criteria: { - billing: "Charges and refunds", - technical: "Bugs and outages", - }, - }, - }, -}); - -const result = await classifier.evaluate({ - state: { message: "I was charged twice. Please refund the extra charge." }, -}); - -console.log(result.answers.department.choice); -``` - -Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. - -Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. - -If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration. - ## Supported Versions - `@sentry/node`: `>=11.0.0`