diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index e496bc386773a6..08274927ea61db 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -21,7 +21,7 @@ This integration is enabled by default when tracing The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers. -If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). +If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/). ```javascript Sentry.init({ diff --git a/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx new file mode 100644 index 00000000000000..f1a543e4be7095 --- /dev/null +++ b/docs/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations.mdx @@ -0,0 +1,52 @@ +--- +title: Mastra Classifier Evaluations +sidebar_title: Classifier Evaluations +description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.evaluate spans." +--- + + + +Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the [Mastra Quick Start](/platforms/javascript/guides/mastra/), no additional integration or setup is needed. + +```javascript {filename:src/mastra/classifiers/ticket-classifier.ts} +import { Classifier } from "@mastra/core/classifier"; +import { typeSafeAi } from "@ai-sdk/typesafe-ai"; + +const classifier = new Classifier({ + id: "ticket-classifier", + model: typeSafeAi.evaluationModel("jev-latest"), + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +const result = await classifier.evaluate({ + state: { message: "I was charged twice. Please refund the extra charge." }, +}); + +console.log(result.answers.department.choice); +``` + +## What Gets Captured + +Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model. + +## Control Recorded Data + +Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the `dataCollection` option. + +## Supported Versions + +- `@sentry/node`: `>=11.6.0` +- `@mastra/core`: `>=1.69.0` + +## Related Integrations + +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration. diff --git a/includes/mastra/what-gets-captured.mdx b/includes/mastra/what-gets-captured.mdx index 4e1440f1d54d9e..00ec9f6d2208a1 100644 --- a/includes/mastra/what-gets-captured.mdx +++ b/includes/mastra/what-gets-captured.mdx @@ -2,16 +2,17 @@ Sentry's Mastra integration maps Mastra's span types to Sentry operations for the Agents dashboards: -| Mastra Span Type | Sentry Operation | -| -------------------- | --------------------- | -| `AGENT_RUN` | `gen_ai.invoke_agent` | -| `WORKFLOW_RUN` | `gen_ai.invoke_agent` | -| `MODEL_GENERATION` | `gen_ai.chat` | -| `TOOL_CALL` | `gen_ai.execute_tool` | -| `MCP_TOOL_CALL` | `gen_ai.execute_tool` | -| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` | -| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` | -| `RAG_EMBEDDING` | `gen_ai.embeddings` | +| Mastra Span Type | Sentry Operation | +| ----------------------- | --------------------- | +| `AGENT_RUN` | `gen_ai.invoke_agent` | +| `WORKFLOW_RUN` | `gen_ai.invoke_agent` | +| `MODEL_GENERATION` | `gen_ai.chat` | +| `TOOL_CALL` | `gen_ai.execute_tool` | +| `MCP_TOOL_CALL` | `gen_ai.execute_tool` | +| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` | +| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` | +| `RAG_EMBEDDING` | `gen_ai.embeddings` | +| `CLASSIFIER_EVALUATION` | `gen_ai.evaluate` | Spans are tagged with `sentry.origin: auto.ai.mastra`.