Repository navigation
feat(server-utils): Instrument the OpenAI decisions API as gen_ai.decide spans #25285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
+437
−34
Open
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
4ca936f
feat(server-utils): Instrument the OpenAI decisions API as gen_ai.dec…
andreiborza 24f4a8b
Match the decisions mock to a real gpt-6-luna response
andreiborza 81b6bc0
Add node-openai-decisions-send-to-sentry e2e test app
andreiborza 99155fb
Match decide spans by input instead of order in the decisions test
andreiborza bf0b8b8
Use the shared test-utils/cli helpers in the decisions e2e app
andreiborza 19ed67b
Inline the trace id lookup in the decisions e2e app
andreiborza 28c77c1
Assert the decisions e2e messages against the full recorded JSON
andreiborza ec536bc
Poll for the request span parent in the decisions e2e test
andreiborza 12dae22
Assert the stored span name in the decisions e2e test
andreiborza 4fefcd6
Lay out decision choice and score answers like Jev answers, keyed by …
andreiborza 95ff42d
Lay out decision questions like Jev questions, with options in criteria
andreiborza 0cf6661
Merge remote-tracking branch 'origin/develop' into ab/openai-decisions
andreiborza 2c6bbaf
Drop image data from top-level decisions input parts
andreiborza 50bba01
Record the decisions input when the span starts
andreiborza b61e351
Write the decisions e2e test steps inline, like the Cloudflare app
andreiborza d09c009
Record decision answers only for decide calls and move the mapping to…
andreiborza 0ba719e
Limit the decisions hook to openai 7.30.0 and later
andreiborza 836145b
Merge branch 'develop' into ab/openai-decisions
andreiborza 10f4154
Merge branch 'develop' into ab/openai-decisions
andreiborza 2eec966
Move the decisions e2e app to its own PR
andreiborza 5112653
Cover cache tokens in the decisions integration test
andreiborza 72df2c5
Keep decision choices without a description in the criteria
andreiborza 13e90d7
Set the decision input and tool definitions without a guard
andreiborza c316841
Record the decisions input images as they are
andreiborza 8cf3f78
Record decision score level descriptions
andreiborza 4ee7395
Key decision score level descriptions by score value
andreiborza File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
9 changes: 9 additions & 0 deletions
9
dev-packages/node-integration-tests/suites/tracing/openai/v7/instrument-with-recording.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| import * as Sentry from '@sentry/node'; | ||
| import { loggingTransport } from '@sentry-internal/node-integration-tests'; | ||
|
|
||
| Sentry.init({ | ||
| dsn: 'https://public@dsn.ingest.sentry.io/1337', | ||
| release: '1.0', | ||
| tracesSampleRate: 1.0, | ||
| transport: loggingTransport, | ||
| }); |
115 changes: 115 additions & 0 deletions
115
dev-packages/node-integration-tests/suites/tracing/openai/v7/scenario-decisions.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| import * as Sentry from '@sentry/node'; | ||
| import express from 'express'; | ||
| import OpenAI from 'openai'; | ||
|
|
||
| function startMockServer() { | ||
| const app = express(); | ||
| app.use(express.json({ limit: '1mb' })); | ||
|
|
||
| app.post('/openai/decisions', (req, res) => { | ||
| const { model } = req.body; | ||
|
|
||
| if (model === 'error-model') { | ||
| res.status(500).set('x-request-id', 'mock-request-error').end('Internal server error'); | ||
| return; | ||
| } | ||
|
|
||
| // Shaped like a real `gpt-6-luna` response. Decisions bills input tokens only. | ||
| res.send({ | ||
| model: 'gpt-6-luna', | ||
| answers: [ | ||
| { type: 'predicate', name: 'wants_refund', probability: 0.97 }, | ||
| { | ||
| type: 'choice', | ||
| name: 'department', | ||
| choice: 'billing', | ||
| probabilities: [ | ||
| { value: 'billing', probability: 0.82 }, | ||
| { value: 'other', probability: 0.18 }, | ||
| ], | ||
| confidence: 0.82, | ||
| }, | ||
| { | ||
| type: 'score', | ||
| name: 'urgency', | ||
| score: 0.95, | ||
| probabilities: [ | ||
| { value: 0, label: 'low', probability: 0.05 }, | ||
| { value: 1, label: 'high', probability: 0.95 }, | ||
| ], | ||
| confidence: 0.93, | ||
| }, | ||
| ], | ||
| usage: { | ||
| input_tokens: 382, | ||
| input_tokens_details: { cached_tokens: 256, cache_write_tokens: 64 }, | ||
| output_tokens: 0, | ||
| output_tokens_details: { reasoning_tokens: 0 }, | ||
| total_tokens: 382, | ||
| }, | ||
| }); | ||
| }); | ||
|
|
||
| return new Promise(resolve => { | ||
| const server = app.listen(0, () => { | ||
| resolve(server); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| const questions = [ | ||
| { type: 'predicate', name: 'wants_refund', instructions: 'Is a refund requested?' }, | ||
| { | ||
| type: 'choice', | ||
| name: 'department', | ||
| instructions: 'Which team should handle this?', | ||
| choices: [{ value: 'billing', description: 'Charges and refunds' }, { value: 'other' }], | ||
| }, | ||
| { | ||
| type: 'score', | ||
| name: 'urgency', | ||
| instructions: 'How urgent is this ticket?', | ||
| levels: [{ label: 'low' }, { label: 'high', description: 'Blocks the customer' }], | ||
| }, | ||
| ]; | ||
|
|
||
| async function run() { | ||
| const server = await startMockServer(); | ||
|
|
||
| await Sentry.startSpan({ op: 'function', name: 'main' }, async () => { | ||
| const client = new OpenAI({ | ||
| baseURL: `http://localhost:${server.address().port}/openai`, | ||
| apiKey: 'mock-api-key', | ||
| }); | ||
|
|
||
| await client.decisions.create({ | ||
| model: 'gpt-6-luna', | ||
| input: 'I was charged twice and want a refund.', | ||
| questions, | ||
| }); | ||
|
|
||
| await client.decisions.create({ | ||
| model: 'gpt-6-luna', | ||
| input: [ | ||
| { | ||
| role: 'user', | ||
| content: [ | ||
| { type: 'input_text', text: 'Is the package damaged?' }, | ||
| { type: 'input_image', image_url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUg==' }, | ||
| ], | ||
| }, | ||
| ], | ||
| questions: [{ type: 'predicate', name: 'damaged', instructions: 'Is there visible damage?' }], | ||
| }); | ||
|
|
||
| try { | ||
| await client.decisions.create({ model: 'error-model', input: 'Error test', questions }); | ||
| } catch { | ||
| // Error is expected and handled | ||
| } | ||
| }); | ||
|
|
||
| server.close(); | ||
| } | ||
|
|
||
| run(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.