Skip to content

Instrument OpenAI Decisions (decisions.create) and Vercel AI experimental_decide with OpenAI models #25253

Open

Description

@andreiborza

OpenAI added the Decisions API (POST /v1/decisions, model gpt-6-luna). It is similar to Jev: the caller sends shared evidence and typed questions (predicate, choice, score), and the model returns typed answers. Users can call it in two ways:

  1. openai SDK: client.decisions.create({ model, input, questions }) (in openai 7.30.0 and later). Today this makes no span. The method is not in OPENAI_METHOD_REGISTRY or in the orchestrion config.
  2. Vercel AI: experimental_decide({ model: openai.decisionModel('gpt-6-luna'), ... }) (@ai-sdk/openai 4.x). The channel subscriber already makes the span, but it drops the answer confidence. withProviderConfidence() only reads providerMetadata.typesafe.confidence, and OpenAI puts it in providerMetadata.openai.confidence.

Target span

Use the same shape as the other gen_ai.decide spans (see #25249 / #25251), so that all paths look the same in the product.

  • Name: decide <model>, op gen_ai.decide, origin auto.ai.openai
  • gen_ai.operation.name: decide, gen_ai.provider.name: openai
  • gen_ai.request.model, gen_ai.response.model (result.model)
  • gen_ai.usage.input_tokens, output_tokens, total_tokens. Also input_tokens.cached (usage.input_tokens_details.cached_tokens) and output_tokens.reasoning (usage.output_tokens_details.reasoning_tokens).
  • gen_ai.input.messages: [{ type: 'evaluation', state: <input>, questions }] (only when recordInputs is on). Map the OpenAI input field to state to keep the shape the same as Jev.
  • gen_ai.output.messages: [{ type: 'evaluation', answers }] (only when recordOutputs is on).
  • Set the span status to error when the call throws. A refusal answer is not an error.

Scope

  • OpenAI integration: add decisions.create to both paths (the Proxy registry in src/ai/openai/ and src/orchestrion/config/openai.ts). It needs its own operation, because the chat response handling does not fit. The API does not stream.
  • Vercel AI: read the confidence from any providerMetadata.<provider>.confidence, not only from typesafe. This also fixes OpenRouter (Keep Jev answer confidence for OpenRouter evaluation models in Vercel AI #24860).
  • Input images: input can contain input_image parts with base64 data URLs. Do not record the raw base64 data.
  • Tests: a node-integration test with a mocked OpenAI server (needs openai@^7.30, the root has 5.18.1), and a Vercel AI case with openai.decisionModel().

Notes

Activity

  1. linear-code commented on Oct 12, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions