Skip to content

list-experiments MCP tool throws: repo_info.dirty is required in Zod schema but absent in API responses #2164

Description

@swini-grammarly

Description

The list-experiments MCP tool fails entirely when any returned experiment has a repo_info object that lacks the dirty field. This happens whenever evals are run without git context enabled — which appears to be the common case.

Error

Every experiment object in the response fails Zod validation with:

[
  {
    "code": "invalid_type",
    "expected": "boolean",
    "received": "undefined",
    "path": ["objects", 0, "repo_info", "dirty"],
    "message": "Required"
  },
  ...
]

The error fires for every experiment in the list, so the tool returns nothing even if results exist.

Root Cause

The repo_info.dirty field in the response schema is marked as a required boolean, but the Braintrust API omits it from experiment records when the eval runner doesn't have git context (no working tree, CI without git, etc.).

Fix

Mark dirty as optional in the repo_info Zod schema:

- dirty: z.boolean()
+ dirty: z.boolean().optional()

Other git metadata fields in repo_info (branch, commit, tag, etc.) may have the same issue and are worth auditing for optionality.

Reproduction

  1. Configure the Braintrust MCP server against any org that runs evals in CI or without a git working tree
  2. Call list-experiments with any valid projectId
  3. Observe: tool throws with Zod validation errors instead of returning the experiment list

Tested against project Agent/SalmonUserJourneyEvalJob-head (id: 3f8b01c4-f6a2-4597-b3ac-42707c174f4c) which has 7+ experiments, all failing validation.

Expected Behavior

list-experiments returns the experiment list with dirty absent or undefined on records where it was not captured.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions