agent-eval-harness

agent-eval-harness is a trajectory-level evaluation harness for agent runs. It evaluates the whole run, derives per-turn state and state diffs, anchors findings to turn spans, measures judge recall against labeled or seeded defects, and wires results into gates.

The default implementation is offline and deterministic so tests and CI do not require a model key. Hosted model access belongs behind the JudgeProvider interface.

Why trajectory-level?

Single-turn judging misses defects that only become visible across turns: stale state after a user update, guardrails that remain active over several actions, ignored tool failures, contradictions, and unresolved commitments. This package makes those cross-turn signals first-class.

flowchart LR
  A[Trajectory JSON/JSONL] --> B[StateExtractor]
  B --> C[State snapshots]
  B --> D[State diffs]
  A --> E[TrajectoryJudge]
  D --> E
  E --> F[Axis scores]
  E --> G[Findings with turn spans]
  G --> H[RecallEvaluator]
  I[Human or seeded labels] --> H
  F --> J[GateWiring]
  G --> J
  H --> J
  K[SeededDefectGenerator] --> I
  K --> A

Loading

Install

bun install
bun test
bun run typecheck

Quickstart

Run the reproducible synthetic benchmark:

bun run demo

Evaluate a trajectory JSON file:

bun run src/cli.ts judge examples/trajectory.json

Inject seeded defects:

bun run src/cli.ts seed examples/trajectory.json state-tracking,guardrail,recovery

Use the library:

import {
  DeterministicTrajectoryJudge,
  RecallEvaluator,
  SeededDefectGenerator,
  createSyntheticTrajectory,
} from "agent-eval-harness";

const base = createSyntheticTrajectory();
const seeded = new SeededDefectGenerator({ seed: 7 }).inject(base, [
  { axis: "state-tracking", severity: "high" },
]);

const judgment = await new DeterministicTrajectoryJudge().judge(seeded.trajectory);
const recall = new RecallEvaluator().evaluate(judgment.findings, seeded.labels);
console.log(recall.overall);

Input Format

A trajectory is a JSON object:

{
  "id": "example-run",
  "metadata": { "source": "synthetic" },
  "turns": [
    {
      "id": "t1",
      "role": "user",
      "content": "destination = north warehouse\nconstraint: do not use overnight shipping"
    },
    {
      "id": "t2",
      "role": "assistant",
      "content": "I will use ground shipping to the north warehouse.",
      "toolCalls": [{ "name": "plan_delivery", "arguments": { "mode": "ground" } }]
    },
    {
      "id": "t3",
      "role": "tool",
      "content": "plan accepted",
      "toolResults": [{ "name": "plan_delivery", "content": "accepted" }]
    }
  ]
}

Roles are system, user, assistant, or tool. Validation errors include the failing path, such as turns[0].content.

Evaluation Axes

• state-tracking: stale task variables, ignored state changes, wrong entity after update.
• guardrail: active constraint or policy instruction violated later in the run.
• recovery: tool or execution failure not acknowledged or recovered.
• consistency: contradictions across assistant commitments or claims.
• safety: unsafe or policy-bypassing instruction surfaced.
• task-completion: trajectory ends with unresolved work or a promise instead of completion.

Methodology

The headline demo uses deterministic synthetic traces rather than heavy external benchmark downloads. The seeded generator injects known defects and emits ground truth labels, allowing recall and precision to be measured without manual labeling. This is not a claim about absolute real-world performance; it is a reproducible demonstration that state-diff trajectory judging detects cross-turn seeded defects that a naive per-turn judge misses.

For open benchmark use, adapt tau-bench, AppWorld, or WebArena-style traces into the trajectory schema, add human-confirmed or seeded labels, and compare:

• NaivePerTurnJudge: sees each turn locally and is intentionally state-blind.
• DeterministicTrajectoryJudge: sees the full trajectory and state diffs.
• MultiJudgePanel: merges multiple judge outputs and deduplicates findings.

RecallEvaluator reports recall, precision, per-axis metrics, matches, warnings, and a Rogan-Gladen style corrected prevalence estimate when sensitivity, specificity, and total trajectory count are supplied. It explicitly flags that zero defects found is uninformative without human or seeded labels.

Model Providers

The core package does not hardcode any provider, host, or key. Implement JudgeProvider to connect a model:

import { AXES, type JudgeProvider, type JudgeProviderInput, type TrajectoryJudgment } from "./src";

export class MyProvider implements JudgeProvider {
  readonly id = "my-provider";

  async judgeTrajectory(input: JudgeProviderInput): Promise<TrajectoryJudgment> {
    // Send input.trajectory and input.state to your model, validate the response,
    // and return a TrajectoryJudgment.
    const axisScores = Object.fromEntries(
      AXES.map((axis) => [
        axis,
        { axis, score: 1, rationale: "Replace with provider output." },
      ]),
    ) as TrajectoryJudgment["axisScores"];

    return {
      trajectoryId: input.trajectory.id,
      judgeId: this.id,
      axisScores,
      findings: [],
    };
  }
}

Keep provider credentials in your own application environment, not in this repository.

Development

bun install
bun test
bun run typecheck
bun run demo

License

MIT