feat(eve-bench): Terminal-Bench runner that separates invalid trials from scores

[ruiconti]

Summary

Harness comparisons were measuring the harness adapters rather than the harnesses. In the last Harbor multi-harness run, pi logged 0 tokens on every trial and was still scored 0. opencode solved 4 of 5 d0 tasks but scored 0/5 because nothing captured its answers. Other harnesses lost their HOME or image environment before grading. This adds eve-bench, a zero-dependency Terminal-Bench runner. It keeps attempts that never measured the harness out of scores, and it runs every harness in the same sandbox environment.

What

• A runner that owns the container, the harness upload, the agent and verifier timeouts, and resume. It keeps the Terminal-Bench layout and reward contract, so results stay comparable with published runs.
• Harnesses: eve (local or published), pinned pi, opencode, and codex CLIs, and oracle.
• Scored or invalid, per trial.
• The same task environment for every harness.
• CI that rejects runs with invalid attempts.

Scored or invalid

	Before	After
Harness never reached the model (401, bad adapter config, crash on launch)	reward 0, counted as a task failure	invalid: harness, excluded from scores
Image, container, upload, or log download failed	reward 0	invalid: environment
Verifier wrote no reward or timed out	counted as 0	invalid: verifier
Any model call completed, then crash, timeout, or wrong answer	scored	scored, unchanged

"Reached the model" means the harness's own event log shows non-zero token usage. The host reads it from events.ndjson for eve, and from <cli>.jsonl for the CLIs. eve now writes its events as they arrive, so usage survives an agent timeout. Reports list every invalid attempt with a reason, JUnit marks them <error>, and diff shows invalid counts next to each task.

Same environment for every harness

• pi, opencode, codex: HOME and XDG_* used to point into a temp folder that was deleted before grading, and the image environment was dropped. Now they inherit the container's HOME, PATH, and image environment. Their config lives in harness-owned paths (CODEX_HOME, PI_CODING_AGENT_DIR, OPENCODE_CONFIG).
• eve bash: it had no timeout and saw the eve server's PORT, HOST, and NITRO_*. Now it has a process-group timeout (default 120 s, exit 124) and sees the task's own values. It also returns when the shell exits, even if a background job still holds the output pipe.
• eve grep: a missing rg used to look exactly like "no matches". It now says rg is missing.

CI

oracle-smoke now builds eve before the tests, because the tests import eve's built exports. It also runs the Docker stress suite. Both smoke jobs run pnpm bench report <job> --fail-on-invalid.

Not in this PR

• No real-model run of pi, opencode, or codex with a valid key yet; the adapter tests use mock binaries. That's the next step: a paid per-harness smoke gated by --fail-on-invalid.
• Codex's turn.completed usage shape was taken from the pinned binary, not a live run.
• Cost isn't comparable across harnesses. The CLIs report none for custom gateway providers, so costUsd is null there. Compare tokens.
• Hermes still fails explicitly until it has a pinned bundle.

Validation

• pnpm run test:unit (tsc + unit): 83 passed.
• pnpm run test:integration: 34 passed, 0 failed. Includes the new scenario tests:
  • trial-validity: runJob against a scripted Docker stub, covering every validity outcome.
  • eve-agent-tools: real subprocesses.
  • cli-report: the --fail-on-invalid exit codes.
• Temporarily breaking the reached-the-model check fails exactly the two harness-invalid scenarios.
• pnpm bench prepare --harness eve --model zai/glm-5.2 built the bundle, and the built server contains the new tool code.
• Replaying the recorded real-model trials (eve-smoke and the invalid-key runs for pi, codex, and opencode): all 6 are now invalid: harness, including a pi run that exited 0 after a 401.
• oxfmt, oxlint, and pnpm guard:invariants are clean for tracked files.
• Not run locally: pnpm run test:stress, because Docker was unavailable. This PR's CI runs it.

Checklist

• I linked an issue with prior discussion confirming this change is wanted
• I ran the relevant checks from CONTRIBUTING.md
• I added tests and documentation where relevant
• I added a changeset if this touches the published eve package (not applicable: @eve/bench is private)
• DCO sign-off passes for every commit (git commit --signoff)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
eve-docs	Ready	Preview, v0	Sep 25, 2026 4:54am UTC
eve-pkg	Ready	Preview, v0	Sep 25, 2026 4:54am UTC

[github-actions]

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

• No notable deltas vs main (32cd5b9).

Delta vs main (32cd5b9)

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	8.44 MB	8.44 MB	-205 B ✅
Package	Unpacked publish size	31.01 MB	31.01 MB	0 B ➖
Package	Installed footprint	77.35 MB	77.35 MB	0 B ➖
Package	Published files	3929	3929	0
Package	Installed files	7893	7893	0
Package	Installed package instances	33	33	0
Package	Distinct installed package names	32	32	0
Package	Installed dependency edges	51	51	0
Package	Installed optional peer edges	9	9	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	19.18 MB	19.18 MB	-80 B ✅
Runtime	Public routes	18	18	0

Changed function payloads vs main (32cd5b9) (2)

Function	Status	Baseline	Current	Delta	Route changes
functions/__server.func	changed	9.59 MB	9.59 MB	-40 B ✅	none
functions/.well-known/workflow/v1/flow.func	changed	9.59 MB	9.59 MB	-40 B ✅	none

eve init install

Metric	Baseline	Current	Delta
Installed footprint	115.57 MB	115.57 MB	0 B ➖
Installed packages	97	97	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	48.44 MB	48.44 MB	0 B ➖
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

• Preset: vercel
• Nitro: nitro@3.0.260903-beta
• Output directory: apps/fixtures/weather-agent/.vercel/output
• Build metadata timestamp: 2026-09-25T04:54:38.927Z
• Route aliases: 18 public, 1 internal (19 total aliases)
• Vercel routes in config: 21
• Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

• Package: eve@0.66.3
• Package directory: packages/eve
• Tarball: 8.44 MB (eve-0.66.3.tgz)
• Unpacked payload: 31.01 MB across 3929 published files
• Installed footprint: 77.35 MB across 7893 installed files
• Installed root package: 30.72 MB
• Installed dependencies: 46.63 MB
• Installed package instances: 33
• Distinct installed package names: 32
• Installed dependency edges: 51
• Installed optional peer edges: 9
• Runtime dependencies: 2
• Peer dependencies: 7 (6 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

• eve: 30.72 MB (39.7%)
• @rolldown/binding-linux-x64-gnu: 19.15 MB (24.8%)
• ai: 7.77 MB (10.0%)
• zod: 6.14 MB (7.9%)
• undici: 3.51 MB (4.5%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [##############..........] 9.70 MB 31.3%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.30 MB 7.4%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js     [#.......................] 1.03 MB 3.3%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 604.6 kB 1.9%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 514.5 kB 1.7%
🔴 Other published files                            [########################] 16.86 MB 54.4%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 30.72 MB 39.7%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.15 MB 24.8%
🔴 ai                              [######..................] 7.77 MB 10.0%
🔴 zod                             [#####...................] 6.14 MB 7.9%
🟠 undici                          [###.....................] 3.51 MB 4.5%
🟠 nitro                           [#.......................] 1.89 MB 2.4%
🔴 Other installed packages        [######..................] 8.16 MB 10.6%

Runtime dependencies (2)

Package	Range	Notes
nitro	3.0.260903-beta
undici	8.9.0

Peer dependencies (7)

Package	Range	Notes
@opentelemetry/api	^1.0.0	optional peer
ai	catalog:
braintrust	^3.0.0	optional peer
chat	^4.41.0	optional peer
dd-trace	^6.13.0	optional peer
just-bash	^3.1.0	optional peer
microsandbox	^0.5.0	optional peer

eve init install drill-down

eve init install details

• Command: eve init my-agent
• Package manager: npm
• Installed footprint: 115.57 MB across 9787 installed files
• Installed packages: 97 total (91 transitive-only)
• dependencies: 4 direct packages totaling 48.44 MB
• devDependencies: 2 direct packages totaling 5.04 MB
• Other transitive package files: 62.09 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

• eve: 30.72 MB (26.6%)
• @typescript/typescript-linux-x64: 27.95 MB (24.2%)
• @rolldown/binding-linux-x64-gnu: 19.15 MB (16.6%)
• zod: 9.76 MB (8.4%)
• ai: 7.77 MB (6.7%)

Installed footprint breakdown

Installed package size
🔴 eve                              [########################] 30.72 MB 26.6%
🔴 @typescript/typescript-linux-x64 [######################..] 27.95 MB 24.2%
🔴 @rolldown/binding-linux-x64-gnu  [###############.........] 19.15 MB 16.6%
🔴 zod                              [########................] 9.76 MB 8.4%
🔴 ai                               [######..................] 7.77 MB 6.7%
🟠 undici                           [###.....................] 3.51 MB 3.0%
🔴 Other installed packages         [#############...........] 16.72 MB 14.5%

dependencies (4)

Package	Range	Installed size	Share
@vercel/connect	2.2.0	194.7 kB	0.2%
ai	^7.0.105	7.77 MB	6.7%
eve	file:eve-0.66.3.tgz	30.72 MB	26.6%
zod	4.5.4	9.76 MB	8.4%

devDependencies (2)

Package	Range	Installed size	Share
@types/node	24.x	2.54 MB	2.2%
typescript	7.0.2	2.50 MB	2.2%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 9.59 MB 50.0%
🔴 functions/__server.func                         [########################] 9.59 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 9.59 MB

Metric	Value
Public routes	/.well-known/workflow/v1/flow
Runtime	nodejs24.x
Handler	index.mjs
Payload	9.59 MB
Function files	9.59 MB across 110 files
Traced dependencies	0 B
Signal	🟠 Bundled file index.mjs is 2.95 MB (30.8%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###################.....] 2.95 MB 30.8%
🟡 _libs/undici.mjs                 [######..................] 980.8 kB 10.2%
🟡 _chunks/sandbox.mjs              [#####...................] 811.4 kB 8.5%
🟡 _chunks/signal-exit-DZKTacNU.mjs [####....................] 616.2 kB 6.4%
🟡 _chunks/zod.mjs                  [###.....................] 455.4 kB 4.7%
🟠 Other bundled files              [########################] 3.77 MB 39.4%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

---

🟠 functions/__server.func • 17 public routes, 1 internal alias • 9.59 MB

Metric	Value
Public routes	/ /.well-known/workflow/v1/webhook/[token] /eve/v1/activity/[token] /eve/v1/callback/[token] /eve/v1/connections/[name]/callback/[attemptId]/[token] /eve/v1/connections/[name]/callback/[token] /eve/v1/health /eve/v1/info /eve/v1/session /eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream /eve/v1/session/[sessionId] /eve/v1/session/[sessionId]/cancel /eve/v1/session/[sessionId]/clear /eve/v1/session/[sessionId]/compact /eve/v1/session/[sessionId]/reset /eve/v1/session/[sessionId]/stream /eve/v1/task-input/[token]
Internal aliases	/__server
Runtime	nodejs24.x
Handler	index.mjs
Payload	9.59 MB
Function files	9.59 MB across 110 files
Traced dependencies	0 B
Signal	🟠 Bundled file index.mjs is 2.95 MB (30.8%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###################.....] 2.95 MB 30.8%
🟡 _libs/undici.mjs                 [######..................] 980.8 kB 10.2%
🟡 _chunks/sandbox.mjs              [#####...................] 811.4 kB 8.5%
🟡 _chunks/signal-exit-DZKTacNU.mjs [####....................] 616.2 kB 6.4%
🟡 _chunks/zod.mjs                  [###.....................] 455.4 kB 4.7%
🟠 Other bundled files              [########################] 3.77 MB 39.4%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

• Build pipeline: 2.98 s -> 2.60 s (-385.5 ms) vs main (32cd5b9).
• Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (32cd5b9)`

Phase	Baseline	Current	Delta
extension.check	10.8 ms	10.7 ms	-0.1 ms
project.resolve	0.3 ms	3.1 ms	+2.8 ms
workspace.create	0.4 ms	1.9 ms	+1.5 ms
host.prepare	818.0 ms	528.1 ms	-289.9 ms
vercel.service-prefix.resolve	1.0 ms	0.9 ms	-0.1 ms
nitro.create	191.6 ms	190.0 ms	-1.6 ms
sandbox.prewarm	265.4 ms	219.1 ms	-46.3 ms
nitro.cache.prepare	0.2 ms	0.2 ms	0.0 ms
nitro.prepare	0.4 ms	0.5 ms	+0.1 ms
nitro.public-assets	0.6 ms	0.5 ms	-0.1 ms
nitro.prerender	0.4 ms	0.4 ms	0.0 ms
nitro.bundle	1.54 s	1.53 s	-12.4 ms
nitro.cache.write	0.2 ms	0.3 ms	+0.1 ms
vercel.workflow-function.materialize	84.1 ms	51.7 ms	-32.4 ms
agent-summary.emit	0.5 ms	0.5 ms	0.0 ms
nitro.close	0.1 ms	0.1 ms	0.0 ms
output.publish	2.5 ms	4.9 ms	+2.4 ms
workspace.remove	1.5 ms	1.6 ms	+0.1 ms

[roboeve]

taskEnv() only removes EVE_BENCH_TASK_ENV_RESTORE. Everything else from the eve server's environment stays, and runner.mjs spawns the server with ...process.env, which includes the forwarded credentials (AI_GATEWAY_API_KEY, VERCEL_OIDC_TOKEN, OPENAI_API_KEY, ANTHROPIC_API_KEY, per harnesses/eve/index.ts:36). So any bash tool command sees the key. An agent that runs env while debugging, which models do all the time, writes the key into action.result in events.ndjson, and the eve-smoke job uploads that directory as an artifact. It also gives the eve harness something the task itself wouldn't have. Can we strip the forwarded credential names here, or better, capture the container's original env before the runner overlays anything and build the tool env from that?

[roboeve]

detached: true puts every tool command in its own process group. When the agent times out, Container.exec kills only the docker exec group (docker.ts:96), so a tool command that's still running (up to MAX_TIMEOUT_SECONDS = 1800) or anything it put in the background keeps running while tests/test.sh executes. That lets post-timeout work change the verified state, which breaks the "verifier reflects what the agent left behind" contract. The runner should kill the tool process groups before the verifier starts. One option: track tool pgids in the server and kill them on SIGTERM. Another: have exec kill every process except PID 1 once the agent step ends.

[roboeve]

If you abort during the verifier, step() records it as status: "failed". These downloads don't take signal, so if the verifier had already written reward.txt, classifyTrial returns a normal scored trial and trial.json gets written. Resume then trusts that record. An interrupted attempt shouldn't be scored. Pass signal here and treat signal.aborted as not recorded, so resume reruns the attempt, or at least as invalid.

[roboeve]

The image tag hashes only the Dockerfile bytes, but docker build gets the whole environment/ directory as its context. buildImage returns early when the tag exists, so if a task edits a file it COPYs without touching the Dockerfile, it silently runs the stale image. That's common in Terminal-Bench tasks and for local --task-dir work. Hash the full build context into the tag, or always run docker build and let the layer cache do its job.

[roboeve]

ensureNode downloads and runs a Node binary without verifying it. cli/prepare.ts installNode already fetches SHASUMS256.txt, checks the archive, and caps the download size. That leaves two Node provisioners for the same NODE_VERSION, and the one eve uses (which runs with model credentials) is the unverified one. It also races: it uses fixed archive and .extract paths in a shared cache dir. Please collapse these into one verified, atomic helper that both harness families use.

[roboeve]

Any --eve value other than local goes straight into dependencies.eve, so --eve latest or --eve ^0.35.0 are accepted, and the bundle key and provenance record only that string. Two fresh caches can then benchmark different eve code under the same identity, which contradicts the README's "exact published version". Validate an exact semver here, and ideally record the resolved version and integrity in provenance.

[roboeve]

SOURCES leaves out usage.ts, but readCliUsage is what decides harness validity for pi/opencode/codex. A change to usage parsing doesn't change sourceSha256, and so doesn't change the job identity. A resumed job would then mix old and new validity rules without noticing. Add usage.ts. It might be better to hash the whole cli/ directory, so the next new file can't get missed.

[roboeve]

stderr goes through the same emit into the same <cli>.jsonl that readCliUsage parses. When a stdout JSON event arrives in two chunks and a stderr chunk lands in between, the event line is corrupted and usage can come back undefined, so a run that did reach the model gets marked invalid: harness. Keep the event log stdout-only and write stderr to a separate diagnostics log.

[roboeve]

--attempts and --concurrency go through bare Number(). --attempts 0 or --attempts nope builds an empty queue, and a non-numeric --concurrency becomes Math.max(1, NaN) = NaN, so runJob starts zero workers (job.ts:107). Either way the job writes a successful result with 0 trials, and --fail-on-invalid passes. Reject anything that isn't a positive integer, and ideally assert the same thing in runJob for tool callers.

[roboeve]

This calls the schema above "the API", but it no longer matches src/core/result.ts: it's missing invalid, the nullable costUsd, and the scored/invalid summary counts. The tool names at line 217 are also wrong (run_trial/get_result/diff vs the shipped run_bench/get_bench_result/diff_bench/list_bench_tasks), and "Out of scope" (line 237) still lists src/tools/ and the CI workflow, both of which this PR ships. Please update it to what shipped, or mark it as superseded by the README.

[vercel]

CLI run coerces --attempts/--concurrency with Number() and no validation, so bad input (0, NaN) silently produces a completed job with 0 trials

[vercel]

Suggested change

	const target = join(input.dir, "trial.json");
	const temporary = join(input.dir, `trial.${process.pid}.tmp`);
	try {
	await writeFile(temporary, `${JSON.stringify(result, null, 2)}\n`, { flag: "wx" });
	await rename(temporary, target);
	} finally {
	await rm(temporary, { force: true });
	// A user-initiated abort tears down the container mid-trial, so this result is
	// an artifact of the interruption rather than a real outcome. Persisting it would
	// make resume treat the trial as a completed (invalid) attempt and never re-run it,
	// so skip the write and let resume pick the trial up cleanly.
	if (!signal.aborted) {
	const target = join(input.dir, "trial.json");
	const temporary = join(input.dir, `trial.${process.pid}.tmp`);
	try {
	await writeFile(temporary, `${JSON.stringify(result, null, 2)}\n`, { flag: "wx" });
	await rename(temporary, target);
	} finally {
	await rm(temporary, { force: true });
	}

A user-initiated abort mid-trial unconditionally writes an invalid trial.json, so on resume the interrupted trial is treated as a completed (invalid) attempt and never re-run.

[vercel]

ensureNode in the eve harness downloads the Node runtime with no checksum verification and reads the response via an unbounded arrayBuffer(), unlike the CLI harness's installNode.

[vercel]

Image tag hashes only the Dockerfile bytes while the whole environment dir is the build context, so changes to COPY'd context files reuse a stale cached image.

[vercel]

The eve harness's model-controlled bash tool inherits the eve server's process env, which still contains the forwarded provider credentials (AI_GATEWAY_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, VERCEL_OIDC_TOKEN), letting a task/model read and exfiltrate them into persisted agent logs.

[vercel]

Suggested change

	const SOURCES = ["plan.ts", "runner.ts", "output.ts", "prepare.ts", "index.ts"];
	const SOURCES = ["plan.ts", "runner.ts", "output.ts", "usage.ts", "prepare.ts", "index.ts"];

SOURCES in the CLI harness omits usage.ts, so editing the behavior-affecting usage parser does not change sourceSha256, leaving the bundle cache key and job identity fingerprint stale.