Repository navigation
Conversation
Terminal-Bench evaluation of the eve harness no longer goes through Harbor and a Python adapter. packages/eve-bench now owns a small Node runner (task loader, docker lifecycle, trial/job scheduler with resume, reporters), an eve harness whose in-container runner speaks plain HTTP to the built server, an oracle harness for reference solutions, pinned dataset locks and cohorts, eve tools over the same core, and a CI workflow. Design in research/eve-bench-runner.md. Signed-off-by: Rui Conti <ruiconti@gmail.com>
…rios Containers now carry eve-bench labels; a job removes leftovers from a crashed previous run of the same name, cleans up by label after an abort, checks the Docker daemon up front, and caps per-trial logs at 64 MiB. The CLI accepts repeatable --task-dir for local Terminal-Bench task directories, which powers a Docker-backed stress suite (pnpm run test:stress) covering timeouts, bad images, SIGINT abort and resume, and concurrency. The eve-smoke CI job can run pre-merge on PRs labeled bench:eve. Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| archive, | ||
| "--strip-components=2", | ||
| "-C", | ||
| extractDir, |
There was a problem hiding this comment.
ensureNode downloads and executes the Node.js tarball with a bare fetch — no checksum verification, no size cap, no timeout, and no redirect control — unlike the sibling installNode in the same package which verifies against SHASUMS256.txt.
![roboeve[bot]](https://avatars.githubusercontent.com/b/10660?s=60&v=4){kind=small}
| run: pnpm install --frozen-lockfile | ||
| working-directory: ${{ github.workspace }} | ||
|
|
||
| - name: Run tests |
There was a problem hiding this comment.
P1: Build eve before the mandatory test step
A clean pnpm install leaves packages/eve/dist absent, but this test script typechecks imports from eve, eve/tools, and eve/sandbox. Those exports all resolve to dist. Reproducing this workflow sequence fails with TS2307, so the oracle benchmark never runs. Add the same pnpm --filter eve run build prerequisite used by eve-smoke, or make the bench tests resolve source intentionally.
| - name: Compare with base job | ||
| if: ${{ inputs.base_job_artifact }} | ||
| run: | | ||
| pnpm bench diff .generated/base-job .generated/jobs/eve-smoke | tee diff.txt |
There was a problem hiding this comment.
P2: Pass job names rather than paths to diff
diff resolves both arguments through jobDir(), whose safe-ID validation rejects /. The exact command here exits with job name must be 1-128 path- and Docker-safe characters. Download the artifact under .generated/jobs/<safe-name> and pass that job name, or add an explicit validated directory-input mode.
|
|
||
| export function resolveTaskPath(path = "."): string { | ||
| const root = taskRoot(); | ||
| const target = resolve(root, path); |
There was a problem hiding this comment.
P1: Resolve links before enforcing the task root
This checks only the lexical path. read_file and write_file follow links afterward, so a link below the task root can escape to /proc/self/environ, /logs, or /installed-agent. In the eve harness, /proc/self/environ contains forwarded provider credentials and the tool result is persisted in agent events. Canonicalize every existing path component and reject symbolic links, as the e0 container backend already does.
| } | ||
| }); | ||
|
|
||
| export default defineTool({ |
There was a problem hiding this comment.
P1: Gate or constrain this credential-bearing tool
With no approval policy, an agent can call this tool without user approval. Its baseUrl accepts any HTTPS host, while the CLI runner injects AI_GATEWAY_API_KEY and OPENAI_API_KEY into clients configured for that host. A prompt-injected caller can therefore redirect a gateway credential to an attacker-controlled endpoint. Require approval for this side effect and restrict credential forwarding to approved origins.
| return task.environment.dockerImage; | ||
| } | ||
| const dockerfile = await readFile(join(task.environment.dockerfileDir, "Dockerfile")); | ||
| const tag = `eve-bench/${task.name}:${createHash("sha256").update(dockerfile).digest("hex").slice(0, 12)}`; |
There was a problem hiding this comment.
P1: Hash the complete Docker build context
This tag covers only Dockerfile, although Docker receives the whole environment/ directory. If a copied script or other context file changes, buildImage() sees the old tag and skips Docker entirely. The job identity changes, but the trial still runs the stale image, silently invalidating the result. Include paths, modes, and contents from the validated build context in the tag.
| if (name === "oracle") return createOracleHarness(); | ||
| if (name === "eve") { | ||
| return createEveHarness( | ||
| eve === "local" ? { kind: "local" } : { kind: "release", version: eve }, |
There was a problem hiding this comment.
P2: Require an immutable eve release selector
This accepts tags, ranges, and other npm specs such as latest, ^0.35.0, and file:../eve. The raw selector also becomes the cache and provenance identity, so the same identity can represent different code across clean hosts or points in time. Require an exact release version, as the CLI harnesses do, or record the resolved package version and integrity before deriving identity.
|
|
||
| /** Fetches the pinned commit with a sparse, blobless checkout of the locked tasks. Idempotent. */ | ||
| export async function syncDataset(root: string, lock: DatasetLock): Promise<string> { | ||
| const dir = datasetDir(root, lock); |
There was a problem hiding this comment.
P2: Publish shared caches atomically
Dataset sync runs before the per-job lock, so different job names can both miss this check and then delete or mutate the same checkout. The fixed ${bundleDir}.building flow in src/harnesses/eve/index.ts has the same race. Use one key-scoped cache publisher that builds in a unique sibling directory and atomically installs the result, or serialize each cache key.
| @@ -0,0 +1,245 @@ | |||
| --- | |||
| issue: TODO | |||
There was a problem hiding this comment.
P2: Link the required prior issue
The repository rules require every research plan to reference an issue, but this approved plan still contains TODO. Link the prior discussion issue before treating the plan as approved.
| } | ||
| ``` | ||
|
|
||
| `eve-bench tasks sync <lock>` clones the pinned commit (shallow, sparse) into |
There was a problem hiding this comment.
P2: Use the supported dataset option
tasks sync does not accept a positional lock. Running this example exits with Unexpected argument 'terminal-bench-2.0'. Update this and the CLI summary below to use tasks sync --dataset terminal-bench-2.0.
|
|
||
| ### Consumers | ||
|
|
||
| - CI: a workflow runs `eve-bench run --cohort smoke --json`, uploads the job |
There was a problem hiding this comment.
P2: Align the approved plan with the implemented CI contract
The workflow does not fetch a base-branch artifact or post a PR diff automatically. It only accepts an optional artifact name during manual dispatch, and the current comparison command fails for the path arguments noted above. Update this contract to describe the implemented opt-in flow, or implement the automatic base comparison described here.
|
Superseded by vercel-labs/eve-bench#7. The benchmark runner belongs in the dedicated eve-bench repository; the replacement PR ports the e0 harness and native-build support onto that repository's existing MCP/compose/artifact-aware runner. |
This branch had an error being deployed
1 participant
Summary
Adds
@eve/bench, an eve-owned, zero-dependency Node runner for Terminal-Bench. It replaces the Harbor/Python adapter with pinned datasets, Docker lifecycle management, resumable jobs, reports, and reusable harnesses for eve, local e0 source, pi, OpenCode, Codex, and oracle controls.@eve/benchprepares a content-addressed harness bundle, runs it in the task container, always runs the verifier, and writes an identity-bound trial record--native-buildrebuilds the reviewed task Dockerfile for every comparison arm and records that environment choiceResume now rejects changed model, harness, source provenance, dataset, task content, or attempt count before touching prior results. Provider credentials are allowlisted per harness, shell subprocesses do not inherit them, paths and identifiers are contained, and logs/output are bounded.
Hermes remains unsupported until its Python environment and provider routing can be pinned. Hosted result publication is also out of scope: raw jobs stay under ignored
.generated/jobs/or CI artifacts.Validation
Direct TypeScript and Node coverage passed with 81 unit/tool tests, 32 integration scenarios plus one optional macOS skip, and all 6 Docker stress cases. A Linux container check exercised the real e0
code__apply_patchtool, and the invariant guard passed with ignored generated caches excluded from its scan.The native-build oracle passed the pinned
fix-gittask. A one-attemptopenai/gpt-5.6-terrapilot on the same native ARM task and high reasoning scored 1/1 for e0, pi 0.85.1, OpenCode 1.18.30, and Codex 0.154.0. Raw run artifacts remain local and are not part of this PR. DeepSeek was not measured because the test team's provider allowlist rejected all available DeepSeek providers.Checklist
CONTRIBUTING.mdevepackagegit commit --signoff)This adds a private benchmark package and does not change the published
evepackage, so no changeset is required.Diff size
Docs — 2 files ·
+416 / -0The package guide documents operation and interpretation; the research plan records the runner contract, lifecycle, and boundaries.
Implementation — 47 files ·
+4505 / -0This includes the runner core, harness implementations, pinned dataset metadata, CLI and agent-tool surfaces, CI workflow, and the minimal eve fixture used as a benchmark subject.
Tests — 80 files ·
+2675 / -0Unit and integration coverage exercises parsing, Docker execution, source identity, safe resume, credential redaction, real e0 tool wiring, CLI adapter contracts, timeouts, abort cleanup, and concurrency. Small Terminal-Bench-shaped fixtures account for much of the file count.