feat: track model benchmark results across models and eve versions by prha · Pull Request #4470 · vercel/eve

prha · 2026-10-07T20:17:21Z

Summary

Currently, we cannot see how eval correctness, latency, and cost change across models and eve versions. CI keeps eval artifacts only on failure, for 7 days. We only run our evals on a couple models, to find regressions on PRs.

This PR adds an eval-benchmark workflow. It runs fixture evals on registry models and on mock-model worlds, nightly and on manual dispatch, on main only. It derives a versioned metric set from the captured event streams and publishes immutable records to private Blob for an internal dashboard.

The only change to published eve is that eval artifacts now record each eval's start and completion time.

Validation

pnpm test:eval-benchmark (33 tests) and node --test .github/scripts/*.test.mjs pass.
pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/evals/runner/artifacts.test.ts passes.
pnpm fmt, pnpm lint, pnpm guard:invariants, pnpm docs:check, pnpm check:deps, and pnpm --filter eve typecheck are clean.
Local mock-model eve eval runs of agent-tools and agent-subagents produced full metric sets. Trimmed copies of those captures are the golden test fixtures.
plan.mjs resolved every registry model against the live AI Gateway catalog.
A temporary, since-removed pull request trigger ran the workflow end to end and published to Blob: one model on agent-tools (runs 37681411702 and 37682053298), then all 10 registry models × 3 fixtures × 1 attempt (run 37705854499, 21.5 minutes, $8.49, 0 gaps).

Checklist

This change was requested or approved by a maintainer
I ran the relevant checks from CONTRIBUTING.md
I added tests and documentation where relevant
I added a changeset if this touches the published eve package
DCO sign-off passes for every commit (git commit --signoff)

vercel · 2026-10-07T20:17:25Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
eve-docs	Ready	Preview, v0	Oct 8, 2026 6:31am UTC
eve-pkg	Ready	Preview, v0	Oct 8, 2026 6:31am UTC

socket-security · 2026-10-07T20:21:55Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github/actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c

View full report

vercel · 2026-10-07T20:31:07Z

+    );
+    const missingReason = dir === undefined ? "job-missing" : "eval-missing";
+    const expectedIds = expected ?? fixture.eval_ids;
+    for (const evalId of new Set([...expectedIds, ...results.keys()]))


buildRecords falls back to the full fixture.eval_ids (which includes real-model-tagged evals) for mock legs that lack an expected.json, fabricating spurious gap records for evals the mock track never runs.

github-actions · 2026-10-07T23:32:05Z

Bundle + Package Summary: `apps/fixtures/weather-agent`

Key takeaways

No notable deltas vs main (89704cb).

Delta vs `main (89704cb)`

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	7.49 MB	7.49 MB	+291 B ⚠️
Package	Unpacked publish size	26.39 MB	26.39 MB	+172 B ⚠️
Package	Installed footprint	74.06 MB	74.06 MB	+172 B ⚠️
Package	Published files	4159	4159	0
Package	Installed files	8160	8160	0
Package	Installed package instances	34	34	0
Package	Distinct installed package names	32	32	0
Package	Installed dependency edges	51	51	0
Package	Installed optional peer edges	9	9	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	19.92 MB	19.92 MB	-120 B ✅
Runtime	Public routes	17	17	0

Changed function payloads vs main (89704cb) (2)

Function	Status	Baseline	Current	Delta	Route changes
`functions/__server.func`	changed	9.96 MB	9.96 MB	-60 B ✅	none
`functions/.well-known/workflow/v1/flow.func`	changed	9.96 MB	9.96 MB	-60 B ✅	none

`eve init` install

Metric	Baseline	Current	Delta
Installed footprint	112.35 MB	112.35 MB	+172 B ⚠️
Installed packages	103	103	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	44.04 MB	44.04 MB	+172 B ⚠️
devDependency package bytes	5.11 MB	5.11 MB	0 B ➖

Build Metadata

Preset: vercel
Nitro: nitro@3.0.260903-beta
Output directory: apps/fixtures/weather-agent/.vercel/output
Build metadata timestamp: 2026-10-08T06:32:00.620Z
Route aliases: 17 public, 1 internal (18 total aliases)
Vercel routes in config: 20
Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

Package: eve@0.74.0
Package directory: packages/eve
Tarball: 7.49 MB (eve-0.74.0.tgz)
Unpacked payload: 26.39 MB across 4159 published files
Installed footprint: 74.06 MB across 8160 installed files
Installed root package: 26.09 MB
Installed dependencies: 47.97 MB
Installed package instances: 34
Distinct installed package names: 32
Installed dependency edges: 51
Installed optional peer edges: 9
Runtime dependencies: 2
Peer dependencies: 7 (6 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

eve: 26.09 MB (35.2%)
@rolldown/binding-linux-x64-gnu: 19.62 MB (26.5%)
ai: 8.01 MB (10.8%)
zod: 6.14 MB (8.3%)
undici: 3.56 MB (4.8%)

Publish payload breakdown

Published file size
🟠 dist/src/compiled/shadcn-registry/index.js       [#####...................] 3.85 MB 14.6%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.29 MB 8.7%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js     [#.......................] 1.03 MB 3.9%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 604.9 kB 2.3%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 521.9 kB 2.0%
🔴 Other published files                            [########################] 18.10 MB 68.6%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 26.09 MB 35.2%
🔴 @rolldown/binding-linux-x64-gnu [##################......] 19.62 MB 26.5%
🔴 ai                              [#######.................] 8.01 MB 10.8%
🔴 zod                             [######..................] 6.14 MB 8.3%
🟠 undici                          [###.....................] 3.56 MB 4.8%
🟠 nitro                           [##......................] 1.89 MB 2.6%
🔴 Other installed packages        [########................] 8.75 MB 11.8%

Runtime dependencies (2)

Package	Range	Notes
`nitro`	`3.0.260903-beta`
`undici`	`8.10.2`

Peer dependencies (7)

Package	Range	Notes
`@opentelemetry/api`	`^1.0.0`	optional peer
`ai`	`catalog:`
`braintrust`	`^3.0.0`	optional peer
`chat`	`^4.41.0`	optional peer
`dd-trace`	`^6.13.0`	optional peer
`just-bash`	`^3.1.0`	optional peer
`microsandbox`	`^0.5.0`	optional peer

eve init install drill-down

`eve init` install details

Command: eve init my-agent
Package manager: npm
Installed footprint: 112.35 MB across 10055 installed files
Installed packages: 103 total (97 transitive-only)
dependencies: 4 direct packages totaling 44.04 MB
devDependencies: 2 direct packages totaling 5.11 MB
Other transitive package files: 63.20 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

@typescript/typescript-linux-x64: 27.95 MB (24.9%)
eve: 26.09 MB (23.2%)
@rolldown/binding-linux-x64-gnu: 19.62 MB (17.5%)
zod: 9.76 MB (8.7%)
ai: 8.01 MB (7.1%)

Installed footprint breakdown

Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 24.9%
🔴 eve                              [######################..] 26.09 MB 23.2%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.62 MB 17.5%
🔴 zod                              [########................] 9.76 MB 8.7%
🔴 ai                               [#######.................] 8.01 MB 7.1%
🟠 undici                           [###.....................] 3.56 MB 3.2%
🔴 Other installed packages         [###############.........] 17.37 MB 15.5%

dependencies (4)

Package	Range	Installed size	Share
`@vercel/connect`	`2.2.0`	188.7 kB	0.2%
`ai`	`^7.0.128`	8.01 MB	7.1%
`eve`	`file:eve-0.74.0.tgz`	26.09 MB	23.2%
`zod`	`4.5.4`	9.76 MB	8.7%

devDependencies (2)

Package	Range	Installed size	Share
`@types/node`	`24.x`	2.61 MB	2.3%
`typescript`	`7.0.2`	2.50 MB	2.2%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 9.96 MB 50.0%
🔴 functions/__server.func                         [########################] 9.96 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 9.96 MB

Metric	Value
Public routes	`/.well-known/workflow/v1/flow`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	9.96 MB
Function files	9.96 MB across 127 files
Traced dependencies	0 B
Signal	🟠 Bundled file `_chunks/vercel.web.mjs` is 2.03 MB (20.4%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/vercel.web.mjs                         [###########.............] 2.03 MB 20.4%
🟠 _libs/undici.mjs                               [#####...................] 1.01 MB 10.1%
🟡 _chunks/sandbox3.mjs                           [####....................] 815.4 kB 8.2%
🟡 _chunks/compiled-artifacts-instrumentation.mjs [####....................] 762.7 kB 7.7%
🟡 _chunks/token-util-CddTZ_ef.mjs                [####....................] 704.7 kB 7.1%
🔴 Other bundled files                            [########################] 4.63 MB 46.5%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 16 public routes, 1 internal alias • 9.96 MB

Metric	Value
Public routes	`/` `/.well-known/workflow/v1/webhook/[token]` `/eve/v1/callback/[token]` `/eve/v1/connections/[name]/callback/[attemptId]/[token]` `/eve/v1/connections/[name]/callback/[token]` `/eve/v1/health` `/eve/v1/info` `/eve/v1/session` `/eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream` `/eve/v1/session/[sessionId]` `/eve/v1/session/[sessionId]/cancel` `/eve/v1/session/[sessionId]/clear` `/eve/v1/session/[sessionId]/compact` `/eve/v1/session/[sessionId]/reset` `/eve/v1/session/[sessionId]/stream` `/eve/v1/task-input/[token]`
Internal aliases	`/__server`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	9.96 MB
Function files	9.96 MB across 127 files
Traced dependencies	0 B
Signal	🟠 Bundled file `_chunks/vercel.web.mjs` is 2.03 MB (20.4%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/vercel.web.mjs                         [###########.............] 2.03 MB 20.4%
🟠 _libs/undici.mjs                               [#####...................] 1.01 MB 10.1%
🟡 _chunks/sandbox3.mjs                           [####....................] 815.4 kB 8.2%
🟡 _chunks/compiled-artifacts-instrumentation.mjs [####....................] 762.7 kB 7.7%
🟡 _chunks/token-util-CddTZ_ef.mjs                [####....................] 704.7 kB 7.1%
🔴 Other bundled files                            [########################] 4.63 MB 46.5%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: `e2e/fixtures/agent-tools-sandbox`

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

Build pipeline: 7.20 s -> 3.47 s (-3.74 s) vs main (89704cb).
Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (89704cb)`

Phase	Baseline	Current	Delta
`extension.check`	0.5 ms	0.5 ms	0.0 ms
`project.resolve`	0.3 ms	0.3 ms	0.0 ms
`workspace.create`	0.5 ms	0.5 ms	0.0 ms
`host.prepare`	4.52 s	820.1 ms	-3.70 s
`vercel.service-prefix.resolve`	1.4 ms	1.5 ms	+0.1 ms
`nitro.create`	562.3 ms	562.7 ms	+0.4 ms
`sandbox.prewarm`	305.6 ms	360.2 ms	+54.6 ms
`nitro.cache.prepare`	0.3 ms	0.3 ms	0.0 ms
`nitro.prepare`	0.7 ms	0.7 ms	0.0 ms
`nitro.public-assets`	0.8 ms	0.8 ms	0.0 ms
`nitro.prerender`	0.5 ms	0.5 ms	0.0 ms
`nitro.bundle`	1.66 s	1.57 s	-91.5 ms
`nitro.cache.write`	0.2 ms	0.3 ms	+0.1 ms
`vercel.workflow-function.materialize`	70.6 ms	70.7 ms	+0.1 ms
`agent-summary.emit`	0.7 ms	0.7 ms	0.0 ms
`connect-manifest.emit`	0.3 ms	0.3 ms	0.0 ms
`nitro.close`	0.1 ms	0.1 ms	0.0 ms
`output.publish`	3.8 ms	3.7 ms	-0.1 ms
`workspace.remove`	2.6 ms	2.5 ms	-0.1 ms

Adds the eval-benchmark workflow: event-derived measurement primitives, eval artifact timing, the standard per-attempt metric set and gap classification, the schema_version 1 record contract, registry planning against the AI Gateway catalog, nightly and manual runs on main, and immutable publication to private Blob. Signed-off-by: prha <prha@vercel.com>

prha added the run-eval-benchmark Run the model benchmark on this PR (temporary, eval-benchmark.yml) label Oct 7, 2026

vercel Bot deployed to Preview – eve-pkg October 7, 2026 20:19

View deployment

vercel Bot deployed to Preview – eve-docs October 7, 2026 20:21

View deployment

prha force-pushed the prha/eval-bench branch from 62c62e5 to a2c2af2 Compare October 7, 2026 20:21

prha added run-eval-benchmark Run the model benchmark on this PR (temporary, eval-benchmark.yml) and removed run-eval-benchmark Run the model benchmark on this PR (temporary, eval-benchmark.yml) labels Oct 7, 2026

vercel Bot had a problem deploying to Preview – eve-docs October 7, 2026 20:22 Failure

vercel Bot deployed to Preview – eve-pkg October 7, 2026 20:24

View deployment

prha force-pushed the prha/eval-bench branch from a2c2af2 to a9a3916 Compare October 7, 2026 20:24

vercel Bot had a problem deploying to Preview – eve-docs October 7, 2026 20:25 Failure

prha added run-eval-benchmark Run the model benchmark on this PR (temporary, eval-benchmark.yml) and removed run-eval-benchmark Run the model benchmark on this PR (temporary, eval-benchmark.yml) labels Oct 7, 2026

vercel Bot deployed to Preview – eve-pkg October 7, 2026 20:27

View deployment

vercel Bot reviewed Oct 7, 2026

View reviewed changes

prha force-pushed the prha/eval-bench branch from a9a3916 to 8b18ec7 Compare October 7, 2026 23:28

This was referenced Oct 7, 2026

chore: full-matrix benchmark run (do not merge) #4485

Closed

chore: full-matrix benchmark run 2 (do not merge) #4486

Closed

vercel Bot deployed to Preview – eve-docs October 7, 2026 23:30

View deployment

vercel Bot deployed to Preview – eve-pkg October 7, 2026 23:30

View deployment

prha force-pushed the prha/eval-bench branch from 8b18ec7 to a25e81e Compare October 7, 2026 23:46

vercel Bot deployed to Preview – eve-pkg October 7, 2026 23:48

View deployment

vercel Bot deployed to Preview – eve-docs October 7, 2026 23:48

View deployment

prha force-pushed the prha/eval-bench branch from a25e81e to 8d60557 Compare October 7, 2026 23:54

vercel Bot deployed to Preview – eve-pkg October 7, 2026 23:56

View deployment

vercel Bot deployed to Preview – eve-docs October 7, 2026 23:56

View deployment

prha force-pushed the prha/eval-bench branch from 8d60557 to 5673210 Compare October 7, 2026 23:59

vercel Bot deployed to Preview – eve-docs October 8, 2026 00:01

View deployment

vercel Bot deployed to Preview – eve-pkg October 8, 2026 00:01

View deployment

prha force-pushed the prha/eval-bench branch from 5673210 to c22e6c6 Compare October 8, 2026 05:21

vercel Bot deployed to Preview – eve-pkg October 8, 2026 05:23

View deployment

vercel Bot deployed to Preview – eve-docs October 8, 2026 05:23

View deployment

prha force-pushed the prha/eval-bench branch from c22e6c6 to c97c374 Compare October 8, 2026 05:47

vercel Bot deployed to Preview – eve-pkg October 8, 2026 05:49

View deployment

vercel Bot deployed to Preview – eve-docs October 8, 2026 05:50

View deployment

prha force-pushed the prha/eval-bench branch from c97c374 to 6b25ea4 Compare October 8, 2026 06:18

vercel Bot deployed to Preview – eve-docs October 8, 2026 06:19

View deployment

vercel Bot deployed to Preview – eve-pkg October 8, 2026 06:20

View deployment

prha force-pushed the prha/eval-bench branch from 6b25ea4 to 33c1491 Compare October 8, 2026 06:29

vercel Bot deployed to Preview – eve-docs October 8, 2026 06:31

View deployment

vercel Bot deployed to Preview – eve-pkg October 8, 2026 06:31

View deployment

Navigation Menu

Repository navigation

feat: track model benchmark results across models and eve versions - #4470

feat: track model benchmark results across models and eve versions#4470
prha wants to merge 1 commit into
mainfrom
prha/eval-bench

prha commented Oct 7, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

vercel Bot Oct 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Package Details

`eve init` install details

Payload Size Graph

Top Function Payloads

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

prha commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Checklist

Uh oh!

vercel Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle + Package Summary: apps/fixtures/weather-agent

Delta vs main (89704cb)

eve init install

Package Details

eve init install details

Payload Size Graph

Top Function Payloads

Build Timing: e2e/fixtures/agent-tools-sandbox

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

prha commented Oct 7, 2026 •

edited

Loading

vercel Bot commented Oct 7, 2026 •

edited

Loading

socket-security Bot commented Oct 7, 2026 •

edited

Loading

vercel Bot Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Bundle + Package Summary: `apps/fixtures/weather-agent`

Delta vs `main (89704cb)`

`eve init` install

`eve init` install details

Build Timing: `e2e/fixtures/agent-tools-sandbox`