Skip to content

fix(evals): bound complete agent run IDs to the server limit - #503

Merged
iuyo5678 merged 1 commit into
Tencent:mainfrom
u7-u7:fix/eval-run-id-length
Oct 28, 2026
Merged

iuyo5678 merged 1 commit into
Tencent:mainfrom
u7-u7:fix/eval-run-id-length

Conversation

@u7-u7

@u7-u7 u7-u7 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Problem

runCommandTask caps adapter, variant and case names individually at 48 characters, then concatenates them with the iteration and UUID. The event server accepts only 120 characters for an entire run ID. Descriptive names can produce an ID of 159 characters; POST /api/events returns 400 and the evaluation fails despite an otherwise successful agent.

Change

Bound the combined human-readable prefix to the remaining 120-character budget, reserving the complete iteration and random suffix. Keep full names in report metadata. Add an end-to-end regression using real Node command adapters and the local evaluation server, checking HTTP acceptance, oracle verification, and distinct event scopes for repeated names.

Verification

  • Regression failed on unchanged base: adapter exit code 1 due to rejected event POST (expected 0).
  • node evals/browser/cli.mjs validate: 10 cases / 19 fixture routes validated.
  • node --test evals/browser/tests/*.test.mjs: 26 passed, 0 failed.
  • Biome 2.4.15 check of both changed files: 2 files checked, no fixes.
  • git diff --check: clean.
  • Rust and extension suites were not run; no runtime extension/Rust code changed.

AI assistance

Prepared and locally tested by OpenAI Codex. No human-review attestation is made. Submitted as a draft without requesting reviewers.

Revalidated the exact patch against current upstream main 6469c47: applies cleanly, 26/26 eval tests pass, 10 cases / 19 routes validate. The branch retains its original base 379b83c.

@u7-u7
u7-u7 marked this pull request as ready for review October 28, 2026 03:07
@iuyo5678
iuyo5678 merged commit 1186399 into Tencent:main Oct 28, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants