Issues

• Frontier benchmarks to add (and existing ones to fix): Terminal-Bench 4.0, TB-Science, HLE-Diamond, Agents' Last Exam, SWE-bench Pro V2, OSWorld-V2 — Nov 2026

  enhancement
  Exgentic/eval-containers#733 · Filter by author alonh opened on Nov 4, 2026
  

• bug: ci — disabled build-sweep.yml is an empty workflow, so every push records a failed run

  bug
  Exgentic/eval-containers#732 · Filter by author agentshift-app opened on Nov 4, 2026

• drift: .agents/edge/RULES.md#7 — edge's final record can be truncated with no response body, breaking replay fixtures

  bug
  Exgentic/eval-containers#716 · Filter by author yoavkatz opened on Oct 19, 2026
  1

• drift: .agents/docs/RULES.md#13: document the existing TASK MCP setup

  Exgentic/eval-containers#700 · Filter by author georgeatparallel opened on Oct 17, 2026

• bug: agents/claude-code — a backgrounded command lets the agent idle until EVAL_TIMEOUT

  bug
  Exgentic/eval-containers#647 · Filter by author elronbandel opened on Oct 8, 2026

• cybergym and handbook get the 300s chart timeout on k8s, not their compose default

  Exgentic/eval-containers#641 · Filter by author alonh opened on Oct 8, 2026

• Observability audit: why a run tells you so little, and what each cause is

  Exgentic/eval-containers#620 · Filter by author elronbandel opened on Sep 23, 2026
  4

• RFC: delivery/RULES.md — no rule governs buildcache/* retention, so ~1,876 packages accumulate unboundedly

  Exgentic/eval-containers#597 · Filter by author yoavkatz opened on Sep 15, 2026

• RFC: delivery/RULES.md#22 — the retention clock is unobservable, so rule 22 cannot be enforced or verified

  Exgentic/eval-containers#596 · Filter by author yoavkatz opened on Sep 15, 2026

• drift: delivery/RULES.md#21 — published eval-<benchmark> artifacts default to :latest, not a hash tag

  Exgentic/eval-containers#595 · Filter by author yoavkatz opened on Sep 15, 2026

• drift: .agents/delivery/RULES.md#20 — a channel run never re-stitches a per-task combo whose per-arch tags outran its :latest

  Exgentic/eval-containers#591 · Filter by author elronbandel opened on Sep 15, 2026

• rfc: .agents/gateways/RULES.md#10-11, compose#8, models#16 — the gateway and the collector become opt-in

  Exgentic/eval-containers#586 · Filter by author elronbandel opened on Sep 15, 2026

• rfc: .agents/gateways/RULES.md#10-11 — retire the otelcol sidecar; the edge's record is the only account of a call

  Exgentic/eval-containers#580 · Filter by author elronbandel opened on Sep 15, 2026

• bug: registry — no retention sweep; ~9,900 packages accumulate superseded untagged versions (58.7% of a sample)

  bug
  Exgentic/eval-containers#578 · Filter by author yoavkatz opened on Sep 15, 2026
  1

• bug: cli/build — per-task benchmarks bypass bake, so --builder (incl. oc) can't build six benchmarks

  bug
  Exgentic/eval-containers#563 · Filter by author elronbandel opened on Sep 14, 2026

• docs: .agents/delivery/build/SKILL.md's combination bake example omits EDGE_IMAGE/PROCESS_COMPOSE_IMAGE and misplaces MODEL_IMAGE

  Exgentic/eval-containers#562 · Filter by author yoavkatz opened on Sep 13, 2026

• drift: delivery/RULES.md#14 — eval.platforms is per-benchmark, but swe-bench's arm64 gap is per-task (80 of 500)

  Exgentic/eval-containers#557 · Filter by author yoavkatz opened on Sep 12, 2026

• bug(chart): eval.runnerImage drops toString, so an all-digit task id renders %!s(float64=N) in the image ref

  Exgentic/eval-containers#555 · Filter by author yoavkatz opened on Sep 10, 2026

• bug(tau-bench): the harness has neither the output volume nor write-result, so a k8s run writes no result.json at all

  Exgentic/eval-containers#551 · Filter by author elronbandel opened on Sep 10, 2026

• bug: agents smoke — every agent fails "the edge wrote no record to /output/model/calls.jsonl" (nightly red since 2026-09-06)

  bug
  Exgentic/eval-containers#517 · Filter by author elronbandel opened on Sep 9, 2026

• rfc: .agents/edge/RULES.md#10a — the edge should keep a running usage total beside the record

  Exgentic/eval-containers#500 · Filter by author elronbandel opened on Sep 8, 2026
  1

• fix(runner): useradd fallback in core/runner/run creates gid 1002, not gid 0

  Exgentic/eval-containers#488 · Filter by author yoavkatz opened on Sep 7, 2026

• drift: tests/run/replay/RULES.md#6,7 — provenance.json and broken.json don't exist

  Exgentic/eval-containers#483 · Filter by author yoavkatz opened on Sep 7, 2026

• bug: ci — a tag release and a main push are in different concurrency groups and both write :latest

  bug
  Exgentic/eval-containers#469 · Filter by author elronbandel opened on Sep 6, 2026

• bug: bifrost's Anthropic→OpenAI translation drops cache_control and reports inconsistent input_tokens, and raw Responses-API input_tokens is cumulative-per-call not net-new

  Exgentic/eval-containers#458 · Filter by author yoavkatz opened on Sep 6, 2026
  1