Issues

• fullsend runner: record fullsend, OpenShell, image and harness provenance with each run

  blockedkind/enhancement
  fullsend-ai/agent-eval-harness#262 · Filter by author astefanutti opened on Oct 20, 2026
  3

• fullsend runner: execute the skill under test through fullsend run

  kind/enhancementpr-open
  fullsend-ai/agent-eval-harness#261 · Filter by author astefanutti opened on Oct 20, 2026
  8

• after_each hook failures never reach run_result.json or the judges, so evals re-implement "the case did not run cleanly" by hand

  featurekind/enhancementtriaged
  fullsend-ai/agent-eval-harness#259 · Filter by author jwm4 opened on Oct 18, 2026
  11

• Stage agent-judge inputs by copying, not by loading every declared output file into memory

  ready-to-code
  fullsend-ai/agent-eval-harness#258 · Filter by author jwm4 opened on Oct 16, 2026
  5

• anthropic:/ on an agent role is only normalised under an OpenRouter plan

  fullsend-ai/agent-eval-harness#242 · Filter by author astefanutti opened on Oct 8, 2026

• hook model ignores the CLI model override: hook_model_for is called with config roles only

  fullsend-ai/agent-eval-harness#241 · Filter by author astefanutti opened on Oct 8, 2026

• Want to add your plugin to awesome-ai-plugins?

  fullsend-ai/agent-eval-harness#211 · Filter by author zerocodefast opened on Aug 29, 2026

• execution.steps: no session re-entrance (--continue/resume) or conditional retry loops

  fullsend-ai/agent-eval-harness#210 · Filter by author not-stbenjam opened on Aug 28, 2026

• Proposal: optional EvalPort export for dataset cases + run results (portable interchange format)

  fullsend-ai/agent-eval-harness#201 · Filter by author adhabnr-ux opened on Aug 23, 2026

• Support with skill / without skill A/B testing similar to agentskills.io evals

  fullsend-ai/agent-eval-harness#178 · Filter by author alexdmccabe opened on Aug 7, 2026

• collect.py: positional prefix grouping breaks when batch-mode skills produce more file prefixes than cases

  fullsend-ai/agent-eval-harness#154 · Filter by author rhuss opened on Jul 16, 2026
  1

• Cross-case output reads: agents can access sibling case directories

  fullsend-ai/agent-eval-harness#151 · Filter by author not-stbenjam opened on Jul 9, 2026

• Case directory names leak expected answers to agents

  fullsend-ai/agent-eval-harness#150 · Filter by author not-stbenjam opened on Jul 9, 2026

• feat: add /eval-security skill for deterministic security scanning of harness components

  fullsend-ai/agent-eval-harness#131 · Filter by author Benkapner opened on Jun 21, 2026
  1

• Surfacing Agent Skills security scanner Nvidia SkillSpector

  fullsend-ai/agent-eval-harness#102 · Filter by author tonykay opened on Jun 2, 2026
  1

• trace_builder.py: MLflow cost charts show $0.00 — missing endpointId and no fallback when per_model_usage is absent

  fullsend-ai/agent-eval-harness#97 · Filter by author abhijeet-dhumal opened on Jun 2, 2026

• log_results.py: AsyncExportQueue import breaks on MLflow >=3.13, traces not flushed before assessment logging

  fullsend-ai/agent-eval-harness#96 · Filter by author abhijeet-dhumal opened on Jun 2, 2026

• score.py: conversation key contains raw tool calls instead of final skill output

  fullsend-ai/agent-eval-harness#94 · Filter by author abhijeet-dhumal opened on Jun 2, 2026

• Replace hand-rolled _md_to_html with mistune

  fullsend-ai/agent-eval-harness#93 · Filter by author rhuss opened on Jun 2, 2026
  2

• feat: Add skill activation testing mode (execution.mode: activation)

  fullsend-ai/agent-eval-harness#87 · Filter by author rhuss opened on May 31, 2026
  2

• Check for all the requirements and stop execution if anything is missing before starting the eval harness

  fullsend-ai/agent-eval-harness#69 · Filter by author bdattoma opened on May 21, 2026

• Report per-case details missing stdout output and baseline diff

  fullsend-ai/agent-eval-harness#64 · Filter by author astefanutti opened on May 15, 2026

• Support sandboxed execution environments (containers, VMs)

  fullsend-ai/agent-eval-harness#4 · Filter by author astefanutti opened on Apr 13, 2026

• Support test suites with multiple eval.yaml files

  fullsend-ai/agent-eval-harness#3 · Filter by author astefanutti opened on Apr 13, 2026