Run shared eval scenarios against every agent, checking each step along the way #65

Component

Something else: an eval harness for every agent

Experience

• Owner: one set of scenarios runs against every agent's command-line tool, and its results are scored against Claude Code's. It checks the final answer and also what happened along the way:
  • which skill loaded;
  • whether a question came before any write;
  • whether a sub-agent started;
  • whether the rules were in context.
• Operator: an agent the plugin claims to support has passed the scenarios Claude Code passes, or its gaps are listed.

Proposed behaviour

• Shared scenarios for skills, sub-agents and rules. They start from the spike's three probe prompts: what the session loaded, a skill called with arguments, and a hand-off followed by a sub-agent. This issue also settles which of the existing cases under evals/ join them.
• One harness that drives each agent headless and reads its event stream. Each of these worked in the spike's runs in Spike: run the plugin in Codex, Cursor and Kiro as well as Claude Code #61:
  • Codex: codex exec --json, with its own CODEX_HOME and standard input closed, or it waits forever.
  • Cursor: agent -p --output-format stream-json --plugin-dir.
  • Kiro: kiro-cli chat --agent-engine v3 --no-interactive.
• Mid-turn checks on those events, as well as on the final result. claude plugin eval judges Claude Code only, mostly by its final answer.
• Weigh the author's Gherkin harness (@freethinkingit/gherkin-harness: Cucumber Features, ready-made steps, step records) against extending this repository's own Python tooling. Gherkin brings Node and TypeScript into a stdlib-Python repository, and it would need new steps that drive an agent and read its events.

Evals stay a token cost: run by hand, one target at a time, never in CI.

Motivating case

Nothing today shows whether the plugin works anywhere but Claude Code, or whether a skill took the right steps on the way to its answer.

Done when

• Every target in Add a target per agent: Cursor and Codex, then Kiro and Devin #64 has scenarios that pass under the parity bar in the strategy. The bar: every shared scenario for skills, sub-agents and rules that passes in Claude Code passes in that agent, through its own support. A failure caused by a recorded gap outside those three parts does not count.
• The harness choice is recorded in this stream's design.

Depends on

Runs alongside #64: a target's scenarios are written before or with it. This is a stream in .freethinking/strategies/multi-tool-plugin.md. Design with /freethinking:hld .freethinking/strategies/multi-tool-plugin.md cross-agent-evals.