From 702c933e0a9be2f239e57be34249544529db4080 Mon Sep 17 00:00:00 2001 From: Dan Sabin Date: Fri, 23 Oct 2026 19:09:29 -0500 Subject: [PATCH] Pivot the multi-tool strategy to a canonical source Every agent's plugin, Claude Code's included, is compiled from one canonical source, with one target folder per agent, in place of converging on the Claude Code layout and compiling only the gaps. Co-Authored-By: Claude Opus 5.5 --- .freethinking/strategies/multi-tool-plugin.md | 44 ++++++++++--------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/.freethinking/strategies/multi-tool-plugin.md b/.freethinking/strategies/multi-tool-plugin.md index f1e7c54..6482c69 100644 --- a/.freethinking/strategies/multi-tool-plugin.md +++ b/.freethinking/strategies/multi-tool-plugin.md @@ -2,7 +2,7 @@ ## Introduction and horizon -The freethinking plugin gives engineers a disciplined way to work: specs approved before code, bugs reproduced before fixes, tests with each change, and team rules in every session. Today it runs only in Claude Code. This strategy takes it to every coding agent its author and the engineers who use it work in, starting with Cursor and Codex, from one source rather than a copy per tool. It serves three people named in the spike, freethinkingit/dev_tools#61: +The freethinking plugin gives engineers a disciplined way to work: specs approved before code, bugs reproduced before fixes, tests with each change, and team rules in every session. Today it runs only in Claude Code. This strategy takes it to every coding agent its author and the engineers who use it work in, starting with Cursor and Codex, from one canonical source compiled into each agent's own format, rather than a copy per tool. It serves three people named in the spike, freethinkingit/dev_tools#61: - the operator, an engineer who moves between agents; - the user, who uses the software the operator builds; - the owner, who maintains the plugin. @@ -20,10 +20,10 @@ This quarter the targets are each agent's command-line tool. The IDEs may become Other gaps, such as a question tool that does not block, are recorded and do not count against parity. -1. **Claude Code, Cursor and Codex at the closest parity each allows.** In Cursor and Codex, an operator installs the plugin with one command, and its rules reach every session and sub-agent. Its skills, agents and hand-offs then behave as they do in Claude Code, as far as the tool allows. Measure: the shared eval cases for skills, sub-agents and rules pass in each tool, scored against Claude Code's results. Worth: the author stops switching back to Claude Code for the plugin. Users get the plugin's discipline whichever of the three their engineer works in. +1. **Claude Code, Cursor and Codex at the closest parity each allows.** In Cursor and Codex, an operator installs the plugin with one command, and its rules reach every session and sub-agent. Its skills, agents and hand-offs then behave as they do in Claude Code, as far as the tool allows. Measure: every shared scenario for skills, sub-agents and rules that passes in Claude Code passes in each tool, with each part delivered through that tool's own support, such as Codex's agents as its own agent files; a failure caused by a recorded gap outside those three parts does not count. Worth: the author, as an operator, stops switching back to Claude Code for the plugin. Users get the plugin's discipline whichever of the three their engineer works in, except where an agent's question tool does not wait for approval, which is recorded as a gap. 2. **Kiro next, then every agent that gains a format.** Kiro is brought up to the closest parity it allows, measured as for goal 1. Every other agent with a plugin or skills format is installed for the weekly format watch and appears in its report with its gaps, whether or not it is brought up yet. No agent is ruled out, because their support changes quickly. Worth: the author built the plugin partly to stop switching between Kiro and Claude Code, and the next agent should cost less than the last. -3. **One source, no hand-kept copies.** Each tool's packaging comes from the one source, either read as it is or generated from it at build time. A free check fails when a generated file is out of date. Measure: no file a tool reads is edited by hand outside the source. Worth: the owner keeps one plugin in step instead of several. -4. **Know within a week when a tool's support changes.** A weekly run reports which plugin parts each tool supports. It catches both a gap that opens, such as a broken hook, and one that closes, such as a tool gaining named agents. Measure: the weekly report runs, and it names any change from the week before. Worth: glue is retired as soon as a tool no longer needs it, and breakage is seen before an operator meets it. +3. **One canonical source, no hand-kept copies.** Every agent's layout, Claude Code's included, is generated at build time from one canonical source. A free check fails when a generated file is out of date or when a target leaves out part of the canonical source. Measure: no file an agent reads is edited by hand; each is generated. Worth: the owner keeps one plugin in step instead of several, and adding an agent means adding one target, not rewriting the plugin. +4. **Know within a week when a tool's support changes.** A weekly run reports which plugin parts each tool supports. It catches both a gap that opens, such as a broken hook, and one that closes, such as a tool gaining named agents. Measure: the weekly report runs, and it names any change from the week before. Worth: a target is updated as soon as an agent's support changes, and breakage is seen before an operator meets it. 5. **One delivery channel every agent shares, where it helps.** An MCP server serves the plugin's parts that MCP can carry, in whichever agent connects to it. MCP, the Model Context Protocol, is a standard way for an agent to connect to tools served by another program. All four tools accept MCP servers, Codex, Cursor and Kiro even inside their own plugin or package formats, according to the documentation read in the spike. Measure: the parts served this way work in each target, by that target's eval cases. Worth: one install line and one place to update for the parts it serves, in every agent, including ones with no plugin format. ## Current state @@ -52,39 +52,43 @@ AWS's [agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws) is t - a CI check that the manifests agree; - shared skills copied from one canonical folder by a sync script with a CI check. -It keeps hooks and agents Claude-only, leaves Claude-only syntax in its shared skill text, and gives Kiro a skills-only install. It also serves skills on demand from an MCP server through a `retrieve_skill` tool. +It keeps hooks and agents Claude-only, leaves Claude-only syntax in its shared skill text, and gives Kiro a skills-only install. It also ships a Devin manifest, so Devin has a plugin format too, and it serves skills on demand from an MCP server through a `retrieve_skill` tool. + +The author's dansabin project, a separate repository of theirs, has a Gherkin harness, `@freethinkingit/gherkin-harness`: it runs Cucumber Features through a library of ready-made steps for HTTP, browser, command and data checks, and records what each step did. It is TypeScript on Node, and it has no steps for driving a coding agent or reading its output. ## Principles and constraints -- Claude Code is the benchmark. Each other tool is measured against Claude Code's results on the same cases, and its glue aims to bring it as close to them as it allows. -- Converge first, compile at the edges. Where a tool reads the plugin's own structure, it reads the source as it is. Where it cannot, the gap is closed by glue generated at build time. Generation is never a first resort. -- Generated output is produced at build time and checked in, and a free check in CI fails when it is out of date. +- Claude Code is the benchmark. Each other tool is measured against Claude Code's results on the same cases, and its target aims to bring it as close to them as it allows. +- One canonical source, compiled for every agent. A canonical folder holds the Markdown of every skill, rule and agent, with neutral markers where agents differ. One target folder per agent, as the author chose, holds the generator that writes that agent's native layout from it. Claude Code is a target like any other. A target generates every file its agent reads, copying unchanged whatever the agent already reads natively. +- Generated output is produced at build time and checked in. A free check in CI fails when it is out of date or when a target is incomplete, that is, when it leaves out part of the canonical source. - No agent is excluded for what it cannot do today. Its gaps are recorded and revisited by the weekly run. -- One set of eval cases is shared by every tool. Each tool has its own runner, plus cases of its own for behaviour only it has. Evals are a token cost: they run by hand, one target at a time, and never in CI. +- One set of eval scenarios is shared by every agent. They run through one harness that drives each agent's command-line tool and checks both the final result and what happened along the way, such as which skill loaded, whether a question came before a write, and whether a sub-agent started. An agent may add scenarios of its own for behaviour only it has. Evals are a token cost: they run by hand, one target at a time, and never in CI. - Free checks are stdlib Python `unittest` files under `checks/`. No check greps a skill's prompt text; a skill's behaviour is proven by an eval case. - The author does the work alone, so one destination is brought up at a time. - The weekly format watch is the one scheduled run that spends tokens, and it is kept to the three probe prompts. ## Streams of work -1. **Core and glue.** Separates the tool-neutral text of skills, rules and agents from what is specific to one tool. It defines the glue each tool needs, such as reading arguments from the request, running prefetch commands as a first step, and naming files relative to the skill. It also adds the build step and its free check for anything generated. Serves goals 1, 2 and 3. First; every other stream builds on it. -2. **Tool packaging and install.** Brings up one destination at a time, each with its own manifest, its hook format and its install path. Cursor and Codex come first, beside Claude Code, then Kiro, then other agents as they gain a format. Serves goals 1 and 2. Starts once the core and glue stream has its first tool-neutral skill. -3. **Cross-tool evals.** The shared eval cases, a runner per tool, and each tool's own cases, scored against Claude Code's results. A destination is done when its runner shows the plugin's skills, sub-agents and rules working in that tool. Serves goals 1 and 2. Runs alongside the packaging stream, one destination ahead of or with it. +1. **Canonical source and the Claude Code target.** Moves the skills, rules and agents into the canonical folder, defines the neutral markers, and builds the compiler with its freshness and completeness check. Its first target is Claude Code: the generated output must match today's `plugins/freethinking/` byte for byte, so the migration changes no behaviour. Serves goals 1, 2 and 3. First; every stream but the format watch builds on it. +2. **Agent targets.** Adds one target folder per agent, one at a time. Each holds that agent's generator, manifest, hook format and install path. Cursor and Codex come first, then Kiro, then other agents as they gain a format, Devin among them, since it already has one. Serves goals 1, 2 and 3. Starts once the Claude Code target reproduces today's plugin. +3. **Cross-agent evals.** The shared scenarios and one harness that runs them against every agent, checking the final result and the events along the way, scored against Claude Code's results. It weighs the author's Gherkin harness against extending this repository's own tooling. An agent target is done when its scenarios show the plugin's skills, sub-agents and rules working in that agent. Serves goals 1 and 2. Runs alongside the agent targets stream: the scenarios for a target are written before or with that target. 4. **Format watch.** Rebuilds the spike's test plugin in the repository from its description in #61's result comment. Then a weekly automation re-runs its three probe prompts in every installed agent and reports what each supports and what changed since the week before. The automation runs in Superset, the author's agent workspace, which schedules agent tasks. Serves goals 2 and 4. Can start now. -5. **MCP delivery channel.** Designs an MCP server that serves the plugin's parts MCP can carry: skill text, templates and possibly rules. It decides whether the server runs locally or is hosted on AWS. It settles four questions first: +5. **MCP delivery channel.** Designs and builds an MCP server that serves the plugin's parts MCP can carry: skill text, templates and possibly rules. It decides whether the server runs locally or is hosted on AWS, or records a decision not to build one. It settles four questions first: - do other agents add MCP server instructions to the session, as Claude Code does; - do sub-agents see them; - which CLIs render elicitation, MCP's way for a server to ask the user a question; - does the model pick skills as reliably through a fetch tool as through descriptions. - Serves goal 5. Starts once the core and glue stream has defined the tool-neutral text. + Serves goal 5. Starts once the canonical source exists, since that is the text it would serve. ## Risks - **A format changes under a shipped destination.** Likely: Cursor's CLI started running plugin hooks in August 2026, according to its changelog. Kiro moved its powers, its package format, to the open Agent Plugins format in August 2026, according to Kiro's blog; a power still carries only skills and MCP servers. Kiro's changelog says its V3 engine becomes the default at the end of October 2026. Costly when an operator meets it first. The weekly format watch is there to see it first. -- **Glue grows into a fork.** The Claude Code text and a tool's generated text could drift until they are two plugins. Moderate likelihood, costly to unwind. The core and glue stream keeps glue to measured gaps, the spike's for Codex, Cursor and Kiro, and the weekly report's for the rest, and the weekly report retires glue a tool no longer needs. -- **The plugin's approval gates do not hold in some tools.** None of the three has a blocking question tool. Cursor's does not block, so a skill can carry on before the author approves. Codex's question tool works only in its plan mode. Kiro has none. Certain today. A gate that does not hold is recorded as a gap in that tool's evals, not hidden by glue, because no glue can make a tool wait for an answer it does not wait for. -- **Eval costs multiply by the number of tools.** Each destination adds its own runs and a paid account for that tool, and the weekly format watch spends tokens in every installed agent each week. Certain, and a moderate cost. The shared cases start from the spike's three probe prompts; #65 settles which of the existing cases under `evals/` join them. A tool's own cases are added only when a problem in that tool needs one. +- **The markers grow into a programming language.** Each agent's difference tempts a new marker, until the canonical source is harder to read than any one agent's text. Moderate likelihood, costly to unwind. Markers are added only for a difference the spike or the weekly report has measured, and the canonical source design sets how few markers there are. +- **Every change goes through a build.** Editing a skill means editing the canonical source and building, even for Claude Code. Certain, and a small cost on every edit. The freshness check in CI catches a forgotten build, and `make build` is one command. +- **The migration changes Claude Code's behaviour.** Moving the plugin into the canonical source could alter the text Claude Code reads. Possible, and costly to the benchmark. The Claude Code target must reproduce today's `plugins/freethinking/` byte for byte before any other target is added. +- **The plugin's approval gates do not hold in some tools.** None of the three has a blocking question tool in its normal mode. Cursor's does not block, so a skill can carry on before the author approves. Codex's question tool exists only in its plan mode, which the plugin's skills do not run in. Kiro has none. Certain today. A gate that does not hold is recorded as a gap in that tool's evals, not hidden by glue, because no glue can make a tool wait for an answer it does not wait for. +- **Eval costs multiply by the number of tools.** Each destination adds its own runs and a paid account for that tool, and the weekly format watch spends tokens in every installed agent each week. Certain, and a moderate cost. The shared cases start from the spike's three probe prompts; the cross-agent evals issue, #65, settles which of the existing cases under `evals/` join them. A tool's own cases are added only when a problem in that tool needs one. - **The MCP server becomes a service to run.** An AWS-hosted server needs authentication and uptime, and every session depends on the network. A single maintainer then carries that alone. Moderate likelihood if hosting is chosen. The MCP design weighs a local server against a hosted one before anything is built. ## Out of scope @@ -96,8 +100,8 @@ Anything this strategy does not name is out of scope this quarter. In particular ## High level designs -- `core-and-glue`: the split between tool-neutral text and per-tool glue, the build step that generates per-tool output, and its free check. -- `tool-packaging`: per-destination manifests, hook formats and install paths, Cursor and Codex first, then Kiro and the agents after. -- `cross-tool-evals`: the shared eval cases, a runner per tool, each tool's own cases, and scoring against Claude Code. +- `canonical-source`: the canonical folder, its neutral markers, the compiler with its freshness and completeness check, and the Claude Code target that reproduces today's plugin byte for byte. +- `agent-targets`: one target folder per agent, each with its generator, manifest, hook format and install path; Cursor and Codex first, then Kiro and the agents after. +- `cross-agent-evals`: the shared scenarios, one harness driving every agent's command-line tool with checks on the final result and the events along the way, and the choice between the Gherkin harness and this repository's own tooling. - `format-watch`: the test plugin in the repository and the weekly Superset automation that re-runs it in each agent and reports changes. - `mcp-delivery`: an MCP server for the parts MCP can carry, local or AWS-hosted, and the four questions it settles first.