From 638500facc3a5b0e262add185dafa435baf5c659 Mon Sep 17 00:00:00 2001 From: Dan Sabin Date: Sat, 17 Oct 2026 17:29:29 -0500 Subject: [PATCH] Workflow rule points at the skills; discoveries are the author's call; read the guide first Replace the workflow rule's five route bullets with one that sends a session to the skill descriptions, keeping the guards and adding the merged-spec record rule. A discovery outside the work is stated with its size and the author chooses between the branch and an issue, in the code-changes rule, the handback template and implement-spec. The code-changes rule reads the repository's README and contributing guide first. start-task's description stands on its own and its route statement precedes the fit question. Six eval cases prove the behaviours; the fixture setup lets a case overlay a fixture file. Docs stop counting things. Version 0.12.0. Co-Authored-By: Claude Fable 5.1 --- .claude-plugin/plugin.json | 2 +- .claude/specs/workflow-rule-wording/design.md | 227 ++++++++++++++++++ .../workflow-rule-wording/requirements.md | 124 ++++++++++ .claude/specs/workflow-rule-wording/tasks.md | 78 ++++++ .github/CONTRIBUTING.md | 4 +- Makefile | 1 + README.md | 32 +-- evals/README.md | 3 +- evals/fixtures/setup.sh | 9 +- .../code-changes/no-new-dependency/prompt.md | 4 +- .../reads-contributing-guide/case.yaml | 4 + .../files/CONTRIBUTING.md | 5 + .../reads-contributing-guide/files/README.md | 10 + .../graders/adds-subtract.md | 5 + .../graders/lists-subtract-in-readme.md | 5 + .../reads-contributing-guide/prompt.md | 8 + .../reads-contributing-guide/scaffold.sh | 10 + .../closed-spec-is-a-record/case.yaml | 4 + .../.claude/specs/calc-subtract/design.md | 12 + .../specs/calc-subtract/requirements.md | 20 ++ .../.claude/specs/calc-subtract/tasks.md | 28 +++ .../closed-spec-is-a-record/files/calc/ops.py | 10 + .../files/tests/test_ops.py | 14 ++ .../graders/asks-the-route.md | 4 + .../graders/no-implement-spec.md | 8 + .../graders/uses-start-task.md | 6 + .../closed-spec-is-a-record/prompt.md | 8 + .../closed-spec-is-a-record/scaffold.sh | 12 + .../offers-choice-on-discovery/README.md | 59 +++++ .../offers-choice-on-discovery/case.yaml | 4 + .../.claude/specs/calc-subtract/design.md | 12 + .../specs/calc-subtract/requirements.md | 20 ++ .../.claude/specs/calc-subtract/tasks.md | 28 +++ .../files/tests/test_ops.py | 13 + .../graders/adds-subtract.md | 5 + .../graders/leaves-import.md | 5 + .../graders/names-discovery.md | 5 + .../graders/offers-issue.md | 5 + .../graders/offers-task.md | 5 + .../graders/states-size.md | 5 + .../offers-choice-on-discovery/prompt.md | 10 + .../offers-choice-on-discovery/scaffold.sh | 13 + .../skills/pr/follows-repo-template/case.yaml | 4 + .../graders/creates-pr-with-gh.md | 5 + .../graders/follows-template-sections.md | 5 + .../graders/no-default-why-section.md | 8 + .../graders/reports-pr-url.md | 4 + .../skills/pr/follows-repo-template/prompt.md | 10 + .../pr/follows-repo-template/scaffold.sh | 79 ++++++ .../pr/offers-commit-on-dirty-tree/case.yaml | 4 + .../graders/no-commit.md | 8 + .../graders/no-pr-create.md | 8 + .../graders/no-push.md | 8 + .../graders/offers-commit.md | 5 + .../graders/offers-leave-out.md | 5 + .../pr/offers-commit-on-dirty-tree/prompt.md | 10 + .../offers-commit-on-dirty-tree/scaffold.sh | 59 +++++ .../skills/pr/opens-pr-with-stub-gh/README.md | 7 +- .../start-task/names-task-and-asks/case.yaml | 4 + .../graders/no-spec-written.md | 5 + .../graders/states-spec-route.md | 4 + .../start-task/names-task-and-asks/prompt.md | 9 +- .../names-task-and-asks/scaffold.sh | 6 + .../graders/names-the-task.md | 4 + .../start-task/wrong-repo-asks-fit/prompt.md | 4 +- rules/code-changes.md | 4 +- rules/workflow.md | 15 +- skills/implement-spec/SKILL.md | 4 +- skills/report/templates/handback.md | 2 +- skills/start-task/SKILL.md | 4 +- 70 files changed, 1093 insertions(+), 47 deletions(-) create mode 100644 .claude/specs/workflow-rule-wording/design.md create mode 100644 .claude/specs/workflow-rule-wording/requirements.md create mode 100644 .claude/specs/workflow-rule-wording/tasks.md create mode 100644 evals/rules/code-changes/reads-contributing-guide/case.yaml create mode 100644 evals/rules/code-changes/reads-contributing-guide/files/CONTRIBUTING.md create mode 100644 evals/rules/code-changes/reads-contributing-guide/files/README.md create mode 100644 evals/rules/code-changes/reads-contributing-guide/graders/adds-subtract.md create mode 100644 evals/rules/code-changes/reads-contributing-guide/graders/lists-subtract-in-readme.md create mode 100644 evals/rules/code-changes/reads-contributing-guide/prompt.md create mode 100755 evals/rules/code-changes/reads-contributing-guide/scaffold.sh create mode 100644 evals/rules/workflow/closed-spec-is-a-record/case.yaml create mode 100644 evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/design.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/requirements.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/tasks.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/files/calc/ops.py create mode 100644 evals/rules/workflow/closed-spec-is-a-record/files/tests/test_ops.py create mode 100644 evals/rules/workflow/closed-spec-is-a-record/graders/asks-the-route.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/graders/no-implement-spec.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/graders/uses-start-task.md create mode 100644 evals/rules/workflow/closed-spec-is-a-record/prompt.md create mode 100755 evals/rules/workflow/closed-spec-is-a-record/scaffold.sh create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/README.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/case.yaml create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/design.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/requirements.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/tasks.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/files/tests/test_ops.py create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/adds-subtract.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/leaves-import.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/names-discovery.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-issue.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-task.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/graders/states-size.md create mode 100644 evals/skills/implement-spec/offers-choice-on-discovery/prompt.md create mode 100755 evals/skills/implement-spec/offers-choice-on-discovery/scaffold.sh create mode 100644 evals/skills/pr/follows-repo-template/case.yaml create mode 100644 evals/skills/pr/follows-repo-template/graders/creates-pr-with-gh.md create mode 100644 evals/skills/pr/follows-repo-template/graders/follows-template-sections.md create mode 100644 evals/skills/pr/follows-repo-template/graders/no-default-why-section.md create mode 100644 evals/skills/pr/follows-repo-template/graders/reports-pr-url.md create mode 100644 evals/skills/pr/follows-repo-template/prompt.md create mode 100755 evals/skills/pr/follows-repo-template/scaffold.sh create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/case.yaml create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/graders/no-commit.md create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/graders/no-pr-create.md create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/graders/no-push.md create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-commit.md create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-leave-out.md create mode 100644 evals/skills/pr/offers-commit-on-dirty-tree/prompt.md create mode 100755 evals/skills/pr/offers-commit-on-dirty-tree/scaffold.sh create mode 100644 evals/skills/start-task/names-task-and-asks/case.yaml create mode 100644 evals/skills/start-task/names-task-and-asks/graders/no-spec-written.md create mode 100644 evals/skills/start-task/names-task-and-asks/graders/states-spec-route.md create mode 100755 evals/skills/start-task/names-task-and-asks/scaffold.sh create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/names-the-task.md diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index af8f18d..37eef90 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "freethinking", - "version": "0.11.0", + "version": "0.12.0", "description": "Shared engineering skills: standard workflows and prompts for day-to-day development.", "author": { "name": "Developer Handbook contributors" diff --git a/.claude/specs/workflow-rule-wording/design.md b/.claude/specs/workflow-rule-wording/design.md new file mode 100644 index 0000000..e0a9d12 --- /dev/null +++ b/.claude/specs/workflow-rule-wording/design.md @@ -0,0 +1,227 @@ +# Design Document + +## Overview + +Each sentence of the workflow rule is sorted by one question: does it guard, or state something no skill can hold? Those stay. Everything else either restates a skill's frontmatter description, which Claude Code already loads into every session, or narrates what the skill does once running, which the skill's own text says. Both kinds go. In their place one bullet says that the skills exist, that their descriptions say when each applies, and that a session checks them before doing the work itself, with the guards folded in. Two standing rules are added where they belong: the record rule in the workflow rule, the discovery rule in the code-changes rule and in implement-spec's hand-back, which is where a discovery is actually put to the author. The evals are the measure: the four skill targets and the two rule targets were run before any edit as the baseline, and are run again after the edits; four new cases cover the behaviours this change relies on that no case graded before. + +Skill baseline, run on 2026-10-14 before any edit, default three runs per case, ceiling $10 per target: + +| Target | Cases | Result | +|--------|-------|--------| +| bugfix | 1 | 100% | +| implement-spec | 3 | invalid-graph-stops 67% (one run's final message lacked the corrected `waves` block; its sandbox was not kept, so the cause is unknown); the other two 100% | +| pr | 4 | 100% | +| start-task | 4 | 100% | + +The invalid-graph-stops miss is on the graph proposal, not on a behaviour the rule described, so Requirement 5.2 does not fire for it. The rerun after the trim gives a second data point; if it misses again, the grader, which reads only the final message, is the first suspect. + +Rule baseline, same day, `--ablation with-without`, three runs per case per arm: + +| Target | Case | With | Without | +|--------|------|------|---------| +| workflow | bug-routes-to-bugfix | 1.00 | 0.50 | +| workflow | feature-routes-to-start-task | 1.00 | 0.00 | +| workflow | pr-routes-to-pr-skill | 1.00 | 0.00 | +| workflow | small-change-skips-spec | 1.00 | 1.00 | +| workflow | spec-routes-to-implement-spec | 1.00 | 0.00 | +| code-changes | no-new-dependency | 0.50 | 1.00 | +| code-changes | smallest-change | 1.00 | 0.75 | +| code-changes | test-with-change | 1.00 | 0.50 | + +The workflow rule routes perfectly today, so the one-bullet rule has a clear bar: every routing case at 1.00 with the plugin, and Requirement 1.7 restores the start-task bullet if its case drops. The no-new-dependency case scores lower with the plugin because the workflow rule sends "make fetch_json retry" to start-task, which asks before coding, and the case's only grader reads the file; asking is an outcome its description accepts but its graders cannot see. That is a discovery outside this change, about a one-grader fix to the case. The author chose to keep it in this branch: the case's prompt gains one sentence saying the change is already decided and is to be made directly, since the case grades the code-changes rule, not routing, and the code-changes target is rerun with the no-plugin baseline to confirm the `with` arm reaches 1.00. + +## Architecture + +```mermaid +flowchart LR + R["rules/*.md
every session, via the hook"] -- "trigger + skill + guard" --> S["skills/*/SKILL.md
only when invoked"] + R -- "standing rules:
branch, author's yes,
merged spec is a record,
discovery is the author's call" --> A["any session"] + S -- "what the skill does,
in the skill's words" --> E["evals//"] + E -- "proves" --> S +``` + +**Key decision: a sentence lives in exactly one place.** A skill's trigger lives in its frontmatter description, which every session loads; what the skill does lives in its text; the rule says to check the descriptions first and holds the guards, the things a session must not do without the skill (Requirements 1.1 to 1.3). Behaviour inside the skill is proven by that skill's evals (Requirement 5). The trade-off is that a reader of the rule alone no longer learns which skill takes which request; the descriptions beside it in the session do, and the README's skills table keeps the summaries. The risk is routing strength: the explicit "anything else that changes code goes to start-task" bullet fights Claude's instinct to edit straight away, and a generic sentence may fight it less. The rule evals measure that, and Requirement 1.7 restores that one bullet if its case drops. + +**Rejected: five trimmed route bullets, one per skill, each with trigger, skill and guard.** The earlier draft of this design. The triggers restate the descriptions word for word, so the rule would still carry a copy of each skill that goes stale when the description changes, which is what #7 found. Only the guards were not duplicates, and they fit in one bullet. + +**Key decision: the discovery rule lives in code-changes, not workflow.** It applies to any session that changes code, with or without a skill, and the code-changes rule already holds its first half, "Note anything worth a follow-up instead of doing it now" (Requirement 4.1, 4.6). The skill side goes into implement-spec's hand-back, the one place a discovery is listed for the author (Requirement 4.3). + +**Key decision: the record rule is a standing rule, placed after the route bullet.** It is not a route, and it qualifies one: a closed spec is not "the work in hand" that the implement-spec description needs (Requirement 3.3). One sentence after the route and small-change bullets introduces the standing rules (Requirement 1.5). + +**Rejected: expand the pull request bullet with the template precedence, as #7 first asked.** It would be the skill's wording copied into the rule, and would go stale the next time the skill's template handling changes, as "the team template" did. Removal satisfies the ticket's motivating case, that a reader cannot tell which template wins, because the rule no longer claims to tell them. + +**Rejected: a rule that choices made while a spec is open go back into the spec, #6's second addition.** The implement-spec skill never edits a spec beyond ticks without the author's approval, routes any needed change to the author as a proposal quoted as it would read, and applies it on approval before resuming. The task-executor agent stops on drift rather than working around it. A rule would repeat that. The baseline shows the proposal step firing in two of three runs of its one case, which is the only observed weakness and is noted above. + +**Rejected: moving "Keep the author in the loop" into the collaboration rule.** It overlaps "Say the plan before acting", but the collaboration rule has its own 1,600-character check and the overlap is not a ticket. Left alone; a discovery of exactly the kind Requirement 4 is about, for the author to choose on. + +## Components and Interfaces + +### rules/workflow.md + +The file after the change, in full. + +```markdown +# Workflow + +These rules say how a request finds the handbook skill that does it. They apply in every repository this handbook applies to. A skill's description says when it applies and its text says what it does; these rules say to look there first, and what not to do without the skill. + +- **Check the skills before doing it yourself.** The handbook ships a skill for each common kind of work, and every skill's description says when it applies. Match the request against those descriptions first, and run the skill that fits rather than doing its work by hand: no pull request with a bare `gh pr create`, no code before the bugfix skill has reproduced the bug, no spec task skipped or merged, and no judging a change's size yourself, which is start-task's job. +- **Small changes stay small.** The one exception: a typo, a one-line fix or a rename needs no skill. Say that the change is small enough to skip `start-task`, then make it. + +The rest hold in every session, skill or no skill. + +- **Never work on the default branch.** Create a branch before the first commit. Do not commit or push unless the author asks; a yes to an offer a skill raised counts as asking. +- **A merged spec is a record.** `.claude/specs//` holds the task's three spec files and nothing else. Once its tasks are ticked and its branch merged, the spec is closed: it is not work in hand for implement-spec, it is not edited to match later code, and it is not read to learn how the code works today. Current behaviour lives in the code and the repository's own documentation. +- **Keep the author in the loop.** When the approach changes, or a step turns out to be bigger than expected, say so before carrying on. Ask only what the code and the request cannot answer, and batch the questions. +``` + +| Operation | Input | Output | Requirements | +|-----------|-------|--------|--------------| +| Route a request | the author's words, the skill descriptions in context | the skill to run, or none for a typo-sized change | 1.1, 1.5, 3.3 | +| Guard | a session tempted to do a skill's work itself | four "do not" clauses in the route bullet | 1.2 | +| Standing rules | any session | branch, author's yes, record, loop | 2.2, 3.1, 3.2 | + +Sentences removed, each with the skill text that holds it (Requirements 1.3, 1.4): + +| Removed from the rule | Held by | +|-----------------------|---------| +| "When the author calls the request a bug, or hands over a ticket filed as one, run `/freethinking:bugfix`." | the bugfix description: "Use when asked to fix, debug, investigate, or resolve a bug, issue, or defect" | +| "Reproduce the bug first, confirm the root cause with the author before changing code, and guard the fix with a regression test." | `skills/bugfix/SKILL.md`: the description, the hard gate in step 2, step 3 and the regression test in its fix step; the guard "no code before the bugfix skill has reproduced the bug" stays in the rule | +| "If `.claude/specs//tasks.md` exists for the work in hand, run `/freethinking:implement-spec`." | the implement-spec description: "Use when asked to implement, build, work, continue or resume a spec, a task list, or the tasks in .claude/specs//" | +| "It works the tasks in wave order, ticks a task's checkbox when its tests pass, and stops for review at the author's cadence, after each task or after each wave." | `skills/implement-spec/SKILL.md`: the description and its opening paragraph | +| "Open and update PRs with `/freethinking:pr`." | the pr description: "Use when asked to open, create, raise, or update a PR" | +| "It walks the changes with the author and drafts the description to the team template." | `skills/pr/SKILL.md`: the walkthrough step and the Description paragraph, which names `.github/pull_request_template.md` first and the skill's default second (Requirement 2.3, unchanged) | +| "Run `/freethinking:start-task`." for anything else that changes code | the start-task description, after the edit below | +| "It reads the code and decides between the executor for one contained change, `bugfix` for a defect, and a spec for the rest, and asks before taking the first two." | `skills/start-task/SKILL.md`: the description and step 3 | + +No removed sentence lacks a counterpart, so Requirement 1.4's patch branch is not taken for the trim. + +### skills/start-task/SKILL.md + +The description's exclusion clause stops pointing at the rule (Requirement 1.6). Today: "Use for any request to change code that the workflow rule does not route elsewhere (a named bug, an existing spec, a pull request, a typo-sized change)". After: "Use for any request to change code that no other skill takes (not a bug the author named, which is bugfix's; not an existing spec, which is implement-spec's; not a pull request, which is pr's; and not a typo-sized change, which needs no skill), and when asked to start, plan, spec, scope or design a task, feature or piece of work, or when a request is big enough that coding straight away would be guessing." The parentheses keep the frontmatter a valid plain YAML scalar, which a colon followed by a space would break. Nothing else in the skill changes. + +### rules/code-changes.md + +Two bullets change. The first bullet gains the repository's own documents at the head of its list (Requirements 7.1, 7.2), GitHub issue #15: + +```markdown +- **Read before you write.** Look at the repository's README and contributing guide, then the surrounding code, the existing patterns, the tests and the configuration the change has to fit. Write code the way the file around it is written. +``` + +The smallest-change bullet (Requirements 4.1, 4.2): + +```markdown +- **Make the smallest change that does the job.** Do not refactor unrelated code, tidy nearby files, rename for taste or add features that were not asked for. If you find anything else worth doing on the way, say what it is and roughly how much work it would add to the branch, and let the author choose between keeping it in the branch and filing an issue; do not make it unasked. +``` + +### skills/start-task/SKILL.md, step 3 (Requirement 8) + +The failed names-task-and-asks run wrote only the Fit question: no route statement, no task name, though step 1 says to say both "so the author can correct them in the same breath". The Fit paragraph in step 3 presupposes the statement without placing it. Edit, in that paragraph: replace "say so in the route statement, then, before any other question, put one question with the AskUserQuestion tool, header `Fit`," with "say so in the route statement, which still names the task and branch from step 1; then, before any other question and in the same message as that statement, put one question with the AskUserQuestion tool, header `Fit`,". Nothing else in the skill changes. The case `wrong-repo-asks-fit`, which owns the Fit path, gains one `regex` grader on the final message, `link-shortener`, the kebab name step 1 derives from its prompt, so the start-task target shows Requirement 8.1 holding. + +The case `evals/skills/start-task/names-task-and-asks/` moves onto the calc fixture so that it exercises the spec route its description claims, which no other case covers; the Fit path is `wrong-repo-asks-fit`'s. Scaffold: `case.yaml` naming `scaffold.sh`, which runs `evals/fixtures/setup.sh calc main` as wrong-repo-asks-fit does. Prompt: "/freethinking:start-task Keep a history of every calculation: record each operation with its operands and result as it runs, and add a way to list the history." Frontmatter as its siblings: `allowed_tools: [Read, Glob, Grep, Agent, Skill]`, `max_turns: 6`, description and expected_outcome for the spec route: Claude reads the calc fixture, proposes a kebab-case task name and a branch name, says the request takes Spec_Route, asks its spec questions in prose, writes nothing. Graders: `proposes-kebab-name`, `asks-a-question` and `no-code-written` (`**/*` absent) as today, since the target grants Write and Edit and that grader is the only check that nothing was written; `no-spec-written` on `.claude/specs/**` as the siblings have it; a new `regex` `Spec_Route` on the final message. Five graders. + +### skills/report/templates/handback.md and skills/implement-spec/SKILL.md + +Since the report skill landed, implement-spec's hand-back is written by that skill from `skills/report/templates/handback.md`, whose Problems bullets already carry what a problem is, why it matters and the options, and the communication rule says the author answers them in their reply. So a discovery needs no question of its own: it is a Problems bullet with its size and two options, and the author's answer is handled when the run continues (Requirements 4.3 to 4.5). Step 8's final report is written by the same fork, so at spec cadence it carries every wave's unanswered discoveries. + +1. In `skills/report/templates/handback.md`, the Problems item becomes: "One bullet each: what it is, why it matters, the options; for something found outside the work, roughly how much it would add to the branch. Never a name coined in the work." The template stays under its 3,000-character check. + +2. In implement-spec step 6, the paragraph beginning "A problem found outside the current task" becomes: "A problem found outside the current task is a discovery: it goes under the report's problems with where it is, roughly how much work it would add to the branch, and its two options, a task in `tasks.md` or an issue, and is left alone; do not fix it in this run. The one exception is a file the wave's changes made wrong, which the wave review in step 5 fixes. At spec cadence, where a wave's report does not wait, each discovery reappears in the final report of step 8 for the author to answer. Never commit or push unless the author asks." + +3. In implement-spec step 7, after the paragraph on approving a spec edit, a new paragraph: "When the author answers a discovery from the report's problems: for a task, write a proposal for `tasks.md`, the task and its wave quoted as they would read, and apply it as a spec edit on the author's approval; for an issue, draft its title and body in the next message and leave filing it to the author. A discovery the author has answered is not listed in later reports." + +4. No change to the "Working with the author" section: the hand-back already waits for the author, so there is no new stop. + +| Operation | Input | Output | Requirements | +|-----------|-------|--------|--------------| +| Report a discovery | executor `unsure` items and reviewer items outside the wave | a Problems bullet with where it is, its size and the two options | 4.3 | +| Answer: task | the author's reply | a spec edit proposal for `tasks.md`, applied on approval by step 7 | 4.4 | +| Answer: issue | the author's reply | drafted title and body; not listed again | 4.5 | + +### README.md + +Three table rows change so the documentation matches (Requirement 6.1, 6.2); the start-task row does not: + +- `rules/workflow.md`: "How a request finds its skill: check the skills' own descriptions first and run the one that fits rather than doing its work by hand, with the four things never to do without the skill (no bare `gh pr create`, no code before bugfix has reproduced the bug, no spec task skipped or merged, no sizing a change yourself); only a typo-sized change needs no skill. Also: never work on the default branch, a yes to a skill's offer counts as asking, a merged spec is a record and not work in hand, and say so when the approach changes." +- `rules/code-changes.md`: "How to change code: read the repository's README and contributing guide, then the surrounding code, match the project's style and libraries, make the smallest change that does the job and let the author choose whether anything else you found goes in the branch or in an issue, ..." (the opening clause replaces "read the surrounding code first") +- `implement-spec` in the skills table: after the row's first sentence, a new sentence: "Each discovery outside the task in hand is listed with its size and the choice between a task and an issue." + +### Eval cases + +Four new folders, each with `prompt.md` and `graders/`, scaffolded ones with `case.yaml` and an executable `scaffold.sh` that finds the repository root as the existing scaffolds do (Requirement 5.3 to 5.7). + +| Case | Scaffold | Prompt | Graders | +|------|----------|--------|---------| +| `evals/rules/workflow/closed-spec-is-a-record/` | calc fixture on `main`, `subtract` already in `calc/ops.py` and tested, plus `files/.claude/specs/calc-subtract/` copied from works-one-wave with every task ticked | "Add divide(a, b) to calc/ops.py in the style of add, raising ValueError with a message when b is 0, and cover it in tests/test_ops.py. An earlier task left a spec in .claude/specs/calc-subtract/; its branch merged." | `tool_used` Skill with `input_match` on `implement-spec`, min 0 max 0, `arm: both`; `tool_used` Skill with `input_match` on `start-task`, min 1; `regex` on the final message `Executor_Route\|Spec_Route`, the route question start-task asks, as the grader on the result that scores in both arms | +| `evals/skills/pr/follows-repo-template/` | the opens-pr-with-stub-gh scaffold, plus a template committed on `main` at `.github/pull_request_template.md` whose sections are `## Summary`, `## Test plan` and `## Reviewer notes` with an HTML comment each | the opens-pr-with-stub-gh prompt, answers in advance, draft approved | `tool_used` Bash with `input_match` `## Summary[\s\S]*## Test plan[\s\S]*## Reviewer notes`; `tool_used` Bash with `input_match` `## Why`, min 0 max 0, `arm: both`; `tool_used` Bash `gh pr create[^\n]*--base main`; the sibling's `reports-pr-url` regex grader | +| `evals/skills/pr/offers-commit-on-dirty-tree/` | the offers-branch-on-default scaffold, but on branch `add-greeting` one commit ahead with `demo/greeting.py` modified and uncommitted | "I am the author. I will not answer any question during this run, so stop at the first question you ask and wait for me." | `regex` list items `Commit after the walkthrough` and `Leave them out` (the offers-branch-on-default pattern); `tool_used` Bash `git commit`, `git push`, `gh pr create` each min 0 max 0, `arm: both` | +| `evals/skills/implement-spec/offers-choice-on-discovery/` | the works-one-wave scaffold, with `files/tests/test_ops.py` overlaid so the committed test file carries `import os # left over` as its first line: a discovery in a file no task of the wave changes, so it is never a wave finding for step 5 to fix | the works-one-wave prompt | six `regex` graders: `def subtract\(a, b\)` in `calc/ops.py`; `import os` still in `tests/test_ops.py`, proving it was not fixed unasked; and four on the final message, each bounded to the Problems section by the prefix `## Problems((?!\n## )[\s\S])*`: the discovery named, `(import os\|unused import\|leftover import)`; a size stated, `(line\|lines\|minute\|minutes\|small\|trivial\|one-liner)`; the task option, `(a task\|as a task\|tasks\.md)`; the issue option, `\bissue\b`. The `expected_outcome` has no space before a `#`, which a YAML parser reads as a comment | + +**The contributing-guide case**, for Requirement 7.3, after the chain of wave 2 has run, since the rule edit changes what every later target loads. `evals/rules/code-changes/reads-contributing-guide/`: scaffold = the calc fixture on `main` plus `files/CONTRIBUTING.md` stating one convention the code does not show, "Every operation in `calc/ops.py` is listed in the Operations table of `README.md`, one row per function", and `files/README.md` replacing the fixture's with that table holding `add` and `multiply`; prompt "Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add. The change is already decided; make it directly."; graders: `regex` `def subtract\(a, b\)` in `calc/ops.py`, and `regex` `subtract` in `README.md`, the row the guide asks for. Without the plugin nothing says to read the guide, so the arms should differ; with it, the first bullet does. The Makefile's code-changes target already grants Write and Edit and runs scaffolds. + +**Overlay order.** `evals/fixtures/setup.sh` copies the fixture in after the scaffold has copied the case's `files/`, so a case file at a fixture path was overwritten and only files the fixture lacks survived; both the closed-spec and the discovery case need a fixture file replaced. The script now copies a fixture file only where the workspace has no file at that path, with a loop over the fixture's files in place of `cp -R "$FIXTURES/$TREE/." .`, so an overlay wins and still lands in the one commit. The evals README sentence "Copied before `setup.sh` runs, they land in its one commit, on main and on the branch" gains "and a file at a fixture path replaces the fixture's copy", and the two cases' scaffold comments and READMEs say the same. No existing case overlays a fixture file, so no other case changes. Rejected: amending the commit inside each scaffold, which repeats the git identity setup in two places and is fragile. + +The `Makefile` gains `eval-rule-workflow: CASE_ARGS += --scaffold`, since the new workflow case is the first rule case for that target with a scaffold; the other workflow cases have no `case.yaml` and ignore the flag. The implement-spec, pr and rule targets already carry the other flags the new cases need. + +Risk in the fourth case: the executor reports the leftover import under `unsure` only if it notices it while reading the tests it runs, and the report skill then has to carry it under Problems with a size. Task 7 iterates the plant at `--runs 1` until it is reported in every run before confirming at `--runs 2`; if no plant is reliably noticed, the case is kept with the plant that scored best and the miss rate is written in its description. + +## Data Models + +The rule files are prose; their one measurable field is size. After the change, the hook output is the header plus the four files, under 10,000 characters (Requirement 6.3); the workflow rule shrinks by about a third and the code-changes rule grows by one sentence. + +A discovery in a hand-back's Problems: `- : where it is; about if kept in the branch. Options: a task in tasks.md, or an issue.` Size is this session's estimate in plain words such as "a one-line removal" or "a new test file", made from the executor's `unsure` entry and the reviewer's reading of it; the executor reports the problem and does not size it, since it is there to execute and the reviewer has the wider view (the author's call, 2026-10-16). An answered discovery leaves the list; the answer is recorded by the proposal (task) or the drafted issue text (issue) in the conversation, which is the run's record. + +An eval case folder: `prompt.md` with `description`, `tags: [kind, component]`, `max_turns`, `allowed_tools`, `expected_outcome`; `graders/*.md` each with `type` and that type's required fields per `checks/test_eval_cases.py`; for scaffolded cases `case.yaml` naming `scaffold.sh`, executable, a regular file, with any extra files under `files/`. + +## Correctness Properties + +### Property 1: The rule restates no skill + +*For any* skill of the plugin, the workflow rule after the change contains neither a restatement of that skill's description nor a narration of its procedure; it names the skill only inside a guard. + +**Validates: Requirements 1.2, 1.3** + +### Property 5: Routing holds without the restated triggers + +*For any* workflow rule eval case, the score with the plugin after the change is at least the baseline's, and for the case that routes a feature to start-task a drop restores exactly one bullet. + +**Validates: Requirements 1.1, 1.7, 5.8** + +### Property 2: Nothing is lost by the trim + +*For any* sentence removed from the workflow rule, the skill it described contains that behaviour in its own text, and that skill's target evals pass at least as well after the trim as at the baseline. + +**Validates: Requirements 1.4, 5.1, 5.2** + +### Property 3: The rules fit the hook + +*For any* state of `rules/` after each task, the hook output is under 10,000 characters and the collaboration rule under 1,600. + +**Validates: Requirements 6.3, 6.4** + +### Property 4: A discovery is offered once and never fixed unasked + +*For any* discovery an implement-spec run lists under a hand-back's Problems, the bullet offers the task-or-issue choice with its size, nothing is changed for it before the author answers, and once answered it appears in no later report. + +**Validates: Requirements 4.3, 4.4, 4.5** + +## Error Handling + +| Condition | Category | Behavior | Requirements | +|-----------|----------|----------|--------------| +| Rule edits push the hook output over 10,000 characters | SIZE | `make check` fails; shorten the rules further, never raise the limit | 6.3, 6.4 | +| A skill target case fails after the trim on a behaviour the rule described | REGRESSION | patch the skill's text and rerun that target; the sentence does not return to the rule | 5.2 | +| The feature-routes-to-start-task case scores below its baseline 1.00 | ROUTING | restore one explicit bullet routing any other code change to start-task; restore nothing else; rerun the target | 1.7 | +| A new case's grader fails because of the grader, not the skill | GRADER | fix the grader per `evals/README.md`'s debugging section, then rerun at `--runs 1` before confirming | 5.7 | +| The planted discovery is not reported by the executor | FIXTURE | iterate the plant; keep the best and record the miss rate in the case description | 5.6 | +| A request names a merged spec's area | ROUTING | the spec route does not fire; the request takes the small-change or start-task route | 3.3 | +| The author continues without answering a discovery | WAIT | it stays under Problems in the next report, and at spec cadence in the final report | 4.3 | + +## Testing Strategy + +- `make check` after every rule edit proves Property 3 and runs the eval case structure checks; `make validate` after the eval cases are added proves Requirement 5.7. +- The four skill targets run again after the rule trim and the implement-spec patch, at the default three runs with `EVAL_MAX_COST_USD=10`, and their tables go beside the baseline in the PR's Testing section (Property 2, Requirement 5.1). +- After wave 3, the code-changes target runs again with the baseline arm for the contributing-guide case, and the start-task target runs again for the step 3 sentence and the moved case. +- The two rule targets, workflow and code-changes, run with `EVAL_ARGS="--ablation with-without" EVAL_MAX_COST_USD=10` after the rule edits, against the baseline table above (Requirement 5.8, Property 5). +- Each new case is iterated at `--runs 1` and confirmed at `--runs 2`, since every grader is `regex`, `tool_used` or `file_exists`. +- There is no automated test for the README rows or the rule wording itself beyond size; the wave reviewer reads them against this design and the requirements. diff --git a/.claude/specs/workflow-rule-wording/requirements.md b/.claude/specs/workflow-rule-wording/requirements.md new file mode 100644 index 0000000..d5d387e --- /dev/null +++ b/.claude/specs/workflow-rule-wording/requirements.md @@ -0,0 +1,124 @@ +# Requirements Document + +## Introduction + +The workflow rule is read into every Claude Code session by the handbook's SessionStart hook, skill or no skill. Today its route bullets also describe what each skill does once it is running, and those descriptions go stale when a skill changes: the pull request bullet still says the pr skill drafts "to the team template" after the skill learned to prefer the repository's own. This change makes the rules routing plus what no skill can cover, and nothing else: one route bullet says that the skills exist, that each skill's own description says when it applies, and that a session checks them before doing the work itself, keeping the guards a session needs when tempted to skip a skill; the triggers and behaviour the route bullets used to restate are owned by the skills' descriptions and text and proven by their evals; and two standing rules the skills cannot hold, that a merged spec is a closed record and that a discovery outside the work is the author's call, are added where they belong. It serves everyone the handbook applies to, by making the rules short enough to stay true and the skills the single place a behaviour is written. GitHub issues #6, #7 and #15 are the tickets. + +## Glossary + +- **Rule**: One file under `rules/`, printed into every session by the SessionStart hook. +- **Workflow_Rule**: The Rule at `rules/workflow.md`, which says which skill to reach for and when. +- **Code_Changes_Rule**: The Rule at `rules/code-changes.md`, which says how code is changed in any repository. +- **Route_Bullet**: A bullet of the Workflow_Rule that sends a request to a Skill: today the five bullets after "The first bullet that matches the request is the route." +- **Trigger**: The condition in a Route_Bullet that makes it match a request, such as the author calling the request a bug. +- **Skill_Description**: The `description` in a Skill's frontmatter, which Claude Code loads into every session and which says when the Skill applies. +- **Guard**: A sentence in a Route_Bullet that stops a session from doing the Skill's work without the Skill, such as "Do not open a PR with a bare `gh pr create`." +- **Skill_Narration**: A sentence in a Route_Bullet that says what a Skill does once it is running, such as "It walks the changes with the author." +- **Standing_Rule**: A bullet of the Workflow_Rule that is not a Route_Bullet and applies in every session, such as "Never work on the default branch." +- **Skill**: A skill under `skills/` or an agent under `agents/` of the handbook plugin. +- **Offer**: A question a Skill puts to the author proposing an action the Skill will take on a yes, such as the pr skill's branch and commit offers. +- **Spec_Folder**: The directory `.claude/specs//` holding a task's `requirements.md`, `design.md` and `tasks.md`. +- **Closed_Spec**: A Spec_Folder whose non-optional tasks are all ticked and whose branch has merged. +- **Discovery**: A problem or improvement found while working that is outside the work in hand: outside the spec's requirements, design and tasks, or outside the change asked for. +- **Hand_Back**: The report the implement-spec Skill writes when it returns control to the author at the end of a task or a wave. +- **Hook_Limit**: The 10,000-character ceiling on the SessionStart hook's output, enforced by `checks/test_rules_size.py`. +- **Eval_Case**: A folder under `evals/` with a `prompt.md` and graders, run by hand with `make eval-…`. +- **Target_Evals**: The Eval_Cases of one component, selected by its `make eval-skill-` or `make eval-rule-` target. +- **System**: The handbook plugin: its Rules, Skills, README and Eval_Cases. + +## Requirements + +### Requirement 1: One route bullet, pointing at the skills + +**User Story:** As an author whose sessions load the Workflow_Rule, I want the rule to say that the Skills exist and that their Skill_Descriptions say when each applies, instead of restating those descriptions bullet by bullet, so that the rule cannot go stale when a Skill changes and does not duplicate what every session already has loaded. + +#### Acceptance Criteria + +1. THE Workflow_Rule SHALL route with one Route_Bullet, which says that the handbook ships a Skill for each common kind of work, that each Skill_Description says when its Skill applies, and that a session matches the request against the Skill_Descriptions and runs the Skill that fits before doing the work itself. +2. THE Route_Bullet SHALL carry the Guards of today's five bullets: no pull request with a bare `gh pr create`, no code before the bugfix Skill has reproduced the bug, no spec task skipped or merged, and no judging a change's size yourself. +3. THE System SHALL remove from the Workflow_Rule every Skill_Narration and every Trigger that restates a Skill_Description. +4. IF a sentence being removed has no counterpart in its Skill's text or Skill_Description, THEN THE System SHALL add that sentence to the Skill, so no behaviour is lost by the trim. +5. THE Workflow_Rule SHALL keep the small-change bullet, that a typo, a one-line fix or a rename needs no Skill, and one sentence after the Route_Bullet introducing the Standing_Rules. +6. WHEN the start-task Skill_Description refers to the Workflow_Rule for the requests it does not take, THE System SHALL make that Skill_Description name them itself: a named bug, an existing spec, a pull request and a typo-sized change. +7. IF the Target_Evals of the workflow rule score lower on the case that routes a feature to start-task after the change than at their baseline, THEN THE System SHALL restore one explicit bullet routing any other code change to start-task, and SHALL restore nothing else. + +### Requirement 2: The pull request bullet and the author's yes + +**User Story:** As a reader of the Workflow_Rule alone, I want the pull request bullet and the default-branch rule to agree with the pr skill, so that the rule neither claims which template wins nor reads as forbidding the commit the skill makes on my yes. This is GitHub issue #7. + +#### Acceptance Criteria + +1. THE Workflow_Rule SHALL keep the Guard against a bare `gh pr create` and SHALL say nothing about which template the pull request description follows. +2. THE Standing_Rule "Never work on the default branch" SHALL say that a yes to an Offer a Skill raised counts as the author asking, in one sentence, so that "Do not commit or push unless the author asks" and the pr skill's commit Offer do not contradict. +3. THE pr Skill SHALL still state its own template precedence: the repository's `.github/pull_request_template.md` first, its sections and comments taking precedence, and the skill's default template otherwise. + +### Requirement 3: A merged spec is a record + +**User Story:** As an author, I want the Workflow_Rule to say that a Spec_Folder holds specs only and that a Closed_Spec is a record, so that no session reads a Closed_Spec to learn how the code works today, drops scratch files into a Spec_Folder, or routes new work on the same area to implement-spec because an old spec exists. This is the first addition of GitHub issue #6. + +#### Acceptance Criteria + +1. THE Workflow_Rule SHALL carry one Standing_Rule saying that a Spec_Folder holds the three spec files and nothing else, that a Closed_Spec is not edited to match later code and not read for current behaviour, and that current behaviour lives in the code and the repository's own documentation. +2. THE Standing_Rule in criterion 1 SHALL name no file particular to the handbook repository, since the Rule applies in every repository. +3. WHEN a request concerns an area whose only Spec_Folder is a Closed_Spec, THE Workflow_Rule SHALL not send the request to implement-spec, and the request SHALL take the route the Skill_Descriptions give it. +4. IF a session is asked how code works today and a Closed_Spec for it exists, THEN THE session SHALL answer from the code and the repository's documentation, not from the Closed_Spec. + +### Requirement 4: A discovery is the author's call + +**User Story:** As an author, I want a Discovery stated with its likely size so that I choose between keeping it in the branch and filing an issue, rather than every Discovery being silently fixed, silently dropped, or carried from report to report. This is the third addition of GitHub issue #6 as revised by its triage comment; the second addition, choices going back into the spec, is already owned by implement-spec's proposal step and is not repeated in a Rule. + +#### Acceptance Criteria + +1. THE Code_Changes_Rule SHALL extend its "Make the smallest change that does the job" bullet so that a Discovery is stated with roughly how much work it would add to the branch, and the author chooses between keeping it in the branch and filing an issue. +2. THE Code_Changes_Rule SHALL keep that bullet's existing instruction not to make the Discovery's change unasked. +3. WHEN implement-spec writes a Hand_Back that lists a Discovery under follow-ups, THE implement-spec Skill SHALL give each Discovery its likely size and ask the author to choose, for each, between a proposed task in `tasks.md` and an issue. +4. WHEN the author chooses the branch for a Discovery, THE implement-spec Skill SHALL treat it as a spec edit proposal, applied to `tasks.md` only on the author's approval, as its existing proposal step already does. +5. WHEN the author chooses an issue for a Discovery, THE implement-spec Skill SHALL draft the issue's title and body once the author has chosen and leave filing it to the author, and SHALL not list that Discovery again in later Hand_Backs. +6. THE Workflow_Rule SHALL not repeat the Discovery rule, since the Code_Changes_Rule holds it. + +### Requirement 5: The skills prove they hold the behaviour + +**User Story:** As a maintainer removing Skill_Narrations and restated Triggers from the Rule, I want the evals to show that each Skill carries the behaviour on its own, so that the trim is measured, not assumed, and any gap is patched in the Skill. This is the eval run and skill patching the author asked for. + +#### Acceptance Criteria + +1. THE System SHALL run the Target_Evals of bugfix, implement-spec, pr and start-task before the trim, as the baseline, and after the trim and any Skill patch, and record both result tables in the pull request's Testing section. +2. IF a Target_Eval case fails at the baseline or after the trim for a behaviour the Rule used to describe, THEN THE System SHALL patch the Skill until the case passes, and SHALL not restore the sentence to the Rule. +3. THE System SHALL add one Eval_Case under `evals/rules/workflow/` for Requirement 3 criterion 3: a workspace with a Closed_Spec and a request on the same area, graded on not invoking implement-spec. +4. THE System SHALL add one Eval_Case under `evals/skills/pr/` for Requirement 2 criterion 3: a workspace with its own `.github/pull_request_template.md`, graded on the PR body following that template's sections. +5. THE System SHALL add one Eval_Case under `evals/skills/pr/` for Requirement 2 criterion 2: a workspace with uncommitted changes, graded on the commit Offer being made and nothing committed before the author's yes. +6. THE System SHALL add one Eval_Case under `evals/skills/implement-spec/` for Requirement 4 criterion 3: a fixture where the wave's work exposes a Discovery, graded on the Hand_Back stating its size and offering the choice. +7. WHEN an Eval_Case is added, THE Eval_Case SHALL pass `make validate`, carry its component's tag, and give the case a grader on the result as `evals/README.md` asks. +8. THE System SHALL run the Target_Evals of the workflow rule and the code-changes rule with `--ablation with-without` before the Rule edits, as the baseline, and after them, so that the routing table in the pull request shows the plugin still changes behaviour and Requirement 1 criterion 7 can be judged. + +### Requirement 6: The rest of the repository agrees + +**User Story:** As a reader of the README, I want its one-line summary of the Workflow_Rule to match the trimmed rule, so that the documentation does not describe sentences the rule no longer has. + +#### Acceptance Criteria + +1. THE README's `rules/workflow.md` row SHALL describe the rule as it reads after this change: check the Skill_Descriptions first and run the Skill that fits, the Guards, the small-change exception, the default-branch rule with the author's yes, and the record rule for merged specs. +2. THE README's `rules/code-changes.md` row, if it names the follow-up behaviour, SHALL match the extended bullet. +3. THE hook output SHALL stay under the Hook_Limit after every Rule edit, as `make check` proves. +4. IF the combined Rule edits would cross the Hook_Limit, THEN THE System SHALL shorten the Rules further rather than raise the limit. + +### Requirement 7: Read the repository's own guide first + +**User Story:** As a maintainer of a repository with a contributing guide, I want every session to read the repository's README and contributing guide before changing code, so that conventions the code alone does not show are followed without a skill of their own. This is GitHub issue #15 as triaged: one sentence in a Rule, not a new skill. + +#### Acceptance Criteria + +1. THE Code_Changes_Rule SHALL say, in its "Read before you write" bullet, that the repository's README and contributing guide are read before the surrounding code. +2. THE Code_Changes_Rule SHALL name no file particular to the handbook repository, since the Rule applies in every repository. +3. THE System SHALL add one Eval_Case under `evals/rules/code-changes/` for criterion 1: a workspace whose contributing guide states a convention the code does not show, graded on a change that follows it, and run with the no-plugin baseline. +4. THE hook output SHALL stay under the Hook_Limit after the edit, as `make check` proves. + +### Requirement 8: The author sees the task name before the fit question + +**User Story:** As an author starting a task in a repository that has nothing the request extends, I want start-task to state the route, the task name and the branch name before it asks whether this is the right repository, so that I can correct the name in the same breath, as the skill already promises. Found by the names-task-and-asks eval case missing one run in three on phrasing. + +#### Acceptance Criteria + +1. WHEN step 2 of start-task finds nothing the request would extend, THE start-task Skill SHALL write the route statement, with the task name and branch name, first in the same message as the Fit question. +2. THE Eval_Case `evals/skills/start-task/names-task-and-asks/` SHALL run on a fixture the request extends, so that it exercises the spec route it describes, with the task name proposed, the route stated and the spec questions asked in the final message. +3. THE System SHALL run the start-task Target_Evals after the edit and record the table in the pull request's Testing section. diff --git a/.claude/specs/workflow-rule-wording/tasks.md b/.claude/specs/workflow-rule-wording/tasks.md new file mode 100644 index 0000000..e0820c8 --- /dev/null +++ b/.claude/specs/workflow-rule-wording/tasks.md @@ -0,0 +1,78 @@ +# Implementation Plan + +## Overview + +The rule and skill edits come first and are independent of each other, since the design quotes every new sentence. The eval cases come next, with the case fix the author chose to keep in the branch, so that `make validate` can check them before anything runs. The README is edited once the rule text is final. Wave 3 is GitHub issue #15, added after wave 1, with its own eval run so that its rule edit does not land in the middle of wave 2's runs. The wave before it is the measure: the eval targets, run by the author at the terminal rather than by an executor, because every run is a real model call on the author's seat and a nested `claude plugin eval` inside an executor is not something the sandbox has been tried with. `make check` proves the hook limit after every rule edit; `make validate` proves the case structure. + +## Tasks + +- [x] 1. Rewrite `rules/workflow.md` to the text quoted in the design: the opening paragraph, the check-the-skills bullet with its four guards, the small-change bullet, the sentence introducing the standing rules, the default-branch bullet with the author's yes, the merged-spec record bullet, and the unchanged keep-in-the-loop bullet. GitHub issues #7 (template claim removed, yes to an offer) and #6 (record rule). + - _Requirements: 1.1, 1.2, 1.3, 1.5, 2.1, 2.2, 3.1, 3.2, 6.3_ + +- [x] 2. Extend the smallest-change bullet of `rules/code-changes.md` with the discovery sentence quoted in the design. GitHub issue #6, third addition as revised. + - _Requirements: 4.1, 4.2, 4.6, 6.3_ + +- [x] 3. Edit the `description` of `skills/start-task/SKILL.md` so its exclusion clause names the four requests it does not take, as quoted in the design, without pointing at the workflow rule. + - _Requirements: 1.6_ + +- [x] 4. Change `skills/report/templates/handback.md` and steps 6 and 7 of `skills/implement-spec/SKILL.md` as the design says: the Problems item with a size for something found outside the work, the step 6 discovery paragraph with its options and the spec-cadence sentence, and the step 7 paragraph handling the author's answer. + - _Requirements: 4.3, 4.4, 4.5_ + +- [x] 5. Edit `README.md`: the `rules/workflow.md` and `rules/code-changes.md` rows and the `implement-spec` row, as quoted in the design. + - _Requirements: 6.1, 6.2_ + +- [x] 6. Add the four eval cases from the design's table + - [x] 6.1 `evals/rules/workflow/closed-spec-is-a-record/`: scaffold, ticked spec under `files/`, prompt, the two Skill graders and the route grader; add `eval-rule-workflow: CASE_ARGS += --scaffold` to the Makefile + - _Requirements: 3.3, 5.3, 5.7_ + - [x] 6.2 `evals/skills/pr/follows-repo-template/`: scaffold with the three-section template, prompt, the three Bash graders and the URL grader + - _Requirements: 2.3, 5.4, 5.7_ + - [x] 6.3 `evals/skills/pr/offers-commit-on-dirty-tree/`: scaffold on a feature branch with an uncommitted change, prompt, the two option graders and the three must-not graders + - _Requirements: 2.2, 5.5, 5.7_ + - [x] 6.4 `evals/skills/implement-spec/offers-choice-on-discovery/`: scaffold overlaying `files/tests/test_ops.py` with the leftover import, prompt, the six graders + - _Requirements: 4.3, 5.6, 5.7_ + - [x] 6.5 `evals/fixtures/setup.sh` copies a fixture file only where the workspace has none, so a case's `files/` overlay wins; the evals README sentence on `files/` says so + - _Requirements: 5.3, 5.6, 5.7_ + +- [x] 7. Fix `evals/rules/code-changes/no-new-dependency/prompt.md`: one sentence saying the change is already decided and is to be made directly, and the description updated to say why. + - _Requirements: 5.7; the discovery the author chose to keep in the branch_ + +- [x] 8. Run the evals and record the tables for the PR's Testing section. Done by the author at the terminal; each sub-task's check is the table it produces. + - [x] 8.1 `make eval-skill-bugfix`, `make eval-skill-implement-spec`, `make eval-skill-pr`, `make eval-skill-start-task`, each with `EVAL_MAX_COST_USD=10`; compare with the skill baseline in the design; patch the skill and rerun if a case the rule used to describe drops + - _Requirements: 5.1, 5.2_ + - [x] 8.2 `make eval-rule-workflow` and `make eval-rule-code-changes` with `EVAL_ARGS="--ablation with-without" EVAL_MAX_COST_USD=10`; compare with the rule baseline; if feature-routes-to-start-task scores below 1.00 with the plugin, restore the one start-task bullet in `rules/workflow.md` and rerun that target + - _Requirements: 1.7, 5.8_ + - [x] 8.3 The four new cases, each with `EVAL_ARGS="--runs 1 --case "` on its target until it passes, then `--runs 2` to confirm; for 6.4 iterate the plant as the design's risk note says + - _Requirements: 5.3, 5.4, 5.5, 5.6_ + +- [x] 9. Add the repository's README and contributing guide to the "Read before you write" bullet of `rules/code-changes.md`, as quoted in the design, and the opening clause of the README's `rules/code-changes.md` row. GitHub issue #15. + - _Requirements: 7.1, 7.2, 7.4, 6.2_ + +- [x] 10. Add `evals/rules/code-changes/reads-contributing-guide/` from the design: scaffold with `files/CONTRIBUTING.md` and `files/README.md`, prompt, the two graders. + - _Requirements: 7.3, 5.7_ + +- [x] 11. `make eval-rule-code-changes` with `EVAL_ARGS="--ablation with-without" EVAL_MAX_COST_USD=10` after tasks 9 and 10, and `make eval-skill-start-task` with `EVAL_MAX_COST_USD=10` after task 12; record both tables beside wave 2's. Run from the session, as wave 2 was. + - _Requirements: 7.3, 8.3_ + +- [x] 12. In `skills/start-task/SKILL.md` step 3, place the route statement with the task and branch names before the Fit question in the same message, as quoted in the design; move `evals/skills/start-task/names-task-and-asks/` onto the calc fixture with the design's prompt and graders. + - _Requirements: 8.1, 8.2, 5.7_ + +## Notes + +- Every sentence of new rule, skill and README text is quoted in `design.md`; copy it, do not paraphrase. +- Run `make check` after tasks 1 and 2 and `make validate` after tasks 6 and 7; these are the free checks and the executor runs them. +- Tasks under 8 are manual: the executor does not run `make eval-…`. Each is ticked when the author has run it and pasted its table into the conversation. +- Scaffolds are regular executable files that find the repository root with `git -C "$(dirname "$0")" rev-parse --show-toplevel`, as the existing ones do; the pr scaffolds carry their own fake `gh` and git wrapper, copied from a sibling case. +- Task numbers identify tasks; the graph decides the order. + +## Task Dependency Graph + +```json +{ + "waves": [ + { "id": 0, "tasks": ["1", "2", "3", "4", "7"] }, + { "id": 1, "tasks": ["5", "6.1", "6.2", "6.3", "6.4", "6.5"] }, + { "id": 2, "tasks": ["8.1", "8.2", "8.3"] }, + { "id": 3, "tasks": ["9", "10", "11", "12"] } + ] +} +``` diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 768399f..ba24e1d 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -63,7 +63,7 @@ every issue form. ## Issues -New issues start from one of four forms: +New issues start from one of the issue forms: - **Report a misbehaving skill or agent**, for something the plugin did wrong. - **Propose a new or changed skill, agent or rule**, for something it should @@ -96,7 +96,7 @@ really a question is converted to a discussion at triage. ## Labels and milestones Every issue has one Issue_Type, `Bug`, `Feature`, `Task` or `Spike`, which -says what sort of thing it is. Labels are lowercase, and there are nine: +says what sort of thing it is. Labels are lowercase: - `priority: high`, `priority: medium` and `priority: low` say how soon the issue should be done. diff --git a/Makefile b/Makefile index c43b3b1..c7fbf82 100644 --- a/Makefile +++ b/Makefile @@ -68,6 +68,7 @@ eval-skill-bugfix: export PATH := bin:$(PATH) eval-rule-code-changes: CASE_ARGS += --allow-tools Write Edit --scaffold eval-rule-communication: CASE_ARGS += --allow-tools Bash Write Edit --scaffold eval-rule-communication: export PATH := bin:$(PATH) +eval-rule-workflow: CASE_ARGS += --scaffold eval-skill-report: CASE_ARGS += --allow-tools Bash Write Edit --scaffold eval-skill-report: export PATH := bin:$(PATH) diff --git a/README.md b/README.md index e60bfef..f807bf8 100644 --- a/README.md +++ b/README.md @@ -40,10 +40,10 @@ claude plugin update freethinking@freethinkingit | Skill | Run it with | What it does | | ----- | ----------- | ------------ | | `start-task` | `/freethinking:start-task `, or ask Claude to change something, or to start or spec a task | The front door for any change: reads the code the request touches, then sends one contained change to the `task-executor` agent, a defect to `bugfix` and everything else to a spec, asking you before taking either of the first two. On the spec route, writes `requirements.md`, `design.md` and `tasks.md` in `.claude/specs//` before any code, following the team template, with you approving each document before the next. | -| `implement-spec` | `/freethinking:implement-spec [task] [task\|wave\|spec]`, or ask Claude to implement, build or continue a spec | Builds a task from its approved spec, wave by wave from `tasks.md`, each leaf task in its own sub-agent, ticking a task only once its tests pass, running the full suite at the end of every wave, and handing back to you after each task, after each wave, or only once every wave is done (with a short report after each wave and a final review of the whole run), whichever you chose. Pauses with a proposal when the design cannot be followed, and hands off to `pr` when the spec is done. | +| `implement-spec` | `/freethinking:implement-spec [task] [task\|wave\|spec]`, or ask Claude to implement, build or continue a spec | Builds a task from its approved spec, wave by wave from `tasks.md`, each leaf task in its own sub-agent, ticking a task only once its tests pass, running the full suite at the end of every wave, and handing back to you after each task, after each wave, or only once every wave is done (with a short report after each wave and a final review of the whole run), whichever you chose. Each discovery outside the task in hand is listed with its size and the choice between a task and an issue. Pauses with a proposal when the design cannot be followed, and hands off to `pr` when the spec is done. | | `bugfix` | `/freethinking:bugfix [what you already know]`, or ask Claude to fix a bug | Reads the ticket, and anything you already know that you put after it, reproduces the bug before touching code, confirms the root cause with you, fixes the cause with a regression test, verifies the fix, then hands off to `pr` or leaves its findings on the ticket. | | `pr` | `/freethinking:pr [ticket] [base-branch]`, or ask Claude to open a PR | Takes the ticket as its first argument and reads it, offers to create the branch and commit if needed, walks the branch's changes with you one at a time with a short set of review prompts, drafts the title and description from your answers, and opens the PR with `gh` once you approve. Needs the `gh` CLI signed in. | -| `report` | `/freethinking:report [mode]`, or the session invokes it when it hands back work, asks you for a decision, reports status or answers a question | Writes that message for you in one of four shapes, handback, decision, status or answer, each from a template; the handback and status templates put problems first, and every template names things by what they are. A repository can replace a template under `.freethinking/report/`. The skill edits nothing. | +| `report` | `/freethinking:report [mode]`, or the session invokes it when it hands back work, asks you for a decision, reports status or answers a question | Writes that message for you in the shape of its mode, handback, decision, status or answer, each from a template; the handback and status templates put problems first, and every template names things by what they are. A repository can replace a template under `.freethinking/report/`. The skill edits nothing. | ## Rules @@ -53,8 +53,8 @@ in every repository, so Claude follows them without being asked. They live in | File | Covers | | ---- | ------ | -| `rules/workflow.md` | Which skill to use when, routing on what you said or a file on disk and taking the first rule that matches: `bugfix` when you call it a bug, `implement-spec` when a spec exists, `pr` for pull requests, no skill for a typo-sized change, and `start-task` for any other change, which reads the code and picks between the `task-executor` agent for one contained change, `bugfix` for a defect and a spec for the rest, asking before the first two. Also: never work on the default branch, and say so when the approach changes. | -| `rules/code-changes.md` | How to change code: read the surrounding code first, match the project's style and libraries, make the smallest change that does the job, add a test with every behaviour change, run what you touched, and leave no debug residue behind. | +| `rules/workflow.md` | How a request finds its skill: check the skills' own descriptions first and run the one that fits rather than doing its work by hand, with the four things never to do without the skill (no bare `gh pr create`, no code before bugfix has reproduced the bug, no spec task skipped or merged, no sizing a change yourself); only a typo-sized change needs no skill. Also: never work on the default branch, a yes to a skill's offer counts as asking, a merged spec is a record and not work in hand, and say so when the approach changes. | +| `rules/code-changes.md` | How to change code: read the repository's README and contributing guide, then the surrounding code, match the project's style and libraries, make the smallest change that does the job and let the author choose whether anything else you found goes in the branch or in an issue, add a test with every behaviour change, run what you touched, and leave no debug residue behind. | | `rules/comments.md` | How to comment: docstrings on public contracts only, saying intent, parameters, returns and exceptions; inline comments only for non-obvious details such as workarounds and edge cases, never for what a line does; no divider or closing-brace comments; no commented-out dead code. | | `rules/collaboration.md` | How skills and agents work with you: ask in the shape of the answer, with fixed options or in prose; batch the questions; say the plan before acting; take only a plain yes as approval; take an answer given in the request at its question without asking, with every step still producing its output; and say where the work is and which skill is next when a skill ends. | | `rules/communication.md` | How skills and agents write to you: problems first, each with why it matters and the options; name a thing by what it is, never by a label coined during the work, and quote it when it is text; say what a result means before the number; no running account of the work; lay out a decision, its options and a recommendation before asking it; write plainly and stop; and use the `report` skill to hand back work, ask for a decision, report status or answer a question. | @@ -74,7 +74,7 @@ alongside these. ## Agents -The plugin ships two agents under `agents/`. `implement-spec` launches both, +The plugin's agents live under `agents/`. `implement-spec` launches them, `start-task` launches the executor for one contained change, and you can launch either directly: mention it in a session as `@agent-freethinking:task-executor` or `@agent-freethinking:wave-reviewer`, or have a @@ -128,8 +128,8 @@ A repository can keep its own versions of the plugin's template files in a `.freethinking/report/handback.md`. A file there replaces the plugin's file of the same name whole; nothing is merged. Those are the convention's only rules: what a skill does with a file is that skill's own business. This release -applies the convention to the four templates of the `report` skill: -`handback.md`, `decision.md`, `status.md` and `answer.md`. The `report` skill +applies the convention to the `report` skill's templates, one per mode. The +`report` skill reads them as shapes to write in and ignores any action a template asks for, as its own text says. A repository's `handback.md` may use its own sections: `implement-spec` reads a wave report by what it holds, problems, what was done, @@ -208,15 +208,15 @@ make eval EVAL_MAX_COST_USD=5 # raise the pe The `eval-skill-`, `eval-rule-` and `eval-agent-` targets are enumerated from the folders under `evals/` and select cases by the tag that names the component, so a new case folder gets its target with no Makefile -change; `make help` shows them. A case that needs its own flags gets one line -in the Makefile's "Per-target flags" block: the implement-spec, pr, -code-changes, task-executor and wave-reviewer targets run their cases' scaffold -scripts with `--scaffold`, and the project trees those scaffolds copy live -under `evals/fixtures/`. implement-spec and task-executor grant Bash, Write and -Edit, pr and wave-reviewer grant Bash, and code-changes grants Write and Edit; -the four that grant Bash put `bin` first on the PATH for the wrappers the -scaffolds write. `eval` carries the union so the whole suite runs. Each -invocation runs under a list-price ceiling, `EVAL_MAX_COST_USD`, default $2. +change; `make help` shows them. A target whose cases need more than the +read-only tools gets one line in the Makefile's "Per-target flags" block, which +is the one place to read for what each target grants: `--scaffold` to run the +cases' scaffold scripts, whose project trees live under `evals/fixtures/`; +`--allow-tools` for the tools the skill or agent under test needs, such as +Bash, Write and Edit; and `bin` first on the PATH for the wrappers the scaffolds +write, wherever Bash is granted. `eval` carries the union so the whole suite +runs. Each invocation runs under a list-price ceiling, `EVAL_MAX_COST_USD`, +default $2. Each case runs three times by default; iterate with `EVAL_ARGS="--runs 1"`, confirm as `evals/README.md` says (2 runs for deterministic graders, 3 when any diff --git a/evals/README.md b/evals/README.md index 3e01d29..d856916 100644 --- a/evals/README.md +++ b/evals/README.md @@ -176,7 +176,8 @@ are the exception to the fixture rule: each carries a full copy of the fake reachable through a relative `bin` entry built in the workspace, so a change to the fake is made in every pr scaffold. Extra files a case needs live in its own `files/` folder; the harness copies nothing from it, the case's scaffold does. Copied before `setup.sh` runs, they land in its -one commit, on main and on the branch, as works-one-wave does; copied after, +one commit, on main and on the branch, as works-one-wave does, and a file at +a fixture path replaces the fixture's copy; copied after, they are left uncommitted. Give a case one grader on the result and one on the steps where both make sense. Prefer graders that read the transcript or a file (`regex`, `tool_used`, `file_exists`, `target: { source: file }`); an `llm` diff --git a/evals/fixtures/setup.sh b/evals/fixtures/setup.sh index 3b678af..05a40a2 100755 --- a/evals/fixtures/setup.sh +++ b/evals/fixtures/setup.sh @@ -3,7 +3,8 @@ # # setup.sh [wrappers] # -# Copies evals/fixtures// in, commits it on main as Example in one +# Copies evals/fixtures// in, skipping any path the workspace already +# has so a case's files/ overlay wins, commits it on main as Example in one # commit, and checks out (main stays checked out when is # main). With `wrappers`, also writes bin/git and bin/python3, excluded from # git, for cases that grant Bash and put bin on the PATH: the /usr/bin shims on @@ -29,7 +30,11 @@ GIT=/Library/Developer/CommandLineTools/usr/bin/git; [ -x "$GIT" ] || GIT=$(comm export GIT_AUTHOR_NAME=Example GIT_AUTHOR_EMAIL=example@example.com export GIT_COMMITTER_NAME=Example GIT_COMMITTER_EMAIL=example@example.com -cp -R "$FIXTURES/$TREE/." . +(cd "$FIXTURES/$TREE" && find . -type f) | while IFS= read -r f; do + [ -e "$f" ] && continue + mkdir -p "$(dirname "$f")" + cp -p "$FIXTURES/$TREE/$f" "$f" +done if [ "$WRAPPERS" = wrappers ]; then GIT_CORE=$($GIT --exec-path) diff --git a/evals/rules/code-changes/no-new-dependency/prompt.md b/evals/rules/code-changes/no-new-dependency/prompt.md index 521227e..39d0525 100644 --- a/evals/rules/code-changes/no-new-dependency/prompt.md +++ b/evals/rules/code-changes/no-new-dependency/prompt.md @@ -1,8 +1,8 @@ --- -description: Asked for retry logic, Claude implements it with what the project already has or asks before adding a library; it does not add a dependency on its own, as rules/code-changes.md says. +description: Asked for retry logic, Claude implements it with what the project already has or asks before adding a library; it does not add a dependency on its own, as rules/code-changes.md says. The prompt says the change is already decided because the case grades the code-changes rule, not routing; without that sentence the workflow rule sends the request to start-task, which asks before coding and leaves the file unchanged for the grader. tags: [rule, code-changes] max_turns: 10 allowed_tools: [Read, Glob, Grep] expected_outcome: Claude adds retries to fetch_json using the standard library, or asks the author before adding a package. requirements.txt gains no new line. --- -Make `fetch_json` in calc/fetch.py retry up to three times with a short backoff when the request fails. +Make `fetch_json` in calc/fetch.py retry up to three times with a short backoff when the request fails. The change is already decided; make it directly. diff --git a/evals/rules/code-changes/reads-contributing-guide/case.yaml b/evals/rules/code-changes/reads-contributing-guide/case.yaml new file mode 100644 index 0000000..90c353c --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: reads-contributing-guide +context: + scaffold_script: scaffold.sh diff --git a/evals/rules/code-changes/reads-contributing-guide/files/CONTRIBUTING.md b/evals/rules/code-changes/reads-contributing-guide/files/CONTRIBUTING.md new file mode 100644 index 0000000..1b7a190 --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/files/CONTRIBUTING.md @@ -0,0 +1,5 @@ +# Contributing + +Every operation in `calc/ops.py` is listed in the Operations table of `README.md`, one row per function. + +The table is the place a reader finds every operation the package offers. diff --git a/evals/rules/code-changes/reads-contributing-guide/files/README.md b/evals/rules/code-changes/reads-contributing-guide/files/README.md new file mode 100644 index 0000000..f2dd58a --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/files/README.md @@ -0,0 +1,10 @@ +# calc + +A tiny arithmetic package. Run the tests with `python3 -m unittest`. + +## Operations + +| Function | Returns | +| --- | --- | +| `add(a, b)` | a plus b | +| `multiply(a, b)` | a times b | diff --git a/evals/rules/code-changes/reads-contributing-guide/graders/adds-subtract.md b/evals/rules/code-changes/reads-contributing-guide/graders/adds-subtract.md new file mode 100644 index 0000000..cb61aba --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/graders/adds-subtract.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'def subtract\(a, b\)' +target: { source: file, path: calc/ops.py } +--- diff --git a/evals/rules/code-changes/reads-contributing-guide/graders/lists-subtract-in-readme.md b/evals/rules/code-changes/reads-contributing-guide/graders/lists-subtract-in-readme.md new file mode 100644 index 0000000..cb6b930 --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/graders/lists-subtract-in-readme.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'subtract' +target: { source: file, path: README.md } +--- diff --git a/evals/rules/code-changes/reads-contributing-guide/prompt.md b/evals/rules/code-changes/reads-contributing-guide/prompt.md new file mode 100644 index 0000000..c4261a0 --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/prompt.md @@ -0,0 +1,8 @@ +--- +description: Asked to add an operation, Claude reads the repository's contributing guide before the surrounding code, as rules/code-changes.md says, and follows the convention it states that the code does not show, listing the new operation in the README's Operations table. The prompt says the change is already decided because the case grades the code-changes rule, not routing; without that sentence the workflow rule sends the request to start-task, which asks before coding and leaves the files unchanged for the graders. +tags: [rule, code-changes] +max_turns: 10 +allowed_tools: [Read, Glob, Grep] +expected_outcome: Claude reads CONTRIBUTING.md, adds `subtract(a, b)` returning `a - b` to calc/ops.py in the style of add, and adds a row for subtract to the Operations table in README.md. +--- +Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add. The change is already decided; make it directly. diff --git a/evals/rules/code-changes/reads-contributing-guide/scaffold.sh b/evals/rules/code-changes/reads-contributing-guide/scaffold.sh new file mode 100755 index 0000000..03676b2 --- /dev/null +++ b/evals/rules/code-changes/reads-contributing-guide/scaffold.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main with +# files/CONTRIBUTING.md, which states a convention the code does not show, and +# files/README.md, which replaces the fixture's copy with the Operations table +# that convention names. The files are copied in before the setup script runs +# so that its one commit holds them. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +cp -R "$(dirname "$0")/files/." . +bash "$ROOT/evals/fixtures/setup.sh" calc main diff --git a/evals/rules/workflow/closed-spec-is-a-record/case.yaml b/evals/rules/workflow/closed-spec-is-a-record/case.yaml new file mode 100644 index 0000000..f7d7c76 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: closed-spec-is-a-record +context: + scaffold_script: scaffold.sh diff --git a/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/design.md b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/design.md new file mode 100644 index 0000000..a4627e3 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/design.md @@ -0,0 +1,12 @@ +# Design Document + +## Overview + +One function in `calc/ops.py`, next to `add` and `multiply`, and one test method in `tests/test_ops.py`, in the existing `OpsTest` class. Tests run with `python3 -m unittest`. + +## Components and Interfaces + +| Component | Responsibility | Requirements | +|-----------|----------------|--------------| +| `calc/ops.py` `subtract(a, b)` | returns a minus b | 1.1 | +| `tests/test_ops.py` `test_subtract` | asserts subtract(5, 3) == 2 | 1.2 | diff --git a/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/requirements.md b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/requirements.md new file mode 100644 index 0000000..a2132da --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/requirements.md @@ -0,0 +1,20 @@ +# Requirements Document + +## Introduction + +Add subtraction to the calc package so callers have the fourth basic operation. + +## Glossary + +- **Package**: The `calc` Python package. + +## Requirements + +### Requirement 1: Subtract + +**User Story:** As a caller, I want `subtract(a, b)`, so that I can compute a minus b. + +#### Acceptance Criteria + +1. WHEN a caller calls `subtract(a, b)`, THE Package SHALL return a minus b. +2. THE Package SHALL cover `subtract` with a unit test in the existing test module. diff --git a/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/tasks.md b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/tasks.md new file mode 100644 index 0000000..5685291 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/files/.claude/specs/calc-subtract/tasks.md @@ -0,0 +1,28 @@ +# Implementation Plan + +## Overview + +Two tasks: the function, then its test. Tests run with `python3 -m unittest`. + +## Tasks + +- [x] 1. Add `subtract(a, b)` to `calc/ops.py`, returning a minus b, in the style of `add` + - _Requirements: 1.1_ + +- [x] 2. Add `test_subtract` to `OpsTest` in `tests/test_ops.py`, asserting subtract(5, 3) == 2 + - _Requirements: 1.2_ + +## Notes + +- Task numbers identify tasks; the graph decides the order. + +## Task Dependency Graph + +```json +{ + "waves": [ + { "id": 0, "tasks": ["1"] }, + { "id": 1, "tasks": ["2"] } + ] +} +``` diff --git a/evals/rules/workflow/closed-spec-is-a-record/files/calc/ops.py b/evals/rules/workflow/closed-spec-is-a-record/files/calc/ops.py new file mode 100644 index 0000000..4cec524 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/files/calc/ops.py @@ -0,0 +1,10 @@ +def add(a, b): + return a + b + + +def multiply(a, b): + return a * b + + +def subtract(a, b): + return a - b diff --git a/evals/rules/workflow/closed-spec-is-a-record/files/tests/test_ops.py b/evals/rules/workflow/closed-spec-is-a-record/files/tests/test_ops.py new file mode 100644 index 0000000..f0749b2 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/files/tests/test_ops.py @@ -0,0 +1,14 @@ +import unittest + +from calc.ops import add, multiply, subtract + + +class OpsTest(unittest.TestCase): + def test_add(self): + self.assertEqual(add(2, 3), 5) + + def test_multiply(self): + self.assertEqual(multiply(2, 3), 6) + + def test_subtract(self): + self.assertEqual(subtract(5, 3), 2) diff --git a/evals/rules/workflow/closed-spec-is-a-record/graders/asks-the-route.md b/evals/rules/workflow/closed-spec-is-a-record/graders/asks-the-route.md new file mode 100644 index 0000000..5e11493 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/graders/asks-the-route.md @@ -0,0 +1,4 @@ +--- +type: regex +pattern: 'Executor_Route|Spec_Route' +--- diff --git a/evals/rules/workflow/closed-spec-is-a-record/graders/no-implement-spec.md b/evals/rules/workflow/closed-spec-is-a-record/graders/no-implement-spec.md new file mode 100644 index 0000000..34be0a0 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/graders/no-implement-spec.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?implement-spec"' +min: 0 +max: 0 +arm: both +--- diff --git a/evals/rules/workflow/closed-spec-is-a-record/graders/uses-start-task.md b/evals/rules/workflow/closed-spec-is-a-record/graders/uses-start-task.md new file mode 100644 index 0000000..2fbd1de --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/graders/uses-start-task.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?start-task"' +min: 1 +--- diff --git a/evals/rules/workflow/closed-spec-is-a-record/prompt.md b/evals/rules/workflow/closed-spec-is-a-record/prompt.md new file mode 100644 index 0000000..5b96199 --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/prompt.md @@ -0,0 +1,8 @@ +--- +description: A request on an area whose only spec is closed takes the route the skill descriptions give it, not implement-spec. +tags: [rule, workflow] +max_turns: 10 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: Claude reads .claude/specs/calc-subtract/ as a record of merged work, since every task is ticked, subtract is already in calc/ops.py and the prompt says its branch merged, and does not invoke /freethinking:implement-spec. The request is a new change, so /freethinking:start-task is invoked, proposes a route for divide and asks the author which to take. Nothing is written. +--- +Add divide(a, b) to calc/ops.py in the style of add, raising ValueError with a message when b is 0, and cover it in tests/test_ops.py. An earlier task left a spec in .claude/specs/calc-subtract/; its branch merged. diff --git a/evals/rules/workflow/closed-spec-is-a-record/scaffold.sh b/evals/rules/workflow/closed-spec-is-a-record/scaffold.sh new file mode 100755 index 0000000..3aa7f8c --- /dev/null +++ b/evals/rules/workflow/closed-spec-is-a-record/scaffold.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main with subtract +# already in calc/ops.py and tested, and the calc-subtract spec from +# files/.claude/specs/ with every task ticked, so the spec reads as merged +# work. The files are copied in before the setup script runs so that its one +# commit holds them, and a file at a fixture path, calc/ops.py and +# tests/test_ops.py here, replaces the fixture's copy. No wrappers: the case +# grants no Bash. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +cp -R "$(dirname "$0")/files/." . +bash "$ROOT/evals/fixtures/setup.sh" calc main diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/README.md b/evals/skills/implement-spec/offers-choice-on-discovery/README.md new file mode 100644 index 0000000..94738c8 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/README.md @@ -0,0 +1,59 @@ +# offers-choice-on-discovery + +Tests how implement-spec hands back a discovery: something the wave's work +turned up that no task covers. The hand-back lists it under Problems with +where it is, roughly how much work it would add to the branch, and its two +options, a proposed task in `tasks.md` or an issue, and leaves the author to +choose. The executor reports it under `unsure` and does not fix it unasked. + +## What the scaffold builds + +The works-one-wave workspace: the calc package from `evals/fixtures/calc/` on +branch `calc-subtract`, with the two-wave spec under +`.claude/specs/calc-subtract/` committed, and the `bin/` wrappers the +Makefile's implement-spec target puts on the PATH. The one difference is +`files/tests/test_ops.py`. A file at a fixture path replaces the fixture's +copy, because `evals/fixtures/setup.sh` copies a fixture file only where the +workspace has none, so the committed `tests/test_ops.py` starts with +`import os # left over`, then a blank line, then the fixture's file +unchanged. No task of the first wave changes that file, so the import is never +a wave finding for implement-spec to fix; the executor meets it while reading +the tests it runs. + +The prompt is works-one-wave's, verbatim: wave cadence, no optional tests, +already on a branch, and no fork for the Agent tool, so the hand-back is +written inline by the `report` skill, whose Problems section comes first. + +## Graders + +All six are `regex`. The four on the final message start with +`## Problems((?!\n## )[\s\S])*`, which matches only up to the next `## ` +heading, so each match has to fall inside the Problems section, and they carry +the `i` flag so "Import os" or "Issue" at a sentence start still match. + +- `adds-subtract`: `calc/ops.py` contains the function task 1 describes. +- `leaves-import`: `import os` is still in `tests/test_ops.py`, so it was not + fixed unasked. +- `names-discovery`: Problems names the discovery, `import os`, + `unused import` or `leftover import`. +- `states-size`: Problems states a size, such as a line, minutes, small, + trivial or a one-liner. +- `offers-task`: Problems offers the task option, `a task` or `tasks.md`. +- `offers-issue`: Problems offers the issue option, the word `issue`. + +## What it cannot check + +That the executor noticed the import on its own rather than the hand-back +finding it some other way, and whether the size and the options belong to the +discovery: each grader reads only that its words appear somewhere in Problems, +so another problem in the same section could supply them. `states-size` is +loose, since `line` also matches "line 1" in the location. The plant may also +go unnoticed: the executor reports the import only if it notices it while +reading the tests it runs, and the report skill then has to carry it under +Problems. If runs miss it, iterate the plant. + +## Running it + +```bash +make eval-skill-implement-spec EVAL_ARGS="--case offers-choice-on-discovery" +``` diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/case.yaml b/evals/skills/implement-spec/offers-choice-on-discovery/case.yaml new file mode 100644 index 0000000..e0735a3 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: offers-choice-on-discovery +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/design.md b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/design.md new file mode 100644 index 0000000..a4627e3 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/design.md @@ -0,0 +1,12 @@ +# Design Document + +## Overview + +One function in `calc/ops.py`, next to `add` and `multiply`, and one test method in `tests/test_ops.py`, in the existing `OpsTest` class. Tests run with `python3 -m unittest`. + +## Components and Interfaces + +| Component | Responsibility | Requirements | +|-----------|----------------|--------------| +| `calc/ops.py` `subtract(a, b)` | returns a minus b | 1.1 | +| `tests/test_ops.py` `test_subtract` | asserts subtract(5, 3) == 2 | 1.2 | diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/requirements.md b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/requirements.md new file mode 100644 index 0000000..a2132da --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/requirements.md @@ -0,0 +1,20 @@ +# Requirements Document + +## Introduction + +Add subtraction to the calc package so callers have the fourth basic operation. + +## Glossary + +- **Package**: The `calc` Python package. + +## Requirements + +### Requirement 1: Subtract + +**User Story:** As a caller, I want `subtract(a, b)`, so that I can compute a minus b. + +#### Acceptance Criteria + +1. WHEN a caller calls `subtract(a, b)`, THE Package SHALL return a minus b. +2. THE Package SHALL cover `subtract` with a unit test in the existing test module. diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/tasks.md b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/tasks.md new file mode 100644 index 0000000..3e95760 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/files/.claude/specs/calc-subtract/tasks.md @@ -0,0 +1,28 @@ +# Implementation Plan + +## Overview + +Two tasks: the function, then its test. Tests run with `python3 -m unittest`. + +## Tasks + +- [ ] 1. Add `subtract(a, b)` to `calc/ops.py`, returning a minus b, in the style of `add` + - _Requirements: 1.1_ + +- [ ] 2. Add `test_subtract` to `OpsTest` in `tests/test_ops.py`, asserting subtract(5, 3) == 2 + - _Requirements: 1.2_ + +## Notes + +- Task numbers identify tasks; the graph decides the order. + +## Task Dependency Graph + +```json +{ + "waves": [ + { "id": 0, "tasks": ["1"] }, + { "id": 1, "tasks": ["2"] } + ] +} +``` diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/files/tests/test_ops.py b/evals/skills/implement-spec/offers-choice-on-discovery/files/tests/test_ops.py new file mode 100644 index 0000000..1ab7fe7 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/files/tests/test_ops.py @@ -0,0 +1,13 @@ +import os # left over + +import unittest + +from calc.ops import add, multiply + + +class OpsTest(unittest.TestCase): + def test_add(self): + self.assertEqual(add(2, 3), 5) + + def test_multiply(self): + self.assertEqual(multiply(2, 3), 6) diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/adds-subtract.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/adds-subtract.md new file mode 100644 index 0000000..cb61aba --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/adds-subtract.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'def subtract\(a, b\)' +target: { source: file, path: calc/ops.py } +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/leaves-import.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/leaves-import.md new file mode 100644 index 0000000..b854f7f --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/leaves-import.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'import os' +target: { source: file, path: tests/test_ops.py } +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/names-discovery.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/names-discovery.md new file mode 100644 index 0000000..6388521 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/names-discovery.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '## Problems((?!\n## )[\s\S])*(import os|unused import|leftover import)' +flags: i +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-issue.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-issue.md new file mode 100644 index 0000000..e5d4423 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-issue.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '## Problems((?!\n## )[\s\S])*\bissue\b' +flags: i +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-task.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-task.md new file mode 100644 index 0000000..636be55 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/offers-task.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '## Problems((?!\n## )[\s\S])*(a task|as a task|tasks\.md)' +flags: i +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/graders/states-size.md b/evals/skills/implement-spec/offers-choice-on-discovery/graders/states-size.md new file mode 100644 index 0000000..0af7001 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/graders/states-size.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '## Problems((?!\n## )[\s\S])*(line|lines|minute|minutes|small|trivial|one-liner)' +flags: i +--- diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/prompt.md b/evals/skills/implement-spec/offers-choice-on-discovery/prompt.md new file mode 100644 index 0000000..fe3da3e --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/prompt.md @@ -0,0 +1,10 @@ +--- +description: On the works-one-wave scaffold, with tests/test_ops.py committed carrying a leftover `import os` as its first line, in a file no task of the first wave changes, implement-spec works the first wave and its hand-back lists the leftover import as a discovery. Graded on the hand-back naming it under Problems with a rough size and its two options, a task in tasks.md or an issue, and on the import being left in place rather than fixed unasked. The Agent tool has no fork in a headless run, so the hand-back is written inline by the report skill. Needs the Makefile's implement-spec target, which grants Bash, Write and Edit, runs the scaffold, and puts the wrappers in bin on the PATH. +tags: [skill, implement-spec] +max_turns: 40 +allowed_tools: [Read, Glob, Grep, Agent, Skill] +expected_outcome: Claude launches the task-executor agent for task 1, which adds subtract to calc/ops.py, runs the tests, notices the unused `import os` marked as left over on the first line of tests/test_ops.py, and reports it under `unsure` without removing it. The wave's hand-back, written inline with the report skill, lists the leftover import under `## Problems` with where it is (tests/test_ops.py, line 1), roughly how much work it would add to the branch, and its two options, a proposed task in tasks.md or an issue, for the author to choose. `import os` is still in tests/test_ops.py at the end of the run. +--- +/freethinking:implement-spec calc-subtract wave + +I will not be able to answer questions during this run. Cadence: one wave. There are no optional tests. The branch is already calc-subtract. Work the first wave and hand back. The Agent tool has no fork in this run; write the hand-back inline. diff --git a/evals/skills/implement-spec/offers-choice-on-discovery/scaffold.sh b/evals/skills/implement-spec/offers-choice-on-discovery/scaffold.sh new file mode 100755 index 0000000..092fa00 --- /dev/null +++ b/evals/skills/implement-spec/offers-choice-on-discovery/scaffold.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the works-one-wave workspace, the calc +# fixture on branch calc-subtract with the two-wave spec committed, except that +# files/tests/test_ops.py replaces the fixture's copy, because setup.sh copies +# a fixture file only where the workspace has none, so the commit carries a +# leftover `import os` in a file no task of the first wave changes: the +# discovery the hand-back is graded on. The spec and the overlay are copied in +# before the setup script runs so that its one commit holds them, on main and +# on the branch. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +cp -R "$(dirname "$0")/files/." . +bash "$ROOT/evals/fixtures/setup.sh" calc calc-subtract wrappers diff --git a/evals/skills/pr/follows-repo-template/case.yaml b/evals/skills/pr/follows-repo-template/case.yaml new file mode 100644 index 0000000..c6c133f --- /dev/null +++ b/evals/skills/pr/follows-repo-template/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: follows-repo-template +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/pr/follows-repo-template/graders/creates-pr-with-gh.md b/evals/skills/pr/follows-repo-template/graders/creates-pr-with-gh.md new file mode 100644 index 0000000..4af3bc4 --- /dev/null +++ b/evals/skills/pr/follows-repo-template/graders/creates-pr-with-gh.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Bash +input_match: 'gh pr create[^\n]*--base main' +--- diff --git a/evals/skills/pr/follows-repo-template/graders/follows-template-sections.md b/evals/skills/pr/follows-repo-template/graders/follows-template-sections.md new file mode 100644 index 0000000..e81ec74 --- /dev/null +++ b/evals/skills/pr/follows-repo-template/graders/follows-template-sections.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Bash +input_match: '## Summary[\s\S]*## Test plan[\s\S]*## Reviewer notes' +--- diff --git a/evals/skills/pr/follows-repo-template/graders/no-default-why-section.md b/evals/skills/pr/follows-repo-template/graders/no-default-why-section.md new file mode 100644 index 0000000..67c315b --- /dev/null +++ b/evals/skills/pr/follows-repo-template/graders/no-default-why-section.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Bash +input_match: '## Why' +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/pr/follows-repo-template/graders/reports-pr-url.md b/evals/skills/pr/follows-repo-template/graders/reports-pr-url.md new file mode 100644 index 0000000..c33aac7 --- /dev/null +++ b/evals/skills/pr/follows-repo-template/graders/reports-pr-url.md @@ -0,0 +1,4 @@ +--- +type: regex +pattern: 'github\.com/example/demo/pull/1' +--- diff --git a/evals/skills/pr/follows-repo-template/prompt.md b/evals/skills/pr/follows-repo-template/prompt.md new file mode 100644 index 0000000..de68cd7 --- /dev/null +++ b/evals/skills/pr/follows-repo-template/prompt.md @@ -0,0 +1,10 @@ +--- +description: On a feature branch with commits, a GitHub origin, a signed-in gh (both provided by scaffold stubs) and the repository's own `.github/pull_request_template.md` on main with Summary, Test plan and Reviewer notes sections, pr drafts the description to that template rather than its default. Needs the Makefile's pr target, which grants Bash, runs the scaffold, and puts the stub bin on the PATH. +tags: [skill, pr] +max_turns: 20 +allowed_tools: [Read, Glob, Grep] +expected_outcome: Claude reads the repository's pull request template, takes the author's advance answers without waiting, and writes a PR body with the template's `## Summary`, `## Test plan` and `## Reviewer notes` sections in that order and not the skill's default `## Why` section, then calls `gh pr create` with `--base main` and a body file and reports the PR URL the stub returns. +--- +/freethinking:pr + +I am the author and I will not be able to answer questions during this run, so here are my answers in advance. They apply to every change you walk, so do not wait for me. Why was it made: the CLI needed a greeting helper, so the branch adds `greet` and its test. What it does not cover: anything beyond a fixed "Hello, name!" string; there is no localisation. What would break it: `greet` no longer returning a string. What was run: I ran `pytest tests/test_greeting.py` and it passed. What a reviewer should question: nothing in particular beyond `demo/greeting.py`. No ticket. Risk and rollout: none. Open it as a normal PR, not a draft. Treat the draft as approved and create the PR. diff --git a/evals/skills/pr/follows-repo-template/scaffold.sh b/evals/skills/pr/follows-repo-template/scaffold.sh new file mode 100755 index 0000000..5c0d5f4 --- /dev/null +++ b/evals/skills/pr/follows-repo-template/scaffold.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: a repo on a feature branch two commits +# ahead of main, a three-section pull request template committed on main, a +# local bare repo acting as "origin", and bin/ holding a fake gh plus a git +# wrapper. The Makefile puts bin on the PATH for the pr target. +set -euo pipefail +# macOS: /usr/bin/git is an xcrun shim that cannot run in the eval sandbox, so use the +# Command Line Tools binary; elsewhere, whatever git is on the PATH. +GIT=/Library/Developer/CommandLineTools/usr/bin/git; [ -x "$GIT" ] || GIT=$(command -v git) +GIT_CORE=$($GIT --exec-path) +export GIT_AUTHOR_NAME=Example GIT_AUTHOR_EMAIL=example@example.com +export GIT_COMMITTER_NAME=Example GIT_COMMITTER_EMAIL=example@example.com + +# --- fake gh: canned answers for the commands the pr skill runs +mkdir -p bin +cat > bin/gh <<'FAKE' +#!/bin/sh +case "$1 $2" in + "auth status") printf 'github.com\n ✓ Logged in to github.com account example-user (keyring)\n - Active account: true\n' ;; + "repo view") echo "main" ;; + "pr view") echo 'no pull requests found for branch "add-greeting"' >&2; exit 1 ;; + "pr list") ;; + "pr create") echo "https://github.com/example/demo/pull/1" ;; + "pr edit") echo "https://github.com/example/demo/pull/1" ;; + "issue view") printf 'title:\tAdd a greeting helper\nstate:\tOPEN\nauthor:\texample-user\nlabels:\tenhancement\nnumber:\t7\n\nThe demo CLI needs a greeting helper. It should take a name and return a friendly hello.\n--\nauthor:\texample-user\nCloses on merge.\n' ;; + *) echo "fake gh: unsupported: $*" >&2; exit 2 ;; +esac +FAKE + +# --- git wrapper: real git, made to work inside the eval sandbox. +# /usr/bin/git is an xcrun shim that needs a cache the sandbox forbids, so call +# the Command Line Tools binary directly, with an absolute PATH so git's own +# child processes find this wrapper too. "origin" is really the local bare repo; +# answer `remote get-url` with the GitHub URL it stands in for. +cat > bin/git < .git/info/exclude +mkdir -p demo tests +printf '# demo\n\nA tiny command-line tool.\n' > README.md +printf 'def main():\n print("demo")\n' > demo/cli.py +printf 'from demo import cli\n\n\ndef test_main_runs():\n cli.main()\n' > tests/test_cli.py +$GIT add -A && $GIT commit -q -m "Initial demo tool" +$GIT remote add origin "$PWD/origin.git" +$GIT config remote.origin.uploadpack "$GIT_CORE/git-upload-pack" +$GIT config remote.origin.receivepack "$GIT_CORE/git-receive-pack" +$GIT push -q origin main + +# Sections unlike the skill's default template, so the body shows which one it +# followed; committing it on main keeps it out of the branch's changes. +mkdir -p .github +cat > .github/pull_request_template.md <<'TEMPLATE' +## Summary + + + +## Test plan + + + +## Reviewer notes + + +TEMPLATE +$GIT add -A && $GIT commit -q -m "Add pull request template" +$GIT push -q origin main + +$GIT checkout -q -b add-greeting +printf 'def greet(name):\n return f"Hello, {name}!"\n' > demo/greeting.py +$GIT add -A && $GIT commit -q -m "Add greet helper" +printf 'from demo.greeting import greet\n\n\ndef test_greet():\n assert greet("Ada") == "Hello, Ada!"\n' > tests/test_greeting.py +$GIT add -A && $GIT commit -q -m "Test greet helper" diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/case.yaml b/evals/skills/pr/offers-commit-on-dirty-tree/case.yaml new file mode 100644 index 0000000..d82b51a --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: offers-commit-on-dirty-tree +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-commit.md b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-commit.md new file mode 100644 index 0000000..34ce0fa --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-commit.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Bash +input_match: 'git commit' +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-pr-create.md b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-pr-create.md new file mode 100644 index 0000000..642161a --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-pr-create.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Bash +input_match: 'gh pr create' +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-push.md b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-push.md new file mode 100644 index 0000000..4da5e96 --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/graders/no-push.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Bash +input_match: 'git push' +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-commit.md b/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-commit.md new file mode 100644 index 0000000..13db0b4 --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-commit.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Commit after the walkthrough\b' +flags: im +--- diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-leave-out.md b/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-leave-out.md new file mode 100644 index 0000000..d753cda --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/graders/offers-leave-out.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Leave them out\b' +flags: im +--- diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/prompt.md b/evals/skills/pr/offers-commit-on-dirty-tree/prompt.md new file mode 100644 index 0000000..d9618ba --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/prompt.md @@ -0,0 +1,10 @@ +--- +description: On a feature branch one commit ahead of main, with a GitHub origin and a signed-in gh (both provided by scaffold stubs) and a file modified but not committed, pr offers to commit the uncommitted change after the walkthrough, and stops at that question without committing anything. Needs the Makefile's pr target, which grants Bash, runs the scaffold, and puts the stub bin on the PATH. +tags: [skill, pr] +max_turns: 8 +allowed_tools: [Read, Glob, Grep] +expected_outcome: Claude runs the starting checks, finds the branch `add-greeting` with `demo/greeting.py` modified and uncommitted, and makes the commit offer, listing the file, with the options `Commit after the walkthrough` (Recommended) and `Leave them out`, rendered as list items in prose since the AskUserQuestion tool is absent in the sandbox. It then stops and waits. It commits nothing, does not push, and does not run `gh pr create`. +--- +/freethinking:pr + +I am the author. I will not answer any question during this run, so stop at the first question you ask and wait for me. diff --git a/evals/skills/pr/offers-commit-on-dirty-tree/scaffold.sh b/evals/skills/pr/offers-commit-on-dirty-tree/scaffold.sh new file mode 100755 index 0000000..71b88a2 --- /dev/null +++ b/evals/skills/pr/offers-commit-on-dirty-tree/scaffold.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: a repo on branch add-greeting, one commit +# ahead of main, with demo/greeting.py modified again and left uncommitted, a +# local bare repo acting as "origin", and bin/ holding a fake gh plus a git +# wrapper. The Makefile puts bin on the PATH for the pr target. +set -euo pipefail +# macOS: /usr/bin/git is an xcrun shim that cannot run in the eval sandbox, so use the +# Command Line Tools binary; elsewhere, whatever git is on the PATH. +GIT=/Library/Developer/CommandLineTools/usr/bin/git; [ -x "$GIT" ] || GIT=$(command -v git) +GIT_CORE=$($GIT --exec-path) +export GIT_AUTHOR_NAME=Example GIT_AUTHOR_EMAIL=example@example.com +export GIT_COMMITTER_NAME=Example GIT_COMMITTER_EMAIL=example@example.com + +# --- fake gh: canned answers for the commands the pr skill runs +mkdir -p bin +cat > bin/gh <<'FAKE' +#!/bin/sh +case "$1 $2" in + "auth status") printf 'github.com\n ✓ Logged in to github.com account example-user (keyring)\n - Active account: true\n' ;; + "repo view") echo "main" ;; + "pr view") echo 'no pull requests found for branch "add-greeting"' >&2; exit 1 ;; + "pr list") ;; + "pr create") echo "https://github.com/example/demo/pull/1" ;; + "pr edit") echo "https://github.com/example/demo/pull/1" ;; + "issue view") printf 'title:\tAdd a greeting helper\nstate:\tOPEN\nauthor:\texample-user\nlabels:\tenhancement\nnumber:\t7\n\nThe demo CLI needs a greeting helper. It should take a name and return a friendly hello.\n--\nauthor:\texample-user\nCloses on merge.\n' ;; + *) echo "fake gh: unsupported: $*" >&2; exit 2 ;; +esac +FAKE + +# --- git wrapper: real git, made to work inside the eval sandbox. +# /usr/bin/git is an xcrun shim that needs a cache the sandbox forbids, so call +# the Command Line Tools binary directly, with an absolute PATH so git's own +# child processes find this wrapper too. "origin" is really the local bare repo; +# answer `remote get-url` with the GitHub URL it stands in for. +cat > bin/git < .git/info/exclude +mkdir -p demo tests +printf '# demo\n\nA tiny command-line tool.\n' > README.md +printf 'def main():\n print("demo")\n' > demo/cli.py +printf 'from demo import cli\n\n\ndef test_main_runs():\n cli.main()\n' > tests/test_cli.py +$GIT add -A && $GIT commit -q -m "Initial demo tool" +$GIT remote add origin "$PWD/origin.git" +$GIT config remote.origin.uploadpack "$GIT_CORE/git-upload-pack" +$GIT config remote.origin.receivepack "$GIT_CORE/git-receive-pack" +$GIT push -q origin main + +$GIT checkout -q -b add-greeting +printf 'def greet(name):\n return f"Hello, {name}!"\n' > demo/greeting.py +$GIT add -A && $GIT commit -q -m "Add greet helper" +printf 'def greet(name):\n """Return a friendly hello for name."""\n return f"Hello, {name}!"\n' > demo/greeting.py diff --git a/evals/skills/pr/opens-pr-with-stub-gh/README.md b/evals/skills/pr/opens-pr-with-stub-gh/README.md index 92f8f79..9ed79bd 100644 --- a/evals/skills/pr/opens-pr-with-stub-gh/README.md +++ b/evals/skills/pr/opens-pr-with-stub-gh/README.md @@ -2,9 +2,10 @@ Tests the pr skill's happy path end to end: starting checks, walkthrough, draft, push, `gh pr create`. Real GitHub cannot be part of an eval, so the case fakes it. -`opens-pr-with-ticket` and `offers-branch-on-default` copy this scaffold, one -adding the pull request template on `main` and one ending on `main` with the -commits, so the fake lives in three files. +Every pr case whose run needs `gh` carries its own copy of the fake and the git +wrapper in its scaffold, because the fake has to be reached through a relative +`bin` built inside the workspace; `grep -l 'fake gh' evals/skills/pr/*/scaffold.sh` +lists them, and a change to the fake is made in each. ## What the scaffold builds diff --git a/evals/skills/start-task/names-task-and-asks/case.yaml b/evals/skills/start-task/names-task-and-asks/case.yaml new file mode 100644 index 0000000..0e9ebc8 --- /dev/null +++ b/evals/skills/start-task/names-task-and-asks/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: names-task-and-asks +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/start-task/names-task-and-asks/graders/no-spec-written.md b/evals/skills/start-task/names-task-and-asks/graders/no-spec-written.md new file mode 100644 index 0000000..7a79050 --- /dev/null +++ b/evals/skills/start-task/names-task-and-asks/graders/no-spec-written.md @@ -0,0 +1,5 @@ +--- +type: file_exists +path: '.claude/specs/**' +exists: false +--- diff --git a/evals/skills/start-task/names-task-and-asks/graders/states-spec-route.md b/evals/skills/start-task/names-task-and-asks/graders/states-spec-route.md new file mode 100644 index 0000000..1529f58 --- /dev/null +++ b/evals/skills/start-task/names-task-and-asks/graders/states-spec-route.md @@ -0,0 +1,4 @@ +--- +type: regex +pattern: 'Spec_Route' +--- diff --git a/evals/skills/start-task/names-task-and-asks/prompt.md b/evals/skills/start-task/names-task-and-asks/prompt.md index e7d8f3c..03845fa 100644 --- a/evals/skills/start-task/names-task-and-asks/prompt.md +++ b/evals/skills/start-task/names-task-and-asks/prompt.md @@ -1,7 +1,8 @@ --- -description: Invoked directly, start-task proposes a kebab-case task name and asks its first round of questions instead of writing the spec or code. +description: Given a request that extends the calc fixture, start-task proposes a kebab-case task name and a branch name, says the request takes Spec_Route, and asks its spec questions instead of writing the spec or code. Needs the Makefile's skill target, which runs the scaffold. tags: [skill, start-task] -max_turns: 4 -expected_outcome: Claude proposes `link-shortener` (or similar kebab-case name) and a branch name, says in one sentence that the request takes Spec_Route, says the spec will live under .claude/specs/, and asks the author questions. It writes nothing. +max_turns: 6 +allowed_tools: [Read, Glob, Grep, Agent, Skill] +expected_outcome: Claude reads the calc fixture, proposes `calculation-history` (or a similar kebab-case name) and a branch name, says in one sentence that the request takes Spec_Route, and asks its spec questions in prose. It writes nothing under .claude/specs/ and edits no code. --- -/freethinking:start-task Link shortener: users submit a long URL and get a short code; visiting the code redirects and counts the click. +/freethinking:start-task Keep a history of every calculation: record each operation with its operands and result as it runs, and add a way to list the history. diff --git a/evals/skills/start-task/names-task-and-asks/scaffold.sh b/evals/skills/start-task/names-task-and-asks/scaffold.sh new file mode 100755 index 0000000..d087bf2 --- /dev/null +++ b/evals/skills/start-task/names-task-and-asks/scaffold.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main, with no spec +# files, so start-task has code to read and a branch to propose. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +bash "$ROOT/evals/fixtures/setup.sh" calc main diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/names-the-task.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/names-the-task.md new file mode 100644 index 0000000..85ed3ee --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/names-the-task.md @@ -0,0 +1,4 @@ +--- +type: regex +pattern: 'link-shortener' +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/prompt.md b/evals/skills/start-task/wrong-repo-asks-fit/prompt.md index d4eca1b..1f5054b 100644 --- a/evals/skills/start-task/wrong-repo-asks-fit/prompt.md +++ b/evals/skills/start-task/wrong-repo-asks-fit/prompt.md @@ -1,8 +1,8 @@ --- -description: Given a request that extends nothing in the calc fixture, start-task says so and asks whether to spec it here or stop because this is the wrong repository, before any spec question, writing nothing. Needs the Makefile's skill target, which runs the scaffold. +description: Given a request that extends nothing in the calc fixture, start-task says so and asks whether to spec it here or stop because this is the wrong repository, before any spec question, writing nothing. The route statement with the task name comes first in the same message as the Fit question. Needs the Makefile's skill target, which runs the scaffold. tags: [skill, start-task] max_turns: 6 allowed_tools: [Read, Glob, Grep, Agent, Skill] -expected_outcome: Claude reads the calc fixture, says in two or three sentences that a link shortener extends nothing in it (no module, pattern or test it builds on), and asks the fit question with Spec it here and Wrong repository as options, before any spec question. It writes nothing under .claude/specs/, edits no code, and launches no sub-agent or skill. +expected_outcome: Claude reads the calc fixture, says in two or three sentences that a link shortener extends nothing in it (no module, pattern or test it builds on), and asks the fit question with Spec it here and Wrong repository as options, before any spec question. The route statement with the task name comes first in the same message as the Fit question. It writes nothing under .claude/specs/, edits no code, and launches no sub-agent or skill. --- /freethinking:start-task Link shortener: users submit a long URL and get a short code; visiting the code redirects and counts the click diff --git a/rules/code-changes.md b/rules/code-changes.md index ab735d4..3631f05 100644 --- a/rules/code-changes.md +++ b/rules/code-changes.md @@ -2,9 +2,9 @@ How to change code in any repository this handbook applies to. -- **Read before you write.** Look at the surrounding code, the existing patterns, the tests and the configuration the change has to fit. Write code the way the file around it is written. +- **Read before you write.** Look at the repository's README and contributing guide, then the surrounding code, the existing patterns, the tests and the configuration the change has to fit. Write code the way the file around it is written. - **Match the project.** Use the languages, libraries, naming and formatting the repository already uses. Do not add a dependency, framework or tool without asking the author first. -- **Make the smallest change that does the job.** Do not refactor unrelated code, tidy nearby files, rename for taste or add features that were not asked for. Note anything worth a follow-up instead of doing it now. +- **Make the smallest change that does the job.** Do not refactor unrelated code, tidy nearby files, rename for taste or add features that were not asked for. If you find anything else worth doing on the way, say what it is and roughly how much work it would add to the branch, and let the author choose between keeping it in the branch and filing an issue; do not make it unasked. - **Every behaviour change comes with a test.** Write it in the project's test framework, the way the project writes tests. A bug-fix test fails before the fix and passes after. If the repository has no test framework, say so and record a manual check instead; do not add a framework unless the author asks. A task from a spec's `tasks.md` is the one exception: there, tests are tasks of their own, so build only what the task says and prove it with an ephemeral check, a scratch script or call that exercises the new code and is deleted before the change is done, plus the existing tests for what you touched. - **Run what you touched.** Run the tests you added or changed, the existing tests that cover what you touched, and any ephemeral check, before saying the change is done. Report the result as it is, including failures and anything you could not run. - **Leave no investigation behind.** Remove debug output, commented-out code, temporary scripts, scratch files and stubs added while working out the change. diff --git a/rules/workflow.md b/rules/workflow.md index ac1f345..25b7450 100644 --- a/rules/workflow.md +++ b/rules/workflow.md @@ -1,13 +1,12 @@ # Workflow -These rules say which handbook skill to reach for and when. They apply in every repository this handbook applies to. +These rules say how a request finds the handbook skill that does it. They apply in every repository this handbook applies to. A skill's description says when it applies and its text says what it does; these rules say to look there first, and what not to do without the skill. -The first bullet that matches the request is the route. +- **Check the skills before doing it yourself.** The handbook ships a skill for each common kind of work, and every skill's description says when it applies. Match the request against those descriptions first, and run the skill that fits rather than doing its work by hand: no pull request with a bare `gh pr create`, no code before the bugfix skill has reproduced the bug, no spec task skipped or merged, and no judging a change's size yourself, which is start-task's job. +- **Small changes stay small.** The one exception: a typo, a one-line fix or a rename needs no skill. Say that the change is small enough to skip `start-task`, then make it. -- **Bug named: reproduce before fixing.** When the author calls the request a bug, or hands over a ticket filed as one, run `/freethinking:bugfix`. Reproduce the bug first, confirm the root cause with the author before changing code, and guard the fix with a regression test. "It works now" is not a fix. -- **Spec exists: follow it.** If `.claude/specs//tasks.md` exists for the work in hand, run `/freethinking:implement-spec`. It works the tasks in wave order; do not skip ahead or merge tasks. -- **Pull request: use the skill.** Open and update PRs with `/freethinking:pr`. It walks the changes with the author and drafts the description to the team template. Do not open a PR with a bare `gh pr create`. -- **Small changes stay small.** A typo, a one-line fix or a rename needs no skill. Say that the change is small enough to skip `start-task`, then make it. -- **Anything else that changes code: start the task.** Run `/freethinking:start-task`. It reads the code and decides between the executor for one contained change, `bugfix` for a defect, and a spec for the rest, and asks before taking the first two. Do not judge the size yourself. -- **Never work on the default branch.** Create a branch before the first commit. Do not commit or push unless the author asks. +The rest hold in every session, skill or no skill. + +- **Never work on the default branch.** Create a branch before the first commit. Do not commit or push unless the author asks; a yes to an offer a skill raised counts as asking. +- **A merged spec is a record.** `.claude/specs//` holds the task's three spec files and nothing else. Once its tasks are ticked and its branch merged, the spec is closed: it is not work in hand for implement-spec, it is not edited to match later code, and it is not read to learn how the code works today. Current behaviour lives in the code and the repository's own documentation. - **Keep the author in the loop.** When the approach changes, or a step turns out to be bigger than expected, say so before carrying on. Ask only what the code and the request cannot answer, and batch the questions. diff --git a/skills/implement-spec/SKILL.md b/skills/implement-spec/SKILL.md index 39316fa..91d64e9 100644 --- a/skills/implement-spec/SKILL.md +++ b/skills/implement-spec/SKILL.md @@ -163,7 +163,7 @@ The failing output, the finding, the proposed edit and the question are already - **A review finding that is still not fixed**: the finding quoted, then what each of the two attempts did; next, the question of how to proceed. - **A design that cannot be followed, or a new dependency the task would need**: the evidence from the code; next, a specific edit to `design.md` or `tasks.md`, quoted as it would read. Make no further code change until the author answers. -A problem found outside the current task is noted under the report's problems, with the reason, and left alone; do not fix it in this run. The one exception is a file the wave's changes made wrong, which the wave review in step 5 fixes. Never commit or push unless the author asks. +A problem found outside the current task is a discovery: it goes under the report's problems with where it is, roughly how much work it would add to the branch, and its two options, a task in `tasks.md` or an issue, and is left alone; do not fix it in this run. The one exception is a file the wave's changes made wrong, which the wave review in step 5 fixes. At spec cadence, where a wave's report does not wait, each discovery reappears in the final report of step 8 for the author to answer. Never commit or push unless the author asks. ## 7. Continue @@ -171,6 +171,8 @@ When the author says to continue, resume at step 5 with the same cadence and the When the author approves a spec edit, apply it to the named file exactly as approved, and resume where the run handed back: at the task that raised it, or at step 3 if the edit was a graph fix made before any work. +When the author answers a discovery from the report's problems: for a task, write a proposal for `tasks.md`, the task and its wave quoted as they would read, and apply it as a spec edit on the author's approval; for an issue, draft its title and body in the next message and leave filing it to the author. A discovery the author has answered is not listed in later reports. + When the author asks to skip a non-optional task or to work out of wave order, say which tasks may depend on it: the graph holds waves, not edges, so that is every task in a later wave. Proceed only after the author confirms. A skipped non-optional task stays unticked and is listed under the report's problems in every later report, and later waves start on the author's say-so despite it. ## 8. Finish diff --git a/skills/report/templates/handback.md b/skills/report/templates/handback.md index 4dbac39..05e1499 100644 --- a/skills/report/templates/handback.md +++ b/skills/report/templates/handback.md @@ -2,7 +2,7 @@ Sections are `## ` headings, in this order. With a Problem the Message opens with `## Problems`; with none, its first line is `No problems.` and `## Done` follows. -1. **Problems.** One bullet each: what it is, why it matters, the options. Never a name coined in the work. +1. **Problems.** One bullet each: what it is, why it matters, the options; for something found outside the work, roughly how much it would add to the branch. Never a name coined in the work. 2. **Done.** Each item with the check that proved it and the result seen; for a spec, ` (Requirements )`. 3. **Checks.** The full-suite result; `Wave review of wave : ` and, at spec cadence, `Final review of the run: `, each followed by `carried: see Problems` or `nothing carried`. No carried item restated. 4. **Files.** The paths changed or created, undescribed. diff --git a/skills/start-task/SKILL.md b/skills/start-task/SKILL.md index af36205..bb80b5a 100644 --- a/skills/start-task/SKILL.md +++ b/skills/start-task/SKILL.md @@ -1,6 +1,6 @@ --- name: start-task -description: Front door for any request to change code. Reads the code the request touches, then decides whether it goes to the task executor, to the bugfix skill or to a spec, and asks the author when the answer is not a spec. Start a new feature or task by writing its spec before any code. Use for any request to change code that the workflow rule does not route elsewhere (a named bug, an existing spec, a pull request, a typo-sized change), and when asked to start, plan, spec, scope or design a task, feature or piece of work, or when a request is big enough that coding straight away would be guessing. Produces requirements.md, design.md and tasks.md in .claude/specs//, following the team's spec template, with the author approving each document before the next is written. Does not write code. +description: Front door for any request to change code. Reads the code the request touches, then decides whether it goes to the task executor, to the bugfix skill or to a spec, and asks the author when the answer is not a spec. Start a new feature or task by writing its spec before any code. Use for any request to change code that no other skill takes (not a bug the author named, which is bugfix's; not an existing spec, which is implement-spec's; not a pull request, which is pr's; and not a typo-sized change, which needs no skill), and when asked to start, plan, spec, scope or design a task, feature or piece of work, or when a request is big enough that coding straight away would be guessing. Produces requirements.md, design.md and tasks.md in .claude/specs//, following the team's spec template, with the author approving each document before the next is written. Does not write code. argument-hint: "[task name or short description]" --- @@ -45,7 +45,7 @@ Before asking any spec question or writing any document, decide which route the State the route you chose and the reason for it in two or three sentences that name the code you read; on Spec_Route, one sentence that names the route is enough. -If step 2 found nothing the request would extend, no module, pattern or test it builds on, say so in the route statement, then, before any other question, put one question with the AskUserQuestion tool, header `Fit`, with two options in this order: `Spec it here`, to write the spec in this repository anyway, and `Wrong repository`, to stop, each with a one-sentence description. Act only on the author's answer. On `Wrong repository`, say where the work belongs if the request or the code says, then stop and write nothing. On `Spec it here`, go on with the route, asking the route question after this one when the route has one, since there is no point sizing work that does not belong here. +If step 2 found nothing the request would extend, no module, pattern or test it builds on, say so in the route statement, which still names the task and branch from step 1; then, before any other question and in the same message as that statement, put one question with the AskUserQuestion tool, header `Fit`, with two options in this order: `Spec it here`, to write the spec in this repository anyway, and `Wrong repository`, to stop, each with a one-sentence description. Act only on the author's answer. On `Wrong repository`, say where the work belongs if the request or the code says, then stop and write nothing. On `Spec it here`, go on with the route, asking the route question after this one when the route has one, since there is no point sizing work that does not belong here. On Bug_Route or Executor_Route, put one question to the author with the AskUserQuestion tool, header `Route`, with two options in this order: the route you chose, marked "(Recommended)", then `Spec_Route`, each with a one-sentence description. The tool adds the free-text answer itself. Act only on the author's answer. If the author picks the other route, take it without arguing for yours. If the free-text answer is neither route, restate the two routes in one sentence and ask once more.