From 8351fb9c7ca0b4306e5bbb769a7cd711f56b45bd Mon Sep 17 00:00:00 2001 From: Dan Sabin Date: Tue, 13 Oct 2026 11:35:34 -0500 Subject: [PATCH 1/3] Spec and wave 0 of standalone-task-routing: collaboration rule, workflow rule cascade The spec for issue #1: start-task triages every request after reading the code, routing a named bug to bugfix, one contained change to task-executor in standalone mode, and the rest to a spec, asking before the first two. Wave 0: rules/collaboration.md with the six defaults for working with the author; rules/workflow.md rewritten so every bullet routes on what the author said or a file on disk, first match wins, with start-task as the default; the contributing guide's routing passage brought into line. Co-Authored-By: Claude Fable 5.1 --- .../specs/standalone-task-routing/design.md | 168 ++++++++++++++++++ .../standalone-task-routing/requirements.md | 112 ++++++++++++ .../specs/standalone-task-routing/tasks.md | 90 ++++++++++ .github/CONTRIBUTING.md | 10 +- rules/collaboration.md | 10 ++ rules/workflow.md | 10 +- 6 files changed, 393 insertions(+), 7 deletions(-) create mode 100644 .claude/specs/standalone-task-routing/design.md create mode 100644 .claude/specs/standalone-task-routing/requirements.md create mode 100644 .claude/specs/standalone-task-routing/tasks.md create mode 100644 rules/collaboration.md diff --git a/.claude/specs/standalone-task-routing/design.md b/.claude/specs/standalone-task-routing/design.md new file mode 100644 index 0000000..d611ce7 --- /dev/null +++ b/.claude/specs/standalone-task-routing/design.md @@ -0,0 +1,168 @@ +# Design Document + +## Overview + +Prompt and configuration only, no code of its own. `skills/start-task/SKILL.md` gains a Triage step between reading the codebase and asking the spec questions, a CX_Block after its ground rules, and two short exit paths: invoke `bugfix` through the Skill tool, or launch the Executor through the Agent tool and report its Summary (Requirements 1 to 3). `rules/workflow.md` is rewritten so that every bullet routes on the Author's words or a file on disk, with `start-task` as the default for everything else (Requirement 4). A fourth rule file, `rules/collaboration.md`, carries the six collaboration defaults and is picked up by the existing hooks without change, since they glob `rules/*.md` (Requirement 6). README rows, the three issue-form dropdowns, two new eval cases, one new eval directory, a Makefile line and the version bump wire it in (Requirements 5 and 6). + +The one thing the design cannot settle from the docs is how `claude plugin eval` treats a call to AskUserQuestion in a headless run: whether the harness answers it, ends the turn, or errors. The eval cases are designed so the grade does not depend on the answer: they grade the Route statement and the absence of files, and accept the question in either form. Task 5.4 in the task list establishes the harness behaviour before the graders are finalised. + +## Architecture + +```mermaid +flowchart TD + A["Author: /freethinking:start-task "] --> B{"spec exists in .claude/specs//?"} + B -- yes --> S["Spec_Route: resume at first unapproved document"] + B -- no --> C["Read the code the Request touches"] + C --> T{"Triage"} + T -- "defect in existing code" --> QB["Question: Bug_Route / Spec_Route / other"] + T -- "Contained_Change" --> QE["Question: Executor_Route / Spec_Route / other"] + T -- otherwise --> S2["Say Spec_Route in one sentence"] --> S + QB -- Bug_Route --> BF["Skill tool: freethinking:bugfix "] + QB -- Spec_Route --> S + QE -- Executor_Route --> EX["Agent tool: freethinking:task-executor, Standalone_Mode"] + QE -- Spec_Route --> S + EX --> SUM{"Summary status"} + SUM -- done --> R["Show Summary and diff; failures first; recommend pr"] + SUM -- stopped --> QS["Question: answer and relaunch / Spec_Route"] + QS -- relaunch --> EX + QS -- Spec_Route --> S +``` + +```mermaid +sequenceDiagram + actor A as Author + participant S as start-task (session) + participant X as task-executor (agent) + A->>S: /freethinking:start-task Add subtract to calc + S->>S: read calc/ops.py, tests/test_ops.py + S-->>A: "Executor_Route: one function beside add, tests in test_ops.py" + S->>A: AskUserQuestion [Executor_Route, Spec_Route, other] + A-->>S: Executor_Route + S->>X: Agent(subagent_type: freethinking:task-executor, prompt: request + files found) + X-->>S: Summary (status: done, files changed, checks run) + S-->>A: Summary, git diff of files changed, "run /freethinking:pr" +``` + +Key decision, Triage lives in the skill, not the rule. The rule is read before any code is, is capped by the hook limit, and has no structured way to ask; the skill has read the code by step 2 and can put a Question (Requirements 1.1, 4.5). Trade-off: a Request that the Author already knows is a bug or a one-liner still goes through the skill unless they name it, which the kept direct routes in the rule cover (Requirement 4.2, 4.3). + +Key decision, Triage asks, except on Spec_Route. A wrong Executor or Bug call costs one Question; a wrong Spec call costs three documents. So the two cheap routes are offered and the expensive one is simply announced, which keeps today's behaviour for today's cases (Requirements 1.4, 1.5). Rejected alternative, launch the Executor without asking: faster, but the Author told us the goal is to earn trust, and the Executor's standalone mode writes code and tests the Author has not yet agreed to. + +Key decision, the Executor is launched from `start-task`, not from a new skill. The agent already exists with the right mode and Summary shape; a new skill would duplicate the launch and review text that `implement-spec` already carries (Requirement 3.1). Rejected alternative, a `/freethinking:task` skill as the human entry point: the Author wants one front door, and `@agent-freethinking:task-executor` already serves anyone who wants to bypass it. + +Key decision, Bug_Route is a hand-off, not a reimplementation. The skill invokes `bugfix` with the Request and stops, so the reproduction discipline stays in one place (Requirements 2.1, 2.2). + +Key decision, a fourth rule file rather than bullets in `workflow.md`. The collaboration defaults apply to agents as much as skills and are not about routing; a file of their own keeps `workflow.md` short and gives the free size check a single file to cap (Requirements 6.1, 6.8). The hooks need no change: `load-rules.sh` concatenates every `rules/*.md`, and the issue-form check derives its dropdown from the same glob, so the three forms gain a `collaboration (rule)` entry or `make validate` fails. + +Key decision, the Question is the AskUserQuestion tool, named in the skill. Nothing in the plugin names the tool today, which is why Claude falls back to prose. Naming it in the CX_Block and in the Triage step is the whole mechanism; the Collaboration_Rule makes it the default for the other skills' follow-ups (Requirements 1.8, 6.2). + +## Components and Interfaces + +| Component | Responsibility | Requirements | +|-----------|----------------|--------------| +| `skills/start-task/SKILL.md`, frontmatter | `description` gains the triage sentence so the skill triggers for any request to change code, not only large ones | 1.2, 4.1 | +| `skills/start-task/SKILL.md`, CX_Block | New section "Working with the author" after Ground rules: the stops, how each asks, what each hand-back holds, the hand-offs | 1.8, 1.9 | +| `skills/start-task/SKILL.md`, step 1 | Unchanged except: an existing spec means Spec_Route without Triage; beside the task name, propose the branch name in the contributing guide's shape, without creating it | 1.7, 1.10 | +| `skills/start-task/SKILL.md`, step 2 | Split in two: 2a read the codebase, 2b Triage: state the Route and reason, put the Question for Bug_Route or Executor_Route, say Spec_Route in a sentence, then act | 1.1 to 1.6 | +| `skills/start-task/SKILL.md`, step 2 Bug_Route | Skill tool, `freethinking:bugfix`, args: the Request; then stop | 2.1, 2.2 | +| `skills/start-task/SKILL.md`, step 2 Executor_Route | Agent tool, `subagent_type: freethinking:task-executor`, the Launch_Prompt below; on `done` show Summary, `git diff -- `, failures first, recommend `pr`; on `stopped` show Summary and put the relaunch Question; never edit, commit, branch | 3.1 to 3.7 | +| `skills/start-task/SKILL.md`, steps 3 to 6 | Unchanged; the spec questions move to the head of step 3 | 1.1 | +| `rules/workflow.md` | Rewritten bullets: bug named, spec exists, PR asked, trivial change, default to `start-task`, never on main, keep the author in the loop | 4.1 to 4.5 | +| `rules/collaboration.md` | The six defaults, under 1,600 characters | 6.1 to 6.8 | +| `README.md` | `start-task` row, `rules/workflow.md` row, new `rules/collaboration.md` row | 5.1, 6.9 | +| `.github/ISSUE_TEMPLATE/*.yml` (three forms) | `collaboration (rule)` added to the component dropdown after `code-changes (rule)`, the alphabetical order `checks/test_issue_forms.py` enforces | 6.1, 5.7 | +| `.claude-plugin/plugin.json` | Version `0.6.0` | 5.2 | +| `evals/skills/start-task/contained-change-offers-executor/` | Scaffolded calc fixture; prompt is a Contained_Change; graders below | 5.3 | +| `evals/skills/start-task/bug-offers-bugfix/` | Scaffolded calc fixture with a planted defect; prompt describes it; graders below | 5.4 | +| `evals/skills/start-task/names-task-and-asks/` | Prompt unchanged (the link shortener is Spec_Route); `expected_outcome` gains the one-sentence Spec_Route statement | 5.5 | +| `evals/rules/workflow/*` | `feature-routes-to-start-task` and `small-change-skips-spec` pass as they are; `bug-routes-to-bugfix` prompt names it as a bug if it does not already | 5.6 | +| `evals/rules/collaboration/short-list-asks-with-tool/` | Prompt leads to a short fixed-list question; graders accept the tool call or prose with the same options | 6.10 | +| `Makefile` | `eval-skill-start-task: CASE_ARGS += --scaffold` so the two new cases get their fixture | 5.3, 5.4 | + +## Data Models + +Launch_Prompt, the text `start-task` gives the Executor on Executor_Route. Prose, in this order, with no sentence saying the task comes from a spec: + +| Part | Content | Requirements | +|------|---------|--------------| +| Request | The Author's words, verbatim, from `$ARGUMENTS` and the message | 3.2 | +| Context found | The files and symbols step 2a read that the change touches, each as `path: what is there` | 3.2 | +| Author's additions | The free-text answer to the Question, if any, verbatim | 3.2 | +| Boundaries | "Make only this change. Commit nothing. Return the Summary." | 3.3, 3.7 | + +CX_Block, the section each skill opens with. For `start-task`: + +| Bullet | Content | +|--------|---------| +| Stops | Triage Question when the Route is Bug or Executor; approval of each of the three documents; the relaunch Question after a `stopped` Summary | +| How it asks | AskUserQuestion with fixed options plus free text at every stop; the spec questions of step 3 in prose, because the answers are the Author's own words | +| Hand-backs | On Executor_Route: the Summary verbatim, the diff, failures first; on Spec_Route: each document in full | +| Hand-offs | `bugfix` on Bug_Route; `pr` after a `done` Summary; `implement-spec` after `tasks.md` is approved. The branch name proposed in step 1 is created by the author, or by `implement-spec` at its set-up, never by this skill | + +Triage_Question, the AskUserQuestion payload: one question, header `Route`, the options in this order: the chosen Route first with "(Recommended)", `Spec_Route` second, each with a one-sentence description; the harness adds the free-text option itself. + +Rule text, `rules/workflow.md`, the routing bullets in order: + +Under the heading, one sentence: "The first bullet that matches the request is the route." The order is the cascade, cheapest wrong match first among what the rule can check. + +1. **Bug named: reproduce before fixing.** `bugfix`, keyed on the author calling it a bug or handing over a ticket filed as one. +2. **Spec exists: follow it.** `implement-spec`, as today. +3. **Pull request: use the skill.** `pr`, as today. +4. **Small changes stay small.** typo, one-line fix, rename: make it, as today. +5. **Anything else that changes code: start the task.** Run `/freethinking:start-task`. It reads the code and decides between the executor for one contained change, `bugfix` for a defect, and a spec for the rest, and asks before taking the first two. Do not judge the size yourself. +6. **Never work on the default branch.** As today. +7. **Keep the author in the loop.** As today. + +Rule text, `rules/collaboration.md`, one bullet per criterion 6.2 to 6.7, each bold-led in the voice of the other rules, headed "Working with the author". + +## Correctness Properties + +### Property 1: No document before the answer + +*For any* run of `start-task` where Triage chooses Bug_Route or Executor_Route, no file under `.claude/specs/` and no source file is created or edited before the Author answers the Question. + +**Validates: Requirements 1.1, 1.4, 2.2, 3.3** + +### Property 2: The Author's answer is the Route + +*For any* answer to the Triage Question, the Route the skill takes next is the one the answer names, whatever the skill recommended. + +**Validates: Requirements 1.4, 1.6** + +### Property 3: Standalone launch + +*For any* Executor launch from `start-task`, the Launch_Prompt contains the Request verbatim and does not contain a sentence saying the task comes from a spec. + +**Validates: Requirements 3.1, 3.2** + +### Property 4: Rules fit the hook + +*For any* state of `rules/`, the concatenated output of `load-rules.sh` is under 10,000 characters and `rules/collaboration.md` is under 1,600. + +**Validates: Requirements 4.4, 6.8** + +### Property 5: Forms match disk + +*For any* set of files under `rules/`, every issue form's component dropdown lists each as ` (rule)`, in order. + +**Validates: Requirement 6.1** + +## Error Handling + +| Condition | Category | Behaviour | Requirements | +|-----------|----------|-----------|--------------| +| Spec directory already exists for the task name | RESUME | Spec_Route, no Triage, resume at first unapproved document | 1.7 | +| Author picks a different Route than recommended | OVERRIDE | Take the Author's Route, no argument | 1.6 | +| Author's free-text answer is neither Route | UNCLEAR | Restate the two Routes in one sentence and ask again, once | 1.4 | +| Executor returns `stopped` | DRIFT | Show Summary; Question: answer and relaunch, or Spec_Route; no unprompted relaunch | 3.5 | +| Executor returns `done` with a failed check | FAILED_CHECK | Say the failure first, show the Summary and diff, then recommend `pr` with the failure named | 3.6 | +| AskUserQuestion is not available (headless run) | FALLBACK | Ask the same question in prose with the same options on separate lines, then wait | 6.2, 6.10 | +| `bugfix` or `task-executor` not found | MISSING | Say the plugin component is missing and stop; do not reimplement | 2.1, 3.1 | + +## Testing Strategy + +- Free checks, run by `make validate`: the existing `checks/test_issue_forms.py` proves Property 5 once `collaboration (rule)` is in the forms; a new check in `checks/test_rules_size.py` proves Property 4 by running `hooks/load-rules.sh` and measuring, and by measuring `rules/collaboration.md` alone. The size check also closes issue #3's first half. `claude plugin validate --strict` proves the frontmatter. +- Eval case `contained-change-offers-executor` (Property 1, 3; Requirement 5.3): scaffold the calc fixture; prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py in the style of add`; `allowed_tools: [Read, Glob, Grep, AskUserQuestion, Agent, Skill]`; `max_turns: 6`. Graders: regex `Executor_Route|executor` in the final output; `file_exists` `.claude/specs/**` false; `tool_used` Agent with max 0; `tool_used` Skill with max 0; and the question grader, `tool_used` AskUserQuestion min 1, or, if task 5.4 finds the harness cannot carry that tool, a regex for the two options on separate lines. +- Eval case `bug-offers-bugfix` (Property 1; Requirement 5.4): same fixture with `add` returning `a - b`; prompt `/freethinking:start-task Adding 2 and 3 in calc returns -1`; graders: regex `Bug_Route|bugfix`; `file_exists` `**/*.py` unchanged by a regex on `calc/ops.py` still containing the planted body; `tool_used` Skill max 0; the same question grader. +- Eval case `short-list-asks-with-tool` under `evals/rules/collaboration/` (Requirement 6.10): no plugin skill invoked; prompt asks Claude to pick between three named options for a config value it cannot infer; grader as the question grader above. +- Existing cases: `names-task-and-asks` run twice; `eval-rule-workflow` run twice; `eval-skill-start-task` twice. Each result table goes into the PR's Testing section. +- Manual check, recorded in the PR: in a session started with `claude --plugin-dir .` against the calc fixture, run the three prompts above by hand and confirm the Question renders as the interactive form and that answering Spec_Route continues to the spec questions (Property 2). diff --git a/.claude/specs/standalone-task-routing/requirements.md b/.claude/specs/standalone-task-routing/requirements.md new file mode 100644 index 0000000..aeaf7bd --- /dev/null +++ b/.claude/specs/standalone-task-routing/requirements.md @@ -0,0 +1,112 @@ +# Requirements Document + +## Introduction + +`start-task` is the handbook's front door for a piece of work, but today it has one exit: three approved spec documents. The only other route is the workflow rule's bullet that lets a typo, a one-line fix or a rename skip the skill, and that decision is made outside the skill, by whoever reads the rule. Work in the middle, one contained change that is too big to just type but too small to design, gets a full spec it does not need, while the `task-executor` agent, which was built for exactly that size of work, is reachable only from `implement-spec`. This change makes `start-task` triage the request after it has read the codebase: a bug goes to `bugfix`, a single contained change goes to `task-executor`, and only work that needs design goes on to the spec. The engineer starts every piece of work the same way and is asked, not told, when the skill reaches a sizing call. The workflow rule changes to match, and the plugin version moves to 0.6.0. + +## Glossary + +- **Skill**: The `start-task` skill described by this document, run as `/freethinking:start-task`. +- **Author**: The engineer whose session runs the Skill and who is accountable for the resulting code. +- **Request**: The work as the Author gave it: the Skill's argument and the Author's surrounding message. +- **Triage**: The Skill's judgement, made after reading the codebase and before writing any document, of which Route the Request takes. +- **Route**: One of the three outcomes of Triage: Bug_Route, Executor_Route or Spec_Route. +- **Bug_Route**: The Route for a Request that describes a defect: observed behaviour that differs from intended behaviour in code that already exists. +- **Executor_Route**: The Route for a Contained_Change: the Skill launches the Executor in Standalone_Mode with the Request and reports its Summary. +- **Spec_Route**: The Route for every other Request: the Skill writes the three spec documents as it does today. +- **Contained_Change**: A Request that is one change, describable in a sentence or two, touching a place in the code the Skill can name after reading the codebase, needing no design decision and no new dependency, framework or tool. +- **Executor**: The `task-executor` plugin agent, addressed as `freethinking:task-executor`. +- **Standalone_Mode**: The Executor's behaviour when its launch prompt does not say the task comes from a spec: the change comes with its test. +- **Summary**: The fixed-shape report the Executor returns: status, files changed, checks run, ephemeral check, things unsure of, and a reason when stopped. +- **Question**: A question the Skill puts to the Author with the AskUserQuestion tool, offering a fixed set of answers plus the Author's own text. +- **Rule**: The handbook workflow rule in `rules/workflow.md`, printed into every session by the plugin's hooks. +- **Collaboration_Rule**: A new handbook rule in `rules/collaboration.md` that sets the defaults for how any skill or agent works with the Author: how it asks, when it batches, when it says its plan, what approval means, how it reports and how it hands off. +- **CX_Block**: A short section near the top of a skill that lists the skill's stops for the Author and states where it departs from the Collaboration_Rule. +- **Eval_Case**: A case under `evals/` that runs the plugin against a prompt and grades the transcript, run by hand with `make eval-*`. + +## Requirements + +### Requirement 1: Triage every Request + +**User Story:** As an Author, I want to start every piece of work with the same skill and have it decide how big the work is, so that I do not sort work myself and small work does not get a spec it does not need. + +#### Acceptance Criteria + +1. WHEN the Skill has read the parts of the codebase the Request touches, THE Skill SHALL perform Triage before asking the Author any of the spec questions and before writing any document. +2. THE Skill SHALL choose Bug_Route WHEN the Request describes a defect, Executor_Route WHEN the Request is a Contained_Change, and Spec_Route otherwise. +3. THE Skill SHALL state its chosen Route and the reason for it, in two or three sentences that name the code it read, before acting on it. +4. WHEN the Route is Bug_Route or Executor_Route, THE Skill SHALL put a Question to the Author offering that Route, Spec_Route and a free-text answer, and SHALL act only on the Author's answer. +5. WHEN the Route is Spec_Route, THE Skill SHALL say so in one sentence and continue to the spec questions without a Question, since that is today's behaviour. +6. IF the Author's answer to the Question changes the Route, THEN THE Skill SHALL take the Route the Author chose without arguing for its own. +7. WHEN a spec for the Request already exists in `.claude/specs//`, THE Skill SHALL take Spec_Route and resume at the first unapproved document, without Triage, since the sizing decision was already made. +8. THE Skill SHALL open with a CX_Block, placed after its ground rules, that lists every point at which it stops for the Author (the Question at Triage, then approval of each document), says that each stop asks with the AskUserQuestion tool except where the answer is the Author's own words, says what each hand-back contains, and names the skills it hands off to. +9. THE CX_Block SHALL state only what is specific to the Skill or departs from the Collaboration_Rule, and SHALL NOT restate the Collaboration_Rule. +10. WHEN the Skill derives the task name, THE Skill SHALL propose a branch name in the contributing guide's shape, the issue number and the task name when the Request names or links an issue and the task name alone otherwise, and SHALL NOT create the branch. + +### Requirement 2: Bug_Route + +**User Story:** As an Author, I want a bug I bring to `start-task` to be handled the bug way, so that it gets reproduced before it is fixed rather than specified. + +#### Acceptance Criteria + +1. WHEN the Author accepts Bug_Route, THE Skill SHALL invoke the `bugfix` skill with the Request as its argument and SHALL do nothing else. +2. THE Skill SHALL NOT reproduce, diagnose or fix the defect itself. + +### Requirement 3: Executor_Route + +**User Story:** As an Author, I want a contained change built by the executor straight away, so that the handbook's disciplined executor does the work without a spec in between. + +#### Acceptance Criteria + +1. WHEN the Author accepts Executor_Route, THE Skill SHALL launch the Executor once with the Agent tool as `subagent_type: freethinking:task-executor`. +2. THE launch prompt SHALL hold the Request in the Author's words, the files and code the Skill found that the change touches, and anything the Author added in the Question's free text, and SHALL NOT say the task comes from a spec, so that the Executor is in Standalone_Mode. +3. THE Skill SHALL NOT edit, create or delete any source file itself on Executor_Route. +4. WHEN the Executor returns a Summary with status `done`, THE Skill SHALL show the Author the Summary and the diff of the files it names, and SHALL recommend opening a pull request with the `pr` skill. +5. WHEN the Executor returns a Summary with status `stopped`, THE Skill SHALL show the Author the Summary and put a Question offering to answer the Executor's question and relaunch, or to take Spec_Route, and SHALL NOT relaunch unprompted. +6. IF the Executor's Summary reports a check that failed, THEN THE Skill SHALL say so plainly before recommending anything. +7. THE Skill SHALL NOT commit, push or create a branch on Executor_Route. + +### Requirement 4: The Rule sends all work through the Skill + +**User Story:** As an Author, I want the workflow rule to agree with the skill, so that Claude reaches for `start-task` first and does not sort work by size on its own. + +#### Acceptance Criteria + +1. THE Rule SHALL direct Claude to run `/freethinking:start-task` for any request to change code that no other bullet of the Rule names, and SHALL say that the skill reads the code and decides between the executor, `bugfix` and a spec. +5. THE Rule SHALL NOT ask Claude to judge the size of a request beyond the trivial cases of criterion 3; every other bullet SHALL route on what the Author said or on a file that exists. +6. THE Rule SHALL say that the first bullet that matches the request is the route, so that a request matching two bullets has one answer. +2. THE Rule SHALL keep direct routes to `/freethinking:bugfix` for a request the Author names as a bug, to `/freethinking:implement-spec` when a `tasks.md` exists, and to `/freethinking:pr` for a pull request. +3. THE Rule SHALL keep the "small changes stay small" bullet for a typo, a one-line fix or a rename, so that trivial work still needs no skill. +4. THE combined text of `rules/*.md` SHALL stay under 10,000 characters, since Claude Code truncates hook output past that. + +### Requirement 5: Documentation, version and evals + +**User Story:** As a contributor, I want the README, the version and the evals to describe the skill that now ships, so that the plugin stays honest about what it does. + +#### Acceptance Criteria + +1. THE README's row for `start-task` and its row for `rules/workflow.md` SHALL describe Triage and its three Routes in the voice of the other rows. +2. THE plugin version in `.claude-plugin/plugin.json` SHALL be `0.6.0`. +3. THE plugin SHALL include an Eval_Case under `evals/skills/start-task/` whose prompt is a Contained_Change against a fixture, and which passes when the transcript shows the Skill stating Executor_Route, asking the Question, and writing no spec document before the answer. +4. THE plugin SHALL include an Eval_Case under `evals/skills/start-task/` whose prompt is a defect, and which passes when the transcript shows the Skill stating Bug_Route and asking the Question without reproducing or fixing anything. +5. THE existing Eval_Case `evals/skills/start-task/names-task-and-asks` SHALL still pass, with its prompt sized so that Triage chooses Spec_Route. +6. THE existing Eval_Cases under `evals/rules/workflow/` SHALL still pass after the Rule changes, with their prompts or graders adjusted only where the Rule's new wording requires it. +7. THE `make validate` target SHALL pass. + +### Requirement 6: The Collaboration_Rule + +**User Story:** As an Author, I want every skill and agent to work with me the same way by default, so that I know how I will be asked, when I will be stopped and what I will be shown, whichever skill is running. + +#### Acceptance Criteria + +1. THE plugin SHALL ship `rules/collaboration.md`, loaded by the same hooks as the other rules, so that it reaches every session and every sub-agent. +2. THE Collaboration_Rule SHALL say to ask with the AskUserQuestion tool when the answers are a short fixed list, offering the options plus free text, and to ask in prose when the answer is the Author's own words. +3. THE Collaboration_Rule SHALL say to batch questions into one round unless an answer decides the next question, in which case to ask one at a time. +4. THE Collaboration_Rule SHALL say to state the plan in two or three sentences before acting on it. +5. THE Collaboration_Rule SHALL say that approval is explicit and that "looks good" with a "but" is a change request. +6. THE Collaboration_Rule SHALL say to report failures and anything not run before any recommendation. +7. THE Collaboration_Rule SHALL say that a skill which ends hands off by naming where the work is and which skill is next. +8. THE Collaboration_Rule SHALL be under 1,600 characters, and the combined text of `rules/*.md` SHALL stay under 10,000 characters. +9. THE README's rules table SHALL gain a row for `rules/collaboration.md` in the voice of the other rows. +10. THE plugin SHALL include an Eval_Case under `evals/rules/collaboration/` whose prompt leads to a short-list question, and which passes when the transcript shows the question asked with the AskUserQuestion tool or, where the eval runs without that tool, in prose that offers the same fixed options. +11. THE `implement-spec`, `bugfix` and `pr` skills SHALL NOT change in this task; their CX_Blocks are issues #16, #17 and #18. diff --git a/.claude/specs/standalone-task-routing/tasks.md b/.claude/specs/standalone-task-routing/tasks.md new file mode 100644 index 0000000..4ce5c58 --- /dev/null +++ b/.claude/specs/standalone-task-routing/tasks.md @@ -0,0 +1,90 @@ +# Implementation Plan + +## Overview + +Rules first, because the skill's CX block refers to the collaboration rule and the free checks cap the rules' size. Then the skill change, then the eval cases and the probe of how the eval harness treats an interactive question, then README, forms, version and the runs. The handbook has no code and no unit-test framework beyond the stdlib `unittest` files under `checks/` that `make validate` runs; one such check is added here. Behaviour is proved by eval cases, run by hand at the task that says so, and by one manual session the author runs. Nothing here touches `implement-spec`, `bugfix` or `pr`, whose CX blocks are issues #16, #17 and #18. + +## Tasks + +- [ ] 1. The rules + - [x] 1.1 Write `rules/collaboration.md` + - Heading "Working with the author", a one-line scope sentence, then six bold-led bullets in the voice of `rules/code-changes.md`: ask with the AskUserQuestion tool when the answers are a short fixed list, offering the options plus free text, and in prose when the answer is the author's own words; batch questions into one round unless an answer decides the next question; state the plan in two or three sentences before acting; approval is explicit and "looks good" with a "but" is a change request; report failures and anything not run before any recommendation; a skill that ends says where the work is and which skill is next. Under 1,600 characters. Prove it with `wc -c` + - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8_ + - [x] 1.2 Rewrite the routing bullets of `rules/workflow.md` + - Replace the "New feature or sizeable task" bullet with the design's bullet 5, "Anything else that changes code: start the task", placed after "Small changes stay small"; retitle the bug bullet "Bug named: reproduce before fixing"; leave the spec, PR, default-branch and keep-in-the-loop bullets as they are. No bullet other than "Small changes stay small" asks Claude to judge size. Prove it by reading the file back and by `sh hooks/load-rules.sh | wc -c` + - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_ + - Wave 1 fix, agreed at the wave 0 hand-back: add "The first bullet that matches the request is the route." under the rule's heading, and a clause in `.github/CONTRIBUTING.md`'s "Make a change" passage saying `start-task` proposes the branch name on every route + - [ ] 1.3 Add `checks/test_rules_size.py` + - A stdlib `unittest` file in the style of `checks/test_issue_forms.py`: one test runs `hooks/load-rules.sh` with `CLAUDE_PLUGIN_ROOT` set to the repository root and asserts its output is under 10,000 characters; one asserts `rules/collaboration.md` is under 1,600. Failure messages give the measured count. Run `make check` + - **Property 4: Rules fit the hook** + - _Requirements: 4.4, 6.8_ + - [ ] 1.4 Add `collaboration (rule)` to the three issue forms + - In `.github/ISSUE_TEMPLATE/eval-case.yml`, `misbehaving-skill-or-agent.yml` and `proposal.yml`, insert `collaboration (rule)` into the component dropdown directly after `code-changes (rule)`, since the check sorts rule stems alphabetically. Run `make check` + - **Property 5: Forms match disk** + - _Requirements: 6.1_ + +- [ ] 2. The skill + - [ ] 2.1 Add the CX block and the triage step to `skills/start-task/SKILL.md` + - Frontmatter `description`: add that the skill is the front door for any request to change code and decides between the executor, bugfix and a spec. After "Ground rules", a section "Working with the author" with the design's four bullets: stops, how it asks (AskUserQuestion with fixed options at every stop; the step 3 spec questions in prose), hand-backs, hand-offs; state only what is specific to this skill. In step 1, say that an existing spec directory means Spec_Route with no triage, and that beside the task name the skill proposes a branch name in the contributing guide's shape (issue number and task name when the request names or links an issue, task name alone otherwise) without creating it; the CX block's hand-offs bullet says the author or `implement-spec` creates it. Split step 2 into "2. Read the code" (today's first paragraph) and "3. Triage" with: the three routes and the Contained_Change and defect definitions from the requirements glossary; state the chosen route and the reason in two or three sentences naming the code read; on Bug_Route or Executor_Route put one AskUserQuestion, header `Route`, chosen route first marked "(Recommended)", `Spec_Route` second, each with a one-sentence description, and act only on the answer; take the author's route without arguing; on Spec_Route say so in one sentence and continue; if the free-text answer is neither route, restate the two in a sentence and ask once more. Renumber the later steps and the spec questions move to the head of the requirements step. Run `make validate` + - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10_ + - [ ] 2.2 Write the Bug_Route and Executor_Route paths in the triage step + - Bug_Route: invoke the Skill tool with `freethinking:bugfix` and the request as its argument, then stop; do not reproduce, diagnose or fix. Executor_Route: launch the Agent tool once with `subagent_type: freethinking:task-executor` and the design's Launch_Prompt (request verbatim, context found as `path: what is there`, the author's free-text additions, then "Make only this change. Commit nothing. Return the Summary."), with no sentence saying the task comes from a spec; edit no source file, commit nothing, create no branch. On `status: done`: show the Summary verbatim, run `git diff -- ` and show it, name any failed check before anything else, then recommend `/freethinking:pr`. On `status: stopped`: show the Summary and put one AskUserQuestion offering "answer and relaunch" and "Spec_Route"; never relaunch unprompted. If either plugin component is missing, say so and stop. Run `make validate` + - _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7_ + +- [ ] 3. The eval cases + - [ ] 3.1 Probe how the eval harness treats AskUserQuestion + - In the scratchpad, make a one-case plugin whose only skill puts an AskUserQuestion, run `claude plugin eval` on it with `allowed_tools` including `AskUserQuestion` and `max_turns: 3`, and record in the task's result: whether the tool is callable headless, whether the harness answers it or ends the turn, whether a `tool_used` grader with `tool: AskUserQuestion` counts it, and whether the transcript a `regex` grader sees contains the question's option labels. Delete the scratch plugin. The answer decides the question grader's shape in 3.2, 3.3 and 3.5 + - _Requirements: 5.3, 5.4, 6.10_ + - [ ] 3.2 Eval case `evals/skills/start-task/contained-change-offers-executor/` + - `case.yaml` with `scaffold_script: scaffold.sh` copying `evals/fixtures/calc` into the workspace, as the implement-spec and task-executor cases do; `prompt.md` with tags `[skill, start-task]`, `max_turns: 6`, `allowed_tools: [Read, Glob, Grep, AskUserQuestion, Agent, Skill]`, prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add`; graders: regex `(?i)executor` on the output; `file_exists` `.claude/specs/**` false; `tool_used` Agent max 0; `tool_used` Skill max 0; the question grader in the shape 3.1 found. Add `eval-skill-start-task: CASE_ARGS += --scaffold` to the Makefile's per-target block + - **Property 1: No document before the answer** + - **Property 3: Standalone launch** + - _Requirements: 5.3_ + - [ ] 3.3 Eval case `evals/skills/start-task/bug-offers-bugfix/` + - Same scaffold, but the script then rewrites `add` in `calc/ops.py` to return `a - b`; prompt `/freethinking:start-task calc.add(2, 3) returns -1; it should return 5`; graders: regex `(?i)bug`; regex `return a - b` on `calc/ops.py` (the defect is still there); `tool_used` Skill max 0; `file_exists` `.claude/specs/**` false; the question grader + - **Property 1: No document before the answer** + - _Requirements: 5.4_ + - [ ] 3.4 Keep `evals/skills/start-task/names-task-and-asks/` on Spec_Route + - Prompt unchanged; `expected_outcome` says the skill states Spec_Route in one sentence before proposing the task name and a branch name and asking its questions; `allowed_tools` adds `AskUserQuestion` if 3.1 found it callable. Check `evals/rules/workflow/bug-routes-to-bugfix/prompt.md` names the request as a bug in the author's words, and change the prompt only if it does not + - _Requirements: 5.5, 5.6_ + - [ ] 3.5 Eval case `evals/rules/collaboration/short-list-asks-with-tool/` + - `prompt.md` with tags `[rule, collaboration]`, `max_turns: 4`, `allowed_tools: [Read, Glob, Grep, AskUserQuestion]`, a prompt that asks Claude to set a log level in a config file to one of three named levels without saying which, so it must ask; graders: the question grader, plus a regex that the three level names appear. The new `eval-rule-collaboration` target comes from the Makefile's directory enumeration and needs no Makefile change + - _Requirements: 6.10_ + +- [ ] 4. Docs and version + - [ ] 4.1 README rows + - Skills table, `start-task` row: it is the front door for any change, reads the code, and decides between the executor for one contained change, `bugfix` for a defect and a spec for the rest, asking before the first two. Rules table, `rules/workflow.md` row: route on what the author said or a file on disk, default to `start-task`. New row for `rules/collaboration.md` after the `comments` row, in the voice of the others. Nothing else in the README changes + - _Requirements: 5.1, 6.9_ + - [ ] 4.2 Version `0.6.0` + - `.claude-plugin/plugin.json` version to `0.6.0`. Run `make validate` + - _Requirements: 5.2_ + +- [ ] 5. Runs + - [ ] 5.1 Run the evals + - `make eval-skill-start-task EVAL_ARGS="--runs 2"`, `make eval-rule-workflow EVAL_ARGS="--runs 2"`, `make eval-rule-collaboration EVAL_ARGS="--runs 2"`. Fix the case or the skill until each case passes both runs, and record the three result tables for the PR's Testing section + - _Requirements: 5.3, 5.4, 5.5, 5.6, 6.10_ + - [ ] 5.2 Manual session, done by the author + - In a session started with `claude --plugin-dir .` in a scratch copy of `evals/fixtures/calc`, run the contained-change prompt, the bug prompt and the link-shortener prompt by hand. Confirm the Route question renders as the interactive form, that answering Spec_Route continues to the spec questions, and that accepting Executor_Route ends with the Summary, the diff and the `pr` recommendation. Record what was seen in the PR + - **Property 2: The Author's answer is the Route** + - _Requirements: 1.4, 1.6, 3.4, 5.7_ + +## Notes + +- Every file here is prose or YAML; "the project's style" means the voice and shape of the neighbouring file, which the executor reads first. +- `make validate` is the free check and runs after every task that touches a manifest, a form, a check or a skill; evals run only at task 5.1. +- The question grader's exact shape is unknown until task 3.1 reports; tasks 3.2, 3.3 and 3.5 take it from that report. +- Issue #1 is the ticket for this work; the branch is `1-standalone-task-routing` and the PR closes #1. + +## Task Dependency Graph + +```json +{ + "waves": [ + { "id": 0, "tasks": ["1.1", "1.2", "3.1"] }, + { "id": 1, "tasks": ["1.3", "1.4", "2.1", "3.4", "3.5", "4.2"] }, + { "id": 2, "tasks": ["2.2", "3.2", "3.3", "4.1"] }, + { "id": 3, "tasks": ["5.1"] }, + { "id": 4, "tasks": ["5.2"] } + ] +} +``` diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index d787c7d..cb5a1a4 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -39,9 +39,13 @@ For how the checks and evals work, see the README's ## Make a change -A small change, such as a typo or a one-line fix: make it. A sizeable change: -run `/freethinking:start-task` to write the spec, then -`/freethinking:implement-spec` to build it. A bug: run `/freethinking:bugfix`. +A small change, such as a typo, a one-line fix or a rename: make it. A bug: +run `/freethinking:bugfix`. Anything else: run `/freethinking:start-task`. It +reads the code and decides between the `task-executor` agent for one contained +change, `bugfix` for a defect, and a spec for the rest, and asks before taking +the first two. When it writes a spec, `/freethinking:implement-spec` builds it. +On every route, `start-task` proposes the branch name, and you or +`implement-spec` create it. Work on a branch named `-` when an issue exists, such as `12-ticket-reference`, and on a short slug otherwise. Never work on the diff --git a/rules/collaboration.md b/rules/collaboration.md new file mode 100644 index 0000000..96ded7a --- /dev/null +++ b/rules/collaboration.md @@ -0,0 +1,10 @@ +# Working with the author + +How any skill or agent works with the author in every repository this handbook applies to. + +- **Ask in the shape of the answer.** When the answers are a short fixed list, ask with the AskUserQuestion tool, offering the options plus free text. When the answer is the author's own words, ask in prose. +- **Batch the questions.** Ask everything you need in one round. Ask one at a time only when an answer decides the next question. +- **Say the plan before acting.** State the plan in two or three sentences before carrying it out, so the author can stop a wrong turn early. +- **Approval is explicit.** Where a step waits for approval, approval is a plain yes. "Looks good" with a "but" is a change request, not an approval. +- **Bad news first.** Report failures and anything you did not run before any recommendation. +- **Hand off when a skill ends.** Say where the work is and which skill is next. diff --git a/rules/workflow.md b/rules/workflow.md index 4016c8c..d73fae9 100644 --- a/rules/workflow.md +++ b/rules/workflow.md @@ -2,10 +2,12 @@ These rules say which handbook skill to reach for and when. They apply in every repository this handbook applies to. -- **New feature or sizeable task: spec first.** Before writing code for a new feature, or for any request large enough that coding straight away would be guessing, run `/freethinking:start-task`. It writes `requirements.md`, `design.md` and `tasks.md` in `.claude/specs//`, each approved by the author before the next. -- **Existing spec: follow it.** If `.claude/specs//tasks.md` exists for the work in hand, run `/freethinking:implement-spec`. It works the tasks in wave order, ticks a task's checkbox when its tests pass, and stops for review at the author's cadence, after each task or after each wave. Do not skip ahead or merge tasks. -- **Bug: reproduce before fixing.** For a bug, issue or defect, run `/freethinking:bugfix`. Reproduce the bug first, confirm the root cause with the author before changing code, and guard the fix with a regression test. "It works now" is not a fix. +The first bullet that matches the request is the route. + +- **Bug named: reproduce before fixing.** When the author calls the request a bug, or hands over a ticket filed as one, run `/freethinking:bugfix`. Reproduce the bug first, confirm the root cause with the author before changing code, and guard the fix with a regression test. "It works now" is not a fix. +- **Spec exists: follow it.** If `.claude/specs//tasks.md` exists for the work in hand, run `/freethinking:implement-spec`. It works the tasks in wave order, ticks a task's checkbox when its tests pass, and stops for review at the author's cadence, after each task or after each wave. Do not skip ahead or merge tasks. - **Pull request: use the skill.** Open and update PRs with `/freethinking:pr`. It walks the changes with the author and drafts the description to the team template. Do not open a PR with a bare `gh pr create`. +- **Small changes stay small.** A typo, a one-line fix or a rename needs no skill. Say that the change is small enough to skip `start-task`, then make it. +- **Anything else that changes code: start the task.** Run `/freethinking:start-task`. It reads the code and decides between the executor for one contained change, `bugfix` for a defect, and a spec for the rest, and asks before taking the first two. Do not judge the size yourself. - **Never work on the default branch.** Create a branch before the first commit. Do not commit or push unless the author asks. -- **Small changes stay small.** A typo, a one-line fix or a rename does not need a spec. Say that the change is small enough to skip the skill, then make it. - **Keep the author in the loop.** When the approach changes, or a step turns out to be bigger than expected, say so before carrying on. Ask only what the code and the request cannot answer, and batch the questions. From d07edd99990c6ef47ba64b26dbe7124579d9a030 Mon Sep 17 00:00:00 2001 From: Dan Sabin Date: Tue, 13 Oct 2026 12:40:02 -0500 Subject: [PATCH 2/3] start-task triages every request: executor, bugfix or spec Waves 1 to 3 of the standalone-task-routing spec (issue #1). start-task is now the front door for a change. After reading the code it states a route and, for a contained change or a defect, asks with the AskUserQuestion tool before launching task-executor in standalone mode or handing off to bugfix; everything else goes to the spec as before. Its "Working with the author" block lists its stops, how each asks, what each hand-back shows and what it hands off to, and step 1 proposes a branch name without creating it. rules/collaboration.md gains the headless fallback (prose with the options on separate lines). The README rows, the executor's description, the three issue forms and the contributing guide follow. A free check caps the rules' size; the start-task eval target may write so its no-write graders can fail. Version 0.6.0. Evals, two runs each, all 100%: the three start-task cases, the five workflow rule cases and the new collaboration rule case. Co-Authored-By: Claude Fable 5.1 --- .claude-plugin/plugin.json | 2 +- .../specs/standalone-task-routing/design.md | 6 +- .../standalone-task-routing/requirements.md | 4 + .../specs/standalone-task-routing/tasks.md | 41 ++++----- .github/ISSUE_TEMPLATE/eval-case.yml | 1 + .../misbehaving-skill-or-agent.yml | 1 + .github/ISSUE_TEMPLATE/proposal.yml | 1 + Makefile | 1 + README.md | 23 +++-- agents/task-executor.md | 2 +- checks/test_rules_size.py | 41 +++++++++ .../graders/asks-a-question.md | 4 + .../graders/does-not-choose.md | 6 ++ .../graders/offers-debug.md | 5 ++ .../graders/offers-info.md | 5 ++ .../graders/offers-warn.md | 5 ++ .../graders/sets-nothing.md | 5 ++ .../short-list-asks-with-tool/prompt.md | 8 ++ .../workflow/bug-routes-to-bugfix/prompt.md | 2 +- .../start-task/bug-offers-bugfix/case.yaml | 4 + .../graders/defect-still-present.md | 5 ++ .../bug-offers-bugfix/graders/names-bug.md | 5 ++ .../bug-offers-bugfix/graders/no-agent.md | 7 ++ .../bug-offers-bugfix/graders/no-skill.md | 7 ++ .../graders/no-spec-written.md | 5 ++ .../graders/offers-bug-route.md | 5 ++ .../graders/offers-spec-route.md | 5 ++ .../start-task/bug-offers-bugfix/prompt.md | 8 ++ .../start-task/bug-offers-bugfix/scaffold.sh | 10 +++ .../case.yaml | 4 + .../graders/no-skill.md | 7 ++ .../graders/no-spec-written.md | 5 ++ .../graders/no-sub-agent.md | 7 ++ .../graders/offers-executor-route.md | 5 ++ .../graders/offers-spec-route.md | 5 ++ .../graders/states-executor.md | 5 ++ .../prompt.md | 8 ++ .../scaffold.sh | 6 ++ .../start-task/names-task-and-asks/prompt.md | 2 +- rules/collaboration.md | 2 +- skills/start-task/SKILL.md | 85 +++++++++++++++---- 41 files changed, 311 insertions(+), 54 deletions(-) create mode 100644 checks/test_rules_size.py create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/asks-a-question.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/does-not-choose.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/offers-debug.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/offers-info.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/offers-warn.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/graders/sets-nothing.md create mode 100644 evals/rules/collaboration/short-list-asks-with-tool/prompt.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/case.yaml create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/defect-still-present.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/names-bug.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/no-agent.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/no-skill.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/no-spec-written.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/offers-bug-route.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/graders/offers-spec-route.md create mode 100644 evals/skills/start-task/bug-offers-bugfix/prompt.md create mode 100755 evals/skills/start-task/bug-offers-bugfix/scaffold.sh create mode 100644 evals/skills/start-task/contained-change-offers-executor/case.yaml create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/no-skill.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/no-spec-written.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/no-sub-agent.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/offers-executor-route.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/offers-spec-route.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/graders/states-executor.md create mode 100644 evals/skills/start-task/contained-change-offers-executor/prompt.md create mode 100755 evals/skills/start-task/contained-change-offers-executor/scaffold.sh diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 9af7093..37c0fdf 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "freethinking", - "version": "0.5.1", + "version": "0.6.0", "description": "Shared engineering skills: standard workflows and prompts for day-to-day development.", "author": { "name": "Developer Handbook contributors" diff --git a/.claude/specs/standalone-task-routing/design.md b/.claude/specs/standalone-task-routing/design.md index d611ce7..7af1a91 100644 --- a/.claude/specs/standalone-task-routing/design.md +++ b/.claude/specs/standalone-task-routing/design.md @@ -68,7 +68,9 @@ Key decision, the Question is the AskUserQuestion tool, named in the skill. Noth | `skills/start-task/SKILL.md`, steps 3 to 6 | Unchanged; the spec questions move to the head of step 3 | 1.1 | | `rules/workflow.md` | Rewritten bullets: bug named, spec exists, PR asked, trivial change, default to `start-task`, never on main, keep the author in the loop | 4.1 to 4.5 | | `rules/collaboration.md` | The six defaults, under 1,600 characters | 6.1 to 6.8 | -| `README.md` | `start-task` row, `rules/workflow.md` row, new `rules/collaboration.md` row | 5.1, 6.9 | +| `README.md` | `start-task` row, `rules/workflow.md` row, new `rules/collaboration.md` row; the rules-size paragraph says `make validate` fails past the limit; the Testing section describes the checks under `checks/` without listing them; the Agents section names `start-task` as a launcher of the executor | 5.1, 5.8, 5.9, 5.10, 6.9 | +| `agents/task-executor.md`, frontmatter `description` | Names `start-task` beside `implement-spec` as a launcher | 5.10 | +| `Makefile`, `eval-skill-start-task` | `CASE_ARGS += --allow-tools Write Edit --scaffold`, so the no-write graders can fail | 5.3, 5.4, 5.11 | | `.github/ISSUE_TEMPLATE/*.yml` (three forms) | `collaboration (rule)` added to the component dropdown after `code-changes (rule)`, the alphabetical order `checks/test_issue_forms.py` enforces | 6.1, 5.7 | | `.claude-plugin/plugin.json` | Version `0.6.0` | 5.2 | | `evals/skills/start-task/contained-change-offers-executor/` | Scaffolded calc fixture; prompt is a Contained_Change; graders below | 5.3 | @@ -161,7 +163,7 @@ Rule text, `rules/collaboration.md`, one bullet per criterion 6.2 to 6.7, each b ## Testing Strategy - Free checks, run by `make validate`: the existing `checks/test_issue_forms.py` proves Property 5 once `collaboration (rule)` is in the forms; a new check in `checks/test_rules_size.py` proves Property 4 by running `hooks/load-rules.sh` and measuring, and by measuring `rules/collaboration.md` alone. The size check also closes issue #3's first half. `claude plugin validate --strict` proves the frontmatter. -- Eval case `contained-change-offers-executor` (Property 1, 3; Requirement 5.3): scaffold the calc fixture; prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py in the style of add`; `allowed_tools: [Read, Glob, Grep, AskUserQuestion, Agent, Skill]`; `max_turns: 6`. Graders: regex `Executor_Route|executor` in the final output; `file_exists` `.claude/specs/**` false; `tool_used` Agent with max 0; `tool_used` Skill with max 0; and the question grader, `tool_used` AskUserQuestion min 1, or, if task 5.4 finds the harness cannot carry that tool, a regex for the two options on separate lines. +- Eval case `contained-change-offers-executor` (Property 1, 3; Requirement 5.3): scaffold the calc fixture; prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py in the style of add`; `allowed_tools: [Read, Glob, Grep, Agent, Skill]`; `max_turns: 6`. Write and Edit come from the Makefile target's `--allow-tools` grant, the only route for them per `evals/README.md`, on purpose: a skill that writes a spec document or edits source before the Question must be caught by the `file_exists` and file-content graders, not prevented by the tool list. Graders: regex `executor` with `flags: i` in the final output (the harness's engine rejects inline `(?i)`); `file_exists` `.claude/specs/**` false; `tool_used` Agent with max 0; `tool_used` Skill with max 0; and the question grader, which task 3.1 settled as two regexes with `flags: im`, one per option as a list item at a line start. No `\?` grader: the prose fallback does not always end in a question mark, and the two option graders are the stronger proof that the question was put. - Eval case `bug-offers-bugfix` (Property 1; Requirement 5.4): same fixture with `add` returning `a - b`; prompt `/freethinking:start-task Adding 2 and 3 in calc returns -1`; graders: regex `Bug_Route|bugfix`; `file_exists` `**/*.py` unchanged by a regex on `calc/ops.py` still containing the planted body; `tool_used` Skill max 0; the same question grader. - Eval case `short-list-asks-with-tool` under `evals/rules/collaboration/` (Requirement 6.10): no plugin skill invoked; prompt asks Claude to pick between three named options for a config value it cannot infer; grader as the question grader above. - Existing cases: `names-task-and-asks` run twice; `eval-rule-workflow` run twice; `eval-skill-start-task` twice. Each result table goes into the PR's Testing section. diff --git a/.claude/specs/standalone-task-routing/requirements.md b/.claude/specs/standalone-task-routing/requirements.md index aeaf7bd..508c21a 100644 --- a/.claude/specs/standalone-task-routing/requirements.md +++ b/.claude/specs/standalone-task-routing/requirements.md @@ -92,6 +92,10 @@ 5. THE existing Eval_Case `evals/skills/start-task/names-task-and-asks` SHALL still pass, with its prompt sized so that Triage chooses Spec_Route. 6. THE existing Eval_Cases under `evals/rules/workflow/` SHALL still pass after the Rule changes, with their prompts or graders adjusted only where the Rule's new wording requires it. 7. THE `make validate` target SHALL pass. +8. THE README's paragraph on the rules' size limit SHALL say that `make validate` fails when the combined rules exceed it, since the size check now runs there. +9. THE README's description of the free checks SHALL describe them as the `unittest` files under `checks/` that `make check` discovers, each one concern named by its module docstring, and SHALL NOT list them by name, so that a new check needs no README change. +10. THE `task-executor` agent's description and the README's Agents section SHALL name `start-task` as a launcher of the executor beside `implement-spec`, since every text that names the executor's launchers must stay true. +11. THE `eval-skill-start-task` Makefile target SHALL grant `Write` and `Edit` with `--allow-tools`, which per `evals/README.md` is the only way a case gets them (its own `allowed_tools` lists read-only tools), so that a Skill that writes a spec document or edits source before the Question is caught by the graders rather than prevented by the tool list. ### Requirement 6: The Collaboration_Rule diff --git a/.claude/specs/standalone-task-routing/tasks.md b/.claude/specs/standalone-task-routing/tasks.md index 4ce5c58..f045845 100644 --- a/.claude/specs/standalone-task-routing/tasks.md +++ b/.claude/specs/standalone-task-routing/tasks.md @@ -6,7 +6,7 @@ Rules first, because the skill's CX block refers to the collaboration rule and t ## Tasks -- [ ] 1. The rules +- [x] 1. The rules - [x] 1.1 Write `rules/collaboration.md` - Heading "Working with the author", a one-line scope sentence, then six bold-led bullets in the voice of `rules/code-changes.md`: ask with the AskUserQuestion tool when the answers are a short fixed list, offering the options plus free text, and in prose when the answer is the author's own words; batch questions into one round unless an answer decides the next question; state the plan in two or three sentences before acting; approval is explicit and "looks good" with a "but" is a change request; report failures and anything not run before any recommendation; a skill that ends says where the work is and which skill is next. Under 1,600 characters. Prove it with `wc -c` - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8_ @@ -14,54 +14,57 @@ Rules first, because the skill's CX block refers to the collaboration rule and t - Replace the "New feature or sizeable task" bullet with the design's bullet 5, "Anything else that changes code: start the task", placed after "Small changes stay small"; retitle the bug bullet "Bug named: reproduce before fixing"; leave the spec, PR, default-branch and keep-in-the-loop bullets as they are. No bullet other than "Small changes stay small" asks Claude to judge size. Prove it by reading the file back and by `sh hooks/load-rules.sh | wc -c` - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_ - Wave 1 fix, agreed at the wave 0 hand-back: add "The first bullet that matches the request is the route." under the rule's heading, and a clause in `.github/CONTRIBUTING.md`'s "Make a change" passage saying `start-task` proposes the branch name on every route - - [ ] 1.3 Add `checks/test_rules_size.py` + - [x] 1.3 Add `checks/test_rules_size.py` - A stdlib `unittest` file in the style of `checks/test_issue_forms.py`: one test runs `hooks/load-rules.sh` with `CLAUDE_PLUGIN_ROOT` set to the repository root and asserts its output is under 10,000 characters; one asserts `rules/collaboration.md` is under 1,600. Failure messages give the measured count. Run `make check` - **Property 4: Rules fit the hook** - _Requirements: 4.4, 6.8_ - - [ ] 1.4 Add `collaboration (rule)` to the three issue forms + - [x] 1.4 Add `collaboration (rule)` to the three issue forms - In `.github/ISSUE_TEMPLATE/eval-case.yml`, `misbehaving-skill-or-agent.yml` and `proposal.yml`, insert `collaboration (rule)` into the component dropdown directly after `code-changes (rule)`, since the check sorts rule stems alphabetically. Run `make check` - **Property 5: Forms match disk** - _Requirements: 6.1_ -- [ ] 2. The skill - - [ ] 2.1 Add the CX block and the triage step to `skills/start-task/SKILL.md` +- [x] 2. The skill + - [x] 2.1 Add the CX block and the triage step to `skills/start-task/SKILL.md` - Frontmatter `description`: add that the skill is the front door for any request to change code and decides between the executor, bugfix and a spec. After "Ground rules", a section "Working with the author" with the design's four bullets: stops, how it asks (AskUserQuestion with fixed options at every stop; the step 3 spec questions in prose), hand-backs, hand-offs; state only what is specific to this skill. In step 1, say that an existing spec directory means Spec_Route with no triage, and that beside the task name the skill proposes a branch name in the contributing guide's shape (issue number and task name when the request names or links an issue, task name alone otherwise) without creating it; the CX block's hand-offs bullet says the author or `implement-spec` creates it. Split step 2 into "2. Read the code" (today's first paragraph) and "3. Triage" with: the three routes and the Contained_Change and defect definitions from the requirements glossary; state the chosen route and the reason in two or three sentences naming the code read; on Bug_Route or Executor_Route put one AskUserQuestion, header `Route`, chosen route first marked "(Recommended)", `Spec_Route` second, each with a one-sentence description, and act only on the answer; take the author's route without arguing; on Spec_Route say so in one sentence and continue; if the free-text answer is neither route, restate the two in a sentence and ask once more. Renumber the later steps and the spec questions move to the head of the requirements step. Run `make validate` - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10_ - - [ ] 2.2 Write the Bug_Route and Executor_Route paths in the triage step + - [x] 2.2 Write the Bug_Route and Executor_Route paths in the triage step - Bug_Route: invoke the Skill tool with `freethinking:bugfix` and the request as its argument, then stop; do not reproduce, diagnose or fix. Executor_Route: launch the Agent tool once with `subagent_type: freethinking:task-executor` and the design's Launch_Prompt (request verbatim, context found as `path: what is there`, the author's free-text additions, then "Make only this change. Commit nothing. Return the Summary."), with no sentence saying the task comes from a spec; edit no source file, commit nothing, create no branch. On `status: done`: show the Summary verbatim, run `git diff -- ` and show it, name any failed check before anything else, then recommend `/freethinking:pr`. On `status: stopped`: show the Summary and put one AskUserQuestion offering "answer and relaunch" and "Spec_Route"; never relaunch unprompted. If either plugin component is missing, say so and stop. Run `make validate` - _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7_ -- [ ] 3. The eval cases - - [ ] 3.1 Probe how the eval harness treats AskUserQuestion +- [x] 3. The eval cases + - [x] 3.1 Probe how the eval harness treats AskUserQuestion - In the scratchpad, make a one-case plugin whose only skill puts an AskUserQuestion, run `claude plugin eval` on it with `allowed_tools` including `AskUserQuestion` and `max_turns: 3`, and record in the task's result: whether the tool is callable headless, whether the harness answers it or ends the turn, whether a `tool_used` grader with `tool: AskUserQuestion` counts it, and whether the transcript a `regex` grader sees contains the question's option labels. Delete the scratch plugin. The answer decides the question grader's shape in 3.2, 3.3 and 3.5 - _Requirements: 5.3, 5.4, 6.10_ - - [ ] 3.2 Eval case `evals/skills/start-task/contained-change-offers-executor/` - - `case.yaml` with `scaffold_script: scaffold.sh` copying `evals/fixtures/calc` into the workspace, as the implement-spec and task-executor cases do; `prompt.md` with tags `[skill, start-task]`, `max_turns: 6`, `allowed_tools: [Read, Glob, Grep, AskUserQuestion, Agent, Skill]`, prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add`; graders: regex `(?i)executor` on the output; `file_exists` `.claude/specs/**` false; `tool_used` Agent max 0; `tool_used` Skill max 0; the question grader in the shape 3.1 found. Add `eval-skill-start-task: CASE_ARGS += --scaffold` to the Makefile's per-target block + - [x] 3.2 Eval case `evals/skills/start-task/contained-change-offers-executor/` + - `case.yaml` with `scaffold_script: scaffold.sh` copying `evals/fixtures/calc` into the workspace, as the implement-spec and task-executor cases do; `prompt.md` with tags `[skill, start-task]`, `max_turns: 6`, `allowed_tools: [Read, Glob, Grep, AskUserQuestion, Agent, Skill]`, prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add`; graders: regex `(?i)executor` on the output; `file_exists` `.claude/specs/**` false; `tool_used` Agent max 0; `tool_used` Skill max 0; the question grader in the shape 3.1 found. Add `eval-skill-start-task: CASE_ARGS += --scaffold` to the Makefile's per-target block. Wave 2 fix, agreed at the wave 2 hand-back: the Makefile line becomes `CASE_ARGS += --allow-tools Write Edit --scaffold`, which per `evals/README.md` is the only way a case gets those tools, so the no-write graders can fail; the cases' `allowed_tools` stay read-only (Requirement 5.11) - **Property 1: No document before the answer** - **Property 3: Standalone launch** - _Requirements: 5.3_ - - [ ] 3.3 Eval case `evals/skills/start-task/bug-offers-bugfix/` + - [x] 3.3 Eval case `evals/skills/start-task/bug-offers-bugfix/` - Same scaffold, but the script then rewrites `add` in `calc/ops.py` to return `a - b`; prompt `/freethinking:start-task calc.add(2, 3) returns -1; it should return 5`; graders: regex `(?i)bug`; regex `return a - b` on `calc/ops.py` (the defect is still there); `tool_used` Skill max 0; `file_exists` `.claude/specs/**` false; the question grader - **Property 1: No document before the answer** - _Requirements: 5.4_ - - [ ] 3.4 Keep `evals/skills/start-task/names-task-and-asks/` on Spec_Route + - [x] 3.4 Keep `evals/skills/start-task/names-task-and-asks/` on Spec_Route - Prompt unchanged; `expected_outcome` says the skill states Spec_Route in one sentence before proposing the task name and a branch name and asking its questions; `allowed_tools` adds `AskUserQuestion` if 3.1 found it callable. Check `evals/rules/workflow/bug-routes-to-bugfix/prompt.md` names the request as a bug in the author's words, and change the prompt only if it does not - _Requirements: 5.5, 5.6_ - - [ ] 3.5 Eval case `evals/rules/collaboration/short-list-asks-with-tool/` + - [x] 3.5 Eval case `evals/rules/collaboration/short-list-asks-with-tool/` - `prompt.md` with tags `[rule, collaboration]`, `max_turns: 4`, `allowed_tools: [Read, Glob, Grep, AskUserQuestion]`, a prompt that asks Claude to set a log level in a config file to one of three named levels without saying which, so it must ask; graders: the question grader, plus a regex that the three level names appear. The new `eval-rule-collaboration` target comes from the Makefile's directory enumeration and needs no Makefile change - _Requirements: 6.10_ -- [ ] 4. Docs and version - - [ ] 4.1 README rows - - Skills table, `start-task` row: it is the front door for any change, reads the code, and decides between the executor for one contained change, `bugfix` for a defect and a spec for the rest, asking before the first two. Rules table, `rules/workflow.md` row: route on what the author said or a file on disk, default to `start-task`. New row for `rules/collaboration.md` after the `comments` row, in the voice of the others. Nothing else in the README changes - - _Requirements: 5.1, 6.9_ - - [ ] 4.2 Version `0.6.0` +- [x] 4. Docs and version + - [x] 4.1 README rows + - Skills table, `start-task` row: it is the front door for any change, reads the code, and decides between the executor for one contained change, `bugfix` for a defect and a spec for the rest, asking before the first two. Rules table, `rules/workflow.md` row: route on what the author said or a file on disk, default to `start-task`. New row for `rules/collaboration.md` after the `comments` row, in the voice of the others. The Rules paragraph's sentence on the 10,000-character limit gains "and `make validate` fails when they go over". Nothing else in the README changes + - _Requirements: 5.1, 5.8, 6.9_ + - Wave 2 fix, agreed at the wave 2 hand-back: the README's Testing section describes the checks as the `unittest` files under `checks/` that `make check` discovers, each one concern named by its module docstring, with no list of file names; the README's Agents section and the `description` in `agents/task-executor.md` name `start-task` as a launcher of the executor beside `implement-spec` + - _Requirements: 5.9, 5.10_ + - [x] 4.2 Version `0.6.0` - `.claude-plugin/plugin.json` version to `0.6.0`. Run `make validate` - _Requirements: 5.2_ - [ ] 5. Runs - - [ ] 5.1 Run the evals + - [x] 5.1 Run the evals - `make eval-skill-start-task EVAL_ARGS="--runs 2"`, `make eval-rule-workflow EVAL_ARGS="--runs 2"`, `make eval-rule-collaboration EVAL_ARGS="--runs 2"`. Fix the case or the skill until each case passes both runs, and record the three result tables for the PR's Testing section + - Run on 2026-10-13: first pass found the harness rejects inline `(?i)` (two graders switched to `flags: i`) and that the prose route question does not always end in `?` (the `\?` grader dropped from the two route cases, whose option graders prove the question). Second pass: all nine cases 100% on both runs, $2.18 total - _Requirements: 5.3, 5.4, 5.5, 5.6, 6.10_ - [ ] 5.2 Manual session, done by the author - In a session started with `claude --plugin-dir .` in a scratch copy of `evals/fixtures/calc`, run the contained-change prompt, the bug prompt and the link-shortener prompt by hand. Confirm the Route question renders as the interactive form, that answering Spec_Route continues to the spec questions, and that accepting Executor_Route ends with the Summary, the diff and the `pr` recommendation. Record what was seen in the PR diff --git a/.github/ISSUE_TEMPLATE/eval-case.yml b/.github/ISSUE_TEMPLATE/eval-case.yml index 6298822..767bac6 100644 --- a/.github/ISSUE_TEMPLATE/eval-case.yml +++ b/.github/ISSUE_TEMPLATE/eval-case.yml @@ -28,6 +28,7 @@ body: - task-executor (agent) - wave-reviewer (agent) - code-changes (rule) + - collaboration (rule) - comments (rule) - workflow (rule) - SessionStart hook diff --git a/.github/ISSUE_TEMPLATE/misbehaving-skill-or-agent.yml b/.github/ISSUE_TEMPLATE/misbehaving-skill-or-agent.yml index c70609e..9129176 100644 --- a/.github/ISSUE_TEMPLATE/misbehaving-skill-or-agent.yml +++ b/.github/ISSUE_TEMPLATE/misbehaving-skill-or-agent.yml @@ -28,6 +28,7 @@ body: - task-executor (agent) - wave-reviewer (agent) - code-changes (rule) + - collaboration (rule) - comments (rule) - workflow (rule) - SessionStart hook diff --git a/.github/ISSUE_TEMPLATE/proposal.yml b/.github/ISSUE_TEMPLATE/proposal.yml index d017aed..8a9ed7b 100644 --- a/.github/ISSUE_TEMPLATE/proposal.yml +++ b/.github/ISSUE_TEMPLATE/proposal.yml @@ -42,6 +42,7 @@ body: - task-executor (agent) - wave-reviewer (agent) - code-changes (rule) + - collaboration (rule) - comments (rule) - workflow (rule) - SessionStart hook diff --git a/Makefile b/Makefile index bd3d177..8e755ed 100644 --- a/Makefile +++ b/Makefile @@ -59,6 +59,7 @@ eval-agent-wave-reviewer: CASE_ARGS += --allow-tools Bash --scaffold eval-agent-wave-reviewer: export PATH := bin:$(PATH) eval-skill-pr: CASE_ARGS += --allow-tools Bash --scaffold eval-skill-pr: export PATH := bin:$(PATH) +eval-skill-start-task: CASE_ARGS += --allow-tools Write Edit --scaffold eval-rule-code-changes: CASE_ARGS += --allow-tools Write Edit --scaffold # Runs all evals and needs all flags diff --git a/README.md b/README.md index 9f527bd..db56d97 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ claude plugin update freethinking@freethinkingit | Skill | Run it with | What it does | | ----- | ----------- | ------------ | -| `start-task` | `/freethinking:start-task `, or ask Claude to start or spec a task | Writes the spec before any code: `requirements.md`, `design.md` and `tasks.md` in `.claude/specs//`, following the team template, with you approving each document before the next. | +| `start-task` | `/freethinking:start-task `, or ask Claude to change something, or to start or spec a task | The front door for any change: reads the code the request touches, then sends one contained change to the `task-executor` agent, a defect to `bugfix` and everything else to a spec, asking you before taking either of the first two. On the spec route, writes `requirements.md`, `design.md` and `tasks.md` in `.claude/specs//` before any code, following the team template, with you approving each document before the next. | | `implement-spec` | `/freethinking:implement-spec [task] [task\|wave]`, or ask Claude to implement, build or continue a spec | Builds a task from its approved spec, wave by wave from `tasks.md`, each leaf task in its own sub-agent, ticking a task only once its tests pass, running the full suite at the end of every wave, and handing back to you after each task or each wave, whichever you chose. Pauses with a proposal when the design cannot be followed, and hands off to `pr` when the spec is done. | | `bugfix` | `/freethinking:bugfix `, or ask Claude to fix a bug | Reads the ticket, reproduces the bug before touching code, confirms the root cause with you, fixes the cause with a regression test, verifies the fix, then hands off to `pr`. | | `pr` | `/freethinking:pr`, or ask Claude to open a PR | Walks the branch's changes with you, asks what the code can't answer, drafts the title and description to the team standard, and opens the PR with `gh` once you approve. Needs the `gh` CLI signed in. | @@ -52,9 +52,10 @@ in every repository, so Claude follows them without being asked. They live in | File | Covers | | ---- | ------ | -| `rules/workflow.md` | Which skill to use when: spec new work with `start-task`, build an existing spec with `implement-spec`, fix bugs with `bugfix`, open PRs with `pr`, never work on the default branch, and when a change is small enough to skip all of that. | +| `rules/workflow.md` | Which skill to use when, routing on what you said or a file on disk and taking the first rule that matches: `bugfix` when you call it a bug, `implement-spec` when a spec exists, `pr` for pull requests, no skill for a typo-sized change, and `start-task` for any other change, which reads the code and picks between the `task-executor` agent for one contained change, `bugfix` for a defect and a spec for the rest, asking before the first two. Also: never work on the default branch, and say so when the approach changes. | | `rules/code-changes.md` | How to change code: read the surrounding code first, match the project's style and libraries, make the smallest change that does the job, add a test with every behaviour change, run what you touched, and leave no debug residue behind. | | `rules/comments.md` | How to comment: docstrings on public contracts only, saying intent, parameters, returns and exceptions; inline comments only for non-obvious details such as workarounds and edge cases, never for what a line does; no divider or closing-brace comments; no commented-out dead code. | +| `rules/collaboration.md` | How skills and agents work with you: ask in the shape of the answer, with fixed options or in prose; batch the questions; say the plan before acting; take only a plain yes as approval; report failures and anything not run before any recommendation; and say where the work is and which skill is next when a skill ends. | Claude Code does not load a `CLAUDE.md` or a `rules/` folder from a plugin by itself, so `hooks/hooks.json` registers two hooks. The `SessionStart` hook runs @@ -64,13 +65,15 @@ a resume, a clear, a compact or a fork). The `SubagentStart` hook runs `hooks/load-rules-subagent.sh`, which puts the same rules into every sub-agent, which is how both agents below get them. Keep the combined rules under 10,000 characters; Claude Code truncates hook output past -that. A rule a repository needs to override belongs in that repository's own -`CLAUDE.md`, which Claude reads alongside these. +that, and `make validate` fails when they go over. A rule a repository needs +to override belongs in that repository's own `CLAUDE.md`, which Claude reads +alongside these. ## Agents The plugin ships two agents under `agents/`. `implement-spec` launches both, -and you can launch either directly: mention it in a session as +`start-task` launches the executor for one contained change, and you can +launch either directly: mention it in a session as `@agent-freethinking:task-executor` or `@agent-freethinking:wave-reviewer`, or have a skill or session launch it with the Agent tool as `subagent_type: freethinking:task-executor` or `freethinking:wave-reviewer`. @@ -87,7 +90,7 @@ What the prompt carries: | Agent | What it does | What it returns | | ----- | ------------ | --------------- | -| `task-executor` | Builds one well-specified change as the smallest change in the project's style, proves it, and deletes its scratch work. Stops and asks instead of guessing when the task is unclear, the design does not fit the code, or the task would need a new dependency, framework or tool. `implement-spec` launches it for every leaf task. | A Summary: `status` (`done` or `stopped`), `files changed`, `checks run` with their results, the ephemeral check it ran, or why none applied, what it is `unsure` of, and, when stopped, the question or evidence under `stopped because`. | +| `task-executor` | Builds one well-specified change as the smallest change in the project's style, proves it, and deletes its scratch work. Stops and asks instead of guessing when the task is unclear, the design does not fit the code, or the task would need a new dependency, framework or tool. `start-task` launches it for one contained change, and `implement-spec` for every leaf task. | A Summary: `status` (`done` or `stopped`), `files changed`, `checks run` with their results, the ephemeral check it ran, or why none applied, what it is `unsure` of, and, when stopped, the question or evidence under `stopped because`. | | `wave-reviewer` | Reviews one set of changes against the requirements and design they implement and the files that cite them, gathering the diff itself and changing nothing. `implement-spec` launches it at the end of every wave. | Findings: a `verdict` (`holds` or `findings`), then each problem with its `file`, an exact `quote`, the `problem` and a `severity` (`blocker`, `should-fix` or `nit`), and what it is `unsure` of. | The executor works in one of two modes, set by the launch prompt alone: @@ -131,12 +134,8 @@ with a description exists. ### Unit tests `checks/` holds standard-library `unittest` files, written for the tooling the -plugin's development and evals depend on: - -- `test_hook_encoder.py`: the sub-agent hook's JSON encoder, with a fixed - table, a seeded fuzz and a real run of the hook. -- `test_agents.py`: every agent's frontmatter, and every `freethinking:` a - skill references names an existing agent or skill. +plugin's development and evals depend on. `make check` discovers every one of +them, and each file covers one concern, named by its module docstring. ```bash make check # all of them diff --git a/agents/task-executor.md b/agents/task-executor.md index 789ecb2..acd70a1 100644 --- a/agents/task-executor.md +++ b/agents/task-executor.md @@ -1,6 +1,6 @@ --- name: task-executor -description: Builds one well-specified change, from a spec task or a single change you describe, as the smallest change in the project's style, proves it, and returns a fixed-shape summary. Use it directly for a contained task; implement-spec launches it for every leaf task of a spec. +description: Builds one well-specified change, from a spec task or a single change you describe, as the smallest change in the project's style, proves it, and returns a fixed-shape summary. Use it directly for a contained task; start-task launches it for one contained change on its executor route, and implement-spec for every leaf task of a spec. tools: Read, Glob, Grep, Edit, Write, Bash model: opus effort: medium diff --git a/checks/test_rules_size.py b/checks/test_rules_size.py new file mode 100644 index 0000000..0334a17 --- /dev/null +++ b/checks/test_rules_size.py @@ -0,0 +1,41 @@ +"""The rules fit the SessionStart hook: Claude Code truncates hook output past 10,000 characters. + +The output of hooks/load-rules.sh stays under 10,000 characters and +rules/collaboration.md under 1,600, both counted as decoded UTF-8 characters. +""" + +import os +import subprocess +import unittest + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +LOAD_RULES = os.path.join(ROOT, "hooks", "load-rules.sh") +COLLABORATION = os.path.join(ROOT, "rules", "collaboration.md") + +HOOK_LIMIT = 10000 +COLLABORATION_LIMIT = 1600 + + +class RulesSizeTest(unittest.TestCase): + """The hook output and the collaboration rule stay under their character limits.""" + + def test_hook_output_is_under_the_limit(self): + env = dict(os.environ, CLAUDE_PLUGIN_ROOT=ROOT) + run = subprocess.run([LOAD_RULES], env=env, capture_output=True, check=True) + count = len(run.stdout.decode("utf-8")) + self.assertLess( + count, HOOK_LIMIT, "load-rules.sh prints %d characters; the limit is %d" % (count, HOOK_LIMIT) + ) + + def test_collaboration_rule_is_under_the_limit(self): + with open(COLLABORATION, encoding="utf-8") as f: + count = len(f.read()) + self.assertLess( + count, + COLLABORATION_LIMIT, + "rules/collaboration.md has %d characters; the limit is %d" % (count, COLLABORATION_LIMIT), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/asks-a-question.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/asks-a-question.md new file mode 100644 index 0000000..ecf03ff --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/asks-a-question.md @@ -0,0 +1,4 @@ +--- +type: regex +pattern: '\?' +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/does-not-choose.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/does-not-choose.md new file mode 100644 index 0000000..50bcb3e --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/does-not-choose.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|[.!]\s+)I set\b|\b(?:I(?:''ve| have) (?:set|chosen|picked|gone with)|I (?:chose|picked|went with)|setting it to|log_level:\s*(?:debug|info|warn))\b' +match: not_contains +flags: im +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-debug.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-debug.md new file mode 100644 index 0000000..360ceeb --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-debug.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*debug\b' +flags: im +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-info.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-info.md new file mode 100644 index 0000000..3ee03a1 --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-info.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*info\b' +flags: im +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-warn.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-warn.md new file mode 100644 index 0000000..0b45c0b --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/offers-warn.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*warn\b' +flags: im +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/graders/sets-nothing.md b/evals/rules/collaboration/short-list-asks-with-tool/graders/sets-nothing.md new file mode 100644 index 0000000..e23503a --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/graders/sets-nothing.md @@ -0,0 +1,5 @@ +--- +type: file_exists +path: 'config.yml' +exists: false +--- diff --git a/evals/rules/collaboration/short-list-asks-with-tool/prompt.md b/evals/rules/collaboration/short-list-asks-with-tool/prompt.md new file mode 100644 index 0000000..d558cd1 --- /dev/null +++ b/evals/rules/collaboration/short-list-asks-with-tool/prompt.md @@ -0,0 +1,8 @@ +--- +description: 'Asked to set a config value from a short fixed list without being told which, Claude asks with the options offered, as rules/collaboration.md says: with the AskUserQuestion tool where it is available, otherwise in prose with the options on separate lines.' +tags: [rule, collaboration] +max_turns: 4 +allowed_tools: [Read, Glob, Grep] +expected_outcome: Claude does not pick a level itself; it asks which of debug, info and warn to use, listing each option on its own line, and sets nothing. +--- +`config.yml` is to be created with one key, `log_level`, and no value has been chosen yet. The service supports three levels: `debug`, `info` and `warn`. I have not decided which one this deployment should use. Set the log level in `config.yml`. diff --git a/evals/rules/workflow/bug-routes-to-bugfix/prompt.md b/evals/rules/workflow/bug-routes-to-bugfix/prompt.md index 0782b86..af6ace0 100644 --- a/evals/rules/workflow/bug-routes-to-bugfix/prompt.md +++ b/evals/rules/workflow/bug-routes-to-bugfix/prompt.md @@ -5,4 +5,4 @@ max_turns: 10 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: Claude invokes /freethinking:bugfix before doing anything else about the bug, restates the symptom and expected behaviour, says it will reproduce first, and claims no fix. --- -Ticket ENG-412: the upload worker crashes with "TypeError: Cannot read properties of null (reading 'name')" when a user uploads a file whose name contains spaces. Expected: the upload succeeds. Please fix it. +Bug ticket ENG-412: the upload worker crashes with "TypeError: Cannot read properties of null (reading 'name')" when a user uploads a file whose name contains spaces. Expected: the upload succeeds. Please fix it. diff --git a/evals/skills/start-task/bug-offers-bugfix/case.yaml b/evals/skills/start-task/bug-offers-bugfix/case.yaml new file mode 100644 index 0000000..2cbb0d9 --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: bug-offers-bugfix +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/defect-still-present.md b/evals/skills/start-task/bug-offers-bugfix/graders/defect-still-present.md new file mode 100644 index 0000000..47cd8d4 --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/defect-still-present.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'return a - b' +target: { source: file, path: calc/ops.py } +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/names-bug.md b/evals/skills/start-task/bug-offers-bugfix/graders/names-bug.md new file mode 100644 index 0000000..9fe7a2d --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/names-bug.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'bug' +flags: i +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/no-agent.md b/evals/skills/start-task/bug-offers-bugfix/graders/no-agent.md new file mode 100644 index 0000000..94838dd --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/no-agent.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Agent +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/no-skill.md b/evals/skills/start-task/bug-offers-bugfix/graders/no-skill.md new file mode 100644 index 0000000..a01eeac --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/no-skill.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/no-spec-written.md b/evals/skills/start-task/bug-offers-bugfix/graders/no-spec-written.md new file mode 100644 index 0000000..7a79050 --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/no-spec-written.md @@ -0,0 +1,5 @@ +--- +type: file_exists +path: '.claude/specs/**' +exists: false +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/offers-bug-route.md b/evals/skills/start-task/bug-offers-bugfix/graders/offers-bug-route.md new file mode 100644 index 0000000..e874f68 --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/offers-bug-route.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Bug_Route\b' +flags: im +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/graders/offers-spec-route.md b/evals/skills/start-task/bug-offers-bugfix/graders/offers-spec-route.md new file mode 100644 index 0000000..4f418ed --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/graders/offers-spec-route.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Spec_Route\b' +flags: im +--- diff --git a/evals/skills/start-task/bug-offers-bugfix/prompt.md b/evals/skills/start-task/bug-offers-bugfix/prompt.md new file mode 100644 index 0000000..7ae6a1a --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/prompt.md @@ -0,0 +1,8 @@ +--- +description: Invoked directly with observed versus intended behaviour that never uses the word bug, start-task triages the defect to Bug_Route, offers the routes as a question, and neither reproduces, fixes, nor invokes bugfix before the author answers. Needs the scaffold, which plants add returning a - b in calc/ops.py. +tags: [skill, start-task] +max_turns: 6 +allowed_tools: [Read, Glob, Grep, Agent, Skill] +expected_outcome: Claude reads calc/ops.py and tests/test_ops.py, states in two or three sentences that this is a bug and takes Bug_Route, naming add in calc/ops.py returning a - b, and asks the author which route to take, with Bug_Route and Spec_Route as options on their own lines. It launches no sub-agent, invokes no skill, writes nothing under .claude/specs/, and leaves calc/ops.py unchanged. +--- +/freethinking:start-task calc.add(2, 3) returns -1; it should return 5 diff --git a/evals/skills/start-task/bug-offers-bugfix/scaffold.sh b/evals/skills/start-task/bug-offers-bugfix/scaffold.sh new file mode 100755 index 0000000..7baff72 --- /dev/null +++ b/evals/skills/start-task/bug-offers-bugfix/scaffold.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main, then plants the +# defect, add returning a - b, so test_add fails. The defect is committed, so the +# workspace is clean and the defect reads as existing code, not a pending edit. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +bash "$ROOT/evals/fixtures/setup.sh" calc main +sed 's/return a + b/return a - b/' calc/ops.py > calc/ops.py.tmp && mv calc/ops.py.tmp calc/ops.py +grep -q 'return a - b' calc/ops.py +git -c user.name=fixture -c user.email=fixture@example.com commit -q -am "Plant the defect: add returns a - b" diff --git a/evals/skills/start-task/contained-change-offers-executor/case.yaml b/evals/skills/start-task/contained-change-offers-executor/case.yaml new file mode 100644 index 0000000..15f7991 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: contained-change-offers-executor +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/no-skill.md b/evals/skills/start-task/contained-change-offers-executor/graders/no-skill.md new file mode 100644 index 0000000..a01eeac --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/no-skill.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/no-spec-written.md b/evals/skills/start-task/contained-change-offers-executor/graders/no-spec-written.md new file mode 100644 index 0000000..7a79050 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/no-spec-written.md @@ -0,0 +1,5 @@ +--- +type: file_exists +path: '.claude/specs/**' +exists: false +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/no-sub-agent.md b/evals/skills/start-task/contained-change-offers-executor/graders/no-sub-agent.md new file mode 100644 index 0000000..94838dd --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/no-sub-agent.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Agent +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/offers-executor-route.md b/evals/skills/start-task/contained-change-offers-executor/graders/offers-executor-route.md new file mode 100644 index 0000000..4c8a063 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/offers-executor-route.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Executor_Route\b' +flags: im +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/offers-spec-route.md b/evals/skills/start-task/contained-change-offers-executor/graders/offers-spec-route.md new file mode 100644 index 0000000..4f418ed --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/offers-spec-route.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Spec_Route\b' +flags: im +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/graders/states-executor.md b/evals/skills/start-task/contained-change-offers-executor/graders/states-executor.md new file mode 100644 index 0000000..e222db6 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/graders/states-executor.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'executor' +flags: i +--- diff --git a/evals/skills/start-task/contained-change-offers-executor/prompt.md b/evals/skills/start-task/contained-change-offers-executor/prompt.md new file mode 100644 index 0000000..dbfa9b8 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/prompt.md @@ -0,0 +1,8 @@ +--- +description: Given a contained change against the calc fixture, start-task triages it as Executor_Route, offers the route question with Executor_Route recommended and Spec_Route as the alternative, and stops there, writing no spec and launching nothing before the author answers. Needs the Makefile's skill target, which runs the scaffold. +tags: [skill, start-task] +max_turns: 6 +allowed_tools: [Read, Glob, Grep, Agent, Skill] +expected_outcome: Claude reads calc/ops.py and the tests, says in two or three sentences that adding subtract next to add is a contained change taking Executor_Route, naming the code it touches, proposes `subtract` (or similar) as the task name and a branch name, and asks the route question with Executor_Route (Recommended) and Spec_Route as options. It writes nothing under .claude/specs/, edits no code, and launches no sub-agent or skill. +--- +/freethinking:start-task Add subtract(a, b) to calc/ops.py returning a minus b, in the style of add diff --git a/evals/skills/start-task/contained-change-offers-executor/scaffold.sh b/evals/skills/start-task/contained-change-offers-executor/scaffold.sh new file mode 100755 index 0000000..d087bf2 --- /dev/null +++ b/evals/skills/start-task/contained-change-offers-executor/scaffold.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main, with no spec +# files, so start-task has code to read and a branch to propose. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +bash "$ROOT/evals/fixtures/setup.sh" calc main diff --git a/evals/skills/start-task/names-task-and-asks/prompt.md b/evals/skills/start-task/names-task-and-asks/prompt.md index ba2e577..e7d8f3c 100644 --- a/evals/skills/start-task/names-task-and-asks/prompt.md +++ b/evals/skills/start-task/names-task-and-asks/prompt.md @@ -2,6 +2,6 @@ description: Invoked directly, start-task proposes a kebab-case task name and asks its first round of questions instead of writing the spec or code. tags: [skill, start-task] max_turns: 4 -expected_outcome: Claude proposes `link-shortener` (or similar kebab-case name), says the spec will live under .claude/specs/, and asks the author questions. It writes nothing. +expected_outcome: Claude proposes `link-shortener` (or similar kebab-case name) and a branch name, says in one sentence that the request takes Spec_Route, says the spec will live under .claude/specs/, and asks the author questions. It writes nothing. --- /freethinking:start-task Link shortener: users submit a long URL and get a short code; visiting the code redirects and counts the click. diff --git a/rules/collaboration.md b/rules/collaboration.md index 96ded7a..71e03ba 100644 --- a/rules/collaboration.md +++ b/rules/collaboration.md @@ -2,7 +2,7 @@ How any skill or agent works with the author in every repository this handbook applies to. -- **Ask in the shape of the answer.** When the answers are a short fixed list, ask with the AskUserQuestion tool, offering the options plus free text. When the answer is the author's own words, ask in prose. +- **Ask in the shape of the answer.** When the answers are a short fixed list, ask with the AskUserQuestion tool, offering the options plus free text. When the answer is the author's own words, ask in prose. When the tool is not available, ask the same question in prose with the options on separate lines, then wait. - **Batch the questions.** Ask everything you need in one round. Ask one at a time only when an answer decides the next question. - **Say the plan before acting.** State the plan in two or three sentences before carrying it out, so the author can stop a wrong turn early. - **Approval is explicit.** Where a step waits for approval, approval is a plain yes. "Looks good" with a "but" is a change request, not an approval. diff --git a/skills/start-task/SKILL.md b/skills/start-task/SKILL.md index 8a273ad..ee018ba 100644 --- a/skills/start-task/SKILL.md +++ b/skills/start-task/SKILL.md @@ -1,49 +1,102 @@ --- name: start-task -description: Start a new feature or task by writing its spec before any code. Use when asked to start, plan, spec, scope or design a task, feature or piece of work, or when a request is big enough that coding straight away would be guessing. Produces requirements.md, design.md and tasks.md in .claude/specs//, following the team's spec template, with the author approving each document before the next is written. Does not write code. +description: Front door for any request to change code. Reads the code the request touches, then decides whether it goes to the task executor, to the bugfix skill or to a spec, and asks the author when the answer is not a spec. Start a new feature or task by writing its spec before any code. Use for any request to change code that the workflow rule does not route elsewhere (a named bug, an existing spec, a pull request, a typo-sized change), and when asked to start, plan, spec, scope or design a task, feature or piece of work, or when a request is big enough that coding straight away would be guessing. Produces requirements.md, design.md and tasks.md in .claude/specs//, following the team's spec template, with the author approving each document before the next is written. Does not write code. argument-hint: "[task name or short description]" --- # Start a task -Turn a task into an approved spec before anyone writes code: requirements first, then a design that satisfies them, then an ordered task list to build it. The author approves each document before the next one starts, and owns the result. +The front door for a change: read the code the request touches, then route the request to `bugfix`, to the executor, or to a spec. On the spec route, turn the task into an approved spec before anyone writes code: requirements first, then a design that satisfies them, then an ordered task list to build it. The author approves each document before the next one starts, and owns the result. Task, as the author gave it: $ARGUMENTS ## Ground rules -- **No code.** This skill ends when `tasks.md` is approved. Do not create or edit source files, and do not start on the first task. +- **No code of your own.** Do not create or edit source files. On Spec_Route the skill ends when `tasks.md` is approved; on Bug_Route it ends at the hand-off; on Executor_Route the executor writes the code and the skill ends after reporting its Summary. - **The template is the standard.** The three files under `${CLAUDE_SKILL_DIR}/spec-template/` are self-describing: each is a valid document whose own prose explains how that kind of document works, using a Link Shortener as the worked example. Read the matching template file before writing each document. Copy its structure and its rules; do not copy its example content or its teaching notes. - **Ask, don't invent.** When something the spec needs is not in the request or the codebase, ask the author. Keep each round of questions short, and batch them. - **Approval is explicit.** "Looks good" with a "but" is a change request. Make the change and ask again. Do not move to the next document until the author says the current one is approved. - **Size to the task.** A small task gets short documents, but each document keeps its sections. For a small task, one diagram in the design is enough; for a large one, use the diagram types the template shows where they add information. +## Working with the author + +- **Stops.** The route question at triage, when the route is Bug_Route or Executor_Route; the approval of each of the three documents on Spec_Route; the relaunch question after a `stopped` Summary on Executor_Route. +- **How it asks.** With the AskUserQuestion tool, fixed options plus free text, at every stop. The spec questions at the head of the requirements step are the exception: ask them in prose, because the answers are the author's own words. +- **Hand-backs.** On Executor_Route, the executor's Summary verbatim and the diff, failures first. On Spec_Route, each document in full. +- **Hand-offs.** `bugfix` on Bug_Route; `pr` after a `done` Summary; `implement-spec` after `tasks.md` is approved. The branch name proposed in step 1 is created by the author before the first commit, or by `implement-spec` at its set-up when the author gives it that name; never by this skill. + ## 1. Name the task and find its spec -Derive a kebab-case task name from the request, for example `link-shortener`, and confirm it with the author along with the first questions below. The spec lives in `.claude/specs//` in the current repository. +Derive a kebab-case task name from the request, for example `link-shortener`. Beside it, propose a branch name: the issue number and the task name, such as `12-link-shortener`, when the request names or links an issue, and the task name alone otherwise. Do not create the branch. Say both in the route statement of step 3, so the author can correct them in the same breath. The spec lives in `.claude/specs//` in the current repository. + +If that directory already exists, the request takes Spec_Route without triage, since its size was already decided. Read what is there and resume at the first document that is missing or that the author says is not approved. Do not overwrite an existing document without asking. + +## 2. Read the code + +Before asking anything, look at the codebase for what the request touches: existing modules, patterns, naming, tests and configuration the task will have to fit. + +## 3. Triage + +Before asking any spec question or writing any document, decide which route the request takes: + +- **Bug_Route** when the request describes a defect: observed behaviour that differs from intended behaviour in code that already exists. +- **Executor_Route** when the request is a contained change: one change, describable in a sentence or two, touching a place in the code you can name after reading it, needing no design decision and no new dependency, framework or tool. +- **Spec_Route** for every other request. + +State the route you chose and the reason for it in two or three sentences that name the code you read; on Spec_Route, one sentence that names the route is enough. + +On Bug_Route or Executor_Route, put one question to the author with the AskUserQuestion tool, header `Route`, with two options in this order: the route you chose, marked "(Recommended)", then `Spec_Route`, each with a one-sentence description. The tool adds the free-text answer itself. Act only on the author's answer. If the author picks the other route, take it without arguing for yours. If the free-text answer is neither route, restate the two routes in one sentence and ask once more. + +On Spec_Route, go on to step 4 without a question. + +On Bug_Route or Executor_Route, once the author has accepted it, the route's own section below takes over. -If that directory already exists, read what is there and resume at the first document that is missing or that the author says is not approved. Do not overwrite an existing document without asking. +### Bug_Route -## 2. Understand the task +Invoke the Skill tool with `skill: freethinking:bugfix` and `args:` the request, in the author's words. This skill's own steps end at that hand-off, and `bugfix`'s procedure takes over from its first step. Do not reproduce, diagnose or fix the defect under this skill's steps: `bugfix` holds that discipline, and the hand-off keeps it in one place. -Before asking anything, look at the codebase for what the request touches: existing modules, patterns, naming, tests and configuration the task will have to fit. Then ask the author only what you still need, in one round: +If the Skill tool cannot find `freethinking:bugfix`, say that the plugin's `bugfix` skill is missing and stop. Do not do its work in its place. + +### Executor_Route + +Launch the agent once, with the Agent tool as `subagent_type: freethinking:task-executor` and a prompt that holds, in this order: + +- **Request**: the author's words, verbatim, from the task above and their message. +- **Context found**: the files and symbols step 2 read that the change touches, each as `path: what is there`. +- **Author's additions**: the free-text answer to the route question, verbatim, if there was one. +- **Boundaries**: the sentence, verbatim, "Make only this change. Commit nothing. Return the Summary." + +Nothing more. Never say, in any words, that the task comes from a spec: that sentence puts the agent in its spec mode, and here it must be in its standalone mode, where the change comes with its test. The agent's own definition carries its procedure, the shape of its Summary and the instruction to stop on drift, so the launch restates none of them. + +If the Agent tool cannot find `freethinking:task-executor`, say that the plugin's `task-executor` agent is missing and stop. Do not make the change yourself. + +On this route, edit, create or delete no source file yourself, commit nothing, push nothing and create no branch. The executor writes the change; the author commits it. + +To show the change, run `git status --short -- ` with every path the Summary lists under `files changed`, and show the output; then run `git diff HEAD -- ` for the tracked paths and show it; then, for each path `git status` marks `??`, run `git diff --no-index -- /dev/null ` and show it (its exit status 1 is normal). Never run `git add`, not even with `-N`: committing is the author's. If `files changed` is empty, show nothing and say so. + +When the Summary comes back, read its `status` first. + +- **`done`**: if any check under `checks run` failed, say so plainly before anything else, naming the check and its result. Then show the Summary verbatim and show the change. Then say that the change sits uncommitted on the current branch, naming that branch, and that if it is the default branch the author creates the branch proposed in step 1 and commits before running `/freethinking:pr`. Then recommend `/freethinking:pr` to open the pull request, naming any failed check again in that recommendation. +- **`stopped`**: show the Summary verbatim, then show the change for anything it lists under `files changed`, and say plainly that those partial edits are in the working tree. Then put one question with the AskUserQuestion tool, header `Relaunch`, with two options in this order: `Answer and relaunch`, to answer the question under `stopped because` and launch the executor again, and `Spec_Route`, to write a spec for the change instead. A free-text answer counts as `Answer and relaunch`, with that text as the answer; if the author picks `Answer and relaunch` with no text, ask for the answer in prose, once, before relaunching. Act only on the author's answer. On `Answer and relaunch`, launch once more with the same prompt, adding under Author's additions the executor's `stopped because` text, quoted, then the author's answer, then the files that already hold partial edits, if any; the launch still never says that the task comes from a spec. Handle that Summary the same way. On `Spec_Route`, say again in one sentence that any partial edits are still in the working tree, then go on to step 4. Never relaunch unprompted. + +## 4. Write the requirements + +Ask the author only what you still need, in one round, in prose: - What the task must do, for whom, and what value it delivers. - What is out of scope. - Constraints: deadlines, compatibility, performance, security, systems it must work with. - Anything the author already knows they want or do not want in the solution. -## 3. Write the requirements - -Read `${CLAUDE_SKILL_DIR}/spec-template/requirements.md`, then write `.claude/specs//requirements.md` with: +Then read `${CLAUDE_SKILL_DIR}/spec-template/requirements.md` and write `.claude/specs//requirements.md` with: - **Introduction**: what the feature is, who uses it, what value it delivers. A short paragraph. No implementation detail. - **Glossary**: every domain noun the requirements use, each defined in one sentence, in `Term_Name` form. Use the terms verbatim in the criteria. - **Requirements**: numbered, each with a user story ("As a ROLE, I want CAPABILITY, so that BENEFIT") and acceptance criteria in EARS form: ubiquitous (`THE System SHALL`), event-driven (`WHEN ..., THE System SHALL`), state-driven (`WHILE ..., THE System SHALL`), optional-feature (`WHERE ..., THE System SHALL`) and unwanted-behaviour (`IF ..., THEN THE System SHALL`). One behaviour per criterion. Quantify what can be quantified. Every requirement covers its unhappy path. -Then show the author the document and ask: do these requirements say what you want built, is anything missing, and is anything there that should not be? Iterate until approved. +Then show the author the document and ask with the AskUserQuestion tool, header `Approve`, options `Approved` and `Changes needed`, whether these requirements say what they want built, with anything missing or wrong in the free text. Iterate until approved. -## 4. Write the design +## 5. Write the design Read `${CLAUDE_SKILL_DIR}/spec-template/design.md`. Read the parts of the codebase the design will touch, so the design fits what exists rather than inventing a parallel structure. Then write `design.md` with: @@ -55,9 +108,9 @@ Read `${CLAUDE_SKILL_DIR}/spec-template/design.md`. Read the parts of the codeba - **Error Handling**: a table of condition, category, behaviour and requirement. - **Testing Strategy**: property-based tests for the properties, example and edge-case tests for the error rows, integration tests for the wiring. -Check traceability both ways before showing it: every requirement is satisfied somewhere in the design, and every decision traces to a requirement. Name anything that does not trace. Ask the author to approve; iterate until they do. +Check traceability both ways before showing it: every requirement is satisfied somewhere in the design, and every decision traces to a requirement. Name anything that does not trace. Then show the author the document and ask with the AskUserQuestion tool, header `Approve`, options `Approved` and `Changes needed`, whether this design is how they want the requirements met, with anything missing or wrong in the free text. Iterate until approved. -## 5. Write the task list +## 6. Write the task list Read `${CLAUDE_SKILL_DIR}/spec-template/tasks.md`, then write `tasks.md` with: @@ -66,8 +119,8 @@ Read `${CLAUDE_SKILL_DIR}/spec-template/tasks.md`, then write `tasks.md` with: - **Notes**: conventions the executor should follow. - **Task Dependency Graph**: the JSON `waves` block, where each wave depends only on earlier waves. Every leaf task appears exactly once, optional test tasks included. Task numbers only identify tasks; the waves decide the order of execution. -Ask the author to approve; iterate until they do. +Then show the author the document and ask with the AskUserQuestion tool, header `Approve`, options `Approved` and `Changes needed`, whether these tasks build the design in an order they accept, with anything missing or wrong in the free text. Iterate until approved. -## 6. Hand off +## 7. Hand off When `tasks.md` is approved, tell the author where the three files are and how to proceed: run `/freethinking:implement-spec `, which builds each task in its own sub-agent in wave order, ticks each checkbox when its tests pass, runs the full suite at the end of each wave, and stops for review after each task or each wave, as the author chooses. Point at the first wave. Then stop. From f2eba2fa5faec983c3199e1675b506d7a35332ff Mon Sep 17 00:00:00 2001 From: Dan Sabin Date: Tue, 13 Oct 2026 13:02:32 -0500 Subject: [PATCH 3/3] start-task asks whether a request fits the repository before speccing it Found in the manual session: asked for a link shortener in a two-function arithmetic package, start-task noted that nothing there would be extended and then went straight to the spec questions. Triage now has a fit check: when the request builds on no module, pattern or test in the codebase, the skill says so and asks, before any spec question, whether to spec it here anyway or stop because this is the wrong repository. New eval case wrong-repo-asks-fit; all four start-task cases pass two runs. Spec records the eval and manual runs, and the spec is complete. Co-Authored-By: Claude Fable 5.1 --- .claude/specs/standalone-task-routing/design.md | 4 +++- .claude/specs/standalone-task-routing/requirements.md | 1 + .claude/specs/standalone-task-routing/tasks.md | 6 ++++-- evals/skills/start-task/wrong-repo-asks-fit/case.yaml | 4 ++++ .../start-task/wrong-repo-asks-fit/graders/no-skill.md | 7 +++++++ .../wrong-repo-asks-fit/graders/no-spec-written.md | 5 +++++ .../wrong-repo-asks-fit/graders/no-sub-agent.md | 7 +++++++ .../wrong-repo-asks-fit/graders/offers-spec-here.md | 5 +++++ .../graders/offers-wrong-repository.md | 5 +++++ evals/skills/start-task/wrong-repo-asks-fit/prompt.md | 8 ++++++++ evals/skills/start-task/wrong-repo-asks-fit/scaffold.sh | 6 ++++++ skills/start-task/SKILL.md | 4 +++- 12 files changed, 58 insertions(+), 4 deletions(-) create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/case.yaml create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/no-skill.md create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/no-spec-written.md create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/no-sub-agent.md create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/offers-spec-here.md create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/graders/offers-wrong-repository.md create mode 100644 evals/skills/start-task/wrong-repo-asks-fit/prompt.md create mode 100755 evals/skills/start-task/wrong-repo-asks-fit/scaffold.sh diff --git a/.claude/specs/standalone-task-routing/design.md b/.claude/specs/standalone-task-routing/design.md index 7af1a91..d174a52 100644 --- a/.claude/specs/standalone-task-routing/design.md +++ b/.claude/specs/standalone-task-routing/design.md @@ -95,7 +95,7 @@ CX_Block, the section each skill opens with. For `start-task`: | Bullet | Content | |--------|---------| -| Stops | Triage Question when the Route is Bug or Executor; approval of each of the three documents; the relaunch Question after a `stopped` Summary | +| Stops | Triage Question when the Route is Bug or Executor; the fit Question when the Request extends nothing in the codebase; approval of each of the three documents; the relaunch Question after a `stopped` Summary | | How it asks | AskUserQuestion with fixed options plus free text at every stop; the spec questions of step 3 in prose, because the answers are the Author's own words | | Hand-backs | On Executor_Route: the Summary verbatim, the diff, failures first; on Spec_Route: each document in full | | Hand-offs | `bugfix` on Bug_Route; `pr` after a `done` Summary; `implement-spec` after `tasks.md` is approved. The branch name proposed in step 1 is created by the author, or by `implement-spec` at its set-up, never by this skill | @@ -155,6 +155,7 @@ Rule text, `rules/collaboration.md`, one bullet per criterion 6.2 to 6.7, each b | Spec directory already exists for the task name | RESUME | Spec_Route, no Triage, resume at first unapproved document | 1.7 | | Author picks a different Route than recommended | OVERRIDE | Take the Author's Route, no argument | 1.6 | | Author's free-text answer is neither Route | UNCLEAR | Restate the two Routes in one sentence and ask again, once | 1.4 | +| Request extends nothing in the codebase | WRONG_PLACE | Say so in the route statement; Question with header `Fit`: `Spec it here` and `Wrong repository`; on the second, stop and say where it belongs if known | 1.11 | | Executor returns `stopped` | DRIFT | Show Summary; Question: answer and relaunch, or Spec_Route; no unprompted relaunch | 3.5 | | Executor returns `done` with a failed check | FAILED_CHECK | Say the failure first, show the Summary and diff, then recommend `pr` with the failure named | 3.6 | | AskUserQuestion is not available (headless run) | FALLBACK | Ask the same question in prose with the same options on separate lines, then wait | 6.2, 6.10 | @@ -164,6 +165,7 @@ Rule text, `rules/collaboration.md`, one bullet per criterion 6.2 to 6.7, each b - Free checks, run by `make validate`: the existing `checks/test_issue_forms.py` proves Property 5 once `collaboration (rule)` is in the forms; a new check in `checks/test_rules_size.py` proves Property 4 by running `hooks/load-rules.sh` and measuring, and by measuring `rules/collaboration.md` alone. The size check also closes issue #3's first half. `claude plugin validate --strict` proves the frontmatter. - Eval case `contained-change-offers-executor` (Property 1, 3; Requirement 5.3): scaffold the calc fixture; prompt `/freethinking:start-task Add subtract(a, b) to calc/ops.py in the style of add`; `allowed_tools: [Read, Glob, Grep, Agent, Skill]`; `max_turns: 6`. Write and Edit come from the Makefile target's `--allow-tools` grant, the only route for them per `evals/README.md`, on purpose: a skill that writes a spec document or edits source before the Question must be caught by the `file_exists` and file-content graders, not prevented by the tool list. Graders: regex `executor` with `flags: i` in the final output (the harness's engine rejects inline `(?i)`); `file_exists` `.claude/specs/**` false; `tool_used` Agent with max 0; `tool_used` Skill with max 0; and the question grader, which task 3.1 settled as two regexes with `flags: im`, one per option as a list item at a line start. No `\?` grader: the prose fallback does not always end in a question mark, and the two option graders are the stronger proof that the question was put. +- Eval case `wrong-repo-asks-fit` (Requirement 1.11): scaffold the calc fixture; prompt the link shortener; graders: the two fit options as list items (`Spec it here`, `Wrong repository`), `file_exists` `.claude/specs/**` false, `tool_used` Agent and Skill max 0. - Eval case `bug-offers-bugfix` (Property 1; Requirement 5.4): same fixture with `add` returning `a - b`; prompt `/freethinking:start-task Adding 2 and 3 in calc returns -1`; graders: regex `Bug_Route|bugfix`; `file_exists` `**/*.py` unchanged by a regex on `calc/ops.py` still containing the planted body; `tool_used` Skill max 0; the same question grader. - Eval case `short-list-asks-with-tool` under `evals/rules/collaboration/` (Requirement 6.10): no plugin skill invoked; prompt asks Claude to pick between three named options for a config value it cannot infer; grader as the question grader above. - Existing cases: `names-task-and-asks` run twice; `eval-rule-workflow` run twice; `eval-skill-start-task` twice. Each result table goes into the PR's Testing section. diff --git a/.claude/specs/standalone-task-routing/requirements.md b/.claude/specs/standalone-task-routing/requirements.md index 508c21a..538f45d 100644 --- a/.claude/specs/standalone-task-routing/requirements.md +++ b/.claude/specs/standalone-task-routing/requirements.md @@ -41,6 +41,7 @@ 7. WHEN a spec for the Request already exists in `.claude/specs//`, THE Skill SHALL take Spec_Route and resume at the first unapproved document, without Triage, since the sizing decision was already made. 8. THE Skill SHALL open with a CX_Block, placed after its ground rules, that lists every point at which it stops for the Author (the Question at Triage, then approval of each document), says that each stop asks with the AskUserQuestion tool except where the answer is the Author's own words, says what each hand-back contains, and names the skills it hands off to. 9. THE CX_Block SHALL state only what is specific to the Skill or departs from the Collaboration_Rule, and SHALL NOT restate the Collaboration_Rule. +11. WHEN Triage finds that the Request would extend nothing in the codebase (no module, pattern or test it builds on), THE Skill SHALL say so in the route statement and put a Question, before any spec question, offering to write the spec here anyway or to stop because this is the wrong repository, and on the second answer SHALL stop after saying where the work belongs if the Request or the code says. 10. WHEN the Skill derives the task name, THE Skill SHALL propose a branch name in the contributing guide's shape, the issue number and the task name when the Request names or links an issue and the task name alone otherwise, and SHALL NOT create the branch. ### Requirement 2: Bug_Route diff --git a/.claude/specs/standalone-task-routing/tasks.md b/.claude/specs/standalone-task-routing/tasks.md index f045845..ac05e69 100644 --- a/.claude/specs/standalone-task-routing/tasks.md +++ b/.claude/specs/standalone-task-routing/tasks.md @@ -27,6 +27,7 @@ Rules first, because the skill's CX block refers to the collaboration rule and t - [x] 2.1 Add the CX block and the triage step to `skills/start-task/SKILL.md` - Frontmatter `description`: add that the skill is the front door for any request to change code and decides between the executor, bugfix and a spec. After "Ground rules", a section "Working with the author" with the design's four bullets: stops, how it asks (AskUserQuestion with fixed options at every stop; the step 3 spec questions in prose), hand-backs, hand-offs; state only what is specific to this skill. In step 1, say that an existing spec directory means Spec_Route with no triage, and that beside the task name the skill proposes a branch name in the contributing guide's shape (issue number and task name when the request names or links an issue, task name alone otherwise) without creating it; the CX block's hand-offs bullet says the author or `implement-spec` creates it. Split step 2 into "2. Read the code" (today's first paragraph) and "3. Triage" with: the three routes and the Contained_Change and defect definitions from the requirements glossary; state the chosen route and the reason in two or three sentences naming the code read; on Bug_Route or Executor_Route put one AskUserQuestion, header `Route`, chosen route first marked "(Recommended)", `Spec_Route` second, each with a one-sentence description, and act only on the answer; take the author's route without arguing; on Spec_Route say so in one sentence and continue; if the free-text answer is neither route, restate the two in a sentence and ask once more. Renumber the later steps and the spec questions move to the head of the requirements step. Run `make validate` - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10_ + - Wave 4 fix, found in the manual session: a fit check in triage. When the request extends nothing in the codebase, the route statement says so and a question with header `Fit` offers `Spec it here` and `Wrong repository` before any spec question; on the second the skill stops and says where the work belongs if known. The CX block's Stops bullet lists it. Eval case `evals/skills/start-task/wrong-repo-asks-fit/`: calc scaffold, the link-shortener prompt, graders for the two options as list items, no spec written, no Agent or Skill call (Requirement 1.11) - [x] 2.2 Write the Bug_Route and Executor_Route paths in the triage step - Bug_Route: invoke the Skill tool with `freethinking:bugfix` and the request as its argument, then stop; do not reproduce, diagnose or fix. Executor_Route: launch the Agent tool once with `subagent_type: freethinking:task-executor` and the design's Launch_Prompt (request verbatim, context found as `path: what is there`, the author's free-text additions, then "Make only this change. Commit nothing. Return the Summary."), with no sentence saying the task comes from a spec; edit no source file, commit nothing, create no branch. On `status: done`: show the Summary verbatim, run `git diff -- ` and show it, name any failed check before anything else, then recommend `/freethinking:pr`. On `status: stopped`: show the Summary and put one AskUserQuestion offering "answer and relaunch" and "Spec_Route"; never relaunch unprompted. If either plugin component is missing, say so and stop. Run `make validate` - _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7_ @@ -61,15 +62,16 @@ Rules first, because the skill's CX block refers to the collaboration rule and t - `.claude-plugin/plugin.json` version to `0.6.0`. Run `make validate` - _Requirements: 5.2_ -- [ ] 5. Runs +- [x] 5. Runs - [x] 5.1 Run the evals - `make eval-skill-start-task EVAL_ARGS="--runs 2"`, `make eval-rule-workflow EVAL_ARGS="--runs 2"`, `make eval-rule-collaboration EVAL_ARGS="--runs 2"`. Fix the case or the skill until each case passes both runs, and record the three result tables for the PR's Testing section - Run on 2026-10-13: first pass found the harness rejects inline `(?i)` (two graders switched to `flags: i`) and that the prose route question does not always end in `?` (the `\?` grader dropped from the two route cases, whose option graders prove the question). Second pass: all nine cases 100% on both runs, $2.18 total - _Requirements: 5.3, 5.4, 5.5, 5.6, 6.10_ - - [ ] 5.2 Manual session, done by the author + - [x] 5.2 Manual session, done by the author - In a session started with `claude --plugin-dir .` in a scratch copy of `evals/fixtures/calc`, run the contained-change prompt, the bug prompt and the link-shortener prompt by hand. Confirm the Route question renders as the interactive form, that answering Spec_Route continues to the spec questions, and that accepting Executor_Route ends with the Summary, the diff and the `pr` recommendation. Record what was seen in the PR - **Property 2: The Author's answer is the Route** - _Requirements: 1.4, 1.6, 3.4, 5.7_ + - Run on 2026-10-13. Headless first, with `claude -p --plugin-dir` against four calc workspaces: the contained change answered Executor_Route ran the executor and ended with the Summary, the diff, the branch-and-commit line and the `pr` recommendation; the same change answered Spec_Route went to four task-sized spec questions; the unnamed defect named `add` at `calc/ops.py:2`, offered Bug_Route, and after the answer `bugfix` reproduced it and stopped to confirm the cause; the link shortener went straight to spec questions, which exposed the missing fit check, added as Requirement 1.11 with its eval case. Then interactively by the author: the Fit form rendered and Wrong repository stopped with nothing written; the Route form rendered and Executor_Route ran through to the `pr` recommendation ## Notes diff --git a/evals/skills/start-task/wrong-repo-asks-fit/case.yaml b/evals/skills/start-task/wrong-repo-asks-fit/case.yaml new file mode 100644 index 0000000..cae5641 --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: wrong-repo-asks-fit +context: + scaffold_script: scaffold.sh diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/no-skill.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-skill.md new file mode 100644 index 0000000..a01eeac --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-skill.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/no-spec-written.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-spec-written.md new file mode 100644 index 0000000..7a79050 --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-spec-written.md @@ -0,0 +1,5 @@ +--- +type: file_exists +path: '.claude/specs/**' +exists: false +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/no-sub-agent.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-sub-agent.md new file mode 100644 index 0000000..94838dd --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/no-sub-agent.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Agent +min: 0 +max: 0 +arm: both +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-spec-here.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-spec-here.md new file mode 100644 index 0000000..2faa34b --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-spec-here.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Spec it here\b' +flags: im +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-wrong-repository.md b/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-wrong-repository.md new file mode 100644 index 0000000..418a366 --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/graders/offers-wrong-repository.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '^\s*(?:[-*•]|\d+[.)])\s*[*_`]*Wrong repository\b' +flags: im +--- diff --git a/evals/skills/start-task/wrong-repo-asks-fit/prompt.md b/evals/skills/start-task/wrong-repo-asks-fit/prompt.md new file mode 100644 index 0000000..d4eca1b --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/prompt.md @@ -0,0 +1,8 @@ +--- +description: Given a request that extends nothing in the calc fixture, start-task says so and asks whether to spec it here or stop because this is the wrong repository, before any spec question, writing nothing. Needs the Makefile's skill target, which runs the scaffold. +tags: [skill, start-task] +max_turns: 6 +allowed_tools: [Read, Glob, Grep, Agent, Skill] +expected_outcome: Claude reads the calc fixture, says in two or three sentences that a link shortener extends nothing in it (no module, pattern or test it builds on), and asks the fit question with Spec it here and Wrong repository as options, before any spec question. It writes nothing under .claude/specs/, edits no code, and launches no sub-agent or skill. +--- +/freethinking:start-task Link shortener: users submit a long URL and get a short code; visiting the code redirects and counts the click diff --git a/evals/skills/start-task/wrong-repo-asks-fit/scaffold.sh b/evals/skills/start-task/wrong-repo-asks-fit/scaffold.sh new file mode 100755 index 0000000..d087bf2 --- /dev/null +++ b/evals/skills/start-task/wrong-repo-asks-fit/scaffold.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Builds the workspace for this case: the calc fixture on main, with no spec +# files, so start-task has code to read and a branch to propose. +set -euo pipefail +ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) +bash "$ROOT/evals/fixtures/setup.sh" calc main diff --git a/skills/start-task/SKILL.md b/skills/start-task/SKILL.md index ee018ba..f78f96a 100644 --- a/skills/start-task/SKILL.md +++ b/skills/start-task/SKILL.md @@ -20,7 +20,7 @@ Task, as the author gave it: $ARGUMENTS ## Working with the author -- **Stops.** The route question at triage, when the route is Bug_Route or Executor_Route; the approval of each of the three documents on Spec_Route; the relaunch question after a `stopped` Summary on Executor_Route. +- **Stops.** The route question at triage, when the route is Bug_Route or Executor_Route; the fit question at triage, when the request extends nothing in the codebase; the approval of each of the three documents on Spec_Route; the relaunch question after a `stopped` Summary on Executor_Route. - **How it asks.** With the AskUserQuestion tool, fixed options plus free text, at every stop. The spec questions at the head of the requirements step are the exception: ask them in prose, because the answers are the author's own words. - **Hand-backs.** On Executor_Route, the executor's Summary verbatim and the diff, failures first. On Spec_Route, each document in full. - **Hand-offs.** `bugfix` on Bug_Route; `pr` after a `done` Summary; `implement-spec` after `tasks.md` is approved. The branch name proposed in step 1 is created by the author before the first commit, or by `implement-spec` at its set-up when the author gives it that name; never by this skill. @@ -45,6 +45,8 @@ Before asking any spec question or writing any document, decide which route the State the route you chose and the reason for it in two or three sentences that name the code you read; on Spec_Route, one sentence that names the route is enough. +If step 2 found nothing the request would extend, no module, pattern or test it builds on, say so in the route statement, then, before any other question, put one question with the AskUserQuestion tool, header `Fit`, with two options in this order: `Spec it here`, to write the spec in this repository anyway, and `Wrong repository`, to stop, each with a one-sentence description. Act only on the author's answer. On `Wrong repository`, say where the work belongs if the request or the code says, then stop and write nothing. On `Spec it here`, go on with the route, asking the route question after this one when the route has one, since there is no point sizing work that does not belong here. + On Bug_Route or Executor_Route, put one question to the author with the AskUserQuestion tool, header `Route`, with two options in this order: the route you chose, marked "(Recommended)", then `Spec_Route`, each with a one-sentence description. The tool adds the free-text answer itself. Act only on the author's answer. If the author picks the other route, take it without arguing for yours. If the free-text answer is neither route, restate the two routes in one sentence and ask once more. On Spec_Route, go on to step 4 without a question.