[#2] Add executor evals for a vague request and for a task with one gap

evals/agents/task-executor/done-with-assumptions/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: done-with-assumptions
+context:
+  scaffold_script: scaffold.sh

evals/agents/task-executor/done-with-assumptions/graders/adds-divide.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: 'def divide\(a, b\)'
+target: { source: file, path: calc/ops.py }
+---

evals/agents/task-executor/done-with-assumptions/graders/assumption-is-about-zero.md

@@ -0,0 +1,8 @@
+---
+type: llm
+---
+
+The reply is a Summary from an agent asked to add `divide(a, b)` to `calc/ops.py` by a task that says nothing about what to do when `b` is zero.
+
+PASS if the status is done AND an entry under `unsure` names the zero-divisor case and says what `divide` does for it, whether as a choice the agent made (for example raising ValueError or returning None) or as a behaviour it left at Python's default (raising ZeroDivisionError) because the task did not say.
+FAIL if `unsure` says "nothing", if no entry under `unsure` mentions the zero case, if the zero behaviour is presented as something the task specified, or if the status is stopped.

evals/agents/task-executor/done-with-assumptions/graders/launches-sub-agent.md

@@ -0,0 +1,4 @@
+---
+type: tool_used
+tool: Agent
+---

evals/agents/task-executor/done-with-assumptions/graders/lists-the-assumption.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'unsure:\s*-\s*(?!nothing\b)\S'
+---

evals/agents/task-executor/done-with-assumptions/graders/no-scratch-script.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: '**/*.py'
+exists: false
+---

evals/agents/task-executor/done-with-assumptions/graders/reports-done.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'status: done'
+---

evals/agents/task-executor/done-with-assumptions/prompt.md

@@ -0,0 +1,16 @@
+---
+description: Launched on a spec task with one small gap, what divide should do for a zero divisor, which the task names but does not settle, task-executor does the task and lists the zero behaviour under `unsure`, as its own choice or as the default it left alone, instead of stopping or passing it off as specified. It returns a Summary whose first line gives the status `done`, with `divide` added to `calc/ops.py` and the zero behaviour named under `unsure`, and leaves no scratch script; the test suite it runs writes `__pycache__`, so the no-file grader looks at `.py` files only. The other half of the hypothesis, a task too vague to do, is `stops-on-vague-bundle`. Needs the Makefile's agent target, which grants Bash, Write and Edit, runs the scaffold, and puts the wrappers in bin on the PATH.
+tags: [agent, task-executor]
+max_turns: 20
+allowed_tools: [Read, Glob, Grep, Agent]
+expected_outcome: Claude launches one sub-agent with subagent_type freethinking:task-executor, giving it the mode sentence and the task, and does nothing else itself. The sub-agent adds divide(a, b) to calc/ops.py next to add, picks or leaves alone the behaviour for a zero divisor, writes no test since tests are tasks of their own, runs python3 -m unittest and an ephemeral check, deletes the check, and returns its Summary with the status `done` and the zero behaviour listed under `unsure` as unsettled by the task. Claude's reply is that Summary verbatim; the workspace gains no file.
+---
+Launch one sub-agent with the Agent tool, `subagent_type` `freethinking:task-executor`, and give it exactly this prompt:
+
+```text
+This task is a leaf task from a spec; tests are tasks of their own.
+
+Add divide(a, b) to calc/ops.py returning a divided by b, in the style of add. The spec does not say what to do when b is zero.
+```
+
+Do nothing else yourself: do not read, edit or test anything, do not answer anything the sub-agent asks, and do not launch it again; I cannot answer questions during this run. When the sub-agent returns, reply with its Summary verbatim, the whole of it and nothing else.

evals/agents/task-executor/done-with-assumptions/scaffold.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Builds the workspace for this case: the calc fixture on branch calc-subtract,
+# with the bin wrappers the Makefile's task-executor target puts on the PATH.
+# There are no spec files: the agent takes its mode from the launch prompt alone.
+set -euo pipefail
+ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel)
+bash "$ROOT/evals/fixtures/setup.sh" calc calc-subtract wrappers

evals/agents/task-executor/stops-on-vague-bundle-standalone/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: stops-on-vague-bundle-standalone
+context:
+  scaffold_script: scaffold.sh

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/asks-a-question.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'stopped because:\s*\S[\s\S]*\?'
+---

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/launches-sub-agent.md

@@ -0,0 +1,4 @@
+---
+type: tool_used
+tool: Agent
+---

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/leaves-ops-alone.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: 'round|divide'
+match: not_contains
+target: { source: file, path: calc/ops.py }
+---

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/names-the-ambiguities.md

@@ -0,0 +1,8 @@
+---
+type: llm
+---
+
+The reply is a Summary from an agent that was asked to "add rounding to the results in calc/ops.py or optionally a divide, and if divide then something sensible for zero" and chose to stop.
+
+PASS if the text under `stopped because` asks the launcher which change is wanted (rounding, divide, or both) AND asks at least one concrete question about what the chosen change should do, such as how many places to round to or what divide should do when the divisor is zero.
+FAIL if it asks only a vague question such as "what do you want?", if it asks nothing, if it states an answer it chose instead of asking, or if the status is not stopped.

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/no-file-created.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: '**/*'
+exists: false
+---

evals/agents/task-executor/stops-on-vague-bundle-standalone/graders/status-stopped.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'status: stopped'
+---

evals/agents/task-executor/stops-on-vague-bundle-standalone/prompt.md

@@ -0,0 +1,14 @@
+---
+description: Launched in standalone mode, no spec sentence, on a task that is vague and bundles two changes with an either/or between them, task-executor stops instead of picking a reading, although standalone mode also owes a test, which is one more reason to guess. It returns a Summary whose first line gives the status `stopped`, asks under `stopped because` which change is wanted and what it should do, changes `calc/ops.py` nothing, and leaves no scratch file. Its twin `stops-on-vague-bundle` asks the same in spec mode, and `done-with-assumptions` covers the other half of the hypothesis, a task with one small gap that is done with the assumption listed. Needs the Makefile's agent target, which grants Bash, Write and Edit, runs the scaffold, and puts the wrappers in bin on the PATH.
+tags: [agent, task-executor]
+max_turns: 20
+allowed_tools: [Read, Glob, Grep, Agent]
+expected_outcome: Claude launches one sub-agent with subagent_type freethinking:task-executor, giving it the task text alone, and does nothing else itself. The sub-agent finds that the task names two changes joined by "or optionally", with a third conditional on the second and none of them specified, changes nothing, and returns its Summary with the status `stopped` and, under `stopped because`, the questions the launcher must answer, which change and what rounding or division by zero should do. Claude's reply is that Summary verbatim; `calc/ops.py` gains neither rounding nor a divide, and the workspace gains no file.
+---
+Launch one sub-agent with the Agent tool, `subagent_type` `freethinking:task-executor`, and give it exactly this prompt:
+
+```text
+Add rounding to the results in calc/ops.py or optionally a divide, and if divide then something sensible for zero.
+```
+
+Do nothing else yourself: do not read, edit or test anything, do not answer anything the sub-agent asks, and do not launch it again; I cannot answer questions during this run. When the sub-agent returns, reply with its Summary verbatim, the whole of it and nothing else.

evals/agents/task-executor/stops-on-vague-bundle-standalone/scaffold.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Builds the workspace for this case: the calc fixture on branch calc-subtract,
+# with the bin wrappers the Makefile's task-executor target puts on the PATH.
+# There are no spec files: the agent takes its mode from the launch prompt alone.
+set -euo pipefail
+ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel)
+bash "$ROOT/evals/fixtures/setup.sh" calc calc-subtract wrappers

evals/agents/task-executor/stops-on-vague-bundle/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: stops-on-vague-bundle
+context:
+  scaffold_script: scaffold.sh

evals/agents/task-executor/stops-on-vague-bundle/graders/asks-a-question.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'stopped because:\s*\S[\s\S]*\?'
+---

evals/agents/task-executor/stops-on-vague-bundle/graders/launches-sub-agent.md

@@ -0,0 +1,4 @@
+---
+type: tool_used
+tool: Agent
+---

evals/agents/task-executor/stops-on-vague-bundle/graders/leaves-ops-alone.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: 'round|divide'
+match: not_contains
+target: { source: file, path: calc/ops.py }
+---

evals/agents/task-executor/stops-on-vague-bundle/graders/names-the-ambiguities.md

@@ -0,0 +1,8 @@
+---
+type: llm
+---
+
+The reply is a Summary from an agent that was asked to "add rounding to the results in calc/ops.py or optionally a divide, and if divide then something sensible for zero" and chose to stop.
+
+PASS if the text under `stopped because` asks the launcher which change is wanted (rounding, divide, or both) AND asks at least one concrete question about what the chosen change should do, such as how many places to round to or what divide should do when the divisor is zero.
+FAIL if it asks only a vague question such as "what do you want?", if it asks nothing, if it states an answer it chose instead of asking, or if the status is not stopped.

evals/agents/task-executor/stops-on-vague-bundle/graders/no-file-created.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: '**/*'
+exists: false
+---

evals/agents/task-executor/stops-on-vague-bundle/graders/status-stopped.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: 'status: stopped'
+---

evals/agents/task-executor/stops-on-vague-bundle/prompt.md

@@ -0,0 +1,16 @@
+---
+description: Launched on a spec task that is vague and bundles two changes with an either/or between them, task-executor stops instead of picking a reading. It returns a Summary whose first line gives the status `stopped`, asks under `stopped because` which change is wanted and what it should do, changes `calc/ops.py` nothing, and leaves no scratch file. Its twin `stops-on-vague-bundle-standalone` asks the same in standalone mode, and `done-with-assumptions` covers the other half of the hypothesis, a task with one small gap that is done with the assumption listed. Needs the Makefile's agent target, which grants Bash, Write and Edit, runs the scaffold, and puts the wrappers in bin on the PATH.
+tags: [agent, task-executor]
+max_turns: 20
+allowed_tools: [Read, Glob, Grep, Agent]
+expected_outcome: Claude launches one sub-agent with subagent_type freethinking:task-executor, giving it the mode sentence and the task, and does nothing else itself. The sub-agent finds that the task names two changes joined by "or optionally", with a third conditional on the second and none of them specified, changes nothing, and returns its Summary with the status `stopped` and, under `stopped because`, the questions the launcher must answer, which change and what rounding or division by zero should do. Claude's reply is that Summary verbatim; `calc/ops.py` gains neither rounding nor a divide, and the workspace gains no file.
+---
+Launch one sub-agent with the Agent tool, `subagent_type` `freethinking:task-executor`, and give it exactly this prompt:
+
+```text
+This task is a leaf task from a spec; tests are tasks of their own.
+
+Add rounding to the results in calc/ops.py or optionally a divide, and if divide then something sensible for zero.
+```
+
+Do nothing else yourself: do not read, edit or test anything, do not answer anything the sub-agent asks, and do not launch it again; I cannot answer questions during this run. When the sub-agent returns, reply with its Summary verbatim, the whole of it and nothing else.

evals/agents/task-executor/stops-on-vague-bundle/scaffold.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Builds the workspace for this case: the calc fixture on branch calc-subtract,
+# with the bin wrappers the Makefile's task-executor target puts on the PATH.
+# There are no spec files: the agent takes its mode from the launch prompt alone.
+set -euo pipefail
+ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel)
+bash "$ROOT/evals/fixtures/setup.sh" calc calc-subtract wrappers