[sabind]

Why

stops-when-unclear covers one kind of stop, a task naming a module that does not exist. Nothing covered a request that is vague and bundles two changes, which the agent's text handles only through the general "do not assume, stop and ask" sentence.

Closes #2

What

Three task-executor eval cases on the calc fixture test both halves of the hypothesis in #2: a request too vague to do makes the agent stop with the questions the launcher must answer, and a task with one named gap gets done with the gap reported under unsure. Each case leaves no scratch file.

Changes

• Add evals/agents/task-executor/stops-on-vague-bundle/: a spec-mode prompt of the shape "add rounding or optionally a divide, and if divide then something sensible for zero", graded on status stopped, a question under stopped because, calc/ops.py untouched, no file created, and an llm judge on whether the questions name which change is wanted and what it should do. The existing stop case only covers a missing module; this is the vague, two-change shape from the issue.
• Add stops-on-vague-bundle-standalone/: the same prompt without the spec sentence, same graders. Standalone mode also owes a test, one more reason to guess; forcing each side of the hypothesis is how we test it.
• Add done-with-assumptions/: "add divide; the spec does not say what to do when b is zero", graded on status done, divide in calc/ops.py, an unsure entry, an llm judge that the entry names the zero behaviour as the agent's choice or as the default it left alone, and no new .py file (the test run writes __pycache__, so the file grader is narrowed). The other half of the hypothesis: a small gap is done and reported, not stopped on. The first version told the agent to say what it decided; dropping that showed the agent reports the gap on its own, and the rubric was widened to accept "left at Python's default".

Testing

make validate: 9 unit tests OK, both validators passed.

make eval-agent-task-executor EVAL_ARGS="--case <name>", one invocation per case since only the last --case flag is honoured:

CASE	SCORE	PASS%	RUNS	COST
stops-on-vague-bundle	1.00	100%	3	$0.54
stops-on-vague-bundle-standalone	1.00	100%	3	$0.52
done-with-assumptions	1.00	100%	3	$0.61

Every grader passed in every run; both llm judges voted PASS three of three each time. Earlier single runs during iteration: the first vague-bundle version passed twice, with a kept transcript showing a numbered list of questions and no change; the done case failed once on **/* because the test run writes __pycache__, and once on the zero rubric when the agent reported the zero case as left at the default rather than chosen. Both were grader fixes, and the kept directories were deleted.

Risk and rollout

None. The cases run by hand through the existing eval-agent-task-executor target; no Makefile change. Each run costs about twenty cents, the agent run itself rather than the judges.

Notes for reviewers

The standalone twin doubles the cost of the stop hypothesis for one extra fact. If it stays green across changes to the agent, it can be dropped or run only when the agent's Mode section changes. Follow-ups from this work are #11, #12 and #13.

🤖 Generated with Claude Code