Repository navigation
bugfix skips its diagnosis and Summary when every answer is given in the request #27
Description
Before you file
- I searched the existing issues.
- I looked in Discussions under Ideas.
Component
bugfix (skill)
Plugin version
0.8.0, branch pick-up-issues-12-17, the PR for #12 and #17.
Prompt
The eval case evals/skills/bugfix/hands-off-to-pr/prompt.md, run with make eval-skill-bugfix EVAL_ARGS="--runs 1 --keep-temp --case hands-off-to-pr": the calc fixture with add returning a - b, ticket #7, and the author's three answers given up front, Go ahead, Confirmed, Open it now, with a line saying the author cannot answer during the run.
What happened
In five of eight runs on 2026-10-15 and 2026-10-16 the skill wrote no assistant text between the reproduction and the Skill tool call. It edited the fix, re-ran the tests, cleaned __pycache__, ran git branch --show-current and called freethinking:pr with #7. No root cause was stated at step 4 and no Summary was written at step 7, under its labels or in other words. In the last failing run the skill's first message said "Root cause is Confirmed (I'll still write out the diagnosis and evidence first)" and then it did not. Failing runs take nine or ten turns, passing runs eleven. Everything else passed in every run: the fix landed, the ticket reached pr, and there was no commit, branch, push or gh pr create.
What was tried, none of which held:
- three wordings in step 7: the Summary as its own labelled message; "an advance answer skips the wait, never the Summary"; "if the Summary has not yet been written when an answer is read, write it first";
- the collaboration-rule bullet "Answers in the request count, steps still run", which the skill acknowledged in its first message and then did not follow;
- a prompt saying the run would be judged afterwards as a whole, as if the author could review the findings.
Results folders under evals/results/ from 2026-10-15T21-45-41-099Z to 2026-10-16T15-02-54-984Z. The kept trace of the last failure was read and deleted.
What you expected
The diagnosis with its evidence before the edit, and the four-part Summary (Root cause, Fix, Prevention, Verification) before the Skill call, as steps 4 and 7 say, whether or not the answers were given in advance.
Does it reproduce?
Sometimes: five of eight runs.
Notes
The eval case ships without a Summary grader for this reason, and the manual check recorded in the PR proves the Summary with the author present. Two leads: place the outputs where the tool calls happen, the diagnosis as the stated reason for the edit in step 5 and the Summary as the first of two ordered actions on Open it now; and test the Summary after a real confirmation with a resumed transcript, which is the subject of #28.
Activity
sabind commentedon Oct 16, 2026
Diagnosis from two runs of the skill on this issue, made as the manual check of the PR for #12 and #17 on 2026-10-16, on branch
pick-up-issues-12-17with the plugin loaded from the working tree.Where the text goes. Both runs reran the eval case
hands-off-to-prwith traces kept and read them. In both, the diagnosis and the Summary were written as the model's private thinking in the moment before the Edit and the moment before the Skill call, never as a message. Verbatim from one trace, right before the Edit ofcalc/ops.py:and right before the Skill call:
Total thinking across a run was about five hundred tokens, so these were brief private notes, not a drafted message that got lost. The run's first message had said "I'll still write out the diagnosis and the summary before acting on those answers", so the step 1 sentence and the collaboration-rule bullet were read and agreed with; the hook had loaded the rule, and the skill text in the sandbox was current.
Root cause. In the skill, the diagnosis (step 4) and the Summary (step 7) exist only as the lead-in to a question: they are the text the author reads before answering
Root causeandPull request. When the answers come in advance, the question is never asked, and the model drops the whole stop block, the question and the text attached to it, keeping only the reasoning it needs to act. Nothing at the two actions themselves, the Edit in step 5 and the Skill call in step 7, requires a sent message first. That is why every wording tried so far failed: each told the skill to write the Summary in relation to the question or the answer ("before the question", "once the Summary is in the conversation", "steps still run"), none in relation to the action. Both runs reached this from different starting points, one from a lead naming step 7, and both found step 4 has the same shape and the same loss.Ruled out. That the rules were not loaded in the eval sandbox: the rule's text is in both traces. That the skill text was stale: the first message reads the argument exactly as the new step 1 says. That step 7 alone is the cause: step 4 loses its output the same way.
Inferred, not observed. That the collapse is driven by the absence of any stop before the tool call, rather than by the prompt's "do not wait for me", is a reading of the pattern; the traces do not record which instruction the model weighed last.
Fix proposed by both runs. Tie each output to its action, not to its question. In step 5, the first fixing edit is made in the same message as the stated root cause and its evidence, and is not made without it. In step 7,
Open it nowbecomes two ordered actions, the Summary message first and the Skill call second, so the Skill call reads the Summary as its precondition. Prevention: the four-label Summary grader goes back intoevals/skills/bugfix/hands-off-to-pr/once the fix holds (it was removed from the case on the PR because it failed five of eight runs). #28 is the longer route, a resumed-transcript case that tests the Summary after a real confirmation.sabind commentedon Oct 16, 2026
Further evidence from the manual checks on 2026-10-16, three runs on a scratch issue with the author present and answering in the AskUserQuestion tool.
The pre-answers are not the cause. In all three runs the hand-off question was asked and the Summary was still not written before it. The question's text was "The fix is in place and verified. What should happen with it?" and the message before it was one line: "Fixed:
addincalc/ops.pyhada - binstead ofa + b(introduced in commit b3fcb4f). The existingtest_addalready catches this regression, and both tests pass now." That is the four parts compressed into one sentence, not the labelled message step 7 asks for. By then the skill text said the Summary is "its own message ... before anything else in this step" and the collaboration rule said steps still produce their output.What did appear, and where. The diagnosis appeared every time, as the Root cause question's own text ("Diagnosis:
addincalc/ops.pyline 2 returnsa - b... Is this right?"). The Summary appeared when it was the ticket comment's body (Stop and comment on the ticket), with all four parts. It never appeared as a free-standing message before a tool call, in eight eval runs (three passes) or three manual runs (none), under five wordings and a rule bullet.Reading. The model writes what something consumes and compresses the rest. A labelled four-part message that restates facts confirmed one turn earlier, immediately before a tool call, is treated as redundant whether or not the author is present.
Change made on the PR for #12 and #17. The Summary becomes the hand-off question's own text, so the author reads the four parts in the dialog before choosing and the transcript keeps them in the call; the comment body keeps them on the comment path.
What remains for this issue. The pre-answered
Open it nowpath, which is what the eval case exercises: nothing consumes the Summary there, so the eval still ships without a Summary grader. The honest fix is a consumer: a hand-off record the pr skill reads before its walkthrough, which is a pr-skill change so the record is not mistaken for the author's words. #28 (legged evals) would let the asked path be graded too.sabind commentedon Oct 16, 2026
Proposed fix, agreed on 2026-10-16 as this issue's work. Give the Summary a consumer on the pre-answered
Open it nowpath, the same principle that fixed the asked path (the Summary as the hand-off question's text) and the comment path (the comment body).A hand-off record the pr skill recognises:
argsas the ticket reference followed by the Summary under a marker, for example#12then a line--- hand-off from bugfix ---then the four labelled parts.hands-off-to-prgets its Summary grader back, on the Skill call'sargsthis time, so the pre-answered path is graded too.Rejected: posting the Summary on the ticket at
Open it nowas well, which needs no pr change since pr reads the ticket's comments, but puts a comment on every bugfix that ends in a PR, duplicating the PR description. Kept out of the PR for #12 and #17 because that spec leaves the pr skill untouched; this is a one-wave change with one eval round.sabind commentedon Oct 16, 2026
Resolved on PR #32 by a different mechanism than the hand-off record proposed above, and closed by that PR.
PR #31 added the
reportskill, which writes every hand-back from one template, and the merge of #31 into #32 made bugfix's step 7 invokefreethinking:report handbackwhen the fix is verified, putting its Message to the author as the hand-off question's text. A Skill invocation is a consumer of exactly the kind this issue showed the Summary needed: the hand-back is no longer a free-standing message before a tool call, it is the output of a skill whose whole job is writing it.Evidence on the merged text, eval case
hands-off-to-prwith every answer given up front, the path that never showed a Summary before: two new graders,invokes-report(the Skill call toreport) andgives-handback(## Donethen## Checksin the trace), passed in three of three with-plugin runs, one single run and a two-run pair against the no-plugin baseline, where the baseline wrote no hand-back. Before the merge the Summary was skipped in five of eight runs under five wordings and a rule bullet.Three runs is thin evidence, so if the hand-back goes missing again on this path in real use, reopen with the trace; the hand-off record read by the pr skill, proposed above, remains the next step if it does. #28 (legged evals) would let the asked path be graded as well.