Skip to content

fix(bgrun): discourage polling a job you just started - #27

Merged
lloydsk merged 4 commits into
mainfrom
fix/no-poll-after-bgrun
Sep 26, 2026
Merged

lloydsk merged 4 commits into
mainfrom
fix/no-poll-after-bgrun

Conversation

@lloydsk

@lloydsk lloydsk commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

Every measured session that used bgrun opened the same way — bgrun, then bgstatus, then bgtail — before the job could possibly have finished. In one run those two calls were 2,231 of 8,049 context chars (28%), spent learning what the wake delivered for free moments later.

Nothing told the agent not to: the guideline mentioned the wake only in passing, the started: result said merely "You'll be woken automatically when it finishes", and the run-bg skill licensed the polls in two places:

  • step 2: "exit: 0 → success. bgtail to confirm." — a read even when the wake already carried the counters;
  • step 3: "If you need to check before the wake (non-blocking): call bgstatus…" — framed as ordinary practice.

Change

  • BGRUN_GUIDELINES gains: "Do not poll a job you just started — no bgstatus/bgtail round after bgrun: the wake is the signal and carries the outcome (exit code, stats, the log's last line, and the digest when the project configures one). Read the log only when the wake is not enough — typically the detail behind a failure."
  • The started: result ends "…no need to poll; the wake carries the exit code, the stats and the log's last line."
  • Skill step 2 makes the wake itself the result and drops "bgtail to confirm"; step 3 is reframed — checking before the wake is rarely worth it, so reach for bgstatus when the job is not one you just started (another session's, one from before a restart, or one whose wake never came).
  • The completion wake's own trailing instruction is now conditional — see below; it was the strongest polling invitation in the system.

The bgtail/bgstatus descriptions are deliberately untouched: they describe what those tools do, not when to reach for them.

Follow-up from review (second commit)

The first commit left the contradiction in place: the wake still ended with an unconditional

Review the result now: call `bgtail` with this job id to see the output, summarize pass/fail, and continue the task that depended on it.

…emitted on exit: 0 as well. That is the last instruction the agent reads on every completion, and it asks for exactly the read this PR discourages. The smokes could not catch it — one-shot sessions exit before the wake arrives, so they only measure the pre-wake decision.

Fixed:

  • the wake ends with the success wording ("…are the result — report it…") or the failure wording ("Analyze the failure: bgtail … bggrep …"), never both;
  • the ❌ test pins the failure wording and the absence of the success wording;
  • digestBlockOf no longer delimits the digest block by that instruction's wording (it broke four digest tests), and the README sample (an exit-1 job) shows the failure variant;
  • the guideline and started: line no longer claim the wake carries the line count, last line and digest unconditionally — each is conditional in the wake construction.

Verification

  • tsc --noEmit clean; 224 tests pass, including the new failure/success wake assertions.
  • Two live smokes asking an agent to start a job — one with skills disabled, one with the skill loaded — each make exactly one call (bgrun) and answer "I'll be woken when it finishes", where every earlier arm polled twice.
  • The wake's two instruction variants are pinned by tests rather than by a smoke, which is the lesson of the follow-up above.

Every measured session that used bgrun opened the same way: `bgrun`, then
`bgstatus`, then `bgtail` — before the job could possibly have finished. In one run
those two calls were 2,231 of 8,049 context chars (28%), spent learning what the
wake delivered for free moments later.

Nothing told the agent not to. BGRUN_GUIDELINES mentioned the wake only in passing,
the `started:` result said merely "You'll be woken automatically when it finishes",
and the run-bg skill actively licensed the polls in two places:

- step 2: "`exit: 0` → success. `bgtail` to confirm." — a read even when the wake
  already carried the counters;
- step 3: "If you need to check before the wake (non-blocking): call `bgstatus`…" —
  framed as ordinary practice.

Now:

- BGRUN_GUIDELINES gains: "Do not poll a job you just started — no bgstatus/bgtail
  round after bgrun. The wake is the signal: it carries the exit code, duration,
  log line count, the log's last line and the project's digest. Read the log after
  the wake only for detail the wake does not carry…"
- The `started:` result ends "…no need to poll; the wake carries the exit code,
  counters and digest."
- Skill step 2 makes the wake itself the result and drops "bgtail to confirm"; step
  3 is reframed — checking before the wake is rarely worth it, so reach for
  `bgstatus` when the job is NOT one you just started (another session's, one from
  before a restart, or one whose wake never came).

Verified: `tsc --noEmit` clean; 224 tests pass (no test asserted the old wording);
and a live smoke where an agent asked to start a job makes exactly one call — bgrun
— and answers "I'll be woken when it finishes — no polling needed", where every
earlier arm polled twice.
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

CI report

Check Result
tsc --noEmit success
tests success
npm pack --dry-run success (7 files in tarball)

Ref: 0baa23b0e69ea81fae882014d31c083d0a076677

…xit code

Review of the first commit found the change contradicted itself: the wake still
ended with an unconditional "Review the result now: call `bgtail` …", emitted on
exit 0 too — the last instruction the agent reads, telling it to make exactly the
read this branch discourages. The smokes could not catch it: they are one-shot
sessions that exit before the wake arrives, so they only measure the pre-wake
decision. The verification was blind by construction.

- The wake now ends with one of two lines: on success "The exit code, stats and
  last output above are the result — report it, and continue the task that depended
  on this…"; on failure "Analyze the failure: `bgtail` for a peek at the end of the
  log, or `bggrep` to search it — then continue…".
- The ❌ test asserts the failure wording AND that the success wording is absent, so
  the two paths cannot silently converge again.
- `digestBlockOf` no longer delimits the digest block by that instruction's wording
  — which is what made four digest tests fail on the first attempt; it now takes
  the block from the `digest (…)` line to the wake's last line.
- README's sample wake (an exit-1 job) shows the failure variant.
- Accuracy fix from the same review: the guideline and the `started:` line claimed
  the wake carries the line count, the last line and the digest unconditionally, but
  all three are conditional in the wake construction (line count when the log is
  present and within the read window, last line when the log has content, digest
  only when a project configures one). Both now carry qualifiers.

Verified: `tsc --noEmit` clean, 224 tests pass, and no remaining reference to the
old instruction anywhere in the tree.
@lloydsk
lloydsk merged commit c6680f2 into main Sep 26, 2026
2 checks passed
@lloydsk
lloydsk deleted the fix/no-poll-after-bgrun branch September 26, 2026 02:19
lloydsk added a commit that referenced this pull request Sep 26, 2026
…grun does not help

Three additions, prompted by the obvious hazard: a fixture we can tune is a
fixture we can tune in the tool's favour.

- **How these numbers are produced** — the protocol as one table (arms, prompt
  template, model, live sessions, n=3 reported as median and range, metric
  definitions, and the command that re-derives every table from the transcripts),
  plus the `vanilla-hinted` control arm: the agent hand-rolling redirect-and-grep,
  which is the strongest baseline in the set.
- **Pre-registered expectations** — nine cells with the predicted winner written
  down before the runs, including the ones where bgrun is expected to lose (R1
  green, R2 tail-shaped failure, R5 fail-fast) and the one where our own story may
  be wrong (D1: if the advantage is reliability-driven it should grow with a weaker
  model). C1-C4 are dialogues — overlap, three jobs at once, a mid-run restart,
  asking where a run is — because no generated fixture can express them.
- **The fixture's design and why each property is under test** — the shared chain
  that makes duration scheduler-independent, the 65% failure position, the
  realistic failure block, and the knobs for the negative cells. Records that the
  earlier long-suite run used the verbose 8-lines-per-test density while the
  realistic default is 2, and that the difference is an axis rather than an
  oversight.
- **Where bgrun helps, and where it does not** — the strongest claim the data
  supports is the removal of the worst case, not a better best case (vanilla's
  `tail -100` run beat every bgrun run on cost). Names the papercuts the measuring
  found: the 28% pre-wake poll (#27 fixed), window-relative bggrep line numbers
  (#25 open), and re-paid log reads before delta `bgtail` (#4).
lloydsk added a commit that referenced this pull request Sep 28, 2026
…grun does not help

Three additions, prompted by the obvious hazard: a fixture we can tune is a
fixture we can tune in the tool's favour.

- **How these numbers are produced** — the protocol as one table (arms, prompt
  template, model, live sessions, n=3 reported as median and range, metric
  definitions, and the command that re-derives every table from the transcripts),
  plus the `vanilla-hinted` control arm: the agent hand-rolling redirect-and-grep,
  which is the strongest baseline in the set.
- **Pre-registered expectations** — nine cells with the predicted winner written
  down before the runs, including the ones where bgrun is expected to lose (R1
  green, R2 tail-shaped failure, R5 fail-fast) and the one where our own story may
  be wrong (D1: if the advantage is reliability-driven it should grow with a weaker
  model). C1-C4 are dialogues — overlap, three jobs at once, a mid-run restart,
  asking where a run is — because no generated fixture can express them.
- **The fixture's design and why each property is under test** — the shared chain
  that makes duration scheduler-independent, the 65% failure position, the
  realistic failure block, and the knobs for the negative cells. Records that the
  earlier long-suite run used the verbose 8-lines-per-test density while the
  realistic default is 2, and that the difference is an axis rather than an
  oversight.
- **Where bgrun helps, and where it does not** — the strongest claim the data
  supports is the removal of the worst case, not a better best case (vanilla's
  `tail -100` run beat every bgrun run on cost). Names the papercuts the measuring
  found: the 28% pre-wake poll (#27 fixed), window-relative bggrep line numbers
  (#25 open), and re-paid log reads before delta `bgtail` (#4).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant