Skip to content

fix(pi): treat torn-down-task wake rows as main-only, not queue corruption - #3

Closed
frasdl wants to merge 2 commits into
mainfrom
fm/fm-branch-swallow
Closed

fix(pi): treat torn-down-task wake rows as main-only, not queue corruption#3
frasdl wants to merge 2 commits into
mainfrom
fm/fm-branch-swallow

Conversation

@frasdl

@frasdl frasdl commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Intent

Investigate and fix the "supervision branch silently swallows task wake rows without emitting an outcome" reliability bug in firstmate. Touches shared tracked material (bin/, .pi/extensions/, docs) - follow firstmate-coding-guidelines.

Part 1 root-cause (evidence-driven): the 2026-08-26 m365-pi-scout incident. Scout finished ~11:47-11:48 (report.md mtime 11:47, done: row queued 11:48:47 rows 53/54). Watcher absorbed signals as benign while the scout provably worked (correct). The branch accepted the done-wake at 11:48:48, handled it through 11:50:32 (outcome seq 22 at 11:50:19). Meanwhile a stale wake (row 55, queued 11:49:19) was accepted and chained behind the handling; when the chain re-scanned eligibility (~11:50:32), teardown (11:50:02) had deleted state/m365-pi-scout.meta, so the stale row's window could not be resolved and scopeForUnreadWake classified the queue as CORRUPTED, producing the false fallback "Supervision branch unavailable, falling back to main: the unread wake queue could not be read safely" (delivered 11:51:14, after main's drain at 11:51:17 already found the queue empty). Main also misread the outcome store ~1s before seq 22 landed and concluded the branch swallowed the rows (duplicate relay; maps to upstream open issues kunchenguid#2977 and kunchenguid#2984, which are NOT fixed here).

Part 2: GitHub issue search on the ORIGINAL repo kunchenguid/firstmate (open AND closed) for watcher/supervision wake absorption, missing outcomes, stuck wake queue, "queued wakes pending", stale swallowing, branch actor not reporting - summarize each with number/title/state/URL; state explicitly if none describe this incident (no issue describes the torn-down-row eligibility poisoning; closest are kunchenguid#2977 open, kunchenguid#2984 open, kunchenguid#2028 open, kunchenguid#2791 open, kunchenguid#2374 open).

Part 3 fix (implemented, EVALUATE AS ACCEPTED): in .pi/extensions/lib/fm-branch-dispatch.ts scopeForUnreadWake, a legible signal/stale row whose task metadata is gone (teardown raced the queued wake) is ordinary main-only content exactly like the check-kind class: excluded from eligibleSeqs, left queued for main, non-vetoing in ordinary scans, and still a heartbeat veto. Structural corruption (unknown kind, malformed line, non-numeric seq) still vetoes the whole scan. bin/fm-wake-drain.sh write_rows_file_locked no longer abandons an empty claim/consume scratch tmp file in state/. tests/fm-pi-branch-live-e2e.test.sh fixture copies lib/fm-calm-visibility.ts (required by 0.84.x imports); docs/verification/runtime-backends.md refreshed with dated 0.84.3 results. Regression tests added: tests/fm-pi-branch-extension.test.sh (unmapped classification, mixed-queue eligibility, heartbeat veto) and tests/fm-wake-queue.test.sh (no scratch tmp left behind). docs/pi-supervision-branch.md autonomy section updated; classification contract remains owned by fm-branch-dispatch.ts header. Constraints: keep the fix minimal and fail-closed for structural corruption; do NOT fix upstream issues kunchenguid#2977/kunchenguid#2984 (out of scope); no agent co-author; shellcheck-clean bin scripts; tests must exercise behavior through the public interface, never assert implementation source bytes; pre-existing local-only failure of the stock-render test in tests/fm-pi-branch-extension.test.sh (Pi 0.84.3 version skew, fails on clean main too) must not be attributed to this change.

What Changed

  • .pi/extensions/lib/fm-branch-dispatch.ts: scopeForUnreadWake now classifies a legible signal/stale wake row whose task metadata is gone (teardown raced the still-queued wake) as ordinary main-only content instead of corruption - it is excluded from eligibleSeqs, left queued for main, and no longer vetoes an ordinary scan, while structurally unreadable rows (unknown kind, malformed line) still veto the whole scan and an unmapped row still defers a heartbeat review to main. This removes the false "Supervision branch unavailable" fallback after a task teardown.
  • bin/fm-wake-drain.sh: write_rows_file_locked now removes the caller's owned scratch file along with the target when the source is empty, so empty main claims and branch consumes no longer abandon .main-eligible-rows.tmp.* / .wake-rows.consume.* litter in state/.
  • Tests and docs: regression coverage added in tests/fm-pi-branch-extension.test.sh (unmapped-row main-only classification, mixed-queue eligibility, heartbeat veto) and tests/fm-wake-queue.test.sh (no scratch tmp left behind); tests/fm-pi-branch-live-e2e.test.sh fixture now copies lib/fm-calm-visibility.ts (required by Pi 0.84.x imports); docs/verification/runtime-backends.md refreshed with dated 0.84.3 evidence and docs/pi-supervision-branch.md wording aligned.

Risk Assessment

⚠️ Medium: The fix itself is minimal, correct against its contract (fail-closed for structural corruption preserved, branch can never acknowledge an unmapped row), regression-tested through public interfaces, and shellcheck-clean; risk is limited to the open ask-user question about where the required Part 2 issue-search summary is delivered, which is a follow-up confirmation rather than a source defect.

Testing

Validated the torn-down-task wake-row fix via the real public interfaces: new classification coverage passes on target (unmapped rows = ordinary main-only, excluded from eligibleSeqs, non-vetoing in mixed queues, still heartbeat vetoes; structural corruption still fail-closed) and reproduces the incident misclassification on base (whole queue corrupted/fallback-to-main, eligible lost), the mixed-queue eligible set [2,3] is published through the real grant bin, the fm-wake-queue suite (including the new no-scratch-tmp test) is green on target and fails on base, and the live e2e guard passes against the real Pi 0.84.3 SDK (fixture now copies fm-calm-visibility.ts). The only failure in the touched suites is the pre-existing Pi 0.84.3 stock-render skew in fm-pi-branch-extension.test.sh (byte-identical in base, explicitly not attributable per the intent), which unfortunately aborts that file before the new classification tests run in normal suite executions - the new tests were proven by direct execution of the same code paths instead.

Evidence: Target vs base scopeForUnreadWake classification of the torn-down (unmapped) wake row and mixed queue

Source: Target vs base scopeForUnreadWake classification of the torn-down (unmapped) wake row and mixed queue

TARGET: unmapped-only {status:unsafe, eligible:false, corrupted:false, eligibleSeqs:[]} / mixed {status:safe, eligible:true, corrupted:false, eligibleSeqs:["2","3"]} / heartbeat-with-unmapped {corrupted:true} / malformed-line {corrupted:true} / snapshot-publish: published rows ["2","3"] BASE: unmapped-only {corrupted:true} / mixed {corrupted:true, eligible:false}

unmapped-only: {"status":"unsafe","eligible":false,"corrupted":false,"eligibleSeqs":[],"projects":[]}
mixed: {"status":"safe","eligible":true,"corrupted":false,"eligibleSeqs":["2","3"],"projects":["/tmp/tmp.mrYajBZjYK/home/projects/approved"]}
heartbeat-with-unmapped: {"status":"unsafe","eligible":false,"corrupted":true,"eligibleSeqs":[],"projects":[]}
malformed-line: {"status":"unsafe","eligible":false,"corrupted":true,"eligibleSeqs":[],"projects":[]}
snapshot-publish: published rows: ["2","3"] == [2,3]: true
Evidence: Base (pre-fix) scopeForUnreadWake misclassification: the 2026-08-26 wake-loss reproduced

Source: Base (pre-fix) scopeForUnreadWake misclassification: the 2026-08-26 wake-loss reproduced

unmapped-only: {"status":"unsafe","eligible":false,"corrupted":true,...} mixed: {"status":"unsafe","eligible":false,"corrupted":true,...} heartbeat-with-unmapped: {"status":"unsafe","eligible":false,"corrupted":true,...}

unmapped-only: {"status":"unsafe","eligible":false,"corrupted":true,"eligibleSeqs":[],"projects":[]}
mixed: {"status":"unsafe","eligible":false,"corrupted":true,"eligibleSeqs":[],"projects":[]}
heartbeat-with-unmapped: {"status":"unsafe","eligible":false,"corrupted":true,"eligibleSeqs":[],"projects":[]}
Evidence: Target bin/fm-wake-drain.sh leaves no empty claim/consume scratch tmp

Source: Target bin/fm-wake-drain.sh leaves no empty claim/consume scratch tmp

TARGET(post-fix): no scratch tmp files left in state/

TARGET(post-fix): no scratch tmp files left in state/
Evidence: Base bin/fm-wake-drain.sh abandons empty claim/consume scratch tmps in state/

Source: Base bin/fm-wake-drain.sh abandons empty claim/consume scratch tmps in state/

BASE(pre-fix): left scratch tmp file(s) in state/: .main-eligible-rows.tmp.P93d9M .wake-rows.consume.qL5HxI

BASE(pre-fix): left scratch tmp file(s) in state/: .main-eligible-rows.tmp.P93d9M .wake-rows.consume.qL5HxI 
- Outcome: ⚠️ 1 info across 1 run (8m26s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 issues (1 warning, 1 info)
⚠️ **Test** - 1 info
  • ℹ️ tests/fm-pi-branch-extension.test.sh:1601 - Pre-existing (not attributable to this change): the stock-render test test_outcomes_tool_uses_stock_execution_and_export_consumers fails on the installed Pi 0.84.3 SDK (version skew; the test body is byte-identical to base commit d88c71f, verified via diff) and, because it is the first test invoked in the file and fail() exits the script, it aborts the entire suite before the newly added unmapped-row classification coverage at line 1603 runs. I verified the full-file run emits only that one not-ok (exit 1), and that all 17 remaining tests - including test_branch_dispatch_classifies_main_only_rows_and_writes_the_eligible_snapshot with the new unmapped/mixed-queue/heartbeat assertions - pass when that invocation is removed. Consequence: the new regression tests added by this change are unreachable in normal suite execution until the version-skew test is repaired; the fix itself is proven by the direct-driver runs below, not by the in-file suite run.
  • bash tests/fm-wake-queue.test.sh - full suite ok (exit 0), including the new test_empty_actor_claims_leave_no_scratch_tmp_behind driving the real bin/fm-wake-drain.sh + bin/fm-wake-grant.sh with an empty claim set and empty consume set
  • bash tests/fm-pi-branch-extension.test.sh - aborts at the first test with not ok - Pi outcomes rendering consumers must preserve stock behavior (Pi 0.84.3 version skew; test body verified byte-identical to base via git diff d88c71f e7e46ff), so nothing else in the file ran
  • Suite re-run with only the stock-render invocation removed (transient scratch copy tests/.fm-pi-branch-ext-skip-stock.test.sh, deleted after): 17/17 ok including test_branch_dispatch_classifies_main_only_rows_and_writes_the_eligible_snapshot (new unmapped-only, unmapped+mixed-queue, heartbeat-veto assertions through the real lib + grant bin)
  • FM_PI_BRANCH_LIVE_E2E=1 bash tests/fm-pi-branch-live-e2e.test.sh - ok - real Pi SDK 0.84.3 accepts the branch session construction and preserves an unpromptable wake, validating the fixture now copies lib/fm-calm-visibility.ts
  • Classification driver (classify-driver.mjs) against target fm-branch-dispatch.ts: unmapped row -> corrupted:false main-only; mixed queue -> eligible:true, eligibleSeqs ["2","3"], snapshot published through the real grant bin; heartbeat with unmapped row -> corrupted:true; malformed line -> corrupted:true (fail-closed)
  • Same driver against base fm-branch-dispatch.ts (git show d88c71f): unmapped row -> corrupted:true and mixed queue -> corrupted:true, eligible:false - the 2026-08-26 m365-pi-scout wake-loss misclassification, i.e. the regression fails before the fix and passes after
  • Drain claims demo (drain-claims-demo.sh) against base bin/fm-wake-drain.sh: leaves .main-eligible-rows.tmp.* and .wake-rows.consume.* in state/ (exit 7); against target bin/fm-wake-drain.sh: no scratch tmp files left in state/ (exit 0)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

francesco.quarta added 2 commits August 26, 2026 12:24
…uption

A wake row can legitimately outlive its task's state files: the watcher
queues a signal/stale row while the task still exists, the branch accepts
the wake, and teardown deletes state/<id>.meta before the serialized
branchChain reaches that wake. scopeForUnreadWake classified any
unmapped signal/stale row as queue corruption (UNSAFE), poisoning the
whole scan and producing a false 'Supervision branch unavailable,
falling back to main: the unread wake queue could not be read safely'
wake (2026-08-26 m365-pi-scout incident).

Legible-but-unmapped rows now behave exactly like the existing
check-kind class: excluded from the branch's eligible set, left queued
for main, non-vetoing in ordinary scans, and still a heartbeat veto.
Structural corruption (unknown kind, malformed line, non-numeric seq)
still vetoes the whole scan. The classification contract stays owned by
fm-branch-dispatch.ts's header; pi-supervision-branch.md documents the
boundary.

Also: bin/fm-wake-drain.sh no longer abandons an empty claim/consume
scratch tmp file in state/ (the .main-eligible-rows.tmp.* and
.wake-rows.consume.* litter), and the live Pi branch e2e fixture copies
lib/fm-calm-visibility.ts its 0.84.x imports require, refreshing the
dated runtime-backends.md evidence for pi-coding-agent 0.84.3.

Regression: tests/fm-pi-branch-extension.test.sh (unmapped-row
classification, mixed-queue eligibility, heartbeat veto) and
tests/fm-wake-queue.test.sh (no scratch tmp left behind).
@frasdl

frasdl commented Aug 26, 2026

Copy link
Copy Markdown
Owner Author

Abandoned per captain direction ("leave the fm branch completely"); closing without merge. Branch/work remains, unlanded.

@frasdl frasdl closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant