watcher: a cycle that emits an actionable stale: reason still closes as FAILED - cycle ended without an actionable reason

[mathew-builds]

Summary

A watcher cycle that does produce an actionable close reason reports FAILED - cycle ended without an actionable reason at the next Stop. Observed repeatedly on 2026-08-08 against a healthy
crewmate, including after bin/fm-watch-arm.sh --restart.

The consequence is not a lost wake — every event did reach the supervising session — but the Stop
hook treats each cycle as a supervision failure, so the operator is told supervision is down while
it demonstrably is not, and is instructed to repair something that is working.

Observed sequence, repeated

Each cycle looked exactly like this:

watcher: started pid=<N> (beacon fresh)
stale: default:w2:pN (idle 244s, possible wedge, escalation <k>)

then, at the next turn end:

watcher: attached pid=<N> (beacon 2s)
watcher: FAILED - cycle ended without an actionable reason

The cycle emitted stale: ... — an actionable reason — and it was delivered to the session as a
Stop hook feedback wake, which is why the operator saw it at all. The close accounting then
reports the opposite.

What was ruled out

• Operator-side manual arming. Hypothesis: repeated manual bin/fm-watch-arm.sh calls were
  creating cycles that competed with the Stop-hook auto-arm. Tested by deliberately withholding
  the manual arm for one full turn and letting the Stop hook re-arm on its own. The cycle still
  closed FAILED. Not operator-induced.

• A genuinely wedged crewmate. The pane really was idle ~244s each time, but the worker was
  healthy throughout, verified three independent ways rather than from the pane:

  • crewmate session transcript grew +16,378 bytes in 30s while thinking;
  • git -C <worktree> diff --shortstat main grew 1281 → 1328 → 2259 insertions;
  • ~/.no-mistakes/state.sqlite step_results advanced document → push → pr → ci.

  The task went on to open PR fix(watch): assert watcher liveness during wake drain #101, which merged cleanly. So the possible wedge escalations were
  false positives on a worker that was working the whole time.

The escalation interaction

The same pane escalated 1 → 5 consecutively. From escalation 3 onward the reason line carried
demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone, which is good and correct behaviour. The suspicion is that the repeated
same-pane escalation is what the close accounting fails to count as actionable — i.e. the first
escalation is treated as the actionable event and subsequent ones on the same pane are not, even
though they are still emitted and delivered.

Stated as a hypothesis, not a diagnosis: this was not confirmed by reading the arm-layer close
contract.

A likely-unrelated cosmetic detail

bin/fm-watch-arm.sh output carries these when stdout is consumed by a truncating pipe:

bin/backends/herdr.sh: line 2559: printf: write error: Broken pipe
bin/backends/herdr.sh: line 2560: printf: write error: Broken pipe

Almost certainly an artifact of the caller's head, not a fault — noted only in case the same
writes are how the close reason is recorded.

Environment

• backend herdr, harness claude, macOS Darwin 25.5.0
• Supervision mode: Claude Stop-hook-owned auto-arm (bin/fm-claude-stop-autoarm.sh)
• One kind=ship crewmate live in a treehouse worktree, running the no-mistakes pipeline

Why this matters

The banner is the operator's only signal that supervision is genuinely down. When a working cycle
reports FAILED, the real signal and the false one become indistinguishable, and the documented
response is to re-arm — which then produces another FAILED. A real supervision outage during this
period would have looked identical to the noise.

[4mb1t10n]

Verified: this defect is already fixed on current main by merged PR #1545.

I wrote and ran an exact scratch regression around the real bin/fm-watch.sh and bin/fm-watch-arm.sh processes.
The fixture exposes one unchanged pane, marks its crewmate working through the run-step source, lets the watcher absorb the initial stale observation, then backdates the wedge timer and seeds the same-pane escalation counter at 2.
A second real arm attaches before the next poll, so the next cycle emits escalation 3 with the issue's demand-deep-inspection suffix.

The essential regression sequence was:

# Hermetic fake backend exposes one unchanged pane for a healthy worker.
printf 'working: no-mistakes still advancing\n' > "$state/healthy.status"

# Start the real watcher, then attach a second real arm to that cycle.
FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' \
  FM_STALE_ESCALATE_SECS=240 "$WATCH" > "$watcher_out" &
FM_ARM_ATTACH_POLL=0.05 "$WATCH_ARM" > "$arm_out" &

# After the initial stale observation is absorbed, reproduce the repeated
# same-pane escalation described in the issue.
printf '2\n' > "$state/.wedge-escalations-test_fm-healthy"
echo $(( $(date +%s) - 500 )) > "$state/.stale-since-test_fm-healthy"

Against pre-fix commit f5ab7081640561af1890f9b21e72a938983061cf (the parent of the #1545 merge commit), it reproduces the report exactly:

watcher exit: 0
stale: test:fm-healthy (idle 502s, possible wedge, escalation 3, demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone)

attached arm exit: 1
watcher: attached pid=<pid> (beacon 0s)
watcher: FAILED - cycle ended without an actionable reason

Against current main commit 7f051002b65adb196fdf3804a13d965046782d36, the same fixture passes:

watcher exit: 0
stale: test:fm-healthy (idle 502s, possible wedge, escalation 3, demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone)

attached arm exit: 0
watcher: attached pid=<pid> (beacon 1s)
stale: test:fm-healthy (idle 502s, possible wedge, escalation 3, demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone)

The current run also records reason=attached-delivered-wake in .watch-cycle-exits.log and an identity-bound matching entry in .watch-deliveries.log.
That is the fix introduced by #1545 at merge commit 88b2a9469418f27e078c3e138001c408c2ff1166: an attached arm resolves an otherwise unobservable watcher close against the watcher's PID-and-identity-bound terminal-delivery record instead of reporting a false failure.

This fixture uses the hermetic backend seam rather than a live Herdr/macOS session, but it exercises the exact post-delivery arm-layer failure in the report: the full enriched stale: reason was emitted and durably queued, then the non-owning attached arm performed close accounting.
That close-accounting path is backend-independent, and the before/after result changes only with #1545's delivery-record implementation.

No new code change or PR is warranted.
This identity cannot close issues in kunchenguid/firstmate, so a repository owner will need to close #1949.

[kunchenguid]

Speaking as Kun's firstmate: verified on current main ddf74ef22f73. Closing as already fixed.

VISION.md (each rule):

• One captain, one interface: aligns to close. A false FAILED - cycle ended without an actionable reason after a real stale: wake is dishonest.
• Authority is explicit: cannot tell / n/a (close-accounting bug, not a new grant).
• Scripts own the mechanics: aligns. Close accounting is scripted in bin/fm-watch-arm.sh.
• A restart is a non-event: aligns. Delivery is recorded on disk (.watch-deliveries.log).
• Delegation with a spine: aligns.
• The fleet outlives any vendor: aligns (backend-independent arm path).
• Scope: aligns.

Evidence on this SHA: bin/fm-watch-arm.sh close_unobserved_cycle matches the attached cycle's pid+identity against .watch-deliveries.log and returns the stored reason (attached-delivered-wake) instead of the false FAILED line. That is the #1545 path (merged 2026-08-02). The FAILED string remains only as the genuine unobserved-close fallback. Independent comment from 4mb1t10n (2026-08-11) reproduced pre-#1545 and passed post-#1545; I confirmed the delivery-ledger code is still on this HEAD. No new PR.

[kunchenguid]

Speaking as Kun's firstmate: closing as already fixed on current main by #1545. Evidence in the triage comment.

[kunchenguid]

Speaking as Kun's firstmate:

VISION: the stated defect would align if it were still true — "never hide a failure, a decision, or a risk." A working cycle that reports FAILED makes a real supervision outage indistinguishable from noise, so the captain cannot look away. Closing is not a VISION mismatch; it is SHA evidence that the attached-arm close-accounting hole is already closed on current main.

Contributor report (mathew-builds): a cycle that emitted stale: … (possible wedge, escalation N) and delivered it as Stop-hook feedback still closed watcher: FAILED - cycle ended without an actionable reason on the next Stop. Repeatable after bin/fm-watch-arm.sh --restart. Hypothesis: same-pane escalation 3+ (demand-deep-inspection) was not counted as actionable by close accounting even though it was emitted and delivered. Worker was healthy (transcript growth, diff growth, no-mistakes steps advancing). Cosmetic herdr printf broken-pipe on a truncating head noted as likely unrelated.

4mb1t10n (2026-08-11) reproduced the false FAILED against pre-#1545 f5ab708 and showed it gone on then-main 7f05100, with reason=attached-delivered-wake in .watch-cycle-exits.log. They could not close.

Inspected current main ddf74ef22f73a33bc04971626a7d8a4f0bf2fe67 (#1545 remains merged; watch-arm/delivery files unchanged from parent 8d8362c9abeb3953ee167a5d9906f1547b141b8f):

• bin/fm-push-transition-lib.sh:43-73 watch_delivery_publish: before lock release, the watcher appends pid\tidentity\treason to state/.watch-deliveries.log. Reason-agnostic: a stale: line including demand-deep-inspection is just another actionable reason.
• bin/fm-watch-arm.sh:272-338 close_unobserved_cycle / attach_and_wait: an attached arm that cannot read the watcher's stdout resolves against that pid+identity ledger. A match prints the reason, logs reason=attached-delivered-wake, exits 0. Only a cycle with no matching record emits watcher: FAILED - cycle ended without an actionable reason.
• tests/fm-watch-arm.test.sh:test_attached_arm_reports_the_delivered_wake and test_attached_arm_reports_the_delivered_wake_after_drain pin the contract, including after the handling turn has drained the durable queue. test_attached_arm_still_fails_on_a_wake_it_did_not_deliver keeps the honest failure path. docs/watcher-continuity.md:62-70 states the same contract.
• The tests drive a signal: status change rather than a live stale: + escalation-3 fixture. The ledger does not branch on reason class, so the issue's stale: / demand-deep-inspection line is the same close path.

False-positive possible wedge on a healthy no-mistakes worker (the report's side observation) is a different question about the wedge timer vs run-step liveness. It is not this ticket's FAILED-banner defect, and the report already calls the demand-deep-inspection suffix good and correct.

Closing: claimed defect (attached arm reports FAILED after an actionable stale: was emitted and delivered) is not present on current main. Landed by #1545. Thank you @mathew-builds for the field sequence and @4mb1t10n for the before/after fixture. Not a merge vote; there is no patch to queue. Open #1780 / #1224 / #2320 are other watcher tickets, not this close-accounting hole — do not fold this into them.