Watcher wedges on every arm; supervision stays down; only a session restart recovers

[mayankasthana]

Symptom

The firstmate watcher process starts and then wedges silently within minutes: it never touches its liveness beacon, never writes to the triage log, and holds the watch lock while idle. Every re-arm produces another wedged watcher. Wake delivery via the Pi extension still works, but the watcher itself is dead, the turn-end guard blocks turns, and the only recovery is restarting the Pi session.

Environment (reported by operator, 2026-08-12)

• Firstmate version: b5d430d (updated 2026-08-12; includes feat(fmx-respond): consume Relay conversation chains (#2206) and fix(bin): prevent watcher recovery acknowledgement livelock (#2212))
• Harness / agent: pi (Pi coding agent CLI) v0.84.1
• Model: deepseek-v4-flash (deepseek provider) — used by the primary and all crewmates
• Runtime backend: herdr (HERDR_ENV=1, experimental session-provider backend; many concurrent herdr panes at wedge time)
• OS: macOS 26.4.1 (Build 25E253)
• Node: v25.9.0 · npm 11.12.1
• Timezone: IST

Reproduction context (observed 2026-08-12, macOS, pi primary, herdr backend)

• Watcher ran normally until ~12:43 IST (triage log active, beats fresh).
• From 12:43 onward: 6+ consecutive arms each wedged within minutes (process alive, S+ state, no beat, no triage-log output after start).
• fm_watch_arm_pi returns "unchanged - extension already owns an arm child" even after all arm/watcher processes were killed; the extension keeps spawning new arms that wedge again.
• Process sample shows bash stuck deep in a wait (child/syscall wait, not resolvable in the sample head).
• Full home-scoped cleanup (kill wedged watcher+arm PIDs, remove stale .watch.lock + owner dir, remove stale .watcher-down lock owner/steal dirs) did NOT help — the next fresh arm wedged the same way.
• Earlier in the day, a wake-drain lock acquisition was observed dying with SIGKILL ("Killed: 9") during ln -s lock creation — possible related resource/state pressure.
• No state/*.check.sh files were present at wedge time (all checks quarantined), so the hang is not a check execution.

Suspected areas

• bin/fm-watch.sh poll loop / lock acquisition deadlock (possibly a stale lock-owner dir or a subprocess it waits on that hangs).
• .pi/extensions/fm-primary-pi-watch.ts arm verification: the arm child "did not exit within 1000ms" and the extension can never verify a ready successor, so it keeps an unwedged-looking arm claim that blocks manual re-arm.
• .watcher-down recovery/lock state interplay with the ack-livelock area recently fixed in fix(bin): prevent watcher recovery acknowledgement livelock #2212.

Impact

Supervision down for hours with live fleet work; turn-end guard blocks; the documented repair (session restart) is disruptive. The watcher should fail loudly and recover, or at least release the lock so a manual re-arm can take over.

[mayankasthana]

Session evidence from the Claude primary (2026-08-12 ~15:28 IST), same home as the Pi report:

• Auto-arm hook (fm-claude-stop-autoarm.sh) fired with epoch=6, ran its 2 bounded attempts, both failed: the forked watcher child acquired the watch lock (owner dir .watch.lock.owner.updNXq created 15:28, pid 92352) and then died BEFORE writing its identity files (fm-home/watcher-path/pid-identity absent from the owner dir) and before touching the liveness beacon. No EXIT-trap cleanup ran (the lock symlink remained pointing at the dead pid), consistent with the child being killed (SIGKILL-style) mid-startup rather than exiting through the arm's cleanup path.
• The arm's confirmation window then timed out with no healthy watcher; hook emitted the one-time failure notice and exited 2. The turn-end guard then required investigation before any blind stop.
• CRITICAL CONTRAST: a direct foreground checkpoint run (bin/fm-watch-checkpoint.sh) at 15:46 started the SAME fm-watch.sh cleanly — beacon fresh, lock acquired and released via EXIT trap, recovery marker transitioned acked->pending:downtime with fresh generation, and the watcher exited with the expected 'check: rearm-resurface' actionable wake. So the watcher startup path itself is NOT wedged; the failure is specific to the harness-owned asyncRewake hook process tree killing/never-verifying the forked arm child.
• Same signature as the Pi side: 'arm child did not exit within 1000ms; extension can never verify a ready successor'. Both harnesses' arm children die or wedge before the watcher's first beacon touch.
• Pr-check migration passes standalone (exit 0); memory pressure normal (60% free, 1GB/2GB swap); no stale check.sh files.

[4mb1t10n]

Independent Linux/CI reproduction of a bin/fm-watch.sh hang

Adding corroborating evidence from an unrelated lane, since the original report is macOS + pi + herdr and this one is a different environment entirely.

While validating an unrelated change (issue #1956), a worker traced a red required check to a 15-minute CI timeout rather than an assertion failure. Root cause it identified:

• An intermittent bin/fm-watch.sh hang.
• The hung process survives SIGTERM.
• Reproduced locally at 2 out of 15 trials.
• It also cancels shards on unchanged main, so it is not specific to any feature branch.

That matches this issue's core symptom — the watcher process stays alive while making no progress — but from GitHub-hosted Linux CI rather than a macOS herdr desktop. Two independent environments showing the same component hanging suggests the defect is in fm-watch.sh itself rather than in a harness- or backend-specific integration.

Two deliberate caveats so this is not over-read:

1. This reproduction did not observe the held-.watch.lock-with-no-beat state described in the original report; it was observed as a job-level timeout. It may be the same root cause or a related one.
2. A separate supervision outage seen on a Claude primary with the tmux backend had no lock held and no live process, which is a different failure shape again. It is not being claimed as the same bug here.

The intermittency (~13%) is consistent with a defect that looks like flaky infrastructure at low fleet load and like a hard outage under sustained supervision, which may explain why it has been easy to write off as CI flakiness.

Filed by Firstmate while routing the finding out of the lane that discovered it, so the evidence is not lost when that unrelated pull request lands.

[4mb1t10n]

Follow-up: the same shard now blocks three independent PRs on the same day

Extending the Linux/CI reproduction posted earlier. What was reported there as one lane's red check is now visible as a repeating, cross-branch pattern.

On 2026-08-12, three unrelated pull requests — different authors, different subsystems, different branches — all failed CI on the same single check, Behavior portable serial 2, while every other check on each PR passed:

PR	Branch	Other checks	Behavior portable serial 2
#2142	fm/fm-1956	13 pass	fail (15-min timeout, no verdict)
#2235	fm/fm-1243	12 pass	fail
#2269	fm/fm-2223	12 pass	fail

Two details that sharpen the diagnosis:

1. The workflow run at test: guard secondmate harness marker isolation #2235's exact head 3ac8d41 is completed / cancelled, not completed / failure. The shard did not assert-fail; the run was killed. That is the timeout signature, not a test defect — and it means the shard produces no verdict at all, so the PR cannot go green even though nothing is wrong with it.
2. No change is common to the three branches. The only shared factor is the shard itself.

Combined with the earlier evidence — the hang survives SIGTERM, reproduces locally at roughly 2 in 15 trials, and also cancels shards on unchanged main — the picture is consistent: bin/fm-watch.sh intermittently hangs, the shard that exercises it never returns, and CI's timeout takes the whole run down with it.

The practical consequence is worth stating plainly, because it is what makes this more than a flaky test: at a ~13% per-run hit rate, a repository landing work through many concurrent PRs will see this strand several of them at once, each needing a manual rerun by someone with admin rights. Contributors without those rights cannot clear it themselves — all three PRs above are complete on the merits and are simply stuck.

No fix is proposed here; this is corroborating evidence routed out of the lanes that hit it so it survives when those PRs land.

[zeeshaanahmad]

We hit this same symptom (watcher wedges silently within minutes of arming, beacon frozen, watch lock held by the wedged pid, every re-arm wedges again) on b5d430d-era code and root-caused it while working on an unrelated change. Sharing in case it saves someone the dig:

Root cause we proved locally: the fleet locks are non-reentrant, and bin/fm-watch.sh traps HUP/INT/TERM as exit 1, which unwinds into watcher_cleanup - and watcher_cleanup re-acquires locks the interrupted flow still holds, so the process spins forever in fm_lock_acquire_wait on a lock its own pid owns. The window is reachable on every poll via _fm_recovery_marker_arm_check, not only via fm_wake_append, which is why it reproduces so readily.

Concrete observation backing this: a wedged watcher whose own pid held both .wake-queue.lock and .watcher-down.lock with a beacon frozen for 64 minutes.

Two fix shapes we scoped (both have sharp edges):

1. A global deferred-signal trap in fm-watch.sh with explicit safe points each poll - correct shape, but any code path that misses a safe point risks a TERM-immune watcher.
2. Making fm_lock_acquire_wait ownership-aware - small change, but roughly a third of its ~65 callers ignore its return value, so naive ownership-awareness silently unlocks critical sections across spawn/teardown/drain; every caller needs auditing.

The afk daemon's SIGTERM shutdown hang (#2270) looks like the same unwind-into-cleanup pattern on the daemon path and may share the fix.