patch(fm-watch): cap wedge escalations to prevent unattended LLM loop drain - #2605
patch(fm-watch): cap wedge escalations to prevent unattended LLM loop drain#2605LorenzoMinghini wants to merge 11 commits into
Conversation
80b39b1 to
bd99eff
Compare
Adds FM_WEDGE_MAX_ESCALATIONS (default 10) to bin/fm-watch.sh. Once wedge escalations for a stale-hash reach this count, the watcher emits ONE terminal wake with a 'PERMANENTLY-WEDGED' marker and writes STATE/.wedge-permanent-<key>, then stops sending further wakes for that hash until the pane's state resets to genuinely active. Resets on the existing handle_paused_stale / clear_pause_tracking paths. Root cause of the 2026-08-18 MiniMax subscription drain (~359M tokens): in LLM-supervised unattended setups, the demand-deep-inspection marker is read but not acted on, so the wake loop never breaks. This cap preserves the existing by-design signal escalation and only adds a safety floor. Tracked in PATCHES.md (patch-wedge-cap-2026-08-19) for revert.
bd99eff to
858dcef
Compare
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (11): Last reviewed commit: "patch(wedge): validate FM_CAP_HORIZON_SE..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…(v2, Greptile review) Addresses two issues from the Greptile 3/5 review on PR kunchenguid#2605: 1. Per-hash marker keying. v1 keyed STATE/.wedge-permanent-<key> on the window only, so a fresh stale hash in the same window was permanently suppressed (contradicting the 'for this hash' semantics documented in PATCHES.md). v2 keys the marker on (window, hash): STATE/.wedge-permanent-<key>-<hash12>. Fresh stale hashes in the same window can still escalate; only this exact stale hash is silenced. Threads the hash as a 6th parameter to wedge_timer_check from all 4 call sites (busy_turn_bound_check, the three main-loop sites). Reset sites (handle_paused_stale, clear_pause_tracking) now glob-remove .wedge-permanent-<key>-* (all hashes for this key) on pane recovery. 2. Atomic marker write. v1 wrote the marker BEFORE fm_wake_append and wake, so a crash or wake failure between marker-write and wake-publish left the pane permanently silent (marker on disk, wake never delivered). v2 writes the marker AFTER wake succeeds, with the ordering invariant documented in a comment. A mid-flow crash leaves no marker, and the next poll re-enters the cap branch and retries. Defensive fallback: if a caller forgets to thread the hash, v2 falls back to the v1 window-scoped marker name (with a triage log) so the cap still suppresses retries for that window. Trade-off documented: that mode is window-scoped and would suppress fresh stale hashes. Files: bin/fm-watch.sh, PATCHES.md
|
v2 pushed (commit 183a8ce). Addresses both Greptile 3/5 findings: 1. Window-scoped marker → per-hash marker. 2. Atomic marker write. Defensive fallback: if a future caller forgets to thread the hash, PATCHES.md updated with the new marker schema and ordering invariant. Ready for re-review. |
…3, Greptile round 2) Addresses the two issues from Greptile's 3/5 re-review on commit 183a8ce: 1. Marker never persisted. v2 wrote the marker AFTER 'wake', but wake() is sourced from fm-push-transition-lib and ends with 'exit 0' - the watcher only emits one wake per cycle. So the marker write was dead code and the cap never persisted: capped hashes kept producing terminal wakes on every poll, the exact drain the cap was supposed to bound. v3 writes the marker AFTER 'fm_wake_append' succeeds but BEFORE 'wake' runs, so the marker is durable when the script exits. Writes after wake-append (not before) to keep v1's safety property: a fs failure during wake-append exits 1 without setting the marker, so the next poll retries cleanly. 2. Unvalidated FM_WEDGE_MAX_ESCALATIONS override. v2/v1 accepted any value. A non-positive integer (0, negative) would fire the cap on the very first escalation, silencing wakes before the demand-deep-inspection marker ever surfaces; a non-integer would make the integer compare error silently (no 'set -e') and the cap would never fire. v3 validates at load: rejects both with a triage_log warning and falls back to the default (10). Files: bin/fm-watch.sh, PATCHES.md
|
v3 pushed (commit d17cf73). Addresses the two Greptile 3/5 findings on v2: 1. Marker never persisted (severe). 2. Unvalidated Files: Ready for re-review. |
…ailure (v4, Greptile round 3) Addresses Greptile's 4/5 review on commit d17cf73: the v3 marker write was unchecked, and wake() exit 0's mid-script, so a fs failure on the marker write persisted nothing and the cap kept firing terminal wakes every ~STALE_ESCALATE_SECS. v4 writes the marker FIRST with an explicit error check: - marker write fails: exit 1, no queue entry, no wake. Next poll retries cleanly from scratch. - marker write OK, fm_wake_append fails: rm -f marker (rollback), exit 1, no queue entry. Next poll retries cleanly. - both succeed: marker durable, queue entry durable, wake runs. Neither failure mode produces the v1 'silent wedge' (marker without queue entry) or the v3 'fire every STALE_ESCALATE_SECS' regression. Both error paths are loud and observable via triage_log. Files: bin/fm-watch.sh, PATCHES.md
|
v4 pushed (commit a040800). Addresses Greptile 4/5 finding on v3. v3 bug: marker write was unchecked, and v4 fix: error-check the marker write + rollback on fm_wake_append failure. Neither failure mode produces v1's "silent wedge" (marker without queue entry) or v3's "fire every STALE_ESCALATE_SECS" regression. Both are loud via Files: |
…Greptile round 4) Addresses Greptile 4/5 finding on commit a040800: the v2-v4 reset sites (rm -f ...wedge-permanent-<key>-*) cleared the cap marker whenever handle_paused_stale or clear_pause_tracking fired, and those fire on AUTOMATIC pause-class transitions, not on actual hash change. Failure case the v4 design allowed: 1. Hash H wedges, escalation count climbs to 10, cap fires, marker .wedge-permanent-<key>-H12 set. 2. Operator types paused: (or pause class auto-transitions for any reason). 3. handle_paused_stale fires -> glob-removes ALL .wedge-permanent-<key>-* including H12's. 4. Operator removes paused:. 5. Hash H still wedged but no marker -> cap fires again, second terminal wake. The cap was supposed to be permanent for that hash. v5: do not clear .wedge-permanent-<key>-* in either reset site. The marker is keyed on (window, hash) and the lookup at the top of wedge_timer_check is always for the CURRENT hash being processed, so old markers are naturally stale once the hash actually changes. Manual operator reset (rm STATE/.wedge-permanent-<key>-H12) is the only legitimate way to lift the cap for a still-wedged hash. The pre-existing clearing of .wedge-escalations- in those reset sites is untouched: it predates this patch and is unrelated to the cap mechanism. (The escalation count resetting is harmless because the cap marker is what gates re-firing, not the count itself.) Files: bin/fm-watch.sh, PATCHES.md
|
v5 pushed (commit a86a393). Addresses Greptile 4/5 finding on v4. v4 bug: the v2-v4 reset sites ( Failure case v4 allowed:
v5 fix: stop clearing The pre-existing Files: |
…ound 5) Addresses Greptile 4/5 finding on commit a86a393: v5 never cleared the marker, so a genuinely recovered pane that later reproduces the same stale hash would have all supervision wakes suppressed. v6 lifts the marker ONLY on unambiguous recovery signals - not on every pause-class transition (which was the v2-v4 over-clearing that Greptile R4 flagged). Two unambiguous sites: 1. New-hash + pause_state_class=working (main loop, line ~1545): capture old_h before clear_pause_tracking wipes .stale-, then rm -f its marker. The wedge was absorbed because an active pipeline exists, so PERMANENTLY-WEDGED for old_h is stale. 2. Same-hash + was-paused + pause_state_class=working (main loop, line ~1568): worker recovered on the SAME hash during a declared pause; rm -f the marker for hash H. The wedge that fired PERMANENTLY-WEDGED earlier is no longer authoritative. Other clear_pause_tracking / handle_paused_stale call sites are untouched: status-verb changes, secondmate path, etc. are ambiguous (can be stale log lines or genuine state) and would re-introduce the R4 over-clearing. Manual operator 'rm STATE/.wedge-permanent-<key>-H12' remains the escape hatch for those cases. Files: bin/fm-watch.sh, PATCHES.md
|
v6 pushed (commit 654b8dd). Addresses Greptile 4/5 finding on v5. v5 overcorrection: markers were never cleared, so a genuinely recovered pane that later reproduces the same stale hash would have all supervision wakes suppressed permanently. v6 middle ground: lift the marker ONLY on unambiguous recovery signals, NOT on every pause-class transition (which was the v2-v4 over-clearing Greptile R4 flagged). Two unambiguous sites:
Other Manual Files: |
|
Speaking as Kun's firstmate: Reviewed HEAD Class: default-behavior. The cap is on by default at VISION (per rule):
What is not cleared:
Security: no. This is waiting on the author, not the captain: drop Merge-eligible: NO. Captain-flag NOW: NO. |
Adds tests/fm-watch-wedge-cap.test.sh with six unit tests covering the
patch's behavior end-to-end:
1. test_wedge_cap_fires_permanently_wedged_after_max_escalations
drives the watcher to the cap and verifies PERMANENTLY-WEDGED +
per-(window, hash) marker are emitted.
2. test_wedge_cap_suppresses_subsequent_polls_for_same_hash
verifies the marker short-circuits further stale wakes for the
capped hash (no LLM loop drain).
3. test_wedge_cap_persists_across_pause_class_transitions
Greptile R4 fix: paused:/unpaused cycles do NOT clear the marker
while the worker is genuinely waiting (FM_FAKE_CREW_STATE=paused,
not working).
4. test_wedge_cap_lifts_on_unambiguous_recovery
Greptile R5 fix v6 site 1: when the pane content changes AND the
pipeline is verifiably active, the marker for the OLD hash is
lifted (handled in the new-stale-detection branch).
5. test_wedge_cap_lifts_on_same_hash_recovery
Greptile R5 fix v6 site 2: when the same hash resumes with an
active pipeline during a declared pause, the marker is lifted
(handled in the same-hash + was-paused + working branch).
6. test_wedge_cap_validates_invalid_override
FM_WEDGE_MAX_ESCALATIONS=0 and =abc fall back to the default
with a triage_log warning instead of firing the cap prematurely
or disabling it.
Tests use the existing test framework (wake-helpers.sh + fakebin +
FM_FAKE_CREW_STATE) and follow the same patterns as the surrounding
wedge tests in tests/fm-watch-triage.test.sh.
Each test run uses a small FM_WEDGE_MAX_ESCALATIONS (3-4) so the cap
is reached in a handful of poll cycles. Validation runs use the
default (10) since the override is rejected.
All six tests pass consistently on the v6 patch.
|
Ship-ready: PR head now 0cff500. New commit adds tests/fm-watch-wedge-cap.test.sh with six focused unit tests covering the patch end-to-end:
All six tests pass consistently. Existing wedge tests in tests/fm-watch-triage.test.sh (consecutive wedge escalations, demand-deep-inspection at threshold, pane-becoming-active resets counter, busy-pane turn-age bound, paused cadence) still pass — no regressions. Failure-mode matrix across all versions, for posterity: v1 (858dcef): per-window marker, marker before fm_wake_append with no check
v2 (183a8ce): per-hash keying, marker AFTER wake
v3 (d17cf73): marker-before-wake, override validation
v4 (a040800): error-check marker write + rollback on fm_wake_append failure
v5 (a86a393): cap persists across pause-class transitions
v6 (654b8dd): cap lifted only on unambiguous recovery (pause_state_class=working)
v6 + tests (0cff500): end-to-end coverage, ship-ready. Branch ready for maintainer review. Merging the PR is recommended — the cap addresses a documented MiniMax subscription drain (~359M tokens on 2026-08-18) and all five Greptile rounds are addressed. |
…v7, Greptile round 6) Addresses Greptile 4/5 finding on commit 0cff500: a pane can genuinely recover WITHOUT entering the declared-pause branch (e.g., the worker recovers via file activity or run-step without paused: ever being declared). v6 only lifted the cap on (a) a new hash detected with an active pipeline, or (b) the same hash resuming during a declared pause. v7 adds a third site that lifts the cap when the same hash resumes with an active pipeline outside any declared pause - the wedge was a misdetection or has been resolved. The same gate (pause_state_class=working) is used as v6 sites 1 and 2, so a recovery here is just as unambiguous: an authoritative 'working' verdict with no declared wait in flight means the wedge for this pane is no longer 'wedged' - it is a static pane in front of an active pipeline. Counter is intentionally NOT reset by the lift. The next wedge episode starts from where the previous one left off, so the cap fires on the first wedge_timer_check call after this lift and the LLM sees ONE 'PERMANENTLY-WEDGED' per wedge episode rather than a continuous drain. If the worker wedges in cycles (wedges, recovers, wedges), each cycle bounded by FM_WEDGE_MAX_ESCALATIONS escalations plus one cap fire - same as the original bounded-wake design, just restarted per recovery event. Tests: updated tests/fm-watch-wedge-cap.test.sh: - test_wedge_cap_suppresses_subsequent_polls_for_same_hash now sets FM_FAKE_CREW_STATE=paused during the suppression check (worker is genuinely still wedged, so v7 site 3 does NOT fire and the cap holds). - new test_wedge_cap_lifts_on_same_hash_worker_active_without_pause exercises v7 site 3: cap fires, FM_FAKE_CREW_STATE stays 'working', no declared pause - marker is lifted on the next wedge_timer_check call. Files: bin/fm-watch.sh, PATCHES.md, tests/fm-watch-wedge-cap.test.sh
|
v7 pushed (commit ce19e14). Addresses Greptile 4/5 round 6. v6 gap: v6 lifted the cap only on (a) new hash + working, or (b) same hash + was-paused + working. v6 missed the case where the pane recovers via file activity or run-step WITHOUT ever declaring v7 fix: same-hash recovery outside any declared pause. Added a third lift site in the same-hash branch where Counter is intentionally NOT reset. The next wedge episode starts from where the previous one left off, so the cap fires on the first wedge_timer_check call after the lift and the LLM sees ONE Test updates:
All 7 cap tests pass; existing wedge tests in Three lift sites total, each gated by the same authoritative Manual |
…d 7) Addresses Greptile 4/5 finding on commit ce19e14: a persistently working-classified stale pane can resume emitting terminal wakes indefinitely. Same-hash recovery (v7 site 3) removes the suppression marker while retaining a counter already at the cap, so the next expired stale timer recreates the marker and emits another terminal wake; continued working classification repeats that cycle. Each wedge/recover cycle produced another cap wake with no bound - exactly the drain the cap was supposed to bound. v8 resets the escalation counter alongside the marker at every lift site (v6 site 1 via clear_pause_tracking, v6 site 2 and v7 site 3 explicitly). Each new wedge episode now has to climb FM_WEDGE_MAX_ESCALATIONS escalations again before the cap fires, so the LLM sees at most ONE 'PERMANENTLY-WEDGED' per wedge episode - the original bounded-wake design intent, restored. Trade-offs: - Short wedges (< STALE_ESCALATE_SECS * FM_WEDGE_MAX_ESCALATIONS) produce only normal escalations, no cap wake. This is fine - short wedges are not the drain problem the cap was designed to solve. - Long wedges (> STALE_ESCALATE_SECS * FM_WEDGE_MAX_ESCALATIONS) produce 10 normal escalations + 1 cap wake per cycle, bounded. - Cycling wedges (wedges, recovers, wedges) bounded by FM_WEDGE_MAX_ESCALATIONS per cycle. Each cycle is observable. Tests: updated tests/fm-watch-wedge-cap.test.sh: - test_wedge_cap_lifts_on_same_hash_recovery (site 2) now also asserts .wedge-escalations is reset. - test_wedge_cap_lifts_on_same_hash_worker_active_without_pause (site 3) now also asserts .wedge-escalations is reset. - new test_wedge_cap_bounded_across_wedge_recover_cycles drives TWO consecutive wedge episodes and verifies each one bounded (cap fires on the second episode after FM_WEDGE_MAX_ESCALATIONS escalations from a fresh counter, not immediately on the first wedge_timer_check after recovery). All 8 cap tests pass. Existing wedge tests in tests/fm-watch-triage.test.sh still pass - no regressions. Files: bin/fm-watch.sh, PATCHES.md, tests/fm-watch-wedge-cap.test.sh
|
v8 pushed (commit 790464a). Addresses Greptile 4/5 round 7. v7 problem Greptile caught: site 3 lifted the cap marker but left the escalation counter at FM_WEDGE_MAX_ESCALATIONS. The next wedge_timer_check call after recovery re-fired the cap immediately (n=11 -> cap branch), then marker re-set; site 3 lifted again; cycle. A worker that wedges/recover/wedges produced a continuous cap wake every STALE_ESCALATE_SECS — exactly the drain the cap was supposed to bound. v8 fix: reset the escalation counter alongside the marker at every lift site.
Each new wedge episode now has to climb FM_WEDGE_MAX_ESCALATIONS escalations again before the cap fires. The LLM sees at most ONE 'PERMANENTLY-WEDGED' per wedge episode — the original bounded-wake design intent, restored. Cycle behavior with v8:
Tests:
All 8 cap tests pass. Existing wedge tests in Iteration analysis (the right thing to do, since you asked): The Greptile iteration loop on this patch has been: each round catches a new edge case in the cap's recovery semantics, my fix adds another special case, Greptile catches the next edge case in the new special case. R1..R7 are all about the relationship between the cap marker and recovery. The root cause: I kept defining "recovery" ad-hoc instead of designing the cap's end-of-life semantics upfront. The cap marker is is a binary state but the recovery semantics need to handle:
v8 is the most comprehensive design I've shipped. If Greptile round 8 finds another edge case, the right move is to question whether the cap-as-marker model is the right abstraction at all — not to add another lift site. PR head |
… Greptile round 8)
Per your guidance, this commit questions the cap-as-marker model
itself rather than adding another lift site.
Greptile R8 (on v8): 'a persistently working-classified stale pane
can clear its cap on the next poll and resume repeated terminal-wake
cycles... pause_state_class=working can remain unchanged throughout
the original wedge; consequently, the permanent marker is removed
without an intervening recovery and the same pane can repeatedly
escalate back to the cap.'
Greptile is right: the v6/v7/v8 lift sites used pause_state_class=
working as a recovery gate, but that verdict is steady-state during a
wedge (the worker is active, the pane is static), not a recovery
signal. Every Greptile round since v5 caught a new edge case in
this design (R5, R6, R7, R8) - the cap-as-binary-marker with
recovery-detection via pause_state_class is fundamentally too brittle.
v9 replaces the model: cap is bound by FM_CAP_HORIZON_SECS
(default 24h). The marker file's content is its cap-fire timestamp;
wedge_timer_check checks marker age and ignores markers older than
the horizon. A new wedge on the same (window, hash) can re-fire the
cap after the horizon elapses. A new hash naturally invalidates the
marker (different key). Operator can manually rm for immediate
re-engagement.
Why this is the right model:
- The cap is bounded: at most 1 cap wake per (window, hash) per
FM_CAP_HORIZON_SECS, regardless of worker recovery behavior.
- The cap is time-bounded, not behavior-bounded - so it doesn't
depend on 'what counts as recovery', which is the question every
Greptile round has been about.
- The hash change + manual rm paths give the LLM/operator
legitimate ways to re-engage.
Trade-offs (deliberate):
- A genuinely recovered pane that wedges again on the same hash
within FM_CAP_HORIZON_SECS gets no cap wake. The LLM has to wait
for the horizon or for the operator.
- For very long STUCK wedges, the cap fires every horizon instead
of once. Bounded, not unbounded.
- For cycling wedges (wedges, recovers, wedges), each cycle bounded
by the horizon. Not a continuous drain.
Tests: rewrote tests/fm-watch-wedge-cap.test.sh around the new
semantics:
- cap fires PERMANENTLY-WEDGED and writes per-(window, hash) marker
- subsequent polls silent within horizon
- cap persists across pause-class transitions when worker is
genuinely waiting (FM_FAKE_CREW_STATE=paused)
- cap re-fires after horizon elapses (backdated marker)
- cap holds within horizon (recent marker)
- new hash invalidates marker naturally (different key)
- operator rm bypasses horizon for immediate re-engagement
- invalid FM_WEDGE_MAX_ESCALATIONS override falls back to default 10
Removed tests for v6/v7/v8 lift sites (no auto-lift in v9).
All 8 cap tests pass. Existing wedge tests in
tests/fm-watch-triage.test.sh still pass - no regressions.
Files: bin/fm-watch.sh, PATCHES.md, tests/fm-watch-wedge-cap.test.sh
|
v9 pushed (commit 929f1d9). Per your guidance, this questions the cap-as-marker model itself rather than adding another lift site. Greptile R8 was right: the v6/v7/v8 lift sites used v9 model: cap is bound by FM_CAP_HORIZON_SECS (default 24h). The marker file's content is its cap-fire timestamp. Removed (deliberately):
Why this is the right model:
Trade-offs (deliberate):
Tests: rewrote tests/fm-watch-wedge-cap.test.sh around the new semantics - cap fires, holds within horizon, re-fires after horizon, hash change invalidates, operator rm bypasses, invalid override falls back. 8 tests, all pass. No regressions in fm-watch-triage.test.sh. Final version arc: v1 (initial) -> v9 (cap horizon). The model changed in v9; v2-v8 were attempts at recovery-detection lift sites that the Greptile loop revealed as fundamentally too brittle. If Greptile R9 finds another edge case, the right move is no longer 'add another lift site' - v9 has no lift sites. The right move is to tune FM_CAP_HORIZON_SECS or accept the trade-off. |
…(v10, Greptile round 9) Addresses Greptile 4/5 finding on commit 929f1d9: FM_CAP_HORIZON_SECS=0 passes the existing 'non-integer' validation regex (0 is all-digits), causing every marker to expire immediately and re-enabling the v3 'cap fires every stale interval' failure mode. The retained escalation count keeps emitting terminal wakes after each stale interval. v10 mirrors the FM_WEDGE_MAX_ESCALATIONS validation pattern from v3: reject 0 (and non-integer), fall back to the default (86400 = 24h), log a triage_log warning so the bad config is visible. The validation is at the top of fm-watch.sh, before any code that uses the value. This is a small fix - the cap-horizon model from v9 is correct. The gap was a missing validation, not a design flaw. The horizon-based design bounds silent-suppression to a fixed time window without depending on an ambiguous recovery verdict. Tests: extended test_wedge_cap_validates_invalid_override to cover both FM_WEDGE_MAX_ESCALATIONS and FM_CAP_HORIZON_SECS (0 and 'abc' each). All 8 cap tests pass. No regressions in fm-watch-triage.test.sh. Files: bin/fm-watch.sh, PATCHES.md, tests/fm-watch-wedge-cap.test.sh
|
v10 pushed (commit a86a381). Addresses Greptile 4/5 round 9. Greptile R9 caught a validation gap: FM_CAP_HORIZON_SECS=0 passed the existing validation (0 is all-digits, regex v10 fix: mirror the FM_WEDGE_MAX_ESCALATIONS validation pattern from v3 — reject 0 and non-integer values for FM_CAP_HORIZON_SECS, fall back to default (86400), log a triage_log warning. case "$FM_CAP_HORIZON_SECS" in
''|*[!0-9]*) triage_log "FM_CAP_HORIZON_SECS='...' is not a positive integer, falling back to 86400"; FM_CAP_HORIZON_SECS=86400 ;;
0) triage_log "FM_CAP_HORIZON_SECS=0 would expire the cap immediately..., falling back to 86400"; FM_CAP_HORIZON_SECS=86400 ;;
esacTests: extended test_wedge_cap_validates_invalid_override to cover FM_CAP_HORIZON_SECS=0 and =abc. All 8 cap tests pass. No regressions in fm-watch-triage.test.sh. This was a gap, not a model flaw. The cap-horizon model from v9 is correct — it bounds silent-suppression to a fixed time window without depending on an ambiguous recovery verdict. The gap was a missing validation, exactly like v3's FM_WEDGE_MAX_ESCALATIONS=0 validation. Mirroring that pattern fixed it. PR head |
|
Speaking as Kun's firstmate: Re-reviewed newer HEAD The newer activity adds focused regression coverage and validates
The terminal wake text also says silence lasts “until pane recovers,” while the implementation is horizon/hash/manual-rm based; that contract should be made accurate. This is waiting on the author, not the captain. Drop |
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
2 participants
Summary
In
bin/fm-watch.sh,wedge_timer_check()increments a per-stale-hash escalation counter and emits astale: ... demand-deep-inspectionwake once it crossesFM_WEDGE_DEMAND_INSPECT_COUNT(default 3). The intent — documented in the constant's comment — is that the marker "forces a closer look instead of another routine supervision resume."In human-supervised setups this works: the marker is visible, a human (or smart supervisor) closes the wedge or kills the pane, and the loop breaks.
In LLM-supervised unattended setups (herdr + an LLM agent receiving wakes as prompt input), the marker is read but never acted on: the agent's default behavior is "respond to every wake." Wakes keep firing every ~
STALE_ESCALATE_SECS(default 240s), the counter grows without bound, and the agent loop hammers the model API until the subscription is drained.Concrete impact
This exact pattern drained ~359M tokens from a paid model subscription on 2026-08-18, driven by herdr panes whose cwds pointed to deleted worktrees (so they were immediately idle/wedged) and a Firstmate pi session configured on a paid model. Observed escalation counts in the session: 70, 72, 112, 113, 128, 129 — all on the same wedged panes, all within one session.
Proposal
Add a per-stale-hash cap. After
FM_WEDGE_MAX_ESCALATIONS(new constant, default 10) consecutive escalations on the same hash:PERMANENTLY-WEDGEDmarker, thenSTATE/.wedge-permanent-<key>marker, thenThe reset sites already exist (
handle_paused_stale,clear_pause_tracking); the patch just adds the permanent-wedge file to theirrm -flists.Why this preserves the existing design
demand-deep-inspectionmarker atFM_WEDGE_DEMAND_INSPECT_COUNTis unchanged — human/smart-supervisor paths see the same signal as today.n=0again.Defaults
FM_WEDGE_MAX_ESCALATIONS = 10≈ 40 min of unattended signaling (10 × defaultSTALE_ESCALATE_SECS=240s). Enough for any human or supervisor to act; short enough to bound the burn.FM_WEDGE_MAX_ESCALATIONS=5(~20 min)FM_WEDGE_MAX_ESCALATIONS=999999(revert to upstream behavior)Patch contents
bin/fm-watch.sh(+38 / -3):FM_WEDGE_MAX_ESCALATIONS(default 10).wedge_timer_checkshort-circuits ifSTATE/.wedge-permanent-<key>exists.n >= FM_WEDGE_MAX_ESCALATIONS: write the permanent marker, emit one terminalPERMANENTLY-WEDGEDwake, return.handle_paused_stale,clear_pause_tracking) extended to remove the permanent marker on genuine recovery.Verified locally:
bash -n bin/fm-watch.shpasses;git apply --checkagainst currentmain(03bb1d8) clean.Revert safety
The patch is non-invasive: it only adds a new constant, two
ifbranches inwedge_timer_check, tworm -fentries on existing reset paths, and a short-circuit at the top of the function. Behavior withFM_WEDGE_MAX_ESCALATIONS=999999is identical to upstream.Why not just disable the v1 watcher / herdr auto-restore?
Either would lose real supervision functionality (idle detection, error surfacing, agent restore on herdr restart). The cap preserves all of that and only bounds the failure mode.