Firstmate burns through my credits too quickly because of excessive watcher events

[gamedevcb]

Summary

When running Firstmate through my chatGPT subscription (Sol= firstmate, Luna= crewmates), it rapidly burned through my weekly allowance. When I investigated where my credits went, it turned out that over 80% of them were used up by Sol, repeatedly resending the cache, thousands of times via watcher events triggering checking-in on crewmates. Not sure if this is a bug, but it is definitely undesirable.

Overview

With Pi as the primary harness, one actionable Firstmate watcher wake can expand into many model provider requests because the injected operational message contains only the event reason and an instruction to run bin/fm-wake-drain.sh.

The model then reconstructs current task state through multiple separate tool/result cycles. Each cycle is another provider request carrying the primary session context.

This is not a request to suppress genuine actionable wakes or weaken lifecycle safety. It is a request to provide one bounded, structured wake-context packet so the primary agent does not have to reconstruct routine operational state incrementally.

Environment

• Firstmate commit: 99533c5d7d3702050e6084429dddff6ea4fe1aa0
• Primary harness: Pi
• Runtime backend: Herdr
• Primary model: GPT-5.6 Sol
• Platform: Linux
• Firstmate home: dedicated profile

Measured evidence

After removing exact copied usage entries introduced by a parent-session clone/fork:

• 1,605 unique Sol provider-activity records
• 1,200 watcher-originated provider calls
• 178,589,184 of 220,382,208 unique cached-input tokens were watcher-originated
• watcher-originated activity therefore represented about 74.8% of calls and 81.0% of cached input

One inspected watcher:signal run expanded into 18 provider requests.

The sequence included:

1. fm-wake-drain.sh
2. read full report
3. fm-crew-state.sh
4. read status log
5. Git/evidence/decision checks
6. teardown
7. drain another wake
8. process and PID investigation
9. creation and dispatch of a follow-up review task
10. worker-state confirmation
11. final user-facing response

Some of these actions were substantively necessary. The performance problem is that routine state acquisition and lifecycle reconciliation required many separate model continuations.

Current behavior

The Pi watcher injects a message equivalent to:

Run bin/fm-wake-drain.sh first and handle the queued wake.

The model must then decide which state files, reports, endpoint checks, decision holds and repository checks to query.

Expected behavior

The watcher/extension should preserve all current durable-queue and safety semantics while delivering one bounded structured packet containing enough deterministic state for the model to make its first substantive decision.

Proposed direction

Add a helper or extension-owned path such as:

• bin/fm-wake-context.sh --json
• schema fm-wake-context.v1

The helper should:

1. Drain the durable wake queue exactly once under the existing authority and locking contract.
2. Coalesce .status and .turn-ended records for the same task.
3. Resolve fm-crew-state.sh once per affected task.
4. Return bounded recent status-event history.
5. Return endpoint/liveness state.
6. Return report path, existence and size, but not full report content.
7. Return decision-hold state where applicable.
8. Return bounded repository state where lifecycle handling requires it.
9. Return watcher-health and additional-pending-wake state.
10. Preserve full evidence on disk and expose paths for targeted reads.

The Pi extension could run this deterministic helper before injecting the operational user message, so the first model request already contains the packet.

Safety requirements

• Do not suppress terminal, failure, blocker, decision or genuinely stale events.
• Do not embed full reports, logs or diffs.
• Preserve queue-before-detector-state ordering.
• Preserve lock-refusal behavior.
• Preserve watcher-continuity and turn-end-guard behavior.
• Keep existing fm-wake-drain.sh compatibility.
• Treat ambiguous state as ambiguous rather than silently working or done.

[xmadscientist]

Having the same issue here.

[josh-padnick]

Second data point on a different harness and runtime, with the same conclusion.

I hit this on the Grok CLI (grok-4.5) as the primary harness, Herdr backend, macOS. Firstmate commit b5d430d6fdcd961ce9b681bf196f365c1825c284. One prepaid balance drained three times in a single day, roughly $100 of it in the final 141 minutes, on a day with almost no substantive work requested.

Measurements

From ~/.grok/logs/unified.jsonl (shell.turn.inference_done records) over a ~15.5h window:

Metric	Value
Model calls	1,187
Total input tokens	290,264,222
Cached input tokens	287,391,488 (99.0%)
Output tokens	245,875
Mean input tokens per call	262,000
Shell tool calls	1,210

Turn-starting prompts, classified by origin:

Origin	Count
Background-task-completion wakes	234
Actual human messages	~24

Nearly every one of those 234 wakes was bin/fm-watch-arm.sh completing. Median background-arm duration was 40 seconds, so the loop is: arm completes -> synthetic user message -> full paid turn on ~262k tokens -> re-arm -> completes 40s later.

Cost attribution over the last top-up window (665 calls, 174,154,757 input tokens, 141 minutes): the balance went from 10,299 credits to 0, about 15 credits per model call, or roughly 59 credits per million input tokens. Because the context is ~99% cached, the spend is almost entirely cache-read on a context that never shrank.

This lines up closely with the original report: watcher-originated activity dominating both call count and cached input.

A Grok-specific amplifier worth noting

The two supervision protocols differ in a way that matters for cost.

docs/supervision-protocols/claude.md states the re-arm happens "with no model command and no model tokens" - it is a Stop hook.

docs/supervision-protocols/grok.md has no equivalent. The arm is a tracked background command, and the protocol documents that "Grok injects a synthetic user message with synthetic_reason: task_completed when the background arm completes," after which step 4 instructs the model to re-arm. So on Grok every watcher cycle close bills a full model turn carrying the entire session context, even when the wake turns out to be benign.

On a 262k-token context at a 40-second median cycle, that is a standing charge of roughly $0.15 per cycle for doing nothing. Codex and Pi appear to sit closer to the Grok shape than the Claude one, which may explain why the original report and this one land on such similar percentages from different vendors.

Support for the proposed direction

The fm-wake-context.sh --json packet proposed above would cut the calls-per-wake multiplier, and I would use it immediately.

It is worth pairing with a second lever, though: the packet reduces how many calls each wake costs, but not what each call costs. With a ~262k-token context, even a perfectly efficient one-call wake handling is expensive, and #1955 / #560 (always-loaded AGENTS.md at ~92KB) are the floor under that number. Both levers multiply.

A third, cheapest lever: on harnesses where the re-arm cannot be free, the arm could avoid waking the model at all when the cycle closes with no actionable reason. Today a benign close still costs a full turn.

Happy to supply the raw aggregation script or fuller per-hour breakdowns if useful.

[Cl3MM]

Independent confirmation from another fleet, with measurements, plus three self-inflicted wake loops the original report does not name.

Environment

• Primary harness: Claude Code (the original report used Pi, so this is not harness-specific)
• Runtime backend: Herdr
• Topology: 1 primary + 2 persistent secondmates, each supervising its own crewmates
• Symptom: ~75% of a weekly plan consumed in a single day, two session-limit hits, for 8 merged MRs

Measured evidence

The watcher itself is not the problem. state/.watch-triage.log shows it absorbing wakes correctly with zero model cost:

588 wakes absorbed without waking a model
  281 absorbed stale
  182 absorbed heartbeat
   63 absorbed busy
   26 absorbed non-terminal stale timer reset
   36 absorbed benign signal

The problem is what agents choose to append to state/<id>.status. Every appended line is a wake, and every wake is a full-context model turn for the primary. Over one day, across two secondmates:

verb	count	actionable?
done	68	rarely
working	58	no
resolved	51	no - see loop 1
needs-decision	38	yes
failed	14	yes
paused	6	no
blocked	5	yes

240 status lines, ~76% of which required no supervisor action. Across both secondmates: 240 lines / 57 actionable.

Startup digest measured at 12,897 estimated tokens against a configured config/startup-memory-budget of 7,500 (72% over), driven mainly by 127KB of accumulated status logs.

Three self-inflicted loops the original report does not describe

Loop 1 - the primary is woken to be shown its own answer.
Answering a decision with bin/fm-send.sh <task> --resolve-key <key> '<answer>' appends a resolved [key=...]: answered: <the primary's own text> line to the status log. That append is a signal, which wakes the primary, which drains, and is presented its own message back. 51 occurrences in one day = 51 full-context model turns to re-read text the primary just wrote.

Loop 2 - a status-discipline instruction cannot be obeyed.
Instructing a secondmate to stop posting non-actionable status lines is itself delivered via fm-send --resolve-key, which opens a pending-reply record. The instruction forbids acknowledging; the mechanism requires a correlated reply. The absent reply becomes pending-reply-missed, a durable blocked: decision that wakes the primary indefinitely. Closing that with another --resolve-key message opens a new pending-reply, which becomes a new blocked:. The loop is self-replicating - observed three generations deep before it was broken by hand.

Loop 3 - the fix for loop 2 fails silently.
pending-reply- is a reserved decision-key namespace (FM_CLASSIFY_RESERVED_KEY_PREFIXES_DEFAULT in bin/fm-classify-lib.sh). A resolved [key=pending-reply-<id>] line is honoured only if its note begins with that namespace's own vocabulary token. A close written in ordinary prose is folded as ordinary status and silently discarded, leaving the decision permanently open. Editing phase= in state/pending-replies/<id> does not close it either. The only working form is:

resolved [key=pending-reply-<id>]: pending-reply-closed: <text>

The reserved-namespace rule is sound - it prevents key hijacking - but it is mute on failure, so a well-intentioned close is indistinguishable from no close at all.

A separate, larger multiplier worth naming in the same issue

config/crew-dispatch.json is optional, and when it is absent every crewmate and secondmate inherits the primary's own harness and model at default effort. For a primary running a frontier model, this silently launches every implementation crewmate on that frontier model - and implementation is by far the highest turn-count work in the fleet. There is no startup diagnostic for "no dispatch profile configured, so all workers inherit the primary model", so an operator has no signal that this is happening. Creating the file cut the dominant cost immediately.

Suggestion: emit a BOOTSTRAP_INFO: line naming the effective crewmate harness/model/effort when no dispatch profile exists, so inheritance is visible rather than implicit.

Supporting the proposed direction, with two additions

The proposed bin/fm-wake-context.sh --json packet would remove the reconstruct-state-incrementally amplification described in the original report, and we saw exactly that shape here (2-4 tool calls per wake, each a full-context request).

Two additions from this data:

1. A wake that carries no actionable content should not reach the model at all. working:/done:/resolved: are already distinguishable from needs-decision:/blocked:/failed: in bin/fm-classify-lib.sh. The away-mode daemon already applies exactly this triage in bash. Applying the same filter when away mode is off would have removed ~76% of this fleet's model wakes without touching any safety-relevant event.
2. Loop 1 is fixable in isolation and cheaply: an appended resolved line whose author is the primary itself should not signal the primary.

[kunchenguid]

Speaking as Kun's firstmate:

This is a real token-efficiency hole. A Pi watcher wake still injects only a reason plus “run fm-wake-drain.sh,” so the primary reconstructs fleet state across many tool rounds. That is the opposite of lean context.

Marking ready-for-pr. The ask is a bounded structured wake-context packet, not suppressing genuine actionable wakes. Related flood bugs (#2615, #2573) stay separate.

[kunchenguid]

Speaking as Kun's firstmate:

This is a real, unfixed token-burn. Independent measurements on Pi, Grok, and Claude all show watcher-originated activity dominating call count and cached input. The ask is not to suppress genuine actionable wakes.

VISION

• Token efficiency is a first-class concern: aligns. Inspected bin/fm-wake-drain.sh on main: drain still emits queued wakes plus OPEN DECISIONS, then the model reconstructs crew/report/endpoint/decision state through many full-context tool turns. No bin/fm-wake-context.sh exists.
• Scripts own the mechanics, agents own the judgment: aligns. Deterministic wake-context belongs in a script; spending a frontier turn on fm-crew-state.sh + status log + liveness is the inversion this rule forbids.
• Peace of mind / looking away: aligns. The defect is always-on operational noise. A bounded packet plus absorbing non-actionable wakes (when that does not hide a failure, decision, or risk) serves ease of looking away.
• Honest interface (batching and silence never hide a failure, decision, or risk): aligns only with the report's safety requirements. Do not suppress terminal, failure, blocker, decision, or genuinely stale events. Treat ambiguous state as ambiguous.
• Fleet outlives any vendor / quota stays captain-owned: aligns. docs/supervision-protocols/grok.md still bills a full synthetic user turn on every background-arm completion; docs/supervision-protocols/claude.md re-arms on Stop with no model tokens. That harness gap is part of the same cost family, not a reason to ignore Pi/Codex.

Related, not a duplicate: #2615 is a Pi recursive re-arm flood of the same wake (already ready-for-pr). This issue is the per-wake reconstruction multiplier plus non-actionable status chatter. No competing implementation PR found.

A sound first PR is a default-off or always-safe fm-wake-context packet that preserves queue/lock/drain contracts, plus not waking the model for already-classifiable non-actionable closes on harnesses where re-arm is not free. Self-wake on the primary's own resolved: line is a cheap, isolated fix called out in-thread.

[Cl3MM]

Implementation findings from a cancelled local attempt

I built a local fm-wake-context.v1 prototype for Claude, Codex, and Pi, then cancelled its automated validation because the review loop itself was consuming an unacceptable amount of model quota.

Nothing from this attempt has been pushed or shipped. The last ordinary local commit is 4b16b7e, but it should not be deployed as-is.

The implementation and independent reproductions established several useful invariants:

• presentation must be staged before any cursor advances
• replay before ACK must be byte-identical
• ACK must commit cursors monotonically and never move a newer cursor backward
• packet limits must be measured in bytes, including UTF-8 payloads
• a post-presentation fallback needs a durable receipt tied to the exact ACK
• Pi must turn ENOBUFS into one canonical bounded fallback rather than forwarding truncated output
• a manual drain must supersede any older fallback receipt and staged cursor

The final review found seven legitimate issues that remain unresolved in the local prototype:

1. presentation.stdout can still embed an entire small status log. It needs a bounded projection with a tail and omitted-line count.
2. Task check wakes can use absolute keys such as $STATE/alpha.check.sh. The current parsing can reject the slash-containing key and return tasks: [].
3. A valid status-only recovery uses --ack-through 0, but the prototype can serialize that ACK as null, preventing cursor commit and causing replay.
4. ACK cursor commit needs a monotonic merge or compare-and-swap under the presentation lock. Blindly restoring an older staged cursor can replay already-presented statuses.
5. Backend liveness probes need a hard timeout. A stalled endpoint can otherwise block both the packet and its fallback. Pi also needs a spawnSync timeout.
6. The wake cardinality cap must be checked against the post-presentation snapshot. A concurrent seventeenth wake can currently enter a packet that advertises a limit of sixteen.
7. The Pi ENOBUFS regression test must exercise the public actionable-close path through sendUserMessage, not a directly exported private helper.

There is no honest before/after provider-token benchmark from this attempt. The only proven result is that the design can reduce deterministic state reconstruction to one executable packet invocation for the tested fixture. It does not prove a quota reduction in a real session.

Anyone continuing should treat the seven findings above as acceptance criteria and should avoid reusing the local patch without fixing them first.

[kunchenguid]

Speaking as Kun's firstmate:

Still real and unfixed on main 1231b6ae7fd4c5ff7e94d2f5ca2159536c4c41cb. Cl3MM's 8/22 comment is a cancelled local fm-wake-context.v1 attempt: nothing was pushed, and the seven review findings remain open. No implementation PR is open, so there is nothing to help and no competing PR to open.

VISION

• Token efficiency is a first-class concern: aligns. Inspected bin/ on main: bin/fm-wake-context.sh is still 404. A Pi/Grok/Codex watcher wake still injects a reason plus run bin/fm-wake-drain.sh, then the primary reconstructs crew/report/endpoint/decision state across many full-context tool turns.
• Scripts own the mechanics, agents own the judgment: aligns. Deterministic wake-context belongs in a script. Spending a frontier turn on fm-crew-state.sh + status log + liveness is the inversion this rule forbids.
• Honest interface (batching and silence never hide a failure, decision, or risk): aligns only with the original safety requirements. Do not suppress terminal, failure, blocker, decision, or genuinely stale events. Treat ambiguous state as ambiguous.
• Peace of mind / looking away: aligns. Watcher-originated cached-input burn is always-on operational noise. Absorbing already-classifiable non-actionable closes (when that does not hide a failure) serves ease of looking away.
• Fleet outlives any vendor / quota stays captain-owned: aligns. Independent measurements on Pi, Grok, and Claude still stand; Cl3MM's cancelled prototype does not change main.

A sound first PR remains a bounded fm-wake-context packet that preserves queue/lock/drain contracts, plus not waking the model for already-classifiable non-actionable closes on harnesses where re-arm is not free. Anyone continuing should treat Cl3MM's seven findings as acceptance criteria and should not reuse the unpushed local patch until those are fixed: bounded status projection (not embedding a whole log), slash-containing check keys, --ack-through 0 must not serialize as null, monotonic ACK cursor commit, hard timeouts on liveness probes, wake-cardinality cap against the post-presentation snapshot, and Pi ENOBUFS coverage through the public sendUserMessage path.

Related, not a duplicate: #2615 (Pi recursive re-arm of the same wake).

Restamping ready-for-pr. This is not a merge vote.

[kunchenguid]

Speaking as Kun's firstmate:

Newer activity from @Cl3MM (cancelled local fm-wake-context.v1 prototype). The defect is still on current main (1231b6ae7fd4): watcher-originated wakes still burn tokens without a durable per-wake context bound. A cancelled local attempt is not a landing.

VISION: token efficiency is first-class; silence and batching must not hide a failure, but they also must not spend tokens on mechanics that are not news. Aligns as a real unfixed burn.

Left open, still ready-for-pr. Help any later PR; do not open a competing one. Distinct from #2356 (quiet-while-present).