buzz-acp: zombie in-flight turn after reconnect + rate-limit; agent acks mentions but never responds

[Tony-ooo]

Summary

buzz-acp gets stuck with a zombie in-flight turn after a WebSocket disconnect + relay rate-limit. The agent process stays online and heartbeat-keepalive, incoming @mentions are accepted with ack=Ok(Success) (steer path), but the agent never produces any output again — every subsequent mention is steered into a dead turn and silently dropped.

Observed Behavior

1. Agent (claude-agent-acp runtime, 10 workers, steering_supported=true) initialized fine and worked normally for hours.
2. Relay connection dropped (Connection reset without closing handshake), then 4 autonomous reconnect attempts (several 502 Bad Gateway), finally reconnect succeeded.
3. Immediately after reconnect, relay sent NOTICE: rate-limited: quota exceeded, and the relay observer logged:

   relay observer publisher lagged dropped=108
   relay observer publisher lagged dropped=2017
   

   → 2017+ events dropped during the reconnect/backpressure window.
4. After this window, every new @mention is handled via the non-cancelling steer path and acked:

   steer success: renewed hard deadline (7200s from now)
   non-cancelling steer ack received channel=<ch> event_id=<id> ack=Ok(Success) release_withheld=false drop_withheld=true signal_fallback=false
   extending in-flight deadline by 7200s + 100s buffer channel_id=<ch>
   

5. No turn completion, no tool calls, no message publishing ever happens again. The claude worker process shows near-zero CPU growth (zombie/idle). presence stays online.

Expected Behavior

After a relay disconnect / rate-limit / event-drop, an in-flight turn should be recovered or expired so that the next @mention starts a fresh turn. The agent should keep responding to mentions rather than silently dead-ending.

Root-cause hypothesis

The in-flight turn state machine believes a turn is still running (extend in-flight deadline keeps firing), so new mentions are treated as mid-turn steer injection into a turn whose claude worker is no longer producing output. The turn is never reclaimed — the deadline just keeps getting renewed 7200s at a time, so the normal recovery path (deadline expiry → recover withheld → cancel+merge) never triggers.

The observer publisher lagged dropped=2017 during reconnect is the likely trigger: critical turn lifecycle events (completion/abort) were among the dropped events, leaving the queue with an orphaned in-flight turn.

Environment

• Platform: Windows 11 (win32)
• buzz-acp via Buzz Desktop (agent harness), agent runtime @agentclientprotocol/claude-agent-acp 0.64.2 (agent_command_override=claude-agent-acp)
• Relay: self-hosted Nostr relay (wss://buzz.tonyooo.xyz), rate-limited (quota exceeded) during the incident
• Config: subscribe=Mentions, meh=Steer, permission_mode=bypassPermissions, respond_to=allowlist, max_turn=7200s

Suggested fix directions

• After reconnect / when observer lagged ... dropped is observed, expire orphaned in-flight turns (or run the same recovery path as deadline expiry) so withheld mentions are re-dispatched as fresh turns.
• Consider a max consecutive-steer-without-output guard: if a turn is being steered repeatedly but produces nothing and its worker is idle, reclaim it.
• Emit a WARN/ERROR log when a mention is steered into a turn that has not produced output for a long time, to make this diagnosable from logs.

Log excerpts (sanitized)

2026-08-03T22:28:49Z WARN buzz_acp::relay: WebSocket error: Connection reset without closing handshake
2026-08-03T22:28:49Z WARN buzz_acp::relay: autonomous reconnect attempt 1/5 failed: HTTP error: 502 Bad Gateway
2026-08-03T22:28:56Z INFO buzz_acp::relay: autonomous reconnect succeeded (attempt 4)
2026-08-03T22:29:36Z WARN buzz_acp: relay observer publisher lagged dropped=108
2026-08-03T22:29:46Z WARN buzz_acp: relay observer publisher lagged dropped=2017
2026-08-03T22:29:46Z WARN buzz_acp::relay: relay NOTICE: rate-limited: quota exceeded; retry in 10s
# hours later, a new mention:
2026-08-04T04:37:35Z INFO buzz_acp::acp: steer success: renewed hard deadline (7200s from now)
2026-08-04T04:37:35Z INFO buzz_acp: non-cancelling steer ack received channel=... event_id=... ack=Ok(Success) release_withheld=false drop_withheld=true signal_fallback=false
2026-08-04T04:37:35Z INFO buzz_acp::queue: extending in-flight deadline by 7200s + 100s buffer
# ...no further log lines, ever. Agent never responds again.