Skip to content

fix: restart unhealthy wrapper instances - #6

Merged
LYJW131 merged 1 commit into
mainfrom
agent/wrapper-self-heal
Jul 18, 2026
Merged

fix: restart unhealthy wrapper instances#6
LYJW131 merged 1 commit into
mainfrom
agent/wrapper-self-heal

Conversation

@LYJW131

@LYJW131 LYJW131 commented Jul 18, 2026

Copy link
Copy Markdown
Member

What changed

  • quarantine a wrapper instance immediately when the manager-owned local I/O deadline expires
  • require repeated cross-connection and cross-Adam-ID evidence before escalating EOF/reset failures
  • remove unhealthy instances from dispatch by pointer and terminate the exact wrapper generation once
  • escalate from SIGINT to SIGKILL when a wrapper does not exit within the grace period
  • release decrypt and M3U8 ports when a wrapper generation exits
  • add timeout, cancellation, concurrency, replacement-generation, routing, and process-termination tests

Root cause

Wrapper context creation enters non-cancelable native FairPlay calls while holding a process-global context mutex. If a native call wedges, new wrapper connection threads remain blocked behind that mutex. The manager previously discarded only the timed-out TCP connection and then reconnected to the same unhealthy process, causing the failure to accumulate until even concurrency one timed out.

Impact

The gRPC API and configuration remain unchanged. A manager-owned 30-second loopback I/O timeout now quarantines and restarts only the affected wrapper. Client cancellation and client-owned deadlines do not trigger recovery. Concurrent failures schedule exactly one restart, and stale failures cannot remove or terminate a replacement with the same wrapper ID.

Validation

  • GOTOOLCHAIN=go1.23.5 go test ./...
  • GOTOOLCHAIN=go1.23.5 go test -race ./...
  • GOTOOLCHAIN=go1.23.5 go vet ./...
  • critical concurrent recovery tests under -race -count=100
  • ROG production candidate: two deterministic SIGSTOP injections each produced exactly one quarantine/down/replacement cycle
  • with normal backend retry policy, the injected-failure 8-track job completed 8/8 without manual manager restart
  • post-recovery real-media soak completed Fate 62/62 followed immediately by Tsukihime 8/8

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: LYJW131 <lyjw2007@gmail.com>
@LYJW131
LYJW131 merged commit 9dfd88c into main Jul 18, 2026
1 check passed
@LYJW131
LYJW131 deleted the agent/wrapper-self-heal branch July 18, 2026 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant