Skip to content

fix(core): resolve a child's exit from /proc when Bun never reports it - #40

Merged
PierrotAWB merged 1 commit into
devfrom
lost-child-exit-watchdog
Sep 3, 2026
Merged

PierrotAWB merged 1 commit into
devfrom
lost-child-exit-watchdog

Conversation

@PierrotAWB

@PierrotAWB PierrotAWB commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator

A tool call whose child process has already exited can no longer wedge a turn forever. When Bun never reports the exit, the spawner now reads it from /proc within a second, and the kill path can no longer wait unboundedly after SIGKILL.

Context

The existing-site-lifecycle canary has failed its "agent turn completes" step in ~15% of runs since Aug 26, every one of them a bash tool call that ran a browser command (replo-browser attach … && playwright-cli snapshot) and never came back. Caught live on the canary sandbox (opencode pid 686, Bun v1.3.14, kernel 6.17): the bash tool's shell had exited and sat as a Z zombie under opencode for 50+ minutes; opencode held pidfds for its other live children but none for the zombie and never called waitpid; nothing else held its pipes; the part stayed running, session/status reported idle, and opencode logged nothing further for that session. Isolated repros (48 direct shell runs, 40 through POST /session/:id/shell, 5 real bash-tool turns) never hung, so it needs the canary turn's context, most likely capture-html's in-process Chromium exiting a few seconds earlier. Bun's own 1.3.14 notes describe this class (lost pidfd exit when multiple children exit close together) as fixed for stdio: "ignore"; it evidently is not for piped, detached children.

The exit Deferred in cross-spawn-spawner.ts resolves only from Node's close event. When that never fires, handle.exitCode never resolves, and the bash tool's 120s timeout fires into an acquire/release finalizer that sends SIGTERM, awaits the same Deferred, escalates to SIGKILL after 3s, then awaits it again with no bound. Finalizers are uninterruptible, so the timeout is swallowed silently.

Why it surfaced now: the canary user's model moved from sonnet-5 to gpt-5.6-terra via the replo-agent-model flag on the same days. Sonnet never ran a browser command on that sandbox; Terra runs one in most turns. All 15 hung runs sit inside Terra windows, zero in Sonnet windows.

What changed

  • getExit(stat) decodes /proc/<pid>/stat: returns the waitpid-shaped [code, signal] from field 52 when the state is Z, otherwise undefined. Verified against the live zombie's stat line.
  • spawn starts a Linux-only, unref'd 1s poller alongside the close listener; whichever observes the exit first completes the Deferred and clears the poller. A reaped-and-gone child yields undefined and the poller keeps waiting for Bun's event, which owns that case.
  • terminate() replaces the identical SIGTERM → wait → SIGKILL → wait sequence in the scope release and handle.kill; the wait after SIGKILL is now bounded by forceKillAfter via Effect.timeoutOption.

Net: one source file, +76/−39 including the dedupe.

What to scrutinize

The poller resolves only on state Z, never on ENOENT, so it cannot race Bun's own reaping into a wrong status. The status decode assumes the Linux waitpid layout (status & 0x7f signal, status >> 8 code); that is what /proc documents for field 52. On macOS and Windows the poller is not created. Zombies stay unreaped until opencode restarts, which on sandboxes is every boot.

Testing done

  • packages/core bun test test/effect/cross-spawn-spawner.test.ts test/process/process.test.ts: 49 pass, including the new getExit decode test (exit code, signal, running, empty).
  • bun run typecheck in packages/core: clean. bun lint on the changed files: only the four pre-existing warnings remain.
  • Fleet signal after the next pin bump: existing-site-lifecycle "agent turn completes" failures on Terra days should go to zero; sandbox-side, Tool execute before tool=bash with no matching after should stop appearing in {job="sandbox", service_name="opencode"}.

🤖 Generated with Claude Code


Summary by cubic

Fixes the existing-site-lifecycle canary's "agent turn completes" failures by stopping a bash tool call from wedging a turn forever when Bun never reports the child's exit. The spawner now reads the exit from /proc/<pid>/stat within a second, and the kill path can no longer wait unboundedly after SIGKILL.

What changed

  • The exit Deferred resolves from Node's close event or a Linux-only 1s poller that decodes field 52 of /proc/<pid>/stat for zombie children.
  • terminate() replaces the duplicated SIGTERM → wait → SIGKILL → wait sequence; the wait after SIGKILL is bounded by forceKillAfter.
  • The poller resolves only on zombie state, so it cannot race Bun's own reaping into a wrong status.
  • The poller only reads, so zombies remain until opencode restarts.
  • getExit is exported with tests covering exit codes, signals, running processes, and empty stat lines.

Testing

  • bun test passes 49 tests across cross-spawn-spawner and process, including the new getExit decode test.
  • Typecheck and lint pass except for pre-existing warnings.

Written for commit 26ec53e. Summary will update on new commits.

Review in cubic

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

The following comment was made by an LLM, it may be inaccurate:

Bun 1.3.14 can drop a child's exit notification and leave the process a
zombie. The spawner's exit Deferred then never resolves, so the tool call,
its timeout, and the whole turn wedge forever.

- poll /proc/<pid>/stat once a second while the exit is unknown and resolve
  the Deferred from field 52 (the waitpid status) when the child is a zombie
- bound the wait after SIGKILL so a release finalizer can never pin a fiber
- one terminate() helper replaces the duplicated kill sequence in the scope
  release and handle.kill

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@PierrotAWB
PierrotAWB force-pushed the lost-child-exit-watchdog branch from 1754d55 to 26ec53e Compare September 3, 2026 21:11

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/core/src/cross-spawn-spawner.ts">

<violation number="1" location="packages/core/src/cross-spawn-spawner.ts:382">
P1: When a scoped command omits `forceKillAfter` and ignores SIGTERM, this branch waits for `signal` forever during scope finalization. Keep finalizer cleanup non-blocking without a bound, or give that path its own timeout.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

const send = (s: NodeJS.Signals) => Effect.catch(killGroup(command, proc, s), () => killOne(command, proc, s))
const attempt = send(opts?.killSignal ?? "SIGTERM").pipe(Effect.andThen(Deferred.await(signal)), Effect.asVoid)
const bound = opts?.forceKillAfter
if (Predicate.isUndefined(bound)) return attempt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: When a scoped command omits forceKillAfter and ignores SIGTERM, this branch waits for signal forever during scope finalization. Keep finalizer cleanup non-blocking without a bound, or give that path its own timeout.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/src/cross-spawn-spawner.ts, line 382:

<comment>When a scoped command omits `forceKillAfter` and ignores SIGTERM, this branch waits for `signal` forever during scope finalization. Keep finalizer cleanup non-blocking without a bound, or give that path its own timeout.</comment>

<file context>
@@ -342,6 +370,24 @@ export const make = Effect.gen(function* () {
+    const send = (s: NodeJS.Signals) => Effect.catch(killGroup(command, proc, s), () => killOne(command, proc, s))
+    const attempt = send(opts?.killSignal ?? "SIGTERM").pipe(Effect.andThen(Deferred.await(signal)), Effect.asVoid)
+    const bound = opts?.forceKillAfter
+    if (Predicate.isUndefined(bound)) return attempt
+    // A child whose exit is never reported must not pin this fiber after SIGKILL.
+    return Effect.timeoutOrElse(attempt, {
</file context>

@PierrotAWB
PierrotAWB merged commit ee04d03 into dev Sep 3, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant