Skip to content

Capture hung-child stacks (SIGABRT) and slim pool-child startup#124

Merged
maximusunc merged 1 commit into
mainfrom
claude/omnicorp-worker-pool-error-ytpcn2
Jul 13, 2026
Merged

Capture hung-child stacks (SIGABRT) and slim pool-child startup#124
maximusunc merged 1 commit into
mainfrom
claude/omnicorp-worker-pool-error-ytpcn2

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Diagnosing the omnicorp "hang" showed the child never reached the overlay ("Start omnicorp" never logged) -- it was wedged in its own cold startup, which a spawned pool child runs before every task (the worker is launched as a script, so spawn re-imports worker.py top-to-bottom). Two changes:

  1. Kill stuck children with SIGABRT before SIGKILL. faulthandler (already enabled per child) catches SIGABRT and dumps the child's C+Python traceback to stderr, so the next hang leaves an actionable stack instead of a silent kill. A 0.5s grace lets the dump land; anything still alive (ignored SIGABRT) is then SIGKILLed so the slot is always freed.

  2. Stop pool children from re-running the parent's startup work:

    • otel.setup_tracer now no-ops in spawned children (they emit no spans -- the task span lives in the parent) and imports the OTLP/grpc stack lazily, so a child never pays the grpc init / cold-start stall.
    • logger only attaches the RotatingFileHandler in the main process. Locally every child re-ran setup_logging and attached the handler, so N processes drove one non-multiprocess-safe handler on the same bind-mounted file -- a prime suspect for the local stall (prod is console-only via the Kubernetes branch, so this was local-only). Children now log to console.

Both use current_process().name, which spawn sets before re-importing the worker module, so it's reliable at import time. This removes the two most likely cold-start hang sources and instruments the path so the next occurrence is self-explaining.

Claude-Session: https://claude.ai/code/session_01TiiFCcToq8LgiEeaQhQM5e

Diagnosing the omnicorp "hang" showed the child never reached the overlay
("Start omnicorp" never logged) -- it was wedged in its own cold startup, which
a spawned pool child runs before every task (the worker is launched as a script,
so spawn re-imports worker.py top-to-bottom). Two changes:

1. Kill stuck children with SIGABRT before SIGKILL. faulthandler (already
   enabled per child) catches SIGABRT and dumps the child's C+Python traceback
   to stderr, so the next hang leaves an actionable stack instead of a silent
   kill. A 0.5s grace lets the dump land; anything still alive (ignored SIGABRT)
   is then SIGKILLed so the slot is always freed.

2. Stop pool children from re-running the parent's startup work:
   - otel.setup_tracer now no-ops in spawned children (they emit no spans -- the
     task span lives in the parent) and imports the OTLP/grpc stack lazily, so a
     child never pays the grpc init / cold-start stall.
   - logger only attaches the RotatingFileHandler in the main process. Locally
     every child re-ran setup_logging and attached the handler, so N processes
     drove one non-multiprocess-safe handler on the same bind-mounted file --
     a prime suspect for the local stall (prod is console-only via the
     Kubernetes branch, so this was local-only). Children now log to console.

Both use current_process().name, which spawn sets before re-importing the worker
module, so it's reliable at import time. This removes the two most likely
cold-start hang sources and instruments the path so the next occurrence is
self-explaining.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TiiFCcToq8LgiEeaQhQM5e
@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 48.64865% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.04%. Comparing base (0b6be20) to head (ac19722).
⚠️ Report is 20 commits behind head on main.

Files with missing lines Patch % Lines
shepherd_utils/otel.py 0.00% 10 Missing ⚠️
shepherd_utils/process_pool.py 59.09% 6 Missing and 3 partials ⚠️
Files with missing lines Coverage Δ
shepherd_utils/logger.py 100.00% <100.00%> (ø)
shepherd_utils/process_pool.py 70.52% <59.09%> (-14.09%) ⬇️
shepherd_utils/otel.py 0.00% <0.00%> (ø)

... and 6 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1b5d2e...ac19722. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@maximusunc
maximusunc merged commit b22f0cf into main Jul 13, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/omnicorp-worker-pool-error-ytpcn2 branch July 13, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants