Capture hung-child stacks (SIGABRT) and slim pool-child startup#124
Merged
Conversation
Diagnosing the omnicorp "hang" showed the child never reached the overlay
("Start omnicorp" never logged) -- it was wedged in its own cold startup, which
a spawned pool child runs before every task (the worker is launched as a script,
so spawn re-imports worker.py top-to-bottom). Two changes:
1. Kill stuck children with SIGABRT before SIGKILL. faulthandler (already
enabled per child) catches SIGABRT and dumps the child's C+Python traceback
to stderr, so the next hang leaves an actionable stack instead of a silent
kill. A 0.5s grace lets the dump land; anything still alive (ignored SIGABRT)
is then SIGKILLed so the slot is always freed.
2. Stop pool children from re-running the parent's startup work:
- otel.setup_tracer now no-ops in spawned children (they emit no spans -- the
task span lives in the parent) and imports the OTLP/grpc stack lazily, so a
child never pays the grpc init / cold-start stall.
- logger only attaches the RotatingFileHandler in the main process. Locally
every child re-ran setup_logging and attached the handler, so N processes
drove one non-multiprocess-safe handler on the same bind-mounted file --
a prime suspect for the local stall (prod is console-only via the
Kubernetes branch, so this was local-only). Children now log to console.
Both use current_process().name, which spawn sets before re-importing the worker
module, so it's reliable at import time. This removes the two most likely
cold-start hang sources and instruments the path so the next occurrence is
self-explaining.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TiiFCcToq8LgiEeaQhQM5e
Codecov Report❌ Patch coverage is
... and 6 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnosing the omnicorp "hang" showed the child never reached the overlay ("Start omnicorp" never logged) -- it was wedged in its own cold startup, which a spawned pool child runs before every task (the worker is launched as a script, so spawn re-imports worker.py top-to-bottom). Two changes:
Kill stuck children with SIGABRT before SIGKILL. faulthandler (already enabled per child) catches SIGABRT and dumps the child's C+Python traceback to stderr, so the next hang leaves an actionable stack instead of a silent kill. A 0.5s grace lets the dump land; anything still alive (ignored SIGABRT) is then SIGKILLed so the slot is always freed.
Stop pool children from re-running the parent's startup work:
Both use current_process().name, which spawn sets before re-importing the worker module, so it's reliable at import time. This removes the two most likely cold-start hang sources and instruments the path so the next occurrence is self-explaining.
Claude-Session: https://claude.ai/code/session_01TiiFCcToq8LgiEeaQhQM5e