Skip to content

feat(GenesisKit, GenesisHub, hub): a burst of child processes is logged as spawn.storm and shown by the dev monitor - #593

Merged
genesiscz merged 6 commits into
masterfrom
feat/2026-10-25-spawn-storm
Oct 26, 2026
Merged

genesiscz merged 6 commits into
masterfrom
feat/2026-10-25-spawn-storm

Conversation

@genesiscz

@genesiscz genesiscz commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #562: adds a storm event kind to src/hub/lib/dev-monitor.ts beside the spin kind that #562 adds.

A burst of child processes is logged as spawn.storm (40 starts in 10 s or 16 running, with the commonest command), the dev monitor shows it, its label leaves out commit ids and revisions, and a storm ends at the first quiet start or finish.

  • 5d20749 feat(GenesisKit, GenesisHub, hub): a burst of child processes is logged as spawn.storm (24 starts in 10 s or 12 running, with the commonest commands) by GenesisKit's launchers, and tools hub dev monitor reports it and reads Genesis Markdown's perf log and hangs too
  • c375b5f fix(GenesisKit): spawn.storm fires at 40 starts in 10 s or 16 running, so the hub opening a pull request (24 starts, 5 running) is not a storm, and git's -c option stays out of the label
  • f5aa647 fix(GenesisKit): a spawn.storm label leaves out commit ids and revisions, so git cat-file counts as git cat-file
  • 1053e0a fix(GenesisKit): a spawn.storm ends at the first start or finish that finds it quiet and is timed to its last start, so a 10 s burst no longer reads "over after 209 s"

Gates on this branch (clean worktree, 2026-10-25): bun scripts/git/check-commits.ts, root tsc --noEmit, biome on the changed files, the related tests, and scripts/ci/placeholder-check.sh. Tests: src/hub/lib/dev-monitor.test.ts 16 pass; swift build of GenesisKit and swift test --filter AgentBlameModelTests pass.

Split out of the unified branch (#507); each commit is a cherry-pick of its unified original.

Summary by CodeRabbit

  • New Features
    • Development monitoring now tracks performance logs and hang reports from both the app and Genesis.
    • Added process-burst reporting, including activity levels and the most frequent command labels.
    • Monitor reports now distinguish Genesis events and detect hang samples across app-specific folders.
  • Bug Fixes
    • Improved grouping of process-burst events and duration reporting for clearer monitor output.

@eve-bot-lovinka

eve-bot-lovinka Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

🐉 eve review — ✅ APPROVE · 1 finding

→ review · run

  • Queued 03:19:38Z
  • Reading diff — 121 files
  • Building repo map
  • Analyzing (find → verify) — 1 candidates → 1 survivors · reviewed by find codex/gpt-6.1-sol + codex/gpt-6-luna, verify codex/gpt-6-luna
  • Posting review
  • Review posted 03:27:07Z (7m 30s)

run wrun_01M5V3H67Q0357S373EQBZNZRV

Previous runs (5)
run head outcome findings took
5 c04a9a8 ✅ APPROVE 0 14m 34s
4 b298491 ✅ APPROVE 1 13m 56s
3 0a23d0f ✅ APPROVE 2 14m 2s
2 b149550 ✅ APPROVE 0 12m 26s
1 1053e0a ✅ APPROVE 4 13m 41s

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 16 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 3 included reviews currently available. Your 48 included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between dfd5ec5 and 421c572.

Files selected for processing: 10
  • .claude/skills/ux/references/findings.md
  • src/hub/commands/dev.ts
  • src/hub/lib/dev-monitor.test.ts
  • src/hub/lib/dev-monitor.ts
  • src/macos/GenesisKit/Sources/GenesisKit/Tools/ToolsLineStream.swift
  • src/macos/GenesisKit/Sources/GenesisKitBase/Perf/SpawnMeter.swift
  • src/macos/GenesisKit/Sources/GenesisKitBase/Tools/ProcessCapture.swift
  • src/macos/GenesisKit/Sources/GenesisKitBase/Tools/ToolsBridge.swift
  • src/macos/GenesisKit/Tests/GenesisKitTests/Diff/AgentBlameModelTests.swift
  • src/macos/GenesisKit/Tests/GenesisKitTests/ToolsBridgeTests.swift
Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — 🟡 Review comments

1053e0a · 4 actionable findings · reviewed by find codex/gpt-6.1-sol + codex/gpt-6-luna, verify codex/gpt-6-luna · view run ↗

Severity Count
🟡 Medium 2
🔵 Low 2

ℹ️ Checks (advisory — CI is authoritative) — typecheck timed out in the check queue: no check slot freed within 600 s; lint timed out in the check queue: no check slot freed within 600 s

Blast radius

  • label (+) → 2 dependent(s) outside this PR: src/browser-router/native/Router.swift, src/macos/GenesisKit/Tests/GenesisKitTests/KitChartsTests.swift
  • registerDevCommands (~) → 1 dependent(s) outside this PR: src/hub/cli.ts
  • started (+) → 1 dependent(s) outside this PR: src/macos/GenesisKit/Tests/GenesisKitTests/SessionTranscriptClientTests.swift
  • graph truncated by caps — dependents above are a lower bound

ℹ️ Claims vs diff — 5 claims could not be checked from the diff (c1, c3, c4, c5, c6).

Comment thread src/hub/lib/dev-monitor.ts
Comment thread src/hub/lib/dev-monitor.ts Outdated
Comment thread src/macos/GenesisKit/Sources/GenesisKitBase/Perf/SpawnMeter.swift Outdated
Comment thread src/macos/GenesisKit/Sources/GenesisKitBase/Perf/SpawnMeter.swift Outdated
genesiscz added a commit that referenced this pull request Oct 26, 2026
@genesiscz

Copy link
Copy Markdown
Owner Author

Review fixes 2026-10-26 02:22 from GPT-6.1 Sol judged by Claude Opus 5.5 (fable-judge)

Round 1. 4 open threads (eve 4, CodeRabbit 0). 4 fixed, 0 rejected, 0 already done. Threads t3 and t4 describe the same bare-HEAD defect. Full review of the nine-file PR found no additional defect. All four changed files were already in the PR. Verification: not run (the lead verifies at integration). No further tests, builds or typechecks were run after the lead's instruction; the commit hook's automatic typecheck was interrupted immediately. Biome and the repository rule and tool-command guards passed. The single commit used a command-local hooks path retaining commit-message validation while excluding the pre-commit typecheck.

Preserve source identity and storm endings when batching (eve-bot-lovinka)

  • Context: Both perf tails used the same storm collapse key, merging different apps and ongoing/over phases. The reducer retained stale text and invented a maximum duration for events without ms.
  • Commit(s): b1495508e
  • Verdict: accepted
  • Code before:
    key: /^spawn\.storm (ongoing|over)/.test(body) ? "storm" : undefined,
  • Code after:
    key: /^spawn\.storm ongoing\b/.test(body) ? `${hangsDir}:storm:ongoing` : undefined,
  • How was this fixed: Only ongoing events collapse, under a key scoped to their source directory. Terminal events remain independent. Collapsed storm updates retain the latest text/time; a maximum is shown only when a duration was measured. Added an integration regression in src/hub/lib/dev-monitor.test.ts that drives both perf tails, repeated ongoing updates and multiple terminal events through runDevMonitor and EventBatcher. An optional sources override isolates all watched paths in fixture directories. Verification: not run (the lead verifies at integration).
  • Confidence: 99%

Deduplicate hang announcements by full path (eve-bot-lovinka)

  • Context: A shared set of basenames suppressed an unannounced hang from the second app when the first app logged a sample with the same timestamp filename.
  • Commit(s): b1495508e
  • Verdict: accepted
  • Code before:
    announced.add(basename(event.file));
    if (!announced.has(basename(path))) {
  • Code after:
    announced.add(event.file);
    if (!announced.has(path)) {
  • How was this fixed: The announcement set stores and compares full watched paths. Added an integration regression in src/hub/lib/dev-monitor.test.ts that logs one app's sample, creates same-named samples in both directories, then asserts the logged sample is suppressed while the other app's unannounced sample is emitted. Verification: not run (the lead verifies at integration).
  • Confidence: 99%

Exclude bare HEAD from storm labels (eve-bot-lovinka)

  • Context: The revision predicate accepted HEAD with a suffix but retained bare HEAD, splitting equivalent cat-file operations into separate command counts.
  • Commit(s): b1495508e
  • Verdict: accepted
  • Code before:
    return base.count >= 7 && base.allSatisfy(\.isHexDigit) || base == "HEAD" && base.count < argument.count
  • Code after:
    return base.count >= 7 && base.allSatisfy(\.isHexDigit) || base == "HEAD"
  • How was this fixed: Bare HEAD now satisfies the same revision predicate as HEAD~2 and HEAD^{commit}. Added SpawnStormTests.testLabelsGroupHeadAndCommitRevisionsTogether to the existing src/macos/GenesisKit/Tests/GenesisKitTests/Diff/AgentBlameModelTests.swift file. It checks HEAD, suffixed HEAD and SHA forms, plus the HEADquarters negative control. Verification: not run (the lead verifies at integration).
  • Confidence: 98%

Group bare HEAD with other cat-file revisions (eve-bot-lovinka)

  • Context: This thread independently reports the same bare-HEAD predicate defect as t3. A HEAD lookup and an equivalent SHA lookup produced different frequency labels.
  • Commit(s): b1495508e
  • Verdict: accepted
  • Code before:
    return base.count >= 7 && base.allSatisfy(\.isHexDigit) || base == "HEAD" && base.count < argument.count
  • Code after:
    return base.count >= 7 && base.allSatisfy(\.isHexDigit) || base == "HEAD"
  • How was this fixed: The shared revision predicate now accepts bare HEAD, resolving both t3 and t4 at the source. SpawnStormTests.testLabelsGroupHeadAndCommitRevisionsTogether checks equivalent HEAD and SHA forms and preserves an ordinary name beginning with HEAD. No new test file was needed. Verification: not run (the lead verifies at integration).
  • Confidence: 98%

Threads: t1 accepted · t2 accepted · t3 accepted · t4 accepted

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — ✅ Approved

b149550 · 0 actionable findings · reviewed by codex/gpt-6.1-sol · view run ↗

Blast radius

  • runDevMonitor (~) → 2 dependent(s) outside this PR: src/hub/commands/dev.ts

genesiscz added a commit that referenced this pull request Oct 26, 2026
@genesiscz
genesiscz force-pushed the feat/2026-10-25-spawn-storm branch from b149550 to 0a23d0f Compare October 26, 2026 02:13
@genesiscz
genesiscz force-pushed the feat/2026-10-24-widget-hub-prereqs branch from 7fb1542 to 9d74bca Compare October 26, 2026 02:25

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — 🟡 Review comments

0a23d0f · 2 actionable findings · reviewed by find codex/gpt-6.1-sol + codex/gpt-6-luna, verify codex/gpt-6-luna · view run ↗

Severity Count
🟡 Medium 2

ℹ️ Checks (advisory — CI is authoritative) — typecheck timed out (tsconfig.json): out of memory, then its 6144 MiB retry ran out of time

Blast radius

  • label (+) → 5 dependent(s) outside this PR: src/browser-router/native/Router.swift, src/macos/GenesisTools/Tests/ReviewKeysTests.swift, src/macos/GenesisTools/Tests/DaemonSettingsTests.swift, +2 more
  • started (+) → 2 dependent(s) outside this PR: src/macos/GenesisTools/Package.swift, src/macos/GenesisTools/Tests/HubNavigationTests.swift
  • registerDevCommands (~) → 1 dependent(s) outside this PR: src/hub/cli.ts
  • graph truncated by caps — dependents above are a lower bound

Comment thread src/macos/GenesisKit/Sources/GenesisKitBase/Perf/SpawnMeter.swift Outdated
Comment thread src/macos/GenesisKit/Sources/GenesisKitBase/Tools/ProcessCapture.swift Outdated
genesiscz added a commit that referenced this pull request Oct 26, 2026
@genesiscz
genesiscz force-pushed the feat/2026-10-25-spawn-storm branch from 0a23d0f to b298491 Compare October 26, 2026 02:27
@genesiscz
genesiscz changed the base branch from feat/2026-10-24-widget-hub-prereqs to master October 26, 2026 02:29

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — 🟡 Review comments

b298491 · 1 actionable finding · reviewed by find codex/gpt-6.1-sol + codex/gpt-6-luna, verify codex/gpt-6-luna · view run ↗

Severity Count
🟡 Medium 1

Blast radius

  • started (+) → 6 dependent(s) outside this PR: native/ax-tool/Tests/RecorderExecutableTests.swift, native/ax-tool/Tests/ContextMenuTests.swift, native/ax-tool/Tests/SnapshotDispatchTests.swift, +3 more
  • label (+) → 4 dependent(s) outside this PR: native/ax-tool/Tests/ObservedTreeBuilderTests.swift, native/ax-tool/Tests/SnapshotSelectionTests.swift, native/ax-tool/Tests/IdentifierTargetTests.swift, +1 more
  • registerDevCommands (~) → 1 dependent(s) outside this PR: src/hub/cli.ts
  • graph truncated by caps — dependents above are a lower bound

ℹ️ Claims vs diff — 6 claims could not be checked from the diff (c1, c2, c3, c4, c5, c6).

Comment thread src/hub/lib/dev-monitor.ts Outdated
genesiscz added a commit that referenced this pull request Oct 26, 2026
…ts its end without more starts, a captured child that ends early or fails to launch keeps the meter balanced, and helper perf lines stay apart from their parent's
@genesiscz

Copy link
Copy Markdown
Owner Author

Review fixes 2026-10-26 03:49 from GPT-6.1 Sol judged by Claude Opus 5.5 (fable-judge)

Round 2 at b298491. 3 unresolved threads (eve 3, CodeRabbit 0). Earlier round-1 findings were already addressed and need no further reply. Added regressions in existing test files; no new test file or dependency. No checks, formatting commands, staging or commits were run. Fixed in c04a9a8; the lead ran the tests and committed. Fixed in c04a9a8; the lead ran the tests and committed.

Report sustained concurrency storms on process finishes (eve-bot-lovinka)

  • Context: The ongoing report deadline was checked only by start(). Long-running children could finish after the report interval while the storm remained active without producing an update.
  • Commit(s): c04a9a81c
  • Verdict: accepted
  • Code before:
    public mutating func finish(at now: Double) -> [String] {
        running = max(0, running - 1)
        prune(now)
        return endIfQuiet().map { [$0] } ?? []
    }
  • Code after:
    public mutating func finish(at now: Double) -> [String] {
        running = max(0, running - 1)
        prune(now)
        if let ended = endIfQuiet() { return [ended] }
        return reportIfDue(at: now).map { [$0] } ?? []
    }
  • How was this fixed: Both start() and finish() use reportIfDue(at:) and the same lastReport deadline. Quiet detection runs first, so an ending emits only an over line. The documentation now explicitly describes event-driven reporting. Added SpawnStormTests.testFinishesReportAnOngoingConcurrencyStormWithoutMoreStarts in the existing AgentBlameModelTests.swift file, covering finish-only updates, a shared start/finish deadline, quiet precedence and no reports after termination. Verification:.
  • Confidence: 96%

Stop counting captured children when they exit (eve-bot-lovinka)

  • Context: The deferred meter completion tracked the lifetime of runCapturing(), including afterStart and pipe draining after child exit, inflating concurrent-running counts and potentially reporting a false storm.
  • Commit(s): c04a9a81c
  • Verdict: accepted
  • Code before:
    terminationHandler = { _ in exited.signal() }
    try run()
    let ended = SpawnMeter.started(executable: executableURL, arguments: arguments ?? [])
    defer { ended() }
  • Code after:
    let ended = meterStarted(executableURL, arguments ?? [])
    terminationHandler = { _ in
        ended()
        exited.signal()
    }
    do { try run() } catch {
        ended(); throw error
    }
  • How was this fixed: Meter registration occurs before launch; successful completion runs in the termination handler before signalling the exit semaphore. A launch failure balances registration in the catch path. The public API is unchanged; an internal overload accepts the meter callback for direct lifecycle assertions. Extended the existing ToolsBridgeTests.swift file outside the original PR diff because it already imports GenesisKitBase with @testable. Added testCapturedChildEndsBeforeAfterStartReturns and testCapturedChildLaunchFailureBalancesMeterRegistration, covering exit while afterStart remains active, registration before launch, failure balancing and no duplicate completion. Verification:.
  • Confidence: 95%

Retain helper process tags in displayed and batched perf events (eve-bot-lovinka)

  • Context: Classification stripped the helper tag and discarded it. Parent and helper events from the shared Genesis perf log therefore displayed identically, and their ongoing storm events could collapse together.
  • Commit(s): c04a9a81c
  • Verdict: accepted
  • Code before:
    const body = line
        .replace(timePattern, "")
        .replace(/^\[[a-z-]+\]\s+/, "")
        .replace(/^mark\s+/, "");
    const short = (text: string) => text.replace(/ recent=\[.*$/, "").slice(0, 300);
  • Code after:
    const message = line.replace(timePattern, "");
    const tag = message.match(/^(\[[a-z-]+\])\s+/)?.[1] ?? "";
    const body = message.replace(/^\[[a-z-]+\]\s+/, "").replace(/^mark\s+/, "");
    const short = (text: string) => `${tag ? `${tag} ` : ""}${text.replace(/ recent=\[.*$/, "")}`.slice(0, 300);
  • How was this fixed: Extracted the tag before normalizing the body and preserved it in every classified perf event's text. Ongoing storm keys include both the log directory and process tag, preventing parent/helper collapse within the shared log. Extended dev-monitor.test.ts to compare otherwise identical tagged and untagged storm, spin and stall events, assert the hang event retains its tag, and exercise parent/helper storm attribution through both tails and EventBatcher. Verification:.
  • Confidence: 97%

Threads: t1 accepted · t2 accepted · t3 accepted

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — ✅ Approved

c04a9a8 · 0 actionable findings · reviewed by codex/gpt-6.1-sol + codex/gpt-6-luna · view run ↗

Blast radius

  • runCapturing (+) → 7 dependent(s) outside this PR: src/macos/GenesisTools/Sources/Parts/Hub/Review/PRRefs.swift, src/macos/GenesisTools/Sources/Parts/Hub/Hub/HubWorktrees.swift, src/macos/GenesisTools/Sources/Host/App/ChildEnvironment.swift, +4 more
  • graph truncated by caps — dependents above are a lower bound

…ed as spawn.storm (24 starts in 10 s or 12 running, with the commonest commands) by GenesisKit's launchers, and tools hub dev monitor reports it and reads Genesis Markdown's perf log and hangs too
…, so the hub opening a pull request (24 starts, 5 running) is not a storm, and git's -c option stays out of the label
…ons, so git cat-file <sha> counts as git cat-file
… finds it quiet and is timed to its last start, so a 10 s burst no longer reads "over after 209 s"
…ts its end without more starts, a captured child that ends early or fails to launch keeps the meter balanced, and helper perf lines stay apart from their parent's
@genesiscz
genesiscz force-pushed the feat/2026-10-25-spawn-storm branch from c04a9a8 to 421c572 Compare October 26, 2026 03:19
@genesiscz
genesiscz merged commit 421c572 into master Oct 26, 2026

@eve-bot-lovinka eve-bot-lovinka Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐉 eve review — 🟡 Review comments

421c572 · 1 actionable finding · reviewed by find codex/gpt-6.1-sol + codex/gpt-6-luna, verify codex/gpt-6-luna · view run ↗

Severity Count
🟡 Medium 1

Blast radius

  • runCapturing (+) → 7 dependent(s) outside this PR: src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/DiskSystemProbes.swift, src/macos/GenesisKit/Sources/GenesisKit/Cmux/CmuxCommandRunner.swift, src/macos/GenesisKit/Sources/GenesisKit/Diff/Provider/GenesisKitGTProvider.swift, +4 more
  • started (+) → 7 dependent(s) outside this PR: src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/ToolsDiskProvider.swift, src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/VolumeSpace.swift, src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/DiskSystemProbes.swift, +4 more
  • label (+) → 6 dependent(s) outside this PR: src/macos/GenesisKit/Sources/GenesisKitOverlay/ScreenEffectView.swift, src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/ToolsDiskActions.swift, src/macos/GenesisKit/Sources/GenesisKitDisk/Disk/DiskHistoryStore.swift, +3 more
  • registerDevCommands (~) → 1 dependent(s) outside this PR: src/hub/cli.ts
  • graph truncated by caps — dependents above are a lower bound

lastStart = now
lastReport = now
total = starts.count
counts = starts.reduce(into: [:]) { $0[$1.label, default: 0] += 1 }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Quality | 🟡 Medium · confidence 90/100

⚠️ Potential issue

Include older running children in concurrency-storm command counts

A storm triggered by the running-child threshold initializes its command counts only from the last 10 seconds of starts, even though running includes older children. For example, start 15 long-lived GenesisTools agents blame children at t=0, then one git status at t=20: the resulting storm reports 16 running but identifies only git status ×1, omitting all 15 processes responsible for the concurrency. Subsequent reports never recover those labels. Track active children by label (and pass the label or an identity into finish), so concurrency-triggered storms can identify the commonest running command rather than only recent arrivals. Add a test with children whose lifetimes exceed the rolling window; the current concurrency test checks only the running count.

🧩 Analysis

Grep evidence: starts\.reduce|starts\.removeAll|running >= Self\.burstRunning

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted. The follow-up will include older running children in concurrency-storm command counts. The pull request was merged after its fix rounds, so this finding waits for a follow-up, and this thread stays open until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant