Skip to content

fix: Flag evaluations agent normal exit leak - #234

Merged
dustinbyrne merged 4 commits into
PostHog:mainfrom
matthewlehner:fix/evaluations-agent-normal-exit-leak
Oct 29, 2026
Merged

dustinbyrne merged 4 commits into
PostHog:mainfrom
matthewlehner:fix/evaluations-agent-normal-exit-leak

Conversation

@matthewlehner

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Right now these Agent are only shut down if the linked process crashes. If it exits normally, they continue living on forever.

This traps exits from the linked process and shuts down the agent when the process shuts down.

💚 How did you test it?

There are tests here!

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted) — or — Fully autonomous

  • Tools: pi coding agent with Claude (claude-opus-5-5). The agent investigated the leak, wrote the fix, tests, and changeset; I directed the approach and reviewed the code. Session not linked
    (local only).
  • A small script confirmed the leak: a linked Agent survives a :normal caller exit but dies on :shutdown or a crash.
  • Considered and rejected for now: replacing the Agent with an :atomics-backed tracker. It removes the per-snapshot process entirely and lets snapshots outlive their creator, but it changes
    the struct and makes accessed/1 drop keys that were read but absent from the snapshot. We went with the minimal fix instead; happy to follow up with the atomics approach if maintainers prefer.
  • Checked the sdk-specs evaluate-flags spec; it doesn't cover how access tracking is stored or how long it lives, so there's no divergence to call out.

Each Evaluations snapshot starts an Agent linked to the calling process
to track accessed keys. A linked process that isn't trapping exits
ignores a :normal exit signal, so whenever the caller exited normally
(a finished request, Task or job) the Agent was orphaned and lived
for the life of the node.

Trap exits in the Agent so it stops when its parent exits for any
reason. Document that a snapshot is only usable while the process that
created it is alive.
@matthewlehner
matthewlehner requested a review from a team as a code owner October 29, 2026 19:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T20:01:03.119734Z 7300476 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7300476475

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@dustinbyrne dustinbyrne changed the title Fix evaluations agent normal exit leak fix: Flag evaluations agent normal exit leak Oct 29, 2026
@dustinbyrne
dustinbyrne merged commit 8d9eb85 into PostHog:main Oct 29, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants