Skip to content

fix(flags): track snapshot access with atomics instead of a process - #235

Merged
marandaneto merged 4 commits into
mainfrom
fix/evaluations-access-tracker-leak
Oct 30, 2026
Merged

marandaneto merged 4 commits into
mainfrom
fix/evaluations-access-tracker-leak

Conversation

@marandaneto

@marandaneto marandaneto commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

💡 Motivation and Context

Follow-up to #234. That PR stops the snapshot's access-tracking Agent when the calling process exits. Two leaks remained:

  • Long-lived callers build up one process per evaluation. Every evaluate_flags/2, Evaluations.only/2, and Evaluations.only_accessed/1 call starts an Agent linked to the caller. The empty snapshot returned when no distinct_id is available starts one too. The Agent only stops when the caller exits, not when the snapshot goes out of scope. In a GenServer, LiveView, channel, or queue consumer, these Agents pile up for the life of that process. In a local repro, 10,000 evaluate_flags/2 calls from one process left 10,000 extra processes (about 32 MB) alive.
  • The OpenFeature provider fills the mailbox of callers that trap exits. The provider calls Agent.stop/1 on every evaluation. Because the Agent is linked, a caller with trap_exit enabled gets an {:EXIT, pid, :normal} message each time. In the repro, 1,000 evaluations left 1,000 unread messages. A GenServer without a matching handle_info logs an "unexpected message" error on every evaluation instead.

Snapshots now track access with :atomics, one counter per flag key in the snapshot. The atomics array is freed when the snapshot is garbage-collected, so no process is started and nothing needs to be stopped. The OpenFeature cleanup is removed because nothing is left to clean up. As a side effect, a snapshot can now be used from any process, including after the process that created it has exited. Before, that raised an exit error.

Behaviour changes

This ships as a patch because it fixes a leak. The changeset describes two visible changes:

  • The documented accessed_pid struct field is kept for compatibility but is deprecated and always nil. It was internal tracking state. Code that reads or pattern-matches it still compiles. Code that passes it to Agent.stop/1, as the OpenFeature provider used to, must remove that call, because there is no process to stop anymore. Tracking state now lives in a new opaque accessed field.
  • Evaluations.accessed/1 now only returns accessed keys that are present in the snapshot. Keys read but missing from the snapshot are no longer listed, because the counters are created from the snapshot's own flags. only_accessed/1 already ignored those keys, so its result is unchanged. Reading a missing key still sends $feature_flag_called with the flag_missing error.

GitHub code search found no public Elixir code using accessed_pid, Evaluations.accessed, or only_accessed outside this repo.

The sdk-specs evaluate-flags spec defines the accessed-key filter only in terms of flags present in the snapshot, so this matches the spec. The spec does not cover how access is stored or how long it lives.

💚 How did you test it?

  • New tests check that snapshots link no processes to the caller, that a snapshot still works after its creating process exits, that an OpenFeature caller trapping exits receives no exit messages, and that accessed/1 leaves out missing keys. Each of these failed on main before the change.
  • Another test checks that accessed_pid is still present and nil on every kind of snapshot. Without the field, it fails to compile.
  • Removed tests that only covered the old Agent lifetime.
  • mix test passes except 7 PostHog.Integrations.PlugTest tests. They also fail on main on my machine because port 8001 is already in use.
  • mix format --check-formatted, mix credo --strict, and mix posthog.public_api --check pass.
  • A local repro script went from 10,000 leftover processes and 1,000 queued exit messages on main to 0 and 0 on this branch.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Tools: pi coding agent with Claude (claude-opus-5-5). The agent audited the SDK for leaks after fix: Flag evaluations agent normal exit leak #234, reproduced both leaks with failing tests, wrote the fix, and ran a second-model autoreview with no findings. I directed the approach and reviewed the code.
  • We first had a separate two-line OpenFeature fix that unlinks the Agent before stopping it. The atomics change removes that code, so both fixes are combined in this PR instead of stacking them.
  • Considered keeping missing keys in accessed/1. That needs a shared mutable store such as ETS or a process, which brings back the cleanup problem, so we accepted the change.
  • We chose a patch release because this is a leak fix. To keep it compatible, accessed_pid stays as a deprecated nil field instead of being removed. The behaviour changes are described in the changeset.

@marandaneto marandaneto self-assigned this Oct 30, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 3327ee3: 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

posthog-elixir Compliance Report

Date: 2026-10-30T08:17:39.448834+00:00
Duration: 108715ms

⚠️ Some Tests Failed

40/47 tests passed, 7 failed


Capture Tests

⚠️ 29/30 tests passed, 1 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 611ms
Format Validation.Event Has Uuid ✅ 611ms
Format Validation.Event Has Lib Properties ✅ 611ms
Format Validation.Distinct Id Is String ✅ 610ms
Format Validation.Token Is Present ✅ 612ms
Format Validation.Custom Properties Preserved ✅ 612ms
Format Validation.Event Has Timestamp ✅ 612ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ❌ 612ms
Retry Behavior.Retries On 503 ✅ 5618ms
Retry Behavior.Does Not Retry On 400 ✅ 2614ms
Retry Behavior.Does Not Retry On 401 ✅ 2613ms
Retry Behavior.Respects Retry After Header ✅ 5615ms
Retry Behavior.Implements Backoff ✅ 15616ms
Retry Behavior.Retries On 500 ✅ 5617ms
Retry Behavior.Retries On 502 ✅ 5616ms
Retry Behavior.Retries On 504 ✅ 5617ms
Retry Behavior.Max Retries Respected ✅ 15626ms
Deduplication.Generates Unique Uuids ✅ 623ms
Deduplication.Preserves Uuid On Retry ✅ 5617ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10618ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5616ms
Deduplication.No Duplicate Events In Batch ✅ 617ms
Deduplication.Different Events Have Different Uuids ✅ 615ms
Compression.Sends Gzip When Enabled ✅ 611ms
Batch Format.Uses Proper Batch Structure ✅ 611ms
Batch Format.Flush With No Events Sends Nothing ✅ 608ms
Batch Format.Multiple Events Batched Together ✅ 615ms
Error Handling.Does Not Retry On 403 ✅ 2613ms
Error Handling.Does Not Retry On 413 ✅ 2613ms
Error Handling.Retries On 408 ✅ 5616ms

Failures

format_validation.non_utc_event_timestamp_is_converted_to_utc

Event 0 field 'timestamp' instant '2026-10-30T08:15:55.353576Z' != expected '2025-01-02T03:04:05Z'

Feature_Flags Tests

⚠️ 11/17 tests passed, 6 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 9ms
Request Payload.Flags Request Uses V2 Query Param ✅ 9ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 8ms
Request Payload.Flags Request Omits Authorization Header ✅ 8ms
Request Payload.Token In Flags Body Matches Init ✅ 8ms
Request Payload.Groups Round Trip ✅ 7ms
Request Payload.Groups Default To Empty Object ❌ 8ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 8ms
Request Payload.Disable Geoip Omitted Defaults To False ❌ 7ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 8ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 609ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 11ms
Request Lifecycle.Mock Response Value Is Returned To Caller ❌ 7ms
Retry Behavior.Retries Flags On 502 ❌ 311ms
Retry Behavior.Retries Flags On 504 ❌ 311ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ❌ 610ms

Failures

request_payload.groups_default_to_empty_object

Field 'groups' not found in /flags request body at path 'groups'. Available keys: ['api_key', 'distinct_id', 'flag_keys_to_evaluate']

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['api_key', 'distinct_id', 'flag_keys_to_evaluate']

request_lifecycle.mock_response_value_is_returned_to_caller

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_502

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_504

Last action result missing field 'value'. Keys: ['error', 'success']

side_effect_events.get_feature_flag_captures_feature_flag_called_event

Expected 1 events with name '$feature_flag_called', got 0

@marandaneto
marandaneto marked this pull request as ready for review October 30, 2026 08:11
@marandaneto
marandaneto requested a review from a team as a code owner October 30, 2026 08:12

@ioannisj ioannisj left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 An AI agent wrote this on behalf of @ioannisj.

Nothing blocking from me. I went through the :atomics index mapping and the nil path for empty snapshots, the accessed_pid field staying nil, the accessed/1 change against the evaluate-flags spec and onlyAccessed in posthog-js, and the OpenFeature provider no longer stopping an agent. The one thing I'd like your take on is whether the changeset should be minor rather than patch, since two runtime behaviours change for existing callers.

@marandaneto
marandaneto merged commit 747c160 into main Oct 30, 2026
30 checks passed
@marandaneto
marandaneto deleted the fix/evaluations-access-tracker-leak branch October 30, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants