Skip to content

fix: drop the oldest AI events while too many wait to be written to disk - #901

Open
turnipdabeets wants to merge 3 commits into
feat/v4-capture-v1from
fix/v4-ai-pending-cap
Open

turnipdabeets wants to merge 3 commits into
feat/v4-capture-v1from
fix/v4-ai-pending-cap

Conversation

@turnipdabeets

@turnipdabeets turnipdabeets commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #881.

💡 Motivation and Context

One single-threaded executor both writes AI events to disk and sends them. During a slow upload (up to captureAiTimeout, 60 s, per request), new captureAi events wait in the executor's task queue, in memory, with no limit. The 100-event / 50 MiB caps only applied once an event was written, so a chat app capturing multi-MiB generations on a slow network could exhaust the heap. PostHog Review flagged this on #881 and #882.

AI events now wait in a bounded list until the executor writes them. Once more than captureAiMaxQueueSize events (100 by default) or 50 MiB wait, the oldest waiting events are dropped and the new one is kept, the same as the disk queue already does (and as the posthog-ios 4.0 AI queue and the sdk-specs capture-ai client note describe). These are the disk queue's limits, so holding more in memory couldn't deliver more events; it only delayed the eviction. captureAi still returns the new event's uuid.

  • PostHogQueue.addMeasured(record, bytes): adds the record to the waiting list, drops the oldest waiting records while over the count or bytes, and queues an executor task that writes the oldest waiting record, only while there are fewer tasks than waiting records, so dropped events leave no tasks behind. Only the AI lane uses it; analytics, replay and logs keep add.
  • captureAi passes the size it already measures for the per-event limit, so there's no second serialization.
  • A drop is logged with the count only, never event content.
  • The changeset bullet, the migration guide and captureAiMaxQueueSize describe the drop.

How the other SDKs bound waiting AI events

SDK Waiting AI events Limit When full
posthog-ios 4.0 None in memory: captureAi writes to disk on the caller's thread Disk: 100 events, 50 MiB Evicts the oldest on disk
posthog-python v1 In-memory queue 1000 events Drops the new event
posthog-go v1 In-memory channel 1000 events Drops the new event, ErrQueueFull
posthog-rs v1 In-memory queue 1000 events Drops the new event, one warning
posthog-android (this PR) Waiting list before the disk write 100 events, 50 MiB Drops the oldest waiting events

This follows posthog-ios and the disk queue (drop the oldest) with mobile limits: a count-only cap of 100 would still allow ~800 MiB of 8 MiB events. A synchronous write like iOS would put disk I/O on Android's main thread. posthog-flutter needs no change: mobile calls the native SDKs, and the desktop Dart lane enforces its caps when it queues.

Splitting writing from sending into separate threads would remove the waiting entirely, but it changes the queue shared by analytics, replay and logs, so it's left out of this PR.

💚 How did you test it?

  • PostHogQueueByteLimitsTest: with the executor blocked, going over the waiting bytes and then the waiting count each drops the oldest waiting record; the queue ends with the newest three.
  • PostHogCaptureAiTest: with the AI executor blocked and captureAiMaxQueueSize = 2, the third captureAi still returns a uuid and the oldest waiting event is dropped.
  • With the eviction removed, both tests fail.
  • ./gradlew apiCheck spotlessCheck :posthog:test :posthog-android:testReleaseUnitTest :posthog-server:test pass locally.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file (updated the unreleased v4 major changeset's captureAi bullet)

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code compared how posthog-ios, posthog-python, posthog-go and posthog-rs bound AI events waiting to be sent, from their v1/4.0 branch sources, and implemented the cap as a follow-up to #881; after review against sdk-specs #147, the maintainer switched it from drop-the-newest to drop-the-oldest. Rejected: a count-only cap (too much memory on a device), a synchronous write (main-thread I/O), separate write and send threads (touches every queue; possible later).

🤖 Generated with Claude Code

One executor both writes and sends AI events, so during a slow upload new
multi-MiB events waited in memory with no limit. captureAi now returns
null while captureAiMaxQueueSize events or 50 MiB are waiting, the same
limits the disk queue keeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@turnipdabeets turnipdabeets self-assigned this Oct 28, 2026
@turnipdabeets
turnipdabeets requested a review from a team as a code owner October 28, 2026 22:39
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

posthog-android Compliance Report

Date: 2026-10-29T01:30:50.719311+00:00
Duration: 450287ms

⚠️ Some Tests Failed

212/219 tests passed, 7 failed


Capture_V1 Tests

⚠️ 98/101 tests passed, 3 failed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 435ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 37ms
Required Headers.Has Authorization Bearer Header ✅ 32ms
Required Headers.Has Content Type Json ✅ 31ms
Required Headers.Has Posthog Sdk Info Format ✅ 32ms
Required Headers.Has Posthog Attempt Header ✅ 26ms
Required Headers.Has Posthog Request Id ✅ 27ms
Required Headers.Has Posthog Request Timestamp ✅ 30ms
Required Headers.Has User Agent ✅ 26ms
Body Format.Body Has Created At And Batch ✅ 28ms
Body Format.No Api Key In Body ✅ 26ms
Body Format.No Sent At In Body ✅ 23ms
Event Format.Event Has Required Root Fields ✅ 25ms
Event Format.Event Uuid Is Valid ✅ 26ms
Event Format.Event Timestamp Is Rfc3339 ✅ 24ms
Event Format.Non Utc Event Timestamp Is Converted To Utc ✅ 26ms
Event Format.Distinct Id Is String ✅ 21ms
Event Format.Distinct Id At Root Not Properties ✅ 24ms
Event Format.Custom Properties Preserved ✅ 24ms
Event Format.Set Properties Preserved ✅ 25ms
Event Format.Set Once Properties Preserved ✅ 26ms
Event Format.Groups Properties Preserved ✅ 25ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 26ms
Event Format.Event Has Required Root Fields Batch ✅ 32ms
Event Format.Event Uuid Is Valid Batch ✅ 32ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 27ms
Event Format.Distinct Id Is String Batch ✅ 28ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 27ms
Event Format.Custom Properties Preserved Batch ✅ 30ms
Event Format.Set Properties Preserved Batch ✅ 30ms
Event Format.Set Once Properties Preserved Batch ✅ 43ms
Event Format.Groups Properties Preserved Batch ✅ 30ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 27ms
Batch Behavior.Multiple Events In Single Batch ✅ 37ms
Batch Behavior.Batch Envelope Smoke ✅ 33ms
Batch Behavior.Flush With No Events Sends Nothing ✅ 19ms
Batch Behavior.Flush At Triggers Batch ✅ 1029ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 21ms
Deduplication.Generates Unique Uuids ✅ 42ms
Deduplication.Different Events Same Content Different Uuids ✅ 23ms
Deduplication.Preserves Uuid On Retry ✅ 7016ms
Deduplication.Preserves Timestamp On Retry ✅ 7018ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 7023ms
Deduplication.No Duplicate Events In Batch ✅ 34ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 17ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 12024ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 7017ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 2029ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 7017ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 19ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 24ms
Response Format Validation.Success No Retry After When All Ok ✅ 20ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 2010ms
Response Format Validation.Success No Retry After When Drop Only ✅ 20ms
Response Format Validation.Response Echoes Request Id ✅ 18ms
Retry Behavior.Retries On 408 ✅ 7017ms
Retry Behavior.Retries On 500 ✅ 7016ms
Retry Behavior.Retries On 503 ✅ 7024ms
Retry Behavior.Retries On 504 ✅ 7017ms
Retry Behavior.Retryable Errors Have Retry After ✅ 4013ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 10024ms
Retry Behavior.Does Not Retry On 400 ✅ 4021ms
Retry Behavior.Does Not Retry On 401 ✅ 4020ms
Retry Behavior.Does Not Retry On 402 ✅ 4020ms
Retry Behavior.Does Not Retry On 413 ✅ 4018ms
Retry Behavior.Does Not Retry On 415 ✅ 4020ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 4018ms
Retry Behavior.Implements Backoff ✅ 17022ms
Retry Behavior.Max Retries Respected ❌ 17037ms
Partial Batch Handling.Handles 200 Full Success ✅ 2017ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3029ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3027ms
Partial Batch Handling.Does Not Retry Limited Events ✅ 3023ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 7013ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 7017ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 7017ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 7015ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 7014ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 7016ms
Partial Batch Handling.Respects Retry After On Partial ✅ 7024ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3022ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry ✅ 3033ms
Compression.Sends Gzip Content Encoding ✅ 15ms
Compression.No Content Encoding When Disabled ✅ 14ms
Compression.Compressed Body Is Decompressible ✅ 15ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 4017ms
Event Options.Cookieless Mode Override ✅ 17ms
Event Options.Disable Skew Correction Override ✅ 13ms
Event Options.Process Person Profile Override ✅ 16ms
Event Options.Product Tour Id Override ✅ 17ms
Event Options.Unset Options Omitted ✅ 15ms
Event Options.Options Override In Batch ✅ 23ms
Event Options.Unknown Option Passes Through ✅ 17ms
Event Options.Option Value Is Not Converted ✅ 16ms
Event Options.Lenient Bool Option Values Are Not Converted ✅ 18ms
Event Options.Option Wins Over Legacy Property ✅ 28ms
Event Options.Legacy Property Fills Unset Option ✅ 23ms
Event Options.Null Option Falls Back To Legacy Property ✅ 22ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties ❌ 23ms
Geoip And Historical Migration.Historical Migration Set In Body ❌ 23ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 21ms

Failures

retry_behavior.max_retries_respected

Expected 4 requests, got 5

geoip_and_historical_migration.geoip_disable_injected_into_properties

Expected $geoip_disable='True', got 'None'. Available properties: ['$is_identified', '$recording_status', '$sdk_debug_pending_queue_size']

geoip_and_historical_migration.historical_migration_set_in_body

Body missing 'historical_migration' field. Body keys: ['created_at', 'batch']

Capture_Ai_V1 Tests

⚠️ 97/101 tests passed, 4 failed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 22ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 21ms
Endpoint And Method.Capture Does Not Reroute Ai Named Events ✅ 20ms
Required Headers.Has Authorization Bearer Header ✅ 18ms
Required Headers.Has Content Type Json ✅ 19ms
Required Headers.Has Posthog Sdk Info Format ✅ 24ms
Required Headers.Has Posthog Attempt Header ✅ 22ms
Required Headers.Has Posthog Request Id ✅ 24ms
Required Headers.Has Posthog Request Timestamp ✅ 20ms
Required Headers.Has User Agent ✅ 23ms
Body Format.Body Has Created At And Batch ✅ 31ms
Body Format.No Api Key In Body ✅ 23ms
Body Format.No Sent At In Body ✅ 17ms
Event Format.Event Has Required Root Fields ✅ 14ms
Event Format.Event Uuid Is Valid ✅ 15ms
Event Format.Event Timestamp Is Rfc3339 ✅ 13ms
Event Format.Non Utc Event Timestamp Is Converted To Utc ✅ 14ms
Event Format.Distinct Id Is String ✅ 15ms
Event Format.Distinct Id At Root Not Properties ✅ 13ms
Event Format.Custom Properties Preserved ✅ 18ms
Event Format.Set Properties Preserved ✅ 18ms
Event Format.Set Once Properties Preserved ✅ 20ms
Event Format.Groups Properties Preserved ✅ 20ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 19ms
Event Format.Capture Ai Keeps Supplied Uuid ❌ 12ms
Event Format.Event Has Required Root Fields Batch ❌ 27ms
Event Format.Event Uuid Is Valid Batch ✅ 24ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 25ms
Event Format.Distinct Id Is String Batch ✅ 21ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 24ms
Event Format.Custom Properties Preserved Batch ✅ 22ms
Event Format.Set Properties Preserved Batch ✅ 30ms
Event Format.Set Once Properties Preserved Batch ✅ 26ms
Event Format.Groups Properties Preserved Batch ✅ 27ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 28ms
Batch Behavior.Multiple Events In Single Batch ✅ 32ms
Batch Behavior.Batch Envelope Smoke ✅ 28ms
Batch Behavior.Flush At Triggers Batch ✅ 1016ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 15ms
Deduplication.Generates Unique Uuids ✅ 32ms
Deduplication.Different Events Same Content Different Uuids ✅ 21ms
Deduplication.Preserves Uuid On Retry ✅ 7016ms
Deduplication.Preserves Timestamp On Retry ✅ 7015ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 7016ms
Deduplication.No Duplicate Events In Batch ✅ 23ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 13ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 12020ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 7013ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 2024ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 7016ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 16ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 22ms
Response Format Validation.Success No Retry After When All Ok ✅ 17ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 2008ms
Response Format Validation.Success No Retry After When Drop Only ✅ 18ms
Response Format Validation.Response Echoes Request Id ✅ 13ms
Retry Behavior.Retries On 408 ✅ 7013ms
Retry Behavior.Retries On 500 ✅ 7015ms
Retry Behavior.Retries On 503 ✅ 7022ms
Retry Behavior.Retries On 504 ✅ 7011ms
Retry Behavior.Retryable Errors Have Retry After ✅ 4013ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 10023ms
Retry Behavior.Does Not Retry On 400 ✅ 4018ms
Retry Behavior.Does Not Retry On 401 ✅ 4021ms
Retry Behavior.Does Not Retry On 402 ✅ 4016ms
Retry Behavior.Does Not Retry On 413 ✅ 4018ms
Retry Behavior.Does Not Retry On 415 ✅ 4015ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 4017ms
Retry Behavior.Implements Backoff ✅ 17027ms
Retry Behavior.Max Retries Respected ❌ 17030ms
Partial Batch Handling.Handles 200 Full Success ✅ 2019ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3023ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3023ms
Partial Batch Handling.Does Not Retry Limited Events ✅ 3021ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 7015ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 7017ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 7014ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 7015ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 7014ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 7015ms
Partial Batch Handling.Respects Retry After On Partial ✅ 7021ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3021ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry ✅ 3026ms
Compression.Sends Gzip Content Encoding ✅ 14ms
Compression.No Content Encoding When Disabled ✅ 12ms
Compression.Compressed Body Is Decompressible ✅ 15ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 4016ms
Event Options.Cookieless Mode Override ✅ 15ms
Event Options.Disable Skew Correction Override ✅ 15ms
Event Options.Process Person Profile Override ✅ 15ms
Event Options.Product Tour Id Override ✅ 16ms
Event Options.Unset Options Omitted ✅ 20ms
Event Options.Options Override In Batch ✅ 23ms
Event Options.Unknown Option Passes Through ✅ 14ms
Event Options.Option Value Is Not Converted ✅ 15ms
Event Options.Lenient Bool Option Values Are Not Converted ✅ 13ms
Event Options.Option Wins Over Legacy Property ✅ 19ms
Event Options.Legacy Property Fills Unset Option ✅ 16ms
Event Options.Null Option Falls Back To Legacy Property ✅ 15ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties ❌ 16ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 16ms

Failures

event_format.capture_ai_keeps_supplied_uuid

Expected last action result '0198c0de-0000-7000-8000-000000000abc', got '01a18544-935f-799d-9c70-c6ea1a7a0e78'

event_format.event_has_required_root_fields_batch

Expected 1 requests, got 2

retry_behavior.max_retries_respected

Expected 4 requests, got 5

geoip_and_historical_migration.geoip_disable_injected_into_properties

Expected $geoip_disable='True', got 'None'. Available properties: ['$is_identified', '$recording_status', '$sdk_debug_session_start', '$sdk_debug_pending_queue_size']

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 35ms
Request Payload.Flags Request Uses V2 Query Param ✅ 22ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 25ms
Request Payload.Flags Request Omits Authorization Header ✅ 21ms
Request Payload.Token In Flags Body Matches Init ✅ 20ms
Request Payload.Groups Round Trip ✅ 23ms
Request Payload.Groups Default To Empty Object ✅ 20ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 20ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 17ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 17ms
Request Lifecycle.No Flags Request On Init Alone ✅ 6ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 16ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 29ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 16ms
Retry Behavior.Retries Flags On 502 ✅ 318ms
Retry Behavior.Retries Flags On 504 ✅ 318ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 20ms

While a slow upload holds the AI executor, captureAi now keeps the new
event and drops the oldest events still waiting to be written once
captureAiMaxQueueSize events or 50 MiB wait, the same as the disk queue
already does. captureAi returns null again only for disabled, opted-out,
beforeSend-dropped or oversized events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@turnipdabeets turnipdabeets changed the title fix: drop an AI event while too many wait to be written to disk fix: drop the oldest AI events while too many wait to be written to disk Oct 28, 2026
Comment thread posthog/src/main/java/com/posthog/internal/PostHogQueue.kt
@veria-ai

veria-ai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Dropping a waiting AI event no longer leaves its executor task behind
for nothing: a dropped event's task writes a newer one instead, so the
executor never holds more tasks than events are waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed cf53ab6: 1 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment on lines +270 to +272
schedule = pendingTasks < pending.size
if (schedule) pendingTasks++
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Submit reserved write tasks before releasing the pending lock

With concurrent captures, A can pause here before submitting its task. B's task then writes A's record, leaving B's record for A's later task. A flush() after B returns misses B, including during close(), when below flushAt. Preserve submission ordering by enqueueing tasks under pendingLock, following the synchronization rule in AGENTS.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants