Repository navigation
fix: drop the oldest AI events while too many wait to be written to disk - #901
turnipdabeets wants to merge 3 commits into
Conversation
One executor both writes and sends AI events, so during a slow upload new multi-MiB events waited in memory with no limit. captureAi now returns null while captureAiMaxQueueSize events or 50 MiB are waiting, the same limits the disk queue keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
posthog-android Compliance ReportDate: 2026-10-29T01:30:50.719311+00:00
|
| Test | Status | Duration |
|---|---|---|
| Endpoint And Method.Targets V1 Endpoint | ✅ | 435ms |
| Endpoint And Method.Does Not Use Legacy Endpoints | ✅ | 37ms |
| Required Headers.Has Authorization Bearer Header | ✅ | 32ms |
| Required Headers.Has Content Type Json | ✅ | 31ms |
| Required Headers.Has Posthog Sdk Info Format | ✅ | 32ms |
| Required Headers.Has Posthog Attempt Header | ✅ | 26ms |
| Required Headers.Has Posthog Request Id | ✅ | 27ms |
| Required Headers.Has Posthog Request Timestamp | ✅ | 30ms |
| Required Headers.Has User Agent | ✅ | 26ms |
| Body Format.Body Has Created At And Batch | ✅ | 28ms |
| Body Format.No Api Key In Body | ✅ | 26ms |
| Body Format.No Sent At In Body | ✅ | 23ms |
| Event Format.Event Has Required Root Fields | ✅ | 25ms |
| Event Format.Event Uuid Is Valid | ✅ | 26ms |
| Event Format.Event Timestamp Is Rfc3339 | ✅ | 24ms |
| Event Format.Non Utc Event Timestamp Is Converted To Utc | ✅ | 26ms |
| Event Format.Distinct Id Is String | ✅ | 21ms |
| Event Format.Distinct Id At Root Not Properties | ✅ | 24ms |
| Event Format.Custom Properties Preserved | ✅ | 24ms |
| Event Format.Set Properties Preserved | ✅ | 25ms |
| Event Format.Set Once Properties Preserved | ✅ | 26ms |
| Event Format.Groups Properties Preserved | ✅ | 25ms |
| Event Format.Sdk Generates Uuid If Not Provided | ✅ | 26ms |
| Event Format.Event Has Required Root Fields Batch | ✅ | 32ms |
| Event Format.Event Uuid Is Valid Batch | ✅ | 32ms |
| Event Format.Event Timestamp Is Rfc3339 Batch | ✅ | 27ms |
| Event Format.Distinct Id Is String Batch | ✅ | 28ms |
| Event Format.Distinct Id At Root Not Properties Batch | ✅ | 27ms |
| Event Format.Custom Properties Preserved Batch | ✅ | 30ms |
| Event Format.Set Properties Preserved Batch | ✅ | 30ms |
| Event Format.Set Once Properties Preserved Batch | ✅ | 43ms |
| Event Format.Groups Properties Preserved Batch | ✅ | 30ms |
| Event Format.Sdk Generates Uuid If Not Provided Batch | ✅ | 27ms |
| Batch Behavior.Multiple Events In Single Batch | ✅ | 37ms |
| Batch Behavior.Batch Envelope Smoke | ✅ | 33ms |
| Batch Behavior.Flush With No Events Sends Nothing | ✅ | 19ms |
| Batch Behavior.Flush At Triggers Batch | ✅ | 1029ms |
| Batch Behavior.Created At Reflects Batch Creation Time | ✅ | 21ms |
| Deduplication.Generates Unique Uuids | ✅ | 42ms |
| Deduplication.Different Events Same Content Different Uuids | ✅ | 23ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 7016ms |
| Deduplication.Preserves Timestamp On Retry | ✅ | 7018ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 7023ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 34ms |
| Header Behavior On Retry.Attempt Header Starts At One | ✅ | 17ms |
| Header Behavior On Retry.Attempt Header Increments On Retry | ✅ | 12024ms |
| Header Behavior On Retry.Request Id Preserved On Retry | ✅ | 7017ms |
| Header Behavior On Retry.Different Requests Have Different Request Ids | ✅ | 2029ms |
| Header Behavior On Retry.Request Timestamp Changes On Retry | ✅ | 7017ms |
| Response Format Validation.Success Response Has Uuid Keyed Results | ✅ | 19ms |
| Response Format Validation.Success Response Has Ok For Each Event | ✅ | 24ms |
| Response Format Validation.Success No Retry After When All Ok | ✅ | 20ms |
| Response Format Validation.Success Retry After Present When Retry Events | ✅ | 2010ms |
| Response Format Validation.Success No Retry After When Drop Only | ✅ | 20ms |
| Response Format Validation.Response Echoes Request Id | ✅ | 18ms |
| Retry Behavior.Retries On 408 | ✅ | 7017ms |
| Retry Behavior.Retries On 500 | ✅ | 7016ms |
| Retry Behavior.Retries On 503 | ✅ | 7024ms |
| Retry Behavior.Retries On 504 | ✅ | 7017ms |
| Retry Behavior.Retryable Errors Have Retry After | ✅ | 4013ms |
| Retry Behavior.Respects Retry After On Retryable Error | ✅ | 10024ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 4021ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 4020ms |
| Retry Behavior.Does Not Retry On 402 | ✅ | 4020ms |
| Retry Behavior.Does Not Retry On 413 | ✅ | 4018ms |
| Retry Behavior.Does Not Retry On 415 | ✅ | 4020ms |
| Retry Behavior.Non Retryable Errors Have No Retry After | ✅ | 4018ms |
| Retry Behavior.Implements Backoff | ✅ | 17022ms |
| Retry Behavior.Max Retries Respected | ❌ | 17037ms |
| Partial Batch Handling.Handles 200 Full Success | ✅ | 2017ms |
| Partial Batch Handling.Handles 200 With All Ok | ✅ | 3029ms |
| Partial Batch Handling.Does Not Retry Dropped Events | ✅ | 3027ms |
| Partial Batch Handling.Does Not Retry Limited Events | ✅ | 3023ms |
| Partial Batch Handling.Prunes Ok Events On Partial Retry | ✅ | 7013ms |
| Partial Batch Handling.Prunes Dropped Events On Partial Retry | ✅ | 7017ms |
| Partial Batch Handling.Retries Only Retry Events From Partial | ✅ | 7017ms |
| Partial Batch Handling.Partial Retry Preserves Uuids | ✅ | 7015ms |
| Partial Batch Handling.Partial Retry Attempt Header Increments | ✅ | 7014ms |
| Partial Batch Handling.Partial Retry Request Id Preserved | ✅ | 7016ms |
| Partial Batch Handling.Respects Retry After On Partial | ✅ | 7024ms |
| Partial Batch Handling.Unknown Result Treated As Terminal | ✅ | 3022ms |
| Partial Batch Handling.Mixed Ok Drop Limited No Retry | ✅ | 3033ms |
| Compression.Sends Gzip Content Encoding | ✅ | 15ms |
| Compression.No Content Encoding When Disabled | ✅ | 14ms |
| Compression.Compressed Body Is Decompressible | ✅ | 15ms |
| Error Handling.Does Not Retry On Unknown 4Xx | ✅ | 4017ms |
| Event Options.Cookieless Mode Override | ✅ | 17ms |
| Event Options.Disable Skew Correction Override | ✅ | 13ms |
| Event Options.Process Person Profile Override | ✅ | 16ms |
| Event Options.Product Tour Id Override | ✅ | 17ms |
| Event Options.Unset Options Omitted | ✅ | 15ms |
| Event Options.Options Override In Batch | ✅ | 23ms |
| Event Options.Unknown Option Passes Through | ✅ | 17ms |
| Event Options.Option Value Is Not Converted | ✅ | 16ms |
| Event Options.Lenient Bool Option Values Are Not Converted | ✅ | 18ms |
| Event Options.Option Wins Over Legacy Property | ✅ | 28ms |
| Event Options.Legacy Property Fills Unset Option | ✅ | 23ms |
| Event Options.Null Option Falls Back To Legacy Property | ✅ | 22ms |
| Geoip And Historical Migration.Geoip Disable Injected Into Properties | ❌ | 23ms |
| Geoip And Historical Migration.Historical Migration Set In Body | ❌ | 23ms |
| Geoip And Historical Migration.Historical Migration Absent By Default | ✅ | 21ms |
Failures
retry_behavior.max_retries_respected
Expected 4 requests, got 5
geoip_and_historical_migration.geoip_disable_injected_into_properties
Expected $geoip_disable='True', got 'None'. Available properties: ['$is_identified', '$recording_status', '$sdk_debug_pending_queue_size']
geoip_and_historical_migration.historical_migration_set_in_body
Body missing 'historical_migration' field. Body keys: ['created_at', 'batch']
Capture_Ai_V1 Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Endpoint And Method.Targets V1 Endpoint | ✅ | 22ms |
| Endpoint And Method.Does Not Use Legacy Endpoints | ✅ | 21ms |
| Endpoint And Method.Capture Does Not Reroute Ai Named Events | ✅ | 20ms |
| Required Headers.Has Authorization Bearer Header | ✅ | 18ms |
| Required Headers.Has Content Type Json | ✅ | 19ms |
| Required Headers.Has Posthog Sdk Info Format | ✅ | 24ms |
| Required Headers.Has Posthog Attempt Header | ✅ | 22ms |
| Required Headers.Has Posthog Request Id | ✅ | 24ms |
| Required Headers.Has Posthog Request Timestamp | ✅ | 20ms |
| Required Headers.Has User Agent | ✅ | 23ms |
| Body Format.Body Has Created At And Batch | ✅ | 31ms |
| Body Format.No Api Key In Body | ✅ | 23ms |
| Body Format.No Sent At In Body | ✅ | 17ms |
| Event Format.Event Has Required Root Fields | ✅ | 14ms |
| Event Format.Event Uuid Is Valid | ✅ | 15ms |
| Event Format.Event Timestamp Is Rfc3339 | ✅ | 13ms |
| Event Format.Non Utc Event Timestamp Is Converted To Utc | ✅ | 14ms |
| Event Format.Distinct Id Is String | ✅ | 15ms |
| Event Format.Distinct Id At Root Not Properties | ✅ | 13ms |
| Event Format.Custom Properties Preserved | ✅ | 18ms |
| Event Format.Set Properties Preserved | ✅ | 18ms |
| Event Format.Set Once Properties Preserved | ✅ | 20ms |
| Event Format.Groups Properties Preserved | ✅ | 20ms |
| Event Format.Sdk Generates Uuid If Not Provided | ✅ | 19ms |
| Event Format.Capture Ai Keeps Supplied Uuid | ❌ | 12ms |
| Event Format.Event Has Required Root Fields Batch | ❌ | 27ms |
| Event Format.Event Uuid Is Valid Batch | ✅ | 24ms |
| Event Format.Event Timestamp Is Rfc3339 Batch | ✅ | 25ms |
| Event Format.Distinct Id Is String Batch | ✅ | 21ms |
| Event Format.Distinct Id At Root Not Properties Batch | ✅ | 24ms |
| Event Format.Custom Properties Preserved Batch | ✅ | 22ms |
| Event Format.Set Properties Preserved Batch | ✅ | 30ms |
| Event Format.Set Once Properties Preserved Batch | ✅ | 26ms |
| Event Format.Groups Properties Preserved Batch | ✅ | 27ms |
| Event Format.Sdk Generates Uuid If Not Provided Batch | ✅ | 28ms |
| Batch Behavior.Multiple Events In Single Batch | ✅ | 32ms |
| Batch Behavior.Batch Envelope Smoke | ✅ | 28ms |
| Batch Behavior.Flush At Triggers Batch | ✅ | 1016ms |
| Batch Behavior.Created At Reflects Batch Creation Time | ✅ | 15ms |
| Deduplication.Generates Unique Uuids | ✅ | 32ms |
| Deduplication.Different Events Same Content Different Uuids | ✅ | 21ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 7016ms |
| Deduplication.Preserves Timestamp On Retry | ✅ | 7015ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 7016ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 23ms |
| Header Behavior On Retry.Attempt Header Starts At One | ✅ | 13ms |
| Header Behavior On Retry.Attempt Header Increments On Retry | ✅ | 12020ms |
| Header Behavior On Retry.Request Id Preserved On Retry | ✅ | 7013ms |
| Header Behavior On Retry.Different Requests Have Different Request Ids | ✅ | 2024ms |
| Header Behavior On Retry.Request Timestamp Changes On Retry | ✅ | 7016ms |
| Response Format Validation.Success Response Has Uuid Keyed Results | ✅ | 16ms |
| Response Format Validation.Success Response Has Ok For Each Event | ✅ | 22ms |
| Response Format Validation.Success No Retry After When All Ok | ✅ | 17ms |
| Response Format Validation.Success Retry After Present When Retry Events | ✅ | 2008ms |
| Response Format Validation.Success No Retry After When Drop Only | ✅ | 18ms |
| Response Format Validation.Response Echoes Request Id | ✅ | 13ms |
| Retry Behavior.Retries On 408 | ✅ | 7013ms |
| Retry Behavior.Retries On 500 | ✅ | 7015ms |
| Retry Behavior.Retries On 503 | ✅ | 7022ms |
| Retry Behavior.Retries On 504 | ✅ | 7011ms |
| Retry Behavior.Retryable Errors Have Retry After | ✅ | 4013ms |
| Retry Behavior.Respects Retry After On Retryable Error | ✅ | 10023ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 4018ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 4021ms |
| Retry Behavior.Does Not Retry On 402 | ✅ | 4016ms |
| Retry Behavior.Does Not Retry On 413 | ✅ | 4018ms |
| Retry Behavior.Does Not Retry On 415 | ✅ | 4015ms |
| Retry Behavior.Non Retryable Errors Have No Retry After | ✅ | 4017ms |
| Retry Behavior.Implements Backoff | ✅ | 17027ms |
| Retry Behavior.Max Retries Respected | ❌ | 17030ms |
| Partial Batch Handling.Handles 200 Full Success | ✅ | 2019ms |
| Partial Batch Handling.Handles 200 With All Ok | ✅ | 3023ms |
| Partial Batch Handling.Does Not Retry Dropped Events | ✅ | 3023ms |
| Partial Batch Handling.Does Not Retry Limited Events | ✅ | 3021ms |
| Partial Batch Handling.Prunes Ok Events On Partial Retry | ✅ | 7015ms |
| Partial Batch Handling.Prunes Dropped Events On Partial Retry | ✅ | 7017ms |
| Partial Batch Handling.Retries Only Retry Events From Partial | ✅ | 7014ms |
| Partial Batch Handling.Partial Retry Preserves Uuids | ✅ | 7015ms |
| Partial Batch Handling.Partial Retry Attempt Header Increments | ✅ | 7014ms |
| Partial Batch Handling.Partial Retry Request Id Preserved | ✅ | 7015ms |
| Partial Batch Handling.Respects Retry After On Partial | ✅ | 7021ms |
| Partial Batch Handling.Unknown Result Treated As Terminal | ✅ | 3021ms |
| Partial Batch Handling.Mixed Ok Drop Limited No Retry | ✅ | 3026ms |
| Compression.Sends Gzip Content Encoding | ✅ | 14ms |
| Compression.No Content Encoding When Disabled | ✅ | 12ms |
| Compression.Compressed Body Is Decompressible | ✅ | 15ms |
| Error Handling.Does Not Retry On Unknown 4Xx | ✅ | 4016ms |
| Event Options.Cookieless Mode Override | ✅ | 15ms |
| Event Options.Disable Skew Correction Override | ✅ | 15ms |
| Event Options.Process Person Profile Override | ✅ | 15ms |
| Event Options.Product Tour Id Override | ✅ | 16ms |
| Event Options.Unset Options Omitted | ✅ | 20ms |
| Event Options.Options Override In Batch | ✅ | 23ms |
| Event Options.Unknown Option Passes Through | ✅ | 14ms |
| Event Options.Option Value Is Not Converted | ✅ | 15ms |
| Event Options.Lenient Bool Option Values Are Not Converted | ✅ | 13ms |
| Event Options.Option Wins Over Legacy Property | ✅ | 19ms |
| Event Options.Legacy Property Fills Unset Option | ✅ | 16ms |
| Event Options.Null Option Falls Back To Legacy Property | ✅ | 15ms |
| Geoip And Historical Migration.Geoip Disable Injected Into Properties | ❌ | 16ms |
| Geoip And Historical Migration.Historical Migration Absent By Default | ✅ | 16ms |
Failures
event_format.capture_ai_keeps_supplied_uuid
Expected last action result '0198c0de-0000-7000-8000-000000000abc', got '01a18544-935f-799d-9c70-c6ea1a7a0e78'
event_format.event_has_required_root_fields_batch
Expected 1 requests, got 2
retry_behavior.max_retries_respected
Expected 4 requests, got 5
geoip_and_historical_migration.geoip_disable_injected_into_properties
Expected $geoip_disable='True', got 'None'. Available properties: ['$is_identified', '$recording_status', '$sdk_debug_session_start', '$sdk_debug_pending_queue_size']
Feature_Flags Tests
✅ 17/17 tests passed
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 35ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 22ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 25ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 21ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 20ms |
| Request Payload.Groups Round Trip | ✅ | 23ms |
| Request Payload.Groups Default To Empty Object | ✅ | 20ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 20ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ✅ | 17ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 17ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 6ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 16ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 29ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 16ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 318ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 318ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 20ms |
While a slow upload holds the AI executor, captureAi now keeps the new event and drops the oldest events still waiting to be written once captureAiMaxQueueSize events or 50 MiB wait, the same as the disk queue already does. captureAi returns null again only for disabled, opted-out, beforeSend-dropped or oversized events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Dropping a waiting AI event no longer leaves its executor task behind for nothing: a dropped event's task writes a newer one instead, so the executor never holds more tasks than events are waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
PostHog Review · Standard · reviewed cf53ab6: 1 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
| schedule = pendingTasks < pending.size | ||
| if (schedule) pendingTasks++ | ||
| } |
There was a problem hiding this comment.
P2 · Submit reserved write tasks before releasing the pending lock
With concurrent captures, A can pause here before submitting its task. B's task then writes A's record, leaving B's record for A's later task. A flush() after B returns misses B, including during close(), when below flushAt. Preserve submission ordering by enqueueing tasks under pendingLock, following the synchronization rule in AGENTS.md.
Stacked on #881.
💡 Motivation and Context
One single-threaded executor both writes AI events to disk and sends them. During a slow upload (up to
captureAiTimeout, 60 s, per request), newcaptureAievents wait in the executor's task queue, in memory, with no limit. The 100-event / 50 MiB caps only applied once an event was written, so a chat app capturing multi-MiB generations on a slow network could exhaust the heap. PostHog Review flagged this on #881 and #882.AI events now wait in a bounded list until the executor writes them. Once more than
captureAiMaxQueueSizeevents (100 by default) or 50 MiB wait, the oldest waiting events are dropped and the new one is kept, the same as the disk queue already does (and as the posthog-ios 4.0 AI queue and the sdk-specscapture-aiclient note describe). These are the disk queue's limits, so holding more in memory couldn't deliver more events; it only delayed the eviction.captureAistill returns the new event's uuid.PostHogQueue.addMeasured(record, bytes): adds the record to the waiting list, drops the oldest waiting records while over the count or bytes, and queues an executor task that writes the oldest waiting record, only while there are fewer tasks than waiting records, so dropped events leave no tasks behind. Only the AI lane uses it; analytics, replay and logs keepadd.captureAipasses the size it already measures for the per-event limit, so there's no second serialization.captureAiMaxQueueSizedescribe the drop.How the other SDKs bound waiting AI events
captureAiwrites to disk on the caller's threadErrQueueFullThis follows posthog-ios and the disk queue (drop the oldest) with mobile limits: a count-only cap of 100 would still allow ~800 MiB of 8 MiB events. A synchronous write like iOS would put disk I/O on Android's main thread. posthog-flutter needs no change: mobile calls the native SDKs, and the desktop Dart lane enforces its caps when it queues.
Splitting writing from sending into separate threads would remove the waiting entirely, but it changes the queue shared by analytics, replay and logs, so it's left out of this PR.
💚 How did you test it?
PostHogQueueByteLimitsTest: with the executor blocked, going over the waiting bytes and then the waiting count each drops the oldest waiting record; the queue ends with the newest three.PostHogCaptureAiTest: with the AI executor blocked andcaptureAiMaxQueueSize = 2, the thirdcaptureAistill returns a uuid and the oldest waiting event is dropped../gradlew apiCheck spotlessCheck :posthog:test :posthog-android:testReleaseUnitTest :posthog-server:testpass locally.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file (updated the unreleased v4 major changeset'scaptureAibullet)🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Claude Code compared how posthog-ios, posthog-python, posthog-go and posthog-rs bound AI events waiting to be sent, from their v1/4.0 branch sources, and implemented the cap as a follow-up to #881; after review against sdk-specs #147, the maintainer switched it from drop-the-newest to drop-the-oldest. Rejected: a count-only cap (too much memory on a device), a synchronous write (main-thread I/O), separate write and send threads (touches every queue; possible later).
🤖 Generated with Claude Code