Repository navigation
feat(flags): report unresolved local flags on evaluate_flags snapshots - #319
Draft
gustavohstrassburger wants to merge 2 commits into
Draft
gustavohstrassburger wants to merge 2 commits into
gustavohstrassburger wants to merge 2 commits into
Conversation
Implements the "Snapshot unresolved flag reporting" and "Warning for flags local evaluation never resolves" requirements of the evaluate-flags spec (PostHog/sdk-specs#85). - Add `FeatureFlagEvaluations#unresolved_flags`, with a `PostHog::UnresolvedFlagReason` for each flag and the spec's fixed precedence. - Report `local_evaluation_inconclusive` instead of `flag_missing` when a snapshot read hits an unresolved flag. - Do not stop a condition at the first inconclusive filter, so a filter that does not match resolves the condition locally. - Check `active` before experience continuity, also for dependencies. - Log one warning per flag that local evaluation can never resolve, repeated only when its definition changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 39793b31-809e-4689-a5d8-b4359cbb22c1
5 of 6 tasks
posthog-ruby-sync Compliance ReportDate: 2026-10-28T03:57:10.598571+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 10ms |
| Format Validation.Event Has Uuid | ✅ | 7ms |
| Format Validation.Event Has Lib Properties | ✅ | 9ms |
| Format Validation.Distinct Id Is String | ✅ | 7ms |
| Format Validation.Token Is Present | ✅ | 7ms |
| Format Validation.Custom Properties Preserved | ✅ | 7ms |
| Format Validation.Event Has Timestamp | ✅ | 10ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 7ms |
| Retry Behavior.Retries On 503 | ✅ | 5322ms |
| Retry Behavior.Does Not Retry On 400 | ❌ | 8ms |
| Retry Behavior.Does Not Retry On 401 | ❌ | 6ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8015ms |
| Retry Behavior.Implements Backoff | ❌ | 339ms |
| Retry Behavior.Retries On 500 | ✅ | 5116ms |
| Retry Behavior.Retries On 502 | ✅ | 5115ms |
| Retry Behavior.Retries On 504 | ✅ | 5116ms |
| Retry Behavior.Max Retries Respected | ❌ | 427ms |
| Deduplication.Generates Unique Uuids | ✅ | 17ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5115ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10319ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5150ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 20ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 10ms |
| Compression.Sends Gzip When Enabled | ✅ | 7ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 7ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 4ms |
| Batch Format.Multiple Events Batched Together | ❌ | 19ms |
| Error Handling.Does Not Retry On 403 | ❌ | 6ms |
| Error Handling.Does Not Retry On 413 | ❌ | 5ms |
| Error Handling.Retries On 408 | ✅ | 5139ms |
Failures
retry_behavior.does_not_retry_on_400
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.does_not_retry_on_401
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.implements_backoff
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.max_retries_respected
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
batch_format.multiple_events_batched_together
Expected 1 requests, got 5
error_handling.does_not_retry_on_403
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
error_handling.does_not_retry_on_413
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 9ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 8ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 9ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 10ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 11ms |
| Request Payload.Groups Round Trip | ✅ | 7ms |
| Request Payload.Groups Default To Empty Object | ✅ | 8ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 7ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 8ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 8ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 7ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 10ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 8ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 112ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 151ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 9ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
5 of 6 tasks
posthog-ruby-async Compliance ReportDate: 2026-10-28T03:57:47.336313+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 11ms |
| Format Validation.Event Has Uuid | ✅ | 106ms |
| Format Validation.Event Has Lib Properties | ✅ | 110ms |
| Format Validation.Distinct Id Is String | ✅ | 107ms |
| Format Validation.Token Is Present | ✅ | 107ms |
| Format Validation.Custom Properties Preserved | ✅ | 112ms |
| Format Validation.Event Has Timestamp | ✅ | 106ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 8ms |
| Retry Behavior.Retries On 503 | ✅ | 5312ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2109ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2108ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8117ms |
| Retry Behavior.Implements Backoff | ✅ | 15818ms |
| Retry Behavior.Retries On 500 | ✅ | 5213ms |
| Retry Behavior.Retries On 502 | ✅ | 5212ms |
| Retry Behavior.Retries On 504 | ✅ | 5208ms |
| Retry Behavior.Max Retries Respected | ✅ | 15622ms |
| Deduplication.Generates Unique Uuids | ✅ | 112ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5212ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10318ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5216ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 112ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 108ms |
| Compression.Sends Gzip When Enabled | ✅ | 106ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 107ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 5ms |
| Batch Format.Multiple Events Batched Together | ✅ | 111ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2107ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2109ms |
| Error Handling.Retries On 408 | ✅ | 5212ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 108ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 107ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 108ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 109ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 110ms |
| Request Payload.Groups Round Trip | ✅ | 107ms |
| Request Payload.Groups Default To Empty Object | ✅ | 107ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 108ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 107ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 108ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 106ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 111ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 108ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 210ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 243ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 109ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
5 of 6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
Why: sdk-specs #85 defines how a server SDK reports flags that local evaluation could not resolve. Today such a flag is absent from the
evaluate_flagssnapshot, so it looks the same as a flag that does not exist, and$feature_flag_calledreportsflag_missingfor both. Withonly_evaluate_locally: truethis is permanent. posthog-node already implements this (PostHog/posthog-js#5234); posthog-python (#1133) and posthog-server (#896) are in review.Changes:
FeatureFlagEvaluations#unresolved_flagsmaps each unresolved flag to aPostHog::UnresolvedFlagReason(experience_continuity,unsupported_definition,unresolved_dependency,missing_context). When several causes apply, the spec's fixed precedence picks one.only/only_accessedsnapshots have no entries, and reading the hash is not an access.local_evaluation_inconclusiveinstead offlag_missing.InconclusiveMatchErrorandRequiresServerEvaluationtake an optionalreason:, set at every raise site. A bad person value ismissing_context; a bad filter value isunsupported_definition.Behavior changes (all in the changeset):
falselocally.unresolved_dependency) at that condition instead of forcing the fallback, as in posthog-node. A later condition can now resolve it.false; it is absent and listed as unresolved. A failed flag with no local definition keeps itsfalsevalue.Divergences from the spec or posthog-node:
false, as in posthog-node.unsupported_definition, even when the person value causes the deferral. Patterns that always need the server are in the load-time warning; posthog-node does not warn about regexes.💚 How did you test it?
New RSpec examples mirror every scenario tagged
@unresolved_flags_capableand@unresolvable_flag_warning_capableinacceptance/public/evaluate-flags.feature, plus precedence, the condition loop, dependencies, warning dedup, failed fallback and unexpected errors. Run locally:bundle exec rspec(1988 examples, 0 failures, 40 pending for the optional OpenTelemetry bundle)bundle exec rubocopbundle exec rake public_api:checkNo manual testing in an app.
📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
PostHog::Utils.convert_to_datetime(@api private) now takes a requiredreason:so date parsing can tell a bad person value from a bad filter value.Created with PostHog Desktop
🤖 Generated with Claude Code