Skip to content

docs: Add the evaluateBatch command and evaluate-batch capability - #446

Draft
kinyoklion wants to merge 1 commit into
v2from
rlamb/evaluate-batch-contract
Draft

docs: Add the evaluateBatch command and evaluate-batch capability#446
kinyoklion wants to merge 1 commit into
v2from
rlamb/evaluate-batch-contract

Conversation

@kinyoklion

Copy link
Copy Markdown
Member

Summary

Adds the evaluateBatch command and the evaluate-batch capability to the test service specification.

evaluateBatch evaluates a list of flags against one context for a given number of passes inside the test service process, and reports timing for every evaluation. It exists for performance measurement: driving evaluations one HTTP request at a time measures the HTTP stack rather than the SDK, since a round trip costs far more than an evaluation. The SDK stress tester (launchdarkly/sdk-stress-tester) is the first consumer.

The contract:

  • Request mirrors evaluate: each element of evaluations carries flagKey, valueType, and defaultValue, with one shared context (or user) and an iterations count of at least 1. The context matrix in the stress tester is covered by issuing one batch per context.
  • Response returns one element per evaluation, in order, with the final pass's value plus totalNs, minNs, and maxNs across passes, and a whole-batch durationNs. Nanosecond integers are used throughout because every SDK runtime exposes a monotonic nanosecond clock and single evaluations are microseconds.
  • Measurement rules pin what the clock covers: request decoding, context construction, and default-value conversion happen before timing; each evaluation is timed individually; evaluations run sequentially on one thread; the response is built after the final pass.
  • The harness only sends the command when the service advertises evaluate-batch.

Deliberately not included in this revision: a detail option and a concurrency parameter. Either can be added later as optional request properties without breaking this shape.

No harness code changes are included. The prototype implementations in five SDK test services are local and will be updated to this contract before they are proposed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant