[petruanica]

Description of the issue

The PromQL endpoint that the OTel suites query returns at most 500 series for one query. It often drops the other series without a warning. A test that reads every series of a metric then sees only part of the cluster, and it can pass without looking at most nodes.

The client also retries a throttled or failed request after a fixed backoff of 1, 2 and 4 seconds. It ignores the Retry-After header. So a retry can come before the server is ready, or the suite can wait longer than the server asks.

Description of changes

OtelMetricsClient.QueryComplete (util/otelmetrics/complete.go) returns every series that a plain vector selector matches. It sends the selector and count() of the selector at the same time. Both use one evaluation time, which the client picks 5 seconds in the past. If the result has fewer series than the count, it fetches the selector again in disjoint shards. It splits by node first. When one node still has too many series, it also splits by other labels of the result. A label that has more values than one response can list, such as instance_type on the Karpenter instance type metrics, is split by value prefix, and so is a label past the first three candidates. The client counts the series per prefix with count by over label_replace, and a shard that is still over the limit gets a longer prefix. If the shards do not add up to the count, it returns an error and no partial result. A query that is not a plain vector selector goes to Query unchanged.

QueryCache reads through QueryComplete, so Get and GetWithFilter return every series. Each HTTP request, including each shard request, holds one slot of the cache's concurrency limit. A failed read for one host type now fails the whole Get. Before, Get dropped that host type without an error.

The suites had 35 direct client.Query calls that run a plain vector selector. They now call QueryComplete: efa (3), filters (1), gpu (3), gpu_filters (3), multi_efa_filters (1), neuron (5), neuron_filters (1), performance (1) and standard (17). Aggregations stay on Query, because the cap does not cut their input.

The retry loop reads Retry-After on HTTP 429 and 5xx responses, as a number of seconds or as an HTTP date. A missing or invalid header keeps the exponential backoff. The wait is at most 30 seconds (maxRetryAfter), so one throttled request cannot stall a suite.

This is PR 1 of 4. PR 2 to PR 4 build on it.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race.

• client_test.go covers retryDelay and a throttled request that waits for Retry-After.
• complete_test.go covers an untruncated read, the parallel selector and count, and sharding by node. It also covers a node over the limit, a label with too many values and inconsistent counts. The other cases are shard totals that do not match, a selector that cannot be split, a selector that matches no series, and queries that are not plain selectors.
• complete_test.go also covers the prefix split: one resource with more instance types than the limit, a prefix shard that needs a longer prefix, values with regex metacharacters, prefix counts that do not add up, a shared value prefix that is shorter than the sample's common prefix, and a label past the candidate limit.
• query_cache_test.go covers truncated reads through the cache, a failed host type, and the concurrency limit for shard requests.

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes.

The standard and gpu suites ran against an EKS test cluster at the top of this stack (PR 4), with the chart from aws-observability/helm-charts main. QueryComplete found reads that the endpoint had cut, for example kube_pod_status_phase and container_cpu_usage_seconds_total, and fetched them in shards. The results are in PR 3.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): make compile and make simple-lint, the targets that the build-check workflow runs, pass on this commit, and so do the util/otelmetrics unit tests with -race.
• All GitHub Actions checks on the PR are passing
• Integration test evidence: N/A for a GitHub Actions run, because this PR changes the test framework itself. The standard and gpu suites ran against an EKS test cluster at the top of this stack, and the results are under Tests in test(otel): run the every-series checks on every node #794.
• New or updated integration test coverage: N/A, this PR is the test change.
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: this is PR 1 of 4. Merge it first, then test(otel): check node and pod labels, the attribute schema and KSM zones against the cluster #793, test(otel): run the every-series checks on every node #794 and test(otel): model the otelFilters label rules and namespace filter in the label tests #803, which build on it. The filter suites from Add OTel Container Insights filters EKS integration test #782 (merged) now use QueryComplete too.
• New functionality has unit tests; bug fixes have a reproducing test
• Config translation changes include updated golden files: N/A, this PR changes no config translation.
• Breaking or customer-visible changes are called out in the PR description: N/A, this PR changes test code only.