[petruanica]

Description of the issue

The PromQL endpoint that the OTel suites query returns at most 500 series for one query. It often drops the other series without a warning. A test that reads every series of a metric then sees only part of the cluster, and it can pass without looking at most nodes.

The client also retries a throttled or failed request after a fixed backoff of 1, 2 and 4 seconds. It ignores the Retry-After header. So a retry can come before the server is ready, or the suite can wait longer than the server asks.

Description of changes

OtelMetricsClient.QueryComplete (util/otelmetrics/complete.go) returns every series that a plain vector selector matches. It sends the selector and count() of the selector at the same time. Both use one evaluation time, which the client picks 5 seconds in the past. If the result has fewer series than the count, it fetches the selector again in disjoint shards. It splits by node first. When one node still has too many series, it also splits by other labels of the result. If the shards do not add up to the count, it returns an error and no partial result. A query that is not a plain vector selector goes to Query unchanged.

QueryCache reads through QueryComplete, so Get and GetWithFilter return every series. Each HTTP request, including each shard request, holds one slot of the cache's concurrency limit. A failed read for one host type now fails the whole Get. Before, Get dropped that host type without an error.

The suites had 29 direct client.Query calls that run a plain vector selector. They now call QueryComplete: efa (3), gpu (3), neuron (5), performance (1) and standard (17). Aggregations stay on Query, because the cap does not cut their input.

The retry loop reads Retry-After on HTTP 429 and 5xx responses, as a number of seconds or as an HTTP date. A missing or invalid header keeps the exponential backoff. The wait is at most 30 seconds (maxRetryAfter), so one throttled request cannot stall a suite.

This is PR 1 of 3. PR 2 and PR 3 build on it.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race.

• client_test.go covers retryDelay and a throttled request that waits for Retry-After.
• complete_test.go covers an untruncated read, the parallel selector and count, and sharding by node. It also covers a node over the limit, a label with too many values and inconsistent counts. The other cases are shard totals that do not match, a selector that cannot be split, and queries that are not plain selectors.
• query_cache_test.go covers truncated reads through the cache, a failed host type, and the concurrency limit for shard requests.

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes.

The standard and gpu suites ran against an EKS test cluster on the top of this stack (PR 3). QueryComplete found reads that the endpoint had cut, for example kube_pod_status_phase, and fetched them in shards. The results are in PR 3.

[petruanica]

Replaced by #792, #793 and #794, a stack of three PRs with one commit each.