[petruanica]

Description of the issue

Several gaps remain in which series the OTel suites test.

1. The suites read node agent metrics through a cache that adds a host.type matcher. The matcher names the instance types that the suite expects, for example t3.medium in the standard suite. So the tests that must hold on every series never see GPU, Neuron or Karpenter nodes.
2. The instance types are fixed in code, so a suite cannot run on a cluster with other instance types.
3. Cluster-scoped metrics get the same host.type matcher, so they can look absent.
4. The gpu suite finds its idle and multi-GPU nodes by the CI instance types.

Description of changes

• OTEL_HOST_TYPES (util/otelmetrics/host_types.go) is a comma-separated list that replaces a suite's host types. Ten suites use it, and each TestMain prints the host types and the variable.
• Cluster-scoped metrics are read with only the cluster matcher (source_registry.go, query_cache.go).
• The standard suite adds allNodesCache, a cache with no host types. The tests that must hold on every series read through it. The presence tests keep the host-type pin.
• TallySchema reads through Get, so the schema rows reuse the cached reads. It fails for a cache that pins host types. WithSharedConcurrency lets two caches share one concurrency limit, so allNodesCache does not double the parallel load on the endpoint.
• The gpu suite reads its node roles from the data (util/otelmetrics/gpu_roles.go). The idle node has no k8s.pod.name. The busy node runs multi-gpu-burn. A multi-GPU node has an nvidia.com/gpu allocatable above 1 in kube-state-metrics. TestDCGMGPUCountMatchesAllocatable compares the DCGM GPU count of every GPU node with its allocatable. With an OTEL_HOST_TYPES override and no multi-GPU node, the multi-GPU tests skip. With the default host types, they fail.

This is PR 3 of 4. It builds on PR 2.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race. They cover these areas:

• the host types variable and node groups
• the GPU roles and allocatable counts
• cluster-scoped reads and the shared concurrency limit
• TallySchema through Get, which fails for a cache that pins host types and shares its read with the other tests that read every series

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes. These commands also pass on PR 1 and PR 2 alone.

The standard and gpu suites ran against an EKS test cluster at the top of this stack (PR 4), with the chart from aws-observability/helm-charts main. The run set otelContainerInsights.enableDRACorrelation=false.

• Standard suite: 186 of 189 tests pass, including the tests that read the Kubernetes API. Three tests fail because the test cluster lacks fixtures, not because of this stack. TestKarpenterGroupingLabelsRetained fails because Karpenter 1.0.11 sets no instance-tenancy label. TestKSM_PodBucket_JobPodName and TestKSM_WorkloadBucket_Job_HasName fail because the cluster has no ksm-test-job.
• gpu suite with OTEL_HOST_TYPES=g5.xlarge, on two single-GPU nodes: 25 tests pass. The 9 multi-GPU tests skip, because no node has more than one GPU.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): make compile and make simple-lint, the targets that the build-check workflow runs, pass on this commit, and so do the util/otelmetrics unit tests with -race.
• All GitHub Actions checks on the PR are passing
• Integration test evidence: N/A for a GitHub Actions run, because this PR changes the test framework itself. The standard and gpu suites ran against an EKS test cluster at the top of this stack, see Tests.
• New or updated integration test coverage: N/A, this PR is the test change.
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: this is PR 3 of 4. Merge it after fix(otelmetrics): return complete query results and honor Retry-After #792 and test(otel): check node and pod labels, the attribute schema and KSM zones against the cluster #793, and before test(otel): model the otelFilters label rules and namespace filter in the label tests #803.
• New functionality has unit tests; bug fixes have a reproducing test
• Config translation changes include updated golden files: N/A, this PR changes no config translation.
• Breaking or customer-visible changes are called out in the PR description: N/A, this PR changes test code only.