[petruanica]

Description of the issue

Several gaps remain in which series the OTel suites test.

1. The suites read node agent metrics through a cache that adds a host.type matcher. The matcher names the instance types that the suite expects, for example t3.medium in the standard suite. So the tests that must hold on every series never see GPU, Neuron or Karpenter nodes.
2. The instance types are fixed in code, so a suite cannot run on a cluster with other instance types.
3. Cluster-scoped metrics get the same host.type matcher, so they can look absent.
4. The gpu suite finds its idle and multi-GPU nodes by the CI instance types.
5. The label tests know only the attribute denylist. A label include list or a namespace filter removes data in other processors. On a cluster that uses them, the label tests report false failures. A customer can also choose to remove a protected label with an exclude, and the denylist test then fails.

Description of changes

• OTEL_HOST_TYPES (util/otelmetrics/host_types.go) is a comma-separated list that replaces a suite's host types. Ten suites use it, and each TestMain prints the host types and the variable.
• Cluster-scoped metrics are read with only the cluster matcher (source_registry.go, query_cache.go).
• The standard suite adds allNodesCache, a cache with no host types. The tests that must hold on every series read through it. The presence tests keep the host-type pin.
• TallySchema reads through Get, so the schema rows reuse the cached reads. It fails for a cache that pins host types. WithSharedConcurrency lets two caches share one concurrency limit, so allNodesCache does not double the parallel load on the endpoint.
• The gpu suite reads its node roles from the data (util/otelmetrics/gpu_roles.go). The idle node has no k8s.pod.name. The busy node runs multi-gpu-burn. A multi-GPU node has an nvidia.com/gpu allocatable above 1 in kube-state-metrics. TestDCGMGPUCountMatchesAllocatable compares the DCGM GPU count of every GPU node with its allocatable. With an OTEL_HOST_TYPES override and no multi-GPU node, the multi-GPU tests skip. With the default host types, they fail.
• The label tests read the k8sattributes label rules and the namespace filter conditions from each agent's ConfigMap (label_extraction.go, namespace_filter.go). A label that no rule extracts must be absent. A pod in a namespace that the filter drops needs no series. If the model cannot read a filter, the subtest skips and names the processor and the reason (ErrFilterNotModeled). The tests read the rendered collector configuration, not the Helm values. ReconcileLabels and ReconcileObjects take a LabelExtraction in place of the label prefix.
• TestDenylistSparesNeverDeniedAttributes stays strict by default. OTEL_POD_LABEL_EXCLUDES and OTEL_NODE_LABEL_EXCLUDES declare the podLabels.exclude and nodeLabels.exclude patterns that a cluster was deployed with. The test logs the denylist entries that those patterns render to. It fails for every other entry that removes a never-denied attribute.

This is PR 3 of 3. It builds on PR 2.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race. They cover these areas:

• the host types and label excludes variables, and node groups
• the GPU roles and allocatable counts
• label extraction rules, namespace filter conditions, and filter shapes that the model cannot read
• cluster-scoped reads and the shared concurrency limit

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes. These commands also pass on PR 1 and PR 2 alone.

The standard and gpu suites ran on this commit against an EKS test cluster, with the chart from main and aws-observability/helm-charts#405:

• Standard suite: 186 of 189 tests pass, including the tests that read the Kubernetes API. Three tests fail because the test cluster lacks fixtures, not because of this stack. TestKarpenterGroupingLabelsRetained fails because Karpenter 1.0.11 sets no instance-tenancy label. TestKSM_PodBucket_JobPodName and TestKSM_WorkloadBucket_Job_HasName fail because the cluster has no ksm-test-job.
• gpu suite with OTEL_HOST_TYPES=g5.xlarge, on two single-GPU nodes: 25 tests pass. The 9 multi-GPU tests skip, because no node has more than one GPU.

[petruanica]

Replaced by #792, #793 and #794, a stack of three PRs with one commit each.