[petruanica]

Description of the issue

The OTel suites test that some attributes exist on some series. Three gaps remain.

1. No test makes sure that every node and pod label that the chart keeps reaches the metrics with the right value. No test makes sure that the labels that the chart removes do not arrive.
2. A metric group can gain or lose an attribute, and no test fails.
3. The cluster scraper runs on one node, so its own availability zone is wrong for most kube-state-metrics series. No test makes sure that each series carries the zone of the node that it names.

Description of changes

Reverse label reconciliation (util/otelmetrics/label_reconcile.go, test/otel/standard/labels_reverse_test.go). TestNodeLabelsReachMetrics and TestPodLabelsReachMetrics start from the Kubernetes API. They list the nodes and pods, and they read the awsattributelimit denylist from each agent's ConfigMap. Each test has two subtests, one per label path:

• The cluster scraper path reads kube_node_info and kube_pod_info.
• The node agent path reads k8s.node.cpu.usage and k8s.pod.cpu.usage.

Every label that the denylist does not match must arrive with the same value. Every label that it matches must not arrive. Every object that a path reports must have a series. On the cluster scraper path, that is every node and pod. On the node agent path, it is every node with a ready agent pod, and the running pods on those nodes. If an object fails two passes 90 seconds apart, the test reports it. One failed pass is not enough, because a label change reaches the API before the next collection. The test skips a series at the attribute cap, because the cap can explain a missing label.

If a denylist entry removes an attribute in NeverDeniedKeys or under NeverDeniedPrefixes, TestDenylistSparesNeverDeniedAttributes fails. When the processor trims a series to the cap, it keeps these attributes. Its unconditional removal runs before that step, so a denylist entry removes them anyway.

Closed attribute schema (util/otelmetrics/attribute_schema.go, util/otelmetrics/schema_keys.go, and TestAttributeSchema in the standard, gpu, neuron and efa suites). Each metric group declares its required, expected and optional keys. It also declares open prefixes, such as k8s.node.label., and implications, such as "k8s.node.uid requires cloud.availability_zone". TestAttributeSchema tests every series of every metric in the group. It fails for an undeclared key, a missing required key or a broken implication. A chart change that adds or removes an attribute turns these tests red until someone edits the matching schema row. That is the intent.

KSM zones (test/otel/standard/ksm_availability_zone_test.go). On every kube-state-metrics series, cloud.availability_zone must be the zone of the node that the series names. The zone comes from the node metadata enricher or the node's zone label, and a pod's own zone label replaces it. A series of a known node with a zone label must carry a zone. A series that names no node, such as a Deployment series, must carry none. This covers aws-observability/helm-charts#405.

This is PR 2 of 4. It builds on PR 1.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race. attribute_schema_test.go covers key schemas, implications and tallies. label_reconcile_test.go covers denylist parsing, label reconciliation, the attribute cap, never-denied violations, object reconciliation and objects without a series.

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes.

These tests ran against an EKS test cluster at the top of this stack (PR 4), with the chart from aws-observability/helm-charts main:

• Both label tests reconcile every node and pod on both paths with no failure, and no object lacks a series.
• TestDenylistSparesNeverDeniedAttributes passes for both agents.
• On the zone test, every node, pod and container series carries the zone of its node, and no workload series carries a zone.
• TestAttributeSchema passes in the standard and gpu suites. The neuron and efa schema tests were not part of this run.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): make compile and make simple-lint, the targets that the build-check workflow runs, pass on this commit, and so do the util/otelmetrics unit tests with -race.
• All GitHub Actions checks on the PR are passing
• Integration test evidence: N/A for a GitHub Actions run, because this PR changes the test framework itself. The tests ran against an EKS test cluster at the top of this stack, see Tests.
• New or updated integration test coverage: N/A, this PR is the test change.
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: this is PR 2 of 4. Merge it after fix(otelmetrics): return complete query results and honor Retry-After #792 and before test(otel): run the every-series checks on every node #794 and test(otel): model the otelFilters label rules and namespace filter in the label tests #803. The zone test covers fix(container-insights): take each series' availability zone and region from its node and pod, not the cluster scraper aws-observability/helm-charts#405 (merged). test(container-insights): pin deprecated-semconv and raw-label invariants across Prometheus pipelines aws-observability/helm-charts#424 checks the chart's denylists at render time against the same never-denied list (NeverDeniedKeys), and links back here.
• New functionality has unit tests; bug fixes have a reproducing test
• Config translation changes include updated golden files: N/A, this PR changes no config translation.
• Breaking or customer-visible changes are called out in the PR description: N/A, this PR changes test code only.