[petruanica]

Description of the issue

The OTel suites test that some attributes exist on some series. Three gaps remain.

1. No test makes sure that every node and pod label that the chart keeps reaches the metrics with the right value. No test makes sure that the labels that the chart removes do not arrive.
2. A metric group can gain or lose an attribute, and no test fails.
3. The cluster scraper runs on one node, so its own availability zone is wrong for most kube-state-metrics series. No test makes sure that each series carries the zone of the node that it names.

Description of changes

Reverse label reconciliation (util/otelmetrics/label_reconcile.go, test/otel/standard/labels_reverse_test.go). TestNodeLabelsReachMetrics and TestPodLabelsReachMetrics start from the Kubernetes API. They list the nodes and pods, and they read the awsattributelimit denylist from each agent's ConfigMap. Each test has two subtests, one per label path:

• The cluster scraper path reads kube_node_info and kube_pod_info.
• The node agent path reads k8s.node.cpu.usage and k8s.pod.cpu.usage.

Every label that the denylist does not match must arrive with the same value. Every label that it matches must not arrive. Every object that a path reports must have a series. On the cluster scraper path, that is every node and pod. On the node agent path, it is every node with a ready agent pod, and the running pods on those nodes. If an object fails two passes 90 seconds apart, the test reports it. One failed pass is not enough, because a label change reaches the API before the next collection. The test skips a series at the attribute cap, because the cap can explain a missing label.

If a denylist entry removes an attribute in NeverDeniedKeys or under NeverDeniedPrefixes, TestDenylistSparesNeverDeniedAttributes fails. When the processor trims a series to the cap, it keeps these attributes. Its unconditional removal runs before that step, so a denylist entry removes them anyway.

Closed attribute schema (util/otelmetrics/attribute_schema.go, util/otelmetrics/schema_keys.go, and TestAttributeSchema in the standard, gpu, neuron and efa suites). Each metric group declares its required, expected and optional keys. It also declares open prefixes, such as k8s.node.label., and implications, such as "k8s.node.uid requires cloud.availability_zone". TestAttributeSchema tests every series of every metric in the group. It fails for an undeclared key, a missing required key or a broken implication. A chart change that adds or removes an attribute turns these tests red until someone edits the matching schema row. That is the intent.

KSM zones (test/otel/standard/ksm_availability_zone_test.go). On every kube-state-metrics series, cloud.availability_zone must be the zone of the node that the series names. The zone comes from the node metadata enricher or the node's zone label, and a pod's own zone label replaces it. A series of a known node with a zone label must carry a zone. A series that names no node, such as a Deployment series, must carry none. This covers aws-observability/helm-charts#405.

This is PR 2 of 3. It builds on PR 1.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

The unit tests in util/otelmetrics run without a cluster and pass with -race. attribute_schema_test.go covers key schemas, implications and tallies. label_reconcile_test.go covers denylist parsing, label reconciliation, the attribute cap, never-denied violations, object reconciliation and objects without a series.

go vet -tags integration passes on util/otelmetrics and on every package under test/otel. gofmt reports nothing, and make compile passes.

These tests ran against an EKS test cluster on the top of this stack (PR 3), with the chart from main and aws-observability/helm-charts#405:

• Both label tests reconcile every node and pod on both paths with no failure, and no object lacks a series.
• TestDenylistSparesNeverDeniedAttributes passes for both agents.
• On the zone test, every node, pod and container series carries the zone of its node, and no workload series carries a zone.
• TestAttributeSchema passes in the standard and gpu suites. The neuron and efa schema tests did not run, because the test cluster has no matching nodes.

[petruanica]

Replaced by #792, #793 and #794, a stack of three PRs with one commit each.