[petruanica]

Important

Breaking change. This change needs a major chart version bump, from 6.x to 7.0.0. It removes the instance-derived node labels that Karpenter and EKS Auto Mode set on the nodes they manage. It also removes 3 pod labels from every pod, whether or not either provisioner is in use. There is no deprecation window and no values.yaml opt-out, so it must not be picked into a minor or patch release.

Issue #, if available:

n/a

Description of changes:

Extends the denylist in awsattributelimit/cw_k8s_ci_v0 in both collector configurations: templates/linux/_otel-container-insights-config.tpl (node agent) and _otel-container-insights-cluster-scraper-config.tpl (cluster scraper). It denies the 21 Karpenter instance labels derived from the instance type and their 16 EKS Auto Mode copies. It also denies 3 karpenter.sh/ node-management flags and the 3 pod labels below. There is no other change in behavior.

Karpenter applies its labels to every node it manages, and the Kubernetes attribute enrichment copies them onto the resource block. So their key text rides every datapoint from a Karpenter-managed node. Most of them are derived from instance type details, so they restate host.type, which these pipelines already publish. The denylist already removes comparable platform-injected families (feature.node.kubernetes.io/, beta.kubernetes.io/, alpha.eksctl.io/, sourceLaunchTemplateId). The Karpenter equivalents were never added.

What is removed and what is kept

The list is complete against upstream's WellKnownLabels. Karpenter defines 28 labels under karpenter.k8s.aws/ and 5 under karpenter.sh/. This change denies 21 and 3. It keeps the other 9, because they are not derived from the instance type:

• karpenter.sh/nodepool, karpenter.k8s.aws/ec2nodeclass and karpenter.sh/capacity-type identify how the node was provisioned. capacity-type is the only spot or on-demand signal.
• karpenter.k8s.aws/capacity-reservation-id, -type and -interruptible describe the reservation a node consumed. They tell whether it is an ODCR or a Capacity Block, and whether EC2 can reclaim it.
• karpenter.k8s.aws/placement-group-id and -partition tell which placement group and partition the instance launched into. A partition is an independent failure domain.
• karpenter.k8s.aws/instance-tenancy is default or dedicated. Tenancy is a launch property: the same instance type runs on shared or dedicated hardware, so host.type does not give it.

Because of these 9, neither domain is denied by prefix. A single karpenter.k8s.aws/instance- prefix also removes instance-tenancy. So the 21 labels are denied by three narrower prefixes and 11 exact keys:

• The prefixes instance-cpu, instance-gpu- and instance-accelerator- cover 10 labels. They also cover CPU, GPU and accelerator labels that Karpenter adds in the future.
• The 11 exact keys are instance-{category,family,generation,size,memory,hypervisor,local-nvme,ebs-bandwidth,network-bandwidth,encryption-in-transit-supported,capability-flex}.

The 3 karpenter.sh/ keys are exact keys, because nodepool and capacity-type are kept in that domain.

EKS Auto Mode

Auto Mode is AWS-managed Karpenter. It sets the same instance-derived labels under eks.amazonaws.com/ instead of karpenter.k8s.aws/. The supported-labels table lists 16 keys: eks.amazonaws.com/instance-{category,family,size,generation,cpu,cpu-manufacturer,memory,hypervisor,local-nvme,ebs-bandwidth,network-bandwidth,encryption-in-transit-supported} and four gpu-* keys. Each restates host.type, as its Karpenter counterpart does, and the table lists no tenancy label. One prefix entry, k8s.node.label.eks.amazonaws.com/instance-, denies them.

The other labels in that domain do not begin with instance-, so the prefix does not catch them. They are compute-type, nodegroup, capacityType, fargate-profile and two capacity-reservation-* labels. compute-type is kept, because it identifies a node as managed by Auto Mode. Auto Mode uses karpenter.sh/capacity-type for the spot or on-demand signal, which this denylist keeps.

The integration tests do not cover the Auto Mode half. None of the clusters available to run them uses Auto Mode. On those clusters, an absence assertion passes even though the keys were never there. It rests on the label table and on the prefix being structurally equivalent to the Karpenter one.

GPU and Neuron dashboards are unaffected. Device identity travels on the datapoint attributes of the DCGM_FI_DEV_* and neuroncore_* metrics, which this change does not touch. Karpenter's Neuron branch also sets the instance-accelerator-* labels, so the accelerator prefix covers Trainium and Inferentia2 as well as Inferentia1.

Pod-label counterparts of already-denied node labels

It also denies three k8s.pod.label.* keys, on every cluster, with or without Karpenter:

• topology.kubernetes.io/region and topology.kubernetes.io/zone are pod-side copies of node labels that this denylist already removes. The admission plugin behind KEP-4742 copies the node's topology labels onto the pod, so they arrive in addition to the denied node form.
• pod-template-generation is a DaemonSet controller revision marker, from the same family as the already-denied pod-template-hash and controller-revision-hash.

Note

This change ships in the same release as #405. On some platforms and pipelines, the pod topology labels are the only region and zone a series has. #405 copies them into cloud.region and cloud.availability_zone before this denylist runs, so no series loses its region or zone.

What a consumer sees. The dimensions surface as the k8s_node_label_karpenter_*, k8s_node_label_eks_amazonaws_com_* and k8s_pod_label_* metric labels, and under the @resource. prefix in PromQL. Both stop resolving for every denied key. For example, a query that groups by karpenter.k8s.aws/instance-family returns one unlabelled series, and an alarm scoped to one instance-size stops matching anything. Queries on a pod's region or zone use cloud.region and cloud.availability_zone instead. Worth a release note.

Testing:

helm lint passes with a region set. The rendered configuration of both collectors contains the three karpenter.k8s.aws/instance- prefixes, the eks.amazonaws.com/instance- prefix and the 17 exact keys. It denies none of the 9 kept Karpenter labels and none of the kept Auto Mode labels. The chart suites pass: flag_matrix.sh 8/8, pod_level_values.sh 22/22, yaml_document_injection.sh 17/17, and the KServe, Knative, vLLM and Neuron pipeline shape tests.

aws/amazon-cloudwatch-agent-test#775 adds four integration tests. Each removal assertion has a guard:

• TestKarpenterInstanceLabelsRemoved checks that no label under the two instance- prefixes remains, except instance-tenancy. TestKarpenterGroupingLabelsRetained requires nodepool, capacity-type, ec2nodeclass and instance-tenancy.
• TestPodLabelCounterpartsRemoved checks the 3 pod labels. TestPodLabelsStillEnriched requires other pod labels to stay.

The guards make the kept-key decisions testable. If a regression widens either denylist to a domain prefix, the removal tests pass and the guards fail.

The tests ran on a live EKS cluster, because the claim is about what reaches CloudWatch, not what the configuration renders to. We applied the configuration to the running agent and cluster scraper and restarted the pods. After 30 minutes, test/otel/standard -computeType=EKS asserted on datapoints read back from CloudWatch.

Assertion	Before	After
TestKarpenterInstanceLabelsRemoved	18 fail	18 pass
TestKarpenterGroupingLabelsRetained (guard)	18 pass	18 pass
TestPodLabelCounterpartsRemoved	not measured	18 pass
TestPodLabelsStillEnriched (guard)	not measured	10 pass

There were no failures and no skips. The Karpenter pair also ran before the change, because an "attribute is absent" assertion passes trivially against empty data. The guard passing on both sides shows that only the intended keys went. The counterpart pair has no baseline from before the change. Its result shows that the keys are absent now, not that they were present before. In this run the denylist also removed instance-tenancy. So the removal results hold for every denied label that the cluster's nodes carry, and the retention of instance-tenancy is checked on the rendered configuration.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Related pull requests

Integration tests	aws/amazon-cloudwatch-agent-test#775
Ships in the same release	#405
Sibling Container Insights changes	#384, #385

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• Chart renders and lints locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: validated end-to-end on a live EKS cluster; baseline and post-change assertion counts are in the Testing section above. There is no public CI link because the integration suite needs a provisioned cluster and credentials that fork PRs cannot reach.
• New or updated integration test coverage: test(otel): cover Karpenter node-label and pod-label counterpart removal aws/amazon-cloudwatch-agent-test#775
• New functionality has unit tests; bug fixes have a reproducing test
• N/A — no config translation in this repo, so no golden files
• Breaking or customer-visible changes are called out in the PR description

[miconeilaws]

Is this a node label already?

[petruanica]

Yes, k8s.node.label.release is already on the denylist a few lines above, so this is its pod-side counterpart.

[miconeilaws]

A warning from my Agent:

k8s.pod.label.release is a common customer label, not just Helm noise. Teams use release: stable / release: canary for canary deployments; it's the standard example in the Kubernetes docs on labels. The node-side release label is near-meaningless, but the pod-side one often carries user intent. The PR itself admits this is the weakest of its arguments ("consistency"). Removing it on every pod, with no opt-out, will break real dashboards and alarms. I'd drop k8s.pod.label.release from this PR. helm.sh/chart is easier to defend, since it's tooling metadata, though the same "not redundant" point applies to it.

Are we sure about this one?

[miconeilaws]

louisall's commit 42d4524 on otel-ci-filters-labels (#394) probably needs updating based on the change around topology.k8s.aws/zone-id

[miconeilaws]

The test PR (#775) is stale against the final design. deniedKarpenterNodeLabels still says it "mirrors ... unconditional_removal_keys

[lorespiz]

Do we want to exclude from this change the pod zone label until cloud.availability_zone is fixed?

[louisall]

louisall's commit 42d4524 on otel-ci-filters-labels (#394) probably needs updating based on the change around topology.k8s.aws/zone-id

Done this. Dropped them from recommended list

[petruanica]

Do we want to exclude from this change the pod zone label until cloud.availability_zone is fixed?

Raised the fix: #405

[lorespiz]

This prefix also matches karpenter.k8s.aws/instance-tenancy. That's a well-known Karpenter label, and it's set on both Karpenter and Auto Mode nodes. Unlike the other instance-* labels, it can't be derived from host.type. It's a launch property, which is the same reason capacity-reservation-* and placement-group-* are kept. With this prefix, customers running dedicated-tenancy NodePools lose the only per-node tenancy dimension. Was that intended? If not, could we list the instance-type-derived keys explicitly instead of using the prefix, or exclude instance-tenancy from it?

[petruanica]

Not intended, thanks. Tenancy is a launch property, so the denylist now keeps it. The single prefix is replaced by instance-cpu, instance-gpu- and instance-accelerator- plus 11 exact keys, which still deny the other 21 instance labels. The Auto Mode prefix stays, because the documented eks.amazonaws.com/instance-* labels include no tenancy key.

[lorespiz]

Dropping k8s.pod.label.topology.kubernetes.io/zone and /region removes the only zone source in pipelines where nothing carries it into cloud.availability_zone. #405 does that only for the KSM pipeline, and not on AKS. The affected cases:

• AKS: the node-agent config has no cloud.availability_zone source, the node zone label is already denied, and fix(container-insights): take each series' availability zone and region from its node and pod, not the cluster scraper #405's copy is skipped on AKS. So AKS series end up with no zone at all.
• k8sMode=K8S off EC2: the ec2 detector fails, so these pod labels were the only region and zone info.
• Cluster-scraper Karpenter/KEDA/KServe/Knative pipelines: their resourcedetection turns off cloud.availability_zone, so the pod label was the only zone on those series.

This only affects clusters where the KEP-4742 PodTopologyLabels admission plugin is on, which is beta and on by default from 1.35. Is that loss acceptable, or should these platforms and pipelines keep the label, or get a copy into cloud.availability_zone like #405 does for KSM?

[petruanica]

Good catch, all three cases are real. #405 now copies the pod's region and zone labels into cloud.region and cloud.availability_zone before the denylist runs, in every pipeline that adds pod labels, on both collectors. It skips digit-only zones, because on AKS nodes outside availability zones the label holds the fault domain. This PR ships in the same release as #405.