[petruanica]

Note

Customer-visible, but not breaking in the way #383 is. Every attribute removed here has an identically-valued replacement already on the same series, so the consumer-side fix is a rename rather than a lost dimension.

Issue #, if available:

n/a

Description of changes:

Removes two groups of attributes that duplicate values the Container Insights pipelines already send.

Deprecated semantic conventions. The Prometheus receiver injects net.host.name, net.host.port and http.scheme on every scrape target alongside their current-spelling twins server.address, server.port and url.scheme. Each pair carries an identical value on the same series. A new transform/cw_k8s_ci_v0_drop_deprecated_semconv deletes the deprecated form and keeps the current one, wired into all fourteen Prometheus-receiver pipelines across both configs. efa is not in that set because it uses awsefareceiver, and kubeletstats is not because it uses kubeletstatsreceiver — neither receiver injects these attributes.

This also corrects an inversion: the karpenter, keda, kserve-controlplane and knative-controlplane promote transforms deleted url.scheme, the current name, while keeping the deprecated http.scheme. Those inline deletes are replaced by the shared processor, so all fourteen pipelines agree. Promote transforms are copied when a solution is added, so keeping the deletes in one processor referenced from each pipeline list means a new solution cannot reintroduce the inversion.

Doubled raw Prometheus labels. groupbyattrs moves each raw label to resource scope; the promote transforms then rename it to its semantic-convention name and copy the raw name back down to the datapoint, but never remove the resource-level copy. This deletes that copy in the seven pipelines that leave it behind: cadvisor, kube-state-metrics, dcgm, karpenter, keda, kserve-controlplane and knative-controlplane. The remaining Prometheus pipelines need no change. node-exporter, apiserver, vllm, knative-dataplane, ebs-csi and lis-csi run no groupbyattrs, so no raw name reaches resource scope at all; neuron runs one, but it lifts the semantic-convention names plus runtime_tag, which neuron_promote renames and then deletes.

The datapoint copies are kept for dashboard compatibility and the semantic-convention names are untouched, so no consumer loses a dimension. Nothing downstream reads the raw resource copies: k8sattributes keys its pod_association on k8s.pod.name / k8s.namespace.name / k8s.node.name, transform/cw_k8s_ci_v0_set_workload reads the k8s.*.name forms, and awsattributelimit keys on the k8s.node.label.* and k8s.pod.label.* prefixes. The new deletes sit in a trailing context: resource block so they run after the datapoint copy-back, since transform statement groups execute in declaration order.

Customer-visible changes

net.host.name, net.host.port and http.scheme disappear from every metric a Prometheus-receiver pipeline emits, so a query or alarm selecting on one stops matching. The replacement is a rename, not a removal: server.address, server.port and url.scheme carry identical values on the same series and are untouched, so the consumer fix is a one-line edit.

The raw-label deletions are narrower: the raw names (pod, namespace, container and the kube-state-metrics set) stay at datapoint scope, which is where dashboards read them. Only the duplicate resource-scope copy goes, so @resource.pod-style selectors stop resolving — that prefix is how PromQL addresses resource attributes — while the plain datapoint label keeps working.

Both are worth a release note; neither removes information that is not carried elsewhere in the payload.

Testing:

Tested with helm template across the chart's flag matrix with all solutions enabled, plus every suite in charts/amazon-cloudwatch-observability/tests: flag_matrix.sh (8 cases), pod_level_values.sh (22 checks), yaml_document_injection.sh (17 checks), kserve_pipeline_shape.sh (21 checks), vllm_pipeline_shape.sh (25 checks), knative_pipeline_shape.sh (22 checks) and neuron_pipeline_shape.sh (15 checks). All pass.

The rendered output of both agents was decoded and asserted directly: every pipeline with a prometheus/ receiver lists transform/cw_k8s_ci_v0_drop_deprecated_semconv, no processor anywhere deletes url.scheme, and each of the seven copy-back promote transforms emits its statement groups in resource then datapoint then resource order with the deletes in the trailing block.

New integration coverage in aws/amazon-cloudwatch-agent-test#776. TestDeprecatedSemconvAttributesRemoved asserts the three deprecated attributes are gone while url.scheme is still present on at least one series — the assertion that would have caught the inversion — and TestRawLabelsOnlyAtDatapointScope asserts the raw Prometheus labels exist only at datapoint scope. That PR also extends the same two checks into the gpu, neuron, ebs-csi, lis-csi, karpenter and keda suites through a shared helper, so each pipeline this PR touches is asserted in its own suite.

The standard-suite assertions were validated on a live EKS cluster, since the claim is about what reaches CloudWatch rather than what the config renders to: config applied to the running agent and cluster scraper, pods restarted, 30 minutes to settle, then test/otel/standard -computeType=EKS asserted on datapoints read back from CloudWatch. The suite was run before the change too, because an "attribute is absent" assertion passes trivially against empty data.

Assertion	Before	After
TestDeprecatedSemconvAttributesRemoved	27 fail	27 pass
TestRawLabelsOnlyAtDatapointScope	18 fail	18 pass

No failures, no skips. TestDeprecatedSemconvAttributesRemoved failing before the change is the result that matters most here: it is the assertion that would have caught the original inversion, and it only passes once the current-spelling twins are the ones left in place.

The dcgm, neuron and karpenter pipelines were checked on the same cluster. The gpu and neuron suites' new tests pass once their host-type lists point at the cluster's g5.xlarge and trn1.32xlarge: TestDCGMDeprecatedSemconvAttributesRemoved and TestDCGMRawLabelsOnlyAtDatapointScope 6 of 6 each, with TestDCGMHasRawPromotedKeys 6 of 6 while a GPU workload ran so that the datapoint copies were observable, and TestNeuronDeprecatedSemconvAttributesRemoved 4 of 4. Karpenter was checked by querying CloudWatch directly, because the Karpenter version on that cluster emits only 13 of the suite's 26 metric names: across 2,765 series, no deprecated attribute and no raw label at resource scope, url.scheme present on every series, and pod, namespace and node present at datapoint scope.

The vllm, kserve_controlplane, knative_dataplane and knative_controlplane pipelines have no live-cluster evidence, because the validation cluster runs none of those solutions. They are covered by the render assertions above.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Related pull requests

Integration tests	aws/amazon-cloudwatch-agent-test#776
Sibling Container Insights changes	#383, #385

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• Chart renders and lints locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: validated end-to-end on a live EKS cluster; baseline and post-change assertion counts are in the Testing section above. There is no public CI link because the integration suite needs a provisioned cluster and credentials that fork PRs cannot reach.
• New or updated integration test coverage: test(otel): cover removal of deprecated and doubled collector attributes aws/amazon-cloudwatch-agent-test#776
• New functionality has unit tests; bug fixes have a reproducing test
• N/A — no config translation in this repo, so no golden files
• Breaking or customer-visible changes are called out in the PR description

[miconeilaws]

Requested changes

Two comments are now wrong:
    _otel-container-insights-config.tpl:622 (dcgm) says "Raw labels are still at resource scope (not deleted), so copy them back down."
    _otel-container-insights-cluster-scraper-config.tpl:440 (KSM) says "Raw labels stay at resource scope and are copied back to datapoint scope."

Both should say the resource copies are deleted after the copy-back.

Squash the history. It is now feat → feat(drop id) → revert. The checklist claims a squashed, logical set of commits, so squash to one commit with the cAdvisor id paragraph removed from the message.

Update aws/amazon-cloudwatch-agent-test#776. Its title still says "cAdvisor cgroup-path attributes". Remove any id assertion there, or it will fail against this chart.

[lorespiz]

Since this ships with the chart bump to agent 1.300074, whose receiver is v0.150 and no longer emits net.host.*/http.scheme, drop_deprecated_semconv only takes effect when a customer overrides agent.image.tag to 1.300073 or earlier. Is it worth keeping as a guard for that case, or should we drop it to keep the pipelines simpler?

[petruanica]

Let's keep it for now, we can always drop it in the future