Skip to content

Drop deprecated and duplicated resource attributes from Container Insights - #377

Open
petruanica wants to merge 1 commit into
aws-observability:mainfrom
petruanica:otel-ci-l1-drop-redundant-attributes
Open

petruanica wants to merge 1 commit into
aws-observability:mainfrom
petruanica:otel-ci-l1-drop-redundant-attributes

Conversation

@petruanica

@petruanica petruanica commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Release coordination (working assumption, not yet decided). This PR is written assuming it ships together with the two sibling Container Insights changes in a single major chart release, v7.0.0. This change is not itself breaking, but it is customer-visible, so bundling it with the major release avoids a customer-visible dimension change landing in a minor version. That bundling has not been agreed yet and needs confirming before merge.

Issue #, if available:

n/a

Description of changes:

Remove two groups of resource attributes that duplicate values the Container Insights pipelines already send.

Deprecated semantic conventions. The Prometheus receiver injects net.host.name, net.host.port and http.scheme on every scrape target alongside their current-spelling twins server.address, server.port and url.scheme. Each pair carries an identical value on the same series. A new transform/cw_k8s_ci_v0_drop_deprecated_semconv deletes the deprecated form and keeps the current one, wired into all ten Prometheus-receiver pipelines across both configs.

This also corrects an inversion: the karpenter and keda promote transforms deleted url.scheme — the current name — while keeping http.scheme, the deprecated one. Those inline deletes are replaced by the shared processor, so all ten pipelines now agree.

Doubled raw Prometheus labels. groupbyattrs moves each raw label to resource scope; the promote transforms then rename it to its semantic-convention name and copy the raw name back down to the datapoint, but never remove the resource-level copy. This deletes that copy in the five pipelines that leave it behind — cadvisor, kube-state-metrics, dcgm, karpenter and keda — bringing them in line with neuron, efa, ebs-csi and lis-csi, which already promote-and-delete.

The datapoint copies are kept for dashboard compatibility and the semantic-convention names are untouched, so no consumer loses a dimension. Nothing downstream reads the raw resource copies: k8sattributes keys its pod_association on k8s.pod.name / k8s.namespace.name / k8s.node.name, transform/cw_k8s_ci_v0_set_workload reads the k8s.*.name forms, and awsattributelimit keys on the k8s.node.label.* and k8s.pod.label.* prefixes.

The new delete statements sit in a trailing context: resource block that runs after the datapoint copy-back, since transform statement groups execute in declaration order.

Customer-visible changes

Three resource attributes disappear from every metric emitted by a Prometheus-receiver pipeline: net.host.name, net.host.port and http.scheme. A query or alarm that selects on one of them stops matching. The replacement is a rename, not a removal — server.address, server.port and url.scheme carry identical values on the same series and are untouched — so the migration is a one-line edit on the consumer side.

The raw-label deletions are narrower: the raw Prometheus names (pod, namespace, container and the kube-state-metrics set) remain available at datapoint scope, which is where dashboards read them. Only the duplicate resource-scope copy goes, so @resource.pod-style selectors stop resolving while the plain datapoint label continues to work.

Both are customer-visible and worth release-note treatment, but neither removes information from the pipeline.

Paired with the equivalent change in aws/amazon-cloudwatch-agent.

Testing:

Tested with helm template across the chart's flag matrix with all solutions enabled, plus the chart's flag_matrix.sh, pod_level_values.sh and yaml_document_injection.sh suites.

New integration coverage in aws/amazon-cloudwatch-agent-test (aws/amazon-cloudwatch-agent-test#776): TestDeprecatedSemconvAttributesRemoved asserts the three deprecated attributes are gone while url.scheme is still present on at least one series — the assertion that would have caught the original karpenter/keda inversion — and TestRawLabelsNotBilledTwice asserts the raw Prometheus labels exist only at datapoint scope for the cadvisor and kube-state-metrics pipelines.

Both were validated on a live EKS cluster, since the claim is about what arrives in CloudWatch rather than what the config renders to. The changed configuration was applied to the running agent and cluster scraper, the pods were restarted, and metrics were given 30 minutes to settle; test/otel/standard -computeType=EKS then asserts on datapoints read back from CloudWatch. The suite was also run before the change, because an "attribute is absent" assertion passes trivially against empty data.

Assertion Before After
TestDeprecatedSemconvAttributesRemoved 27 fail 27 pass
TestRawLabelsNotBilledTwice 18 fail 18 pass

No failures, no skips. TestDeprecatedSemconvAttributesRemoved failing before the change is the result that matters most here: it is the assertion that would have caught the original inversion, and it only passes once the current-spelling twins are the ones left in place.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

PR Checklist

  • Commits are squashed into a logical, reviewable set (one commit for a single change)
  • Commits and PR description comply with Amazon internal guidelines
  • Chart renders and lints locally
  • All GitHub Actions checks on the PR are passing
  • Integration test evidence: validated end-to-end on a live EKS cluster; baseline and post-change assertion counts are in the Testing section above. There is no public CI link because the integration suite needs a provisioned cluster and credentials that fork PRs cannot reach.
  • New or updated integration test coverage: test(otel): cover removal of deprecated and doubled collector attributes aws/amazon-cloudwatch-agent-test#776
  • New functionality has unit tests; bug fixes have a reproducing test
  • N/A — no config translation in this repo, so no golden files
  • Breaking or customer-visible changes are called out in the PR description

…butes

Two groups of resource attributes that carry no information the pipeline
is not already sending.

Deprecated semantic conventions. The Prometheus receiver injects
net.host.name, net.host.port and http.scheme on every scrape target
alongside their current-spelling twins server.address, server.port and
url.scheme. Each pair carries an identical value on the same series.
Add transform/cw_k8s_ci_v0_drop_deprecated_semconv, deleting the
deprecated form and keeping the current one, and wire it into all ten
Prometheus-receiver pipelines across both configs.

This also corrects an inversion. The karpenter and keda promote
transforms already deleted net.host.name and net.host.port, but deleted
url.scheme — the current name — while keeping http.scheme, the deprecated
one. Those three inline deletes are replaced by the shared processor, so
the two pipelines now match the other eight.

Doubled raw Prometheus labels. groupbyattrs moves each raw label to
resource scope; the promote transforms then rename it to its semantic
convention name and copy the raw name back down to the datapoint, but
never remove the resource-level copy. The result is three billed copies
of the same fact. Delete the resource-level copy in the five pipelines
that leave it behind — cadvisor, kube-state-metrics, dcgm, karpenter and
keda — bringing them in line with neuron, efa, ebs-csi and lis-csi, which
already do promote-and-delete correctly.

The datapoint copy is kept for dashboard compatibility, and the semantic
convention names are untouched, so no consumer loses a dimension. The new
delete statements sit in a trailing `context: resource` block that runs
after the datapoint copy-back, since transform statement groups execute in
declaration order.

Nothing downstream reads the raw resource copies: k8sattributes keys its
pod_association on k8s.pod.name / k8s.namespace.name / k8s.node.name,
transform/cw_k8s_ci_v0_set_workload reads the k8s.*.name forms, and
awsattributelimit keys on k8s.node.label.* and k8s.pod.label.* prefixes.
@petruanica
petruanica force-pushed the otel-ci-l1-drop-redundant-attributes branch from a1ec099 to 1ffd04f Compare October 5, 2026 13:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant