[louisall]

Description

Adds the top-level otelFilters block with its first filter, otelFilters.namespaces. The filter has one include list and one exclude list, and the OTel Container Insights metrics and logs share them. The chart also restarts the pods of an agent when the final otelConfig of that agent changes.

• Entries: an exact namespace, a prefix that ends in *, or * alone. An empty include list, or one that contains *, includes every namespace. Exclude wins over include. A duplicate entry renders once, and an entry cannot be in both lists. Records without k8s.namespace.name always pass.
• Metrics: filter/cw_k8s_ci_v0_metrics_namespaces runs in the kube-state-metrics, cAdvisor, kubelet stats, vLLM and Knative data plane pipelines, after the namespace promotion and before k8sattributes.
• Logs: filter/cw_k8s_ci_v0_logs_namespaces runs in the application logs pipeline, before k8sattributes.
• Devices: the DCGM, Neuron and EFA pipelines never drop data for a namespace, because a device belongs to a pod only for a while. In these pipelines, transform/cw_k8s_ci_v0_device_namespaces runs in the same place, with the same conditions. When the pod of a device is in a filtered namespace, it deletes k8s.pod.name, k8s.container.name and k8s.namespace.name from the resource, and the raw pod, namespace and container copies from the DCGM datapoints. So the device keeps its metrics with its node attributes only, and k8sattributes finds no pod for it. The Neuron runtime tag stays.
• EFA: groupbyattrs/cw_k8s_ci_v0_efa moves the pod, namespace and container of each card to one resource per pod, before the device transform. On main it runs only when otelContainerInsights.enableDRACorrelation is true. This PR runs it always and removes transform/cw_k8s_ci_v0_efa_promote, which copied the pod of one card onto the resource of the whole node. When the group step runs first, as on main by default, efa_promote finds no pod keys on the datapoints and does nothing. The datapoints of a card without a pod keep the node resource.
• Not filtered: node exporter, EBS CSI, LIS CSI, the API server, Karpenter, KEDA, the KServe and Knative control planes, and host logs. Their records have no namespace, or the namespace belongs to the controller and not to a workload.
• Restart: the agent reads its configuration only at startup, and the hash of the operator does not include otelConfig. So the chart adds the pod annotation cloudwatch-agent/otel-config-sha256 to each agent that has an otelConfig. Its value is the SHA-256 of the final otelConfig, after the merge with agents[].otelConfig. When that configuration changes, the operator rolls the pods of that agent. The annotation joins the podAnnotations of the agent. cloudwatch-agent.modify-otel-config returns unquoted YAML, so the hash covers the same text that the custom resource gets.
• The validation and the translation are in the new templates/_otel-container-insights-filters.tpl, which validate-flags calls. helm install fails, with a message that names the setting, for a wrong shape or an unknown key under otelFilters (so a typo such as excludes fails), an entry that is not a DNS-1123 label, an entry in both lists, otelContainerInsights.filters (with a pointer to otelFilters), and podAnnotations that are not a map.

Customer-visible: otelFilters.namespaces is a new opt-in value. With default values and Container Insights off, the rendered agent configurations are the same as on main. With Container Insights on, there are two changes. The cloudwatch-agent and cloudwatch-agent-cluster-scraper agents get the restart annotation, so the first upgrade to this chart restarts their pods once. After that, every change to the final otelConfig of an agent restarts its pods. Also, with otelContainerInsights.enableDRACorrelation=false, the EFA metrics of each card now carry the pod and namespace of the pod that uses that card, and a card without a pod carries none. Before, with that setting, every card on a node carried the pod of one card. With the default (true), the EFA metrics are the same as on main. With a namespace filter, the GPU, Neuron and EFA metrics of a pod in a filtered namespace stay, without the pod, namespace and container attributes.

Testing

• tests/otel_ci_filters.sh (make helm-otel-ci-filters): 67 new cases, all pass. They cover the rendered processors and their place in every pipeline, the restart annotation and every validation error.
• All ten chart test targets pass: helm-lint, helm-flag-matrix, helm-yaml-injection, helm-neuron-shape, helm-kserve-shape, helm-vllm-shape, helm-knative-shape, helm-otel-ci-filters, helm-availability-zone and helm-dcgm-metrics-csv.
• Default render against main: the same with Container Insights off. With it on, only the restart annotation changes, and transform/cw_k8s_ci_v0_efa_promote is gone from the EFA pipeline. A scratch render check (not committed) also covers otelContainerInsights.enableDRACorrelation=false and dcgmExporter.enableDRACorrelation=false, with and without a namespace filter.
• Sample data (scratch test, not committed): the rendered DCGM, Neuron and EFA processors kept every datapoint. Only devices of pods in a filtered namespace lost their pod attributes, and k8sattributes added none back.
• EKS integration test: Add OTel Container Insights filters EKS integration test aws/amazon-cloudwatch-agent-test#782.

This is the first of three stacked PRs: this PR, then #394 (pod and node label filters), then #395 (metric name filters). aws/amazon-cloudwatch-agent-test#782 merges after all three.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint)
• All GitHub Actions checks on the PR are passing
• Integration test evidence: link to a passing run, or state N/A with the reason — all Minikube and EKS integration jobs pass on ee8d7af: https://github.com/aws-observability/helm-charts/actions/runs/39225085075 (Minikube scenarios needed re-runs for the known helm-install hang, see ci: retry hung Minikube scenarios and capture cluster state on failure #404)
• New or updated integration test coverage: link to the corresponding amazon-cloudwatch-agent-test PR: Add OTel Container Insights filters EKS integration test aws/amazon-cloudwatch-agent-test#782
• New functionality has unit tests; bug fixes have a reproducing test
• Config translation changes include updated golden files: render tests assert the exact generated processors
• Breaking or customer-visible changes are called out in the PR description

[lorespiz]

Since this branched, main added metrics/cw_k8s_ci_v0_vllm (#398) and metrics/cw_k8s_ci_v0_knative_dataplane (#401). Both set resource.attributes["k8s.namespace.name"] from the pod's namespace label, so they need filter/cw_k8s_ci_v0_metrics_namespaces like the other workload pipelines

[lorespiz]

^([a-z0-9][-a-z0-9]*\*?|\*)$ lets an exact name end in a hyphen, which no namespace can, if it's used as include rule, it wouldn't match anything

[petruanica]

Fixed: an exact name must now end in a letter or digit, so team- fails the render with a clear message. A prefix such as team-* still works.

[EduVencovsky]

values.yaml lists EFA among the "workload metrics" filtered by namespace and says "Records without a Kubernetes namespace are never filtered."

helm-charts/charts/amazon-cloudwatch-observability/values.yaml

Lines 1263 to 1270 in 149e7b3

	## Records without a Kubernetes namespace are never filtered.
	filters:
	metrics:
	## Namespaces of workload metrics: every metric with a Kubernetes
	## namespace (kube-state-metrics, cAdvisor, kubelet, GPU, Neuron, EFA,
	## vLLM, Knative data plane). Node-level metrics and the Karpenter, KEDA,
	## KServe control plane and Knative control plane metrics are not
	## filtered by namespace.

But efa_promote runs per datapoint and writes each card's namespace onto the resource the whole node shares, so the last card's namespace wins.

helm-charts/charts/amazon-cloudwatch-observability/templates/linux/_otel-container-insights-config.tpl

Lines 997 to 1007 in 149e7b3

	transform/cw_k8s_ci_v0_efa_promote:
	error_mode: ignore
	metric_statements:
	- context: datapoint
	statements:
	- set(resource.attributes["k8s.pod.name"], attributes["k8s.pod.name"]) where attributes["k8s.pod.name"] != nil
	- set(resource.attributes["k8s.namespace.name"], attributes["k8s.namespace.name"]) where attributes["k8s.namespace.name"] != nil
	- set(resource.attributes["k8s.container.name"], attributes["k8s.container.name"]) where attributes["k8s.container.name"] != nil
	- delete_key(attributes, "k8s.pod.name") where attributes["k8s.pod.name"] != nil
	- delete_key(attributes, "k8s.namespace.name") where attributes["k8s.namespace.name"] != nil
	- delete_key(attributes, "k8s.container.name") where attributes["k8s.container.name"] != nil

As a result, this filter judges every EFA card on the node by that one namespace. Cards used by pods in other namespaces, and cards used by no pod, are kept or dropped along with it. Was the single-namespace EFA attribution intentional? If not, splitting by pod with groupbyattrs the way Neuron does would fix both.

[petruanica]

Thanks, fixed. The EFA pipeline now groups datapoints by pod with groupbyattrs, as Neuron does, so each card keeps its own pod and namespace when the filter runs.

[petruanica]

Both now apply the namespace filter: metrics/cw_k8s_ci_v0_vllm and metrics/cw_k8s_ci_v0_knative_dataplane. The Knative control-plane pipeline does not, because its namespace is the controller's, not a workload's.