[miconeilaws]

Description of the issue

aws-observability/helm-charts#425 makes the amazon-cloudwatch-observability chart serve an OTLP endpoint on the node agent by default (4317 gRPC, 4318 HTTP). Workloads send to the cloudwatch-agent Service, and the chart's OTLP pipelines enrich the telemetry like Container Insights enriches the same pod, set the service identity per the OpenTelemetry Kubernetes conventions, apply otelFilters, compute span metrics, and route logs to /aws/cwagent/<cluster>/otlp. otlp.enabled: false turns it off, and default:otel becomes an alias of default.

No suite covers this today. The AKS and GKE default:otel suites send over hostNetwork to 127.0.0.1, so they never go through the Service or exercise pod association from a normal pod, and nothing checks labels, otelFilters or the identity rules on OTLP telemetry.

Description of changes

New EKS suite test/otel/otlp + terraform/eks/daemon/otel-otlp (registered in the generator). Chart defaults plus otelContainerInsights.enabled=true, two nodes. Ordinary pods (no hostNetwork, no pod attributes) post OTLP/HTTP through cloudwatch-agent.amazon-cloudwatch:4318, one workload per identity case:

Case	Sender	Expected service.name
app	Deployment with app.kubernetes.io/name and /version labels, one pod per node	the label
annotation	resource.opentelemetry.io/service.name and /service.version annotations plus app labels	the annotation
explicit	sends service.name, has an app label	the sent value
unknown	sends unknown_service:sh	the Deployment name
daemonset	DaemonSet, no identity labels	the DaemonSet name
cronjob	CronJob, no identity labels	the CronJob name

The tests check:

• the Service and the agent's host network have 4317 and 4318, and the senders' posts are accepted;
• the resource attributes of the app sender's metrics, logs and spans from each pod, against the pod and its node from the Kubernetes API (pod name and UID, node, instance ID, instance type, zone, cloud.resource_id, k8s.workload.*, service.*, deployment.environment.name, pod and node labels), that labels on the built-in exclusion list are absent, and that no internal _tmp.* or aws.log.* attribute leaks;
• scope tags cloudwatch.source=cloudwatch-agent, cloudwatch.solution=otel-otlp;
• the identity cases above;
• span metrics with the span's HTTP dimensions and the sender's resource attributes, labels included;
• the exact set of metric names with the OTLP scope for the cluster;
• logs in /aws/cwagent/<cluster>/otlp, stream <namespace>/<service.namespace>/<service.name>, including the stream of the unknown_service case;
• traces in Transaction Search (aws/spans);
• Container Insights is unchanged: its metrics for the sender pods still arrive with its own scope and without the OTLP-only attributes, and its application log group gets the senders' stdout but no OTLP log record.

New EKS suite test/otel/otlp_disabled + terraform/eks/daemon/otel-otlp-disabled (registered in the generator), following the one-terraform-directory-per-chart-configuration precedent. Same install with otlp.enabled=false: no OTLP Service or host ports, every post through the Service fails, and no metric with the OTLP scope arrives once Container Insights metrics are seen.

test/otel/filters gains an OTLP sender container in every workload, so OTLP telemetry from each namespace carries the same pod labels. New tests check that the namespace filter, the pod and node label filters and the metric name filter apply to OTLP metrics, logs and spans. metricNames.include now also names otlp_e2e_requests; the unlisted otlp_e2e_unlisted and the span metrics must be dropped.

test/azure/aks (containerinsights mode), the mode that installs the chart with the values scripts/azure/setup.sh uses: a sender through the Service, and checks of its metrics, span metrics, the OTLP metric name set, logs and traces with the AKS resource attributes. The install values are unchanged; agents[0].config=default:otel stays valid because it is now an alias of default. The raw-agent default:otel mode (test/azure/aks) and test/gcp/gke do not install the chart, so the chart change does not affect them and they are unchanged.

Shared pieces: terraform/otlp_service_sender.sh (the sender; it logs the HTTP status of every post, which the tests read) and test/otel/otlpendpoint (Kubernetes and log helpers, with unit tests). The new EKS terraform takes helm_chart_repo (precedent: otel-llm-serving) so a chart branch on a fork can be tested, and stops with a clear error when the chart has no otlp values block.

Note for reviewers: the first records a new pod sends can reach the agent before its pod cache has the pod's IP. Those records are exported without pod attributes (seen locally as a default log stream and as span metrics without k8s.*). The identity and attribute checks therefore select telemetry the agent matched to a pod, and the CronJob sender sends three rounds per run.

Related PRs and merge order

• feat(chart): serve an OTLP endpoint on the node agent by default aws-observability/helm-charts#425: the chart change these suites test. Merge this PR after kueue metrics integration tests #425 is merged and released as the new chart major: the EKS suites install the latest release-* chart branch in CI (and fail fast if it has no otlp block), and the AKS containerinsights suite installs the chart from main.
• chore(scripts): stop passing the no-op default:otel to the Kubernetes installs amazon-cloudwatch-agent#2402: removes the now no-op default:otel from the setup scripts. No order between the two: the AKS install values here keep default:otel, which is valid with and without #2402.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

• go vet -tags integration clean for every changed package; make simple-lint (license and import order) passes; go test -run=NO_MATCH ./... compiles every package; go test ./test/otel/otlpendpoint/ passes; terraform fmt -check and terraform validate pass for the four changed terraform directories; go run ./generator emits both new EKS suites.
• The sender script posts valid OTLP JSON (checked against a local JSON server) and logs 000 when the connection fails.
• Local cluster run against the kueue metrics integration tests #425 branch (minikube, OTLP endpoint only, exports fail without AWS credentials, so the pipelines were tapped with a debug exporter): the senders from otel-otlp resolved through the Service with the expected service.name for all six cases and the attributes the tests expect; with the otel-filters values only the kept namespaces, kept labels and otlp_e2e_requests came through on metrics, logs and spans; with otlp.enabled=false the Service lost 4317/4318 and every probe post returned 000, and after re-enabling the posts returned 200. The otlpendpoint helpers were run against that cluster.
• Not run yet: the EKS and AKS suites against real accounts. They need the chart from kueue metrics integration tests #425; a run will be linked here.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): make simple-lint, test compile, unit tests
• All GitHub Actions checks on the PR are passing
• Integration test evidence: not run yet, see Tests (needs the kueue metrics integration tests #425 chart and an AWS account)
• New or updated integration test coverage: this PR is the integration test coverage for feat(chart): serve an OTLP endpoint on the node agent by default aws-observability/helm-charts#425
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: kueue metrics integration tests #425 and #2402 linked with the order above; back-links pending
• New functionality has unit tests; bug fixes have a reproducing test: test/otel/otlpendpoint
• Config translation changes include updated golden files: N/A, no translator change
• Breaking or customer-visible changes are called out in the PR description: N/A, test-only change