[petruanica]

Description of the issue

The eks_daemon:otel_standard_test job has two problems in terraform/eks/daemon/otel/main.tf.

First, the Karpenter node sends no node-agent metrics. The EC2NodeClass selects the security group by the karpenter.sh/discovery tag. The job puts this tag on the shared VPC security group. The managed nodes have only the EKS cluster security group, and that group accepts traffic only from its own members. The Karpenter node therefore cannot reach the managed nodes. The node agent uses host networking with the ClusterFirstWithHostNet DNS policy, and CoreDNS runs on the managed nodes. On the Karpenter node, the agent cannot resolve the CloudWatch endpoint and exports nothing. Its readiness probe is local, so the pod stays Ready. As a result, TestKarpenterInstanceLabelsRemoved and TestKarpenterGroupingLabelsRetained fail for every node-agent metric. The kube-state-metrics subtests pass, because the cluster scraper runs on a managed node.

The shared tag also has a second problem. Parallel CI jobs tag the one shared security group, each with its own cluster name. So the last job to apply wins, and the destroy of one job removes the tag for the others. The subnet tags keep this problem, and a follow-up handles them.

Second, the job hides suite failures. The null_resource.validator step runs three suites in one local-exec command: test/otel/standard, test/otel/solutions/karpenter and test/otel/solutions/keda. The command returns the exit status of its last command, which is the KEDA go test. When the standard suite or the Karpenter suite fails, the job still reports success. For example, on main the standard suite prints FAIL github.com/aws/amazon-cloudwatch-agent-test/test/otel/standard for the two Karpenter tests above, and the job is green.

Description of changes

• The EC2NodeClass selects the EKS cluster security group by ID: - id: "${aws_eks_cluster.this.vpc_config[0].cluster_security_group_id}". The Karpenter node then has the same security group as the managed nodes. The aws_ec2_tag.sg_karpenter resource is deleted, because nothing reads that tag anymore. The subnet selector and its karpenter.sh/discovery subnet tags do not change.
• The validator command sets rc=0, appends || rc=1 to each of the three go test commands and ends with exit $rc. All three suites still run. If any of them fails, the step fails. The command uses POSIX shell only, because local-exec runs it with /bin/sh. The sleeps and the test flags do not change.

The two changes are in one PR. Without the security group change, the validator change turns the job red for the two Karpenter tests.

Customer-visible. None. This change affects only the CI terraform of the EKS OTel daemon tests.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

• terraform fmt -check -recursive terraform/eks/daemon/otel and make check-fmt in terraform/ pass.
• terraform init -backend=false and terraform validate pass in terraform/eks/daemon/otel.
• The karpenter_nodepool command, rendered with terraform console, passes sh -n, and its two YAML documents parse. The EC2NodeClass has securityGroupSelectorTerms: [{id: <cluster security group ID>}] and the unchanged subnet selector.
• The validator command, rendered with terraform console, passes sh -n. It ran with stub go and sleep commands. With no failing suite, it exits 0. With the standard suite, the Karpenter suite, the KEDA suite, or the standard and KEDA suites failing, it exits 1. In each case, all three go test commands run. When the standard suite fails, the command on main exits 0.
• make compile passes.
• Test Artifacts run 42682310261 of ./test/otel/standard on this commit: the eks_daemon:otel_standard_test job passes. TestKarpenterInstanceLabelsRemoved and TestKarpenterGroupingLabelsRetained pass, with all 22 subtests each, including the 18 node-agent metrics. The standard, Karpenter and KEDA suites print ok, and no test fails.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): make compile and the terraform format check pass. This PR changes no Go code.
• All GitHub Actions checks on the PR are passing
• Integration test evidence: the Test Artifacts run of ./test/otel/standard is under Tests.
• New or updated integration test coverage: N/A, this PR fixes the test environment and how the CI step reports the existing suites. The existing Karpenter label tests in test/otel/standard cover it.
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: N/A, this PR has no related PRs and no merge order.
• New functionality has unit tests; bug fixes have a reproducing test: the Karpenter label tests in test/otel/standard fail on main and pass with this change. The stub run under Tests reproduces the green job with a failing suite.
• Config translation changes include updated golden files: N/A, this PR changes no config translation.
• Breaking or customer-visible changes are called out in the PR description: none, see Customer-visible.