[louisall]

Description of the issue

The Helm chart adds the otelFilters block for OTel Container Insights: namespaces (aws-observability/helm-charts#393), pod and node labels (aws-observability/helm-charts#394) and metric names (aws-observability/helm-charts#395). Those PRs test how the values become the generated OTel configuration. This PR makes sure, on a real EKS cluster, that the telemetry that reaches CloudWatch matches the filter values.

The suites need the chart changes of aws-observability/helm-charts#393, aws-observability/helm-charts#394 and aws-observability/helm-charts#395, which are on main. The Terraform clones the chart from helm_chart_branch, and the default is main.

Description of changes

• terraform/eks/daemon/otel-filters, patterned on otel-attr-limit: its own cluster with one 2-node group that carries test node labels. The chart values are one otelFilters block with an include list and an exclude list for each filter, each with at least one prefix entry. A logging workload with test pod labels runs one pod per node in each of 4 namespaces, 2 kept and 2 filtered. A small DaemonSet writes a marker line to /var/log/messages on every node, because the AL2023 nodes have no host log files by default. A stand-in inference pod runs on each node in one kept and one filtered namespace. It looks like a KServe predictor with a queue-proxy, and two nginx containers serve fixed vLLM and Knative data plane metrics.
• test/otel/filters reads metrics from kube-state-metrics, cAdvisor, kubelet stats and node exporter, and the application and host logs, from the node agents and the cluster scraper:
  • Namespaces: data is present for the included namespaces and absent for the excluded ones. The dropped namespace filters-exact-other makes the test fail if the exact entry filters-exact matches as a prefix.
  • Labels: the test expects the same labels on metrics and logs. Included labels are present. Excluded labels, and the pod label filters-e2e-other outside podLabels.include, are absent. With "*" in nodeLabels.include, kubernetes.io/arch and kubernetes.io/os are present. The built-in labels kubernetes.io/hostname, node.kubernetes.io/instance-type, topology.kubernetes.io/region and topology.kubernetes.io/zone are absent. pod-template-hash is on the include list and on the built-in list, and it is present, because a named include entry wins over the built-in list. Two fixture node labels fall inside the built-in prefix karpenter.k8s.aws/instance-gpu-. The named instance-gpu-name is present and instance-gpu-count is absent, which tests transform/cw_k8s_ci_v0_metrics_labels and the logs transform. Other cases fail if the escaped key_regex of a dotted prefix does not match, or if an exact entry matches as a prefix.
  • Metric names: the included names are present, also for the dotted prefix k8s.pod.*. The excluded names, the names outside the include list, and node_load15, which starts with the exact include entry node_load1, are absent, from both agents.
  • vLLM and Knative data plane: vllm:num_requests_running and revision_request_count are present for the kept namespace and absent for the filtered one. The test fails if filter/cw_k8s_ci_v0_metrics_namespaces is missing from either pipeline or runs before the promote transform.
  • Each absence check runs only after the matching kept data arrives from every node. So a pipeline that does not report fails the test instead of passing it.
  • The test reads metrics with client.Query per name rather than queryCache. The absence checks need fresh results, and the source registry expects every metric of a source, which a metric name filter removes by design.
• terraform/eks/daemon/otel-multi-efa-filters, patterned on otel-multi-efa: its own cluster with two nodes that have two EFA cards each, and otelFilters.namespaces.exclude names efa-drop. On node 1, one efaburn pod in efa-keep and one in efa-drop use one card each. On node 2, one more efa-drop pod uses one card, and the other card is free. The chart installs only after all three pods run. The efaburn image defaults to efaburn:latest in the ECR registry of the account and region that run the test.
• test/otel/multi_efa_filters: the setup fails unless node 1 runs only the efa-keep and efa-drop pods, node 2 runs only the second efa-drop pod, and each node has two EFA devices. Each check is scoped to one node, for each of 4 EFA metrics. On node 1, one card carries the efa-keep pod and namespace, and the other card has no k8s.pod.name, k8s.namespace.name or k8s.container.name at resource or data point scope. This tests the per-card pods of groupbyattrs/cw_k8s_ci_v0_efa, and the old transform/cw_k8s_ci_v0_efa_promote fails it in either card order. On node 2, both cards are present and neither has pod attributes, so the old drop filter fails it. On both nodes, no series carries an efa-drop namespace, pod name or pod UID. Because no agent runs before every pod holds its card, series without a pod from the agent start cannot hide a dropped card on node 1. On node 2, the checks wait until the agent has run longer than the query lookback.
• terraform/eks/daemon/otel-gpu-filters, patterned on otel-gpu: its own cluster with one g4dn.12xlarge node (4 GPUs). One gpu-burn pod in gpu-keep and one in gpu-drop use one GPU each, two GPUs are free, and otelFilters.namespaces.exclude names gpu-drop. otelFilters.metricNames.exclude names DCGM_FI_DEV_SM_CLOCK and dcgm_fi_dev_gpu_temp. The chart installs only after both pods run.
• test/otel/gpu_filters: the setup fails unless both pods run on the GPU node and it has 4 GPUs. For 3 DCGM metrics, the test expects 4 GPU UUIDs. One GPU carries the gpu-keep pod and namespace at resource scope and as the raw pod and namespace at data point scope. The other three have no pod attributes at either scope, and no series carries a gpu-drop namespace, pod name or pod UID. The old drop filter fails it, and so does a transform that cleans only the resource, because transform/cw_k8s_ci_v0_dcgm_promote copies the pod keys onto every DCGM data point. Metric names are case-sensitive: DCGM_FI_DEV_SM_CLOCK is absent, and DCGM_FI_DEV_GPU_TEMP is present from all 4 GPUs.
• terraform/eks/daemon/otel-neuron-filters, patterned on otel-neuron: its own cluster with one standard node and one inf2.xlarge Neuron node (2 cores) with test node labels. One neuron-burn pod in neuron-keep and one in neuron-drop use one core each and carry test pod labels. otelFilters.namespaces.exclude names neuron-drop, and podLabels and nodeLabels each have exact, prefix and exclude entries. The chart installs only after both pods run.
• test/otel/neuron_filters: for 2 core metrics, the test expects 2 cores. One core carries the neuron-keep pod, namespace and container and every kept pod label. The other core has no pod attributes at resource or data point scope and no pod labels. Both cores carry the kept node labels. No series has a dropped label or a neuron-drop namespace, pod name or pod UID, except in aws.neuron.runtime.tag.
• Generator entries under eks_daemon: otel/filters next to otel/attr_limit, otel/multi_efa_filters next to otel/multi_efa, otel/gpu_filters (g4dn.12xlarge) next to otel/gpu, and otel/neuron_filters (inf2.xlarge) next to otel/neuron.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

• gofmt -l: no files.
• go vet -tags integration on ./test/otel/filters/, ./test/otel/multi_efa_filters/, ./test/otel/gpu_filters/, ./test/otel/neuron_filters/ and ./generator/: passes. The four test packages compile with the integration tag.
• Sample data (outside this PR): the rendered EFA processors, with groupbyattrs and transform v0.150.0, ran on one node with four cards for exclude, include and prefix lists and all 24 card orders. Every data point stayed. Only the cards of filtered pods lost their pod attributes, and k8sattributes added none back.
• All four Test Artifacts runs below applied and destroyed their Terraform configurations. The runs use helm_chart_branch=glf-dra-off-test, which is main at 57ef424 with otelContainerInsights.enableDRACorrelation: false, because the agent build that the tests use does not accept the DRA settings of awsdevicepodcorrelation yet.
• Test Artifacts run with test_repo_branch=otel-ci-filters-e2e, helm_chart_branch=glf-dra-off-test and test_dir_filter=./test/otel/filters: EKS job: all 7 tests pass (TestNamespaceFilterMetrics, TestNamespaceFilterInferenceMetrics, TestNamespaceFilterLogs, TestLabelFilterMetrics, TestLabelFilterLogs, TestLabelFilterHostLogs and TestMetricNameFilter).
• Test Artifacts run with test_repo_branch=otel-ci-filters-e2e, helm_chart_branch=glf-dra-off-test and test_dir_filter=./test/otel/multi_efa_filters: EKS job: both tests pass (TestEFANamespaceFilter and TestEFAFreeCardNamespaceFilter), each for all 4 EFA metrics.
• Test Artifacts run with test_repo_branch=otel-ci-filters-e2e, helm_chart_branch=glf-dra-off-test and test_dir_filter=./test/otel/gpu_filters: EKS job: both tests pass (TestGPUNamespaceFilter, for all 3 DCGM metrics, and TestMetricNameFilterCase).
• Test Artifacts run with test_repo_branch=otel-ci-filters-e2e, helm_chart_branch=glf-dra-off-test and test_dir_filter=./test/otel/neuron_filters: EKS job: TestNeuronFilters passes for both Neuron core metrics.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): go vet
• All GitHub Actions checks on the PR are passing
• Integration test evidence: link to a passing run: filters, multi_efa_filters, gpu_filters, neuron_filters
• New or updated integration test coverage: this PR
• New functionality has unit tests; bug fixes have a reproducing test: N/A, test-only
• Config translation changes include updated golden files: N/A
• Breaking or customer-visible changes are called out in the PR description: none

[bhavya76]

The Terraform config uses "filters-e2e.example.com/*" as a dotted prefix in the exclude list, but the include list only has "filters-e2e*" (no dots) and "kubernetes.io/os" (exact match). So the key_regex rendering path, the one with the double-escaping concern in the helm chart change, is never exercised by a presence check. If that path is broken, dotted prefix includes would silently fail to extract labels and this test wouldn't catch it.

Should we add one dotted-prefix include case?

In main.tf, add a keep label to node_labels:
+ "filters-e2e.example.com/keep-x" = "v"

In the chart values, add it to both include lists:
- include: ["filters-e2e*", "kubernetes.io/os"]
+ include: ["filters-e2e*", "filters-e2e.example.com/keep*", "kubernetes.io/os"]

And in filters_test.go:

    keptLabels = []string{
         "k8s.node.label.filters-e2e-node-keep",
  +     "k8s.node.label.filters-e2e.example.com/keep-x",
         "k8s.node.label.kubernetes.io/os",

This directly exercises the key_regex + regexQuoteMeta code path and would catch the escaping bug as a failing assertLabels call rather than needing a manual cluster test.

[petruanica]

Added, with one change: the node exclude filters-e2e.example.com/* removes that node label, so the test uses the pod label filters-e2e.example.com/keep-x with the include filters-e2e.example.com/keep*.

[bhavya76]

This only queries the application log group. The transform/cw_k8s_ci_v0_logs_labels processor is also wired into the host logs pipeline, but host log resource attributes are never checked. Should we add a small helper that queries /aws/otel/containerinsights/%s/host and asserts that k8s.node.label.filters-e2e-node-keep is present and k8s.node.label.filters-e2e-node-drop is absent?

[petruanica]

Added TestLabelFilterHostLogs. The AL2023 nodes have no /var/log/messages by default, so a small DaemonSet writes a marker line there on each node.

[bhavya76]

resources here come entirely from container_cpu_usage_seconds_total, which is a node agent metric. The cluster scraper (deployment, not daemonset) has its own k8sattributes processor instances and its own awsattributelimit, both rendered from the same template helpers. Should we add kube_pod_info to the assertLabels check, querying its resources in filters-keep-a and asserting the same kept/dropped labels? It would close the loop on the scraper side I believe.

[petruanica]

Added: the label test now also checks kube_pod_info from the cluster scraper, next to container_cpu_usage_seconds_total from the node agent.

[bhavya76]

The optional solution pipelines (DCGM, Neuron, EFA, vLLM, KServe, Knative) aren't enabled in the Terraform config, so the metric name filter's wiring into those pipelines isn't covered here.

[petruanica]

The e2e cluster runs none of those workloads, so their pipelines stay off. The chart suite asserts the metric name filter in every metrics pipeline with DCGM, Neuron, EFA, vLLM, KServe and Knative on.

[bhavya76]

does PR description need an update since our new commits?

[petruanica]

Yes, thanks. I updated it for the Neuron suite, the vLLM, Knative and metric name case tests, and the merged chart PRs. The Tests section links the passing jobs on the current head.