Repository navigation
test(otel/pernode): per-node, CRD-management and scraper-routing E2E suites - #773
Aakash-Dantre wants to merge 1 commit into
Conversation
6a4a17f to
41fb73b
Compare
| } | ||
|
|
||
| // Core per-node invariant: scraped by the agent on the pod's own node. | ||
| if targetNode != agentNode { |
There was a problem hiding this comment.
I don't think this can detect cross-node scraping. The Prometheus receiver sets resource k8s.node.name from the scraped target's discovery labels (__meta_kubernetes_pod_node_name; see addKubernetesResource in prometheusreceiver/internal/prom_to_otlp.go), and nothing in the prometheuscr pipeline overwrites it. target_node comes from the same label via the relabel in workload.yaml, so the two are always equal, wherever the series was scraped. The comment on agentNodeResourceKey in metrics_test.go (that it comes from ${env:K8S_NODE_NAME}) doesn't match the pipeline either.
Could we assert allocation at the TA instead? /jobs/<job>/targets?collector_id=<agent pod> returns each collector's targets. Comparing each target's __meta_kubernetes_pod_node_name with that agent pod's spec.nodeName from the K8s API tests the allocator directly, and the /jobs probe pod you already have can do it.
There was a problem hiding this comment.
You're right, thanks. In a432c10 it checks allocation at the TA as you suggested: each target's pod node against the assigned agent pod's spec.nodeName. Verified on a live cluster: passes with per-node, fails when switched to consistent-hashing.
| done | ||
| kubectl -n amazon-cloudwatch patch AmazonCloudWatchAgent cloudwatch-agent --type='json' \ | ||
| -p='[{"op": "replace", "path": "/spec/image", "value": "${var.cwagent_image_repo}:${var.cwagent_image_tag}"}]' | ||
| kubectl -n amazon-cloudwatch patch AmazonCloudWatchAgent cloudwatch-agent --type=merge \ |
There was a problem hiding this comment.
The harness forces allocationStrategy: per-node and the TA image by patching the CR after install, so the suite would still pass if the chart's own per-node wiring regressed. aws-observability/helm-charts#376 already renders per-node by default, and the TA image is a chart value (agent.prometheus.targetAllocator.image.repositoryDomainMap.public / .repository / .tag). The comment at line 196 saying it isn't one is out of date. Could the harness set these through helm_release set entries and drop the CR patch, so the E2E covers the chart as shipped?
There was a problem hiding this comment.
Done in a432c10: images and prometheusScrape are set through helm_release values, no CR patch. I'll run the Terraform harness end to end before asking for re-review.
a432c10 to
f18f0d3
Compare
f18f0d3 to
de58fdb
Compare
de58fdb to
10b63c6
Compare
Aakash-Dantre
changed the title
|
In |
9f1514d to
7d2149a
Compare
…E2E suites
Adds the EKS end-to-end coverage for the Target Allocator ServiceMonitor/PodMonitor
work: per-node allocation (asserted at the Target Allocator: every assigned target
is on its agent's node), zero-step CRDs created by the operator, Target Allocator
health and monitor discovery, and annotation-based routing of monitors between the
per-node and cluster-scraper agents.
test/otel/pernode/ per_node, crd_management, monitor_config,
ta_resilience, scraper_routing, metrics + k8s helpers
terraform/eks/daemon/otel-pernode/ EKS harness for the suite
The CRD suite asserts that the operator (--manage-prometheus-crds) created the
ServiceMonitor and PodMonitor CRDs: its crd-managed-by label, crd-applied-version
and crd-applied-generation annotations, the operator as the only spec field
manager, none of the markers other install tools leave (app.kubernetes.io/managed-by,
meta.helm.sh/release-name, kubectl.kubernetes.io/last-applied-configuration), and no
owner reference, so uninstall leaves them in place. It also checks that the operator
emitted a PrometheusCRDCreated Event for each CRD and no PrometheusCRDNotManaged Event,
which proves the operator recognizes the CRDs it just created as its own.
A config suite checks that both agents' SM/PM pipeline sets cloudwatch.solution
k8s-otel-prometheus-monitors and cloudwatch.pipeline prometheus-monitors, and, when the
harness runs with otel_container_insights_enabled=false, that it is the only pipeline
each agent runs. A metrics check queries CloudWatch and asserts that every ServiceMonitor
and PodMonitor workload series is stored with those two scope attributes. The harness
enables otelPrometheusMonitor.enabled.
The suites are tagged `integration` and are not yet registered in the generated
test matrix, so they do not run in CI until a matrix entry is added.
The routing /jobs probe presents the agents' Target Allocator client certificate,
because the Target Allocator's HTTPS server requires and verifies one.
The harness configures the operator and Target Allocator images and
prometheusScrape through chart values rather than patching the agent CR after
install, so the suite exercises the chart's own wiring. The workload step waits
for the operator-created CRDs to exist before waiting for them to be Established.
7d2149a to
40d306b
Compare
Summary
End-to-end coverage for the Target Allocator ServiceMonitor/PodMonitor work: per-node allocation, zero-step ServiceMonitor/PodMonitor CRDs created by the operator, Target Allocator health and monitor discovery, and annotation-based routing of monitors between the per-node and cluster-scraper agents.
Supersedes #720 and #724, combined and re-applied onto current
main. #724 as it stands was committed from an older tree, so it also deletes 77 files that exist onmain(including the Azure VM/AKS suites andtest/e2e/containerinsights) and rolls back later changes to others. This PR carries only the new files and is purely additive: 15 files, +1932/-0.What
test/otel/pernode/per_node,crd_management,monitor_config,ta_resilience,scraper_routing,metricstests + k8s helpers and workload manifeststerraform/eks/daemon/otel-pernode/otelPrometheusMonitor.enabledare set through chart values (otel_container_insights_enabledturns Container Insights on or off), so the suite exercises the chart as shipped (no post-install CR patch). The harness never installs the SM/PM CRDs; the operator must create them.All suites carry the
integrationbuild tag.TestPerNodeAllocation,TestPerNodeCoverageAcrossNodesTestServiceMonitorPodMonitorCRDsCreatedByOperatorcloudwatch.aws.amazon.com/crd-managed-bylabel, matchingcrd-applied-versionandcrd-applied-generationannotations, the operator as the onlyspecfield manager, none of the markers other install tools leave (app.kubernetes.io/managed-by,meta.helm.sh/release-name,kubectl.kubernetes.io/last-applied-configuration), and no owner reference (so uninstall leaves them in place); the operator emitted aPrometheusCRDCreatedEvent for each and noPrometheusCRDNotManagedEvent, so it recognizes the CRDs it just created as its ownTestPrometheusMonitorMetricsCarrySolutioncloudwatch.solution=k8s-otel-prometheus-monitorsandcloudwatch.pipeline=prometheus-monitorsTestPrometheusMonitorPipelineConfigcloudwatch.solution=k8s-otel-prometheus-monitorsandcloudwatch.pipeline=prometheus-monitors; withotel_container_insights_enabled=falseit is the only pipeline each agent runsTestOperatorManagesPrometheusCRDs--manage-prometheus-crdsTestTargetAllocatorHealthyOnZeroStepInstall,TestTargetAllocatorDiscoversMonitorsTestScraperRoleWiringscraperRole=cluster-scraper; the per-node agent carries the default roleTestClusterScraperTargetAllocatorHealthyTestAnnotationRoutingPartition/jobs, the annotated monitor is owned only by the cluster-scraper TA and the unannotated one only by the per-node TARunning
Through the harness (see
test/otel/pernode/README.mdfor the variables):Or against a cluster with the operator and chart changes below deployed:
Not yet wired into CI
The suites are not registered in
generator/or any workflow, so they do not run in CI as submitted. A matrix entry for an EKS target is the remaining follow-up.Dependencies and merge order
Exercises these, which must be deployed first:
--manage-prometheus-crds)Merge after #435 and #406, and run against images and a chart that include them. The public operator and Target Allocator images do not have these changes yet, so the harness takes custom images.
PR Checklist
Commits are squashed into a logical, reviewable set (one commit for a single change) - one commit.
Commits and PR description comply with Amazon internal guidelines - checked.
makepasses locally (build, unit tests, lint) - this repo has nomaketarget for test packages.go vet -tags integration ./test/otel/pernode/is clean, andterraform fmt -checkandterraform validatepass for the harness.All GitHub Actions checks on the PR are passing - workflow runs on fork PRs need a maintainer to approve them.
Integration test evidence - the harness, run locally with Terraform on fresh EKS 1.35 clusters, passed 12/12 tests twice with this commit's tests (
TestPrometheusMonitorMetricsCarrySolutionthen gained a log line and was rerun on both clusters), with 0 Target Allocator restarts and metrics from both monitor types reaching CloudWatch. It used operator and Target Allocator images built from operatormainat5710ae66(which includes feat(crdmanager): create ServiceMonitor and PodMonitor CRDs when missing amazon-cloudwatch-agent-operator#435) and the feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator aws-observability/helm-charts#406 chart atc1cdf70:otel_container_insights_enabled=false: each agent ran only the SM/PM pipeline;otel_container_insights_enabled=true: withotelContainerInsights.enableDRACorrelation=falseadded locally, becausemain's DRA correlation keys are not accepted by the published agent images yet (unrelated to these PRs).In both runs CloudWatch returned the sm-app and pm-app series with
cloudwatch.solution=k8s-otel-prometheus-monitorsandcloudwatch.pipeline=prometheus-monitors.Negative controls on an earlier run: after a server-side apply under another field manager,
TestServiceMonitorPodMonitorCRDsCreatedByOperatorfails on the extraspecfield manager; after addingapp.kubernetes.io/managed-by=Helmto the ServiceMonitor CRD, it fails on the marker and on the operator'sPrometheusCRDNotManagedEvent. Not run in CI (not in the test matrix, see above).New or updated integration test coverage - this PR is the integration test coverage for the operator and chart PRs above.
Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated - feat(crdmanager): create ServiceMonitor and PodMonitor CRDs when missing amazon-cloudwatch-agent-operator#435 and feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator aws-observability/helm-charts#406 link back; merge order above.
New functionality has unit tests; behaviour changes have a reproducing test - N/A: test-only change.
Config translation changes include updated golden files - N/A: no translator changes.
Breaking or customer-visible changes are called out in the PR description - none. Test-only and purely additive; no existing test is modified or removed.