[amazon-cloudwatch-agent-operator] Route SM/PM by cloudwatch.aws/scraper annotation by wenegiemepraise · Pull Request #399 · aws/amazon-cloudwatch-agent-operator

wenegiemepraise · 2026-07-15T11:34:43Z

Summary

Route ServiceMonitor/PodMonitor discovery across CloudWatch agents by the
cloudwatch.aws/scraper annotation on the monitor CR. A monitor annotated
cloudwatch.aws/scraper: cluster-scraper is scraped only by the cluster-scraper agent's Target
Allocator; all others only by the per-node agent's.

What

New spec.targetAllocator.prometheusCR.scraperRole field (Target Allocator config scraper_role).
Client-side annotation filter in the TA watcher (annotationRoleMatches) applied during monitor
discovery in LoadConfig. cluster-scraper role keeps only annotated monitors; the empty
(default) role keeps only unannotated ones — complementary, so every monitor is owned by exactly
one agent (no double-scrape, no gap).

Why annotation (not label)

The routing key is behavioural config, not identity. Both agents' TAs already list all monitors
into their cache and filter client-side, and the two roles' selections union to the full set — so
server-side label filtering would gain nothing here. Filtering by annotation is a few lines in the
existing ListAll callback.

Testing

go build ./... and unit tests pass, incl. new TestAnnotationRoleMatches (8 cases + partition invariant).
Live-verified end to end on an EKS cluster: an annotated PodMonitor appeared only in the
cluster-scraper TA /jobs; an unannotated one only in the per-node TA /jobs.
Local build uses GOPROXY=direct GOSUMDB=off (offline proxy); CI is the authoritative gate.

Dependencies / stacking

Stacks on the per-node allocation operator work (#398, ta-per-node-allocation). Companion helm PR:
[amazon-cloudwatch-observability] Route SM/PM to cluster-scraper by annotation.

musa-asad · 2026-07-21T15:43:22Z

+	// not so annotated. This lets a heavy/singleton monitor be routed to the central cluster-scraper
+	// agent while all others stay on the per-node agent.
+	// +optional
+	ScraperRole string `json:"scraperRole,omitempty"`


scraperRole has no enum validation, so a typo like cluster-scaper quietly falls back to the default role and that monitor gets scraped by nobody. The chart hardcodes the right value today, but could we add +kubebuilder:validation:Enum=cluster-scraper and regenerate the CRD?

musa-asad · 2026-07-21T15:43:23Z

+	if !annotationRoleMatches(w.scraperRole, annotations) {
+		return false
+	}
+	if w.scraperRole == clusterScraperRole {


selectsMonitor logs an Info line for every monitor the cluster-scraper claims, and LoadConfig reruns on each reconcile, so a big annotated set means a lot of repeated lines. Low risk, but could we log only on membership change or drop it to V(1)?

musa-asad · 2026-07-21T15:43:24Z

+// TestLoadConfigScraperRouting exercises the annotation filter through the real LoadConfig path
+// (matching TestLoadConfig's harness): an annotated ServiceMonitor is discovered by the
+// cluster-scraper role and excluded by the default role.
+func TestLoadConfigScraperRouting(t *testing.T) {


The unannotated and PodMonitor cases never actually go through LoadConfig even though PodMonitor discovery mirrors ServiceMonitor, and the informer wait is a HasSynced spin loop with no deadline that hangs until CI kills it. Could we add both LoadConfig cases and use cache.WaitForCacheSync with a timeout?

musa-asad · 2026-07-21T15:43:25Z

+// Allocator, based on its scraperRole. cluster-scraper role selects only monitors annotated
+// cloudwatch.aws/scraper: cluster-scraper; the default role (empty) selects only monitors that are
+// not so annotated, so the two roles partition monitors with no overlap and no gap.
+func annotationRoleMatches(scraperRole string, annotations map[string]string) bool {


Routing is binary here (only cluster-scraper routes, everything else falls to per-node), so a future gpu-scraper would silently land on per-node. Could we add a note that annotationRoleMatches needs to become an explicit role to value match before a third role shows up?

musa-asad · 2026-07-21T15:43:26Z

 	}

+	if len(params.OtelCol.Spec.TargetAllocator.PrometheusCR.ScraperRole) > 0 {
+		taConfig["scraper_role"] = params.OtelCol.Spec.TargetAllocator.PrometheusCR.ScraperRole


The operator writes the scraper_role key, which an older Target Allocator image rejects as unknown and then CrashLoopBackOffs on, taking the cluster-scraper path down. Could we make sure the chart pins the operator and Target Allocator images from this commit or newer together?

Aakash-Dantre · 2026-09-23T10:58:20Z

Checklist posted for the PR author to move into the PR description under a ## PR Checklist heading.

PR Checklist

Commits are squashed into a logical, reviewable set — one commit for this PR's own work; the three inherited commits belong to the unmerged Fix Target Allocator startup crashes, PrometheusCR watcher, and Prometheus config pod restart #386 and are left attributed to their author
Commits and PR description comply with Amazon internal guidelines
make passes locally — go build ./... clean, make manifests idempotent (CRD regenerated), make impi and make checklicense pass, unit tests pass
All GitHub Actions checks on the PR are passing
Integration test evidence: link to a passing run, or state N/A with the reason
New or updated integration test coverage: [amazon-cloudwatch-agent-test] Add scraper routing integration tests amazon-cloudwatch-agent-test#724 (scraper_routing_test.go)
New functionality has unit tests; bug fixes have a reproducing test
Config translation changes include updated golden files — N/A, no translator changes
Breaking or customer-visible changes are called out in the PR description

Note for the description: scraperRole is constrained by +kubebuilder:validation:Enum=cluster-scraper
with no empty member, so the default (per-node) role can only be expressed by omitting the field —
setting scraperRole: "" explicitly is rejected by the API server. Worth documenting, or widening
the enum to include "".

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399.

…er startup The target-allocator declared the enable-prometheus-cr-watcher flag name as a constant but never registered it on the flag set, while the operator passes --enable-prometheus-cr-watcher whenever PrometheusCR.enabled is true. Because args are parsed with pflag.ExitOnError, the unregistered flag caused the binary to print 'unknown flag' and exit(2), putting the target-allocator pod into CrashLoopBackOff. This change registers the flag and ORs it with the YAML prometheus_cr.enabled setting, then fixes three latent defects that were previously unreachable because the binary crashed first: - promOperator: set a non-empty Namespace on the synthetic Prometheus object so the prometheus-operator config generator no longer panics with 'namespace can't be empty' in store.ForNamespace. - promOperator: set EvaluationInterval so the generated config does not render an empty global.evaluation_interval, which the prometheus config parser rejects with 'empty duration string'. - main: create and register service-discovery metrics and pass them to discovery.NewManager; passing a nil sdMetrics map makes every SD provider fail to register, yielding zero discovered targets. RELEASE_NOTES updated. (cherry picked from commit 1376451)

Add a regression test asserting that loading a Target Allocator config whose static scrape job omits scrape_protocols still yields a non-empty ScrapeProtocols on every loaded scrape config. This is defaulted by the pinned Prometheus library during yaml.UnmarshalStrict into the prometheus Config type, so the distributed /scrape_configs payload is never empty and the agent's prometheus-receiver validation passes. The test fails fast if a future dependency or load-path change drops this defaulting. (cherry picked from commit 0405f4d)

The pod-template restart-trigger sha256 was computed from Spec.Config only, so a change to Spec.Prometheus (rendered into a separate ConfigMap) left the pod template byte-identical and the workload controller did not roll the pods. Fold the serialized Spec.Prometheus (PrometheusConfig.Yaml()) into the hash input when it is non-empty, so a Prometheus-only change bumps the pod-template annotation and triggers a rolling restart, matching agent-config behavior. When no Prometheus config is set the hash input is byte-identical to the agent config alone, leaving non-Prometheus agents unaffected. (cherry picked from commit d61d693)

Adds a per-node allocation strategy so each CloudWatch Agent scrapes only the ServiceMonitor/PodMonitor targets on its own node, eliminating cross-node and cross-AZ scrape traffic. Targets with no resolvable node fall back to consistent-hashing so nothing is silently dropped, and unassigned targets are surfaced via a gauge. Also changes collector registration for ALL strategies, including the existing consistent-hashing default: the initial List now skips pods with an empty spec.NodeName, watch.Modified is handled so a pod scheduled after Add is picked up, and pods are dropped as soon as they carry a DeletionTimestamp rather than on Deleted. This releases a terminating agent's targets immediately instead of at the end of its grace period, at the cost of more target churn during rollouts. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. (cherry picked from commit 1595a87)

Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. (cherry picked from commit cd3e40b)

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399. otelContainerInsights.prometheusScrape.enabled defaults to false until operator and Target Allocator images with prometheusCR discovery, per-node allocation and scraper_role are released; enabling it requires pinning those images together. The per-node prometheusCR pipeline applies the same unit normalization as the cluster-scraper one, so a monitor's units do not depend on which agent scrapes it.

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws.amazon.com/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399. otelContainerInsights.prometheusScrape.enabled defaults to false until operator and Target Allocator images with prometheusCR discovery, per-node allocation and scraper_role are released; enabling it requires pinning those images together. The per-node prometheusCR pipeline applies the same unit normalization as the cluster-scraper one, so a monitor's units do not depend on which agent scrapes it.

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws.amazon.com/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399. otelContainerInsights.prometheusScrape.enabled defaults to false until operator and Target Allocator images with prometheusCR discovery, per-node allocation and scraper_role are released; enabling it requires pinning those images together. The per-node prometheusCR pipeline applies the same unit normalization as the cluster-scraper one, so a monitor's units do not depend on which agent scrapes it. otelContainerInsights.prometheusScrape.crds.install defaults to never: the ServiceMonitor/PodMonitor CRDs are a prerequisite (installed by prometheus-operator, kube-prometheus-stack, or the two kubectl apply commands in values.yaml), and the Target Allocator starts discovery once they exist. auto and always remain available for direct helm install/upgrade, where the skip-if-exists lookup and helm.sh/resource-policy: keep apply. Rendered without cluster access (helm template, Argo CD, Flux, the EKS add-on), lookup returns nothing, so the chart does not install the CRDs by default.

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws.amazon.com/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399 and #435. otelContainerInsights.prometheusScrape.enabled defaults to false until operator and Target Allocator images with prometheusCR discovery, per-node allocation and scraper_role are released; enabling it requires pinning those images together. The per-node prometheusCR pipeline applies the same unit normalization as the cluster-scraper one, so a monitor's units do not depend on which agent scrapes it. The chart does not install the ServiceMonitor/PodMonitor CRDs. When prometheusScrape is enabled, it starts the operator with --manage-prometheus-crds and grants it customresourcedefinitions create/get/list/watch, so the operator creates the CRDs only if they are missing (amazon-cloudwatch-agent-operator#435). A copy installed by prometheus-operator, kube-prometheus-stack, Argo CD or kubectl is left unchanged, and the operator never deletes the CRDs, so uninstalling the chart or the EKS add-on leaves them and every ServiceMonitor/PodMonitor in place.

…PrometheusMonitor Adds a top-level otelPrometheusMonitor.enabled switch (default false) that scrapes the targets selected by ServiceMonitor and PodMonitor objects (monitoring.coreos.com/v1) through the Target Allocator's prometheusCR discovery and sends them to CloudWatch over OTLP. It works with otelContainerInsights on or off. Monitors annotated cloudwatch.aws.amazon.com/scraper: cluster-scraper are scraped by the cluster-scraper agent, which gets its own Target Allocator with scraperRole: cluster-scraper and consistent-hashing allocation. The per-node agent keeps the default (empty) role and per-node allocation. The allocation strategies are set internally. With otelContainerInsights off, each agent's config is the SM/PM pipeline alone, cut from the same template that defines it for otelContainerInsights, so the shared processor, exporter and extension definitions are identical either way and no other otelContainerInsights pipeline renders. The cluster-scraper agent renders when either feature is on. The SM/PM series carry cloudwatch.solution k8s-otel-prometheus-monitors and cloudwatch.pipeline prometheus-monitors, and both pipelines apply the same unit normalization, so a monitor's units do not depend on which agent scrapes it. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399 and #435. otelPrometheusMonitor.enabled defaults to false until operator and Target Allocator images with prometheusCR discovery, per-node allocation, scraper_role and --manage-prometheus-crds are released; enabling it requires pinning those images together. The chart does not install the ServiceMonitor/PodMonitor CRDs. When otelPrometheusMonitor is enabled, it starts the operator with --manage-prometheus-crds and grants it customresourcedefinitions create/get/list/watch, so the operator creates the CRDs only if they are missing (amazon-cloudwatch-agent-operator#435). A copy installed by prometheus-operator, kube-prometheus-stack, Argo CD or kubectl is left unchanged, and the operator never deletes the CRDs, so uninstalling the chart or the EKS add-on leaves them and every ServiceMonitor/PodMonitor in place.

This was referenced Jul 15, 2026

[amazon-cloudwatch-observability] Route SM/PM to cluster-scraper by annotation aws-observability/helm-charts#339

Open

[amazon-cloudwatch-agent-test] Add scraper routing integration tests aws/amazon-cloudwatch-agent-test#724

Open

musa-asad reviewed Jul 21, 2026

View reviewed changes

musa-asad previously approved these changes Jul 24, 2026

View reviewed changes

wenegiemepraise dismissed musa-asad’s stale review via 6950def September 4, 2026 10:28

Aakash-Dantre force-pushed the ta-annotation-scraper-routing branch 2 times, most recently from 329e93f to e4fe2d5 Compare September 23, 2026 12:41

Aakash-Dantre force-pushed the ta-annotation-scraper-routing branch from e4fe2d5 to cd3e40b Compare September 23, 2026 12:51

musa-asad and others added 5 commits September 25, 2026 15:37

Aakash-Dantre force-pushed the ta-annotation-scraper-routing branch from cd3e40b to afa7000 Compare September 25, 2026 14:39

Aakash-Dantre mentioned this pull request Oct 1, 2026

feat(target-allocator): per-node allocation and annotation-based SM/PM routing #415

Merged

8 of 9 tasks

miconeilaws closed this Oct 7, 2026

Navigation Menu

Repository navigation

[amazon-cloudwatch-agent-operator] Route SM/PM by cloudwatch.aws/scraper annotation - #399

[amazon-cloudwatch-agent-operator] Route SM/PM by cloudwatch.aws/scraper annotation#399
wenegiemepraise wants to merge 5 commits into
aws:mainfrom
spanaik:ta-annotation-scraper-routing

wenegiemepraise commented Jul 15, 2026

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

wenegiemepraise commented Jul 15, 2026

Summary

What

Why annotation (not label)

Testing

Dependencies / stacking

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aakash-Dantre commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading