[amazon-cloudwatch-observability] Route SM/PM to cluster-scraper by annotation by wenegiemepraise · Pull Request #339 · aws-observability/helm-charts

wenegiemepraise · 2026-07-15T11:35:58Z

Summary

Let a ServiceMonitor/PodMonitor be routed to the central cluster-scraper agent (instead of the default per-node agent) via the cloudwatch.aws/scraper: cluster-scraper annotation on the monitor CR.

What

Give the clusterScraperAgent its own Target Allocator + prometheusCR (otelCIScrapeEnabled now covers it) with scraperRole: cluster-scraper and a prometheuscr receiver/pipeline in its OTEL config. The per-node agent keeps the default (empty) role.
Render the TA ClusterRole/ClusterRoleBinding once (all TAs share the SA target-allocator-service-acct), so a second TA-bearing agent no longer produces a duplicate-named object.
Restore the monitor-type disable sentinel selectors; per-agent allocationStrategy (targetAgent per-node, clusterScraper consistent-hashing). Extend the CRD (scraperRole) and the render-test matrix.

Note: separate commit

This branch also contains fix(observability): use non-hyphenated operator feature-gate identifiers — needed on this base for the operator to start. It is likely a no-op on current main (main already uses the non-hyphenated IDs); drop that commit if this PR's base already has them.

Testing

helm lint clean; tests/prometheus_crds_matrix.sh 23 pass / 1 pre-existing fail (the CRD-RBAC line belongs to a separate CRD-watch RBAC change not on this base). So it goes green when the base also contains feat(observability): grant Target Allocator CRD watch RBAC #327's CRD-watch RBAC (i.e. once both the helm stack and feat(observability): grant Target Allocator CRD watch RBAC #327 land on main). On this isolated stack it can't pass.
Live-verified: annotated PodMonitor scraped only by the cluster-scraper TA; unannotated only by the per-node TA.

Dependencies / stacking

Stacks on the helm scraping stack (top = #332, helm-flag-restructure). Companion operator PR: aws/amazon-cloudwatch-agent-operator#399.

musa-asad · 2026-07-21T15:39:25Z

          - set(attributes["cloudwatch.solution"], "k8s-otel-container-insights")
          - set(attributes["cloudwatch.pipeline"], "efa")

+  transform/cw_k8s_ci_v0_set_scope_prometheuscr:


The prometheuscr processor is declared in processors: but only its pipeline is behind the prometheusScrape gate, so with scrape off it renders with zero pipelines and the collector rejects the unused component and won't start. Could we wrap the processor in that same gate?

musa-asad · 2026-07-21T15:39:25Z

                          Default: "30s"
                        format: duration
                        type: string
+                      scraperRole:


scraperRole and the new per-node allocationStrategy only land in the crds/ copy Helm doesn't update on upgrade, so the API server prunes them from the rendered CR and routing silently fails. Could we move this CRD into templates/ with helm.sh/resource-policy: keep?

musa-asad · 2026-07-21T15:39:26Z

+  annotations:
+    helm.sh/resource-policy: keep
+    controller-gen.kubebuilder.io/version: v0.20.1
+    operator.prometheus.io/version: 0.91.0


This CRD pins operator.prometheus.io/version: 0.91.0 but the operator go.mod is on v0.92.0, so the API server prunes any monitor field added between the two. Could we vendor both CRDs at v0.92.0 to match? Same in servicemonitor-crd.yaml.

musa-asad · 2026-07-21T15:39:28Z


+{{- if dig "prometheusScrape" "enabled" true .Values.otelContainerInsights }}
+  # ServiceMonitor/PodMonitor scraping routed to the cluster-scraper via the
+  # cloudwatch.aws/scraper: cluster-scraper label. This agent's Target Allocator serves the


This comment says routing goes by the cloudwatch.aws/scraper: cluster-scraper label, but the operator reads it as an annotation, so anyone who follows it sets a label and gets no routing. Could we say annotation here to match the others?

musa-asad · 2026-07-21T15:39:28Z

+
+printf "\n${Y}== Target Allocator CRD RBAC ==${N}\n"
+# The TA needs customresourcedefinitions get;list;watch on the prometheus-CR path.
+expect_count "CRD RBAC present when otelCI on"       "$CRD_RBAC" 1 --set otelContainerInsights.enabled=true


The CRD RBAC assertions on lines 76 to 78 expect customresourcedefinitions RBAC from a separate branch, so this branch renders zero and set -euo pipefail fails the suite. Could we move them into that branch or guard them until the rule exists?

musa-asad · 2026-07-21T15:39:29Z

  {{ end }}
  - name: K8S_CLUSTER_NAME
    value: {{ $.Values.clusterName | quote }}
  {{- if $.Values.otelContainerInsights.enabled }}


POD_NAME is injected under the otelContainerInsights.enabled guard, so every agent gets it even though only the scraper agents' prometheusCR receivers use ${env:POD_NAME}. It's harmless, but could we scope the guard to $otelCIScrape so it lives where it's consumed?

musa-asad · 2026-07-21T15:39:30Z

+{{- if or (and (hasKey ($customAgent.prometheus) "targetAllocator") (hasKey ($customAgent.prometheus.targetAllocator) "enabled") $customAgent.prometheus.targetAllocator.enabled) $otelCIScrape }}
+{{- $needsTA = true }}
+{{- end }}
+{{- if or (and (hasKey ($customAgent.prometheus) "targetAllocator") (hasKey ($customAgent.prometheus.targetAllocator) "prometheusCR") $customAgent.prometheus.targetAllocator.prometheusCR.enabled) $otelCIScrape }}


$needsCR keys on prometheusCR.enabled or $otelCIScrape but, unlike $needsTA, doesn't also require targetAllocator.enabled, so a stale prometheusCR.enabled on a disabled agent over grants the monitoring.coreos.com read rule. Could we mirror $needsTA here to keep it least privilege?

musa-asad · 2026-07-21T15:39:31Z

+{{- define "cloudwatch-agent.otelCIScrapeEnabled" -}}
+{{- $ctx := .context -}}
+{{- $agentName := .agentName -}}
+{{- if and $ctx.Values.otelContainerInsights.enabled (dig "prometheusScrape" "enabled" true $ctx.Values.otelContainerInsights) (or (eq $agentName $ctx.Values.otelContainerInsights.targetAgent) (eq $agentName $ctx.Values.otelContainerInsights.clusterScraperAgent)) -}}


otelCIScrapeEnabled keys on prometheusScrape.enabled alone, so with serviceMonitor.enabled=false and podMonitor.enabled=false it still renders an idle target-allocator that bundles CRDs but discovers nothing. Could we short circuit or reject that both false case?

musa-asad · 2026-07-21T15:39:32Z

    verbs: ["get"]
-  {{- if and (hasKey ($customAgent.prometheus.targetAllocator) "prometheusCR") $customAgent.prometheus.targetAllocator.prometheusCR.enabled }}
+  {{- if $needsCR }}
  - apiGroups: [ "monitoring.coreos.com"]


This branch's TA ClusterRole has the SM/PM rule under $needsCR but not the customresourcedefinitions rule, which is in #327, so #327 needs to land alongside this for the target-allocator to list CRDs. Could we make sure they merge together so discovery can start?

…ion-scraper-routing Re-sync aws-observability#339 (scraper routing) onto the full updated stack. Preserve aws-observability#339's annotation-routing feature (scraperRole: cluster-scraper) while adopting the older/settled fixes: the prometheusScrape restructure, legacy-key aliases + deprecation notice, the serviceMonitorEnabled/podMonitorEnabled helpers for the SM/PM selector guards, the feature-gate/CRD-RBAC/render-once/v0.92.0-CRD fixes, and per-node scoping. Verified: lint clean, CRD matrix 24/0, flag matrix 8/8, scraperRole renders, legacy prometheusCRDs.install alias honored.

Aakash-Dantre · 2026-09-23T10:58:47Z

Checklist posted for the PR author to move into the PR description under a ## PR Checklist heading.

PR Checklist

Commits are squashed into a logical, reviewable set — single commit, authored by the PR author
Commits and PR description comply with Amazon internal guidelines
make passes locally — helm lint clean, helm template renders, tests/prometheus_crds_matrix.sh 30 passed / 0 failed
All GitHub Actions checks on the PR are passing
Integration test evidence: link to a passing run, or state N/A with the reason
New or updated integration test coverage: [amazon-cloudwatch-agent-test] Add scraper routing integration tests aws/amazon-cloudwatch-agent-test#724 (scraper_routing_test.go)
New functionality has unit tests; bug fixes have a reproducing test — chart render tests under tests/
Config translation changes include updated golden files — N/A, chart-only change
Breaking or customer-visible changes are called out in the PR description

Notes for the description: (1) the bundled AmazonCloudWatchAgent CRD now carries
enum: [cluster-scraper] on scraperRole, matching the operator-generated schema — previously the
chart shipped it as a bare string so the constraint was unenforced on-cluster. (2) The commit
fix(observability): use non-hyphenated operator feature-gate identifiers is unrelated to
annotation routing and would be cleaner as its own PR.

Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages.

Routes a ServiceMonitor/PodMonitor to the central cluster-scraper agent via the cloudwatch.aws/scraper: cluster-scraper annotation on the monitor CR. The cluster-scraper gets its own Target Allocator and prometheusCR with scraperRole: cluster-scraper and consistent-hashing allocation; the per-node agent keeps the default (empty) role and per-node allocation. The Target Allocator ClusterRole/ClusterRoleBinding render once (all Target Allocators share the target-allocator-service-acct service account), so a second Target-Allocator-bearing agent no longer produces a duplicate-named object. The bundled CRD's scraperRole field carries the enum: [cluster-scraper] constraint generated by the operator. Companion: aws/amazon-cloudwatch-agent-operator#399.

Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. (cherry picked from commit cd3e40b)

…per annotation Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws.amazon.com/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. (cherry picked from commit cd3e40b)

…per annotation Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws.amazon.com/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages.

…M routing (#415) * feat(target-allocator): add per-node allocation strategy Adds a per-node allocation strategy so each CloudWatch Agent scrapes only the ServiceMonitor/PodMonitor targets on its own node, eliminating cross-node and cross-AZ scrape traffic. Targets with no resolvable node fall back to consistent-hashing so every target is still scraped, and unassigned targets are surfaced via a gauge. Also changes collector registration for ALL strategies, including the existing consistent-hashing default: the initial List now skips pods with an empty spec.NodeName, watch.Modified is handled so a pod scheduled after Add is picked up, and pods are dropped as soon as they carry a DeletionTimestamp rather than on Deleted. This releases a terminating agent's targets immediately instead of at the end of its grace period, at the cost of more target churn during rollouts. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. Agent pods are tracked so that a node's targets never stay with an agent that can't scrape them. A pod counts as a collector only when it is scheduled, not being deleted, Running and Ready; the initial List and the watch events share that check, and a pod that stops meeting it is removed. Every watch session starts from a fresh List that replaces the collector set and resumes from its resourceVersion, so a pod deleted while no watch was open does not stay registered. * feat(target-allocator): route SM/PM by cloudwatch.aws.amazon.com/scraper annotation Partitions ServiceMonitor/PodMonitor discovery across CloudWatch Agents by the cloudwatch.aws.amazon.com/scraper annotation on the monitor CR. A monitor annotated cluster-scraper is scraped only by the cluster-scraper agent's Target Allocator; all others only by the per-node agent's. The two roles are complementary, so every monitor is owned by exactly one agent with no double-scrape and no gap. Adds spec.targetAllocator.prometheusCR.scraperRole (Target Allocator config key scraper_role) and a client-side annotation filter applied during monitor discovery in LoadConfig. Note: scraperRole is constrained by an enum with no empty member, so the default per-node role is expressed by omitting the field; setting scraperRole: "" explicitly is rejected by the API server. Companion: aws-observability/helm-charts#339. Verified: go build ./... clean; make impi and make checklicense pass; unit tests pass across the target-allocator and manifests packages. * feat(webhook): require daemonset mode for per-node allocation per-node assigns each target to the collector on the target's node, so it needs one collector per node. Reject allocationStrategy per-node unless mode is daemonset, as upstream does. Also stop warning about running the Target Allocator outside a StatefulSet for the two supported setups: per-node on a DaemonSet, and an agent with a prometheusCR scraperRole. Other non-StatefulSet setups still warn. --------- Co-authored-by: Praise Wenegieme <praisewenegieme29@gmail.com>

musa-asad previously approved these changes Jul 21, 2026

View reviewed changes

musa-asad self-requested a review July 21, 2026 06:52

musa-asad reviewed Jul 21, 2026

View reviewed changes

wenegiemepraise dismissed musa-asad’s stale review via 2093fc8 July 23, 2026 09:57

musa-asad previously approved these changes Jul 24, 2026

View reviewed changes

wenegiemepraise mentioned this pull request Aug 13, 2026

[amazon-cloudwatch-observability] feat: add ServiceMonitor/PodMonitor v2 OTLP scraping path #329

Open

wenegiemepraise dismissed musa-asad’s stale review via f51b930 September 4, 2026 10:28

Aakash-Dantre force-pushed the helm-annotation-scraper-routing branch from a44b845 to 4711e91 Compare September 23, 2026 12:28

Aakash-Dantre force-pushed the helm-annotation-scraper-routing branch from 4711e91 to 13cf24d Compare September 23, 2026 12:42

Aakash-Dantre force-pushed the helm-annotation-scraper-routing branch from 13cf24d to b6edae9 Compare September 23, 2026 12:51

Aakash-Dantre force-pushed the helm-annotation-scraper-routing branch from b6edae9 to 8129142 Compare September 25, 2026 14:34

This was referenced Sep 25, 2026

CI verification only: 339 build/lint (do not merge) Aakash-Dantre/helm-charts#1

Open

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #376

Closed

Aakash-Dantre mentioned this pull request Oct 20, 2026

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #406

Open

10 tasks done

Navigation Menu

Repository navigation

[amazon-cloudwatch-observability] Route SM/PM to cluster-scraper by annotation - #339

[amazon-cloudwatch-observability] Route SM/PM to cluster-scraper by annotation#339
wenegiemepraise wants to merge 1 commit into
aws-observability:mainfrom
wenegiemepraise:helm-annotation-scraper-routing

wenegiemepraise commented Jul 15, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

musa-asad Jul 21, 2026 •

edited

Loading

Uh oh!

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wenegiemepraise commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What

Note: separate commit

Testing

Dependencies / stacking

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aakash-Dantre commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wenegiemepraise commented Jul 15, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading