[amazon-cloudwatch-observability] feat: add ServiceMonitor/PodMonitor v2 OTLP scraping path by wenegiemepraise · Pull Request #329 · aws-observability/helm-charts

wenegiemepraise · 2026-07-09T10:33:28Z

Summary

Add the ServiceMonitor/PodMonitor v2 OTLP scraping path to the chart: a prometheuscr
receiver + pipeline driven by the Target Allocator's PrometheusCR discovery, plus the
enablement flags and gating helper. This is the foundation that lets the CloudWatch Agent
scrape community monitoring.coreos.com ServiceMonitor/PodMonitor CRs into CloudWatch via
the OTEL Container Insights path.

What this enables

When otelContainerInsights.enabled is set, the chart wires the Target Allocator +
prometheuscr receiver so SM/PM CRs are discovered and scraped (default consistent-hashing
allocation). ServiceMonitor and PodMonitor discovery can each be toggled via
otelContainerInsights.serviceMonitor.enabled / podMonitor.enabled (default on).

Scope

prometheuscr receiver + metrics pipeline in the OTEL Container Insights config
otelCIScrape gating helper
otelContainerInsights.serviceMonitor/podMonitor.enabled values (only take effect when
otelContainerInsights.enabled)
Read-only clusterrole additions for the discovery path

This is the base of a small stack; per-node allocation + node-enrichment transforms land
in a follow-up that builds on this path.

Testing

helm template:

otelContainerInsights.enabled=true → the prometheuscr receiver/pipeline renders.
default (disabled) → the path does not render (0 references).

helm lint clean (pre-existing fluent-bit-windows region warning is unrelated).

musa-asad · 2026-07-21T15:34:13Z

        - "--auto-instrumentation-nodejs-image={{ template "auto-instrumentation-nodejs.image" . }}"
        - "--target-allocator-image={{ template "target-allocator.image" (merge .Values.agent.prometheus.targetAllocator.image (dict "region" $.Values.region)) }}"
-        - "--feature-gates=operator.autoinstrumentation.multiinstrumentation,operator.autoinstrumentation.multiinstrumentation.skipcontainervalidation"
+        - "--feature-gates=operator.autoinstrumentation.multi-instrumentation,operator.autoinstrumentation.multi-instrumentation.skip-container-validation"


These --feature-gates values are hyphenated (multi-instrumentation), but the pinned operator only registers the unhyphenated forms, so it rejects the gate and CrashLoopBackOffs and nothing reconciles. This also looks out of scope here. Can we restore the unhyphenated line?

musa-asad · 2026-07-21T15:34:14Z

+  ## Allocator's prometheusCR discovery. Defaults to true: when
+  ## otelContainerInsights is enabled, ServiceMonitor scraping is on unless
+  ## explicitly disabled here. Only takes effect when otelContainerInsights.enabled is true.
+  serviceMonitor:


serviceMonitor.enabled and podMonitor.enabled default to true, so the moment someone turns on otelContainerInsights they silently get prometheusCR discovery, whose CRD and RBAC this PR doesn't set up yet. Could we default them to false, or gate the path on the CRDs being present?

The default-on behavior is intentional, the product goal is that enabling otelContainerInsights gives you working SM/PM scraping out of the box, and the surrounding work is built to make that safe: the operator's CRD-resilience change(#394) makes missing CRDs non-fatal (no crash/loop), and CRD bundling (#331) puts the CRDs in-cluster. So I'd like to keep SM/PM default-on rather than invert to opt-in.

musa-asad · 2026-07-21T15:34:16Z

          - set(attributes["cloudwatch.solution"], "k8s-otel-container-insights")
          - set(attributes["cloudwatch.pipeline"], "efa")

+  transform/cw_k8s_ci_v0_set_scope_prometheuscr:


The scraping receiver and pipeline are guarded on serviceMonitor.enabled/podMonitor.enabled, but the transform/cw_k8s_ci_v0_set_scope_prometheuscr processor isn't, so turning both monitors off leaves it orphaned and the collector warns. Can we guard it the same way via otelCIScrapeEnabled?

musa-asad · 2026-07-21T15:34:17Z

 {{- if .Values.agent.enabled }}
 {{- range $i, $customAgent := .Values.agents }}
-{{- if and (and (hasKey ($customAgent.prometheus) "targetAllocator") (hasKey ($customAgent.prometheus.targetAllocator) "enabled")) $customAgent.prometheus.targetAllocator.enabled }}
+{{- $otelCIScrape := eq (include "cloudwatch-agent.otelCIScrapeEnabled" (dict "agentName" $customAgent.name "context" $)) "true" }}


This renders the ClusterRole inside range .Values.agents with a static name, so two matching agents produce duplicate ClusterRoles and Kubernetes just keeps the last write. Could we namespace the name per agent (same for the ClusterRoleBinding), or document that only one agent is supported?

musa-asad · 2026-07-21T15:34:18Z

+{{- define "cloudwatch-agent.otelCIScrapeEnabled" -}}
+{{- $ctx := .context -}}
+{{- $agentName := .agentName -}}
+{{- if and $ctx.Values.otelContainerInsights.enabled (eq $agentName $ctx.Values.otelContainerInsights.targetAgent) (or $ctx.Values.otelContainerInsights.serviceMonitor.enabled $ctx.Values.otelContainerInsights.podMonitor.enabled) -}}


This helper reads otelContainerInsights.serviceMonitor.enabled directly. Fine today since the PR ships those keys as defaults, but it goes nil if a user replaces the whole map, so a dig with a default fallback might be safer.

Since the subkeys ship as chart defaults and Helm deep-merges them, they're present under any partial override, so this is safe as-is I'll leave the direct access

musa-asad · 2026-07-21T15:34:19Z

+  prometheus/cw_k8s_ci_v0_prometheuscr:
+    target_allocator:
+      endpoint: https://{{ .Values.otelContainerInsights.targetAgent }}-target-allocator-service:80
+      interval: {{ .Values.otelContainerInsights.metricResolution }}


interval here is the collector to Target Allocator refresh cadence, but it reuses otelContainerInsights.metricResolution, which is really the scrape cadence (prometheusCR.scrapeInterval). Worth a comment or a separate field so the two can diverge later?

Agreed they're distinct knobs. I'd rather not add a second field preemptively, but the shared value being non-obvious is fair

wenegiemepraise · 2026-07-21T17:51:54Z

Makes sense. The pinned cloudwatch-agent-target-allocator:1.0.0 predates the discovery path, and enabling scraping against it CrashLoops the TA rather than degrading, so a discovery-capable TA image must be pinned before this path is enabled (landing in the same release that exposes it, or the path stays off until then).

…node-allocation Re-sync aws-observability#330 onto the updated scraping-path branch: inherit the feature-gate ID fix, co-gated CRD RBAC render-once ClusterRole/Binding, the $$$1 comment correction, and the prometheuscr scope-processor guard, alongside aws-observability#330's per-node work.

…bundling Re-sync onto the updated scraping-path branch to inherit the feature-gate ID fix, the co-gated customresourcedefinitions RBAC rule (render-once ClusterRole/ Binding), the $$$1 comment correction, and the prometheuscr scope-processor guard. Resolves the CRD-RBAC matrix assertion (now 11/0) and the feature-gate blocker on this branch.

…bundling Re-sync aws-observability#331 onto aws-observability#330 so the stack is linear (aws-observability#329 -> aws-observability#330 -> aws-observability#331): inherit the per-node allocationStrategy scoping/gating, the SM/PM node-stamping removal, and the context-aware allocationStrategy default, on top of aws-observability#331's CRD bundling.

…cture Re-sync aws-observability#332 onto the updated stack (aws-observability#329->aws-observability#330->aws-observability#331). Resolutions favor the settled older-PR fixes applied onto aws-observability#332's prometheusScrape.* restructure: - prometheusCRDsEnabled: keep aws-observability#331's explicit auto branch + invalid-value fail, gated on aws-observability#332's otelContainerInsights.enabled && prometheusScrape.enabled. - allocationStrategy: use aws-observability#332's precomputed $allocationStrategy (per-node on the otelCI path via prometheusScrape.allocationStrategy, consistent-hashing for direct-TA) -- supersedes aws-observability#330's ternary while preserving its intent. - Fix a stale old-key guard (otelContainerInsights.serviceMonitor.enabled) left by auto-merge in the prometheuscr scope-processor to prometheusScrape.enabled. - Keep aws-observability#330's SM/PM node-stamping removal; port aws-observability#331's GitOps opt-out caveat into the new prometheusScrape.crds doc. Verified: lint clean, CRD matrix 21/0, flag matrix 8/8.

movence

Every other Container Insights metrics pipeline runs clear_schema_url, set_cloud_resource_id, and awsattributelimit before batch, but this prometheuscr one skips all three. That means ServiceMonitor/PodMonitor metrics ship uncapped and without the schema-url and cloud-resource-id normalization the rest of the pipelines get. Could we add those same three processors here?

(_otel-container-insights-config.tpl, prometheuscr pipeline ~L863)

wenegiemepraise · 2026-08-13T21:06:40Z

Fixed the issue around: clear_schema_url, set_cloud_resource_id, and awsattributelimit in the latest commit

For the issue with the looping over the raw .Values.agents I dealt with it in PR no.339: #339

movence · 2026-08-14T13:33:44Z

+*/}}
+{{- $needsTA := false }}
+{{- $needsCR := false }}
 {{- range $i, $customAgent := .Values.agents }}


This loops over the raw .Values.agents entries, but the CR template merges the shared agent defaults first (mergeOverwrite (deepCopy .Values.agent) at cloudwatch-agent-custom-resource.yaml:67). So if someone turns on the target allocator or prometheusCR through the shared agent block, the CR gets created but this RBAC does not, and the TA's CRD informer ends up denied at runtime. Can we merge .Values.agent here the same way?

…node-allocation Picks up aws-observability#329's latest commits (TA RBAC via the singular agent block, and the normalization/cap processors on the prometheuscr pipeline) plus origin/main. Conflicts resolved by keeping this branch's per-node allocationStrategy ternary and adopting main's quoted 'include target-allocator.image ... | quote'.

…bundling Brings in aws-observability#330, aws-observability#329's latest commits and origin/main. The values.yaml conflict was two non-overlapping additions at the end of the file: main's nested otelContainerInsights.solutions block and this branch's top-level prometheusCRDs block. Both kept, with solutions left inside otelContainerInsights and prometheusCRDs at top level.

…cture Completes the chain: brings in aws-observability#331, aws-observability#330 and aws-observability#329's latest commits. Conflicts resolved in favour of this branch's restructure, which is its purpose: the $allocationStrategy variable form is kept, and the top-level prometheusCRDs block is intentionally NOT re-added since this PR moves it to otelContainerInsights.prometheusScrape.crds.

…ion-scraper-routing Brings in the full chain (aws-observability#332, aws-observability#331, aws-observability#330, aws-observability#329) and origin/main. The cluster-scraper OTEL config conflicted structurally: main added the Karpenter/KEDA solution pipelines (aws-observability#325, aws-observability#345) in the same receivers / processors / pipelines sections where this branch adds its prometheuscr blocks, and git interleaved the {{- if }}/{{- end }} boundaries. Rather than untangle the markers, the file was reconstructed from main's version with this branch's three self-contained, individually-gated prometheuscr blocks re-inserted at their anchors (after each kubeStateMetrics block). Verified: all five cluster-scraper pipelines render (prometheuscr, karpenter, keda, apiserver, kube_state_metrics), scraperRole is still set only on the cluster-scraper, and the TA ClusterRole/Binding still render exactly once.

Aakash-Dantre · 2026-09-23T10:58:42Z

Checklist posted for the PR author to move into the PR description under a ## PR Checklist heading.

PR Checklist

Commits are squashed into a logical, reviewable set — single commit, authored by the PR author
Commits and PR description comply with Amazon internal guidelines
make passes locally — helm lint clean, helm template renders, tests/prometheus_crds_matrix.sh 30 passed / 0 failed
All GitHub Actions checks on the PR are passing
Integration test evidence: link to a passing run, or state N/A with the reason
New or updated integration test coverage: [test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness aws/amazon-cloudwatch-agent-test#720 (per_node_test.go, crd_bundling_test.go)
New functionality has unit tests; bug fixes have a reproducing test — chart render tests under tests/
Config translation changes include updated golden files — N/A, chart-only change
Breaking or customer-visible changes are called out in the PR description

Adds the prometheuscr receiver and metrics pipeline driven by the Target Allocator's PrometheusCR discovery, the otelCIScrape gating helper, the serviceMonitor/podMonitor enablement values, and the read-only RBAC the discovery path needs (including the CRD-watch grant, and rendering the Target Allocator ClusterRole/Binding once rather than per agent). Verified: helm lint clean; helm template renders the prometheuscr receiver/pipeline when otelContainerInsights.enabled=true and omits it by default.

musa-asad reviewed Jul 21, 2026

View reviewed changes

wenegiemepraise mentioned this pull request Jul 21, 2026

feat(observability): grant Target Allocator CRD watch RBAC #327

Open

musa-asad previously approved these changes Jul 24, 2026

View reviewed changes

movence reviewed Aug 13, 2026

View reviewed changes

This comment was marked as duplicate.

Sign in to view

wenegiemepraise dismissed musa-asad’s stale review via c98b417 August 13, 2026 21:03

wenegiemepraise requested review from movence and musa-asad August 14, 2026 08:37

movence reviewed Aug 14, 2026

View reviewed changes

wenegiemepraise requested a review from movence August 14, 2026 16:08

Aakash-Dantre force-pushed the otel-ci-smpm-scrape-path branch from b18ffd2 to 651229b Compare September 23, 2026 12:28

Aakash-Dantre force-pushed the otel-ci-smpm-scrape-path branch from 651229b to 7e17bbd Compare September 23, 2026 12:42

Aakash-Dantre mentioned this pull request Oct 1, 2026

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #376

Closed

8 of 9 tasks

Aakash-Dantre mentioned this pull request Oct 20, 2026

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #406

Open

10 tasks done

Conversation

wenegiemepraise commented Jul 9, 2026

Summary

What this enables

Scope

Testing

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise commented Jul 21, 2026

Uh oh!

movence left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

wenegiemepraise commented Aug 13, 2026

Uh oh!

movence Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Aakash-Dantre commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading