[amazon-cloudwatch-observability] feat: bundle ServiceMonitor/PodMonitor CRDs (zero-step install) by wenegiemepraise · Pull Request #331 · aws-observability/helm-charts

wenegiemepraise · 2026-07-09T11:21:52Z

Summary

Bundle the community monitoring.coreos.com ServiceMonitor/PodMonitor CRDs in the chart
as templated, gated resources — so customers who don't already run prometheus-operator get
them with no manual CRD prerequisite (matching the prometheus-operator experience). This
completes the "zero-step install" of the SM/PM scraping path.

templates/prometheus-crds/{servicemonitor,podmonitor}-crd.yaml — the CRD schemas
(vendored/pinned), templated + gated, and skipped when an unmanaged copy already
exists (Helm lookup) so a prometheus-operator-managed CRD is never clobbered. Annotated
helm.sh/resource-policy: keep so uninstall does not cascade-delete the SM/PM CRs.
values.yaml — prometheusCRDs.install (auto|always|never, default auto); auto
bundles only when otelContainerInsights is enabled.
_helpers.tpl — prometheusCRDsEnabled gating helper.
tests/prometheus_crds_matrix.sh — render tests for the gating.

The Target Allocator CRD-watch RBAC is intentionally not here — it ships in #327.

Dependencies

Part of the SM/PM scraping stack; depends on #329 (v2 OTLP scraping path) and #330
(per-node). Until those merge, this PR's diff includes them. Reviewable bottom-up.

Testing

helm template:

otelContainerInsights.enabled=true → 2 CRD manifests render.
default (disabled) → 0 CRD manifests render (correctly gated).

(The skip-if-exists lookup is a no-op under helm template/--dry-run by design; it takes
effect on real install/upgrade.)

musa-asad · 2026-07-21T15:36:54Z

        - "--auto-instrumentation-nodejs-image={{ template "auto-instrumentation-nodejs.image" . }}"
        - "--target-allocator-image={{ template "target-allocator.image" (merge .Values.agent.prometheus.targetAllocator.image (dict "region" $.Values.region)) }}"
-        - "--feature-gates=operator.autoinstrumentation.multiinstrumentation,operator.autoinstrumentation.multiinstrumentation.skipcontainervalidation"
+        - "--feature-gates=operator.autoinstrumentation.multi-instrumentation,operator.autoinstrumentation.multi-instrumentation.skip-container-validation"


Same --feature-gates issue as #329: hyphenated multi-instrumentation IDs the pinned operator doesn't register, so it CrashLoopBackOffs. Since it comes from #329, could we fix it there and rebase?

musa-asad · 2026-07-21T15:36:55Z

@@ -0,0 +1,1442 @@
+{{- /*
+ServiceMonitor CRD (monitoring.coreos.com/v1), pinned to prometheus-operator v0.91.0 to


The header pins these CRDs to prometheus operator v0.91.0, but go.mod is actually on v0.92.0, so the API server prunes any field added between the two from a user's CR. Could we refetch both CRDs from the v0.92.0 tag (or fix the header)? Same in podmonitor-crd.yaml.

musa-asad · 2026-07-21T15:36:56Z

+*/ -}}
+{{- if eq (include "amazon-cloudwatch-observability.prometheusCRDsEnabled" .) "true" }}
+{{- $crdName := "servicemonitors.monitoring.coreos.com" }}
+{{- $existing := lookup "apiextensions.k8s.io/v1" "CustomResourceDefinition" "" $crdName }}


lookup is empty under helm template, so the bundled CRD always applies and can downgrade a GitOps user's newer schema, and the adoption check only reads meta.helm.sh/release-name. Could we add a clean opt out and also match meta.helm.sh/release-namespace so a cluster scoped CRD can't get cross adopted?

musa-asad · 2026-07-21T15:36:57Z

+
+printf "\n${Y}== Target Allocator CRD RBAC ==${N}\n"
+# The TA needs customresourcedefinitions get;list;watch on the prometheus-CR path.
+expect_count "CRD RBAC present when otelCI on"  "$CRD_RBAC" 1 --set otelContainerInsights.enabled=true


The CRD RBAC assertion on line 73 expects customresourcedefinitions, but that RBAC lands in a separate PR so this branch renders zero, and set -euo pipefail fails the whole suite right here. Could we move those two assertions into the RBAC PR, or guard them until the rule exists?

The CRD_RBAC assertions are correct, they only fail because this branch hasn't re-synced onto #329 yet, where the CRD watch RBAC now lives

musa-asad · 2026-07-21T15:36:58Z

@@ -0,0 +1,78 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.


This new script uses bash only features but has no shebang, and the SM_CRD/PM_CRD grep patterns are unanchored with unescaped dots that can over count. Could we add #!/usr/bin/env bash and anchor the patterns with ^/$ and escaped dots?

musa-asad · 2026-07-21T15:36:59Z

+Returns the string "true" when CRDs should be rendered, empty otherwise.
+*/}}
+{{- define "amazon-cloudwatch-observability.prometheusCRDsEnabled" -}}
+{{- $install := "auto" -}}


prometheusCRDsEnabled only matches always and never exactly, so anything else falls through and bundles the CRDs, meaning a typo like nevr silently bundles when otelCI is on. Could we add a fail branch for anything that isn't auto, always, or never, like the sibling key already does?

split auto into its own branch and added a catch-all else that fails with prometheusCRDs.install must be one of "auto", "always", or "never", got: , matching the metricResolution sibling.

musa-asad · 2026-07-21T15:36:59Z

+## annotated helm.sh/resource-policy: keep so `helm uninstall` does NOT delete
+## them (which would cascade-delete every ServiceMonitor/PodMonitor in the
+## cluster).
+prometheusCRDs:


This adds prometheusCRDs.install at the top level, but a later PR moves it under otelContainerInsights.prometheusScrape.crds.install with no alias, so someone who sets never now has it silently ignored after upgrading. Could we hold off calling it stable, or add an alias in that later PR?

I'll ship an alias when #332 relocates it: otelContainerInsights.prometheusScrape.crds.install will fall back to the legacy prometheusCRDs.install and emit a deprecation warning, so a user's never keeps working across the upgrade

musa-asad · 2026-07-21T15:37:01Z

  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
-  {{- if and (hasKey ($customAgent.prometheus.targetAllocator) "prometheusCR") $customAgent.prometheus.targetAllocator.prometheusCR.enabled }}
+  {{- if or (and (hasKey ($customAgent.prometheus) "targetAllocator") (hasKey ($customAgent.prometheus.targetAllocator) "prometheusCR") $customAgent.prometheus.targetAllocator.prometheusCR.enabled) $otelCIScrape }}


This branch's TA ClusterRole has the SM/PM rule but not a customresourcedefinitions rule, which lives in #327, so on its own the target-allocator can't list CRDs to start discovery. Worth making sure #327 lands alongside this so they gate together.

handled on #329

…bundling Re-sync aws-observability#331 onto aws-observability#330 so the stack is linear (aws-observability#329 -> aws-observability#330 -> aws-observability#331): inherit the per-node allocationStrategy scoping/gating, the SM/PM node-stamping removal, and the context-aware allocationStrategy default, on top of aws-observability#331's CRD bundling.

…cture Re-sync aws-observability#332 onto the updated stack (aws-observability#329->aws-observability#330->aws-observability#331). Resolutions favor the settled older-PR fixes applied onto aws-observability#332's prometheusScrape.* restructure: - prometheusCRDsEnabled: keep aws-observability#331's explicit auto branch + invalid-value fail, gated on aws-observability#332's otelContainerInsights.enabled && prometheusScrape.enabled. - allocationStrategy: use aws-observability#332's precomputed $allocationStrategy (per-node on the otelCI path via prometheusScrape.allocationStrategy, consistent-hashing for direct-TA) -- supersedes aws-observability#330's ternary while preserving its intent. - Fix a stale old-key guard (otelContainerInsights.serviceMonitor.enabled) left by auto-merge in the prometheuscr scope-processor to prometheusScrape.enabled. - Keep aws-observability#330's SM/PM node-stamping removal; port aws-observability#331's GitOps opt-out caveat into the new prometheusScrape.crds doc. Verified: lint clean, CRD matrix 21/0, flag matrix 8/8.

…cture Completes the chain: brings in aws-observability#331, aws-observability#330 and aws-observability#329's latest commits. Conflicts resolved in favour of this branch's restructure, which is its purpose: the $allocationStrategy variable form is kept, and the top-level prometheusCRDs block is intentionally NOT re-added since this PR moves it to otelContainerInsights.prometheusScrape.crds.

…ion-scraper-routing Brings in the full chain (aws-observability#332, aws-observability#331, aws-observability#330, aws-observability#329) and origin/main. The cluster-scraper OTEL config conflicted structurally: main added the Karpenter/KEDA solution pipelines (aws-observability#325, aws-observability#345) in the same receivers / processors / pipelines sections where this branch adds its prometheuscr blocks, and git interleaved the {{- if }}/{{- end }} boundaries. Rather than untangle the markers, the file was reconstructed from main's version with this branch's three self-contained, individually-gated prometheuscr blocks re-inserted at their anchors (after each kubeStateMetrics block). Verified: all five cluster-scraper pipelines render (prometheuscr, karpenter, keda, apiserver, kube_state_metrics), scraperRole is still set only on the cluster-scraper, and the TA ClusterRole/Binding still render exactly once.

Aakash-Dantre · 2026-09-23T10:58:44Z

Checklist posted for the PR author to move into the PR description under a ## PR Checklist heading.

PR Checklist

Commits are squashed into a logical, reviewable set — single commit, authored by the PR author
Commits and PR description comply with Amazon internal guidelines
make passes locally — helm lint clean, helm template renders, tests/prometheus_crds_matrix.sh 30 passed / 0 failed
All GitHub Actions checks on the PR are passing
Integration test evidence: link to a passing run, or state N/A with the reason
New or updated integration test coverage: [test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness aws/amazon-cloudwatch-agent-test#720 (per_node_test.go, crd_bundling_test.go)
New functionality has unit tests; bug fixes have a reproducing test — chart render tests under tests/
Config translation changes include updated golden files — N/A, chart-only change
Breaking or customer-visible changes are called out in the PR description

…tep install Ships the community monitoring.coreos.com ServiceMonitor/PodMonitor CRDs as templated, gated resources so they are not a manual prerequisite. Bundled copies are skipped when an unmanaged copy already exists (Helm lookup checks the release-name and release-namespace annotations), so a prometheus-operator-managed CRD is never clobbered, and they carry helm.sh/resource-policy: keep so uninstall does not cascade-delete ServiceMonitor/PodMonitor CRs. Note: lookup returns empty under helm template/--dry-run, so in GitOps render-then-apply flows set crds.install=never if you manage these CRDs yourself.

wenegiemepraise mentioned this pull request Jul 9, 2026

[amazon-cloudwatch-observability] refactor: single-switch SM/PM scraping under otelContainerInsights.prometheusScrape #332

Open

musa-asad reviewed Jul 21, 2026

View reviewed changes

wenegiemepraise mentioned this pull request Jul 21, 2026

[amazon-cloudwatch-observability] feat: add ServiceMonitor/PodMonitor v2 OTLP scraping path #329

Open

musa-asad previously approved these changes Jul 24, 2026

View reviewed changes

Aakash-Dantre dismissed musa-asad’s stale review via 0702b44 September 23, 2026 10:10

Aakash-Dantre force-pushed the helm-crd-bundling branch from 0702b44 to 8c8e1e7 Compare September 23, 2026 12:28

Aakash-Dantre force-pushed the helm-crd-bundling branch from 8c8e1e7 to 654e9e7 Compare September 23, 2026 12:42

Aakash-Dantre mentioned this pull request Oct 1, 2026

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #376

Closed

8 of 9 tasks

Aakash-Dantre mentioned this pull request Oct 20, 2026

feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #406

Open

10 tasks done

		@@ -0,0 +1,1442 @@
		{{- /*
		ServiceMonitor CRD (monitoring.coreos.com/v1), pinned to prometheus-operator v0.91.0 to

		@@ -0,0 +1,78 @@
		# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Conversation

wenegiemepraise commented Jul 9, 2026

Summary

Contents

Dependencies

Testing

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musa-asad Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenegiemepraise Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Aakash-Dantre commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

wenegiemepraise Jul 22, 2026 •

edited

Loading

musa-asad Jul 21, 2026 •

edited

Loading

Aakash-Dantre commented Sep 23, 2026 •

edited

Loading