[Aakash-Dantre]

Adds an opt-in controller (--manage-prometheus-crds, off by default) that creates the monitoring.coreos.com/v1 ServiceMonitor and PodMonitor CRDs when they don't exist, so SM/PM scraping works with no manual CRD install.

• Embeds the prometheus-operator v0.92.0 CRDs, matching the client in go.mod. A unit test fails if they drift apart.
• Creates with field manager amazon-cloudwatch-agent-operator, label cloudwatch.aws.amazon.com/crd-managed-by, and annotations crd-applied-version and crd-applied-generation. Sets no owner reference.
• Never updates, patches or deletes an existing CRD. When a CRD isn't the operator's, it logs and emits an Event once.
• Treats a CRD as another tool's when it carries kubectl.kubernetes.io/last-applied-configuration, meta.helm.sh/release-name or app.kubernetes.io/managed-by. A client-side kubectl apply or Helm --take-ownership with identical content changes no spec field and no generation, so only these markers show it.
• Recreates a CRD after it is deleted. AlreadyExists on create counts as success, so two operator pods running at once (for example during a rolling upgrade) are safe without leader election.
• Watches CRD metadata only, filtered to the two names.
• RBAC: customresourcedefinitions get, list, watch and create, with no update, patch or delete.

Testing

Unit tests for the ownership check, create path, step-back and version pinning, and an envtest suite that runs the reconciler and the ownership check against a real API server (pkg/crdmanager/envtest_test.go):

Case	Result
Reconcile on an empty cluster	Both CRDs created; generation 1, amazon-cloudwatch-agent-operator is the only spec field manager, ownership check passes
Another manager adds a label only	generation unchanged, ownership check passes
Another field manager server-side applies with force	Ownership check fails
Another client changes the spec	generation 2, ownership check fails
operator.prometheus.io/version changes	Ownership check fails
Client-side apply with identical content (adds last-applied-configuration)	generation unchanged, no extra spec manager, ownership check fails on the marker
Helm ownership annotations and label	Ownership check fails on the marker
Each failing case, reconciled 3 times	Zero writes, resourceVersion unchanged, exactly one PrometheusCRDNotManaged Event
CRD deleted, then reconcile	Recreated as a new object with fresh ownership metadata; ownership check passes

With the marker check disabled, the client-side apply and Helm cases fail. golangci-lint is clean.

On EKS 1.35 with aws-observability/helm-charts#406:

• The CRDs were created on a fresh cluster.
• CRDs installed by other tools were left unchanged.
• A newer applied CRD wasn't downgraded.
• The CRDs and monitors survived helm uninstall.
• With two operator pods, each CRD was created exactly once.

Related: aws-observability/helm-charts#406, aws/amazon-cloudwatch-agent-test#773

Merge order

1. This PR, then an operator and Target Allocator image release that includes it.
2. feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator aws-observability/helm-charts#406 passes --manage-prometheus-crds when otelPrometheusMonitor.enabled is true. That switch is off by default, so Updated Release Notes for 3.7.0 #406 can merge before this releases. Turning scraping on requires an operator image that includes this PR; older images reject the unknown flag at startup.
3. test(otel/pernode): per-node, CRD-management and scraper-routing E2E suites amazon-cloudwatch-agent-test#773 checks the CRD ownership metadata this PR writes, so it runs against images and a chart that include both.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change) - one commit.
• Commits and PR description comply with Amazon internal guidelines - checked.
• make passes locally (build, unit tests, lint) - make test stops at generate → crdoc because fybrik.io is unreachable from my network. I ran the target's remaining steps directly:
  • make fmt vet envtest passes, with no changes to files in this PR.
  • go test on the target's package list, with envtest 1.33, passes 32/32 packages, including the new pkg/crdmanager. main passes 31/31 on the same machine.
  • golangci-lint reports 0 issues.
  • make manifests regenerates only the new customresourcedefinitions rule in config/rbac/role.yaml.
• All GitHub Actions checks on the PR are passing - PR Build and Operator Integration Test are waiting for a maintainer to approve workflow runs from a fork.
• Integration test evidence - the test(otel/pernode): per-node, CRD-management and scraper-routing E2E suites amazon-cloudwatch-agent-test#773 per-node harness, run locally with Terraform against a fresh EKS 1.35 cluster using images built from this PR and feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator aws-observability/helm-charts#406, passed 9/9. That covers CRD ownership, per-node allocation, scraper routing, Target Allocator health, and ServiceMonitor and PodMonitor metrics reaching CloudWatch. The CRD scenarios in the Testing section above were run manually on EKS. There is no GitHub Actions link because this harness is not in the CI matrix yet.
• New or updated integration test coverage - test(otel/pernode): per-node, CRD-management and scraper-routing E2E suites amazon-cloudwatch-agent-test#773 (TestServiceMonitorPodMonitorCRDsCreatedByOperator, TestOperatorManagesPrometheusCRDs).
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated - feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator aws-observability/helm-charts#406 and test(otel/pernode): per-node, CRD-management and scraper-routing E2E suites amazon-cloudwatch-agent-test#773; merge order above.
• New functionality has unit tests; behaviour changes have a reproducing test - pkg/crdmanager/reconciler_test.go (unit) and pkg/crdmanager/envtest_test.go (real API server).
• Config translation changes include updated golden files - N/A: operator change, no translator changes.
• Breaking or customer-visible changes are called out in the PR description - no behavior change unless --manage-prometheus-crds is set; the new flag defaults to false. The manager ClusterRole in config/rbac/role.yaml gains cluster-wide create on customresourcedefinitions (plus get, list, watch). Kubernetes cannot limit create to specific names, so this needs security review.

[miconeilaws]

Blocking: add envtest coverage for the ownership check (pkg/crdmanager)

The unit tests use the fake client and hand-build managedFields and generation in ownedMeta(), so nothing proves that a CRD this operator creates passes IsOwned on a real API server. If it doesn't, every cluster that runs this release reports the CRD as not managed, and the follow-up that upgrades owned CRDs can never upgrade them. The label and annotations become a contract once this ships, so we can't fix that afterwards.

The repo already runs envtest from make test. Please add a suite in pkg/crdmanager (pattern: pkg/instrumentation/instrumentation_suite_test.go) that covers:

1. Reconcile on an empty cluster creates both CRDs. On the stored objects, generation is 1, managedFields has an entry for amazon-cloudwatch-agent-operator, and IsOwned returns true.
2. Another manager adds a label only. generation is unchanged and IsOwned is still true.
3. Another field manager server-side applies the CRD with force. IsOwned is false.
4. Another client updates the spec, for example a schema description. generation increases and IsOwned is false.
5. A newer upstream manifest is applied. operator.prometheus.io/version changes and IsOwned is false.
6. Another client writes content identical to ours with a client-side apply, which adds kubectl.kubernetes.io/last-applied-configuration. generation is unchanged and IsOwned is false (see the IsOwned markers comment).
7. After any of 3 to 6, reconcile makes zero writes and emits one PrometheusCRDNotManaged Event.
8. Delete the CRD and reconcile. It is recreated with fresh ownership metadata, and IsOwned is true.

[miconeilaws]

IsOwned (pkg/crdmanager/reconciler.go): step back on other tools' markers

A client-side kubectl apply (Argo CD's default sync) or Helm 3 --take-ownership that writes content identical to ours leaves generation unchanged and adds no spec entry to managedFields. All four checks then pass on a CRD another tool now manages, and the follow-up upgrade would fight that tool.

Please also return false when the CRD has the annotation kubectl.kubernetes.io/last-applied-configuration, the annotation meta.helm.sh/release-name, or the label app.kubernetes.io/managed-by. This only changes what the check reports, so Phase 1 behavior is the same. It belongs in this PR because the follow-up reuses IsOwned unchanged.

[miconeilaws]

Non-blocking: drop --leader-elect (main.go and the package doc)

The chart runs a single operator replica, so this feature doesn't need leader election, and nothing passes the flag. Please remove --leader-elect along with the LeaderElection and LeaderElectionID manager options, and drop it from the PR description.

The package doc also says "Run only in the elected leader", which isn't true as shipped. Suggest: "Safe with more than one operator pod (for example during a rolling upgrade): only a missing CRD is created, and AlreadyExists counts as success."

[Aakash-Dantre]

Blocking: add envtest coverage for the ownership check (pkg/crdmanager)

Added an envtest suite in pkg/crdmanager (crdmanager_suite_test.go, same pattern as pkg/instrumentation). envtest_test.go covers your 8 cases against a real API server:

• create on an empty cluster: generation 1, the operator is the only spec field manager, IsOwned true;
• label-only change;
• forced server-side apply;
• spec change;
• version change;
• client-side apply with identical content;
• Helm markers;
• delete and recreate.

Each failing case is reconciled 3 times, with zero writes and exactly one PrometheusCRDNotManaged Event. With the marker check disabled, the client-side apply and Helm cases fail.

[Aakash-Dantre]

IsOwned (pkg/crdmanager/reconciler.go): step back on other tools' markers

Done: IsOwned now returns false when the CRD has kubectl.kubernetes.io/last-applied-configuration, meta.helm.sh/release-name or the app.kubernetes.io/managed-by label. The envtest case confirms that a client-side apply with identical content leaves generation at 1 and adds no spec manager, so only this check catches it.

[Aakash-Dantre]

Non-blocking: drop --leader-elect (main.go and the package doc)

Removed --leader-elect and the LeaderElection/LeaderElectionID options, updated the package doc to your suggested wording, and dropped it from the description.

[AshuSrv]

Let's fix the tests if they are flaky.