test(e2e): add monitoring-on e2e coverage for the otel-collector sidecar

[WentingWu666666]

Summary

The OpenTelemetry Collector sidecar currently has no end-to-end test coverage. The sidecar is only injected when spec.monitoring.enabled: true, but every e2e spec runs with monitoring off (the base manifest test/e2e/manifests/base/documentdb.yaml.template has no monitoring block, and no spec sets Monitoring.Enabled = true).

As a result:

• The otel-collector's hardened SecurityContext is never admission-validated against a real PSA-restricted namespace end-to-end (only the always-injected documentdb-gateway is).
• More broadly, otel injection, base config, and OTLP/Prometheus wiring are only covered by unit tests in the sidecar-injector module.

Background

Surfaced during review of #411 (fix for #387, hardening injected sidecars for PSA restricted compliance). Reviewer note:

The otel-collector runtime hardening (relying on the image's baked-in UID 10001 + runAsNonRoot) is only unit-tested — the deploy e2e runs with monitoring off, so it's never admission-validated end-to-end. A monitoring-on e2e would close the loop.

This is a pre-existing gap; #411 did not remove otel e2e coverage, it simply could not add admission validation for a path the suite never exercises.

Current coverage (for reference)

• TestNewOtelCollectorSidecar_Hardened (unit) — constructed container is PSA-restricted, no forced UID.
• injectGatewayOTelEnv unit tests — gateway-side OTLP env wiring.

Proposed work

Add a monitoring-on e2e spec (suggest the observability/telemetry area) that:

1. Deploys a DocumentDB with spec.monitoring.enabled: true (plus an exporter, e.g. Prometheus) in a PSA-restricted labeled namespace.
2. Waits for the cluster to become Ready.
3. Asserts the otel-collector sidecar is present and PSA-restricted.

The existing helper AssertInjectedSidecarsPSARestricted (test/e2e/pkg/e2eutils/assertions/assertions.go) already checks otel-collector when present, so once a monitoring-on cluster is deployed it will validate the otel sidecar automatically — no new assertion logic required, just a spec that turns monitoring on.

Optional follow-on

• Assert the collector is actually scrapeable / exporting (Prometheus endpoint reachable) to validate functional behavior, not just admission.

Acceptance criteria

• An e2e spec deploys a DocumentDB with monitoring enabled in a PSA-restricted namespace.
• The otel-collector sidecar is admission-validated end-to-end (cluster reaches Ready) and asserted PSA-restricted via AssertInjectedSidecarsPSARestricted.

Related: #387, #411

[AshrafAhmed9]

Hi, I'd like to work on this! My plan is to add a new e2e spec that deploys a DocumentDB cluster with spec.monitoring.enabled: true in a PSA-restricted namespace, waits for Ready, and calls the existing AssertInjectedSidecarsPSARestricted helper to validate the otel-collector sidecar end-to-end. I'll follow the structure of the existing e2e specs as a template.

Could you confirm this approach works for you before I start? Happy to discuss any specifics.