Skip to content

feat: update charts for enabling monitoring - #236

Merged
Fei-Guo merged 1 commit into
mainfrom
fguo-dev
Oct 7, 2026
Merged

Fei-Guo merged 1 commit into
mainfrom
fguo-dev

Conversation

@Fei-Guo

@Fei-Guo Fei-Guo commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Reason for Change:

Adds GPU and inference observability wiring for Azure Monitor managed Prometheus across the two chart layers:

  • charts/modelharness (per-namespace): new templates/azure/podmonitor.yaml renders an azmonitoring.coreos.com/v1 PodMonitor that scrapes every pod in the workload namespace across the EPP, Istio gateway, and vLLM metric surfaces, lifting the inferencepool / gateway / shadow-pod / workspace pod labels onto the emitted series. Gated on cloudprovider=azure + a new azure.podMonitor toggle, and additionally on .Capabilities.APIVersions.Has "azmonitoring.coreos.com/v1/PodMonitor" so an install against a cluster without the Azure Managed Prometheus add-on silently skips the CR instead of failing on an unknown kind. Values and values.schema.json updated (enabled / name / interval).
  • charts/productionstack (cluster-wide): new templates/azure/ama-metrics-settings-configmap.yaml renders the add-on's well-known ama-metrics-settings-configmap in kube-system enabling the dcgmexporter default scrape target, so the add-on collects raw NVIDIA DCGM GPU metrics cluster-wide. Gated on cloudprovider=azure via a new azure.amaMetricsSettings toggle (default true; renders nothing on non-Azure installs).

This gives per-namespace inference metrics plus cluster-wide raw GPU metrics in Azure Monitor managed Prometheus, with both templates safe to apply on clusters where the add-on / CRD is absent.

Requirements

  • added unit tests and e2e tests (if applicable).

Notes for Reviewers:

  • PodMonitor label caps (labelLimit: 63, labelNameLengthLimit: 511, labelValueLengthLimit: 1023) are hand-picked guardrails, not Azure/Prometheus defaults (Prometheus default is 0 = unlimited). Flag if you'd prefer them tunable or dropped.
  • ConfigMap ownership: productionstack now owns ama-metrics-settings-configmap. If a cluster already has one (platform- or operator-managed), helm install will conflict — set azure.amaMetricsSettings.enabled=false there. Default is true.
  • .Capabilities caveat: capability discovery needs a live API server, so a bare helm template (no --validate/--api-versions) will not render the PodMonitor even on a cluster that has the CRD. Relevant for CI snapshot tests.
  • No values.schema.json exists for charts/productionstack, so only charts/modelharness's schema was updated.
  • The modelharness podMonitor.name default is all-model-pods; the ama settings ConfigMap name/namespace are pinned (the add-on only reads that exact name in kube-system).

@Fei-Guo
Fei-Guo merged commit f9d4b18 into main Oct 7, 2026
13 checks passed

This branch was successfully deployed

1 active deployment
e2e-test — 0f8b1298 Deployed Oct 6, 2026 by Fei-Guo via e2e-tests #622
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants