[Bug] Preserve stable DCGM discovery across ClusterPolicy and GPUCluster deployments

[rahulait]

Problem

The GPUCluster DRA stack uses different Service names and pod labels from the existing ClusterPolicy stack.

ClusterPolicy uses:

• DCGM Service: nvidia-dcgm
• DCGM pod label: app=nvidia-dcgm
• Exporter Service: nvidia-dcgm-exporter
• Exporter pod label: app=nvidia-dcgm-exporter

GPUCluster uses:

• DCGM Service: nvidia-dcgm-dra
• DCGM pod label: app=nvidia-dcgm-dra
• Exporter Service: nvidia-dcgm-exporter-dra
• Exporter pod label: app=nvidia-dcgm-exporter-dra

The established DNS names and labels are used by downstream services, runbooks, readiness checks, and monitoring configurations. Consumers must now know which allocation stack is active or probe both identities.

This will become more complicated when DRA and non-DRA nodes are supported in the same cluster.

Expected behavior

GPU Operator should provide allocation-mode-independent discovery for DCGM and DCGM Exporter.

Consumers should be able to:

• Use a stable DNS name without detecting the allocation mode.
• Discover either implementation with a common component label.
• Continue using the DRA-specific DNS names where they have already been adopted.
• Avoid duplicate Prometheus scraping when both Service aliases exist.

DaemonSet names, selectors, and app labels can remain implementation-specific.

Proposed common labels

Add the following label to both DCGM DaemonSet metadata and pod-template metadata:

nvidia.com/gpu-operator.dcgm: "true"

Add the following label to both DCGM Exporter DaemonSet metadata and pod-template metadata:

nvidia.com/gpu-operator.dcgm-exporter: "true"

Apply these labels to both:

• The ClusterPolicy manifests under assets/state-dcgm*
• The GPUCluster manifests under manifests/state-dcgm*

These labels should be reserved so user-provided DaemonSet labels cannot override them.

Proposed DNS compatibility

For the initial implementation:

The legacy ClusterPolicy stack should continue exposing:

• nvidia-dcgm
• nvidia-dcgm-exporter

The DRA GPUCluster stack should expose:

• nvidia-dcgm
• nvidia-dcgm-dra
• nvidia-dcgm-exporter
• nvidia-dcgm-exporter-dra

This restores the established DNS names for DRA deployments without unnecessarily exposing DRA-specific names from the legacy stack.

Upgrade safety

The initial implementation should not immediately change existing Service selectors to the new common labels.

During an upgrade, existing pods do not receive the new labels until their DaemonSets roll out. If a Service selector changes before that rollout completes, the Service can temporarily lose all or some endpoints.

This is particularly disruptive for DCGM because its Service uses internalTrafficPolicy: Local.

For the initial phase:

• Existing ClusterPolicy Services should retain their existing app selectors.
• Existing DRA Services should retain their DRA-specific app selectors.
• New compatibility aliases in the DRA path should initially select the DRA-specific app labels.
• Common Service selectors should be introduced only after labeled pods have rolled out.

Future coexistence and shared ownership

When ClusterPolicy and GPUCluster are allowed to coexist, the stable Services will become shared resources.

The current reconciliation paths cannot safely co-own these Services:

• Both use controller owner references, but Kubernetes permits only one controller owner.
• Both perform whole-object updates that can overwrite owner references.
• ClusterPolicy directly deletes disabled Services.
• GPUCluster cleanup deletes objects by state label.
• Deleting either CR could remove a Service still needed by the other stack.

Simply replacing SetControllerReference with SetOwnerReference is insufficient.

A future phase should introduce a single shared-Service reconciler that:

• Computes desired Services from both CRs.
• Maintains deterministic non-controller owner references.
• Removes only the disabled or deleted CR's ownership.
• Deletes a Service only when no configuration requires it.
• Preserves API-assigned Service fields.
• Resolves conflicting exporter Service configurations deterministically.

Shared ownership must be implemented before ClusterPolicy and GPUCluster coexistence is enabled.

Monitoring considerations

If both exporter Service aliases select the same pods, monitoring must avoid scraping every exporter twice.

The implementation should define one canonical scrape Service, preferably nvidia-dcgm-exporter, and ensure:

• Only the canonical Service participates in annotation-based discovery.
• Operator-managed ServiceMonitors select only the canonical Service.
• Conflicting ServiceMonitor configurations are reported instead of using last-writer-wins behavior.

Acceptance criteria

• Both DCGM implementations carry nvidia.com/gpu-operator.dcgm="true".
• Both exporter implementations carry nvidia.com/gpu-operator.dcgm-exporter="true".
• Common labels exist on DaemonSet metadata and pod templates.
• User-provided labels cannot override the common labels.
• A DRA deployment exposes both established and DRA-specific DNS names.
• The legacy stack does not expose DRA-specific DNS names.
• Existing Service selectors are not changed before labeled pods roll out.
• Both DCGM DNS names preserve internalTrafficPolicy: Local.
• Exporter aliases preserve supported Service configuration.
• Exporter pods are not scraped twice.
• Regression tests cover both stacks and the upgrade transition.
• Shared ownership is implemented before ClusterPolicy and GPUCluster coexistence is enabled.

[kvnloo]

I like separating stable discovery identity from the implementation-specific app labels.

I would probably stage this in two phases rather than changing selectors and aliases together.

Phase 1: identity

Add reserved common labels to both ClusterPolicy and GPUCluster pods:

nvidia.com/gpu-operator.dcgm=true

nvidia.com/gpu-operator.dcgm-exporter=true

but leave all existing Service selectors unchanged.

That gives external discovery a stable label contract without creating an upgrade endpoint gap.

Phase 2: DNS aliases / shared ownership

Add compatibility Service aliases only after defining who owns those resources when ClusterPolicy and GPUCluster coexist.

The tricky invariant is:

one stable DNS identity must never have two controllers fighting over the same Service object

because whole-object reconciliation + controller owner references makes accidental co-ownership unsafe.

I would test upgrade ordering separately from steady-state discovery:

• old pods + new operator;
• mixed old/new pods during rollout;
• all new pods;
• eventual ClusterPolicy/GPUCluster coexistence.

That keeps the first contribution useful without requiring the final shared-resource ownership model immediately.