Support NVIDIADriver reconciliation and driver upgrades without a ClusterPolicy by karthikvetrivel · Pull Request #2572 · NVIDIA/gpu-operator

karthikvetrivel · 2026-06-23T15:51:30Z

Stacked on #2571 (kv-gpuclusterconfig-crd). This PR's diff is just the two commits below. GitHub will retarget it onto main once #2571 lands.

Summary

This PR lets the NVIDIADriver controller reconcile without a ClusterPolicy, so a GPUCluster (the DRA stack) can use operator-managed drivers standalone.

The driver host root is now sourced via the info catalog instead of the ClusterPolicy CR. The driver renderer consumes a new InfoTypeHostRoot catalog entry rather than reading the ClusterPolicy directly, which decouples it from the gpuv1 types. When both CRs exist, the ClusterPolicy's hostPaths.rootFS takes precedence. Otherwise the GPUCluster's value is used.
Enabling DRA on a cluster requires that all driver management in that cluster is done through NVIDIADriver CRs. A ClusterPolicy may coexist with a GPUCluster CR, but only if it delegates driver management by setting driver.useNvidiaDriverCRD=true. A ClusterPolicy that owns driver daemonsets directly (useNvidiaDriverCRD=false) alongside a GPUCluster CR is an invalid configuration. There is no supported topology where ClusterPolicy-owned drivers and NVIDIADriver-owned drivers run in the same cluster.

Testing

There is unit coverage for the host-root sourcing and the gate transitions, plus two live validations on a single-node Tesla T4 cluster (Kubernetes 1.34, containerd 2.2).

Standalone end-to-end: with no ClusterPolicy, a GPUCluster plus a default NVIDIADriver rolled out the driver DaemonSet. The DRA kubelet-plugin came up against the operator-installed driver root (devRoot=/run/nvidia/driver) and published its DeviceClasses and ResourceSlice. A pod with a DRA claim ran nvidia-smi against the operator-installed driver.
Gate matrix, observed via .status.state transitions on a cluster converted from a released operator install. A ClusterPolicy that is not delegating with no GPUCluster produced disabled. A GPUCluster alone produced ready. With neither CR, the NVIDIADriver went notReady with the expected message. The ClusterPolicy plus GPUCluster coexistence path is covered by the unit tests.
All transitions were watch-driven. Deleting or creating a GPUCluster flipped the NVIDIADriver state without manual intervention, and a single GPUCluster creation transitioned two NVIDIADriver instances at once.

coveralls · 2026-07-01T15:31:22Z

coverage: 35.899% (+0.6%) from 35.256% — kv-nvidiadriver-standalone into main

tariq1890 · 2026-07-24T18:53:13Z

 		return reconcile.Result{}, err
 	}

+	if clusterPolicy == nil {


We can merge two if blocks into one with this change

Suggested change

if clusterPolicy == nil {

if clusterPolicy == nil || clusterPolicy.Spec.Driver.UseNvidiaDriverCRDType() {

Actually, isn't it possible now to have both ClusterPolicy-backed drivers and NVIDIADriver-backed drivers (we can continue to debate whether this is actually desired or not in a separate thread)? IIUC the upgrade-controller, as currently implemented, does not account for that use case -- it only calls reconcileNVIDIADriverUpgrades() OR reconcileClusterPolicyDriverUpgrades.

The problematic scenario -- ClusterPolicy and GPUCluster exist; clusterpolicy.spec.driver.useNvidiaDriverCRD=false; NVIDIADriver CRs exist which target DRA nodes. In this case, the upgrade-controller will not be able to manage the upgrades for both the ClusterPolicy-owned driver daemonsets and the NVIDIADriver-owned driver daemonsets.

Good catch. You are right that the upgrade controller does not handle that.

As we discussed offline, we decided not to support it. To summaize, enabling DRA now requires that all driver management in the cluster is done through NVIDIADriver CRs. A ClusterPolicy with useNvidiaDriverCRD=false alongside a GPUCluster is treated as an invalid configuration.

tariq1890 · 2026-07-24T18:58:34Z

 		}
+		hostRoot = clusterPolicy.Spec.HostPaths.RootFS
+	case gpuCluster != nil:
+		hostRoot = gpuCluster.Spec.HostPaths.RootFS


Correct me if I am wrong, but in the scenario where both ClusterPolicy and GPUCluster exist, will the .spec.hostPaths.rootFS from GPUCluster have the higher order of precedence?

ClusterPolicy takes precedence today. Now, I believe this combination is only valid when the ClusterPolicy sets useNvidiaDriverCRD=true, and in that case ClusterPolicy's hostPaths.rootFS is used.

I see that this hostRoot value gets used here

- name: host-root hostPath: path: {{ .HostRoot | default "/" }}

Thinking about this more, it doesn't make much sense to make this global value and we are better off getting this value from the NVIDIADriver CR. For this, we would need to add a new field called hostRoot to the NVIDIADriver CR. This is the behaviour I envision in the following scenarios:

If the GPUCluster resource is present (whether by itself or along with ClusterPolicy, then we get the hostRoot from the NVIDIADriver CR, else we fallback to the GPUCluster hostRoot. Here we will never get the hostRoot from ClusterPolicy. This way we ensure zero dependency on ClusterPolicy in a GPUCluster flow

If only ClusterPolicy is present, then we either get the hostRoot from the NVIDIADriver CR (the newly added field), else we fallback to the ClusterPolicy.Spec.HostPaths.RootFS

This can be done in a follow-up PR

Note that when we add hostRoot to the NVIDIADriver CR, it'll be an additive change and it will be an optional field. If this is empty, it will be defaulted to "/" as we currently do today

Note, the host root is needed for many of our daemonsets, not just the driver. So we still need this field to be present in the GPUCluster CR. What is the benefit in having users configure this field in multiple places (GPUCluster and NVIDIADriver CRs)?

Yes, the rest of the HostPaths fields are used correctly in operands whether they come from GPUCluster or ClusterPolicy. For the driver daemonset however, it's just the .HostPaths.RootFS field that is relevant

I'm in favor of moving hostRoot to NVIDIADriver and removing it entirely from GPUCluster.

In the legacy ClusterPolicy path, we can get the hostRoot from either NVIDIADriver CR (if specified) or from ClusterPolicy.Spec.HostPaths.RootFS when useNvidiaDriverCRD=true, and from ClusterPolicy.Spec.HostPaths.RootFS only when useNvidiaDriverCRD=false.

After discussing internally, we have come to a consensus on how to move forward with this. The main issue we that we want to address (in a follow-up PR) is that there should be no cross-CR or circular dependency between the custom resources.

The hostRoot will be hard-coded as / for now.

We had between two approaches to fix this problem. Either adding a new hostRoot field to the NVIDIADriver CRD or adding a new environment variable to determine the hostRoot to be set in both the driver daemonset and the DRA driver kubelet plugin daemonset. This is still ongoing.

Done. The host root is now hard-coded to / whenever a GPUCluster is present, including when a ClusterPolicy coexists with it, so NVIDIADriver never reads it from another CR. Only the ClusterPolicy-only legacy path still sources spec.hostPaths.rootFS. Also added a ClusterPolicy-only test case for that path.

tariq1890 · 2026-07-24T19:01:35Z

Thanks @karthikvetrivel! Can you also squash your commit history here?

tariq1890 · 2026-07-28T15:02:34Z

+	if err := r.List(ctx, clusterPolicies); err != nil {
+		return "", fmt.Errorf("error listing ClusterPolicy objects: %w", err)
+	}
+	for i := range clusterPolicies.Items {


Does it make sense to loop through every possible clusterPolicy? Don't we only need to care about the active/singleton clusterPolicy?

I was thinking about this as well, but we don't ever enforce a singleton ClusterPolicy. You're right — there is only one active ClusterPolicy and extra CPs get status.state = Ignored, but only asynchronously after the CP controller reconciles them so filtering on status could lead to a potential races.

This is somewhat related to the TODO above resolveActiveConfig, which we can address in a follow up.

Can you add the TODO here too?

tariq1890 · 2026-07-28T15:49:51Z

+	if err := r.List(ctx, clusterPolicies); err != nil {
+		return "", fmt.Errorf("error listing ClusterPolicy objects: %w", err)
+	}
+	for i := range clusterPolicies.Items {


Suggested change

for i := range clusterPolicies.Items {

for _, clusterPolicy := range clusterPolicies.Items {

cdesiniotis · 2026-07-28T17:12:13Z

+	default:
+		err := fmt.Errorf("no ClusterPolicy or GPUCluster object found in the cluster")
+		logger.Error(err, "failed to retrieve hostPaths information. No ClusterPolicy or GPUCluster object found in the cluster")


Question -- should we not default to / in this case too? That way, one could deploy NVIDIADriver CRs without having to deploy either a ClusterPolicy or GPUCluster first?

Yes, I think so. I added it.

…sterPolicy Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

karthikvetrivel requested review from cdesiniotis, rahulait, rajathagasthya, shivamerla and tariq1890 as code owners June 23, 2026 15:51

karthikvetrivel marked this pull request as draft June 23, 2026 15:54

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 3 times, most recently from 8ffcfc3 to 479562c Compare June 24, 2026 19:05

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from 5691bbc to e5dcecd Compare July 1, 2026 15:24

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 479562c to 0e75d4c Compare July 1, 2026 15:24

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 6 times, most recently from 3b7cc55 to 4ad2c9d Compare July 8, 2026 18:48

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 8 times, most recently from 1a06b01 to b101993 Compare July 16, 2026 16:01

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 0e75d4c to 7f38f41 Compare July 16, 2026 16:11

karthikvetrivel marked this pull request as ready for review July 16, 2026 16:48

karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from b101993 to 7f26b3c Compare July 20, 2026 18:31

karthikvetrivel mentioned this pull request Jul 20, 2026

Add GPUCluster CRD and controller for DRA-based stack #2571

Merged

Base automatically changed from kv-gpuclusterconfig-crd to main July 23, 2026 23:05

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from fdb7c6e to a9629e2 Compare July 24, 2026 16:19

tariq1890 reviewed Jul 24, 2026

View reviewed changes

Comment thread controllers/nvidiadriver_controller.go Outdated

tariq1890 reviewed Jul 24, 2026

View reviewed changes

rahulait assigned karthikvetrivel Jul 27, 2026

rahulait added this to the v26.7 milestone Jul 27, 2026

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 3 times, most recently from f9e8163 to aeca33f Compare July 27, 2026 16:36

tariq1890 reviewed Jul 28, 2026

View reviewed changes

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from aeca33f to cb4dbc6 Compare July 28, 2026 15:28

tariq1890 reviewed Jul 28, 2026

View reviewed changes

cdesiniotis reviewed Jul 28, 2026

View reviewed changes

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 3 times, most recently from 9043f10 to 0380d19 Compare July 28, 2026 18:36

cdesiniotis approved these changes Jul 28, 2026

View reviewed changes

tariq1890 reviewed Jul 28, 2026

View reviewed changes

Comment thread controllers/nvidiadriver_controller.go Outdated

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 0380d19 to 77b8ebb Compare July 28, 2026 18:46

Support NVIDIADriver reconciliation and driver upgrades without a Clu…

edfdd7a

…sterPolicy Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>

karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 77b8ebb to edfdd7a Compare July 28, 2026 19:01

rajathagasthya approved these changes Jul 28, 2026

View reviewed changes

cdesiniotis approved these changes Jul 28, 2026

View reviewed changes

tariq1890 approved these changes Jul 28, 2026

View reviewed changes

karthikvetrivel enabled auto-merge July 28, 2026 19:11

karthikvetrivel mentioned this pull request Jul 28, 2026

Enforce singleton GPUCluster name and select ClusterPolicy by creation time #2689

Merged

karthikvetrivel merged commit 786f4d9 into main Jul 28, 2026
21 checks passed

rajathagasthya deleted the kv-nvidiadriver-standalone branch July 28, 2026 20:23

	if clusterPolicy == nil {
	if clusterPolicy == nil \|\| clusterPolicy.Spec.Driver.UseNvidiaDriverCRDType() {

	for i := range clusterPolicies.Items {
	for _, clusterPolicy := range clusterPolicies.Items {

Conversation

karthikvetrivel commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

coveralls commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tariq1890 commented Jul 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

karthikvetrivel commented Jun 23, 2026 •

edited

Loading

coveralls commented Jul 1, 2026 •

edited

Loading

karthikvetrivel Jul 27, 2026 •

edited

Loading

karthikvetrivel Jul 28, 2026 •

edited

Loading