[BUG][multi-cluster] VolumeExpansion ops fails

[jinuxstyle]

Describe the bug

In multi-cluster deployment with enableInstanceAPI: true, VolumeExpansion OpsRequest fails or stays at progress: 0/N until timeout. The Ops Action successfully updates Cluster / Component / InstanceSet volumeClaimTemplates to the target size, but only one member-cluster PVC is actually expanded. Ops cannot detect any PVC progress and eventually fails with a 30-minute timeout.

Reproduced on: mypg — fresh 3-replica PostgreSQL cluster on community latest main (2026-07-07), first VolumeExpansion after cluster reached Running.

# control plane
kubectl -n mypg get ops
NAME                  TYPE              CLUSTER   STATUS   PROGRESS   AGE
expand-mypg-tklpw     VolumeExpansion   mypg      Failed   0/3        38m

kubectl -n mypg get cluster,component,instanceset
mypg                Updating
mypg-postgresql     Updating

NAME                 DESIRED   UP-TO-DATE   READY   AVAILABLE
mypg-postgresql      3         1            3       3

Spec vs actual storage after volume expansion (10Gi → 11Gi):

# cluster.spec.componentSpecs[0].volumeClaimTemplates[0]   = 11Gi  ✅
# component.spec.volumeClaimTemplates[0]                   = 11Gi  ✅
# instanceset.spec.volumeClaimTemplates[0]                 = 11Gi  ✅
# instanceset.status.updatedReplicas                       = 1     ❌ (expected 3)

Member-cluster PVCs (only 1 of 3 expanded):

# kb-member-cluster1
data-mypg-postgresql-0   10Gi   Bound

# kb-member-cluster2
data-mypg-postgresql-1   10Gi   Bound

# kb-member-cluster3
data-mypg-postgresql-2   11Gi   Bound   # only this one

Instance CR VCT spec:

mypg-postgresql-0   VCT=10Gi  kubeblocks.io/generation=1
mypg-postgresql-1   VCT=10Gi  kubeblocks.io/generation=1
mypg-postgresql-2   VCT=11Gi  kubeblocks.io/generation=2

OpsRequest status:

Status:
  Phase:     Failed
  Progress:  0/3
  Conditions:
    Message: Timed out waiting for volume expansion to complete, the timeout value is 30 minutes
  Components:
    postgresql: {}

To Reproduce

1. Install KubeBlocks (latest main) on control plane + member clusters with multi-cluster enabled:

   --multi-cluster-kubeconfig=/var/run/secrets/kubeblocks.io/multicluster/kubeconfig
   --multi-cluster-contexts=kb-member-cluster1,kb-member-cluster2,kb-member-cluster3
   

2. Create a new running multi-cluster PostgreSQL cluster with enableInstanceAPI: true, 3 replicas, 10Gi data volumes:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mypg
  namespace: mypg
  annotations:
    apps.kubeblocks.io/multi-cluster-placement: kb-member-cluster1,kb-member-cluster2,kb-member-cluster3
spec:
  clusterDef: postgresql
  topology: replication
  terminationPolicy: Delete
  componentSpecs:
    - name: postgresql
      enableInstanceAPI: true
      replicas: 3
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: openebs-lvmpv
            accessModes: [ReadWriteOnce]
            resources:
              requests:
                storage: 10Gi

3. Wait until cluster is Running with 3 Pods on member clusters.

4. Create volume expansion OpsRequest (10Gi → 11Gi):

kubectl create -f - <<EOF
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: expand-mypg-
  namespace: mypg
spec:
  clusterName: mypg
  type: VolumeExpansion
  volumeExpansion:
    - componentName: postgresql
      volumeClaimTemplates:
        - name: data
          storage: 11Gi
EOF

5. Observe:
   • Cluster / Component / InstanceSet spec storage becomes 11Gi
   • Only one member-cluster PVC expands (in our case postgresql-2 on kb-member-cluster3)
   • Other PVCs stay 10Gi
   • InstanceSet.status.updatedReplicas stays 1
   • OpsRequest progress stays 0/3 with empty status.components.postgresql
   • After 30 minutes, OpsRequest phase: Failed with timeout message

Expected behavior

• InstanceSet controller updates all Instance CRs' volumeClaimTemplates to 11Gi.
• Instance controller patches all member-cluster PVCs to request 11Gi.
• Storage provider expands volumes; Ops detects N/N PVCs at target size.
• OpsRequest completes with phase: Succeed and progress: 3/3.
• Cluster returns to Running.

Root cause (analysis)

Two independent bugs in the enableInstanceAPI + instanceset2 path:

Bug 1: Ops cannot discover PVCs created by Instance controller

pkg/operations/ops_runtime.go → loadVolumes lists PVCs with:

client.MatchingLabels{
    constant.AppInstanceLabelKey:    clusterName,      // app.kubernetes.io/instance
    constant.KBAppComponentLabelKey: compName,         // apps.kubeblocks.io/component-name
}

PVCs created by pkg/controller/instance/utils.go → buildInstancePVCs only have:

apps.kubeblocks.io/instance-name=<instance-name>
apps.kubeblocks.io/pod-name=<instance-name>
apps.kubeblocks.io/vct-name=data

They lack app.kubernetes.io/instance and apps.kubeblocks.io/component-name.

In handleVCTExpansionProgress (pkg/operations/volume_expansion.go):

• GetInstance finds the Pod via multi-cluster Get
• newPodInstance → loadVolumes returns empty map
• GetVolume("data") returns false → continue without writing progressDetails
• completedProgressCount stays 0 → progress: 0/N until 30-minute timeout → Failed

Bug 2: instanceset2 update budget exhausted after first Instance update

pkg/controller/instanceset2/reconciler_update.go limits rolling updates per reconcile:

unavailable := maxUnavailable - currentUnavailable  // maxUnavailable defaults to 1
// ...
if updatingInstances >= min(replicas, unavailable, updateCount) {
    break
}

With memberUpdateStrategy: BestEffortParallel and roleful InstanceSet, updateCount = 1 per reconcile.

When the first Instance (mypg-postgresql-2) is updated, its Generation advances while ObservedGeneration lags and UpToDate becomes false. IsInstanceAvailable returns false, so currentUnavailable = 1 and unavailable = 0. All subsequent reconciles can no longer update any Instance; the remaining PVCs stay at the old size.

KubeBlocks logs show only mypg-postgresql-2 was updated during the entire 30-minute window:

update Instance mypg-postgresql-2 in InstanceSet mypg-postgresql successful  (x7)

Suggested fixes

1. Ops: Extend loadVolumes to find PVCs by pod volume claim names or Instance API labels (apps.kubeblocks.io/pod-name); alternatively add cluster/component labels when Instance controller creates PVCs.

2. instanceset2: For storage-only VCT changes (volume expansion), bypass the unavailable gate and update all Instance CRs in parallel; or fix the unavailable = maxUnavailable - currentUnavailable formula so a temporarily unavailable instance does not zero out the update budget permanently.

Environment

• KubeBlocks: community latest main (2026-07-07 build)
• Addon: postgresql-16-1.1.0-main-186f-jun18
• Cluster: mypg, namespace mypg
• Multi-cluster placement: kb-member-cluster1,kb-member-cluster2,kb-member-cluster3
• StorageClass: openebs-lvmpv (allowVolumeExpansion: true)
• enableInstanceAPI: true
• memberUpdateStrategy: BestEffortParallel

Additional context

Attachment has more info about the cluster.

mypg-volume-expansion-debug-bundle-2026-07-07.zip

[jinuxstyle]

@leon-ape @shanshanying Tested the fix, and the issue is not fixed.

===>Cluster
              storage: 11Gi
          storageClassName: openebs-lvmpv

===>Component
            storage: 11Gi
        storageClassName: openebs-lvmpv

===>InstanceSet
            storage: 11Gi
        storageClassName: openebs-lvmpv

===>Instances
            storage: 10Gi
        storageClassName: openebs-lvmpv
            storage: 10Gi
        storageClassName: openebs-lvmpv
            storage: 11Gi
        storageClassName: openebs-lvmpv

===>PVC
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
data-mypg3-postgresql-0   Bound    pvc-ff86bd6a-08f9-4b11-9a50-07bc2c798172   10Gi       RWO            openebs-lvmpv   <unset>                 112m
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
data-mypg3-postgresql-1   Bound    pvc-229326c0-759a-42e7-b358-bd8b986e7a6e   10Gi       RWO            openebs-lvmpv   <unset>                 112m
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
data-mypg3-postgresql-2   Bound    pvc-4b5ec6e2-154c-402e-873a-86405485bcbb   11Gi       RWO            openebs-lvmpv   <unset>                 112m

describe its:

Status:
  Available Replicas:  3
  Conditions:
    Last Transition Time:  2026-07-21T09:55:38Z
    Message:
    Observed Generation:   2
    Reason:                Ready
    Status:                True
    Type:                  InstanceReady
    Last Transition Time:  2026-07-21T09:55:38Z
    Message:
    Observed Generation:   2
    Reason:                Available
    Status:                True
    Type:                  InstanceAvailable
  Current Replicas:        2
  Current Revision:        75b5dc8648
  Current Revisions:
    mypg3-postgresql-0:  787d569ffd
    mypg3-postgresql-1:  55fc4d7c7f
    mypg3-postgresql-2:  5bb7788bbc
  Init Replicas:         0
  Instance Status:
    Pod Name:           mypg3-postgresql-0
    Role:               primary
    Pod Name:           mypg3-postgresql-1
    Role:               secondary
    Pod Name:           mypg3-postgresql-2
    Role:               secondary
  Observed Generation:  2
  Ready Replicas:       3
  Replicas:             3
  Update Revision:      5bb7788bbc
  Update Revisions:
    mypg3-postgresql-0:  76658b5449
    mypg3-postgresql-1:  64874487d4
    mypg3-postgresql-2:  5bb7788bbc
  Updated Replicas:      1
Events:
  Type    Reason            Age                From                       Message
  ----    ------            ----               ----                       -------
  Normal  SuccessfulUpdate  18m (x7 over 18m)  instance-set-controller-2  update Instance mypg3-postgresql-2 in InstanceSet mypg3-postgresql successful

[jinuxstyle]

@shanshanying @leon-ape any progress?

[leon-ape]

Thanks for testing and reporting back. The latest result shows the PVC discovery issue is likely no longer the only blocker. The remaining symptom looks like the InstanceSet only updates one Instance's volumeClaimTemplates, while the other Instances keep the old storage size.

Could you provide the minimal YAML below so we can confirm where the update is stuck?

kubectl -n mypg get instanceset mypg3-postgresql -o yaml
kubectl -n mypg get instance mypg3-postgresql-0 -o yaml
kubectl -n mypg get instance mypg3-postgresql-2 -o yaml
kubectl -n mypg get pvc data-mypg3-postgresql-0 -o yaml
kubectl -n mypg get pvc data-mypg3-postgresql-2 -o yaml

One old-size Instance/PVC and one expanded Instance/PVC should be enough for comparison.

[jinuxstyle]

instance-mypg3-postgresql-0.yaml
instance-mypg3-postgresql-2.yaml
its-mypg3-postgresql.yaml
pvc-data-mypg3-postgresql-0.yaml
pvc-data-mypg3-postgresql-2.yaml