EVE-k: app reaches BOOTING while PVC still Pending (volume marked CREATED_VOLUME on PVC existence, not Bound)

[eriknordmark]

EVE-k: app advances to BOOTING while its PVC is still Pending (volume marked CREATED_VOLUME on PVC existence, not Bound)

Summary

On EVE-k, an app instance can reach BOOTING even though its backing
PersistentVolumeClaim has never been provisioned (Pending), because the
volume's EVE-side state is promoted to CREATED_VOLUME as soon as the PVC
object exists, without waiting for it to reach Bound. The app then sits in
BOOTING indefinitely — kubevirt never creates a VMI/virt-launcher because the
PVC can't be mounted — with no error surfaced on the EVE side.

Observed

Single-node EVE-k, longhorn unhealthy (no longhorn StorageClass yet), so the
app's PVC stays Pending:

• kubectl -n eve-kube-app get pvc → …-pvc-0 Pending (hours)
• kubectl -n eve-kube-app get vmi,pods → No resources found
• EVE AppInstanceStatus → State: 114 (BOOTING), Activated: true, no error
• EVE VolumeStatus for the ref → State: 110 (CREATED_VOLUME),
  PendingAdd: false, no error

So EVE believes the volume is ready and starts the domain, while k8s has not
provisioned the PVC.

Root cause

The deferred volume-create path gates CREATED_VOLUME on Populate():

• pkg/pillar/cmd/volumemgr/handlevolume.go:133 — handleDeferredVolumeCreate
  calls Populate(); line 145 sets status.State = types.CREATED_VOLUME
  whenever Populate() returns created == true.

Populate()'s non-replicated branch only checks PVC existence:

• pkg/pillar/volumehandlers/csihandler.go:282-294 — non-replicated branch calls
  kubeapi.FindPVC(pvcName); if it does not error, falls through to
  return true, nil (line 294).
• kubeapi.FindPVC is a plain PersistentVolumeClaims().Get() — returns true
  if the PVC object exists, with no .Status.Phase == Bound check.

So a Pending PVC that exists → Populate() reports created → CREATED_VOLUME
→ zedmanager marks the app INSTALLED and advances it to BOOTING.

The replicated branch (csihandler.go:268-280) already loops on
WaitForPVCReady; only the non-replicated branch is existence-only.

Related: pkg/pillar/cmd/volumemgr/updatestatus.go:477-478 sets
CREATED_VOLUME immediately for IsReplicated volumes with no local readiness
check (delegated to the owner node).

Not the cause

The fresh create-worker path is fine: CreateVolume() →
kubeapi.RolloutDiskToPVC (pkg/pillar/kubeapi/vitoapiserver.go:217) runs
virtctl image-upload and then waitForPVCUploadComplete, both of which require
the PVC to provision/bind and the data to upload. A volume created through that
path genuinely had a bound PVC.

Suggested fix

In Populate()'s non-replicated branch, require Bound rather than mere
existence — check .Status.Phase == corev1.ClaimBound after FindPVC (return
false, nil while Pending), or reuse WaitForPVCReady as the replicated
branch does. Consider the same Bound gate for the IsReplicated immediate-set
at updatestatus.go:477 for consistency.

Impact

App stuck in BOOTING with no EVE-side error whenever the PVC can't bind (e.g.
longhorn not yet healthy / no StorageClass) — masks the real failure as a
"booting" state instead of a volume/storage error.