EVE-k: Longhorn declared ready before the node's instance-manager can serve volumes

[eriknordmark]

EVE-k declares Longhorn ready before the node's instance-manager can serve volumes, so the first app's volume attach fails for minutes.

Symptom

On a freshly-converted (or freshly-installed) EVE-k node, WaitForKubernetes returns and the Longhorn StorageClass is present, so app deployment proceeds — but the first volume cannot attach. The app's CDI upload pod repeats:

Warning  FailedAttachVolume  pod/cdi-upload-<uuid>-pvc-0
  AttachVolume.Attach failed for volume "pvc-<uuid>" :
  rpc error: code = Aborted desc = volume pvc-<uuid> is not ready for workloads

while the node's instance-manager pod is still coming up:

longhorn-system  instance-manager-<hash>  0/1  ContainerCreating  20m

It self-resolves once the pod runs. This is a readiness-accuracy problem — storage is declared ready too early — not a hang.

Why it takes so long

The instance-manager pod is pulling longhornio/longhorn-instance-manager:v1.9.1, which is 441,724,959 bytes — by a wide margin the largest image in the Longhorn set. Measured on-device:

Pulling  image "longhornio/longhorn-instance-manager:v1.9.1"
Pulled   image "longhornio/longhorn-instance-manager:v1.9.1" in 9m51.194s
         Image size: 441724959 bytes

So the readiness signal fires while a ~440 MB pull is still in flight. On slower storage the same pull took over 20 minutes.

Root cause

checkLonghornReady() (pkg/pillar/kubeapi/kubeapi.go:276) gates only on DaemonSets. It builds a fixed expectation set:

var lhExpectedDaemonsets = map[string]bool{
	"longhorn-manager":    false,
	"longhorn-csi-plugin": false,
	"engine-image":        false,
}

and requires one Running+Ready pod per DaemonSet on this node.

The instance-manager is not a DaemonSet — it is a pod owned by an InstanceManager CR — so this sweep cannot observe it. Longhorn runs a volume's engine and replica processes inside that pod, so no volume can be served until it is running.

checkLonghornReady is what WaitForKubernetes (kubeapi.go:180) waits on (call at kubeapi.go:249), which is why the gap is visible to every consumer of that wait. Other callers: kubeapi.go:377, descheduler.go:95 (IsDeschedulerReady), vitoapiserver.go:108.

The shell path has the same gap: Longhorn_is_ready() in pkg/kube/longhorn-utils.sh:163 checks that all DaemonSets are ready ("$lhStatus" != "truetruetrue"), that EdgeNodeInfo exists, that nodes.longhorn.io/<node> exists, and that the engine image is deployed per engineimage.status.nodeDeploymentMap — but never looks at the instance-manager.

Suggested fix

Require a running InstanceManager for this node, in addition to the DaemonSet sweep. Everything needed is already vendored in pillar — no new dependency:

• github.com/longhorn/longhorn-manager/k8s/pkg/apis/longhorn/v1beta2 defines InstanceManagerStateRunning = InstanceManagerState("running"), InstanceManagerSpec.NodeID, and InstanceManagerStatus.CurrentState.
• The vendored versioned clientset exposes LonghornV1beta2().InstanceManagers(namespace) (.../typed/longhorn/v1beta2/longhorn_client.go:96).
• SetLonghornNodeDrainPolicy in longhornconfig.go already shows the construction pattern: gate on longhornAPIExists(), then GetKubeConfig() + versioned.NewForConfig(config).

So the added check is roughly: list InstanceManagers in longhorn-system, and require at least one whose Spec.NodeID matches this node and whose Status.CurrentState == InstanceManagerStateRunning.

Two properties worth stating, because they are what make this safe:

• Longhorn creates the InstanceManager CR eagerly during node setup, not lazily on first volume request (verified on-device: the CR is running with no volumes present). So waiting on it cannot deadlock against a volume that is itself gated on storage readiness.
• The check is per-node, matching the existing per-node DaemonSet check, so it does not couple this node's readiness to other nodes in a cluster.

The equivalent gate belongs in Longhorn_is_ready() for the shell path.

Prior art in-tree

The Go kube-init rewrite (#5971) already gates on exactly this. Its Longhorn readiness requires all three of:

longhornDaemonSetsReady(waitCtx) && longhornNodeExists(waitCtx) &&
	longhornInstanceManagerRunning(waitCtx)

with longhornInstanceManagerRunning matching spec.nodeID and status.currentState == "running". That covers kube-init's own bring-up, but pillar's checkLonghornReady is a separate path and still has the gap — which is why the symptom above is still observable on a build carrying #5971.

How it was found

Observed on both ZFS legs of a 7-leg kvm→EVE-k conversion matrix (all 7 legs passed; this cost wall-clock time, not correctness):

leg	instance-manager	step (c) "app RUNNING"
twodisk-zfs	ContainerCreating 20+ min	~21 min
zfs-grow	pull 9m51s	~11 min

In both cases the escript's "volumemgr ready (incl kubeapi.WaitForKubernetes)" and "Longhorn StorageClass ready" steps passed while the instance-manager pod was still in ContainerCreating.