HV=k: container app data volumes not mounted at MountDir when the app has no cloud-init user-data

[eriknordmark]

Summary

On HV=k (kubevirt), a container app that has an additional data volume with a
MountDir set (e.g. --mount …,dst=/data) does not get that volume mounted at its
MountDir — unless the app also carries cloud-init user-data. Without cloud-init the
volume is attached to the shim VMI as a raw block device (/dev/vdb) but never mounted at
/data. The volume's data is fully intact; it is simply not mounted.

Root cause

In pkg/pillar/cmd/domainmgr/domainmgr.go, configToStatus() (added in 6016fc51b,
"Pass cloud-init env vars and mount volumes on HV=k shim VMs"), the #EVE_VOLMOUNT
marker generation is nested inside the cloud-init gate:

if config.IsCipher || config.CloudInitUserData != nil {   // <-- gate
    ... fetchCloudInit / parse WriteFiles / EnvVariables / FML ...
    if status.OCIConfigDir != "" {                         // container
        ...
        if ctx.hvTypeKube && config.VirtualizationMode != types.NOHYPER {
            mountUserData := mountDirsToUserData(status.DiskStatusList)   // #EVE_VOLMOUNT markers
            envUserData   := envVarsToUserData(status.EnvVariables)
            userData := mountUserData + envUserData
            if userData != "" {
                // append the cloudinit-nocloud disk that the shim initrd reads
            }
        }
    }
}

On HV=k the shim VMI's eve-external-boot-image initrd rewrites /mnt/mountPoints from the
#EVE_VOLMOUNT lines on the cloudinit-nocloud disk, and pkg/xen-tools/initrd/mount_disk.sh
then maps each /dev/vd* to its MountDir and mounts it. With no cloud-init user-data,
config.CloudInitUserData == nil, the whole block is skipped, no cloudinit-nocloud disk is
added, /mnt/mountPoints stays empty, and mount_disk.sh leaves the volume unmounted.

The mount marker depends only on the disks' MountDir, not on cloud-init — so coupling it to
the cloud-init gate is the bug.

Reproduce

Deploy a container app on an EVE-k node with a data volume and no metadata:

eden pod deploy docker://lfedge/eden-eclient:<tag> -n ctrapp -p 2224:22 \
     --mount=src=file://<eden.root>/data.qcow2,dst=/data

On the device: /data does not exist in the container and the volume is an unmounted
/dev/vdb. Adding any --metadata=… (so CloudInitUserData != nil) makes /data mount.

Confirmed on the device for the failing case:

• MountDir=/data is set by the controller and present in DomainStatus.DiskStatusList
  (disk = Devtype:"hdd" MountDir:"/data"), VirtualizationMode != NOHYPER;
• no cloudinit-nocloud disk in DiskStatusList;
• /dev/vdb is a valid ext4 filesystem with the app's data intact, just not mounted.

Discovered via the kvm→k app-volume-migration test (update_eve_image_kvm_to_k_volmig)
extended to migrate a container app's data volume: the volume migrates to a Bound Longhorn
PVC and the data survives, but it is not mounted at MountDir post-conversion.

Proposed fix

Move the container mount-marker block (mountDirsToUserData + appending the
cloudinit-nocloud disk) out of the if config.IsCipher || config.CloudInitUserData != nil
gate, so it runs whenever status.OCIConfigDir != "" && ctx.hvTypeKube && config.VirtualizationMode != types.NOHYPER. Keep the cloud-init parsing
(fetchCloudInit, WriteFiles, EnvVariables, FML, cipher) gated as-is; build a single
cloudinit-nocloud disk that combines mountUserData (always) with envUserData (only
populated when cloud-init is present).

Why this is low-risk (investigated)

• The cloudinit-nocloud disk is attached as a SATA CDROM (hypervisor/kubevirt.go,
  CDRom{Bus:"sata"} → /dev/sr*), not a /dev/vd*, so adding it does not perturb the
  /dev/vd[b,c,…] ordering that mount_disk.sh maps #EVE_VOLMOUNT lines against.
• status.EnvVariables is sourced only from cloud-init, so it is naturally empty when there
  is no cloud-init — envUserData is "" and no env is lost.
• There is a single append site for the cloudinit-nocloud disk, so a correct restructure
  adds it exactly once (combining mount + env).
• VM apps (OCIConfigDir == "") and NOHYPER containers (native pod, mounted via the k8s
  VolumeMount{MountPath: MountDir} path in CreateReplicaPodConfig) are guarded
  independently of cloud-init and are unaffected.

Impact

Container apps on EVE-k that have data volumes but no cloud-init user-data: their volumes are
attached but not mounted at MountDir. Data is not lost.