feat: support Cloud Hypervisor as a microVM provider - #13
Merged
Merged
Conversation
Add MICROVM_PROVIDER (firecracker | cloudhypervisor) to select the flintlock VM provider per run. The spec builder now sets MicroVMSpec.provider, and each host installs both firecracker and cloud-hypervisor so flintlockd registers both providers. Cloud Hypervisor often needs a PVH-capable kernel, so add optional MICROVM_CH_KERNEL_IMAGE / MICROVM_CH_KERNEL_FILENAME overrides that fall back to the firecracker kernel when unset. Wire the new knobs through .env.example, the e2e workflow dispatch inputs, and the README, and cover provider parsing / validation / kernel resolution with offline tests.
There was a problem hiding this comment.
Pull request overview
Adds Cloud Hypervisor as an alternative microVM provider in the flintlock acceptance test suite, allowing runs to exercise microVM lifecycle against either Firecracker (default) or Cloud Hypervisor via an env switch.
Changes:
- Introduces
MICROVM_PROVIDERwith validation and wiresMicroVMSpec.providerinto spec generation. - Adds optional Cloud Hypervisor-specific kernel overrides (
MICROVM_CH_KERNEL_IMAGE/MICROVM_CH_KERNEL_FILENAME) with fallback to the base kernel when unset. - Updates host bootstrap, docs, CI workflow inputs, and adds offline tests covering provider parsing and kernel override resolution.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_config.py | Adds offline tests for provider defaulting, case-insensitive parsing, override behavior, and invalid provider validation. |
| tests/test_cleanup.py | Extends spec-builder offline tests to assert provider propagation and CH kernel override wiring. |
| README.md | Documents the new hypervisor provider switch and CH kernel override knobs. |
| liquidmetal_at/flintlock/spec.py | Sets MicroVMSpec.provider and uses provider-aware effective kernel image/filename. |
| liquidmetal_at/config.py | Adds provider/override config fields, effective-kernel properties, and provider validation in load(). |
| liquidmetal_at/bootstrap/templates/provision_host.sh.j2 | Installs Cloud Hypervisor and passes --cloudhypervisor-bin to flintlockd so both providers can be registered. |
| .github/workflows/e2e.yml | Adds workflow-dispatch inputs/env wiring for provider selection and CH kernel overrides. |
| .env.example | Documents and adds the new provider and CH kernel override environment variables. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+40
to
+43
| # Always install Cloud Hypervisor too, so flintlockd can register both providers | ||
| # and the per-VM MicroVMSpec.provider field selects the hypervisor at create time. | ||
| # 'all' installs firecracker only, so this runs regardless of the branch above. | ||
| ./hack/scripts/provision.sh cloudhypervisor || true |
Complete the Cloud Hypervisor provider support so `make test` passes end-to-end (20/20) with MICROVM_PROVIDER=cloudhypervisor. Provider fixes: - spec.py: force ds=nocloud on CH guests (CH delivers cloud-init only via the FAT32 cidata NoCloud disk; no MMDS/cmdline datasource hint like FC), and set a unique per-VM guest_mac so flintlock's netplan binds the NIC by MAC instead of name — FC names its NIC eth1 (virtio-mmio, pci=off) but CH is PCI-based (ens4), so match-by-name never binds and the guest is unreachable. - provision_host.sh.j2: symlink cloud-hypervisor-static -> cloud-hypervisor so flintlockd's --cloudhypervisor-bin can exec the installed asset. flintlock v0.10.0 pin (fixes upstream #999 socket churn/orphaning): - .env FLINTLOCK_REF drives the clone; host.py derives a version from a semver ref and provision_host.sh.j2 exports FLINTLOCK so provision.sh installs that exact flintlockd release deterministically, not `latest`. v0.10.0's provider.State() maps CH "Created" -> Running, stopping the reconcile re-fire that spawned a second CH on the same api-socket. Diagnostics/artifacts: - logs.py: ch-sockets-999 probe (sock->PID via fuser + per-VM "already in use" grep-count) and collect_vm_consoles() snapshot before a test's finally deletes its VMs. - remote/ssh.py: drain stdout before recv_exit_status() to avoid a paramiko deadlock on large guest-console output. - docs/flintlock-999-ch-reconcile.md: #999 root-cause write-up.
Comment on lines
+26
to
+31
| {% if flintlock_version|default('') %} | ||
| # Pin the flintlockd release provision.sh installs (it reads ${FLINTLOCK:-latest}). Keeps the | ||
| # version under test deterministic instead of tracking whatever the latest GitHub release is. | ||
| export FLINTLOCK={{ flintlock_version }} | ||
| echo "== pinning flintlockd release to $FLINTLOCK ==" | ||
| {% endif %} |
Comment on lines
+96
to
+99
| try: | ||
| logs.collect_vm_consoles(config, cluster.droplets, tag="ssh-reachability") | ||
| except Exception: # noqa: BLE001 | ||
| pass |
Comment on lines
216
to
+222
| _, fclogs, _ = ssh.run( | ||
| "for f in $(find /var/lib/flintlock -name firecracker.log 2>/dev/null); do " | ||
| 'echo "== $f =="; cat "$f"; echo; done', | ||
| "for f in $(find /var/lib/flintlock \\( -name '*.log' -o -name '*.stderr' " | ||
| "-o -name '*.stdout' \\) 2>/dev/null); do " | ||
| 'echo "== $f =="; head -c 262144 "$f"; echo; done', | ||
| check=False, | ||
| ) | ||
| (out / f"host{i}-firecracker.log").write_text(fclogs) | ||
| (out / f"host{i}-vm-hypervisor.log").write_text(fclogs) |
Comment on lines
+79
to
+86
| # Drain both streams BEFORE reading the exit status. recv_exit_status() blocks until | ||
| # the remote command exits, but a command that writes more than the channel window | ||
| # (~a few MB — e.g. cat-ing a large cloudhypervisor.stdout) blocks on write until we | ||
| # read, so calling recv_exit_status() first deadlocks. read() drains to EOF, then the | ||
| # exit status is already available. | ||
| out = stdout.read().decode(errors="replace") | ||
| err = stderr.read().decode(errors="replace") | ||
| rc = stdout.channel.recv_exit_status() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Let a run exercise flintlock microVM creation against Cloud Hypervisor as well as Firecracker, and make the CH path pass e2e.
Changes
MICROVM_PROVIDERenv (firecrackerdefault |cloudhypervisor). The spec builder setsMicroVMSpec.provider; invalid values fail fast in config.MICROVM_CH_KERNEL_IMAGE/MICROVM_CH_KERNEL_FILENAMEoverrides (CH needs a PVH-capable kernel; the liquidmetalcloudhypervisor-kernel-k8simage shipsvmlinux.binat root, notboot/vmlinux); fall back to the firecracker kernel when unset.flintlockdgains--cloudhypervisor-bin. Symlinkcloud-hypervisor-static→cloud-hypervisorso the flag can exec the installed release asset.ds=nocloud(CH delivers cloud-init only via the FAT32 cidata disk, no MMDS/cmdline hint like FC) and set a unique per-VMguest_macso flintlock's netplan binds the NIC by MAC. FC names its NICeth1(virtio-mmio,pci=off) but CH is PCI-based (ens4), so match-by-name never binds → guest unreachable.State()misclassified CHCreatedasPending, re-firing the create step onto the same api-socket → orphaned socket → VM never reaches CREATED, ~1-in-3 flake). v0.10.0'sState()mapsCreated→Running..envFLINTLOCK_REFdrives the clone;host.pyderives a version from a semver ref andprovision_host.sh.j2exportsFLINTLOCKsoprovision.shinstalls that exactflintlockdrelease deterministically, notlatest.logs.pych-sockets-999probe (sock→PID viafuser+ per-VM "already in use" count) andcollect_vm_consoles()snapshot before a test'sfinallydeletes its VMs;remote/ssh.pydrains stdout beforerecv_exit_status()to avoid a paramiko deadlock on large guest-console output.docs/flintlock-999-ch-reconcile.mdroot-cause write-up..env.example,e2e.ymldispatch inputs,README.md.Verification
make lint— clean.test_cleanup.py,test_config.py) — green.MICROVM_PROVIDER=cloudhypervisor+FLINTLOCK_REF=v0.10.0— 20/20 passed (373s). SSH-reachability (guest fs + egress) green on both nodes.