Repository navigation
Conversation
HULL_HVI_ROOTFS_UPPER selects how an hvi container boot gets a writable root. Unset keeps the per-instance APFS clone. "virtiofs" and "block" share the cached image read-only as the lower and give the instance its own overlay upper: a directory exported as the writable tag rootfs-upper, or a sparse 16G ext4 image attached as the guest's disk. Nothing is copied before the VMM starts. The upper is created on first boot and reused after; mke2fs is told the fresh sparse file is already zeroed, so it writes only metadata. vz-init picks the upper from /urunc-rootfs-upper. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
urunc moves to a58f31a on feat/multi-disk-darwin, one commit on top of
feat/initrd-hvi-backend-v0.8.0 6c062f3:
- ExecArgs gains BlockDevs, an ordered list of BlockDevSpec{Path,
ReadOnly}. BlockDevPath stays as the single writable disk when
BlockDevs is empty.
- The hvi and vz darwin builders emit --disk or --disk-ro per entry, in
order. On vz these sit outside the --rootfs chain, so a root share is
no longer dropped when disks are present.
- Both builders refuse an empty or duplicate disk path.
Linux hvi.go is unchanged. Callers that set only BlockDevPath get the
same argv as before.
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
hvi-vmm moves to 1fe6a49, brig-sh/hvi-vmm feat/multi-disk on main 4562f5b. It includes main from c53a857 up to 4562f5b: the vCPU lifecycle and kick signal shared across backends, and each device's interrupt line raised under its lock. - --disk and --disk-ro attach up to four disks, each its own virtio-mmio device, in argv order. A fifth disk fails the boot. - A read-only disk is opened O_RDONLY and advertises VIRTIO_BLK_F_RO; writes get VIRTIO_BLK_S_IOERR. - GET_ID answers disk<N> by argv position, so the guest finds its disks in /sys/block/vd*/serial. - Unknown requests get VIRTIO_BLK_S_UNSUPP. - virtio-fs is capped at 16 shares on macOS; a 17th fails the boot. A single --disk boots as before. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
- --disk <img> and --disk-ro <img>, repeatable, append virtio-blk devices after the --rootfs/EFI disk, with blockDeviceIdentifier disk<N> counted over --disk* only. --disk-ro attaches read-only. - Checkpoint clones each writable --disk to disk<N>.img and lists all of them under "disks" in latest.json; read-only ones are recorded by path. Restore puts the writable clones back and fails when one is missing. - Without --disk* the devices and the manifest are unchanged. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
ensureLowerRootfs builds <image dir>/rootfs.ext4 from the store rootfs with the block path's mke2fs -d, debugfs ownership pass and repair, factored out of buildBlockRootfs. No journal, root owned by 0:0, sized from the block-rounded contents plus 20%, mode 0400. The build runs under an flock on rootfs.ext4.lock into a temporary file renamed into place, and is redone when the image is missing or older than the image's unpack-schema stamp. The per-instance block rootfs is built as before. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
On hvi and vz the container root becomes an overlay of two disks: the image's cached read-only ext4 as disk0 and the instance's sparse 16 GiB ext4 upper as disk1, passed as ExecArgs.BlockDevs in that order. - /urunc-rootfs-lower (block:disk0:ext4) and /urunc-rootfs-upper (block:disk1) are staged into the instance initrd. - No rootfs share is exported; home and project shares are unchanged. - rdinit stays /vz-init. - Checkpoint refuses an overlay-block instance by name. The mode refuses HULL_HVI_ROOTFS_UPPER and --rootfs-type block, and is ignored with a warning outside a container boot. With neither env set the boot is unchanged. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
- cli.md lists both variables. - storage.md adds the overlay rows to the root filesystem table, the cached lower and the instance upper to the store layout, and how the lower is built and invalidated. - checkpoint-restore.md lists the vz-runner --disk clones. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
lowerRootfsGeometry added a file's size once per name, so busybox with its hundreds of applet links sized a 1 MiB image at 657 MiB. Count a file once per device and inode. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
A guest supervisor that runs several images in one VM can take each image's cached read-only ext4 as a disk instead of a virtio-fs share. The disks follow the root's pair in argv order, so their serials are fixed, and <instance>/disks.json lists them. Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HULL_ROOTFS_MODE=overlay-block(#117) gives the root image a read-only ext4lower and a writable upper disk. A caller that runs containers inside the VM,
like Briglet's guest supervisor, wants the same for each container image:
boot faster, write to disk instead of guest memory, and stat at local-disk
speed. This adds
--image-diskto attach extra image lowers, and fixes thelower sizing, which counted every hard link as a separate file.
This branch stacks on two pieces of open work, so it is opened against the
gateway branch (#119):
content (one hand merge in
docs/storage.md, where main addedrootfs-entriesto the store layout);Merge #117 and #119 first; this then reduces to two commits.
Measured standalone on an M-series Mac (busybox base, one container, n=3):
boot to container running drops from 0.47 s to 0.30 s for busybox and from
0.49 s to 0.32 s for nginx:alpine; a 100 MB write in a container with a 64 MB
memory limit fails with ENOSPC on the tmpfs upper and succeeds on the disk
upper, costing 3 MB of guest memory instead of 67. A Kubernetes v1.37
conformance run on Briglet with this mode passes 401 of 433 tests.
Related issues
Refs #117, #119
Changes
--image-disk IMAGE(repeatable): pull per--pulland--platform, buildor reuse
<image>/rootfs.ext4, attach it read-only. Serials follow argvorder:
disk0/disk1for the overlay-block root pair, thendisk2,disk3for images; without an overlay-block root they start atdisk0.hvi takes four disks, so two images fit next to the root pair; more are
refused before any pull.
<instance>/disks.jsonrecords serial, role, path, read-only flag, imageand digest for every disk, so the guest and the caller agree on the map.
--rootfs-type block, andHULL_HVI_ROOTFS_UPPERset together with--image-disk.hundreds of applet links to one binary, goes from 657 MB to 72 MB.
docs/storage.mddescribes the flag, the serials and the manifest.Checklist
make testpassesmake macosbuilds hull, vz-runner and hvi, if Go, Swift or Rust code changedtest/*.py) for changes touching boot, console, or the run path (pending; exercised live through Briglet's conformance run)docs/)