Skip to content

feat(run): Add HULL_ROOTFS_MODE=overlay-block - #117

Draft
ananos wants to merge 7 commits into
mainfrom
feat/overlay-block-rootfs
Draft

feat(run): Add HULL_ROOTFS_MODE=overlay-block#117
ananos wants to merge 7 commits into
mainfrom
feat/overlay-block-rootfs

Conversation

@ananos

@ananos ananos commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Each container boot clones the image rootfs with APFS and shares the
clone read-write over virtio-fs. The clone costs 1-6 s per launch and
0.3-1.5 s per rm, and on the vz backend the virtio-fs root can never
carry setuid, so sudo does not work there.

This PR adds HULL_ROOTFS_MODE=overlay-block. The root becomes an
overlay of a read-only ext4 image built once per image digest (lower)
and a per-instance sparse 16 GiB ext4 (upper), both attached as
virtio-blk; the home and project shares stay on virtio-fs. The initrd
finds the two disks by serial (disk0, disk1), so it needs an initrd
that reads the markers: NOFireAI/hull-assets#26, or urunit's single
binary (NOFireAI/urunit stack/5-overlay-block). hvi-vmm moves to main,
where brig-sh/hvi-vmm#276 landed (--disk path=..,readonly=on,serial=..,
--fs). The urunc pin moves to urunc-dev/urunc feat/urunit-single-binary
(770a597): the darwin series with the ordered disk list (#1094, ported)
and urunit's single binary on top.

Measured on claude-code-stock (n=5): ready 0.32 s vs 3.46 s for the
clone, hull rm 0.03 s vs 0.84 s, warm stat of the whole root 91 ms vs
710 ms, host footprint 553 MiB vs 1579 MiB. sudo -n true works as the
image user on vz. The default with the variable unset is unchanged, and
HULL_HVI_ROOTFS_UPPER keeps its behavior.

Out of scope here: hull checkpoint/restore refuse an overlay-block
instance (vz-runner itself handles the two disks); DISCARD, so the upper
only grows until rm; the store unpack-schema bump; Linux.

Related issues

Depends on urunc-dev/urunc feat/urunit-single-binary and
NOFireAI/hull-assets#26 or NOFireAI/urunit stack/5-overlay-block.
brig-sh/hvi-vmm#276 is merged.

Changes

  • chore(deps): urunc v0.8.2-0.20261018212029-770a597f0a1d
    (feat/urunit-single-binary); hvi-vmm submodule 590c45d (main).
  • feat(vz-runner): --disk / --disk-ro, repeatable, after the
    rootfs/EFI disk, blockDeviceIdentifier = disk<N>; checkpoint clones
    only writable disks and restore fails if a clone is missing.
  • feat(run): cmd/hull/lowerrootfs.go builds <image dir>/rootfs.ext4
    (mke2fs -d, no journal, the existing debugfs ownership pass, flock,
    tmp + rename, rebuilt when older than unpack-schema); blockrootfs.go
    split into mkfsFromDir + applyBlockFixups with a staging dir.
  • feat(run): HULL_ROOTFS_MODE=overlay-block for container boots on
    hvi and vz; BlockDevs = [lower ro, upper rw]; markers
    /urunc-rootfs-lower = block:disk0:ext4, /urunc-rootfs-upper =
    block:disk1; refused together with HULL_HVI_ROOTFS_UPPER; a
    warning when set outside a container boot.
  • Docs: docs/cli.md, docs/storage.md, README.

Checklist

  • make test passes
  • make macos builds hull + vz-runner, if Go or Swift code changed
  • I have added or updated tests covering the change
  • I have run the PTY / shared-folder e2e harnesses (test/*.py) for changes touching boot, console, or the run path. Not run; boot smoke on hvi and vz with the new initrd instead.
  • I have updated the affected docs (README, docs/)

LLM usage

Claude Code (Opus 4.8 for the implementation, a second model for
review). Review pending on my side.

@ananos
ananos force-pushed the feat/overlay-block-rootfs branch from dc2d786 to 320a0a8 Compare October 9, 2026 13:20
This was referenced Oct 9, 2026
Merged
Draft
Draft
ananos added 7 commits October 19, 2026 00:41
@ananos
feat(run): Add overlay upper layers for hvi roots
219dd08 
HULL_HVI_ROOTFS_UPPER selects how an hvi container boot gets a writable
root. Unset keeps the per-instance APFS clone. "virtiofs" and "block"
share the cached image read-only as the lower and give the instance its
own overlay upper: a directory exported as the writable tag
rootfs-upper, or a sparse 16G ext4 image attached as the guest's disk.
Nothing is copied before the VMM starts.

The upper is created on first boot and reused after; mke2fs is told
the fresh sparse file is already zeroed, so it writes only metadata.
vz-init picks the upper from /urunc-rootfs-upper.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
chore(deps): Pin urunc to the darwin series with the single binary
d7f2357 
urunc moves to 770a597 on feat/urunit-single-binary. That is the darwin
series (feat/hvi-darwin-linux-parity 71c4d2d, on
feat/unchanged_containers-exec-fixes) with urunit's single binary on
top. hull leaves the v0.8.0 line it tracked so far.

For the disks of an overlay-block root:
- ExecArgs gains BlockDevs, an ordered list of BlockDevSpec{Path,
  ReadOnly}. BlockDevPath stays as the single writable disk when
  BlockDevs is empty.
- hvi gets one --disk path=<p>[,readonly=on],serial=disk<N> per entry,
  the form hvi-vmm takes since its multi-disk support landed. Shares go
  out as --fs tag=<t>,path=<p>[,readonly=on][,uid=..,gid=..].
- vz gets --disk or --disk-ro per entry, after the root share, so a
  root share is no longer dropped when disks are present.
- Both refuse an empty or a duplicate disk path; hvi refuses a path
  with a comma.

The newer urunc requires newer modules than hull did, and minimum version
selection takes them: urfave/cli v3.13.0, logrus v1.10.2, the
golang.org/x modules, grpc v1.84.0 and their dependencies.

urunc now writes urunit.conf as NUL-terminated URUNIT1 records. An
image that boots its own urunit (the BuildUrunitConfig paths in run.go)
needs a urunit from NOFireAI/urunit da5e62f or later. An older one runs
the command without its environment, user and working directory.
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
chore(deps): Bump hvi-vmm to main for disks and --fs
1ba952f 
hvi-vmm moves to 590c45d, brig-sh/hvi-vmm main, from c53a857. For this
branch:

- --disk path=<p>[,readonly=on|off][,serial=<id>] attaches up to four
  disks, each its own virtio-mmio device. A fifth fails the boot.
- A read-only disk is opened O_RDONLY and advertises VIRTIO_BLK_F_RO;
  writes get VIRTIO_BLK_S_IOERR. GET_ID answers the serial, so the guest
  finds its disks in /sys/block/vd*/serial.
- --fs tag=<t>,path=<p>[,readonly=on|off][,uid=..,gid=..] names a
  virtio-fs export and its owner.
- arm64 guests get the host's wall-clock time from a PL031 RTC.

It also brings main's refactors since c53a857: the vCPU lifecycle and
kick signal shared across backends, and a run that owns its stop.
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
feat(vz-runner): Add --disk and --disk-ro block devices
254f256 
- --disk <img> and --disk-ro <img>, repeatable, append virtio-blk
  devices after the --rootfs/EFI disk, with blockDeviceIdentifier
  disk<N> counted over --disk* only. --disk-ro attaches read-only.
- Checkpoint clones each writable --disk to disk<N>.img and lists all
  of them under "disks" in latest.json; read-only ones are recorded by
  path. Restore puts the writable clones back and fails when one is
  missing.
- Without --disk* the devices and the manifest are unchanged.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
feat(run): Cache a read-only ext4 image per image digest
d6c4bb3 
ensureLowerRootfs builds <image dir>/rootfs.ext4 from the store rootfs
with the block path's mke2fs -d, debugfs ownership pass and repair,
factored out of buildBlockRootfs. No journal, root owned by 0:0, sized
from the block-rounded contents plus 20%, mode 0400.

The build runs under an flock on rootfs.ext4.lock into a temporary file
renamed into place, and is redone when the image is missing or older
than the image's unpack-schema stamp. The per-instance block rootfs is
built as before.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
feat(run): Add HULL_ROOTFS_MODE=overlay-block for container boots
492d9f1 
On hvi and vz the container root becomes an overlay of two disks: the
image's cached read-only ext4 as disk0 and the instance's sparse 16 GiB
ext4 upper as disk1, passed as ExecArgs.BlockDevs in that order.

- /urunc-rootfs-lower (block:disk0:ext4) and /urunc-rootfs-upper
  (block:disk1) are staged into the instance initrd.
- No rootfs share is exported; home and project shares are unchanged.
- rdinit stays /vz-init.
- Checkpoint refuses an overlay-block instance by name.

The mode refuses HULL_HVI_ROOTFS_UPPER and --rootfs-type block, and is
ignored with a warning outside a container boot. With neither env set
the boot is unchanged.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
docs: Document HULL_ROOTFS_MODE and HULL_HVI_ROOTFS_UPPER
607cc45 
- cli.md lists both variables.
- storage.md adds the overlay rows to the root filesystem table, the
  cached lower and the instance upper to the store layout, and how the
  lower is built and invalidated.
- checkpoint-restore.md lists the vz-runner --disk clones.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
ananos force-pushed the feat/overlay-block-rootfs branch from 320a0a8 to 607cc45 Compare October 18, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ananos