Repository navigation
Conversation
clearStaleUnixSocket treats ECONNREFUSED as "a socket nobody is listening on" and removes the path before binding. On macOS a connect to a regular file fails with ENOTSOCK, which falls through to the refusal, so the existing TestClaimUnixSocketRefusesANonSocketPath passes there. Linux answers the same connect with ECONNREFUSED, so on a Linux host the gateway deleted whatever regular file sat at the path it was told to bind. The test fails on Linux without this change. Lstat the path on ECONNREFUSED and refuse unless it is a socket. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
5 of 6 tasks
pmoust
force-pushed
the
feat/teleport
branch
5 times, most recently
from
October 20, 2026 16:02
8fef906 to
e498e3a
Compare
runGateway made a netgw.Network and never closed its gvisor stack, so the stack's TCP workers outlived the gateway. A gateway runs in its own process, where that cost nothing, but one cmd/hull test runs two of them inside the test binary. Under -race their 30 idle workers spin on about eight cores for the rest of the package, which made every later test ten times slower: the streamed move tests broke their wall-clock bounds and the race step hit its 10 minute package timeout. netgw.Network now has Close, which closes the stack, and runGateway defers it. Close does not wait for the endpoints, since a member still attached at shutdown could open a connection that nothing aborts, and the wait would then hold the gateway's exit. Both packages' tests check that no TCP worker is left behind. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A doubled blank line there since the first commit made `gofmt -l` list the file, so the gofmt gate could not pass on the whole tree. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
cmd/hull is darwin-only because a handful of helpers in otherwise portable files call macOS interfaces: the termios ioctl numbers, the hdiutil store mount, the ps-based process identity check, the kern.procargs2 and kern.osproductversion sysctls, Stat_t.Ctimespec, Statfs_t.Fstypename, the Homebrew e2fsprogs paths, and urunc's darwin-only VzVmm and VzDarwin symbols. Moving each behind a small _darwin.go file is what lets the shared files build for Linux in the next change, with a Linux counterpart beside each one. Nothing changes on macOS. The moved code is the same code, with three small differences. processIsAVMM and reapOrphanVMM now read process facts through inspectProcess and listProcesses: the ps parsing they did inline lives in proc_darwin.go, and the identity rules (base name, recorded argv, start-time slack) stay in stop.go, where both platforms share them. e2fsck is named by a constant like mke2fs and debugfs instead of an inline Homebrew path, and buildBlockRootfs checks that all three exist before mke2fs writes a 15 GiB image. A missing debugfs used to surface only after the image was written, as an error about the wrong step. The command list and usage line come from commands_darwin.go. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Brig's teleport plan needs a destination runtime on Linux/KVM that drives hvi the way hull does on a Mac. cmd/hull did not compile for Linux: the untagged files called darwin-only helpers. The shared files drop their darwin tag and each helper moved into a _darwin.go file gets a Linux counterpart: termios through TCGETS and TCSETS, process identity from /proc, and a plain store directory in place of the hdiutil mount. hull builds the Linux hvi argv itself instead of using urunc's Linux monitor, which is written for the containerd shim and passes no --agent-sock, --net-gateway or --net-mac. hviLinuxArgv is the darwin argv minus the virtio-fs shares, and both builds take the vz name from one constant. A darwin test fails if urunc's darwin argv grows, drops or reorders a flag for the same share-less arguments. hull on Linux offers the one configuration hvi on KVM can run: hvi with a block rootfs. It refuses other backends, virtiofs and 9pfs roots, host shares and --net shared without a gateway by name, before the store is opened or the image pulled, and it refuses an image annotated for another hypervisor instead of booting it under hvi. make linux cross-builds the CLI, CI builds, vets, lints and unit-tests hull for GOOS=linux, and backends.md and build.md describe what Linux hull does, what differs from macOS and what was tested on a Raspberry Pi 5. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Run as an ordinary user on Linux, an unpack cannot chown, so every file came out owned by the user who ran the pull and mke2fs copied that into the guest's block image. On a Raspberry Pi 5 alpine's /bin/busybox and /etc/shadow were 1000:1000 in the guest, where sudo refuses to run and a setuid binary drops its caller. The unpack now records the tar header's mode, uid and gid in the user.com.nofire.hvi.linux-attr attribute, the only namespace an unprivileged process may write, and the block image build applies the recorded owners and modes. Files keep owner read and directories owner rwx on the host, so images that ship /etc/shadow at 0000 (Fedora, RHEL and their relatives) can be read by mke2fs, and the builder writes the image's own mode into the image with debugfs. Symlinks, which take no user.* attributes, get their owners from a side record, and the ownership record itself is stripped from the image so the guest never sees it. A Linux test drives the unprivileged unpack through to the block image, and the docs say what Linux hull is for and what was checked on it. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Unlinking an entry needs write permission on its directory, and an image is free to ship directories at 0555 (Fedora has dozens under /usr/lib). The unpack puts that mode on the host, so os.RemoveAll by the owner stopped with EACCES, and hull rmi, prune, instance removal and the pull's own sweeps of old and staging directories left the tree behind. RemoveTree retries a removal that failed on a permission error after adding owner rwx to every directory, walking through an os.Root so a directory symlink in the image cannot point the chmod at a host directory. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hull could only signal an hvi VMM. Every hvi run now gets --control-sock and --profile-out in its 0700 instance directory, and the record keeps both paths. hull pause and hull resume work on an hvi instance, and hull stop asks hvi to stop before it signals. The client sends one request per connection with "v":1, since hvi serves one connection at a time, and reads the reply under a timeout longer than the pause deadline hvi gives. A refusal comes back with hvi's code, and a reply that never came is told apart from a refusal, because the op may still complete in hvi. hull ps shows a paused or fenced VM, which keeps its record at running, and hull exec refuses to dial into one instead of hanging on its agent. The control socket path is checked before the image is pulled, as the agent socket path already was. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hull checkpoint and hull restore now take an hvi instance as well as a Vz one. Both verbs move to portable files and dispatch on the record, and the Vz paths are unchanged. A checkpoint sends hvi's checkpoint op with the checkpoint's txid into a 0700 bundle directory under the --checkpoint-root every hvi run now gets, since hvi's confinement lets it create files only there. --then fence leaves the guest parked for good, the source side of a move, and --then continue resumes it. Directory roots and host shares are refused, as no bundle captures them. A restore builds hvi's command line from the one the instance last ran through an allowlist, so a flag hull does not know is refused rather than dropped, binds the restore to the bundle's manifest and txid, and can adopt a later latest.json. It waits on the restore's own VMM, sees a detached hvi that exits at once, and dials the gateway. The control flags are passed only to an hvi that takes them, because the hvi hull pins answers them with "unknown boot arg" and exits. A per-instance migration lock serialises checkpoints and restores of one instance, and a restored guest is recorded as run before anything can run it, so a second restore cannot put the bundle's RAM over a disk the guest has written since. checkpoint-restore.md becomes a page for both backends and says what an hvi bundle holds, which restore states exist, what the lock covers and what was checked live. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
`hull migrate NAME --to USER@HOST` drives a move: preflight on both hosts, reserve, fence, transfer, restore paused on the destination, then commit. The fence comes before the transfer because the bundle is the fenced checkpoint; there is no pre-copy in this form. Both hosts keep <instance>/migration.json, written through a synced rename before the step that relies on it, with a transition table per side so no record can go back past the fence or the commit. The instance record gains the migrating, migrated and incoming states and an owner epoch that hvi records with the fence, so a destination can refuse a commit meant for another move. A move's txid is 128 random bits, migrate.lock serialises its steps, and the plain verbs refuse an instance a move holds. The source frames the bundle and a sparse disk into its own stream, since archive/tar cannot write sparse entries, and runs the destination's hidden verbs (migrate-probe, migrate-receive, migrate-prepare, migrate-commit, migrate-abort) over ssh in batch mode with every argument shell-quoted. Each step finds its effect already there when run again, so `hull migrate NAME --txid T` resumes from the record. A refused probe leaves no record behind, and the probe says how to move an instance back to the host it left. migration.md describes the steps, the states, the stream, a failure table and what ran live through the commit on HVF. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A destination kept a move's record only in the instance's migration.json. When the source lost the commit's acknowledgement and the destination then moved the instance on or removed it, nothing there remembered the commit, and an abort from the source could run the guest on two hosts. The destination now keeps a ledger of the moves it committed, with the state change each entry records, and an abort that the ledger refuses says whether the instance moved on or was removed. A committing source stays fenced unless the destination confirms the abort, and takes "unknown txid" as no answer. A destination answers an abort only once nothing of the move can run there. A retry that finds the copy verified checks the disk against the fence, a second --abort finishes what the first left undone, and a txid resolves only to the instance that holds it. migration.md describes the ledger, what an abort needs, and the live runs of a lost commit acknowledgement. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hvi's portable-arm64-v1 profile shows the guest the same machine on an Apple host and on a Linux arm64 host, and its guest keeps time through the hvi-time device, so it needs a kernel built with that driver. `hull assets pull --profile portable-arm64-v1` installs that kernel as a variant of the boot bundle with its own provenance, from an OCI artifact or a local file, checks the artifact holds a kernel before it writes anything, and names the right pull when the kernel fails its check. `hull run --hypervisor hvi --profile portable-arm64-v1`, or the com.nofire.hvi.profile annotation, passes --profile to hvi and records the profile in state.json. The run boots only the profile's kernel, checked by its contents whatever names it, and refuses other backends, non-block roots, host shares, an hvi without --profile and an hvi without the control socket. A move carries the profile. The probe refuses a portable instance when the destination's hvi takes no --profile, relaxes the same-OS check to the architecture for a portable guest, and the destination checks the received bundle's profile against the move. The docs describe profiles, the kernel, what require refuses, and the moves run on the Linux pair and on this Mac. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A move sent the guest's whole disk every time, although most of it is the image hull built and both hosts could hold already. To send only the blocks the guest changed, both hosts need that image and a name for it that does not depend on where it is stored. The disk digest hashes the size and every non-zero 64 KiB block with its index, so a hole and written zeros hash alike, and a clone, a sparse copy and a received image hash the same. An hvi run with a control socket on a block rootfs now keeps its image under <store>/bases before launch, cloned where the filesystem can, records the digest and whether the VMM booted from it, and passes --disk-base-sha256 to an hvi that takes it. A restore clears the "from boot" mark, since hvi's record of dirty blocks restarts there. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A move should not spend its time carrying what the destination could have held before it started: the image hull built the disk from, the portable kernel a later boot there needs, the OCI image. On the Pis the B4 demo had to install the kernel by hand, and every move sent the whole disk. `hull migrate-prepare NAME --to USER@HOST` now puts them there while the guest keeps running: the instance's base by its disk digest, sent as lz4-compressed non-zero blocks and kept on the destination only when what arrived hashes to that digest; the profile kernel this store has installed, checked against its sha256 and installed through the same import as a local kernel; and the image, pulled there by its reference unless the store has its digest. Each step finds its effect already there and sends nothing, and --json reports what was done and what the destination holds. The probe now lists the bases and profile kernels a store holds, whether its hvi restores from a stream, and a capability (stream-v1) that a hull from before this change does not list, so a source can tell an old destination apart without a protocol bump that would make the two refuse each other. The destination's hidden step of a move keeps its name: `migrate-prepare --txid T` still brings a received checkpoint up paused. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hvi's streamed checkpoint (format 2) is read by `hvi restore --from-stream -` on its standard input, and hull had no way to give a VMM one: a detached VMM always read /dev/null, and a restore always named a bundle directory and checked its manifest first. launchVMM now takes the file a detached VMM reads as stdin. A restore reads a stream in two cases. A streamed move's own restore on the destination (an incoming instance whose command line restores from a stream) reads its own stdin, which `migrate-receive --stream` feeds, and starts while the move is still receiving: hvi checks every component as it reads, and the commit still waits for the move to be verified. A source that takes back a streamed move reads the stream it spooled, but only once that spool was sealed; one that never was is refused. The recorded --from-stream is dropped when the command line is mapped into the next restore, as --from is. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A whole-bundle move stops the guest for the whole transfer: 5.3 to 9.4 s for the 1 GiB ubuntu guest between the Pis, 46 to 54 s for the claude-code guest from the Mac. hvi now streams its checkpoint with RAM pre-copy rounds, so the guest only has to stop for the last one. When both hosts can, `hull migrate` runs one ssh session to `migrate-receive --stream` in the HULLMIG2 format: disk batches that differ from the base and hvi's components relayed with hvi's own headers and checked against hvi's sha256, then a trailer with a running hash and an account of every disk round that the source checks against hvi's digests. The source spools the stream and seals it, so a fenced guest whose move fails can be restored from the spool, and holds off SIGINT, SIGTERM and SIGHUP while hvi's stream is not on record. The destination accepts the header before hvi's migrate begins, so a refusal there costs no fence. The downtime fixes move every synced write they can out of the pause: the spool's and the disk's syncs, the fence record and hvi's documents, leaving one synced write per host between the seal and the commit, with writeback started as the spool and the disk grow so the host does not throttle. The last disk pre-copy round runs right before hvi's migrate. A fake hvi models hvi's restore and pre-boot writes for the tests. migration.md, cli.md and storage.md describe streamed moves, bases, migrate-prepare and what a streamed move records. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Every step of a move ran its own ssh and paid for a TCP and key exchange handshake: 0.23 s from this Mac to rp1, against 0.02 s for a command over an open connection. A streamed move runs its commit with the guest stopped, so that handshake was downtime. The steps now share one connection (ControlMaster=auto) kept for a minute after the last one. The mux socket lives in a 0700 directory only this user owns, not under a predictable name in /tmp, where another local user who made the socket first could be handed the move's stream, guest RAM included. HULL_MIGRATE_SSH_MUX=off gives every step its own connection again. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
migrate-prepare installed the source's portable kernel whenever the destination's differed. A profile run boots only the kernel its store has installed (see the machine profile commit), and brig's BRIG_PORTABLE_KERNEL must have those bytes, so preparing a host that keeps its own (newer, or built for it) kernel replaced it without a word, and the portable runs there then failed the kernel check. The prepare now sends the kernel only to a destination that has none for the profile, and reports a different one as kept. The destination verb refuses to replace an installed kernel even when asked. A move needs neither: the kernel is in the guest's RAM. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Each hvi run kept its own base, a copy of the disk it had just built with the instance's /urunit.conf (its environment and credentials) inside. mke2fs writes a fresh UUID and timestamps into every image, so no two instances shared a base and every new instance needed a whole base sent before its first move. The block rootfs is now built in two parts. The image part is built once per image digest and size and kept read-only under <store>/bases, and each instance's disk is a clone of it with the instance's own files injected. Bases are kept only while something needs them: an instance or move record that names one, an image still in the store, or a base a prepare received. hull rm removes an instance's base when nothing else needs it, and hull prune removes the rest, the entries of removed images and half-written copies. storage.md describes the per-image bases and what removes them, and migration.md gives the streamed moves measured with the hvi that can make them, with the guest's downtime taken from the rise in hvi-time's downtime total. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A streamed receive copied the base to the instance's disk before it accepted the stream. Where the filesystem has no clones (ext4 on the Pis) that copy of the 1.6 GB claude-code base took 22 s of a 28 s move. A destination now keeps a spare copy of a base it holds, made when a prepare sends the base or by the new migrate-spare-base step when the base is already there. The receive takes the spare by renaming it and copies the base only when there is none. A base's passed check is recorded beside it, so a move hashes it once. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hull drove hvi's disk-dirty and disk-blocks ops itself, and since hvi serves one control connection at a time the last disk round was read after the fence, holding everything the guest wrote during the RAM pre-copy, outside hvi's manifest and all of it downtime. hvi now sends a disk round before each RAM round and takes the final one in the pause, and its manifest covers them. The source relays that one stream as hvi writes it, and the destination hands each frame to `hvi restore --from-stream - --disk <copy> --disk-base-sha256 <base>` as it arrives. hvi puts disk rounds in the stream only for a disk booted with --disk-base-sha256. A run on an image base compares the instance's disk with the base after the injection, writes that bitmap beside the instance, and passes it as --disk-written with the base digest, so hvi's blocks written since boot include hull's own writes. A move that has to send the whole bundle says why. migration.md describes the stream-v3 move and its numbers. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
hvi fences a guest once its migrate manifest is in the socket's buffer. A hull that dies before its spool holds the whole stream now recovers the guest from the fenced hvi, which streams it again only for the txid and epoch that fenced it. hull reads the first manifest's digest from the stats of that reply and checks the second stream's documents against the first stream's, which the record of the fence now keeps. A move that a killed hull left verified on the destination commits that copy, and its summary leaves out a stream it never sent. When hull itself breaks a stream it asks hvi whether it fenced instead of saying the guest runs on, a replay counts its downtime from the first attempt's pause, and a failed second stream is never reported as a guest that runs on. migration.md records an abort through a second stream, a guest whose run wrote over its base, the fence check and a replay's pause. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A streamed move from the Mac to rp2 spent 2.9 to 3.1 s between hvi's seal and the destination's resume, against 55 to 90 ms on the Pi pair, and nothing on record said where. The destination now records when the manifest arrived, when it recorded verified and each part of its commit, the source keeps those steps with its own, and prints them as one line in milliseconds after the seal. The destination's resume is timed apart from its commit writes, the downtime counts to the destination's resume, and the pause counts from the end of hvi's I/O gate (gate_ms), which the hvi pinned at the end of this series holds with the vCPUs running. migration.md has the Mac to rp2 claude-code moves made through brig. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
On rp1 to rp2 with 100 ms and 10 Mbit/s of netem, a move spent 3043 ms from the seal to the resume, 2566 ms of it before the destination read the manifest: about 3 MB sat in hull's buffer, ssh's channel window and the kernel when hvi sealed. The commit then took a new ssh session, and the source polled migrate-status over new sessions. A destination that lists stream-session now takes `migrate-receive --stream --session` and writes accepted, acked and verified lines on stdout. The source holds what it relays to a window of the fastest recent rate times twice the round trip, measured again each second and never under 20 ms, reads hvi's stream through 64 KiB, and sends the commit in the same session. The destination waits less for that commit than migrate-commit waits for the move's lock. The destination saves the epoch with the manifest it received, and a restore polls hvi every 20 ms. migration.md and cli.md describe the session and give the moves re-run with it on the LAN and on the emulated WAN, counted from hvi's seal. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
`hull migrate --status` during a streamed move's rounds said the source was fenced while the guest still ran, and took hvi's silence during its migrate op as a guest still running, also through the seal and the fence. For a streamed move in transferring the status now asks: a fence on the record or hvi saying fenced is a fence, hvi saying running or paused is a guest that runs here, and a silent hvi gets what is known of the op's phase. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A move back to the host a guest came from found no spare there and copied the base first, which paused the guest 22.9 s on the Pi pair. Once a streamed move commits, the source starts `hull migrate-spare-base` in a session of its own, which makes the spare and then checks the base, so the move back neither copies nor hashes it with the guest paused. A spare is made once, synced, and only for a held base, under a lock that a move back's receive waits on, and a failed lock is recorded as a wait given up. The destination's steps say whether a receive took the spare or copied the base, and hull prune sweeps half-written spares. migration.md says when a move back still waits for that step. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The hvi this series pins waits up to 10 s for its I/O threads at a pause and 2 s for the vCPUs, and asks clients for a reply timeout of at least 13 s. hull gave every op 5 s, so it could report as failed a pause that then paused the guest. pause, resume, status and profile now get 15 s. hvi gives the part of a move's stream it writes with the guest paused 10 s plus twice the last running round, and hull now waits for that part only as long. hvi also closes the connection when a migrate fails after its stream began, so hull takes that close as hvi's failure and asks hvi's status after every close instead of reading frame bytes as a refusal. The stub hvi's test cleanup waits for its goroutines and closes its connections without a map. migration.md has the moves measured with that hvi (94faf8f before hvi-vmm's history was rebuilt). Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The source's window and its writes into ssh waited with no deadline, so a destination hull stopped with SIGSTOP held the move's lock and its signals until it was continued. A write to the destination now has an idle bound: hvi's own reader bound of 30 s plus a margin while hvi's migrate runs, the paused part's bound when that is shorter, and 2 minutes for a replay from the spool. migration.md measures the stalled destination with the source's cut and the claude-code move from the Mac on the LAN. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The instance's record of the fence is written on a goroutine of its own and reads hvi's reply from the move's stats, while the transfer wrote that field again once the destination answered. go test -race caught the race in about one run of 30. The transfer no longer rewrites the reply that send stored at the seal, and the replay path stores it the same way, so nothing writes the field while the record runs. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
brig now pins the five-patch portable kernel and refuses to boot any other at its default place, and a prepare left a host's older kernel in place. migrate-prepare takes --kernel-sha256, the profile kernel the destination is to hold. When this host holds that kernel and the destination another, the kernel is sent with --replace and migrate-take-kernel installs it over the other one. A destination lists the kernel-replace capability when it takes that, and one that does not is told what to run. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The move tests run their destination inside the test binary, so its probe asked the test host whether it can open /dev/kvm. The CI runners cannot, and every test that moves an instance failed there with "it cannot run a VM". The unit tests boot nothing: the probe's hypervisor check is now a variable, and the Linux TestMain answers it as a host that can. The real check moves to hostHypervisorAvailable on both platforms. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The crash tests kill a hull once hvi has fenced the guest, then abort or rerun the move at once. On macOS the dead process's flock on the move is released a few ms after its parent has reaped it, so the next step found the lock held and answered "a hull migrate of vm1 is running". The harness now takes and releases the lock itself, with a wait, before it goes on. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
pmoust
force-pushed
the
feat/teleport
branch
2 times, most recently
from
October 20, 2026 18:12
a52b05c to
de5a196
Compare
hull's code in this series drives hvi's control socket, checkpoint and restore, the migrate stream with its disk rounds and gate, the fence owner's second stream, and the close of a failed stream. da59b45 is hvi-vmm's feat/teleport with all of that landed, rebased onto main's PL031 RTC, ledger writer thread, panic abort and sandbox that stays on for the life of the process. A native bundle now carries the PL031 in its device state; the portable profile's devices are unchanged. A move still needs exactly one writable disk, and hull gives a run one bare --disk and carries it into every restore. The Go build does not read the submodule, so the commits before this one build against main's pin, c53a857. That hvi boots hull's runs, since hull probes for the control flags, but cannot make a move. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
brig runs sandboxes through hull on a Mac. To move a running sandbox to a Linux host and back, hull has to build and run there, and it has to drive the checkpoint, restore and live-move ops that hvi's
feat/teleportbranch adds on its control socket. This branch does both.cmd/hullbuilds for arm64 Linux and runs thehvibackend with a block rootfs (the macOS-only helpers moved into_darwinfiles; the ociclient keeps image ownership on unprivileged unpacks). New verbs pause, resume, checkpoint and restore an hvi instance over its control socket.hull migrate NAME --to USER@HOSTmoves a running instance over ssh: it relays hvi's migrate stream tohvi restore --from-streamon the destination in one ssh session with an ack window, keeps a disk base per image by content digest with a spare for the way back, prepares a destination ahead of a move (bases, the pinned portable kernel), records both hosts' steps in a ledger with a txid and epoch so the guest never runs on two hosts, and recovers a fenced guest through the fence owner's restream when the relay dies.Measured on a Pi pair over the LAN: an idle ubuntu guest moves in about 2.2 s with 47 to 58 ms of downtime, one writing 10 MiB/s to its disk in about 4.1 s with 77 to 91 ms; the way back without a prepare takes 2.4 to 3.1 s. A claude-code guest moved from a Mac to a Pi through
brig teleportin 8.4 to 9.9 s with 188 to 387 ms of downtime, sending 208 to 260 MiB. docs/migration.md is the protocol, the records, the failure modes and the operator recovery, with the measured tables.Related issues
Refs brig-sh/hvi-vmm#304, the hvi branch this one pins (
hvi-vmmat da59b45, its head).Changes
cmd/hullwith the hvi backend and a block rootfs; alinuxCI lane (build, vet, lint, unit tests on ubuntu-24.04-arm).hull pauseandresumeof an hvi instance,stopover its control socket,checkpointinto a bundle with--then continue|fence, andrestorefrom a fenced bundle.hull migrate --to,--status,--abort,--txidandhull migrate-prepare: the streamed move with pre-copy rounds and the disk rounds inside hvi's stream, one ssh session from the header to the resume, bounded waits on hvi's pause and on a stalled destination, and the ledger of committed moves.migrate-prepare --kernel-sha256replaces a destination's portable kernel with the pinned one, so a moved guest runs the Image brig pins.--profile portable-arm64-v1).go.mod: cilium/ebpf held at v0.18.0 for urunc's cgroups code on the native Linux build (the reason is in go.mod).Notes for review
make hvi_vmmbefore it builds main's pin.feat/agent-migration-tcp-reset). This branch keeps go.mod's urunc pseudo-version unchanged; moving that pin together with hull-assets and community-images is a separate change.restorerow of docs/cli.md still say an hvi or KVM bundle cannot be restored, which this branch makes untrue; a docs fix follows.go vetand golangci-lint for darwin and linux/arm64,make test, and CI's race step (go test -race -p 2 ./..., cmd/hull in 162 s). The race step used to time out because a gateway test leaked two gvisor stacks whose idle workers starved every later test under-race; the gateway now closes its stack when it stops. Linux tests ran on a Pi 5. The live Pi-pair and Mac-to-Pi moves above ran on earlier heads of the same series.Checklist
make testpassesmake macosbuilds hull, vz-runner and hvi, if Go, Swift or Rust code changedtest/*.py) for changes touching boot, console, or the run pathdocs/)