Skip to content

machine: snapshot master FPU at prepare time for cross-process forks#85

Merged
fwsGonzo merged 2 commits into
masterfrom
prepared-fpu-cross-process-fork
Jul 6, 2026
Merged

machine: snapshot master FPU at prepare time for cross-process forks#85
fwsGonzo merged 2 commits into
masterfrom
prepared-fpu-cross-process-fork

Conversation

@perbu

@perbu perbu commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

Fork construction (Machine(const Machine&, const MachineOptions&)) and fork
reset (reset_to) copy the master's FPU state with a live KVM_GET_FPU against
the master's vCPU fd. That only works in the process that created the VM: KVM
vCPU ioctls require the caller's mm to be the VM-creating mm. A child that
inherited the master over fork() and builds its own VM from the master's
copy-on-write memory gets -EIO on that ioctl and cannot construct a fork.

Change

Snapshot the master's FPU registers into the Machine at
prepare_copy_on_write() time — while the master's vCPU fd is still ioctl-able —
and have fork construction and reset_to() read that snapshot via the new
prepared_fpu_registers() accessor instead of a live KVM_GET_FPU.

  • Behavior-preserving for the in-process case. The master is frozen after
    prepare and never runs again, so the snapshot is bit-identical to what a live
    KVM_GET_FPU would return at fork time.
  • Precondition made explicit. prepared_fpu_registers() asserts m_prepped,
    and the snapshot is only taken in prepare_copy_on_write().
  • sregs guard. The special-registers half of cross-process safety is already
    covered by TINYKVM_USE_SYNCED_SREGS (reads the mmap'd kvm_run page instead
    of KVM_GET_SREGS). Added a #error guard so disabling that define can't
    silently reintroduce the -EIO on the sregs path.

Cost is one tinykvm_fpuregs (~fxsave/xsave layout) per Machine, always.

Testing

  • cmake --build build clean (only pre-existing sign-compare warnings in
    src/tests.cpp).
  • tests/run_unit_tests.sh: test_reset passes 46/47 assertions. The one
    failing assertion (output_is_hello_world) also fails identically on master
    — it comes from a test that compiles a static guest ELF with gcc at runtime
    and is environmental on my host, unrelated to this change. Same for
    test_tegridy. All other suites pass.

Motivation

Enables a process-per-VM model where a single-threaded zygote fork()s worker
processes that each construct a TinyKVM fork from a master inherited over
fork() — isolating tenant workloads in separate address spaces / jailed
processes rather than threads.

Fork construction and fork reset copied the master's FPU state with a live
KVM_GET_FPU against the master's vCPU fd. That works only in the process that
created the VM: KVM vCPU ioctls require the caller's mm to be the VM-creating
mm, so a child that inherited the master over fork() and builds its own VM
from the master's copy-on-write memory gets -EIO.

Snapshot the FPU registers into the Machine at prepare_copy_on_write() time,
while the master's vCPU fd is still ioctl-able, and have fork construction and
reset_to() read that snapshot via prepared_fpu_registers() instead. The master
is frozen after prepare and never runs again, so the snapshot equals a live
KVM_GET_FPU -- behavior is unchanged for the in-process case. The sregs half of
this is already covered by TINYKVM_USE_SYNCED_SREGS (kvm_run page instead of
KVM_GET_SREGS); add a hard guard so disabling it can't silently reintroduce the
-EIO on the sregs path.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread lib/tinykvm/vcpu.cpp
cross-process fork construction breaks with -EIO. */
#if !TINYKVM_USE_SYNCED_SREGS
#error "TINYKVM_USE_SYNCED_SREGS must be enabled for cross-process fork construction"
#endif

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this break ARM? I thought I saw that ARM didn't support this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. this is true.

checked on native ARM64. The existing ARM test suite passed, but I found one ARM-specific gap: PR 85 only snapshotted prepared FPU state in the non-ARM vcpu.cpp; ARM’s prepare_copy_on_write() was still leaving the prepared FP/SIMD snapshot zeroed. I pushed bb38a40, which snapshots it on ARM too and adds a regression test for fork construction and reset_to(). /tests/run_unit_tests.sh passes on ARM64.

sloppy.

@fwsGonzo fwsGonzo merged commit 27a7c67 into master Jul 6, 2026
4 checks passed
@perbu perbu deleted the prepared-fpu-cross-process-fork branch July 6, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants