Skip to content

Log VM status after the VM has an IP, not right after tart run - #28

Open
jabou wants to merge 1 commit into
khoi:mainfrom
jabou:fix/no-list-while-vm-starts
Open

jabou wants to merge 1 commit into
khoi:mainfrom
jabou:fix/no-list-while-vm-starts

Conversation

@jabou

@jabou jabou commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

What

  • Moves the VM status log (logVMStatusAfterBoot) from right after tart run to right after the VM has an IP. The log line stays the same; it now shows up after VM IP.

Why

tart status runs tart list, which runs diskutil image info on every VM's disk. When that lands in the first ~0.1-0.3 s after tart run, a VM with an ASIF disk can't lock its own disk and Virtualization.framework stops it:

Cannot acquire exclusive lock, errno 35
VZErrorDomain Code=1 "Internal Virtualization error"

Raw disks are not affected. ASIF disks come with macOS 27 base images (e.g., Cirrus base images), so a runner on such an image failed to boot almost every time and its slot never came online.

Once the VM has an IP it holds its disk lock, and tart list can't hurt it anymore (it already runs against running VMs in the health check). That's also where CI tools usually check a VM's status: after it has an IP, not right after it starts.

Plain Tart, no sand (3 tries each, same on Tart 2.40.1 and 2.41.1):

tart list after tart run VM booted
0 s 0/3
0.1 s 0/3
0.3 s 3/3
none 3/3

Testing

  • swift build and swift test: all 60 tests pass.
  • On a host with 2 runners on a macOS 27 (ASIF) image: before, one runner's VM failed on every boot; after, 41/41 boots, each with the status line after VM IP. With tart list running in a loop next to it: 21/23 booted first try, the other 2 recovered on the restart 1 s later.
  • Those 2 are the remaining window: another runner's tart status (health check) can still land in a starting VM's first fraction of a second. It's rare and sand restarts the VM, so this PR doesn't cover it.
  • Running on our fleet (4 hosts, 8 runners) since 2026-10-26.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant