Skip to content

Retry failed runner runs instead of retiring the slot - #26

Merged
khoi merged 1 commit into
khoi:mainfrom
jabou:fix/retry-failed-run
Oct 12, 2026
Merged

khoi merged 1 commit into
khoi:mainfrom
jabou:fix/retry-failed-run

Conversation

@jabou

@jabou jabou commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

What

  • In endless mode, a failed runOnce now goes through RestartBackoff (1s, doubling up to 60s, reset after a successful run) instead of ending that runner slot. The VM is cleaned up before the retry.
  • Before cloning, sand deletes any leftover VM with the runner's name (left by a crash or a failed clone), since tart clone won't overwrite it. Mount paths are now checked before cloning, so a bad path doesn't leave a cloned VM behind.
  • Config validation: a runner name must not match any runner's local vm.source.name, and runner names must be unique ignoring case.
  • stopAfter mode still fails fast on the first error.

Why

One failed run (e.g. two slots cloning the same image at once) used to retire its slot for good. sand kept running the other slots, so launchd never restarted it and the slot stayed dead silently.

The name checks protect the new leftover-VM delete: a runner named like a local source image would otherwise delete that image. They're case-insensitive because Tart VMs are directories and macOS volumes are case-insensitive by default.

Breaking: configs with runner names that differ only in case (Mac / mac) are now rejected at startup. They already shared one VM directory, so they never worked.

Testing

  • swift test: all 60 tests pass, including new RunnerRetryTests (slot retries after a failure, deletes the leftover VM before each clone, stops on cancel) and the name-collision validator tests.
  • New e2e case Tests/cases/stale_vm_replaced.t. Locally I could only verify the leftover-VM delete and re-clone: the VM got no IP on my machine (host networking issue), so the job itself never ran. Please run it on a host with working Tart networking.
  • Note: like the other e2e cases that only check the exit code, this case also passes when the VM never gets an IP, because an IP failure counts as the one stopAfter run. Check the log for a VM IP line.

In endless mode a failed runOnce now goes through RestartBackoff
(1s doubling to 60s) instead of ending that slot for good.
Leftover VMs with the runner's name are deleted before clone.
Runner names must not match any local vm.source.name and are
unique ignoring case.
@khoi
khoi merged commit 96c2e9f into khoi:main Oct 12, 2026
@jabou
jabou deleted the fix/retry-failed-run branch October 12, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants