Skip to content

fix(backups): reclaim only staged bundles old enough to be abandoned - #67

Merged
swimmesberger merged 1 commit into
mainfrom
wt/backup-staging-reclaim-flake
Aug 28, 2026
Merged

fix(backups): reclaim only staged bundles old enough to be abandoned#67
swimmesberger merged 1 commit into
mainfrom
wt/backup-staging-reclaim-flake

Conversation

@swimmesberger

Copy link
Copy Markdown
Owner

Fixes the CI flake that failed the post-merge run on main (run 33156562598): BackupBundleTests.CarriesTheInstanceArchiveAndEveryStacksNewestOne asserting BundleExportState.Current is null after a successful export.

What was happening

The bundle staging root is a fixed path in the machine's shared temp directory ($TMPDIR/watchtower-bundle), and CleanStagingDirectory() — run at startup from Program — deleted the whole tree. dotnet test Watchtower.slnx runs both test assemblies in parallel processes on one runner, and every Watchtower.Api.Tests fixture boots the real Program. So an Api test host starting up wiped the bundle a concurrently running Application test had just staged; BundleExportState.Current self-nulls when the staged file has vanished, producing the Assert.IsType … Actual: null failure. Timing-dependent, hence flaky. The same hazard exists for any two Watchtower processes sharing a temp directory.

The fix

Startup cleanup now deletes only per-export staging directories whose last write is older than two hours (StagingReclaimAge) — age is the only signal distinguishing a dead process's leftovers from a live sibling's freshly staged bundle. A fresh leftover from a crashed process just waits for the next startup and costs only temp-directory disk in the meantime.

The logic is extracted as ReclaimStaleStagingEntries(root, minAge) so it's directly testable; two new tests cover reclaim-old/spare-fresh and tolerance of a missing root.

Reviewer notes

  • This commit was originally pushed to the feat(ci): ship host-derived buildkit defaults into runners #66 branch minutes after that PR merged, so it never landed; this is the same change cherry-picked onto main (-x trailer points at the original).
  • No behaviour change for the normal containerized install (fresh temp dir per container); the two-hour window only matters on shared-temp hosts and in CI.

The bundle staging root is a fixed path in the machine's shared temp
directory, and startup cleanup deleted the whole tree. Two processes
sharing a temp dir therefore clobbered each other's staged bundles —
which is exactly how CI flaked: dotnet test runs both test assemblies
in parallel on one runner, every Api test host boots the real Program
(whose startup runs the cleanup), and it wiped the bundle a parallel
Application test had just staged, so BundleExportState.Current saw the
file gone and returned null.

Startup cleanup now deletes only per-export directories whose last
write is older than two hours: age is the only thing distinguishing a
dead process's leftovers from a live sibling's staged bundle. A fresh
leftover from a crashed process just waits for the next startup, and
only costs temp-directory disk in the meantime.

(cherry picked from commit 2e96375)
@swimmesberger
swimmesberger merged commit f70f254 into main Aug 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant