fix(start-core): make install rollback crash-recoverable, and snapshot a stopped service - #3598
Merged
Merged
Conversation
dr-bonez
reviewed
Aug 6, 2026
helix-nine
force-pushed
the
fix/rollback-quiesce-and-recover
branch
3 times, most recently
from
August 6, 2026 19:21
2b6113e to
d0ed86c
Compare
Contributor
Author
|
Cleaned up per review:
Behavior is unchanged from the reviewed version apart from the timeout removal; 561 lib tests pass. (Ignore the double force-push — the first grabbed a stale local branch by name; head is |
helix-nine
force-pushed
the
fix/rollback-quiesce-and-recover
branch
from
August 6, 2026 19:49
d0ed86c to
62e1dea
Compare
Contributor
Author
|
Reworked the quiesce to go through the actor per your comment — no more direct
561 lib tests pass, fmt clean. |
…t a stopped service A user cancelled a long package update and lost that service's database. The rollback is what deletes data: `restore_volumes_from_install_backup` deleted the live volume root and then renamed the backup into place, so an interruption between the two left the service with neither — and because `handle_installed` and `Bind::pre_mount` recreate a missing volume dir as an empty one, the result was indistinguishable from a completed rollback. The next update then discarded the surviving backup as stale, which is where the data actually went. Restore is now a two-rename swap: move the live root to `<pkg>.restore-old`, rename the backup into place, drop the aside tree. Between the renames both copies exist, and `<pkg>.restore-old` is a marker that survives an empty-skeleton recreation, so `resolve_pending_restore` can finish any interruption point deterministically instead of guessing which tree is authoritative. It runs from `ensure_volume_root` — the funnel every path uses before it trusts or creates the live root — as well as from the snapshot, the boot sweep, and `remove_install_backup`, which now refuses to discard a backup while a restore is in flight. The snapshot no longer deletes the previous rollback point as its first act: it snapshots to `<pkg>.install-backup-tmp` and swaps, so the old backup is only dropped once its replacement exists. A rollback that fails is now fatal to the load and notifies the user, rather than being swallowed by `log_err` and then having its rollback point deleted by the `Ok` path that followed. A failed first-time install over pre-existing data keeps those volumes for the restore instead of deleting them first. The boot sweep is no longer readdir-order dependent: it resolves every pending restore before any branch inspects a backup, so the orphan-reaping arm can't delete a backup that an unvisited marker still depends on. Finally, `ServiceRef::quiesce` stops the service's main chain before the snapshot, leaving the package's uninit for `uninstall`. The snapshot already predated uninit and is atomic, so this is not about the snapshot's internal consistency: it means nothing is writing to the tree a later rollback will rename or delete, formats that aren't crash-safe on their own survive, and the rollback point is the state the user had when they pressed update. It writes nothing to the status — clearing `started` would let the actor race a restart back in, and setting `desired` to Stopped would leave the service stopped after a successful update — guards on `is_initialized` (an uninitialized container's `stop` throws), and is bounded, because proceeding un-quiesced beats hanging an update on a wedged daemon.
…ate waits Three places hand-rolled the same peek-then-changed() loop to block until the database reached some state, and two more waited on a single revision as a proxy for a condition. wait_for is that loop: it peeks and marks seen before awaiting, so it can neither miss a revision nor spin on the free pass DbWatch hands an unseen value — which the two tunnel loops were doing, since they peeked without marking. The predicate takes the deserialized value rather than the model because a fallible predicate over &T::Model cannot be inferred at the call site: with the error generic (E: From<patch_db::Error>) start-core offers rustc several candidate impls, so every call needs a turbofish. Taking T instead keeps the predicate infallible and still propagates deserialization failures. The two revision-proxy waits change behavior. add_tunnel gated its default-outbound write on any revision under the new interface's ipInfo, which a write that left the address absent satisfies; it now waits for the address itself. NetworkInterfaceController::forget returned once anything under gateways changed, so a concurrent update to a different gateway could let it return before its own removal was durable; it now waits for its key to be gone.
Updating now records whether the service was running when the update began, exactly like BackingUp: stop()/start() during an update flip the payload instead of the variant, and init() restores Running or Stopped accordingly — so a power loss mid-update boots the service back to the state the user last asked for.
dr-bonez
force-pushed
the
fix/rollback-quiesce-and-recover
branch
from
August 7, 2026 19:59
0b963aa to
26a502a
Compare
dr-bonez
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The rollback is the only thing in the stack that deletes a populated volume during an update, and it was doing it non-atomically:
Interrupt between those two and the service has neither copy. Worse, it doesn't look interrupted:
handle_installedandBind::pre_mountboth recreate a missingdata/<vol>as an empty dir, so the next boot sees a live root that exists, and the sweep's "backup with no live volume" test — the one recovery we had — doesn't fire. The stranded backup then gets deleted by the next attempt's snapshot, whose first act was "remove any stale backup". That last step is where the data actually goes.What changed
The protocol now lives on one type,
volume::InstallBackup, which owns the four paths (live, backup, backup-tmp, restore-old) and the operations over them:.restore-old, backup → live, drop the aside tree..restore-oldmarks a restore in flight and survives the empty-skeleton recreation, soresolve_pending()maps every interruption point to exactly one resolution — including refusing to pick a winner when the live tree and the backup both hold real files. It runs fromensure_volume_root(the funnel every path goes through before trusting the live root), the snapshot, the boot sweep, andremove, which refuses to discard a backup while a restore is in flight..install-backup-tmpand swaps, so the previous rollback point is dropped only once its replacement exists.log_err'd into looking like a clean revert — after which the success path deleted the rollback point it still needed.Installingarm rancleanup(soft=false)— an unconditional whole-root delete — before checking for a backup. That arm is what a 0.3.x→0.4.0 package conversion runs under..restore-oldmarker still needs.ServiceRef::quiescestops the main chain before the snapshot, leaving uninit touninstall.On the ordering
Snapshot-before-uninit was already true, and a CoW snapshot is atomic, so uninit's writes can't reach it. The new thing is the quiesce: nothing is writing to the tree a later rollback will rename or delete, and the rollback point is the state the user had when they pressed update.
quiescegoes through the actor, which owns run state: it marks the desired statusUpdating(a newDesiredStatusvariant, modeled onBackingUp) and waits for the actor to perform the stop.init()mapsUpdatingback toRunning, so the replacement service starts after a successful update — and the old one starts again if a crash lands mid-update.Tests
10 unit tests over the state machine, run on a plain tmpdir — no btrfs, no root — across {live present / skeleton / absent} × {backup} × {restore-old}.
cargo test -p start-core --lib: 561 passed.Deliberately not in this PR
## [0.4.0.2]CHANGELOG heading; left the manifest half alone — say the word and I'll add it as a second commit.cancel_installis an abandon, not a cancel: the migration keeps running while a detachedload(Undo)does the renames. Two writers, one volume tree, no barrier. Wants its own change.Updating → Installed(old)flip still commits before the renames. The on-disk marker makes that recoverable, but arollback: boolonUpdatingStatewould also stop a crash mid-rollback from resuming a cancelled update.Bind::pre_mount);snapshot_subvolumeis non-recursive, so a nested subvolume lands in a backup as an empty dir.Test plan
cargo test -p start-core --lib.kill -9startd between the two renames (or drop a.restore-oldby hand), reboot, and confirm the boot sweep finishes the restore rather than starting on empty data.