Skip to content

fix(start-core): make install rollback crash-recoverable, and snapshot a stopped service - #3598

Merged
dr-bonez merged 3 commits into
masterfrom
fix/rollback-quiesce-and-recover
Aug 10, 2026
Merged

fix(start-core): make install rollback crash-recoverable, and snapshot a stopped service#3598
dr-bonez merged 3 commits into
masterfrom
fix/rollback-quiesce-and-recover

Conversation

@helix-nine

@helix-nine helix-nine commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@dr-bonez — this is the rollback hardening from the Nextcloud data-loss investigation, with the ordering you asked for.

Why

The rollback is the only thing in the stack that deletes a populated volume during an update, and it was doing it non-atomically:

btrfs::delete_tree(&dst).await?;              // whole package volume root
crate::util::io::rename(&backup, &dst).await?;

Interrupt between those two and the service has neither copy. Worse, it doesn't look interrupted: handle_installed and Bind::pre_mount both recreate a missing data/<vol> as an empty dir, so the next boot sees a live root that exists, and the sweep's "backup with no live volume" test — the one recovery we had — doesn't fire. The stranded backup then gets deleted by the next attempt's snapshot, whose first act was "remove any stale backup". That last step is where the data actually goes.

What changed

The protocol now lives on one type, volume::InstallBackup, which owns the four paths (live, backup, backup-tmp, restore-old) and the operations over them:

  • Restore is a two-rename swap: live → .restore-old, backup → live, drop the aside tree. .restore-old marks a restore in flight and survives the empty-skeleton recreation, so resolve_pending() maps every interruption point to exactly one resolution — including refusing to pick a winner when the live tree and the backup both hold real files. It runs from ensure_volume_root (the funnel every path goes through before trusting the live root), the snapshot, the boot sweep, and remove, which refuses to discard a backup while a restore is in flight.
  • The snapshot stages at .install-backup-tmp and swaps, so the previous rollback point is dropped only once its replacement exists.
  • A failed rollback notifies and propagates instead of being log_err'd into looking like a clean revert — after which the success path deleted the rollback point it still needed.
  • A failed first-time install over pre-existing data keeps that data: the Installing arm ran cleanup(soft=false) — an unconditional whole-root delete — before checking for a backup. That arm is what a 0.3.x→0.4.0 package conversion runs under.
  • The boot sweep resolves every pending restore in a first pass, so readdir order can't reap a backup a .restore-old marker still needs.
  • ServiceRef::quiesce stops the main chain before the snapshot, leaving uninit to uninstall.

On the ordering

Snapshot-before-uninit was already true, and a CoW snapshot is atomic, so uninit's writes can't reach it. The new thing is the quiesce: nothing is writing to the tree a later rollback will rename or delete, and the rollback point is the state the user had when they pressed update. quiesce goes through the actor, which owns run state: it marks the desired status Updating (a new DesiredStatus variant, modeled on BackingUp) and waits for the actor to perform the stop. init() maps Updating back to Running, so the replacement service starts after a successful update — and the old one starts again if a crash lands mid-update.

Tests

10 unit tests over the state machine, run on a plain tmpdir — no btrfs, no root — across {live present / skeleton / absent} × {backup} × {restore-old}. cargo test -p start-core --lib: 561 passed.

Deliberately not in this PR

  • The version bump. Added the ## [0.4.0.2] CHANGELOG heading; left the manifest half alone — say the word and I'll add it as a second commit.
  • cancel_install is an abandon, not a cancel: the migration keeps running while a detached load(Undo) does the renames. Two writers, one volume tree, no barrier. Wants its own change.
  • Durability ordering: the Updating → Installed(old) flip still commits before the renames. The on-disk marker makes that recoverable, but a rollback: bool on UpdatingState would also stop a crash mid-rollback from resuming a cancelled update.
  • Dependency mounts can recreate a volume root while a restore is pending (Bind::pre_mount); snapshot_subvolume is non-recursive, so a nested subvolume lands in a backup as an empty dir.

Test plan

  1. cargo test -p start-core --lib.
  2. On a VM: update a running service, confirm it comes back running and healthy.
  3. Cancel an update mid-init; confirm the service returns to its old version with its data, and the backup is gone afterwards.
  4. Simulate the incident: kill -9 startd between the two renames (or drop a .restore-old by hand), reboot, and confirm the boot sweep finishes the restore rather than starting on empty data.
  5. Confirm a failed rollback surfaces as an error notification and leaves the package unloaded.

Comment thread shared-libs/crates/start-core/src/service/mod.rs Outdated
Comment thread shared-libs/crates/start-core/src/service/mod.rs Outdated
Comment thread shared-libs/crates/start-core/src/service/mod.rs Outdated
Comment thread shared-libs/crates/start-core/src/volume.rs Outdated
@helix-nine
helix-nine force-pushed the fix/rollback-quiesce-and-recover branch 3 times, most recently from 2b6113e to d0ed86c Compare August 6, 2026 19:21
@helix-nine

Copy link
Copy Markdown
Contributor Author

Cleaned up per review:

  • Comments cut to one-liners per CONTRIBUTING; doc comments kept only where the why is non-obvious.
  • The restore protocol now lives on a single volume::InstallBackup type that owns the four paths (live/backup/backup_tmp/restore_old, built with with_extension) and the operations over them (snapshot/restore/resolve_pending/remove/exists) — replaces the free functions + sibling_of + the _in test variants, which also made the snapshot's three identical degrade-and-warn arms collapsible into one.
  • Quiesce timeout dropped; quiesce is now infallible and just stops.

Behavior is unchanged from the reviewed version apart from the timeout removal; 561 lib tests pass. (Ignore the double force-push — the first grabbed a stale local branch by name; head is d0ed86c.)

@helix-nine
helix-nine force-pushed the fix/rollback-quiesce-and-recover branch from d0ed86c to 62e1dea Compare August 6, 2026 19:49
@helix-nine

Copy link
Copy Markdown
Contributor Author

Reworked the quiesce to go through the actor per your comment — no more direct persistent_container.stop().

DesiredStatus gains an Updating variant, modeled on BackingUp:

  • quiesce marks the status Updating (only if it was Running/Restarting) and waits for the actor to report started: None. The actor's existing stop arm performs the stop, so run state stays actor-owned.
  • init() maps Updating back to Running (same as Restarting), which covers both exits: the replacement service starts after a successful update, and a crash mid-update boots the rolled-back old version into Running rather than leaving it stopped.
  • A service that was Stopped stays Stopped throughout — quiesce leaves the status alone and its wait is immediately satisfied.
  • The UI needed no changes: during an update stateInfo.state is already updating, and Updating never survives into the Installed state. Regenerated DesiredStatus.ts is the only binding change.

561 lib tests pass, fmt clean.

helix-nine and others added 3 commits August 7, 2026 13:59
…t a stopped service

A user cancelled a long package update and lost that service's database. The
rollback is what deletes data: `restore_volumes_from_install_backup` deleted the
live volume root and then renamed the backup into place, so an interruption
between the two left the service with neither — and because `handle_installed`
and `Bind::pre_mount` recreate a missing volume dir as an empty one, the result
was indistinguishable from a completed rollback. The next update then discarded
the surviving backup as stale, which is where the data actually went.

Restore is now a two-rename swap: move the live root to `<pkg>.restore-old`,
rename the backup into place, drop the aside tree. Between the renames both
copies exist, and `<pkg>.restore-old` is a marker that survives an empty-skeleton
recreation, so `resolve_pending_restore` can finish any interruption point
deterministically instead of guessing which tree is authoritative. It runs from
`ensure_volume_root` — the funnel every path uses before it trusts or creates the
live root — as well as from the snapshot, the boot sweep, and
`remove_install_backup`, which now refuses to discard a backup while a restore is
in flight.

The snapshot no longer deletes the previous rollback point as its first act: it
snapshots to `<pkg>.install-backup-tmp` and swaps, so the old backup is only
dropped once its replacement exists.

A rollback that fails is now fatal to the load and notifies the user, rather than
being swallowed by `log_err` and then having its rollback point deleted by the
`Ok` path that followed. A failed first-time install over pre-existing data keeps
those volumes for the restore instead of deleting them first.

The boot sweep is no longer readdir-order dependent: it resolves every pending
restore before any branch inspects a backup, so the orphan-reaping arm can't
delete a backup that an unvisited marker still depends on.

Finally, `ServiceRef::quiesce` stops the service's main chain before the snapshot,
leaving the package's uninit for `uninstall`. The snapshot already predated uninit
and is atomic, so this is not about the snapshot's internal consistency: it means
nothing is writing to the tree a later rollback will rename or delete, formats
that aren't crash-safe on their own survive, and the rollback point is the state
the user had when they pressed update. It writes nothing to the status —
clearing `started` would let the actor race a restart back in, and setting
`desired` to Stopped would leave the service stopped after a successful update —
guards on `is_initialized` (an uninitialized container's `stop` throws), and is
bounded, because proceeding un-quiesced beats hanging an update on a wedged
daemon.
…ate waits

Three places hand-rolled the same peek-then-changed() loop to block until the
database reached some state, and two more waited on a single revision as a proxy
for a condition. wait_for is that loop: it peeks and marks seen before awaiting,
so it can neither miss a revision nor spin on the free pass DbWatch hands an
unseen value — which the two tunnel loops were doing, since they peeked without
marking.

The predicate takes the deserialized value rather than the model because a
fallible predicate over &T::Model cannot be inferred at the call site: with the
error generic (E: From<patch_db::Error>) start-core offers rustc several
candidate impls, so every call needs a turbofish. Taking T instead keeps the
predicate infallible and still propagates deserialization failures.

The two revision-proxy waits change behavior. add_tunnel gated its
default-outbound write on any revision under the new interface's ipInfo, which a
write that left the address absent satisfies; it now waits for the address
itself. NetworkInterfaceController::forget returned once anything under gateways
changed, so a concurrent update to a different gateway could let it return
before its own removal was durable; it now waits for its key to be gone.
Updating now records whether the service was running when the update
began, exactly like BackingUp: stop()/start() during an update flip the
payload instead of the variant, and init() restores Running or Stopped
accordingly — so a power loss mid-update boots the service back to the
state the user last asked for.
@dr-bonez
dr-bonez force-pushed the fix/rollback-quiesce-and-recover branch from 0b963aa to 26a502a Compare August 7, 2026 19:59
@dr-bonez
dr-bonez merged commit 630a966 into master Aug 10, 2026
31 checks passed
@dr-bonez
dr-bonez deleted the fix/rollback-quiesce-and-recover branch August 10, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants