Back up Watchtower itself, and restore an instance from a bundle (ADR-0027)

[swimmesberger]

Watchtower has had mature per-stack backups since ADR-0016, but nothing that backed up Watchtower. Since ADR-0024 every fact it owns — the stacks and their environment variables, templates, products and releases, routes, accounts, certificates and keys — lives in its PostgreSQL. So an operator with a nightly schedule and a year of archives could restore every stack's volumes onto a new box and still have nothing that knew how to deploy them.

ADR-0024 noted in passing that backing up Watchtower's own state "becomes a PostgreSQL concern". Nothing was built, and docs/backups.md told operators to run pg_dump by hand — next to a stale paragraph telling them to also keep the watchtower-data volume, which by then held nothing.

Design and rationale: ADR-0027.

What this adds

1. Instance self-backup. A pg_dumpall of Watchtower's own database, wrapped in the same archive format, encryption and storage the stack backups use, written to {instance}/_watchtower/ beside them — so one storage folder per instance holds the whole picture. On the same schedule, governed by Backup:IncludeSelf (default on).

• Nothing is stopped. The dump is consistent by construction, which it has to be, since Watchtower is what runs it.
• The archive carries no volumes: since ADR-0024 there is no file state left to snapshot.
• Encryption is mandatory here, unlike for a stack. The dump carries every database role's password hash, the data-protection key ring, the identity signing key and every certificate's private key. A run without a passphrase is refused rather than quietly downgraded; a scheduled window without one is skipped and logged.
• Finding the database is a detection with a loud failure. A managed or host-installed PostgreSQL has no container to exec into, and the message says so — and admits it is also what an unreachable daemon looks like from here. A self-backup that quietly does nothing is invisible until the day it is needed. Backup:SelfPostgresContainer is the override.

2. An exportable bundle. One plain (uncompressed) tar — its members are already compressed and encrypted — holding a fresh instance dump, the newest archive of every stack, a manifest and secrets.json:

bundle-manifest.json     <- which Watchtower wrote it, against which schema, every archive's SHA-256
secrets.json             <- key-protection secret, backup passphrase, storage credentials
watchtower/watchtower_20260826T033000Z.tar.gz.enc
stacks/prod/blog/blog_20260826T033000Z.tar.gz.enc

Each stack archive keeps its storage-relative path, so an import can put it back exactly where the restored database's BackupDirectory already points, instead of rewriting paths it cannot verify.

3. Restore. An uploaded bundle is validated before anything is touched — an instance that cannot read the bundle it was handed must still be the instance it was — and refused on:

• a schema this build does not know (decided on the migration id, not a version comparison: migrations only roll forward, so "this binary knows that migration" is exact where comparing versions guesses);
• a KeyProtectionSecret mismatch — the sharpest edge in the feature. The stored certificates and keys are AES-GCM under an env-only secret that cannot be changed at runtime, so the message names the variable and says a restart is part of the fix;
• a missing archive, a checksum mismatch, or an archive the bundle's own passphrase cannot open.

The replay itself cannot be done by this process (pg_dumpall --clean terminates every session and drops every database; Watchtower's pool would reconnect into the middle of it). A --restore-self sibling container — modelled on the existing self-update coordinator — takes a safety dump, stops Watchtower, replays, and always restarts it in a finally. It stops and starts rather than recreating, deliberately: the container's filesystem survives, and with it the marker the restarted process reads. A nonce written into the doomed database is what proves on the way back up whether the replay committed — nothing else knows that value, so nothing else could have removed it.

4. Recovery. A checklist deploys each stack from git and then restores its newest archive, in that order — only the deploy creates the volumes the restore needs, and a deploy alone leaves the stack running on empty ones.

Notes for review

• BackupEvent.StackId becomes nullable rather than growing a parallel table: the queue, the startup sweep, retention and both history views already speak BackupEvent, and a second table would have duplicated every one of them. The wire DTO gains a kind (stack/instance) derived from the null, and backups.events an optional filter. Unfiltered history is unchanged.
• Shared, not duplicated. BackupRetentionRunner and BackupArchiveReader were extracted so the stack and instance paths prune and read archives identically. PostgresDumpService gained ReplayRemoteAsync (replay a file already inside the container) so the coordinator runs the shipped implementation rather than a second copy of it — which moved session termination to after the SQL is staged, shortening the window in which something can reconnect before the DROP. One existing test asserted the old order; it now locates calls instead of assuming offsets.
• _watchtower is refused as a stack name, since a stack sanitizing onto it would write its archives into the instance directory, and retention prunes a directory.
• The bundle is radioactive by design. secrets.json carries the key-protection secret, the backup passphrase and the storage credentials in plain text, so one artifact plus its passphrase is a complete instance. The trade is against the alternative: an operator who restores onto a new box and finds their certificates unreadable because a secret they never knew about stayed behind. Export and download are admin-only (a new SystemAdminPolicy, not merely operator-only) and audited, and the UI says what the file is.
• Accepted risk: the pg password reaches the restore coordinator as an env var on its create body, visible to docker inspect — i.e. to anyone who already owns the Docker socket, and therefore the host.

Verification

• 2137 backend tests pass, ~60 of them new: the locator's choice rule, the manifest shapes, the schedule window and cursor, the stackless event and history filter, the bundle's contents and digests, the endpoint's auth matrix (anonymous 401 / non-admin operator 403 / admin 200), the full validation refusal matrix, the completion pass's nonce branches, and the revival state machine.
• The 37 remaining failures are the pre-existing environmental set on the author's Windows box (ACME/certificate X509 chain-building, one CRLF assertion) — zero regressions. Worth confirming against Linux CI.
• rpc-schema.json is regenerated; the frontend typechecks clean (tsc --noEmit, exit 0) against a client regenerated from it.

Not yet verified: the end-to-end manual test — two compose instances, a real coordinator container — could not run on the author's machine, since the coordinator needs Watchtower itself containerised. That is the riskiest path in the change and is worth exercising before merge: build a bundle on one instance, restore it into a second with the same KEYPROTECTIONSECRET, and check the negative cases (wrong secret, newer bundle, coordinator killed mid-replay).