diff --git a/docs/backups.md b/docs/backups.md index beea2c2..8893f98 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -22,21 +22,117 @@ the tar ([ADR-0019](decisions/0019-pause-quiesce-and-parallel-stops.md), see ## Backing up Watchtower itself -This whole document is about **your stacks**. Watchtower's own records — stacks, routes, realms, -accounts, credentials, audit trail, metrics history — live in the PostgreSQL it is configured against -([ADR-0024](decisions/0024-postgresql-only-and-state-in-the-database.md)), and nothing here touches -that. Back it up the way you back up any database: +Everything Watchtower knows — the stacks and their environment variables, templates, products and +releases, routes, realms, accounts, credentials, certificates and keys, the audit trail and the +metrics history — lives in the PostgreSQL it is configured against +([ADR-0024](decisions/0024-postgresql-only-and-state-in-the-database.md)). Backing up your stacks +without it restores their data but nothing that deploys them. + +Watchtower backs that database up itself +([ADR-0027](decisions/0027-full-instance-backup-and-restore.md)). Under **Settings → Watchtower's own +database**, "Include in the backup schedule" (on by default) adds a `pg_dumpall` of it to the same +schedule your stacks run on, written to `{instance}/_watchtower/` on the same storage — so one folder +per instance holds the whole picture. "Back up Watchtower now" runs one immediately. + +Two things to know: + +- **An encryption passphrase is required.** The dump carries every database role's password hash, the + data-protection key ring, the identity signing key and every certificate's private key. Without a + passphrase set (see [Encryption](#encryption)) the schedule skips it and the button is disabled. +- **Nothing is stopped.** The dump is taken while Watchtower keeps serving — it has to be, since + Watchtower is what runs it. + +This needs your PostgreSQL to be **a container on the same Docker daemon**, which it is in the shipped +compose file. Watchtower finds it from its own connection string; if you run several database +containers and it picks wrong (or cannot choose), name the right one in **Database container** or +`WATCHTOWER__BACKUP__SELFPOSTGRESCONTAINER`. A managed PostgreSQL (RDS, Neon, a host-installed +server) cannot be dumped this way — back it up with whatever your provider offers. + +### The full backup bundle + +"Build bundle" (same card, admins only) produces **one file** holding a fresh dump of this database, the +newest archive of every stack, and the secrets that live outside the database — everything a new +Watchtower needs to become this one. Take one before migrating to a new host, and keep it wherever you +keep passwords. + +``` +bundle-manifest.json ← which Watchtower wrote it, against which schema, and every archive's SHA-256 +secrets.json ← key-protection secret, backup passphrase, storage credentials +watchtower/watchtower_20260826T033000Z.tar.gz.enc +stacks/prod/blog/blog_20260826T033000Z.tar.gz.enc +stacks/prod/shop/globex/shop-globex_20260826T033100Z.tar.gz.enc +``` + +It is a plain (uncompressed) tar — its members are already compressed and encrypted — so `tar -tf` lists +it and `tar -xf` unpacks it anywhere. Each stack archive keeps the path it had on the backup storage, so +a restore can put it back exactly where the restored database expects to find it. + +> **The bundle is the instance.** `secrets.json` holds the key-protection secret, the backup passphrase +> and your storage credentials in plain text — deliberately, because a bundle that restores into an +> instance whose certificates and keys are unreadable is not a backup. Treat the file as a credential. + +A stack that has never been backed up appears in the manifest with no archive: its *definition* comes +back with the database, its *data* does not. The card says how many, so you can back those up and build +again. + +The bundle is kept in Watchtower's own container and one is staged at a time, so it is lost on restart — +download it when it is ready, or build a fresh one later. + +### Restoring a whole instance + +**Settings → Watchtower's own database → Restore this Watchtower…** takes a bundle and makes this +Watchtower into the one it came from. Before touching anything it checks the bundle and refuses on: + +- a bundle written by a **newer Watchtower** than this one — a database only ever migrates forward, so + update this instance first; +- a **key-protection secret** that does not match. The certificates, ACME account key and signing key in + the bundle are encrypted under the source instance's + `WATCHTOWER__AUTH__KEYPROTECTIONSECRET`; set that variable to the value in the bundle's + `secrets.json` and restart Watchtower before restoring. It cannot be changed while running; +- an archive that is missing, that does not match its checksum, or that the bundle's own passphrase + cannot open. + +It warns — but does not stop — when this Watchtower already manages stacks. Restoring replaces its whole +database; the containers it deployed keep running, unmanaged, until the checklist redeploys them. + +The restore itself takes a few seconds: a helper container stops Watchtower, replays the dump, and +starts it again. It takes a safety dump of the current database first and replays that back if the +restore's own replay fails, so a failed restore leaves the instance as it was. The page waits for +Watchtower to come back and sends you to the sign-in form — **sign in with an account from the instance +the bundle came from**; the accounts this Watchtower had are gone with its database. + +Watchtower must be running as a container on the same Docker daemon for this: it is stopped and started +around the replay. If it is not, restore the dump by hand (below). + +#### Bringing the stacks back + +After a restore, Settings shows a checklist of every stack in the restored database. Each one is +**deployed from git and then restored from its newest archive**, in that order — only the deploy creates +the volumes the restore needs, and a deploy on its own leaves the stack running on empty ones. Do them +one at a time or press **Revive all**; a stack you are handling yourself can be skipped, and the whole +checklist dismissed when you are done. What happened stays in the audit trail. + +### Restoring it by hand + +The archive is an ordinary Watchtower archive: decrypt it with stock OpenSSL as under +[Restoring a backup](#restoring-a-backup), and `backup/_dumps/watchtower.sql` inside it is a +`pg_dumpall` script. ```bash -docker compose exec postgres pg_dump -U watchtower -Fc watchtower > watchtower-$(date +%F).dump +openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -md sha256 \ + -in watchtower_20260826T033000Z.tar.gz.enc -pass pass:'YOUR PASSPHRASE' \ + | tar -xzO backup/_dumps/watchtower.sql > watchtower.sql +docker compose exec -T postgres psql -U watchtower -d postgres < watchtower.sql ``` -Restore with `pg_restore -U watchtower -d watchtower --clean` into an empty database, then start -Watchtower — it migrates on startup, so a dump from an older version comes forward on its own. +Then start Watchtower: it migrates on startup, so a dump from an older version comes forward on its +own. **Carry `WATCHTOWER__AUTH__KEYPROTECTIONSECRET` across with it.** The certificates, the ACME +account key and the signing key are encrypted in the database under that secret, and an instance +restored without it throws on every one of them. It is an environment variable, never stored in the +database, and it cannot be changed at runtime. -Take the `watchtower-data` volume with it: the certificates, the ACME account key and the -data-protection key ring still live there, and a database restored without them signs everyone out and -reissues every certificate. Neither half is much use alone. +(The `watchtower-data` volume is not part of this. It held the certificates and key ring before +ADR-0024; it holds nothing Watchtower needs now.) ## Setting it up @@ -57,6 +153,8 @@ variables — env vars pin their setting read-only in the UI, see | Helper image | `WATCHTOWER__BACKUP__HELPERIMAGE` | Image for the never-started helper container (default `busybox:stable`); any pullable image works. | | Stop grace | `WATCHTOWER__BACKUP__STOPTIMEOUTSECONDS` | How long a container *stopped* for the snapshot gets to exit on SIGTERM before SIGKILL (`docker stop -t`). Default `5` (the daemon's own default is 10); clamped to 1 … 300. Not in the UI. A service that needs longer belongs on a dump or on `pause`, not on a longer window. | | Provider | `WATCHTOWER__BACKUP__PROVIDER` | `sftp` (default) or `local`. | +| Include Watchtower's own database | `WATCHTOWER__BACKUP__INCLUDESELF` | Adds a dump of Watchtower's own PostgreSQL to the schedule (default on). Needs an encryption passphrase; see [Backing up Watchtower itself](#backing-up-watchtower-itself). | +| Database container | `WATCHTOWER__BACKUP__SELFPOSTGRESCONTAINER` | Names the container holding Watchtower's own database, when detection cannot pick one. Blank = detect it. | Then opt each stack in on its **Backups tab**: include it in the schedule, optionally give it a **schedule override** (its own cron expression instead of the instance one), and choose whether its diff --git a/docs/decisions/0027-full-instance-backup-and-restore.md b/docs/decisions/0027-full-instance-backup-and-restore.md new file mode 100644 index 0000000..7e7b755 --- /dev/null +++ b/docs/decisions/0027-full-instance-backup-and-restore.md @@ -0,0 +1,173 @@ +# ADR-0027: Watchtower backs itself up, and a bundle restores it somewhere else + +- Status: Accepted (implemented) +- Date: 2026-08-26 +- Related: [ADR-0016](0016-stack-backups.md) (the stack backup machinery this reuses whole), + [ADR-0017](0017-database-aware-dumps.md) (the `pg_dumpall` path it points at Watchtower's own + database), [ADR-0018](0018-cron-backup-schedule.md) (the schedule it joins), + [ADR-0024](0024-postgresql-only-and-state-in-the-database.md) (why the database *is* the instance), + [docs/backups.md](../backups.md) (the operator-facing description). + +## Context + +Since ADR-0024 every fact Watchtower owns lives in one PostgreSQL database: the stacks and their +environment variables, the templates, products and releases, the routes and their access grants, the +accounts and sessions, and — after decision 4 of that ADR — the certificates, the ACME account key, +the identity signing key and the data-protection key ring. The container's `/data` volume holds +nothing Watchtower needs any more. + +Against that, the backup feature covered **only stacks**. An operator with a nightly schedule and a +year of archives could restore every stack's volumes onto a new box and still have nothing that knew +how to deploy them: no repository URLs, no environment variables, no routes, no accounts. ADR-0024 +noted in passing that backing up Watchtower's own state "becomes a PostgreSQL concern"; nothing was +built, and `docs/backups.md` told operators to run `pg_dump` by hand — next to a stale paragraph +telling them to also keep the `watchtower-data` volume, which by then held nothing. + +Three things made this awkward rather than obvious: + +1. **Watchtower cannot back itself up through its own stack machinery.** `SelfProjectNameProvider` + reserves Watchtower's own compose project against stack use (so no stack can read its containers + through the App API), which also means the database cannot be registered as a stack and dumped + like any other. +2. **A dump of this database is not an ordinary archive.** `pg_dumpall` carries every role's password + hash, and the tables carry the data-protection key ring, the signing key and every certificate's + private key. It is the instance. +3. **Restoring it cannot be done by the process that holds it open.** `pg_dumpall --clean` drops and + recreates every database after terminating every other session — including Watchtower's own EF + connection pool, which would immediately reconnect into the middle of the replay. + +## Decision + +### 1. A scheduled, always-encrypted dump of Watchtower's own database, beside the stack archives + +`InstanceBackupService` takes a `pg_dumpall` of Watchtower's database over the Docker exec API, +wraps it in the same archive format, gzip and OpenSSL-compatible encryption a stack backup uses, and +uploads it through the same `IBackupStorage` to `{instance}/_watchtower/watchtower_{ts}.tar.gz.enc` +— a sibling of the stack directories under the same instance root. **One storage folder per instance +therefore holds everything a rebuild needs.** The same retention applies, through the same +`BackupRetentionRunner` the stack runs now use. + +Consequences of the "same everything" choice, each deliberate: + +- **The archive carries no volumes.** Since ADR-0024 there is no file state to snapshot, and + `BackupArchiveService` already supported a dumps-only archive (a stack whose only state is a dumped + database produces one). +- **Nothing is stopped or paused.** The dump is consistent by construction, which is what lets + Watchtower keep serving through its own backup — as it must, being the thing running it. +- **Encryption is mandatory, not optional as it is for a stack.** A run without a passphrase is + refused rather than silently downgraded, for the reason in context §2. +- **It shares the single-flight backup queue.** An instance dump waits behind a large stack backup, + which is the right way round: a queued dump is a delayed dump, whereas two runs racing for the + spool disk is a failed one. + +The schedule is the instance-wide cron (there is one instance, so there is nothing to override it +with), governed by `Watchtower:Backup:IncludeSelf` (default on). Its cursor is a settings row rather +than a column, since there is no instance table to put one on; it is read through the settings +manager rather than the options snapshot, so a value written last tick is certainly seen this tick. +A window that opens with no passphrase configured is **skipped and logged, and the cursor still +moves** — the alternative is a run that fails every night, and a window that re-fires every minute. + +### 2. `BackupEvent.StackId` becomes nullable rather than growing a parallel table + +An instance run has no stack. The history views, the single-flight queue, the retention pass and the +startup sweep all already speak `BackupEvent`, and a second table would have duplicated every one of +them. The wire DTO gains a `kind` (`stack` | `instance`) derived from the null, so the UI branches on +a word rather than on an absence, and `backups.events` gains an optional `kind` filter. Unfiltered +history is unchanged and still returns both — an instance run is part of "what has this Watchtower +been backing up". + +The stack relationship still cascades, so a deleted stack takes its own history with it; only the +stackless rows outlive every stack. + +`_watchtower` is refused as a stack name (`BackupNaming.IsReserved`, checked where the compose +project name already is), because a stack sanitizing onto it would write its archives into the +instance directory, and retention prunes a *directory*. + +### 3. Finding the database is a detection with a loud failure, not a configuration + +`SelfPostgresLocator` parses Watchtower's own connection string and looks for a running PostgreSQL +container that answers to its `Host` — by compose service, container name, or the +`{project}-{service}-{replica}` name Compose generates — among the containers of Watchtower's own +compose project, or among all running containers when it is not under Compose. One unmatched +candidate still wins (a service aliased differently from the host is ordinary); several do not, and +the run fails naming them, because the loser would be dumped and the dump would look healthy. +`Watchtower:Backup:SelfPostgresContainer` is the override. + +A managed or host-installed PostgreSQL has no container to exec into. That fails **loudly**, with a +message that says so and also admits it is what an unreachable daemon looks like from here: a +self-backup that quietly does nothing is invisible until the day it is needed. + +### 4. An exportable bundle carries an instance to another machine + +An admin can export one plain (uncompressed) tar containing the fresh instance archive, the newest +archive of every stack, a `bundle-manifest.json` and a `secrets.json`. Plain tar because its members +are already compressed and encrypted, and because the point of the artifact is to be handed to the +import on the other side. + +`bundle-manifest.json` records `bundleFormatVersion`, the instance name, `appVersion`, +`lastMigrationId`, and per-archive sizes and SHA-256s. **`lastMigrationId` is what an import decides +on**, not the version string: migrations only roll forward, so "this binary knows that migration" is +exact where comparing versions guesses. The version string is for the operator's error message. + +### 5. Restore runs from a sibling coordinator container + +The running Watchtower pre-stages everything it still can — re-uploading the bundle's stack archives +to storage at their recorded paths, extracting the SQL into the database container, writing a nonce +into the database it is about to lose — then spawns a `--restore-self` coordinator from its own +image, modelled on the `--self-update` coordinator: Docker socket, no network, group ids from +`/proc/self/status`. The coordinator takes a safety dump, stops Watchtower, replays, and restarts it +in a `finally` whatever happened. On the way back up, the nonce's absence is what proves the replay +committed. + +Validation happens **before** any of that, and refuses rather than warns on: a bundle whose +`lastMigrationId` this binary does not know, and a `KeyProtectionSecret` that differs from the one +this instance runs with. The second is the sharpest edge in the whole feature — the DB's protected +rows are AES-GCM under an env-only secret, so restoring without it yields an instance that throws on +every certificate and key it touches. The message names the variable and says it needs a restart. + +### 6. Restore is offered after login, never anonymously + +A fresh instance already creates a bootstrap admin (`AuthBootstrapService`) whose password is set by +env or printed once to the log. The restore wizard lives behind that login, and behind +`[RequireRole(Admin)]`, rather than on an anonymous "is this instance empty" endpoint: an unauthenticated +restore endpoint is an unauthenticated way to replace an instance, and "the instance looked empty" is +not an authorization decision. The wizard is offered on a fresh-looking instance and is also always +reachable from Settings. + +After the restart, a recovery checklist walks the stacks: redeploy from git (the definitions are in +the restored database), then restore each stack's newest volume archive. + +## Consequences + +- **The bundle is radioactive, by design.** `secrets.json` carries the key-protection secret, the + backup passphrase and the storage credentials in plain text, so that one artifact plus its + passphrase is a complete instance. Export is admin-only and audited, and the UI says what the file + is. This is a deliberate trade against the alternative — an operator who restores into a new box + and discovers their certificates are unreadable because a secret they never knew about stayed + behind. +- **The pg password reaches the restore coordinator as an env var on its create body**, visible to + anyone who can `docker inspect` it. Accepted: that is anyone who already owns the Docker socket, + and therefore the host. +- **Anyone who can place an instance archive on the backup storage can walk away with the + instance.** `backups.runInstance` is admin-only for that reason, where the stack runs are not. +- **Restoring an older dump into a newer binary is the supported direction** and works by itself: + migrations run on startup, so the restored schema rolls forward. The reverse is refused. +- **A restored instance carries stale coordination rows** — scheduler claims, role leases — and a + rolled-back backup cursor. The completion pass clamps the cursors and bumps the routes version; + the leases expire on their own. +- **`docs/backups.md`'s "Backing up Watchtower itself" section is replaced**, including the stale + instruction to keep the `watchtower-data` volume for certificates and keys, which have been rows + since ADR-0024. + +## Alternatives considered + +- **A second `instance_backup_events` table.** Rejected: it duplicates the queue, the sweep, the + retention pass and both history views to avoid one nullable column. +- **Restoring from storage only, with no bundle.** Simpler, and still the right path for an instance + restoring itself in place — but it requires the new box to already have the storage credentials and + the passphrase, which is exactly what an operator rebuilding after a loss does not have to hand. + The bundle exists to be the one thing they need. (Restore-from-storage remains a natural + complement, not built.) +- **An anonymous first-run restore endpoint.** Rejected — see decision 6. +- **Registering Watchtower's own compose project as a stack.** Rejected: the project reservation + exists to stop exactly that, and undoing it would hand any stack Watchtower's own containers. diff --git a/rpc-schema.json b/rpc-schema.json index fb4ee04..c982dce 100644 --- a/rpc-schema.json +++ b/rpc-schema.json @@ -148,6 +148,22 @@ ] } }, + "backups.dismissRecovery": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "dismissed": { + "type": "boolean" + } + }, + "required": [ + "dismissed" + ] + } + }, "backups.events": { "params": { "type": "object", @@ -169,6 +185,13 @@ "null" ], "default": null + }, + "kind": { + "type": [ + "string", + "null" + ], + "default": null } } }, @@ -184,10 +207,16 @@ "type": "integer" }, "stackId": { - "type": "integer" + "type": [ + "integer", + "null" + ] }, "stackName": { - "type": "string" + "type": [ + "string", + "null" + ] }, "triggeredBy": { "type": "string" @@ -223,6 +252,9 @@ "null" ], "format": "date-time" + }, + "kind": { + "type": "string" } }, "required": [ @@ -235,7 +267,8 @@ "sizeBytes", "output", "startedAt", - "finishedAt" + "finishedAt", + "kind" ] } } @@ -245,6 +278,78 @@ ] } }, + "backups.exportBundle": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "export": { + "type": "object", + "properties": { + "backupEventId": { + "type": "integer" + }, + "status": { + "type": "string" + } + }, + "required": [ + "backupEventId", + "status" + ] + } + }, + "required": [ + "export" + ] + } + }, + "backups.getBundleStatus": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "bundle": { + "type": [ + "object", + "null" + ], + "properties": { + "fileName": { + "type": "string" + }, + "sizeBytes": { + "type": "integer" + }, + "createdAtUtc": { + "type": "string", + "format": "date-time" + }, + "stackCount": { + "type": "integer" + }, + "missingStackCount": { + "type": "integer" + } + }, + "required": [ + "fileName", + "sizeBytes", + "createdAtUtc", + "stackCount", + "missingStackCount" + ] + } + }, + "required": [ + "bundle" + ] + } + }, "backups.getConfig": { "params": { "type": "object" @@ -330,6 +435,18 @@ "items": { "type": "string" } + }, + "includeSelf": { + "type": "boolean" + }, + "selfPostgresContainer": { + "type": [ + "string", + "null" + ] + }, + "instanceDirectory": { + "type": "string" } }, "required": [ @@ -344,7 +461,10 @@ "provider", "sftp", "localBasePath", - "pinnedPaths" + "pinnedPaths", + "includeSelf", + "selfPostgresContainer", + "instanceDirectory" ] } }, @@ -513,6 +633,186 @@ ] } }, + "backups.getRecoveryChecklist": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "checklist": { + "type": [ + "object", + "null" + ], + "properties": { + "restoredAtUtc": { + "type": "string", + "format": "date-time" + }, + "sourceInstance": { + "type": "string" + }, + "dismissed": { + "type": "boolean" + }, + "stacks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + }, + "name": { + "type": "string" + }, + "status": { + "type": "string" + }, + "detail": { + "type": [ + "string", + "null" + ] + }, + "deployEventId": { + "type": [ + "integer", + "null" + ] + }, + "backupEventId": { + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "stackId", + "name", + "status", + "detail", + "deployEventId", + "backupEventId" + ] + } + } + }, + "required": [ + "restoredAtUtc", + "sourceInstance", + "dismissed", + "stacks" + ] + } + }, + "required": [ + "checklist" + ] + } + }, + "backups.getRestoreStatus": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "freshInstance": { + "type": "boolean" + }, + "staged": { + "type": [ + "object", + "null" + ], + "properties": { + "canRestore": { + "type": "boolean" + }, + "blocking": { + "type": "array", + "items": { + "type": "object", + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ] + } + }, + "warnings": { + "type": "array", + "items": { + "$ref": "#/properties/staged/properties/blocking/items" + } + }, + "instanceName": { + "type": "string" + }, + "appVersion": { + "type": "string" + }, + "createdAtUtc": { + "type": "string", + "format": "date-time" + }, + "stackCount": { + "type": "integer" + }, + "missingStackCount": { + "type": "integer" + }, + "stackNames": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "canRestore", + "blocking", + "warnings", + "instanceName", + "appVersion", + "createdAtUtc", + "stackCount", + "missingStackCount", + "stackNames" + ] + }, + "lastOutcome": { + "type": "string" + }, + "lastError": { + "type": [ + "string", + "null" + ] + }, + "recoveryPending": { + "type": "boolean" + } + }, + "required": [ + "freshInstance", + "staged", + "lastOutcome", + "lastError", + "recoveryPending" + ] + } + }, "backups.getStackConfig": { "params": { "type": "object", @@ -618,7 +918,51 @@ } }, "required": [ - "config" + "config" + ] + } + }, + "backups.listInstance": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "files": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "sizeBytes": { + "type": "integer" + }, + "takenAt": { + "type": "string", + "format": "date-time" + }, + "encrypted": { + "type": "boolean" + } + }, + "required": [ + "name", + "sizeBytes", + "takenAt", + "encrypted" + ] + } + }, + "directory": { + "type": "string" + } + }, + "required": [ + "files", + "directory" ] } }, @@ -889,6 +1233,151 @@ ] } }, + "backups.reviveAll": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "revived": { + "type": "integer" + }, + "checklist": { + "type": [ + "object", + "null" + ], + "properties": { + "restoredAtUtc": { + "type": "string", + "format": "date-time" + }, + "sourceInstance": { + "type": "string" + }, + "dismissed": { + "type": "boolean" + }, + "stacks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + }, + "name": { + "type": "string" + }, + "status": { + "type": "string" + }, + "detail": { + "type": [ + "string", + "null" + ] + }, + "deployEventId": { + "type": [ + "integer", + "null" + ] + }, + "backupEventId": { + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "stackId", + "name", + "status", + "detail", + "deployEventId", + "backupEventId" + ] + } + } + }, + "required": [ + "restoredAtUtc", + "sourceInstance", + "dismissed", + "stacks" + ] + } + }, + "required": [ + "revived", + "checklist" + ] + } + }, + "backups.reviveStack": { + "params": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + } + }, + "required": [ + "stackId" + ] + }, + "result": { + "type": "object", + "properties": { + "stack": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + }, + "name": { + "type": "string" + }, + "status": { + "type": "string" + }, + "detail": { + "type": [ + "string", + "null" + ] + }, + "deployEventId": { + "type": [ + "integer", + "null" + ] + }, + "backupEventId": { + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "stackId", + "name", + "status", + "detail", + "deployEventId", + "backupEventId" + ] + } + }, + "required": [ + "stack" + ] + } + }, "backups.run": { "params": { "type": "object", @@ -925,6 +1414,34 @@ ] } }, + "backups.runInstance": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "backup": { + "type": "object", + "properties": { + "backupEventId": { + "type": "integer" + }, + "status": { + "type": "string" + } + }, + "required": [ + "backupEventId", + "status" + ] + } + }, + "required": [ + "backup" + ] + } + }, "backups.setServiceOverride": { "params": { "type": "object", @@ -1357,6 +1874,83 @@ ] } }, + "backups.skipRecoveryStack": { + "params": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + } + }, + "required": [ + "stackId" + ] + }, + "result": { + "type": "object", + "properties": { + "stack": { + "type": "object", + "properties": { + "stackId": { + "type": "integer" + }, + "name": { + "type": "string" + }, + "status": { + "type": "string" + }, + "detail": { + "type": [ + "string", + "null" + ] + }, + "deployEventId": { + "type": [ + "integer", + "null" + ] + }, + "backupEventId": { + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "stackId", + "name", + "status", + "detail", + "deployEventId", + "backupEventId" + ] + } + }, + "required": [ + "stack" + ] + } + }, + "backups.startInstanceRestore": { + "params": { + "type": "object" + }, + "result": { + "type": "object", + "properties": { + "sourceInstance": { + "type": "string" + } + }, + "required": [ + "sourceInstance" + ] + } + }, "backups.testStorage": { "params": { "type": "object" @@ -1463,6 +2057,20 @@ "null" ], "default": null + }, + "includeSelf": { + "type": [ + "boolean", + "null" + ], + "default": null + }, + "selfPostgresContainer": { + "type": [ + "string", + "null" + ], + "default": null } }, "required": [ @@ -1556,6 +2164,18 @@ "items": { "type": "string" } + }, + "includeSelf": { + "type": "boolean" + }, + "selfPostgresContainer": { + "type": [ + "string", + "null" + ] + }, + "instanceDirectory": { + "type": "string" } }, "required": [ @@ -1570,7 +2190,10 @@ "provider", "sftp", "localBasePath", - "pinnedPaths" + "pinnedPaths", + "includeSelf", + "selfPostgresContainer", + "instanceDirectory" ] } }, diff --git a/src/Watchtower.Api.Tests/InstanceBundleEndpointTests.cs b/src/Watchtower.Api.Tests/InstanceBundleEndpointTests.cs new file mode 100644 index 0000000..419cf2a --- /dev/null +++ b/src/Watchtower.Api.Tests/InstanceBundleEndpointTests.cs @@ -0,0 +1,131 @@ +using System.Net; +using System.Net.Http.Json; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +// The non-admin account is created through the same helper the access tests use, so it is made the way +// the login endpoint makes one rather than by hand. + +namespace Watchtower.Api.Tests; + +/// +/// Who may download a full backup bundle (ADR-0027 §4). This is the sharpest authorization edge in the +/// feature: the tar carries the key-protection secret, the backup passphrase and the storage credentials +/// in plain text, so anyone who can fetch it can stand the instance up somewhere else. It is therefore +/// gated on admin of the operator realm, not merely on holding a valid session. +/// +public sealed class InstanceBundleEndpointTests { + private const string BundleUrl = "/api/instance/bundle"; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static WatchtowerApiFactory AuthEnabled() => new(("Watchtower:Auth:Enabled", "true")); + + /// Stages a bundle file so the endpoint has something to serve. + private static string StageBundle(WatchtowerApiFactory factory, string content = "bundle-bytes") { + var directory = Directory.CreateTempSubdirectory("wt-bundle-endpoint").FullName; + var path = Path.Combine(directory, "watchtower-bundle_test.tar"); + File.WriteAllText(path, content); + factory.Services.GetRequiredService().Replace(new StagedBundle( + path, "watchtower-bundle_test.tar", content.Length, DateTimeOffset.UtcNow, + StackCount: 1, MissingStackCount: 0)); + return path; + } + + /// Signs in and returns the __wt_sso cookie pair. + private static async Task SignInAsync(HttpClient client, string userName, string password) { + var response = await client.PostAsJsonAsync( + "/api/auth/login", new { userName, password }, Ct); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + return Assert.Single(response.Headers.GetValues("Set-Cookie")).Split(';')[0]; + } + + private static async Task GetAsync(HttpClient client, string? cookie) { + var request = new HttpRequestMessage(HttpMethod.Get, BundleUrl); + if (cookie is not null) request.Headers.Add("Cookie", cookie); + return await client.SendAsync(request, Ct); + } + + /// The password gives a new account. + private const string OperatorPassword = "correct-horse-battery"; + + [Fact] + public async Task AnAnonymousCallerIsChallenged() { + using var factory = AuthEnabled(); + using var client = factory.CreateApiClient(); + StageBundle(factory); + + Assert.Equal(HttpStatusCode.Unauthorized, (await GetAsync(client, cookie: null)).StatusCode); + } + + [Fact] + public async Task AnOperatorWithoutTheAdminRoleIsRefused() { + // A signed-in operator can see the backup history and run stack backups. Taking the whole + // instance off the box is a different privilege, and this is where the two part company. + using var factory = AuthEnabled(); + using var client = factory.CreateApiClient(); + StageBundle(factory); + await factory.AddUserAsync("olive", password: OperatorPassword); + + var cookie = await SignInAsync(client, "olive", OperatorPassword); + + Assert.Equal(HttpStatusCode.Forbidden, (await GetAsync(client, cookie)).StatusCode); + } + + [Fact] + public async Task AnAdminGetsTheTarWithItsFileNameAndLength() { + using var factory = AuthEnabled(); + using var client = factory.CreateApiClient(); + StageBundle(factory); + var cookie = await SignInAsync(client, "admin", WatchtowerApiFactory.AdminPassword); + + var response = await GetAsync(client, cookie); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Equal("application/x-tar", response.Content.Headers.ContentType?.MediaType); + Assert.Equal("watchtower-bundle_test.tar", response.Content.Headers.ContentDisposition?.FileName?.Trim('"')); + // Declared up front so the browser can show real progress on a file that is typically large. + Assert.Equal("bundle-bytes".Length, response.Content.Headers.ContentLength); + Assert.Equal("bundle-bytes", await response.Content.ReadAsStringAsync(Ct)); + } + + [Fact] + public async Task TheDownloadIsAudited() { + // The bundle leaving the box is the event worth being able to point at afterwards. + using var factory = AuthEnabled(); + using var client = factory.CreateApiClient(); + StageBundle(factory); + var cookie = await SignInAsync(client, "admin", WatchtowerApiFactory.AdminPassword); + + Assert.Equal(HttpStatusCode.OK, (await GetAsync(client, cookie)).StatusCode); + + await using var scope = factory.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + // The audit writer is best-effort and out-of-band, so give it a moment to land its row. + AuditEvent? row = null; + for (var attempt = 0; attempt < 50 && row is null; attempt++) { + row = await db.AuditEvents.AsNoTracking() + .FirstOrDefaultAsync(a => a.Action == "bundle.download", Ct); + if (row is null) await Task.Delay(100, Ct); + } + + Assert.NotNull(row); + Assert.Equal("backups", row.Category); + Assert.Equal("admin", row.Actor); + Assert.Contains("watchtower-bundle_test.tar", row.Detail); + } + + [Fact] + public async Task WithNothingStagedAnAdminGetsA404() { + // Not an empty 200: "there is no bundle" and "here is an empty bundle" are different answers. + using var factory = AuthEnabled(); + using var client = factory.CreateApiClient(); + var cookie = await SignInAsync(client, "admin", WatchtowerApiFactory.AdminPassword); + + Assert.Equal(HttpStatusCode.NotFound, (await GetAsync(client, cookie)).StatusCode); + } +} diff --git a/src/Watchtower.Api/Authentication/WatchtowerSessionAuthenticationHandler.cs b/src/Watchtower.Api/Authentication/WatchtowerSessionAuthenticationHandler.cs index b99fce3..be5d3b6 100644 --- a/src/Watchtower.Api/Authentication/WatchtowerSessionAuthenticationHandler.cs +++ b/src/Watchtower.Api/Authentication/WatchtowerSessionAuthenticationHandler.cs @@ -27,6 +27,15 @@ public static class WatchtowerSessionDefaults { /// so there is one rule and not two. /// public const string SystemRealmPolicy = "WatchtowerSystemRealm"; + + /// + /// plus the admin role — the middleware-side equivalent of a handler's + /// [RequireRole(WatchtowerClaims.AdminRole)], for the endpoints that hand out an entire + /// instance (ADR-0027): the full backup bundle carries the key-protection secret, the backup + /// passphrase and the storage credentials in plain text, so downloading one is not something an + /// ordinary operator session should be able to do. + /// + public const string SystemAdminPolicy = "WatchtowerSystemAdmin"; } /// diff --git a/src/Watchtower.Api/Endpoints/InstanceBackupEndpoints.cs b/src/Watchtower.Api/Endpoints/InstanceBackupEndpoints.cs new file mode 100644 index 0000000..f7bd146 --- /dev/null +++ b/src/Watchtower.Api/Endpoints/InstanceBackupEndpoints.cs @@ -0,0 +1,91 @@ +using Elarion.Abstractions.Identity; +using Microsoft.AspNetCore.Mvc; +using Watchtower.Api.Authentication; +using Watchtower.Application.Modules.Backups; +using Watchtower.Application.Services; + +namespace Watchtower.Api.Endpoints; + +/// +/// The instance backup surfaces that move whole files rather than JSON, and so cannot be JSON-RPC +/// handlers (ADR-0027): downloading the full backup bundle, and — from stage 3 — uploading one back. +/// +/// +/// Admin-only through , not merely +/// operator-only: the bundle carries the key-protection secret, the backup passphrase and the storage +/// credentials in plain text, so downloading one is downloading the instance. With authentication +/// disabled nothing gates it, which is the same posture as the rest of the management plane — an +/// unauthenticated Watchtower is a Watchtower on a trusted network by the operator's choice. +/// +public static class InstanceBackupEndpoints { + /// Maps the instance backup file endpoints. + /// The application to map onto. + /// Whether the session scheme is registered; policies exist only then. + public static WebApplication MapInstanceBackupEndpoints(this WebApplication app, bool authEnabled) { + foreach (var route in new[] { MapBundleDownload(app), MapBundleUpload(app) }) + if (authEnabled) route.RequireAuthorization(WatchtowerSessionDefaults.SystemAdminPolicy); + return app; + } + + /// + /// Accepts a full backup bundle, unpacks it and answers with this instance's verdict on restoring it + /// (ADR-0027 §5). Nothing is replaced here — the upload is staged, and a separate confirmed call + /// (backups.startInstanceRestore) is what acts on it. + /// + /// + /// The body is the tar itself rather than a multipart form: there is exactly one file, and streaming + /// it straight to disk keeps a multi-gigabyte upload out of memory. The request size limit is lifted + /// for this route alone, since a bundle is as large as the estate it carries. + /// + private static RouteHandlerBuilder MapBundleUpload(WebApplication app) => + app.MapPost("/api/instance/restore/bundle", async ( + HttpRequest request, InstanceRestoreService restore, AuditLog audit, ICurrentUser currentUser, + CancellationToken ct) => { + try { + var validation = await restore.StageAsync(request.Body, ct); + await audit.RecordAsync( + BackupService.AuditCategory, "instance.restore.upload", + InstanceRestoreService.AuditTarget, + $"bundle from '{validation.InstanceName}' ({validation.AppVersion}) uploaded — " + + (validation.CanRestore + ? $"restorable, {validation.StackCount} stack archive(s)" + : $"refused: {string.Join(" ", validation.Blocking.Select(b => b.Code))}"), + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + return Results.Ok(RestoreValidationDto.From(validation)); + } catch (Exception ex) when (ex is not OperationCanceledException) { + // The upload was not a bundle at all, or could not be unpacked. A 400 with the reason, + // not a 500: the file is the caller's, and so is the fix. + return Results.Problem( + title: "The upload is not a usable backup bundle", detail: ex.Message, + statusCode: StatusCodes.Status400BadRequest); + } + }) + // Kestrel's 30 MB default would reject any real bundle. The body is streamed straight to disk, + // so the ceiling is the container's temp space rather than memory. + .WithMetadata(new RequestSizeLimitAttribute(long.MaxValue)); + + /// + /// Streams the staged full backup bundle. The tar is built by backups.exportBundle onto the + /// container's own filesystem and only ever read from here — its path is never in an API response, + /// so there is no name to traverse with. + /// + private static RouteHandlerBuilder MapBundleDownload(WebApplication app) => + app.MapGet("/api/instance/bundle", async ( + HttpResponse response, BundleExportState state, AuditLog audit, ICurrentUser currentUser, + CancellationToken ct) => { + if (state.Current is not { } staged) return Results.NotFound(); + + await audit.RecordAsync( + BackupService.AuditCategory, "bundle.download", BackupBundleService.AuditTarget, + $"{staged.FileName} · {staged.SizeBytes} bytes", + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + + response.ContentType = "application/x-tar"; + response.Headers.ContentDisposition = $"attachment; filename=\"{staged.FileName}\""; + // Known up front, so the browser can show real progress on what is typically a large file. + response.ContentLength = staged.SizeBytes; + await using var file = File.OpenRead(staged.Path); + await file.CopyToAsync(response.Body, ct); + return Results.Empty; + }); +} diff --git a/src/Watchtower.Api/Program.cs b/src/Watchtower.Api/Program.cs index 522dd02..32bf30f 100644 --- a/src/Watchtower.Api/Program.cs +++ b/src/Watchtower.Api/Program.cs @@ -25,6 +25,13 @@ if (CoordinatorMode.IsApplicable(args)) await CoordinatorMode.RunAndExitAsync(args); +// ── Restore-coordinator mode ────────────────────────────────────────────────── +// The same trick for the instance restore (ADR-0027): Watchtower cannot replay a dump over the database +// its own connection pool is holding open, so a sibling stops it, replays, and starts it again. The web +// host is NOT started in this mode either. +if (RestoreCoordinatorMode.IsApplicable(args)) + await RestoreCoordinatorMode.RunAndExitAsync(args); + // ── Schema export mode ────────────────────────────────────────────────────────── // Generates rpc-schema.json (consumed by the frontend client generator) and exits without // starting the web server or touching the database. @@ -156,14 +163,23 @@ .AddAuthentication(WatchtowerSessionDefaults.AuthenticationScheme) .AddScheme( WatchtowerSessionDefaults.AuthenticationScheme, configureOptions: null); - builder.Services.AddAuthorization(o => o.AddPolicy( - WatchtowerSessionDefaults.SystemRealmPolicy, - p => p - // Authenticated first, so an anonymous caller still gets the 401 challenge it always did - // rather than a 403 that tells it a session would not have helped. - .RequireAuthenticatedUser() - // …and then the same operator-realm rule the handler pipeline applies, read off the principal. - .RequireAssertion(context => WatchtowerClaims.IsSystemRealm(context.User)))); + builder.Services.AddAuthorization(o => { + o.AddPolicy( + WatchtowerSessionDefaults.SystemRealmPolicy, + p => p + // Authenticated first, so an anonymous caller still gets the 401 challenge it always did + // rather than a 403 that tells it a session would not have helped. + .RequireAuthenticatedUser() + // …and then the same operator-realm rule the handler pipeline applies, read off the principal. + .RequireAssertion(context => WatchtowerClaims.IsSystemRealm(context.User))); + // The same, plus the admin role — for the endpoints that hand out a whole instance (ADR-0027). + o.AddPolicy( + WatchtowerSessionDefaults.SystemAdminPolicy, + p => p + .RequireAuthenticatedUser() + .RequireAssertion(context => WatchtowerClaims.IsSystemRealm(context.User)) + .RequireRole(WatchtowerClaims.AdminRole)); + }); // Per-IP throttle on the login endpoint (design.md §9). Registered only in this mode because the // route it protects is only mapped here; the policy is attached to that one route, not global. builder.Services.AddWatchtowerLoginRateLimiter(); @@ -251,6 +267,8 @@ app.MapElarionEndpoints(app.Configuration); // Webhook, SSE streams, and health. app.MapWatchtowerHttpEndpoints(authEnabled); +// Full backup bundle download (ADR-0027) — a file, so not a JSON-RPC handler. Admin-only. +app.MapInstanceBackupEndpoints(authEnabled); // SPA fallback: any unmatched route returns index.html so the client router handles it. app.MapFallbackToFile("index.html"); @@ -280,6 +298,10 @@ await db.BackupEvents .SetProperty(e => e.Output, e => (e.Output ?? "") + "\n[Reset: process restarted while backup was in progress]")); + // A bundle staged by a previous process (ADR-0027 §4) is unreachable — the state that knew about it + // went with the process — so it would only occupy the disk until the container is recreated. + scope.ServiceProvider.GetRequiredService().CleanStagingDirectory(); + // ADR-0022's upgrade guard: an instance that was serving routes under the old implicit `caddy` // default is pinned to caddy once, so the default flip cannot switch a working proxy silently. // Here rather than in a hosted service so the ordering is stated rather than inherited: after the diff --git a/src/Watchtower.Api/RestoreCoordinatorMode.cs b/src/Watchtower.Api/RestoreCoordinatorMode.cs new file mode 100644 index 0000000..211aa08 --- /dev/null +++ b/src/Watchtower.Api/RestoreCoordinatorMode.cs @@ -0,0 +1,137 @@ +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Services; + +namespace Watchtower.Api; + +/// +/// Entry point for restore-coordinator mode (--restore-self, ADR-0027 §5). +/// +/// +/// +/// Watchtower cannot replay a dump over its own database: pg_dumpall --clean terminates every +/// session and drops every database, and Watchtower's connection pool would reconnect into the middle of +/// that. So it spawns a sibling container (same image, same Docker socket) running in this mode, which +/// stops Watchtower, replays, and starts it again. +/// +/// +/// It stops and starts the container rather than recreating it, unlike the self-update +/// coordinator. That is deliberate: the container's filesystem survives, and with it the marker file the +/// restarted Watchtower reads to find out what happened here. +/// +/// +/// The dump is already inside the database container when this starts — placed there by the process +/// that spawned this one, which had the archive and its passphrase. This mode only replays. +/// +/// +internal static class RestoreCoordinatorMode { + /// + /// What the database is called in this mode's output. The dump/replay code names a compose service; + /// here there is only ever one database, and it is Watchtower's own. + /// + private const string Service = "watchtower"; + + /// Returns true when the process was launched in restore-coordinator mode. + internal static bool IsApplicable(string[] args) => + args.Contains(RestoreCoordinatorEnvironment.Flag); + + /// Runs the restore and exits the process. Never returns. + internal static async Task RunAndExitAsync(string[] args) { + var watchtowerId = Required(args, "--container-id"); + var postgresId = Required(args, "--postgres-id"); + var sqlPath = Required(args, "--sql"); + var user = Required(args, "--db-user"); + var execUser = GetArg(args, "--db-exec-user"); + var expected = GetAll(args, "--expect-db"); + var password = Environment.GetEnvironmentVariable( + RestoreCoordinatorEnvironment.PostgresPassword); + + var apiVersion = Environment.GetEnvironmentVariable("WATCHTOWER__DOCKERAPIVERSION") ?? "1.43"; + using var docker = new DockerEngineClient( + Options.Create(new WatchtowerOptions { DockerApiVersion = apiVersion })); + var ct = CancellationToken.None; + var connection = new PostgresConnection(user, password, execUser) { Databases = expected }; + // The shipped dump/replay implementation, not a second one: this mode has no DI and no logging + // sink, which is all the null logger costs it. + var postgres = new PostgresDumpService(docker, NullLogger.Instance); + void Log(string line) => Console.WriteLine(line); + + // Let the request that started this return before its container is stopped. + await Task.Delay(TimeSpan.FromSeconds(3), ct); + + // A dump of what is there now, taken before anything is dropped. It turns the worst failure + // mode — a half-replayed --clean script — into something recoverable. + Console.WriteLine("Taking a safety dump of the current database…"); + try { + await postgres.DumpToContainerFileAsync( + postgresId, connection, RestoreCoordinatorEnvironment.SafetyDumpPath, ct); + } catch (Exception ex) { + // Nothing has been touched yet, so the safe thing is to stop here rather than replay + // without a way back. + Console.WriteLine($"Could not take a safety dump: {ex.Message}"); + Console.WriteLine("Nothing was changed — Watchtower is still running on its own database."); + Environment.Exit(1); + } + + Console.WriteLine($"Stopping Watchtower ({Short(watchtowerId)}) for the replay…"); + await docker.StopContainerAsync(watchtowerId, ct); + + var restored = false; + try { + await postgres.WaitReadyAsync(postgresId, connection, Service, Log, ct); + Console.WriteLine("Replaying the dump…"); + await postgres.ReplayRemoteAsync(postgresId, connection, Service, sqlPath, expected, Log, ct); + restored = true; + Console.WriteLine("Replay complete."); + } catch (Exception ex) { + Console.WriteLine($"Replay failed: {ex.Message}"); + Console.WriteLine("Rolling back to the safety dump."); + try { + // No expected-database check on the way back: the safety dump is whatever was there, and + // the point is to restore it rather than to assert what it held. + await postgres.ReplayRemoteAsync( + postgresId, connection, Service, RestoreCoordinatorEnvironment.SafetyDumpPath, + expectedDatabases: [], Log, ct); + Console.WriteLine("Rollback complete — the database is as it was before the restore."); + } catch (Exception rollbackEx) { + Console.WriteLine( + $"Rollback failed too: {rollbackEx.Message}. The database may be in a partial state; " + + $"the pre-restore dump is at {RestoreCoordinatorEnvironment.SafetyDumpPath} inside " + + "the database container."); + } + } finally { + // Always, whatever happened: an instance that is down is worse than one that is unchanged. + Console.WriteLine($"Starting Watchtower ({Short(watchtowerId)}) again…"); + try { + await docker.StartContainerAsync(watchtowerId, ct); + } catch (Exception ex) { + Console.WriteLine( + $"Could not start Watchtower again: {ex.Message}. Start container {watchtowerId} by hand."); + } + } + + Console.WriteLine(restored + ? "Instance restore complete — Watchtower is coming up on the restored database." + : "Instance restore failed — Watchtower is coming up on the database it had."); + Environment.Exit(restored ? 0 : 1); + } + + private static string Short(string id) => id.Length >= 12 ? id[..12] : id; + + private static string Required(string[] args, string name) => + GetArg(args, name) + ?? throw new InvalidOperationException($"{name} is required in restore-coordinator mode"); + + private static string? GetArg(string[] args, string name) { + var index = Array.IndexOf(args, name); + return index >= 0 && index + 1 < args.Length ? args[index + 1] : null; + } + + /// Every value of a repeatable flag, in order. + private static string[] GetAll(string[] args, string name) => [ + .. args.Index() + .Where(x => x.Item == name && x.Index + 1 < args.Length) + .Select(x => args[x.Index + 1]), + ]; +} diff --git a/src/Watchtower.Application.Tests/BackupBundleTests.cs b/src/Watchtower.Application.Tests/BackupBundleTests.cs new file mode 100644 index 0000000..7981aff --- /dev/null +++ b/src/Watchtower.Application.Tests/BackupBundleTests.cs @@ -0,0 +1,270 @@ +using System.Formats.Tar; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// The exportable full backup bundle (ADR-0027 §4). What it has to get right is what an import on the +/// other side depends on: every archive present under the storage-relative path the restored database +/// will look for it at, a manifest that says which Watchtower wrote it and against which schema, and +/// the out-of-database secrets without which the restored instance cannot read its own keys. +/// +public sealed class BackupBundleTests : IDisposable { + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private readonly string _storageRoot = Directory.CreateTempSubdirectory("wt-bundle-tests").FullName; + + public void Dispose() { + try { + Directory.Delete(_storageRoot, recursive: true); + } catch (IOException) { + // A temp directory the OS will reclaim; never worth failing a passing test over. + } + } + + private AuthTestHost Start(params (string, string?)[] more) => + AuthTestHost.Start(FakeInstanceBackup.Register, [ + ("Watchtower:Backup:Provider", "local"), + ("Watchtower:Backup:Local:BasePath", _storageRoot), + ("Watchtower:Backup:InstanceName", "prod"), + ("Watchtower:Backup:EncryptionPassphrase", "s3cret"), + .. more, + ]); + + private async Task AddStackAsync(AuthTestHost host, string name, string? backupDirectory = null) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var stack = new Stack { + Name = name, + ComposeProjectName = name, + Product = TestProducts.New(name), + BackupDirectory = backupDirectory, + }; + db.Stacks.Add(stack); + await db.SaveChangesAsync(Ct); + return stack.Id; + } + + /// Puts a stand-in stack archive on the storage, the way a real run would have left one. + private async Task SeedArchiveAsync( + string directory, string stem, DateTimeOffset takenAt, string content) { + var name = BackupNaming.FileName(stem, takenAt, encrypted: true); + var path = Path.Combine(_storageRoot, directory.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(path); + await File.WriteAllTextAsync(Path.Combine(path, name), content, Ct); + return name; + } + + /// Runs an export the way the queue would, and returns the staged bundle. + private async Task ExportAsync(AuthTestHost host) { + int eventId; + await using (var scope = host.Services.CreateAsyncScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + var evt = new BackupEvent { + StackId = null, + TriggeredBy = BackupTriggers.BundleExport, + Status = BackupStatuses.Queued, + StartedAt = DateTimeOffset.UtcNow, + }; + db.BackupEvents.Add(evt); + await db.SaveChangesAsync(Ct); + eventId = evt.Id; + } + + await host.Services.GetRequiredService().ExecuteExportAsync(eventId, Ct); + + await using (var scope = host.Services.CreateAsyncScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.AsNoTracking().SingleAsync(e => e.Id == eventId, Ct); + Assert.True( + evt.Status == BackupStatuses.Success, + $"The export failed. Run log:\n{evt.Output}"); + } + + var staged = host.Services.GetRequiredService().Current; + return Assert.IsType(staged); + } + + /// Every entry in the tar, by name, with its bytes. + private static async Task> ReadTarAsync(string path) { + var entries = new Dictionary(StringComparer.Ordinal); + await using var file = File.OpenRead(path); + await using var reader = new TarReader(file); + while (await reader.GetNextEntryAsync() is { } entry) { + using var content = new MemoryStream(); + if (entry.DataStream is { } data) await data.CopyToAsync(content); + entries[entry.Name] = content.ToArray(); + } + return entries; + } + + private static T Json(Dictionary entries, string name) => + JsonSerializer.Deserialize(entries[name], BackupBundle.JsonOptions)!; + + [Fact] + public async Task CarriesTheInstanceArchiveAndEveryStacksNewestOne() { + using var host = Start(); + await AddStackAsync(host, "blog"); + await AddStackAsync(host, "shop"); + var stale = await SeedArchiveAsync( + "prod/blog", "blog", new DateTimeOffset(2026, 8, 20, 3, 30, 0, TimeSpan.Zero), "old-blog"); + var newest = await SeedArchiveAsync( + "prod/blog", "blog", new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), "new-blog"); + await SeedArchiveAsync( + "prod/shop", "shop", new DateTimeOffset(2026, 8, 25, 3, 31, 0, TimeSpan.Zero), "shop-bytes"); + + var staged = await ExportAsync(host); + var entries = await ReadTarAsync(staged.Path); + + Assert.Equal(2, staged.StackCount); + Assert.Equal(0, staged.MissingStackCount); + Assert.Contains(BackupBundle.ManifestEntry, entries.Keys); + Assert.Contains(BackupBundle.SecretsEntry, entries.Keys); + + // The storage-relative path is preserved under stacks/, so an import can put each archive back + // exactly where the restored database's BackupDirectory already points. + Assert.Equal("new-blog", Encoding.UTF8.GetString(entries[$"stacks/prod/blog/{newest}"])); + Assert.DoesNotContain($"stacks/prod/blog/{stale}", entries.Keys); + Assert.Equal( + FakeInstanceBackup.Content, + Encoding.UTF8.GetString(entries.Single(e => e.Key.StartsWith("watchtower/", StringComparison.Ordinal)).Value)); + } + + [Fact] + public async Task TheManifestNamesTheBuildTheSchemaAndEveryArchivesDigest() { + using var host = Start(); + await AddStackAsync(host, "blog"); + await SeedArchiveAsync( + "prod/blog", "blog", new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), "blog-bytes"); + + var entries = await ReadTarAsync((await ExportAsync(host)).Path); + var manifest = Json(entries, BackupBundle.ManifestEntry); + + Assert.Equal(BackupBundle.FormatVersion, manifest.BundleFormatVersion); + Assert.Equal("watchtower", manifest.Tool); + Assert.Equal("prod", manifest.InstanceName); + Assert.False(string.IsNullOrWhiteSpace(manifest.AppVersion)); + // The migration id is what an import decides on, so it has to be the *applied* one, not a guess. + Assert.False(string.IsNullOrWhiteSpace(manifest.LastMigrationId)); + + var stack = Assert.Single(manifest.Stacks); + Assert.Equal("blog", stack.Name); + var archive = Assert.IsType(stack.Archive); + Assert.True(archive.Encrypted); + Assert.Equal( + Convert.ToHexStringLower(SHA256.HashData(entries[archive.Entry])), + archive.Sha256); + Assert.Equal(entries[archive.Entry].Length, archive.SizeBytes); + Assert.Equal( + Convert.ToHexStringLower(SHA256.HashData(entries[manifest.Instance.Entry])), + manifest.Instance.Sha256); + } + + [Fact] + public async Task AStackWithNoArchiveIsRecordedRatherThanOmitted() { + // Its definition still comes back with the database, so the operator has to be told that the + // data did not — silently listing one stack out of two would read as "there was only one". + using var host = Start(); + await AddStackAsync(host, "blog"); + await AddStackAsync(host, "never-backed-up"); + await SeedArchiveAsync( + "prod/blog", "blog", new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), "blog-bytes"); + + var staged = await ExportAsync(host); + var manifest = Json(await ReadTarAsync(staged.Path), BackupBundle.ManifestEntry); + + Assert.Equal(1, staged.StackCount); + Assert.Equal(1, staged.MissingStackCount); + var missing = Assert.Single(manifest.Stacks, s => s.Name == "never-backed-up"); + Assert.Null(missing.Archive); + Assert.Equal("no archive on the backup storage", missing.Reason); + } + + [Fact] + public async Task ATenantsStampedDirectoryIsHonoured() { + // BackupDirectory is stamped once and never recomputed, so the bundle has to read it rather than + // derive a path from the stack's current name — the two differ for every tenant. + using var host = Start(); + await AddStackAsync(host, "shop-globex", backupDirectory: "prod/shop/globex"); + var name = await SeedArchiveAsync( + "prod/shop/globex", "shop-globex", + new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), "globex-bytes"); + + var entries = await ReadTarAsync((await ExportAsync(host)).Path); + + Assert.Equal("globex-bytes", Encoding.UTF8.GetString(entries[$"stacks/prod/shop/globex/{name}"])); + } + + [Fact] + public async Task TheSecretsFileCarriesWhatTheDatabaseCannot() { + using var host = Start(("Watchtower:Auth:KeyProtectionSecret", "key-protection-secret")); + + var entries = await ReadTarAsync((await ExportAsync(host)).Path); + var secrets = Json(entries, BackupBundle.SecretsEntry); + + // Without this one the restored instance throws on every certificate and key it touches. + Assert.Equal("key-protection-secret", secrets.KeyProtectionSecret); + Assert.Equal("s3cret", secrets.BackupEncryptionPassphrase); + Assert.Equal("prod", secrets.BackupInstanceName); + Assert.Equal("local", secrets.Storage.Provider); + Assert.Equal(_storageRoot, secrets.Storage.LocalBasePath); + + var manifest = Json(entries, BackupBundle.ManifestEntry); + Assert.True(manifest.KeyProtectionSecretConfigured); + } + + [Fact] + public async Task AnInstanceWithNoKeyProtectionSecretSaysSoRatherThanLookingLikeALostOne() { + using var host = Start(); + + var entries = await ReadTarAsync((await ExportAsync(host)).Path); + + Assert.Null(Json(entries, BackupBundle.SecretsEntry).KeyProtectionSecret); + Assert.False(Json(entries, BackupBundle.ManifestEntry).KeyProtectionSecretConfigured); + } + + [Fact] + public async Task ASecondExportReplacesTheFirstAndDeletesIt() { + // One bundle is staged at a time: they are large, and an operator downloading "the" bundle + // should never be handed a stale one. + using var host = Start(); + var first = await ExportAsync(host); + var second = await ExportAsync(host); + + Assert.NotEqual(first.Path, second.Path); + Assert.False(File.Exists(first.Path)); + Assert.True(File.Exists(second.Path)); + } + + [Fact] + public async Task TheEventRecordsTheBundleAndTheAuditTrailNamesIt() { + using var host = Start(); + await AddStackAsync(host, "blog"); + await SeedArchiveAsync( + "prod/blog", "blog", new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), "blog-bytes"); + + var staged = await ExportAsync(host); + + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.AsNoTracking().SingleAsync(Ct); + Assert.Null(evt.StackId); + Assert.Equal(staged.SizeBytes, evt.SizeBytes); + Assert.Equal(staged.FileName, evt.RemotePath); + + var row = await db.AuditEvents.AsNoTracking() + .SingleAsync(a => a.Action == "bundle.export", Ct); + Assert.True(row.Success); + Assert.Contains("1 stack archive(s)", row.Detail); + // Never the passphrase or the key-protection secret, whatever else the row says. + Assert.DoesNotContain("s3cret", row.Detail); + } +} diff --git a/src/Watchtower.Application.Tests/BackupNamingAndRetentionTests.cs b/src/Watchtower.Application.Tests/BackupNamingAndRetentionTests.cs index a2e3f84..9bfa442 100644 --- a/src/Watchtower.Application.Tests/BackupNamingAndRetentionTests.cs +++ b/src/Watchtower.Application.Tests/BackupNamingAndRetentionTests.cs @@ -54,6 +54,43 @@ public void ForeignFileNamesDoNotParse(string name) => public void SanitizeYieldsASafeSingleSegment(string input, string expected) => Assert.Equal(expected, BackupNaming.Sanitize(input)); + // ── The instance's own directory (ADR-0027) ────────────────────────────── + + [Fact] + public void TheInstanceDirectoryIsASiblingOfTheStackDirectories() { + // One folder per instance holds everything a rebuild needs: Watchtower's database next to the + // stacks whose volumes it describes. + Assert.Equal("prod/_watchtower", BackupNaming.InstanceDirectory("prod")); + Assert.Equal("prod/web-app", BackupNaming.StackDirectory("prod", "web-app")); + } + + [Fact] + public void InstanceArchivesParticipateInRetentionLikeAnyOther() { + // Retention and the remote listing key off the timestamp suffix alone, so the instance stem is + // not a special case for either — which is what lets both paths share one implementation. + var taken = new DateTimeOffset(2026, 8, 26, 3, 30, 0, TimeSpan.Zero); + var name = BackupNaming.FileName(BackupNaming.InstanceFileStem, taken, encrypted: true); + + Assert.Equal("watchtower_20260826T033000Z.tar.gz.enc", name); + Assert.Equal(taken, BackupNaming.ParseTimestamp(name)); + } + + [Theory] + [InlineData("_watchtower")] + [InlineData("_WATCHTOWER")] + [InlineData("_watchtower ")] + public void TheInstanceDirectoryNameIsReservedAgainstStacks(string stackName) => + // A stack sanitizing onto it would write its archives among Watchtower's own, and retention — + // which prunes a directory, not a stack — would then count the two as one set. + Assert.True(BackupNaming.IsReserved(stackName)); + + [Theory] + [InlineData("watchtower")] + [InlineData("_watchtower-backups")] + [InlineData("web-app")] + public void OrdinaryStackNamesAreNotReserved(string stackName) => + Assert.False(BackupNaming.IsReserved(stackName)); + // ── Retention ──────────────────────────────────────────────────────────── [Fact] diff --git a/src/Watchtower.Application.Tests/BackupScheduleJobTests.cs b/src/Watchtower.Application.Tests/BackupScheduleJobTests.cs index 5b1de48..c787eb6 100644 --- a/src/Watchtower.Application.Tests/BackupScheduleJobTests.cs +++ b/src/Watchtower.Application.Tests/BackupScheduleJobTests.cs @@ -84,6 +84,83 @@ public async Task EnqueuesEachStackOnceForAWindowAndMovesTheCursor() { Assert.Equal(Utc(17, 15, 30), await CursorAsync(host, web)); } + // ── Watchtower's own database (ADR-0027) ───────────────────────────────── + + private static IReadOnlyList InstanceEnqueued(AuthTestHost host) => + ((RecordingBackupQueue)host.Services.GetRequiredService()).InstanceEnqueued; + + /// Enabled, with an encryption passphrase — what the self-backup additionally needs. + private static (string, string?)[] WithSelfBackup(params (string, string?)[] more) => + Enabled([("Watchtower:Backup:EncryptionPassphrase", "s3cret"), .. more]); + + [Fact] + public async Task TheInstanceRunsOnTheSameWindowAsTheStacksAndKeepsItsOwnCursor() { + using var host = Start(WithSelfBackup()); + var web = await AddStackAsync(host, "web", last: Utc(16, 15, 30)); + + Assert.Equal(0, await TickAsync(host, Utc(17, 3, 29))); + // Two enqueues for one window: the stack and Watchtower's own database. + Assert.Equal(2, await TickAsync(host, Utc(17, 3, 30, 15))); + Assert.Equal(1, EnqueuedCount(host, web)); + Assert.Equal(["schedule"], InstanceEnqueued(host)); + + // Same window on the next tick enqueues neither — the instance cursor moved with the stack's. + Assert.Equal(0, await TickAsync(host, Utc(17, 3, 31, 15))); + Assert.Single(InstanceEnqueued(host)); + + Assert.Equal(2, await TickAsync(host, Utc(17, 15, 30, 40))); + Assert.Equal(["schedule", "schedule"], InstanceEnqueued(host)); + } + + [Fact] + public async Task TheInstanceWindowRunsWithNoStacksAtAll() { + // A fresh instance with nothing deployed still has state worth backing up — the whole point of + // ADR-0027 — so the stackless early return must not swallow it. + using var host = Start(WithSelfBackup()); + + Assert.Equal(1, await TickAsync(host, Utc(17, 3, 30, 15))); + Assert.Equal(["schedule"], InstanceEnqueued(host)); + } + + [Fact] + public async Task IncludeSelfOffLeavesTheStacksAlone() { + using var host = Start(WithSelfBackup(("Watchtower:Backup:IncludeSelf", "false"))); + var web = await AddStackAsync(host, "web", last: Utc(16, 15, 30)); + + Assert.Equal(1, await TickAsync(host, Utc(17, 3, 30, 15))); + Assert.Equal(1, EnqueuedCount(host, web)); + Assert.Empty(InstanceEnqueued(host)); + } + + [Fact] + public async Task WithoutAPassphraseTheWindowIsSkippedButStillConsumed() { + // The dump carries every role's password hash, so it is never written unencrypted. The cursor + // still moves: an unconsumed window would re-open on every tick for the rest of the day. + using var host = Start(Enabled()); + Assert.Equal(0, await TickAsync(host, Utc(17, 3, 30, 15))); + Assert.Empty(InstanceEnqueued(host)); + + // Passphrase appears; the window that was skipped stays skipped rather than firing late. + using var configured = host.Restart(WithSelfBackup()); + Assert.Equal(0, await TickAsync(configured, Utc(17, 3, 31))); + Assert.Empty(InstanceEnqueued(configured)); + + // The next window runs normally. + Assert.Equal(1, await TickAsync(configured, Utc(17, 15, 30, 20))); + Assert.Equal(["schedule"], InstanceEnqueued(configured)); + } + + [Fact] + public async Task ARestartDoesNotDoubleFireTheInstanceWindow() { + using var host = Start(WithSelfBackup()); + Assert.Equal(1, await TickAsync(host, Utc(17, 3, 30, 10))); + + // The cursor is a settings row, so it survives the process exactly like a stack's column does. + using var restarted = host.Restart(WithSelfBackup()); + Assert.Equal(0, await TickAsync(restarted, Utc(17, 3, 31))); + Assert.Empty(InstanceEnqueued(restarted)); + } + [Fact] public async Task TheProductionQueueGetsAQueuedEventPerWindow() { // One window through the real queue (its worker never starts here): the enqueue is the diff --git a/src/Watchtower.Application.Tests/BackupTestDoubles.cs b/src/Watchtower.Application.Tests/BackupTestDoubles.cs index 1bb5a42..26da598 100644 --- a/src/Watchtower.Application.Tests/BackupTestDoubles.cs +++ b/src/Watchtower.Application.Tests/BackupTestDoubles.cs @@ -1,10 +1,71 @@ +using System.Text; +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; using Watchtower.Application.Services; namespace Watchtower.Application.Tests; +/// +/// Answers "where is Watchtower's own database" without a Docker daemon, so a test of what a restore +/// decides is not also a test of container detection (which +/// covers on its own). +/// +internal sealed class FakeSelfPostgresLocator( + DockerEngineClient docker, SelfProjectNameProvider selfProjects, IConfiguration configuration, + IOptionsMonitor options, ILogger logger) + : SelfPostgresLocator(docker, selfProjects, configuration, options, logger) { + /// Replaces the real locator on a test host. + public static readonly Action Register = services => + services.Replace(ServiceDescriptor.Singleton()); + + public override Task LocateAsync(Action log, CancellationToken ct) => + Task.FromResult(new SelfPostgresTarget( + "test-postgres", "watchtower-postgres-1", "postgres:18-alpine", "postgres", "watchtower", + "watchtower")); +} + +/// +/// Produces a stand-in instance archive on the configured storage instead of dumping a real database, +/// so a test of what a bundle contains needs neither a Docker daemon nor a second PostgreSQL. +/// The bytes are a marker string rather than a real archive: nothing under test opens it — the bundle +/// carries archives, it does not read them. +/// +internal sealed class FakeInstanceBackup( + BackupArchiveService archiveService, PostgresDumpService postgres, SelfPostgresLocator locator, + BackupStorageFactory storageFactory, BackupRetentionRunner retention, IServiceScopeFactory scopeFactory, + IOptionsMonitor options, AuditLog audit, ILogger logger) + : InstanceBackupService( + archiveService, postgres, locator, storageFactory, retention, scopeFactory, options, audit, logger) { + /// The content every fake instance archive is written with. + public const string Content = "fake-instance-archive"; + + /// Held explicitly rather than captured, so the base class owns the only captured copy. + private readonly BackupStorageFactory _storageFactory = storageFactory; + + /// Replaces the real service on a test host. + public static readonly Action Register = services => + services.Replace(ServiceDescriptor.Singleton()); + + public override async Task RunAsync( + BackupOptions backup, Action log, CancellationToken ct) { + var takenAt = DateTimeOffset.UtcNow; + var directory = BackupNaming.InstanceDirectory(backup.ResolveInstanceName()); + var fileName = BackupNaming.FileName(BackupNaming.InstanceFileStem, takenAt, encrypted: true); + var relativePath = $"{directory}/{fileName}"; + + using var storage = _storageFactory.Create(backup); + await storage.UploadAsync(relativePath, async (stream, token) => + await stream.WriteAsync(Encoding.UTF8.GetBytes(Content), token), ct); + log($"Fake instance archive written to {relativePath}"); + return new InstanceArchiveResult( + relativePath, fileName, directory, Content.Length, takenAt, ["watchtower"]); + } +} + /// /// Records enqueues instead of queueing them; the worker loop never starts in these tests. /// @@ -15,10 +76,13 @@ namespace Watchtower.Application.Tests; /// contract of those paths, and it is invisible from a real queue that has not run yet. /// internal sealed class RecordingBackupQueue( - BackupService backupService, BackupChainCoordinator chain, IServiceScopeFactory scopeFactory, + BackupService backupService, InstanceBackupService instanceBackupService, + BackupBundleService bundleService, BackupChainCoordinator chain, IServiceScopeFactory scopeFactory, ILogger logger) - : BackupQueueService(backupService, chain, scopeFactory, logger) { + : BackupQueueService(backupService, instanceBackupService, bundleService, chain, scopeFactory, logger) { private readonly List<(int StackId, string TriggeredBy, BackupChainStep? Chain)> _enqueued = []; + private readonly List _instanceEnqueued = []; + private readonly List _bundleEnqueued = []; /// Replaces the real queue on a test host. public static readonly Action Register = services => @@ -34,6 +98,11 @@ internal sealed class RecordingBackupQueue( get { lock (_enqueued) return [.. _enqueued]; } } + /// Every instance self-backup enqueue (ADR-0027), by trigger, in call order. + public IReadOnlyList InstanceEnqueued { + get { lock (_enqueued) return [.. _instanceEnqueued]; } + } + public override BackupEnqueueResult Enqueue( int stackId, string triggeredBy, BackupChainStep? chainStep = null) { lock (_enqueued) { @@ -41,4 +110,25 @@ public override BackupEnqueueResult Enqueue( return new BackupEnqueueResult(_enqueued.Count, "queued"); } } + + /// Every bundle export enqueue (ADR-0027 §4), by trigger, in call order. + public IReadOnlyList BundleEnqueued { + get { lock (_enqueued) return [.. _bundleEnqueued]; } + } + + public override BackupEnqueueResult EnqueueBundleExport(string triggeredBy) { + lock (_enqueued) { + _bundleEnqueued.Add(triggeredBy); + return new BackupEnqueueResult(_enqueued.Count + _bundleEnqueued.Count, "queued"); + } + } + + public override BackupEnqueueResult EnqueueInstance(string triggeredBy) { + // Recorded without coalescing, for the reason the stack enqueues are: nothing drains this queue, + // so a coalesced second window would be invisible to a schedule test. + lock (_enqueued) { + _instanceEnqueued.Add(triggeredBy); + return new BackupEnqueueResult(_enqueued.Count + _instanceEnqueued.Count, "queued"); + } + } } diff --git a/src/Watchtower.Application.Tests/InstanceBackupEventTests.cs b/src/Watchtower.Application.Tests/InstanceBackupEventTests.cs new file mode 100644 index 0000000..6a92383 --- /dev/null +++ b/src/Watchtower.Application.Tests/InstanceBackupEventTests.cs @@ -0,0 +1,122 @@ +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Watchtower.Application.Entities; +using Watchtower.Application.Modules.Backups; +using Watchtower.Application.Modules.Backups.Handlers; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// The stackless half of the backup history (ADR-0027): the queue writes an instance run's event with no +/// stack, and the history views can ask for one kind or the other without either becoming a special case. +/// +public sealed class InstanceBackupEventTests { + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static async Task AddStackAsync(AuthTestHost host, string name) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var stack = new Stack { Name = name, ComposeProjectName = name, Product = TestProducts.New(name) }; + db.Stacks.Add(stack); + await db.SaveChangesAsync(Ct); + return stack.Id; + } + + private static async Task> EventsAsync(AuthTestHost host, string? kind) { + await using var scope = host.Services.CreateAsyncScope(); + var handler = ActivatorUtilities.CreateInstance(scope.ServiceProvider); + var result = await handler.HandleAsync(new ListBackupEvents.Query(Kind: kind), Ct); + Assert.True(result.IsSuccess); + return result.Value!.Events; + } + + [Fact] + public async Task TheQueueWritesAQueuedEventWithNoStack() { + using var host = AuthTestHost.Start(); + var queue = host.Services.GetRequiredService(); + + var enqueued = queue.EnqueueInstance(BackupTriggers.Manual); + + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.AsNoTracking().SingleAsync(e => e.Id == enqueued.BackupEventId, Ct); + Assert.Null(evt.StackId); + Assert.Equal("manual", evt.TriggeredBy); + Assert.Equal("queued", evt.Status); + } + + [Fact] + public async Task ASecondRequestCoalescesOntoTheWaitingRun() { + // Same reason the stack backups coalesce: a caller who asks twice wants the backup that is about + // to happen, not two of them competing for the disk. + using var host = AuthTestHost.Start(); + var queue = host.Services.GetRequiredService(); + + var first = queue.EnqueueInstance(BackupTriggers.Manual); + var second = queue.EnqueueInstance(BackupTriggers.Schedule); + + Assert.Equal(first.BackupEventId, second.BackupEventId); + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + Assert.Equal(1, await db.BackupEvents.CountAsync(Ct)); + } + + [Fact] + public async Task TheHistoryReportsTheKindAndCanBeNarrowedToEither() { + using var host = AuthTestHost.Start(); + var stackId = await AddStackAsync(host, "web"); + var queue = host.Services.GetRequiredService(); + queue.Enqueue(stackId, BackupTriggers.Manual); + queue.EnqueueInstance(BackupTriggers.Manual); + + // Unfiltered is unchanged for every existing caller: an instance run is part of "what has this + // Watchtower been backing up", so it belongs in the instance-wide list. + var all = await EventsAsync(host, kind: null); + Assert.Equal(2, all.Count); + + var instance = Assert.Single(await EventsAsync(host, BackupEventKinds.Instance)); + Assert.Equal("instance", instance.Kind); + Assert.Null(instance.StackId); + Assert.Null(instance.StackName); + + var stack = Assert.Single(await EventsAsync(host, BackupEventKinds.Stack)); + Assert.Equal("stack", stack.Kind); + Assert.Equal(stackId, stack.StackId); + Assert.Equal("web", stack.StackName); + } + + [Fact] + public async Task AnUnknownKindIsRefusedRatherThanIgnored() { + // Silently returning everything would make a typo look like "there are no instance backups". + using var host = AuthTestHost.Start(); + await using var scope = host.Services.CreateAsyncScope(); + var handler = ActivatorUtilities.CreateInstance(scope.ServiceProvider); + + var result = await handler.HandleAsync(new ListBackupEvents.Query(Kind: "watchtower"), Ct); + + Assert.False(result.IsSuccess); + Assert.Contains("Kind must be", result.Error!.Message); + } + + [Fact] + public async Task DeletingAStackLeavesTheInstanceHistoryStanding() { + // The relationship still cascades, so a stack takes its own history with it — but the stackless + // rows outlive every stack, which is the point of keeping them in the same table. + using var host = AuthTestHost.Start(); + var stackId = await AddStackAsync(host, "web"); + var queue = host.Services.GetRequiredService(); + queue.Enqueue(stackId, BackupTriggers.Manual); + queue.EnqueueInstance(BackupTriggers.Manual); + + await using (var scope = host.Services.CreateAsyncScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + await db.Stacks.Where(s => s.Id == stackId).ExecuteDeleteAsync(Ct); + } + + var remaining = Assert.Single(await EventsAsync(host, kind: null)); + Assert.Equal("instance", remaining.Kind); + } +} diff --git a/src/Watchtower.Application.Tests/InstanceBackupManifestTests.cs b/src/Watchtower.Application.Tests/InstanceBackupManifestTests.cs new file mode 100644 index 0000000..625bdf2 --- /dev/null +++ b/src/Watchtower.Application.Tests/InstanceBackupManifestTests.cs @@ -0,0 +1,70 @@ +using System.Text.Json; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// The instance archive's manifest (ADR-0027). It has to be self-describing enough that a restore can +/// refuse it: kind tells a reader it is not a stack archive, and lastMigrationId is what a +/// target instance checks before replaying, since migrations only roll forward. +/// +public sealed class InstanceBackupManifestTests { + private static readonly DateTimeOffset TakenAt = new(2026, 8, 26, 3, 15, 0, TimeSpan.Zero); + + private static SelfPostgresTarget Target() => new( + "abc123", "watchtower-postgres-1", "postgres:18-alpine", "postgres", "watchtower", "watchtower"); + + private static BackupService.BackupDumpEntry Dump() => new( + "watchtower", DumpEngine.Postgres, "_dumps/watchtower.sql", "postgres:18-alpine", "watchtower", + "watchtower-postgres-1", Volumes: [], ["postgres", "watchtower"], 65536); + + private static JsonElement Parse(string json) => JsonDocument.Parse(json).RootElement; + + [Fact] + public void DeclaresItsKindVersionAndSchema() { + var manifest = Parse(InstanceBackupService.BuildManifest( + "prod", Target(), TakenAt, Dump(), "20260826192454_AddInstanceBackupEvents")); + + Assert.Equal(1, manifest.GetProperty("formatVersion").GetInt32()); + // The discriminator: a reader holding one archive must be able to tell which of the two it is + // without inferring it from which keys happen to be missing. + Assert.Equal("watchtower-instance", manifest.GetProperty("kind").GetString()); + Assert.Equal("watchtower", manifest.GetProperty("tool").GetString()); + Assert.Equal("prod", manifest.GetProperty("instance").GetString()); + Assert.Equal("watchtower", manifest.GetProperty("database").GetString()); + Assert.Equal("2026-08-26T03:15:00.0000000Z", manifest.GetProperty("createdAtUtc").GetString()); + Assert.Equal( + "20260826192454_AddInstanceBackupEvents", manifest.GetProperty("lastMigrationId").GetString()); + // Always true: the run refuses to produce an unencrypted instance archive at all. + Assert.True(manifest.GetProperty("encrypted").GetBoolean()); + Assert.False(string.IsNullOrWhiteSpace(manifest.GetProperty("appVersion").GetString())); + } + + [Fact] + public void CarriesTheDumpInTheSameShapeAStackArchiveDoes() { + // Same node builder as the stack manifest, so tooling that reads one reads the other. + var dumps = Parse(InstanceBackupService.BuildManifest("prod", Target(), TakenAt, Dump(), "m")) + .GetProperty("dumps"); + + var dump = Assert.Single(dumps.EnumerateArray()); + Assert.Equal("watchtower", dump.GetProperty("service").GetString()); + Assert.Equal("postgres", dump.GetProperty("engine").GetString()); + Assert.Equal("_dumps/watchtower.sql", dump.GetProperty("file").GetString()); + Assert.Equal("watchtower-postgres-1", dump.GetProperty("container").GetString()); + Assert.Equal(65536, dump.GetProperty("sizeBytes").GetInt64()); + Assert.Equal( + ["postgres", "watchtower"], + dump.GetProperty("databases").EnumerateArray().Select(d => d.GetString())); + Assert.Empty(dump.GetProperty("volumes").EnumerateArray()); + } + + [Fact] + public void RecordsANullMigrationRatherThanOmittingTheKey() { + // A database with no migrations applied is a real state, and a key that is sometimes absent is + // harder for a reader to handle than one that is sometimes null. + var manifest = Parse(InstanceBackupService.BuildManifest("prod", Target(), TakenAt, Dump(), null)); + + Assert.Equal(JsonValueKind.Null, manifest.GetProperty("lastMigrationId").ValueKind); + } +} diff --git a/src/Watchtower.Application.Tests/InstanceRestoreValidationTests.cs b/src/Watchtower.Application.Tests/InstanceRestoreValidationTests.cs new file mode 100644 index 0000000..9f2ede9 --- /dev/null +++ b/src/Watchtower.Application.Tests/InstanceRestoreValidationTests.cs @@ -0,0 +1,238 @@ +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// What an instance refuses to restore, and why (ADR-0027 §5). Every one of these decisions is made +/// before anything is touched, which is the point: an instance that cannot read the bundle it +/// was handed has to still be the instance it was. The messages are asserted as well as the outcomes — +/// a refusal an operator cannot act on is only half a refusal. +/// +public sealed class InstanceRestoreValidationTests { + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static AuthTestHost Start(params (string, string?)[] settings) => + AuthTestHost.Start(FakeSelfPostgresLocator.Register, settings); + + /// The migration this build actually applied, so a default bundle is a valid one. + private static async Task LastMigrationAsync(AuthTestHost host) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + return await InstanceVersion.LastMigrationAsync(db, Ct); + } + + private static async Task StageAsync(AuthTestHost host, byte[] bundle) { + using var stream = new MemoryStream(bundle); + return await host.Services.GetRequiredService().StageAsync(stream, Ct); + } + + private static async Task StageDefaultAsync( + AuthTestHost host, TestBundles.Options? options = null) => + await StageAsync(host, TestBundles.Build(await LastMigrationAsync(host), options)); + + private static string? Blocking(RestoreValidation validation, string code) => + validation.Blocking.FirstOrDefault(f => f.Code == code)?.Message; + + [Fact] + public async Task AValidBundleIsAccepted() { + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options( + Stacks: ["blog", "shop"], MissingStacks: ["never-backed-up"])); + + Assert.True(validation.CanRestore, string.Join(" ", validation.Blocking.Select(b => b.Message))); + Assert.Empty(validation.Blocking); + Assert.Equal("source", validation.InstanceName); + Assert.Equal("9.9.9-test", validation.AppVersion); + Assert.Equal(2, validation.StackCount); + Assert.Equal(1, validation.MissingStackCount); + Assert.Equal(["blog", "shop", "never-backed-up"], validation.StackNames); + } + + [Fact] + public async Task ABundleFromANewerSchemaIsRefused() { + // Migrations only roll forward, so this is exact rather than a version comparison: replaying a + // schema this binary has never known would leave a database it cannot migrate. + using var host = Start(); + + var validation = await StageDefaultAsync( + host, new TestBundles.Options(LastMigrationId: "29990101000000_FromTheFuture")); + + Assert.False(validation.CanRestore); + var message = Blocking(validation, "newer-schema"); + Assert.NotNull(message); + Assert.Contains("9.9.9-test", message); + Assert.Contains("Update this Watchtower", message); + } + + [Fact] + public async Task AMismatchedKeyProtectionSecretIsRefusedWithTheRemedy() { + // The sharpest edge in the feature: the stored certificates and keys are AES-GCM under this + // secret, and it cannot be changed at runtime — so the message has to name the variable and say + // that a restart is part of the fix. + using var host = Start(("Watchtower:Auth:KeyProtectionSecret", "this-instances-secret")); + + var validation = await StageDefaultAsync( + host, new TestBundles.Options(KeyProtectionSecret: "the-source-instances-secret")); + + Assert.False(validation.CanRestore); + var message = Blocking(validation, "key-protection-secret"); + Assert.NotNull(message); + Assert.Contains("WATCHTOWER__AUTH__KEYPROTECTIONSECRET", message); + Assert.Contains("restart", message); + // Never the secret itself, from either side. + Assert.DoesNotContain("this-instances-secret", message); + Assert.DoesNotContain("the-source-instances-secret", message); + } + + [Fact] + public async Task AMatchingKeyProtectionSecretIsAccepted() { + using var host = Start(("Watchtower:Auth:KeyProtectionSecret", "shared-secret")); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(KeyProtectionSecret: "shared-secret")); + + Assert.True(validation.CanRestore); + } + + [Fact] + public async Task ABundleWithNoSecretIntoAnInstanceThatHasOneIsAWarningNotARefusal() { + // The restored rows are readable as they are, and later writes are encrypted. Worth saying, + // not worth stopping for. + using var host = Start(("Watchtower:Auth:KeyProtectionSecret", "this-instances-secret")); + + var validation = await StageDefaultAsync(host); + + Assert.True(validation.CanRestore); + Assert.Contains(validation.Warnings, w => w.Code == "key-protection-secret-new"); + } + + [Fact] + public async Task AnArchiveThatDoesNotMatchItsChecksumIsRefused() { + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(CorruptInstanceDigest: true)); + + Assert.False(validation.CanRestore); + Assert.Contains("damaged in transit or altered", Blocking(validation, "corrupt-archive")); + } + + [Fact] + public async Task AnArchiveTheManifestPromisesButTheTarLacksIsRefused() { + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(OmitInstanceArchive: true)); + + Assert.False(validation.CanRestore); + Assert.Contains("incomplete or was repacked", Blocking(validation, "missing-archive")); + } + + [Fact] + public async Task AnArchiveTheBundlesOwnPassphraseCannotOpenIsRefused() { + // Proving the passphrase now is the whole reason the probe exists: discovering it after the + // database has been dropped would be discovering it too late. + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(WrongPassphrase: true)); + + Assert.False(validation.CanRestore); + Assert.NotNull(Blocking(validation, "unreadable-archive")); + } + + [Fact] + public async Task AnArchiveWithNoDumpInItIsRefused() { + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(WithoutDump: true)); + + Assert.False(validation.CanRestore); + Assert.Contains("nothing to restore from", Blocking(validation, "no-dump")); + } + + [Fact] + public async Task AnUnknownBundleFormatIsRefused() { + using var host = Start(); + + var validation = await StageDefaultAsync(host, new TestBundles.Options(BundleFormatVersion: 99)); + + Assert.False(validation.CanRestore); + Assert.Contains("format version 99", Blocking(validation, "bundle-format")); + } + + [Fact] + public async Task RestoringOverAWatchtowerThatIsInUseWarnsAboutWhatItReplaces() { + using var host = Start(); + await using (var scope = host.Services.CreateAsyncScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + db.Stacks.Add(new Stack { + Name = "already-here", ComposeProjectName = "already-here", + Product = TestProducts.New("already-here"), + }); + await db.SaveChangesAsync(Ct); + } + + var validation = await StageDefaultAsync(host); + + // A warning, not a refusal: replacing a working instance is a thing an operator may legitimately + // mean to do. The confirmation dialog is where they say so. + Assert.True(validation.CanRestore); + Assert.Contains("keep running unmanaged", validation.Warnings.Single(w => w.Code == "not-fresh").Message); + } + + [Fact] + public async Task AFreshInstanceDoesNotGetTheWarning() { + using var host = Start(); + + var validation = await StageDefaultAsync(host); + + Assert.DoesNotContain(validation.Warnings, w => w.Code == "not-fresh"); + } + + [Theory] + [InlineData("../escaped.json")] + [InlineData("stacks/../../escaped.json")] + [InlineData("/etc/watchtower-escaped.json")] + public async Task ATarThatWouldWriteOutsideItsDirectoryIsRejectedOutright(string entryName) { + // The entry names come from a file an operator uploaded, so this is the one check that has to + // happen before anything is written rather than after everything is. Refused rather than + // sanitized: a name that tries to escape is a name to stop on, not one to quietly rewrite. + using var host = Start(); + using var stream = new MemoryStream(TestBundles.TraversalBundle(entryName)); + + var error = await Assert.ThrowsAsync(() => + host.Services.GetRequiredService().StageAsync(stream, Ct)); + + Assert.Contains("written outside it", error.Message); + } + + [Fact] + public async Task AnUploadThatIsNotABundleSaysSo() { + using var host = Start(); + using var stream = new MemoryStream(TestBundles.NotABundle()); + + var error = await Assert.ThrowsAsync(() => + host.Services.GetRequiredService().StageAsync(stream, Ct)); + + Assert.Contains("not a Watchtower backup bundle", error.Message); + } + + [Fact] + public async Task AStagedBundleCanBeRevalidatedWithoutReUploadingIt() { + // What the wizard does on a page load: the bundle is already here, and re-hashing gigabytes to + // answer that would be the wrong trade. + using var host = Start(); + await StageDefaultAsync(host, new TestBundles.Options(Stacks: ["blog"])); + + var staging = host.Services.GetRequiredService(); + var staged = Assert.IsType(staging.Current); + var revalidated = await host.Services.GetRequiredService() + .ValidateAsync(staged, Ct); + + Assert.True(revalidated.CanRestore); + Assert.Equal(1, revalidated.StackCount); + } +} diff --git a/src/Watchtower.Application.Tests/PostgresDumpServiceTests.cs b/src/Watchtower.Application.Tests/PostgresDumpServiceTests.cs index 3cb809c..c908689 100644 --- a/src/Watchtower.Application.Tests/PostgresDumpServiceTests.cs +++ b/src/Watchtower.Application.Tests/PostgresDumpServiceTests.cs @@ -1,4 +1,4 @@ -using System.Formats.Tar; +using System.Formats.Tar; using System.Net; using System.Net.Http.Headers; using System.Text; @@ -332,6 +332,19 @@ private static string WriteSql(string sql) { return path; } + /// + /// Where the exec carrying in its command sits among the recorded + /// requests. Found rather than counted: one exec is two HTTP calls, and which client a step rides + /// is a transport decision that has already moved once — fixed offsets would say more about that + /// than about the order under test. + /// + private static int ExecIndex(DockerClientEstate estate, string fragment) { + var index = estate.Default.Bodies.FindIndex( + b => b is not null && b.Contains(fragment, StringComparison.Ordinal)); + Assert.True(index >= 0, $"no exec was recorded whose command contains '{fragment}'"); + return index; + } + [Fact] public async Task TheReplayTerminatesSessions_CopiesTheSql_RunsPsql_AndCleansUp() { using var estate = Estate(); @@ -352,14 +365,6 @@ public async Task TheReplayTerminatesSessions_CopiesTheSql_RunsPsql_AndCleansUp( File.Delete(sql); } - // Every session has to go first: --clean cannot DROP DATABASE under a live connection, and - // psql would then merge the dump into the old database instead of replacing it. - using var terminate = JsonDocument.Parse(estate.Default.Bodies[0]!); - Assert.Contains( - "pg_terminate_backend", - terminate.RootElement.GetProperty("Cmd").EnumerateArray().Last().GetString() ?? ""); - Assert.Contains(log, l => l.StartsWith("WARNING: closed 2 open session(s) on 'db'", StringComparison.Ordinal)); - // The SQL goes in as a tar at /tmp, streamed from the host file — over the untimed client, // because a dump is as large as the database it captures. var put = estate.LongRunning.Requests.FindIndex(r => r.Contains("/archive?path=", StringComparison.Ordinal)); @@ -371,12 +376,16 @@ public async Task TheReplayTerminatesSessions_CopiesTheSql_RunsPsql_AndCleansUp( // 0600: the dump carries every role's password hash. Assert.Equal(UnixFileMode.UserRead | UnixFileMode.UserWrite, entry.Mode); - // The psql exec is the second exec created (after the session terminate). - var psqlCreate = estate.Default.Requests - .Select((request, index) => (request, index)) - .Where(x => x.request.EndsWith("/containers/db-id/exec", StringComparison.Ordinal)) - .ElementAt(1).index; - using var replay = JsonDocument.Parse(estate.Default.Bodies[psqlCreate]!); + // Every session has to go before psql: --clean cannot DROP DATABASE under a live connection, + // and the script would then merge the dump into the old database instead of replacing it. The + // execs are found by what they run rather than counted, so the staging step moving onto the + // untimed client (and off this recorder) cannot silently shift what is being asserted. + var terminateAt = ExecIndex(estate, "pg_terminate_backend"); + var replayAt = ExecIndex(estate, "ON_ERROR_STOP=0"); + Assert.True(terminateAt < replayAt, "the sessions should be closed before psql runs"); + Assert.Contains(log, l => l.StartsWith("WARNING: closed 2 open session(s) on 'db'", StringComparison.Ordinal)); + + using var replay = JsonDocument.Parse(estate.Default.Bodies[replayAt]!); Assert.Equal( new string?[] { "psql", "-U", "app", "-d", "postgres", "-w", "-v", "ON_ERROR_STOP=0", "-f", "/tmp/db.sql" }, replay.RootElement.GetProperty("Cmd").EnumerateArray().Select(e => e.GetString()).ToArray()); diff --git a/src/Watchtower.Application.Tests/RestoreCompletionTests.cs b/src/Watchtower.Application.Tests/RestoreCompletionTests.cs new file mode 100644 index 0000000..63d7ba4 --- /dev/null +++ b/src/Watchtower.Application.Tests/RestoreCompletionTests.cs @@ -0,0 +1,241 @@ +using Elarion.Settings; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Watchtower.Application.Config; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// How the first start after a restore works out whether the restore happened (ADR-0027 §5), and what +/// it repairs when it did. +/// +/// +/// The verdict rests entirely on the nonce: the restore writes one into the database it is about to +/// replace, and only a replay can remove it. That is the whole reason this can be decided at all — the +/// process that would have watched the coordinator is the process the coordinator stopped. +/// +public sealed class RestoreCompletionTests : IDisposable { + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private readonly string _stagingRoot = + Directory.CreateTempSubdirectory("wt-restore-completion").FullName; + + public void Dispose() { + try { + Directory.Delete(_stagingRoot, recursive: true); + } catch (IOException) { + // Scratch space the OS reclaims; never worth failing a passing test over. + } + } + + /// A host whose restore staging is this test's own directory, not the shared temp one. + private AuthTestHost Start() => + AuthTestHost.Start( + services => services.Replace(ServiceDescriptor.Singleton(sp => + new InstanceRestoreStaging( + sp.GetService>() + ?? NullLogger.Instance, + _stagingRoot)))); + + private static InstanceRestoreStaging Staging(AuthTestHost host) => + host.Services.GetRequiredService(); + + /// Writes the marker a restore leaves behind, as the real one does. + private static Task MarkInFlightAsync(AuthTestHost host, string nonce, params string[] stacks) => + Staging(host).WriteProgressAsync( + new RestoreProgress(nonce, DateTimeOffset.UtcNow, "source", "coordinator-id", stacks), Ct); + + /// Writes the nonce row into the database, as the real restore does before handing over. + private static async Task WriteNonceAsync(AuthTestHost host, string nonce) { + await using var scope = host.Services.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + await settings.SetStringAsync( + WatchtowerSettingPaths.RestorePendingNonce, nonce, SettingsScope.Global, + expectedVersion: null, Ct); + } + + private static async Task ReadSettingAsync(AuthTestHost host, string path) { + await using var scope = host.Services.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + return await settings.GetStringAsync(path, SettingsScope.Global, Ct); + } + + /// Runs the completion pass the way the host start does. + private static async Task CompleteAsync(AuthTestHost host) { + var completion = host.Services.GetRequiredService(); + await completion.StartAsync(Ct); + return completion; + } + + private static async Task AddStackAsync(AuthTestHost host, string name, DateTimeOffset? cursor) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var stack = new Stack { + Name = name, + ComposeProjectName = name, + Product = TestProducts.New(name), + LastScheduledBackupAt = cursor, + }; + db.Stacks.Add(stack); + await db.SaveChangesAsync(Ct); + return stack.Id; + } + + private static async Task RestoreAuditAsync(AuthTestHost host) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + return await db.AuditEvents.AsNoTracking() + .FirstOrDefaultAsync(a => a.Action == "instance.restore", Ct); + } + + [Fact] + public async Task WithNoMarkerNothingHappens() { + // The ordinary start, which every instance that has never restored anything does forever. + using var host = Start(); + + var completion = await CompleteAsync(host); + + Assert.Equal(RestoreOutcome.None, completion.LastOutcome); + Assert.Null(await RestoreAuditAsync(host)); + } + + [Fact] + public async Task TheNonceBeingGoneMeansTheReplayCommitted() { + // A replayed database is the source instance's, and the source instance never knew this nonce. + using var host = Start(); + await MarkInFlightAsync(host, "nonce-abc", "blog", "shop"); + + var completion = await CompleteAsync(host); + + Assert.Equal(RestoreOutcome.Succeeded, completion.LastOutcome); + Assert.Null(completion.LastError); + var row = await RestoreAuditAsync(host); + Assert.NotNull(row); + Assert.True(row.Success); + Assert.Contains("restored from a bundle taken from 'source'", row.Detail); + } + + [Fact] + public async Task TheNonceStillBeingThereMeansTheDatabaseWasNeverReplaced() { + // The coordinator failed and rolled back, or never got that far. Either way this instance is + // exactly as it was — which is a failure to report, not a silent no-op. + using var host = Start(); + await WriteNonceAsync(host, "nonce-abc"); + await MarkInFlightAsync(host, "nonce-abc"); + + var completion = await CompleteAsync(host); + + Assert.Equal(RestoreOutcome.Failed, completion.LastOutcome); + Assert.Contains("running on the database it had", completion.LastError); + var row = await RestoreAuditAsync(host); + Assert.NotNull(row); + Assert.False(row.Success); + // The marker row is this instance's own litter now, and means nothing in a database that stays. + Assert.Null(await ReadSettingAsync(host, WatchtowerSettingPaths.RestorePendingNonce)); + } + + [Fact] + public async Task AFailedRestoreKeepsTheUploadSoItCanBeRetried() { + using var host = Start(); + var uploadDirectory = Staging(host).NewUploadDirectory(); + await WriteNonceAsync(host, "nonce-abc"); + await MarkInFlightAsync(host, "nonce-abc"); + + await CompleteAsync(host); + + Assert.True(Directory.Exists(uploadDirectory)); + // The marker is gone, so a later restart does not re-report the same failure. + Assert.Null(Staging(host).ReadProgress()); + } + + [Fact] + public async Task ASucceededRestoreClearsTheBundleAndTheMarker() { + // The bundle carries every secret the source instance had; once it has been used it is only a + // copy of the instance lying around in a container. + using var host = Start(); + await MarkInFlightAsync(host, "nonce-abc"); + + await CompleteAsync(host); + + Assert.Null(Staging(host).ReadProgress()); + Assert.Null(Staging(host).Current); + } + + [Fact] + public async Task TheBackupCursorsAreClampedSoTheRestoreDoesNotFireAFleetOfBackups() { + // The restored rows carry the source instance's cursors. Left alone, every window between its + // dump and now looks missed, and the misfire grace would back up every stack at once — against + // volumes that have not been redeployed yet. + using var host = Start(); + var stale = new DateTimeOffset(2020, 1, 1, 0, 0, 0, TimeSpan.Zero); + var stackId = await AddStackAsync(host, "blog", stale); + await MarkInFlightAsync(host, "nonce-abc"); + + await CompleteAsync(host); + + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var cursor = await db.Stacks.AsNoTracking() + .Where(s => s.Id == stackId).Select(s => s.LastScheduledBackupAt).SingleAsync(Ct); + Assert.NotNull(cursor); + Assert.True(cursor > stale, "the stack's backup cursor should have been moved to now"); + Assert.NotNull(await ReadSettingAsync(host, WatchtowerSettingPaths.BackupSelfLastScheduledAt)); + } + + [Fact] + public async Task TheProxyPlaneIsToldToReprojectTheRestoredRoutes() { + // The routes table arrived wholesale; without the bump the proxy keeps serving what this + // instance had before the restore. + using var host = Start(); + var before = await ReadSettingAsync(host, WatchtowerSettingPaths.ProxyRoutesVersion); + await MarkInFlightAsync(host, "nonce-abc"); + + await CompleteAsync(host); + + var after = await ReadSettingAsync(host, WatchtowerSettingPaths.ProxyRoutesVersion); + Assert.NotNull(after); + Assert.NotEqual(before, after); + } + + [Fact] + public async Task TheRecoveryChecklistIsSeededFromTheRestoredDatabase() { + // From the database, not from the bundle's manifest: the ids the checklist has to act on are the + // restored ones. + using var host = Start(); + var blog = await AddStackAsync(host, "blog", cursor: null); + var shop = await AddStackAsync(host, "shop", cursor: null); + await MarkInFlightAsync(host, "nonce-abc", "whatever-the-manifest-said"); + + await CompleteAsync(host); + + await using var scope = host.Services.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + var checklist = await StackRevivalState.LoadAsync(settings, Ct); + Assert.NotNull(checklist); + Assert.False(checklist.Dismissed); + Assert.Equal("source", checklist.SourceInstance); + Assert.Equal([blog, shop], checklist.Stacks.Select(s => s.StackId)); + Assert.All(checklist.Stacks, s => Assert.Equal(RevivalStatus.Pending, s.Status)); + } + + [Fact] + public async Task AFailedRestoreLeavesNoChecklist() { + using var host = Start(); + await AddStackAsync(host, "blog", cursor: null); + await WriteNonceAsync(host, "nonce-abc"); + await MarkInFlightAsync(host, "nonce-abc"); + + await CompleteAsync(host); + + await using var scope = host.Services.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + Assert.Null(await StackRevivalState.LoadAsync(settings, Ct)); + } +} diff --git a/src/Watchtower.Application.Tests/SelfPostgresLocatorTests.cs b/src/Watchtower.Application.Tests/SelfPostgresLocatorTests.cs new file mode 100644 index 0000000..0300709 --- /dev/null +++ b/src/Watchtower.Application.Tests/SelfPostgresLocatorTests.cs @@ -0,0 +1,107 @@ +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// Which container the instance self-backup dumps (ADR-0027). The rule is pure, so the whole of it is +/// exercised here without a daemon — and it has to be exact: the container this picks is the one whose +/// contents become "the backup of Watchtower", so choosing a neighbouring database would produce an +/// archive that looks entirely healthy and restores the wrong instance. +/// +public sealed class SelfPostgresLocatorTests { + private static DockerContainerInfo Container( + string name, string image = "postgres:18-alpine", string? service = null, string state = "running") => + new() { + Id = $"id-{name}", + Names = [$"/{name}"], + Image = image, + State = state, + Status = $"Up 2 hours", + Labels = service is null + ? [] + : new Dictionary { ["com.docker.compose.service"] = service }, + }; + + [Fact] + public void PicksTheContainerWhoseComposeServiceIsTheConnectionHost() { + var postgres = Container("watchtower-postgres-1", service: "postgres"); + var other = Container("shop-db-1", service: "db"); + + Assert.Same(postgres, SelfPostgresLocator.Choose([other, postgres], "postgres")); + } + + [Fact] + public void PicksTheContainerNamedAfterTheConnectionHost() { + var postgres = Container("watchtower-pg"); + Assert.Same(postgres, SelfPostgresLocator.Choose([Container("other-db"), postgres], "watchtower-pg")); + } + + [Fact] + public void SeesThroughComposesReplicaSuffix() { + // "Host=postgres" resolves to the service; the container it created is "{project}-postgres-1". + var postgres = Container("watchtower-postgres-1"); + Assert.Same(postgres, SelfPostgresLocator.Choose([postgres, Container("shop-db-1")], "postgres")); + } + + [Fact] + public void ASingleCandidateWinsEvenWhenTheHostNamesNothing() { + // A Compose install whose service is aliased differently from the host is ordinary, and with one + // database on the daemon there is nothing to confuse it with. + var postgres = Container("db-1", service: "db"); + Assert.Same(postgres, SelfPostgresLocator.Choose([postgres], "postgres.internal")); + } + + [Fact] + public void RefusesToGuessBetweenSeveralUnmatchedCandidates() { + var error = Assert.Throws(() => + SelfPostgresLocator.Choose([Container("a-db-1"), Container("b-db-1")], "postgres.internal")); + + Assert.Contains("more than one to choose from", error.Message); + Assert.Contains("a-db-1", error.Message); + Assert.Contains("b-db-1", error.Message); + // The way out is always named, so the message is actionable rather than merely correct. + Assert.Contains("Watchtower:Backup:SelfPostgresContainer", error.Message); + } + + [Fact] + public void RefusesWhenSeveralContainersAnswerToTheSameHost() { + var error = Assert.Throws(() => + SelfPostgresLocator.Choose( + [Container("postgres", service: "postgres"), Container("wt-postgres-1", service: "postgres")], + "postgres")); + + Assert.Contains("More than one", error.Message); + Assert.Contains("Watchtower:Backup:SelfPostgresContainer", error.Message); + } + + [Fact] + public void FailsLoudlyWhenThereIsNoDatabaseContainerAtAll() { + // The managed-PostgreSQL case, and also the daemon-unreachable case. Both have to say so: a + // self-backup that quietly does nothing is worse than one that fails, because it is invisible + // until the day it is needed. + var error = Assert.Throws(() => + SelfPostgresLocator.Choose([], "db.eu-central-1.rds.amazonaws.com")); + + Assert.Contains("db.eu-central-1.rds.amazonaws.com", error.Message); + Assert.Contains("managed or host-installed PostgreSQL", error.Message); + Assert.Contains("Docker daemon could not be reached", error.Message); + } + + [Fact] + public void TheDumpTargetCarriesAStableServiceIdentity() { + // Whatever the container is called, the SQL lands at backup/_dumps/watchtower.sql — so a restore + // looks for one name rather than for whatever the source instance happened to name its container. + var target = new SelfPostgresTarget( + "abc123", "watchtower-postgres-1", "postgres:18-alpine", "postgres", "watchtower", "watchtower"); + var dump = target.ToDumpTarget(); + + Assert.Equal("watchtower", dump.Service); + Assert.Equal("abc123", dump.ContainerId); + Assert.Equal(DumpEngine.Postgres, dump.Engine); + // No volumes: an instance archive is the dump and nothing else, so there is no file snapshot for + // a data volume to be excluded from. + Assert.Null(dump.DataVolume); + Assert.Empty(dump.MountedVolumes); + } +} diff --git a/src/Watchtower.Application.Tests/StackRevivalTests.cs b/src/Watchtower.Application.Tests/StackRevivalTests.cs new file mode 100644 index 0000000..ba30546 --- /dev/null +++ b/src/Watchtower.Application.Tests/StackRevivalTests.cs @@ -0,0 +1,287 @@ +using Elarion.Settings; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; +using Xunit; + +namespace Watchtower.Application.Tests; + +/// +/// Bringing the stacks back after an instance restore (ADR-0027 §6). The order is the contract: a stack +/// is deployed first, because only a deploy creates the volumes a restore needs, and restored second, +/// because a deploy alone leaves it running on empty ones. Everything else here is about saying clearly +/// which of the two went wrong. +/// +public sealed class StackRevivalTests : IDisposable { + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private readonly string _storageRoot = Directory.CreateTempSubdirectory("wt-revival-tests").FullName; + + public void Dispose() { + try { + Directory.Delete(_storageRoot, recursive: true); + } catch (IOException) { + // Scratch space the OS reclaims. + } + } + + private AuthTestHost Start() => AuthTestHost.Start( + ("Watchtower:Backup:Provider", "local"), + ("Watchtower:Backup:Local:BasePath", _storageRoot), + ("Watchtower:Backup:InstanceName", "prod")); + + /// + /// A coordinator over queues whose runs are already finished. The state machine is what is under + /// test; how long a real deploy takes is the queues' business, and waiting for one here would only + /// test . + /// + private static (StackRevivalCoordinator Coordinator, TerminalDeployQueue Deploys, TerminalBackupQueue Backups) + Coordinator(AuthTestHost host, string deployStatus = "success", string restoreStatus = "success") { + var scopeFactory = host.Services.GetRequiredService(); + var deploys = new TerminalDeployQueue(host.Services, deployStatus); + var backups = new TerminalBackupQueue(host.Services, restoreStatus); + return ( + new StackRevivalCoordinator( + deploys, backups, + host.Services.GetRequiredService(), + scopeFactory, + host.Services.GetRequiredService>(), + TimeProvider.System, + NullLogger.Instance, + stepTimeout: TimeSpan.FromSeconds(10), + pollInterval: TimeSpan.FromMilliseconds(5)), + deploys, backups); + } + + private static async Task AddStackAsync(AuthTestHost host, string name) { + await using var scope = host.Services.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var stack = new Stack { Name = name, ComposeProjectName = name, Product = TestProducts.New(name) }; + db.Stacks.Add(stack); + await db.SaveChangesAsync(Ct); + return stack.Id; + } + + /// Seeds the checklist the completion pass would have written. + private static async Task SeedChecklistAsync(AuthTestHost host, params (int Id, string Name)[] stacks) { + await using var scope = host.Services.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + await new StackRevivalState( + DateTimeOffset.UtcNow, "source", Dismissed: false, + [.. stacks.Select(s => new RevivalStack(s.Id, s.Name, RevivalStatus.Pending))]) + .SaveAsync(settings, Ct); + } + + /// Puts an archive on the storage where the stack's restore would look for it. + private async Task SeedArchiveAsync(string stackName) { + var name = BackupNaming.FileName( + stackName, new DateTimeOffset(2026, 8, 25, 3, 30, 0, TimeSpan.Zero), encrypted: true); + var directory = Path.Combine(_storageRoot, "prod", stackName); + Directory.CreateDirectory(directory); + await File.WriteAllTextAsync(Path.Combine(directory, name), "archive", Ct); + return name; + } + + [Fact] + public async Task AStackIsDeployedThenRestoredFromItsNewestArchive() { + using var host = Start(); + var stackId = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (stackId, "blog")); + var archive = await SeedArchiveAsync("blog"); + var (coordinator, deploys, backups) = Coordinator(host); + + var result = await coordinator.ReviveAsync(stackId, Ct); + + Assert.NotNull(result); + Assert.Equal(RevivalStatus.Done, result.Status); + Assert.Contains(archive, result.Detail); + Assert.Equal([stackId], deploys.Enqueued); + Assert.Equal([(stackId, archive)], backups.Restored); + // Both runs are linked from the row, so the operator can read either log. + Assert.NotNull(result.DeployEventId); + Assert.NotNull(result.BackupEventId); + } + + [Fact] + public async Task AStackWithNoArchiveIsDeployedAndSaysNothingWasRestored() { + // Its definition came back with the database; its data never existed on the storage. Reporting + // that as done-with-a-note is honest — reporting it as failed would not be. + using var host = Start(); + var stackId = await AddStackAsync(host, "fresh"); + await SeedChecklistAsync(host, (stackId, "fresh")); + var (coordinator, deploys, backups) = Coordinator(host); + + var result = await coordinator.ReviveAsync(stackId, Ct); + + Assert.Equal(RevivalStatus.Done, result!.Status); + Assert.Contains("nothing was restored", result.Detail); + Assert.Equal([stackId], deploys.Enqueued); + Assert.Empty(backups.Restored); + } + + [Fact] + public async Task AFailedDeployStopsBeforeTheRestore() { + // Restoring into volumes a failed deploy never created would either fail confusingly or, worse, + // succeed against the wrong ones. + using var host = Start(); + var stackId = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (stackId, "blog")); + await SeedArchiveAsync("blog"); + var (coordinator, _, backups) = Coordinator(host, deployStatus: "failed"); + + var result = await coordinator.ReviveAsync(stackId, Ct); + + Assert.Equal(RevivalStatus.Failed, result!.Status); + Assert.Contains("The deploy failed", result.Detail); + Assert.Empty(backups.Restored); + } + + [Fact] + public async Task AFailedRestoreIsReportedAsSuch() { + using var host = Start(); + var stackId = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (stackId, "blog")); + await SeedArchiveAsync("blog"); + var (coordinator, _, _) = Coordinator(host, restoreStatus: "failed"); + + var result = await coordinator.ReviveAsync(stackId, Ct); + + Assert.Equal(RevivalStatus.Failed, result!.Status); + Assert.Contains("The restore failed", result.Detail); + } + + [Fact] + public async Task ReviveAllTakesThePendingAndFailedOnesAndLeavesTheRest() { + using var host = Start(); + var blog = await AddStackAsync(host, "blog"); + var shop = await AddStackAsync(host, "shop"); + var done = await AddStackAsync(host, "already-done"); + await SeedChecklistAsync(host, (blog, "blog"), (shop, "shop"), (done, "already-done")); + + // Mark one done and one failed, as a half-finished pass would have left them. + await using (var scope = host.Services.CreateAsyncScope()) { + var settings = scope.ServiceProvider.GetRequiredService(); + var checklist = (await StackRevivalState.LoadAsync(settings, Ct))!; + await checklist + .With(new RevivalStack(done, "already-done", RevivalStatus.Done, "Deployed.")) + .With(new RevivalStack(shop, "shop", RevivalStatus.Failed, "The deploy failed.")) + .SaveAsync(settings, Ct); + } + + var (coordinator, deploys, _) = Coordinator(host); + var revived = await coordinator.ReviveAllAsync(Ct); + + Assert.Equal(2, revived); + // The pending one and the failed one — a failed stack is exactly what "revive all" should retry. + Assert.Equal([blog, shop], deploys.Enqueued.Order()); + var checklistAfter = await coordinator.LoadAsync(Ct); + Assert.All(checklistAfter!.Stacks, s => Assert.Equal(RevivalStatus.Done, s.Status)); + } + + [Fact] + public async Task ASkippedStackIsLeftAloneByReviveAll() { + using var host = Start(); + var blog = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (blog, "blog")); + var (coordinator, deploys, _) = Coordinator(host); + + var skipped = await coordinator.SkipAsync(blog, Ct); + Assert.Equal(RevivalStatus.Skipped, skipped!.Status); + + Assert.Equal(0, await coordinator.ReviveAllAsync(Ct)); + Assert.Empty(deploys.Enqueued); + } + + [Fact] + public async Task DismissingKeepsTheChecklistButStopsOfferingIt() { + // The record of what happened is the audit trail; this is only the prompt. + using var host = Start(); + var blog = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (blog, "blog")); + var (coordinator, _, _) = Coordinator(host); + + await coordinator.DismissAsync(Ct); + + var checklist = await coordinator.LoadAsync(Ct); + Assert.NotNull(checklist); + Assert.True(checklist.Dismissed); + } + + [Fact] + public async Task AStackThatIsNotOnTheChecklistIsNotRevived() { + using var host = Start(); + var blog = await AddStackAsync(host, "blog"); + await SeedChecklistAsync(host, (blog, "blog")); + var (coordinator, deploys, _) = Coordinator(host); + + Assert.Null(await coordinator.ReviveAsync(stackId: 9999, Ct)); + Assert.Empty(deploys.Enqueued); + } +} + +/// A deploy queue whose runs are already over, with an outcome the test chose. +internal sealed class TerminalDeployQueue(IServiceProvider services, string status) + : DeployQueueService( + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService>(), + NullLogger.Instance) { + private readonly List _enqueued = []; + + /// The stacks a deploy was asked for, in call order. + public IReadOnlyList Enqueued { + get { lock (_enqueued) return [.. _enqueued]; } + } + + public override DeployEnqueueResult Enqueue( + int stackId, string triggeredBy, IReadOnlyList? removeVolumes = null) { + using var scope = services.GetRequiredService().CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var deployEvent = new DeployEvent { + StackId = stackId, TriggeredBy = triggeredBy, Status = status, + StartedAt = DateTimeOffset.UtcNow, FinishedAt = DateTimeOffset.UtcNow, + }; + db.DeployEvents.Add(deployEvent); + db.SaveChanges(); + lock (_enqueued) _enqueued.Add(stackId); + return new DeployEnqueueResult(deployEvent.Id, status); + } +} + +/// A backup queue whose restores are already over, with an outcome the test chose. +internal sealed class TerminalBackupQueue(IServiceProvider services, string status) + : BackupQueueService( + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService(), + NullLogger.Instance) { + private readonly List<(int StackId, string FileName)> _restored = []; + + /// Every restore asked for, in call order. + public IReadOnlyList<(int StackId, string FileName)> Restored { + get { lock (_restored) return [.. _restored]; } + } + + public override BackupEnqueueResult? TryEnqueueRestore(int stackId, string fileName) { + using var scope = services.GetRequiredService().CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var backupEvent = new BackupEvent { + StackId = stackId, TriggeredBy = BackupTriggers.Restore, Status = status, + StartedAt = DateTimeOffset.UtcNow, FinishedAt = DateTimeOffset.UtcNow, + }; + db.BackupEvents.Add(backupEvent); + db.SaveChanges(); + lock (_restored) _restored.Add((stackId, fileName)); + return new BackupEnqueueResult(backupEvent.Id, status); + } +} diff --git a/src/Watchtower.Application.Tests/TestBundles.cs b/src/Watchtower.Application.Tests/TestBundles.cs new file mode 100644 index 0000000..e8ad4d8 --- /dev/null +++ b/src/Watchtower.Application.Tests/TestBundles.cs @@ -0,0 +1,192 @@ +using System.Formats.Tar; +using System.IO.Compression; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Tests; + +/// +/// Builds full backup bundles in memory, so the restore's refusals can be tested one at a time against +/// a bundle that is correct in every other respect. +/// +/// +/// The archives are assembled here rather than produced by a real backup run: the format is what the +/// restore reads, so writing it directly is the honest way to test reading it — and a real run would +/// need a Docker daemon and a second PostgreSQL to produce one byte of it. +/// +internal static class TestBundles { + /// The passphrase every bundle this class builds is encrypted with. + public const string Passphrase = "bundle-passphrase"; + + /// What a valid instance archive's dump file is called inside it. + private const string DumpFile = "_dumps/watchtower.sql"; + + /// + /// An encrypted, gzipped instance archive — the real layout: a manifest and a dump under + /// backup/. + /// + /// The dump's content. Its bytes are never parsed, only carried. + /// False writes an archive carrying no dump at all. + public static byte[] InstanceArchive(string sql = "-- pg_dumpall output", bool withDump = true) { + var manifest = InstanceBackupService.BuildManifest( + "source", new SelfPostgresTarget( + "abc", "watchtower-postgres-1", "postgres:18-alpine", "postgres", "watchtower", "watchtower"), + DateTimeOffset.UtcNow, + new BackupService.BackupDumpEntry( + "watchtower", DumpEngine.Postgres, DumpFile, "postgres:18-alpine", "watchtower", + "watchtower-postgres-1", [], ["watchtower"], sql.Length), + // The archive's *own* manifest. Nothing reads this id — the restore decides on the bundle + // manifest's, which Build takes from the caller — so it is deliberately not a real one. + lastMigrationId: "00000000000000_TestArchive"); + + using var plain = new MemoryStream(); + using (var gzip = new GZipStream(plain, CompressionLevel.Fastest, leaveOpen: true)) + using (var writer = new TarWriter(gzip, TarEntryFormat.Pax, leaveOpen: true)) { + WriteEntry(writer, "backup/backup-manifest.json", Encoding.UTF8.GetBytes(manifest)); + if (withDump) WriteEntry(writer, $"backup/{DumpFile}", Encoding.UTF8.GetBytes(sql)); + } + + using var encrypted = new MemoryStream(); + var cipher = BackupEncryption.CreateEncryptingStream(encrypted, Passphrase); + plain.Position = 0; + plain.CopyTo(cipher); + cipher.Dispose(); + return encrypted.ToArray(); + } + + /// How one bundle differs from the valid one, for a test that pins a single refusal. + /// Null keeps the migration this build knows. + /// The secret the source instance's keys are under. + /// Null keeps the current format version. + /// Records a checksum the archive does not have. + /// Lists the archive in the manifest but leaves it out of the tar. + /// Encrypts the archive with a passphrase the secrets file does not name. + /// Writes an instance archive that carries no dump. + /// Stack names to carry archives for. + /// Stack names to describe with no archive. + internal sealed record Options( + string? LastMigrationId = null, + string? KeyProtectionSecret = null, + int? BundleFormatVersion = null, + bool CorruptInstanceDigest = false, + bool OmitInstanceArchive = false, + bool WrongPassphrase = false, + bool WithoutDump = false, + IReadOnlyList? Stacks = null, + IReadOnlyList? MissingStacks = null); + + /// Builds one bundle tar. + /// The migration this build actually knows, so the default is valid. + /// How this bundle should differ from a valid one. + public static byte[] Build(string? lastMigrationId, Options? options = null) { + var o = options ?? new Options(); + var instanceBytes = o.WrongPassphrase + ? Reencrypt(InstanceArchive(withDump: !o.WithoutDump), "a-different-passphrase") + : InstanceArchive(withDump: !o.WithoutDump); + var instanceEntry = $"{BackupBundle.InstanceDirectory}/watchtower_20260826T033000Z.tar.gz.enc"; + var instance = new BundleArchive( + instanceEntry, "source/_watchtower/watchtower_20260826T033000Z.tar.gz.enc", + instanceBytes.Length, + o.CorruptInstanceDigest ? new string('0', 64) : Sha256(instanceBytes), + DateTimeOffset.UtcNow, Encrypted: true); + + var members = new List<(string Entry, byte[] Content)>(); + if (!o.OmitInstanceArchive) members.Add((instanceEntry, instanceBytes)); + + var stacks = new List(); + foreach (var (name, index) in (o.Stacks ?? []).Select((n, i) => (n, i))) { + var content = Encoding.UTF8.GetBytes($"archive-for-{name}"); + var storagePath = $"source/{name}/{name}_2026082{index}T033000Z.tar.gz.enc"; + var entry = $"{BackupBundle.StacksDirectory}/{storagePath}"; + members.Add((entry, content)); + stacks.Add(new BundleStack( + index + 1, name, name, + new BundleArchive( + entry, storagePath, content.Length, Sha256(content), DateTimeOffset.UtcNow, true), + null)); + } + foreach (var (name, index) in (o.MissingStacks ?? []).Select((n, i) => (n, i))) + stacks.Add(new BundleStack( + 100 + index, name, name, null, "no archive on the backup storage")); + + var manifest = new BundleManifest( + o.BundleFormatVersion ?? BackupBundle.FormatVersion, + "watchtower", + DateTimeOffset.UtcNow, + "source", + "9.9.9-test", + o.LastMigrationId ?? lastMigrationId, + KeyProtectionSecretConfigured: o.KeyProtectionSecret is { Length: > 0 }, + instance, + stacks); + + var secrets = new BundleSecrets( + SecretsFormatVersion: 1, + KeyProtectionSecret: o.KeyProtectionSecret, + BackupEncryptionPassphrase: Passphrase, + BackupInstanceName: "source", + Storage: new BundleStorageSecrets( + "local", new BundleSftpSecrets(null, 22, null, null, null, null, ""), "/backups")); + + using var tar = new MemoryStream(); + using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true)) { + WriteEntry(writer, BackupBundle.ManifestEntry, Json(manifest)); + WriteEntry(writer, BackupBundle.SecretsEntry, Json(secrets)); + foreach (var (entry, content) in members) WriteEntry(writer, entry, content); + } + return tar.ToArray(); + } + + /// + /// A tar whose entry name would escape the directory it is unpacked into. Both shapes are here: a + /// leading ../, and one buried after a legitimate-looking segment — the second is what a + /// prefix-stripping guard would miss. + /// + public static byte[] TraversalBundle(string entryName = "stacks/../../escaped.json") { + using var tar = new MemoryStream(); + using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true)) + WriteEntry(writer, entryName, Encoding.UTF8.GetBytes("{}")); + return tar.ToArray(); + } + + /// A tar with no manifest in it at all. + public static byte[] NotABundle() { + using var tar = new MemoryStream(); + using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true)) + WriteEntry(writer, "readme.txt", Encoding.UTF8.GetBytes("not a bundle")); + return tar.ToArray(); + } + + private static byte[] Reencrypt(byte[] archive, string passphrase) { + // Decrypt with the class passphrase and re-encrypt with another, so the bytes are a real + // archive that simply cannot be opened with what the secrets file names. + using var source = new MemoryStream(archive); + using var plain = new MemoryStream(); + var decrypting = BackupEncryption.CreateDecryptingStream(source, Passphrase); + decrypting.CopyTo(plain); + decrypting.Dispose(); + + using var result = new MemoryStream(); + var cipher = BackupEncryption.CreateEncryptingStream(result, passphrase); + plain.Position = 0; + plain.CopyTo(cipher); + cipher.Dispose(); + return result.ToArray(); + } + + private static void WriteEntry(TarWriter writer, string name, byte[] content) { + using var data = new MemoryStream(content); + writer.WriteEntry(new PaxTarEntry(TarEntryType.RegularFile, name) { + Mode = UnixFileMode.UserRead | UnixFileMode.UserWrite, + DataStream = data, + }); + } + + private static byte[] Json(T value) => + JsonSerializer.SerializeToUtf8Bytes(value, BackupBundle.JsonOptions); + + private static string Sha256(byte[] content) => + Convert.ToHexStringLower(SHA256.HashData(content)); +} diff --git a/src/Watchtower.Application/Config/WatchtowerOptions.cs b/src/Watchtower.Application/Config/WatchtowerOptions.cs index 8b27f42..2af6116 100644 --- a/src/Watchtower.Application/Config/WatchtowerOptions.cs +++ b/src/Watchtower.Application/Config/WatchtowerOptions.cs @@ -255,6 +255,22 @@ public sealed record BackupOptions { /// The default for . public const int DefaultStopTimeoutSeconds = 5; + /// + /// Whether the schedule also backs up Watchtower's own database (ADR-0027), as a dumps-only archive + /// under . On by default: an instance whose + /// stacks are backed up but whose own state is not can restore every stack's data and none of the + /// configuration that deploys it. Needs — the dump carries every + /// database role's password hash — and a PostgreSQL that runs as a container on this daemon. + /// + public bool IncludeSelf { get; init; } = true; + + /// + /// The container running Watchtower's own PostgreSQL, by name or id, when + /// cannot work it out — several database containers on + /// one daemon and none of them answering to the connection string's host. Blank = detect it. + /// + public string? SelfPostgresContainer { get; init; } + /// Storage backend the archives are shipped to: sftp (default) or local. public string Provider { get; init; } = "sftp"; diff --git a/src/Watchtower.Application/Config/WatchtowerSettingPaths.cs b/src/Watchtower.Application/Config/WatchtowerSettingPaths.cs index 04b8a20..e127ca2 100644 --- a/src/Watchtower.Application/Config/WatchtowerSettingPaths.cs +++ b/src/Watchtower.Application/Config/WatchtowerSettingPaths.cs @@ -108,6 +108,30 @@ public static class WatchtowerSettingPaths { public const string BackupEncryptionPassphrase = "Watchtower:Backup:EncryptionPassphrase"; public const string BackupHelperImage = "Watchtower:Backup:HelperImage"; public const string BackupProvider = "Watchtower:Backup:Provider"; + /// Whether the schedule also dumps Watchtower's own database (ADR-0027). + public const string BackupIncludeSelf = "Watchtower:Backup:IncludeSelf"; + /// Explicit container for Watchtower's own PostgreSQL, when detection cannot pick one. + public const string BackupSelfPostgresContainer = "Watchtower:Backup:SelfPostgresContainer"; + /// + /// The instance self-backup's schedule cursor — the due time of the last window that was enqueued, + /// the stackless counterpart of . Written by the + /// schedule tick, never offered in the UI. + /// + public const string BackupSelfLastScheduledAt = "Watchtower:Backup:SelfLastScheduledAt"; + + /// + /// The nonce an in-flight instance restore writes into the database it is about to replace + /// (ADR-0027 §5). After the restart its absence is the proof that the replay committed — + /// nothing else can remove it, because nothing else knows it. Never offered in the UI. + /// + public const string RestorePendingNonce = "Watchtower:Restore:PendingNonce"; + + /// + /// The post-restore recovery checklist (ADR-0027 §6), as JSON. A settings row because there is at + /// most one, it has to survive the restart the restore itself causes, and a table for it would be a + /// schema change carried by every instance that never restores anything. + /// + public const string RestoreRecovery = "Watchtower:Restore:Recovery"; public const string BackupSftpHost = "Watchtower:Backup:Sftp:Host"; public const string BackupSftpPort = "Watchtower:Backup:Sftp:Port"; public const string BackupSftpUsername = "Watchtower:Backup:Sftp:Username"; diff --git a/src/Watchtower.Application/Entities/BackupEvent.cs b/src/Watchtower.Application/Entities/BackupEvent.cs index 8e00d78..1c9fc2f 100644 --- a/src/Watchtower.Application/Entities/BackupEvent.cs +++ b/src/Watchtower.Application/Entities/BackupEvent.cs @@ -19,10 +19,19 @@ public static class BackupStatuses { public const string Failed = "failed"; } -/// Records the status and outcome of a single stack backup run (ADR-0016). +/// +/// Records the status and outcome of a single backup run — a stack's (ADR-0016) or the instance's own +/// (ADR-0027). +/// public sealed class BackupEvent { public int Id { get; set; } - public int StackId { get; set; } + + /// + /// The stack this run belongs to, or null for a run that has no stack: the instance self-backup and + /// the bundle export back up Watchtower itself (ADR-0027), so there is no stack to point at. The + /// history views and the single-flight queue both read this as the run's kind. + /// + public int? StackId { get; set; } public Stack? Stack { get; set; } /// Who triggered the backup: "manual" or "schedule". public required string TriggeredBy { get; set; } diff --git a/src/Watchtower.Application/Modules/Backups/BackupScheduleJob.cs b/src/Watchtower.Application/Modules/Backups/BackupScheduleJob.cs index a31234c..0425d7f 100644 --- a/src/Watchtower.Application/Modules/Backups/BackupScheduleJob.cs +++ b/src/Watchtower.Application/Modules/Backups/BackupScheduleJob.cs @@ -1,5 +1,7 @@ using System.Collections.Concurrent; +using System.Globalization; using Elarion.Abstractions.Scheduling; +using Elarion.Settings; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -33,6 +35,7 @@ namespace Watchtower.Application.Modules.Backups; public sealed class BackupScheduleJob( WatchtowerDbContext db, BackupQueueService queue, + ISettingsManager settings, IOptionsMonitor options, TimeProvider timeProvider, ILogger logger) { @@ -43,6 +46,9 @@ public sealed class BackupScheduleJob( // not move for a skip, so without this the same skip would be logged every minute until the next run. private static readonly ConcurrentDictionary LoggedMisses = new(); + /// The instance self-backup's missed window, deduplicated like . + private static DateTimeOffset? _loggedInstanceMiss; + [ScheduledJob(JobName, FixedRate = "1m", Overlap = ScheduledJobOverlap.Skip)] public async ValueTask RunAsync(CancellationToken ct) => await TickAsync(timeProvider.GetUtcNow(), TimeZoneInfo.Local, ct); @@ -64,6 +70,12 @@ public async ValueTask TickAsync(DateTimeOffset now, TimeZoneInfo timeZone, } var grace = BackupSchedule.ResolveMisfireGrace(backup); + // Watchtower's own database runs on the instance-wide expression — there is one instance, so + // there is nothing to override it with (ADR-0027). + var enqueuedInstance = globalCron is null + ? 0 + : await TickInstanceAsync(backup, globalCron, now, grace, timeZone, ct); + // The ladder in SQL: a stack that says yes, or a tenant that says nothing over a template that // says yes. Kept as a predicate rather than "load everything and resolve in memory" because this // runs once a minute against every stack on the box. `BackupPolicyResolver` is still the only @@ -74,9 +86,9 @@ public async ValueTask TickAsync(DateTimeOffset now, TimeZoneInfo timeZone, || (s.BackupEnabled == null && s.Template != null && s.Template.BackupEnabled == true)) .OrderBy(s => s.Name) .ToListAsync(ct); - if (stacks.Count == 0) return 0; + if (stacks.Count == 0) return enqueuedInstance; - var enqueued = 0; + var enqueued = enqueuedInstance; foreach (var stack in stacks) { var policy = BackupPolicyResolver.Resolve(stack, stack.Template); if (!policy.Enabled) continue; @@ -119,4 +131,74 @@ public async ValueTask TickAsync(DateTimeOffset now, TimeZoneInfo timeZone, if (enqueued > 0) await db.SaveChangesAsync(ct); return enqueued; } + + /// + /// The same window evaluation for Watchtower's own database (ADR-0027). Returns 1 when it enqueued. + /// + /// + /// The cursor is a settings row rather than a column, because there is no instance table to put it on + /// — , read through the settings manager + /// rather than the options monitor so a value written last tick is certainly seen this tick (the + /// configuration snapshot reloads asynchronously, and a stale cursor would fire the window twice). + /// + private async ValueTask TickInstanceAsync( + BackupOptions backup, CronExpression cron, DateTimeOffset now, TimeSpan grace, + TimeZoneInfo timeZone, CancellationToken ct) { + if (!backup.IncludeSelf) return 0; + + var cursor = ParseCursor(await settings.GetStringAsync( + WatchtowerSettingPaths.BackupSelfLastScheduledAt, SettingsScope.Global, ct)); + + ScheduleDecision decision; + try { + decision = BackupSchedule.Evaluate(cron, now, cursor, grace, timeZone); + } catch (InvalidOperationException ex) { + logger.LogWarning(ex, "The backup schedule has no upcoming window for Watchtower's own database"); + return 0; + } + + if (decision.MissedAt is { } missed && _loggedInstanceMiss != missed) { + _loggedInstanceMiss = missed; + logger.LogInformation( + "Backup window {Window:o} for Watchtower's own database was missed (older than the {Grace} " + + "misfire grace); skipped", missed, grace); + } + + if (decision.DueAt is not { } due) return 0; + + // Refused rather than run: without a passphrase the run would fail every night and fill the + // history with failures, and the dump it would have written carries every role's password hash. + // The cursor still moves — a window that was evaluated is a window that is over, and leaving it + // open would re-fire it (and re-log) on every tick until the passphrase appears. + await StoreCursorAsync(due, ct); + if (string.IsNullOrEmpty(backup.EncryptionPassphrase)) { + logger.LogWarning( + "Backup window {Window:o} open for Watchtower's own database, but no encryption passphrase " + + "is configured — skipping. Set one under Settings → Backups.", due); + return 0; + } + + var result = queue.EnqueueInstance(BackupTriggers.Schedule); + logger.LogInformation( + "Backup window {Window:o} open — enqueued Watchtower's own database (event {EventId})", + due, result.BackupEventId); + return 1; + } + + /// Writes the instance cursor round-trip formatted, so it parses back exactly. + private async ValueTask StoreCursorAsync(DateTimeOffset due, CancellationToken ct) => + await settings.SetStringAsync( + WatchtowerSettingPaths.BackupSelfLastScheduledAt, due.UtcDateTime.ToString("O"), + SettingsScope.Global, expectedVersion: null, ct); + + /// + /// The stored cursor, or null when there is none — or when it is unreadable, which is treated as + /// "never ran": the misfire grace then bounds how far back the first window can be. + /// + private static DateTimeOffset? ParseCursor(string? stored) => + DateTimeOffset.TryParse( + stored, CultureInfo.InvariantCulture, + DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var cursor) + ? cursor + : null; } diff --git a/src/Watchtower.Application/Modules/Backups/BackupsModule.cs b/src/Watchtower.Application/Modules/Backups/BackupsModule.cs index 6bc5d71..ca268cf 100644 --- a/src/Watchtower.Application/Modules/Backups/BackupsModule.cs +++ b/src/Watchtower.Application/Modules/Backups/BackupsModule.cs @@ -34,7 +34,10 @@ public sealed record BackupConfigDto( string Provider, BackupSftpConfigDto Sftp, string LocalBasePath, - string[] PinnedPaths) { + string[] PinnedPaths, + bool IncludeSelf, + string? SelfPostgresContainer, + string InstanceDirectory) { internal static BackupConfigDto From(BackupOptions backup, EnvironmentSettingPins pins) => new( Enabled: backup.Enabled, Cron: BackupSchedule.ResolveGlobalExpression(backup), @@ -53,7 +56,12 @@ public sealed record BackupConfigDto( HasPrivateKey: !string.IsNullOrEmpty(backup.Sftp.PrivateKey), BasePath: backup.Sftp.BasePath), LocalBasePath: backup.Local.BasePath, - PinnedPaths: ResolvePinnedPaths(pins)); + PinnedPaths: ResolvePinnedPaths(pins), + IncludeSelf: backup.IncludeSelf, + SelfPostgresContainer: backup.SelfPostgresContainer, + // Shown, not configured: the operator needs to know where to look on the storage, and the layout + // is derived from the instance name rather than settable (ADR-0027). + InstanceDirectory: BackupNaming.InstanceDirectory(backup.ResolveInstanceName())); /// /// The pinned paths, with the legacy Backup:Time env var reported as pinning the schedule @@ -77,18 +85,44 @@ public sealed record BackupSftpConfigDto( bool HasPrivateKey, string BasePath); -/// One backup run for the history views (per stack and instance-wide). +/// One backup run for the history views (per stack, per product, and instance-wide). +/// The event's id. +/// +/// The stack the run belongs to, or null for a run that backed up Watchtower itself (ADR-0027). +/// +/// The stack's name, or null when there is no stack — see . +/// Who or what started the run (see ). +/// "queued", "running", "success", or "failed". +/// Provider-relative path of the archive, once it has one. +/// Size of the uploaded archive. +/// The run log. +/// When the run started. +/// When it reached a terminal state, or null while it has not. +/// +/// stack or instance: what this run backed up. Derived from +/// so the UI can branch on a word rather than on a null. +/// public sealed record BackupEventDto( int Id, - int StackId, - string StackName, + int? StackId, + string? StackName, string TriggeredBy, string Status, string? RemotePath, long? SizeBytes, string? Output, DateTimeOffset StartedAt, - DateTimeOffset? FinishedAt); + DateTimeOffset? FinishedAt, + string Kind); + +/// The values takes. +public static class BackupEventKinds { + /// A stack's volumes and database dumps (ADR-0016). + public const string Stack = "stack"; + + /// Watchtower's own database (ADR-0027). + public const string Instance = "instance"; +} /// /// A stack's backup participation: schedule opt-in, the stop-for-snapshot flag, how its stateful @@ -398,6 +432,81 @@ public sealed record BackupRemoteFileDto(string Name, long SizeBytes, DateTimeOf /// Returned immediately after a run is enqueued; the event tracks progress. public sealed record BackupRunAcceptedDto(int BackupEventId, string Status); +/// +/// The full backup bundle staged for download (ADR-0027 §4). Never carries a path: the file lives in +/// Watchtower's own container and is only ever reached through GET /api/instance/bundle. +/// +/// The name it downloads as. +/// Its size. +/// When the export finished. +/// How many stack archives it carries. +/// +/// How many stacks it describes but has no archive for — a stack that has never been backed up. Its +/// definition still comes back with the database; only its data is absent. +/// +public sealed record BackupBundleDto( + string FileName, long SizeBytes, DateTimeOffset CreatedAtUtc, int StackCount, int MissingStackCount); + +/// One stack on the post-restore recovery checklist (ADR-0027 §6). +/// Its id in the restored database. +/// Its name. +/// +/// pending, deploying, restoring, done, failed or skipped. +/// +/// What last happened to it, in a sentence. +/// The deploy this revival started, for a link to its log. +/// The restore this revival started, for a link to its log. +public sealed record RecoveryStackDto( + int StackId, string Name, string Status, string? Detail, int? DeployEventId, int? BackupEventId) { + internal static RecoveryStackDto From(RevivalStack stack) => new( + stack.StackId, stack.Name, stack.Status.ToString().ToLowerInvariant(), stack.Detail, + stack.DeployEventId, stack.BackupEventId); +} + +/// The checklist an operator works through after an instance restore (ADR-0027 §6). +public sealed record RecoveryChecklistDto( + DateTimeOffset RestoredAtUtc, + string SourceInstance, + bool Dismissed, + IReadOnlyList Stacks) { + internal static RecoveryChecklistDto From(StackRevivalState state) => new( + state.RestoredAtUtc, state.SourceInstance, state.Dismissed, + [.. state.Stacks.Select(RecoveryStackDto.From)]); +} + +/// One reason a bundle cannot be restored here, or one caveat about doing so (ADR-0027 §5). +/// A stable key the UI can branch on, e.g. key-protection-secret. +/// The operator-facing sentence, which always names what to do about it. +public sealed record RestoreFindingDto(string Code, string Message) { + internal static RestoreFindingDto From(RestoreFinding finding) => new(finding.Code, finding.Message); +} + +/// +/// An uploaded bundle and this instance's verdict on it (ADR-0027 §5) — everything the wizard needs to +/// say what would happen, before anything does. +/// +public sealed record RestoreValidationDto( + bool CanRestore, + IReadOnlyList Blocking, + IReadOnlyList Warnings, + string InstanceName, + string AppVersion, + DateTimeOffset CreatedAtUtc, + int StackCount, + int MissingStackCount, + IReadOnlyList StackNames) { + internal static RestoreValidationDto From(RestoreValidation validation) => new( + validation.CanRestore, + [.. validation.Blocking.Select(RestoreFindingDto.From)], + [.. validation.Warnings.Select(RestoreFindingDto.From)], + validation.InstanceName, + validation.AppVersion, + validation.CreatedAtUtc, + validation.StackCount, + validation.MissingStackCount, + validation.StackNames); +} + /// JSON serializer context for Backups module request/response types. [JsonSourceGenerationOptions( PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, @@ -418,6 +527,33 @@ public sealed record BackupRunAcceptedDto(int BackupEventId, string Status); [JsonSerializable(typeof(ListBackupEvents.Response), TypeInfoPropertyName = "ListBackupEventsResponse")] [JsonSerializable(typeof(RunBackup.Command), TypeInfoPropertyName = "RunBackupCommand")] [JsonSerializable(typeof(RunBackup.Response), TypeInfoPropertyName = "RunBackupResponse")] +[JsonSerializable(typeof(RunInstanceBackup.Command), TypeInfoPropertyName = "RunInstanceBackupCommand")] +[JsonSerializable(typeof(RunInstanceBackup.Response), TypeInfoPropertyName = "RunInstanceBackupResponse")] +[JsonSerializable(typeof(ListInstanceBackups.Query), TypeInfoPropertyName = "ListInstanceBackupsQuery")] +[JsonSerializable(typeof(ListInstanceBackups.Response), TypeInfoPropertyName = "ListInstanceBackupsResponse")] +[JsonSerializable(typeof(BackupBundleDto))] +[JsonSerializable(typeof(ExportBackupBundle.Command), TypeInfoPropertyName = "ExportBackupBundleCommand")] +[JsonSerializable(typeof(ExportBackupBundle.Response), TypeInfoPropertyName = "ExportBackupBundleResponse")] +[JsonSerializable(typeof(GetBundleStatus.Query), TypeInfoPropertyName = "GetBundleStatusQuery")] +[JsonSerializable(typeof(GetBundleStatus.Response), TypeInfoPropertyName = "GetBundleStatusResponse")] +[JsonSerializable(typeof(RestoreFindingDto))] +[JsonSerializable(typeof(RestoreValidationDto))] +[JsonSerializable(typeof(GetInstanceRestoreStatus.Query), TypeInfoPropertyName = "GetInstanceRestoreStatusQuery")] +[JsonSerializable(typeof(GetInstanceRestoreStatus.Response), TypeInfoPropertyName = "GetInstanceRestoreStatusResponse")] +[JsonSerializable(typeof(StartInstanceRestore.Command), TypeInfoPropertyName = "StartInstanceRestoreCommand")] +[JsonSerializable(typeof(StartInstanceRestore.Response), TypeInfoPropertyName = "StartInstanceRestoreResponse")] +[JsonSerializable(typeof(RecoveryStackDto))] +[JsonSerializable(typeof(RecoveryChecklistDto))] +[JsonSerializable(typeof(GetRecoveryChecklist.Query), TypeInfoPropertyName = "GetRecoveryChecklistQuery")] +[JsonSerializable(typeof(GetRecoveryChecklist.Response), TypeInfoPropertyName = "GetRecoveryChecklistResponse")] +[JsonSerializable(typeof(ReviveStack.Command), TypeInfoPropertyName = "ReviveStackCommand")] +[JsonSerializable(typeof(ReviveStack.Response), TypeInfoPropertyName = "ReviveStackResponse")] +[JsonSerializable(typeof(ReviveAllStacks.Command), TypeInfoPropertyName = "ReviveAllStacksCommand")] +[JsonSerializable(typeof(ReviveAllStacks.Response), TypeInfoPropertyName = "ReviveAllStacksResponse")] +[JsonSerializable(typeof(SkipRecoveryStack.Command), TypeInfoPropertyName = "SkipRecoveryStackCommand")] +[JsonSerializable(typeof(SkipRecoveryStack.Response), TypeInfoPropertyName = "SkipRecoveryStackResponse")] +[JsonSerializable(typeof(DismissRecovery.Command), TypeInfoPropertyName = "DismissRecoveryCommand")] +[JsonSerializable(typeof(DismissRecovery.Response), TypeInfoPropertyName = "DismissRecoveryResponse")] [JsonSerializable(typeof(BackupRemoteFileDto))] [JsonSerializable(typeof(ListRemoteBackups.Query), TypeInfoPropertyName = "ListRemoteBackupsQuery")] [JsonSerializable(typeof(ListRemoteBackups.Response), TypeInfoPropertyName = "ListRemoteBackupsResponse")] diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/ExportBackupBundle.cs b/src/Watchtower.Application/Modules/Backups/Handlers/ExportBackupBundle.cs new file mode 100644 index 0000000..4806eb1 --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/ExportBackupBundle.cs @@ -0,0 +1,51 @@ +using Elarion.Abstractions.Authorization; +using Elarion.Abstractions.Identity; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// Builds a full backup bundle (ADR-0027 §4) — a fresh dump of Watchtower's own database plus the +/// newest archive of every stack, in one tar — and stages it for download at +/// GET /api/instance/bundle. Returns the tracking event immediately; the export runs on the +/// single-flight backup queue, since it dumps and downloads for as long as that takes. +/// +/// +/// Admin-only, and audited on both sides: the bundle carries the key-protection secret, the backup +/// passphrase and the storage credentials in plain text, so producing one is producing a portable copy +/// of the instance. +/// +[Handler("backups.exportBundle")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class ExportBackupBundle( + BackupQueueService queue, + IOptionsMonitor options, + AuditLog audit, + ICurrentUser currentUser) + : IHandler> { + public sealed record Command; + + public sealed record Response(BackupRunAcceptedDto Export); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) { + // Refused here rather than in the run, so the operator sees why in the dialog. Both conditions + // are about the archives the bundle is made of, not about the bundle itself. + var backup = options.CurrentValue.Backup; + if (string.IsNullOrEmpty(backup.EncryptionPassphrase)) + return AppError.Validation( + "A full backup bundle needs an encryption passphrase: it carries a dump of Watchtower's " + + "own database, which holds every database role's password hash, the data-protection " + + "key ring and every certificate's private key. Set one under Settings → Backups first."); + + var result = queue.EnqueueBundleExport(BackupTriggers.BundleExport); + // Recorded when the export is *asked for* as well as when it finishes: the request is the + // decision, and a bundle that fails to build is still an attempt to take one off the box. + await audit.RecordAsync( + BackupService.AuditCategory, "bundle.request", BackupBundleService.AuditTarget, + $"export requested (event {result.BackupEventId})", + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + return new Response(new BackupRunAcceptedDto(result.BackupEventId, result.Status)); + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/GetBackupConfig.cs b/src/Watchtower.Application/Modules/Backups/Handlers/GetBackupConfig.cs index 40b8e3b..fc3e7d1 100644 --- a/src/Watchtower.Application/Modules/Backups/Handlers/GetBackupConfig.cs +++ b/src/Watchtower.Application/Modules/Backups/Handlers/GetBackupConfig.cs @@ -36,6 +36,8 @@ public sealed record Response(BackupConfigDto Config); WatchtowerSettingPaths.BackupSftpPrivateKeyPassphrase, WatchtowerSettingPaths.BackupSftpBasePath, WatchtowerSettingPaths.BackupLocalBasePath, + WatchtowerSettingPaths.BackupIncludeSelf, + WatchtowerSettingPaths.BackupSelfPostgresContainer, ]; public ValueTask> HandleAsync(Query query, CancellationToken ct) { diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/GetBundleStatus.cs b/src/Watchtower.Application/Modules/Backups/Handlers/GetBundleStatus.cs new file mode 100644 index 0000000..d93a60f --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/GetBundleStatus.cs @@ -0,0 +1,25 @@ +using Elarion.Abstractions.Authorization; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// Whether a full backup bundle is staged for download, and what it holds (ADR-0027 §4) — what the +/// Settings card polls while an export runs and reads afterwards to offer the link. +/// +[Handler("backups.getBundleStatus")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class GetBundleStatus(BundleExportState state) + : IHandler> { + public sealed record Query; + + public sealed record Response(BackupBundleDto? Bundle); + + public ValueTask> HandleAsync(Query query, CancellationToken ct) => + ValueTask.FromResult>(new Response( + state.Current is { } staged + ? new BackupBundleDto( + staged.FileName, staged.SizeBytes, staged.CreatedAtUtc, staged.StackCount, + staged.MissingStackCount) + : null)); +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/GetInstanceRestoreStatus.cs b/src/Watchtower.Application/Modules/Backups/Handlers/GetInstanceRestoreStatus.cs new file mode 100644 index 0000000..3b58a23 --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/GetInstanceRestoreStatus.cs @@ -0,0 +1,50 @@ +using Elarion.Abstractions.Authorization; +using Elarion.Settings; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// What the restore wizard needs to decide what to show (ADR-0027): whether this instance looks brand +/// new, whether a bundle is waiting to be restored and what it holds, how the last restore ended, and +/// whether there is a recovery checklist still to work through. +/// +[Handler("backups.getRestoreStatus")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class GetInstanceRestoreStatus( + InstanceRestoreService restore, + InstanceRestoreStaging staging, + RestoreCompletionService completion, + ISettingsManager settings) + : IHandler> { + public sealed record Query; + + /// + /// No stacks, no deploys and one account — a Watchtower nobody has used yet. A hint for what to + /// offer, never a permission: the restore is gated on being an admin either way. + /// + /// The uploaded bundle, checked against this instance, or null. + /// How the last restore this instance attempted ended. + /// Why, when it failed. + /// Whether a post-restore checklist is still open. + public sealed record Response( + bool FreshInstance, + RestoreValidationDto? Staged, + string LastOutcome, + string? LastError, + bool RecoveryPending); + + public async ValueTask> HandleAsync(Query query, CancellationToken ct) { + RestoreValidationDto? staged = null; + if (staging.Current is { } current) + staged = RestoreValidationDto.From(await restore.ValidateAsync(current, ct)); + + var recovery = await StackRevivalState.LoadAsync(settings, ct); + return new Response( + FreshInstance: await restore.IsFreshAsync(ct), + Staged: staged, + LastOutcome: completion.LastOutcome.ToString().ToLowerInvariant(), + LastError: completion.LastError, + RecoveryPending: recovery is { Dismissed: false }); + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/ListBackupEvents.cs b/src/Watchtower.Application/Modules/Backups/Handlers/ListBackupEvents.cs index 9fe00a0..8139dc2 100644 --- a/src/Watchtower.Application/Modules/Backups/Handlers/ListBackupEvents.cs +++ b/src/Watchtower.Application/Modules/Backups/Handlers/ListBackupEvents.cs @@ -6,10 +6,11 @@ namespace Watchtower.Application.Modules.Backups.Handlers; /// /// Returns backup history, newest first — for one stack (), for every /// deployment of one product (, the product Backups tab's fleet history), -/// or instance-wide. Output is included, so the UI can show the run log inline. +/// for Watchtower's own database (), or instance-wide. Output is included, so +/// the UI can show the run log inline. /// /// -/// The two filters are independent and both optional, which is what keeps this additive: every existing +/// The filters are independent and all optional, which is what keeps this additive: every existing /// caller passes a stack id or nothing and gets exactly what it always did. Passing both narrows to the /// intersection rather than being refused — a stack of the product answers both questions, and a stack /// of another product legitimately answers neither. @@ -20,7 +21,12 @@ public sealed class ListBackupEvents(WatchtowerDbContext db) /// One stack, or null for every stack the other filters allow. /// Row cap, clamped to 1…500. /// Every deployment of this product, or null for no product filter. - public sealed record Query(int? StackId = null, int Limit = 50, int? ProductId = null); + /// + /// stack for the stack runs, instance for Watchtower's own (ADR-0027), or null for + /// both. Unfiltered stays the default so the existing history views are unchanged: an instance run + /// is part of "what has this Watchtower been backing up" and belongs in the instance-wide list. + /// + public sealed record Query(int? StackId = null, int Limit = 50, int? ProductId = null, string? Kind = null); public sealed record Response(IReadOnlyList Events); @@ -30,18 +36,27 @@ public async ValueTask> HandleAsync(Query query, CancellationTo if (query.ProductId is { } productId && !await db.Products.AnyAsync(p => p.Id == productId, ct)) return AppError.NotFound($"Product {productId} not found"); + var kind = query.Kind?.Trim().ToLowerInvariant(); + if (kind is not (null or "" or BackupEventKinds.Stack or BackupEventKinds.Instance)) + return AppError.Validation( + $"Kind must be '{BackupEventKinds.Stack}', '{BackupEventKinds.Instance}', or omitted."); + var instanceOnly = kind == BackupEventKinds.Instance; + var stackOnly = kind == BackupEventKinds.Stack; + var limit = Math.Clamp(query.Limit, 1, 500); // Id breaks ties: a stack-wide run writes several events within the same clock tick, and the // limit below only means something over a total order. var events = await db.BackupEvents.AsNoTracking() .Where(e => query.StackId == null || e.StackId == query.StackId) .Where(e => query.ProductId == null || e.Stack!.ProductId == query.ProductId) + .Where(e => (!instanceOnly || e.StackId == null) && (!stackOnly || e.StackId != null)) .OrderByDescending(e => e.StartedAt) .ThenByDescending(e => e.Id) .Take(limit) .Select(e => new BackupEventDto( - e.Id, e.StackId, e.Stack!.Name, e.TriggeredBy, e.Status, e.RemotePath, e.SizeBytes, - e.Output, e.StartedAt, e.FinishedAt)) + e.Id, e.StackId, e.Stack == null ? null : e.Stack.Name, e.TriggeredBy, e.Status, + e.RemotePath, e.SizeBytes, e.Output, e.StartedAt, e.FinishedAt, + e.StackId == null ? BackupEventKinds.Instance : BackupEventKinds.Stack)) .ToListAsync(ct); return new Response(events); } diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/ListInstanceBackups.cs b/src/Watchtower.Application/Modules/Backups/Handlers/ListInstanceBackups.cs new file mode 100644 index 0000000..0a8b6a9 --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/ListInstanceBackups.cs @@ -0,0 +1,46 @@ +using Elarion.Abstractions.Authorization; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// Lists the archives of Watchtower's own database present on the configured storage, newest first +/// (ADR-0027) — the instance counterpart of , and for the same reason the +/// storage rather than backups.events is the source of truth: retention deletes files behind old +/// events, and an archive written by a different instance of this Watchtower is still restorable here. +/// +[Handler("backups.listInstance")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class ListInstanceBackups( + IOptionsMonitor options, + BackupStorageFactory storageFactory) + : IHandler> { + public sealed record Query; + + /// The archives, newest first. + /// The provider-relative directory they were listed from. + public sealed record Response(IReadOnlyList Files, string Directory); + + public async ValueTask> HandleAsync(Query query, CancellationToken ct) { + var backup = options.CurrentValue.Backup; + var directory = BackupNaming.InstanceDirectory(backup.ResolveInstanceName()); + try { + using var storage = storageFactory.Create(backup); + var files = (await storage.ListFilesAsync(directory, ct)) + .Select(f => (File: f, TakenAt: BackupNaming.ParseTimestamp(f.Name))) + .Where(x => x.TakenAt is not null) + .OrderByDescending(x => x.TakenAt) + .Select(x => new BackupRemoteFileDto( + x.File.Name, + x.File.SizeBytes, + x.TakenAt!.Value, + Encrypted: x.File.Name.EndsWith(".enc", StringComparison.Ordinal))) + .ToList(); + return new Response(files, directory); + } catch (Exception ex) when (ex is not OperationCanceledException) { + return AppError.Validation($"Could not list the backup storage: {ex.Message}"); + } + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/RecoveryChecklistHandlers.cs b/src/Watchtower.Application/Modules/Backups/Handlers/RecoveryChecklistHandlers.cs new file mode 100644 index 0000000..f5f4076 --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/RecoveryChecklistHandlers.cs @@ -0,0 +1,113 @@ +using Elarion.Abstractions.Authorization; +using Elarion.Abstractions.Identity; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// The post-restore recovery checklist (ADR-0027 §6): every stack the restored database knows about, +/// waiting to be redeployed from git and restored from its newest archive. +/// +[Handler("backups.getRecoveryChecklist")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class GetRecoveryChecklist(StackRevivalCoordinator revival) + : IHandler> { + public sealed record Query; + + /// The checklist, or null when there is nothing to recover. + public sealed record Response(RecoveryChecklistDto? Checklist); + + public async ValueTask> HandleAsync(Query query, CancellationToken ct) => + new Response(await revival.LoadAsync(ct) is { } state ? RecoveryChecklistDto.From(state) : null); +} + +/// +/// Revives one stack: deploy it from git, then restore its newest archive into the volumes that deploy +/// created (ADR-0027 §6). Runs to completion before returning — a single stack is one deploy and one +/// restore, both of which the UI already knows how to wait on. +/// +[Handler("backups.reviveStack")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class ReviveStack(StackRevivalCoordinator revival, AuditLog audit, ICurrentUser currentUser) + : IHandler> { + public sealed record Command(int StackId); + + public sealed record Response(RecoveryStackDto Stack); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) { + if (await revival.ReviveAsync(command.StackId, ct) is not { } stack) + return AppError.NotFound($"Stack {command.StackId} is not on the recovery checklist."); + + await audit.RecordAsync( + BackupService.AuditCategory, "recovery.revive", stack.Name, + $"{stack.Status.ToString().ToLowerInvariant()} — {stack.Detail}", + success: stack.Status is not RevivalStatus.Failed, + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + return new Response(RecoveryStackDto.From(stack)); + } +} + +/// +/// Revives every stack still pending or failed, one after another (ADR-0027 §6). A failure does not +/// stop the rest: the stacks are independent, and stopping at the first would leave the operator to +/// work out which of the others had been tried. +/// +[Handler("backups.reviveAll")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class ReviveAllStacks( + StackRevivalCoordinator revival, AuditLog audit, ICurrentUser currentUser) + : IHandler> { + public sealed record Command; + + /// How many stacks ended up fully back. + /// The checklist as it now stands. + public sealed record Response(int Revived, RecoveryChecklistDto? Checklist); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) { + var revived = await revival.ReviveAllAsync(ct); + var checklist = await revival.LoadAsync(ct); + await audit.RecordAsync( + BackupService.AuditCategory, "recovery.revive", "all stacks", + $"{revived} of {checklist?.Stacks.Count ?? 0} stack(s) deployed and restored", + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + return new Response(revived, checklist is null ? null : RecoveryChecklistDto.From(checklist)); + } +} + +/// Marks one stack as handled outside Watchtower, so "revive all" leaves it alone. +[Handler("backups.skipRecoveryStack")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class SkipRecoveryStack(StackRevivalCoordinator revival) + : IHandler> { + public sealed record Command(int StackId); + + public sealed record Response(RecoveryStackDto Stack); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) => + await revival.SkipAsync(command.StackId, ct) is { } stack + ? new Response(RecoveryStackDto.From(stack)) + : AppError.NotFound($"Stack {command.StackId} is not on the recovery checklist."); +} + +/// +/// Puts the checklist away. It is a prompt, not a record — what was restored and revived is in the audit +/// trail, which is where that question belongs. +/// +[Handler("backups.dismissRecovery")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class DismissRecovery( + StackRevivalCoordinator revival, AuditLog audit, ICurrentUser currentUser) + : IHandler> { + public sealed record Command; + + public sealed record Response(bool Dismissed); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) { + await revival.DismissAsync(ct); + await audit.RecordAsync( + BackupService.AuditCategory, "recovery.dismiss", InstanceRestoreService.AuditTarget, + "recovery checklist dismissed", + actor: await audit.ActorAsync(currentUser, ct), ct: ct); + return new Response(true); + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/RunInstanceBackup.cs b/src/Watchtower.Application/Modules/Backups/Handlers/RunInstanceBackup.cs new file mode 100644 index 0000000..5b0870f --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/RunInstanceBackup.cs @@ -0,0 +1,39 @@ +using Elarion.Abstractions.Authorization; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// Enqueues a backup of Watchtower's own database on the single-flight backup queue (ADR-0027) and +/// returns the tracking event immediately. Works regardless of the schedule master switch and of +/// Backup:IncludeSelf — both govern the schedule, and an explicit run is an operator's decision. +/// +/// +/// Admin-only, unlike the stack runs. The archive it produces carries every database role's password +/// hash, the data-protection key ring and every certificate's private key, so the ability to place one on +/// the backup storage is the ability to walk away with the instance. +/// +[Handler("backups.runInstance")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class RunInstanceBackup(BackupQueueService queue, IOptionsMonitor options) + : IHandler> { + public sealed record Command; + + public sealed record Response(BackupRunAcceptedDto Backup); + + public ValueTask> HandleAsync(Command command, CancellationToken ct) { + // Refused here rather than in the run, so the operator gets the reason in the dialog instead of + // in a failed event's log. + if (string.IsNullOrEmpty(options.CurrentValue.Backup.EncryptionPassphrase)) + return ValueTask.FromResult>(AppError.Validation( + "Backing up Watchtower itself needs an encryption passphrase: the dump carries every " + + "database role's password hash, the data-protection key ring and every certificate's " + + "private key. Set one under Settings → Backups first.")); + + var result = queue.EnqueueInstance(BackupTriggers.Manual); + return ValueTask.FromResult>( + new Response(new BackupRunAcceptedDto(result.BackupEventId, result.Status))); + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/StartInstanceRestore.cs b/src/Watchtower.Application/Modules/Backups/Handlers/StartInstanceRestore.cs new file mode 100644 index 0000000..fcd5401 --- /dev/null +++ b/src/Watchtower.Application/Modules/Backups/Handlers/StartInstanceRestore.cs @@ -0,0 +1,57 @@ +using Elarion.Abstractions.Authorization; +using Elarion.Abstractions.Identity; +using Microsoft.EntityFrameworkCore; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services; + +namespace Watchtower.Application.Modules.Backups.Handlers; + +/// +/// Replaces this Watchtower's database with the one in the uploaded bundle (ADR-0027 §5). The +/// destructive half of the restore: everything this instance currently knows — its stacks, accounts, +/// routes, settings and keys — is replaced by the bundle's, and the containers it deployed keep running +/// unmanaged until the recovery checklist redeploys them. +/// +/// +/// Returns as soon as the coordinator container has been started; Watchtower stops answering a few +/// seconds later and comes back on the restored database, where the caller's session no longer exists. +/// The UI is expected to wait for the restart and send the operator to the login page. +/// +[Handler("backups.startInstanceRestore")] +[RequireRole(WatchtowerClaims.AdminRole)] +public sealed class StartInstanceRestore( + WatchtowerDbContext db, + InstanceRestoreService restore, + InstanceRestoreStaging staging, + AuditLog audit, + ICurrentUser currentUser) + : IHandler> { + public sealed record Command; + + /// The instance the bundle came from, for the "restarting" banner. + public sealed record Response(string SourceInstance); + + public async ValueTask> HandleAsync(Command command, CancellationToken ct) { + if (staging.Current is not { } staged) + return AppError.Validation("Upload a backup bundle first."); + + // Nothing that writes may be in flight: a deploy or backup finishing against a database that is + // being dropped would leave its own event row in a state nothing can explain afterwards. + if (await db.DeployEvents.AnyAsync(e => e.Status == "running" || e.Status == "queued", ct)) + return AppError.Conflict("A deploy is in progress — restore once it has finished."); + if (await db.BackupEvents.AnyAsync( + e => e.Status == BackupStatuses.Running || e.Status == BackupStatuses.Queued, ct)) + return AppError.Conflict("A backup or restore is in progress — restore once it has finished."); + + var actor = await audit.ActorAsync(currentUser, ct); + try { + await restore.StartAsync(actor, ct); + } catch (InvalidOperationException ex) { + // Everything the restore refuses is a configuration or bundle problem the operator can act + // on, and none of it has touched the database yet. + return AppError.Validation(ex.Message); + } + return new Response(staged.Manifest.InstanceName); + } +} diff --git a/src/Watchtower.Application/Modules/Backups/Handlers/UpdateBackupConfig.cs b/src/Watchtower.Application/Modules/Backups/Handlers/UpdateBackupConfig.cs index c3cd541..8532f24 100644 --- a/src/Watchtower.Application/Modules/Backups/Handlers/UpdateBackupConfig.cs +++ b/src/Watchtower.Application/Modules/Backups/Handlers/UpdateBackupConfig.cs @@ -39,7 +39,9 @@ public sealed record Command( string? SftpPrivateKey = null, string? SftpPrivateKeyPassphrase = null, string? SftpBasePath = null, - string? LocalBasePath = null); + string? LocalBasePath = null, + bool? IncludeSelf = null, + string? SelfPostgresContainer = null); public sealed record Response(BackupConfigDto Config); @@ -90,6 +92,10 @@ void Check(string path, bool changed) { Check(WatchtowerSettingPaths.BackupSftpPrivateKeyPassphrase, command.SftpPrivateKeyPassphrase is not null); Check(WatchtowerSettingPaths.BackupSftpBasePath, Changed(command.SftpBasePath, sftp.BasePath)); Check(WatchtowerSettingPaths.BackupLocalBasePath, Changed(command.LocalBasePath, backup.Local.BasePath)); + Check(WatchtowerSettingPaths.BackupIncludeSelf, + command.IncludeSelf is { } includeSelf && includeSelf != backup.IncludeSelf); + Check(WatchtowerSettingPaths.BackupSelfPostgresContainer, + Changed(command.SelfPostgresContainer, backup.SelfPostgresContainer)); if (violations.Count > 0) return EnvironmentSettingPins.PinnedError(violations); @@ -126,6 +132,12 @@ await SetUnlessPinnedAsync(WatchtowerSettingPaths.BackupRetentionMaxCount, await SetUnlessPinnedAsync(WatchtowerSettingPaths.BackupSftpBasePath, command.SftpBasePath.Trim(), ct); if (command.LocalBasePath is not null) await SetUnlessPinnedAsync(WatchtowerSettingPaths.BackupLocalBasePath, command.LocalBasePath.Trim(), ct); + if (command.IncludeSelf is { } includeSelfValue) + await SetUnlessPinnedAsync( + WatchtowerSettingPaths.BackupIncludeSelf, includeSelfValue ? "true" : "false", ct); + if (command.SelfPostgresContainer is not null) + await SetUnlessPinnedAsync( + WatchtowerSettingPaths.BackupSelfPostgresContainer, command.SelfPostgresContainer.Trim(), ct); // Echo the written values (the config provider reloads asynchronously — same reasoning as // proxy.updateConfig): immediately consistent for the caller. @@ -151,6 +163,8 @@ await SetUnlessPinnedAsync(WatchtowerSettingPaths.BackupRetentionMaxCount, Local = backup.Local with { BasePath = Coalesce(command.LocalBasePath, backup.Local.BasePath) ?? "", }, + IncludeSelf = command.IncludeSelf ?? backup.IncludeSelf, + SelfPostgresContainer = Coalesce(command.SelfPostgresContainer, backup.SelfPostgresContainer), }; // Recorded post-write so the trail answers "what was the configuration at that time" — @@ -169,6 +183,7 @@ await SetUnlessPinnedAsync(WatchtowerSettingPaths.BackupRetentionMaxCount, + $" · provider {provider}" + $" · {BackupService.RetentionSummary(echoed)}" + (string.IsNullOrEmpty(echoed.EncryptionPassphrase) ? "" : " · encrypted") + + $" · Watchtower's own database {(echoed.IncludeSelf ? "included" : "excluded")}" + (updatedSecrets.Count > 0 ? $" · secrets updated: {string.Join(", ", updatedSecrets)}" : ""); await audit.RecordAsync(BackupService.AuditCategory, "config.update", "backup settings", detail, actor: await audit.ActorAsync(currentUser, ct), ct: ct); diff --git a/src/Watchtower.Application/Modules/Stacks/Handlers/CreateStack.cs b/src/Watchtower.Application/Modules/Stacks/Handlers/CreateStack.cs index 34f1dc2..d9591a6 100644 --- a/src/Watchtower.Application/Modules/Stacks/Handlers/CreateStack.cs +++ b/src/Watchtower.Application/Modules/Stacks/Handlers/CreateStack.cs @@ -58,7 +58,8 @@ public async ValueTask> HandleAsync(Command command, Cancellati // visibility. Enforced here because the default name is the lowercased stack name. Watchtower's // own project is reserved for the same reason. var projectName = StackMapping.ResolveProjectName(command.Name, command.ComposeProjectName); - if (await StackProjectNames.ValidateAsync(db, selfProjects, projectName, excludeStackId: null, ct) + if (await StackProjectNames.ValidateAsync( + db, selfProjects, projectName, excludeStackId: null, ct, command.Name) is { } projectNameError) return AppError.Validation(projectNameError); diff --git a/src/Watchtower.Application/Modules/Stacks/Handlers/UpdateStack.cs b/src/Watchtower.Application/Modules/Stacks/Handlers/UpdateStack.cs index 67d33ce..6e2b741 100644 --- a/src/Watchtower.Application/Modules/Stacks/Handlers/UpdateStack.cs +++ b/src/Watchtower.Application/Modules/Stacks/Handlers/UpdateStack.cs @@ -70,7 +70,8 @@ public async ValueTask> HandleAsync(Command command, Cancellati // own — which would make them share containers (and App API visibility). Checked before // anything is mutated. var projectName = StackMapping.ResolveProjectName(command.Name, command.ComposeProjectName); - if (await StackProjectNames.ValidateAsync(db, selfProjects, projectName, excludeStackId: stack.Id, ct) + if (await StackProjectNames.ValidateAsync( + db, selfProjects, projectName, excludeStackId: stack.Id, ct, command.Name) is { } projectNameError) return AppError.Validation(projectNameError); diff --git a/src/Watchtower.Application/Persistence/Configurations/WatchtowerEntityConfigurations.cs b/src/Watchtower.Application/Persistence/Configurations/WatchtowerEntityConfigurations.cs index cf3d146..ef84649 100644 --- a/src/Watchtower.Application/Persistence/Configurations/WatchtowerEntityConfigurations.cs +++ b/src/Watchtower.Application/Persistence/Configurations/WatchtowerEntityConfigurations.cs @@ -394,9 +394,12 @@ public void Configure(EntityTypeBuilder b) { // The history view reads newest-first per stack; the startup sweep scans by status. b.HasIndex(x => new { x.StackId, x.StartedAt }); b.HasIndex(x => x.Status); + // Optional since ADR-0027: an instance self-backup has no stack. Still cascading, so a deleted + // stack takes its own history with it — only the stackless rows outlive every stack. b.HasOne(x => x.Stack) .WithMany() .HasForeignKey(x => x.StackId) + .IsRequired(false) .OnDelete(DeleteBehavior.Cascade); } } diff --git a/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.Designer.cs b/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.Designer.cs new file mode 100644 index 0000000..09ce3fd --- /dev/null +++ b/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.Designer.cs @@ -0,0 +1,2411 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; +using Watchtower.Application.Persistence; + +#nullable disable + +namespace Watchtower.Application.Persistence.Migrations +{ + [DbContext(typeof(WatchtowerDbContext))] + [Migration("20260826192544_AddInstanceBackupEvents")] + partial class AddInstanceBackupEvents + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("Elarion.Coordination.PostgreSql.RoleLeaseEntity", b => + { + b.Property("Role") + .HasMaxLength(128) + .HasColumnType("character varying(128)") + .HasColumnName("role"); + + b.Property("Address") + .HasMaxLength(512) + .HasColumnType("character varying(512)") + .HasColumnName("address"); + + b.Property("ExpiresOnUtc") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_on_utc"); + + b.Property("Owner") + .IsRequired() + .HasMaxLength(256) + .HasColumnType("character varying(256)") + .HasColumnName("owner"); + + b.HasKey("Role") + .HasName("pk_elarion_role_leases"); + + b.ToTable("elarion_role_leases", (string)null); + }); + + modelBuilder.Entity("Elarion.Scheduling.EntityFrameworkCore.SchedulerClaimEntity", b => + { + b.Property("JobName") + .HasMaxLength(256) + .HasColumnType("character varying(256)") + .HasColumnName("job_name"); + + b.Property("OccurrenceUtc") + .HasColumnType("timestamp with time zone") + .HasColumnName("occurrence_utc"); + + b.Property("ClaimedAtUtc") + .HasColumnType("timestamp with time zone") + .HasColumnName("claimed_at_utc"); + + b.HasKey("JobName", "OccurrenceUtc") + .HasName("pk_elarion_scheduler_claims"); + + b.HasIndex("OccurrenceUtc") + .HasDatabaseName("ix_elarion_scheduler_claims_purge"); + + b.ToTable("elarion_scheduler_claims", (string)null); + }); + + modelBuilder.Entity("Elarion.Settings.EntityFrameworkCore.Setting", b => + { + b.Property("Kind") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("kind"); + + b.Property("Owner") + .HasMaxLength(256) + .HasColumnType("character varying(256)") + .HasColumnName("owner"); + + b.Property("Key") + .HasMaxLength(512) + .HasColumnType("character varying(512)") + .HasColumnName("key"); + + b.Property("UpdatedOnUtc") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_on_utc"); + + b.Property("Value") + .HasColumnType("text") + .HasColumnName("value"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("integer") + .HasColumnName("version"); + + b.HasKey("Kind", "Owner", "Key") + .HasName("pk_elarion_settings"); + + b.ToTable("elarion_settings", (string)null); + }); + + modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("FriendlyName") + .HasColumnType("text") + .HasColumnName("friendly_name"); + + b.Property("Xml") + .HasColumnType("text") + .HasColumnName("xml"); + + b.HasKey("Id") + .HasName("pk_data_protection_keys"); + + b.ToTable("data_protection_keys", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.AcmeAccount", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AccountUrl") + .HasColumnType("text") + .HasColumnName("account_url"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("DirectoryUrl") + .IsRequired() + .HasColumnType("text") + .HasColumnName("directory_url"); + + b.Property("PrivateKey") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("private_key"); + + b.Property("Protection") + .IsRequired() + .HasColumnType("text") + .HasColumnName("protection"); + + b.HasKey("Id") + .HasName("pk_acme_accounts"); + + b.HasIndex("DirectoryUrl") + .IsUnique() + .HasDatabaseName("ix_acme_accounts_directory_url"); + + b.ToTable("acme_accounts", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.AcmeHttpChallenge", b => + { + b.Property("Token") + .HasColumnType("text") + .HasColumnName("token"); + + b.Property("ExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_at"); + + b.Property("Host") + .IsRequired() + .HasColumnType("text") + .HasColumnName("host"); + + b.Property("KeyAuthorization") + .IsRequired() + .HasColumnType("text") + .HasColumnName("key_authorization"); + + b.HasKey("Token") + .HasName("pk_acme_http_challenges"); + + b.HasIndex("ExpiresAt") + .HasDatabaseName("ix_acme_http_challenges_expires_at"); + + b.ToTable("acme_http_challenges", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.AuditEvent", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Action") + .IsRequired() + .HasColumnType("text") + .HasColumnName("action"); + + b.Property("Actor") + .HasColumnType("text") + .HasColumnName("actor"); + + b.Property("Category") + .IsRequired() + .HasColumnType("text") + .HasColumnName("category"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Detail") + .HasColumnType("text") + .HasColumnName("detail"); + + b.Property("Error") + .HasColumnType("text") + .HasColumnName("error"); + + b.Property("Success") + .HasColumnType("boolean") + .HasColumnName("success"); + + b.Property("Target") + .IsRequired() + .HasColumnType("text") + .HasColumnName("target"); + + b.HasKey("Id") + .HasName("pk_audit_events"); + + b.HasIndex("Category") + .HasDatabaseName("ix_audit_events_category"); + + b.HasIndex("CreatedAt") + .HasDatabaseName("ix_audit_events_created_at"); + + b.ToTable("audit_events", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.AuthSession", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("ExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_at"); + + b.Property("Kind") + .IsRequired() + .HasColumnType("text") + .HasColumnName("kind"); + + b.Property("RouteId") + .HasColumnType("integer") + .HasColumnName("route_id"); + + b.Property("TokenHash") + .IsRequired() + .HasColumnType("text") + .HasColumnName("token_hash"); + + b.Property("UserId") + .HasColumnType("integer") + .HasColumnName("user_id"); + + b.HasKey("Id") + .HasName("pk_auth_sessions"); + + b.HasIndex("ExpiresAt") + .HasDatabaseName("ix_auth_sessions_expires_at"); + + b.HasIndex("RouteId") + .HasDatabaseName("ix_auth_sessions_route_id"); + + b.HasIndex("TokenHash") + .IsUnique() + .HasDatabaseName("ix_auth_sessions_token_hash"); + + b.HasIndex("UserId") + .HasDatabaseName("ix_auth_sessions_user_id"); + + b.ToTable("auth_sessions", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.BackupEvent", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("FinishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("finished_at"); + + b.Property("Output") + .HasColumnType("text") + .HasColumnName("output"); + + b.Property("RemotePath") + .HasColumnType("text") + .HasColumnName("remote_path"); + + b.Property("SizeBytes") + .HasColumnType("bigint") + .HasColumnName("size_bytes"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("Status") + .IsRequired() + .HasColumnType("text") + .HasColumnName("status"); + + b.Property("TriggeredBy") + .IsRequired() + .HasColumnType("text") + .HasColumnName("triggered_by"); + + b.HasKey("Id") + .HasName("pk_backup_events"); + + b.HasIndex("Status") + .HasDatabaseName("ix_backup_events_status"); + + b.HasIndex("StackId", "StartedAt") + .HasDatabaseName("ix_backup_events_stack_id_started_at"); + + b.ToTable("backup_events", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.BackupPausedContainer", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("ContainerId") + .IsRequired() + .HasColumnType("text") + .HasColumnName("container_id"); + + b.Property("ContainerName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("container_name"); + + b.Property("PausedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("paused_at"); + + b.Property("StackName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("stack_name"); + + b.HasKey("Id") + .HasName("pk_backup_paused_containers"); + + b.HasIndex("ContainerId") + .HasDatabaseName("ix_backup_paused_containers_container_id"); + + b.ToTable("backup_paused_containers", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.CiRepo", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AllowDockerSocket") + .HasColumnType("boolean") + .HasColumnName("allow_docker_socket"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CredentialId") + .HasColumnType("integer") + .HasColumnName("credential_id"); + + b.Property("Enabled") + .HasColumnType("boolean") + .HasColumnName("enabled"); + + b.Property("ExtraLabels") + .HasColumnType("text") + .HasColumnName("extra_labels"); + + b.Property("LastRegistrySyncError") + .HasColumnType("text") + .HasColumnName("last_registry_sync_error"); + + b.Property("LastWarmError") + .HasColumnType("text") + .HasColumnName("last_warm_error"); + + b.Property("LastWarmedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_warmed_at"); + + b.Property("MaxConcurrentRunners") + .HasColumnType("integer") + .HasColumnName("max_concurrent_runners"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("Owner") + .IsRequired() + .HasColumnType("text") + .HasColumnName("owner"); + + b.Property("RegistrySyncedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("registry_synced_at"); + + b.Property("RegistrySyncedHash") + .HasColumnType("text") + .HasColumnName("registry_synced_hash"); + + b.Property("RunnerImage") + .HasColumnType("text") + .HasColumnName("runner_image"); + + b.Property("SyncRegistryUrl") + .HasColumnType("text") + .HasColumnName("sync_registry_url"); + + b.Property("ToolchainDetectedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("toolchain_detected_at"); + + b.Property("ToolchainProfileJson") + .HasColumnType("text") + .HasColumnName("toolchain_profile_json"); + + b.Property("WarmedProfileHash") + .HasColumnType("text") + .HasColumnName("warmed_profile_hash"); + + b.HasKey("Id") + .HasName("pk_ci_repos"); + + b.HasIndex("CredentialId") + .HasDatabaseName("ix_ci_repos_credential_id"); + + b.HasIndex("Owner", "Name") + .IsUnique() + .HasDatabaseName("ix_ci_repos_owner_name"); + + b.ToTable("ci_repos", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Credential", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("Token") + .IsRequired() + .HasColumnType("text") + .HasColumnName("token"); + + b.Property("Username") + .IsRequired() + .HasColumnType("text") + .HasColumnName("username"); + + b.HasKey("Id") + .HasName("pk_credentials"); + + b.HasIndex("Name") + .HasDatabaseName("ix_credentials_name"); + + b.ToTable("credentials", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.DeployEvent", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("FinishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("finished_at"); + + b.Property("Output") + .HasColumnType("text") + .HasColumnName("output"); + + b.Property("ReleaseId") + .HasColumnType("integer") + .HasColumnName("release_id"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("Status") + .IsRequired() + .HasColumnType("text") + .HasColumnName("status"); + + b.Property("TriggeredBy") + .IsRequired() + .HasColumnType("text") + .HasColumnName("triggered_by"); + + b.HasKey("Id") + .HasName("pk_deploy_events"); + + b.HasIndex("ReleaseId") + .HasDatabaseName("ix_deploy_events_release_id"); + + b.HasIndex("Status") + .HasDatabaseName("ix_deploy_events_status"); + + b.HasIndex("StackId", "StartedAt") + .HasDatabaseName("ix_deploy_events_stack_id_started_at"); + + b.ToTable("deploy_events", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Group", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("NormalizedName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("normalized_name"); + + b.Property("RealmId") + .HasColumnType("integer") + .HasColumnName("realm_id"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_groups"); + + b.HasIndex("RealmId", "NormalizedName") + .IsUnique() + .HasDatabaseName("ix_groups_realm_id_normalized_name"); + + b.ToTable("groups", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.GroupMember", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("GroupId") + .HasColumnType("integer") + .HasColumnName("group_id"); + + b.Property("UserId") + .HasColumnType("integer") + .HasColumnName("user_id"); + + b.HasKey("Id") + .HasName("pk_group_members"); + + b.HasIndex("UserId") + .HasDatabaseName("ix_group_members_user_id"); + + b.HasIndex("GroupId", "UserId") + .IsUnique() + .HasDatabaseName("ix_group_members_group_id_user_id"); + + b.ToTable("group_members", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.LoginCode", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CodeHash") + .IsRequired() + .HasColumnType("text") + .HasColumnName("code_hash"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("ExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_at"); + + b.Property("RedirectUri") + .IsRequired() + .HasColumnType("text") + .HasColumnName("redirect_uri"); + + b.Property("RouteId") + .HasColumnType("integer") + .HasColumnName("route_id"); + + b.Property("UserId") + .HasColumnType("integer") + .HasColumnName("user_id"); + + b.HasKey("Id") + .HasName("pk_login_codes"); + + b.HasIndex("CodeHash") + .IsUnique() + .HasDatabaseName("ix_login_codes_code_hash"); + + b.HasIndex("ExpiresAt") + .HasDatabaseName("ix_login_codes_expires_at"); + + b.HasIndex("RouteId") + .HasDatabaseName("ix_login_codes_route_id"); + + b.HasIndex("UserId") + .HasDatabaseName("ix_login_codes_user_id"); + + b.ToTable("login_codes", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.MetricContainerSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("ContainerName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("container_name"); + + b.Property("CpuPercent") + .HasColumnType("double precision") + .HasColumnName("cpu_percent"); + + b.Property("MemLimitBytes") + .HasColumnType("bigint") + .HasColumnName("mem_limit_bytes"); + + b.Property("MemUsedBytes") + .HasColumnType("bigint") + .HasColumnName("mem_used_bytes"); + + b.Property("StackName") + .HasColumnType("text") + .HasColumnName("stack_name"); + + b.Property("TUnixSeconds") + .HasColumnType("bigint") + .HasColumnName("t_unix_seconds"); + + b.Property("TierSeconds") + .HasColumnType("integer") + .HasColumnName("tier_seconds"); + + b.HasKey("Id") + .HasName("pk_metric_container_samples"); + + b.HasIndex("TierSeconds", "TUnixSeconds", "ContainerName") + .IsUnique() + .HasDatabaseName("ix_metric_container_samples_tier_seconds_t_unix_seconds_contai"); + + b.ToTable("metric_container_samples", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.MetricHostSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CpuPercent") + .HasColumnType("double precision") + .HasColumnName("cpu_percent"); + + b.Property("LoadAvg1") + .HasColumnType("double precision") + .HasColumnName("load_avg1"); + + b.Property("LoadAvg5") + .HasColumnType("double precision") + .HasColumnName("load_avg5"); + + b.Property("MemPercent") + .HasColumnType("double precision") + .HasColumnName("mem_percent"); + + b.Property("MemUsedBytes") + .HasColumnType("bigint") + .HasColumnName("mem_used_bytes"); + + b.Property("TUnixSeconds") + .HasColumnType("bigint") + .HasColumnName("t_unix_seconds"); + + b.Property("TierSeconds") + .HasColumnType("integer") + .HasColumnName("tier_seconds"); + + b.HasKey("Id") + .HasName("pk_metric_host_samples"); + + b.HasIndex("TierSeconds", "TUnixSeconds") + .IsUnique() + .HasDatabaseName("ix_metric_host_samples_tier_seconds_t_unix_seconds"); + + b.ToTable("metric_host_samples", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Product", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("ActionsSyncedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("actions_synced_at"); + + b.Property("ActionsSyncedHash") + .HasColumnType("text") + .HasColumnName("actions_synced_hash"); + + b.Property("CiRepoId") + .HasColumnType("integer") + .HasColumnName("ci_repo_id"); + + b.Property("ComposeFilePath") + .IsRequired() + .HasColumnType("text") + .HasColumnName("compose_file_path"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CredentialId") + .HasColumnType("integer") + .HasColumnName("credential_id"); + + b.Property("DefaultBranch") + .IsRequired() + .HasColumnType("text") + .HasColumnName("default_branch"); + + b.Property("Description") + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("LastActionsSyncError") + .HasColumnType("text") + .HasColumnName("last_actions_sync_error"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("ReleaseMode") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("text") + .HasDefaultValue("Git") + .HasColumnName("release_mode"); + + b.Property("ReleaseWebhookEnabled") + .HasColumnType("boolean") + .HasColumnName("release_webhook_enabled"); + + b.Property("ReleaseWebhookToken") + .HasColumnType("text") + .HasColumnName("release_webhook_token"); + + b.Property("RepositoryUrl") + .IsRequired() + .HasColumnType("text") + .HasColumnName("repository_url"); + + b.Property("RetainReleases") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasDefaultValue(50) + .HasColumnName("retain_releases"); + + b.Property("SyncReleaseSecrets") + .HasColumnType("boolean") + .HasColumnName("sync_release_secrets"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_products"); + + b.HasIndex("CiRepoId") + .HasDatabaseName("ix_products_ci_repo_id"); + + b.HasIndex("CredentialId") + .HasDatabaseName("ix_products_credential_id"); + + b.HasIndex("Name") + .IsUnique() + .HasDatabaseName("ix_products_name"); + + b.HasIndex("ReleaseWebhookToken") + .IsUnique() + .HasDatabaseName("ix_products_release_webhook_token"); + + b.HasIndex(new[] { "CiRepoId" }, "ix_products_ci_repo_id_sync_release_secrets") + .IsUnique() + .HasDatabaseName("ix_products_ci_repo_id_sync_release_secrets") + .HasFilter("\"sync_release_secrets\""); + + b.ToTable("products", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.ProxyCertificate", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CertificatePem") + .IsRequired() + .HasColumnType("text") + .HasColumnName("certificate_pem"); + + b.Property("Host") + .IsRequired() + .HasColumnType("text") + .HasColumnName("host"); + + b.Property("InstalledAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("installed_at"); + + b.Property("Issuer") + .IsRequired() + .HasColumnType("text") + .HasColumnName("issuer"); + + b.Property("NotAfter") + .HasColumnType("timestamp with time zone") + .HasColumnName("not_after"); + + b.Property("NotBefore") + .HasColumnType("timestamp with time zone") + .HasColumnName("not_before"); + + b.Property("PrivateKey") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("private_key"); + + b.Property("Protection") + .IsRequired() + .HasColumnType("text") + .HasColumnName("protection"); + + b.Property("Source") + .IsRequired() + .HasColumnType("text") + .HasColumnName("source"); + + b.Property("Thumbprint") + .IsRequired() + .HasColumnType("text") + .HasColumnName("thumbprint"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_proxy_certificates"); + + b.HasIndex("Host") + .IsUnique() + .HasDatabaseName("ix_proxy_certificates_host"); + + b.ToTable("proxy_certificates", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Realm", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("IsSystem") + .HasColumnType("boolean") + .HasColumnName("is_system"); + + b.Property("LoginRouteId") + .HasColumnType("integer") + .HasColumnName("login_route_id"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("Slug") + .IsRequired() + .HasColumnType("text") + .HasColumnName("slug"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_realms"); + + b.HasIndex("LoginRouteId") + .IsUnique() + .HasDatabaseName("ix_realms_login_route_id") + .HasFilter("\"login_route_id\" IS NOT NULL"); + + b.HasIndex("Slug") + .IsUnique() + .HasDatabaseName("ix_realms_slug"); + + b.ToTable("realms", (string)null); + + b.HasData( + new + { + Id = 1, + CreatedAt = new DateTimeOffset(new DateTime(2026, 8, 10, 0, 0, 0, 0, DateTimeKind.Unspecified), new TimeSpan(0, 0, 0, 0, 0)), + IsSystem = true, + Name = "Operator", + Slug = "operator", + Xmin = 0u + }); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Registry", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CredentialId") + .HasColumnType("integer") + .HasColumnName("credential_id"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("Url") + .IsRequired() + .HasColumnType("text") + .HasColumnName("url"); + + b.HasKey("Id") + .HasName("pk_registries"); + + b.HasIndex("CredentialId") + .HasDatabaseName("ix_registries_credential_id"); + + b.HasIndex("Name") + .HasDatabaseName("ix_registries_name"); + + b.ToTable("registries", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Release", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Branch") + .IsRequired() + .HasColumnType("text") + .HasColumnName("branch"); + + b.Property("CommitSha") + .HasColumnType("text") + .HasColumnName("commit_sha"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedVia") + .IsRequired() + .HasColumnType("text") + .HasColumnName("created_via"); + + b.Property("Fingerprint") + .IsRequired() + .HasColumnType("text") + .HasColumnName("fingerprint"); + + b.Property("Notes") + .HasColumnType("text") + .HasColumnName("notes"); + + b.Property("ProductId") + .HasColumnType("integer") + .HasColumnName("product_id"); + + b.Property("PublishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("published_at"); + + b.Property("SourceRunUrl") + .HasColumnType("text") + .HasColumnName("source_run_url"); + + b.Property("Version") + .IsRequired() + .HasColumnType("text") + .HasColumnName("version"); + + b.HasKey("Id") + .HasName("pk_releases"); + + b.HasIndex("ProductId", "Fingerprint") + .IsUnique() + .HasDatabaseName("ix_releases_product_id_fingerprint"); + + b.HasIndex("ProductId", "Id") + .HasDatabaseName("ix_releases_product_id_id"); + + b.HasIndex("ProductId", "Version") + .IsUnique() + .HasDatabaseName("ix_releases_product_id_version"); + + b.ToTable("releases", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.ReleaseImage", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Digest") + .IsRequired() + .HasColumnType("text") + .HasColumnName("digest"); + + b.Property("ReleaseId") + .HasColumnType("integer") + .HasColumnName("release_id"); + + b.Property("Repository") + .IsRequired() + .HasColumnType("text") + .HasColumnName("repository"); + + b.Property("Tag") + .HasColumnType("text") + .HasColumnName("tag"); + + b.HasKey("Id") + .HasName("pk_release_images"); + + b.HasIndex("ReleaseId", "Repository") + .IsUnique() + .HasDatabaseName("ix_release_images_release_id_repository"); + + b.ToTable("release_images", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Route", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AccessMode") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("text") + .HasDefaultValue("Public") + .HasColumnName("access_mode"); + + b.Property("BypassPaths") + .HasColumnType("text") + .HasColumnName("bypass_paths"); + + b.Property("CertNotAfter") + .HasColumnType("timestamp with time zone") + .HasColumnName("cert_not_after"); + + b.Property("ContainerPort") + .HasColumnType("integer") + .HasColumnName("container_port"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Domain") + .IsRequired() + .HasColumnType("text") + .HasColumnName("domain"); + + b.Property("IdentityHeaderMode") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("text") + .HasDefaultValue("None") + .HasColumnName("identity_header_mode"); + + b.Property("IsPrimary") + .HasColumnType("boolean") + .HasColumnName("is_primary"); + + b.Property("Kind") + .IsRequired() + .HasColumnType("text") + .HasColumnName("kind"); + + b.Property("RealmId") + .HasColumnType("integer") + .HasColumnName("realm_id"); + + b.Property("ServiceName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("service_name"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("Status") + .IsRequired() + .HasColumnType("text") + .HasColumnName("status"); + + b.Property("StatusDetail") + .HasColumnType("text") + .HasColumnName("status_detail"); + + b.Property("Target") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("text") + .HasDefaultValue("Service") + .HasColumnName("target"); + + b.Property("TlsEnabled") + .HasColumnType("boolean") + .HasColumnName("tls_enabled"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_routes"); + + b.HasIndex("Domain") + .IsUnique() + .HasDatabaseName("ix_routes_domain"); + + b.HasIndex("RealmId") + .HasDatabaseName("ix_routes_realm_id"); + + b.HasIndex("StackId") + .HasDatabaseName("ix_routes_stack_id"); + + b.ToTable("routes", null, t => + { + t.HasCheckConstraint("ck_routes_target", "(\"target\" = 'Watchtower' AND \"stack_id\" IS NULL AND \"realm_id\" IS NOT NULL AND \"access_mode\" = 'Public')\r\nOR (\"target\" = 'Service' AND \"stack_id\" IS NOT NULL AND \"realm_id\" IS NULL)"); + }); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.RouteAccessGrant", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("GroupId") + .HasColumnType("integer") + .HasColumnName("group_id"); + + b.Property("RouteId") + .HasColumnType("integer") + .HasColumnName("route_id"); + + b.Property("UserId") + .HasColumnType("integer") + .HasColumnName("user_id"); + + b.HasKey("Id") + .HasName("pk_route_access_grants"); + + b.HasIndex("GroupId") + .HasDatabaseName("ix_route_access_grants_group_id"); + + b.HasIndex("UserId") + .HasDatabaseName("ix_route_access_grants_user_id"); + + b.HasIndex("RouteId", "GroupId") + .IsUnique() + .HasDatabaseName("ix_route_access_grants_route_id_group_id") + .HasFilter("\"group_id\" IS NOT NULL"); + + b.HasIndex("RouteId", "UserId") + .IsUnique() + .HasDatabaseName("ix_route_access_grants_route_id_user_id") + .HasFilter("\"user_id\" IS NOT NULL"); + + b.ToTable("route_access_grants", null, t => + { + t.HasCheckConstraint("ck_route_access_grants_subject", "(\"user_id\" IS NOT NULL) <> (\"group_id\" IS NOT NULL)"); + }); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.SigningKey", b => + { + b.Property("Purpose") + .HasColumnType("text") + .HasColumnName("purpose"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("KeyId") + .IsRequired() + .HasColumnType("text") + .HasColumnName("key_id"); + + b.Property("PrivateKey") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("private_key"); + + b.Property("Protection") + .IsRequired() + .HasColumnType("text") + .HasColumnName("protection"); + + b.HasKey("Purpose") + .HasName("pk_signing_keys"); + + b.ToTable("signing_keys", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Stack", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AppApiEnabled") + .HasColumnType("boolean") + .HasColumnName("app_api_enabled"); + + b.Property("AppApiToken") + .HasColumnType("text") + .HasColumnName("app_api_token"); + + b.Property("AutoDeployMode") + .IsRequired() + .HasColumnType("text") + .HasColumnName("auto_deploy_mode"); + + b.Property("AutoDeployTime") + .HasColumnType("text") + .HasColumnName("auto_deploy_time"); + + b.Property("BackupCron") + .HasColumnType("text") + .HasColumnName("backup_cron"); + + b.Property("BackupDirectory") + .HasColumnType("text") + .HasColumnName("backup_directory"); + + b.Property("BackupEnabled") + .HasColumnType("boolean") + .HasColumnName("backup_enabled"); + + b.Property("BackupQuiesceMode") + .HasColumnType("text") + .HasColumnName("backup_quiesce_mode"); + + b.Property("BackupStopContainers") + .HasColumnType("boolean") + .HasColumnName("backup_stop_containers"); + + b.Property("BranchOverride") + .HasColumnType("text") + .HasColumnName("branch_override"); + + b.Property("ComposeProjectName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("compose_project_name"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("DesiredState") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("text") + .HasDefaultValue("Running") + .HasColumnName("desired_state"); + + b.Property("LastDeployStatus") + .HasColumnType("text") + .HasColumnName("last_deploy_status"); + + b.Property("LastDeployedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_deployed_at"); + + b.Property("LastDeployedCommit") + .HasColumnType("text") + .HasColumnName("last_deployed_commit"); + + b.Property("LastDeployedReleaseId") + .HasColumnType("integer") + .HasColumnName("last_deployed_release_id"); + + b.Property("LastScheduledBackupAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_scheduled_backup_at"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("PinnedReleaseId") + .HasColumnType("integer") + .HasColumnName("pinned_release_id"); + + b.Property("ProductId") + .HasColumnType("integer") + .HasColumnName("product_id"); + + b.Property("TemplateId") + .HasColumnType("integer") + .HasColumnName("template_id"); + + b.Property("TenantSlug") + .HasColumnType("text") + .HasColumnName("tenant_slug"); + + b.Property("WebhookEnabled") + .HasColumnType("boolean") + .HasColumnName("webhook_enabled"); + + b.Property("WebhookToken") + .HasColumnType("text") + .HasColumnName("webhook_token"); + + b.Property("Xmin") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.HasKey("Id") + .HasName("pk_stacks"); + + b.HasIndex("AppApiToken") + .IsUnique() + .HasDatabaseName("ix_stacks_app_api_token"); + + b.HasIndex("LastDeployedReleaseId") + .HasDatabaseName("ix_stacks_last_deployed_release_id"); + + b.HasIndex("Name") + .IsUnique() + .HasDatabaseName("ix_stacks_name"); + + b.HasIndex("PinnedReleaseId") + .HasDatabaseName("ix_stacks_pinned_release_id"); + + b.HasIndex("ProductId") + .HasDatabaseName("ix_stacks_product_id"); + + b.HasIndex("TemplateId", "TenantSlug") + .IsUnique() + .HasDatabaseName("ix_stacks_template_id_tenant_slug"); + + b.ToTable("stacks", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackBackupServiceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Dump") + .HasColumnType("text") + .HasColumnName("dump"); + + b.Property("Exclude") + .HasColumnType("boolean") + .HasColumnName("exclude"); + + b.Property("Service") + .IsRequired() + .HasColumnType("text") + .HasColumnName("service"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("Stop") + .HasColumnType("text") + .HasColumnName("stop"); + + b.HasKey("Id") + .HasName("pk_stack_backup_service_overrides"); + + b.HasIndex("StackId", "Service") + .IsUnique() + .HasDatabaseName("ix_stack_backup_service_overrides_stack_id_service"); + + b.ToTable("stack_backup_service_overrides", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackEnvVar", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Key") + .IsRequired() + .HasColumnType("text") + .HasColumnName("key"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("Value") + .IsRequired() + .HasColumnType("text") + .HasColumnName("value"); + + b.HasKey("Id") + .HasName("pk_stack_env_vars"); + + b.HasIndex("StackId", "Key") + .IsUnique() + .HasDatabaseName("ix_stack_env_vars_stack_id_key"); + + b.ToTable("stack_env_vars", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackTemplate", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("BackupCron") + .HasColumnType("text") + .HasColumnName("backup_cron"); + + b.Property("BackupEnabled") + .HasColumnType("boolean") + .HasColumnName("backup_enabled"); + + b.Property("BackupQuiesceMode") + .HasColumnType("text") + .HasColumnName("backup_quiesce_mode"); + + b.Property("BackupStopContainers") + .HasColumnType("boolean") + .HasColumnName("backup_stop_containers"); + + b.Property("BranchOverride") + .HasColumnType("text") + .HasColumnName("branch_override"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("DefaultPinnedReleaseId") + .HasColumnType("integer") + .HasColumnName("default_pinned_release_id"); + + b.Property("DomainPattern") + .IsRequired() + .HasColumnType("text") + .HasColumnName("domain_pattern"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("ProductId") + .HasColumnType("integer") + .HasColumnName("product_id"); + + b.Property("RealmId") + .HasColumnType("integer") + .HasColumnName("realm_id"); + + b.Property("TargetPort") + .HasColumnType("integer") + .HasColumnName("target_port"); + + b.Property("TargetServiceName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("target_service_name"); + + b.HasKey("Id") + .HasName("pk_stack_templates"); + + b.HasIndex("DefaultPinnedReleaseId") + .HasDatabaseName("ix_stack_templates_default_pinned_release_id"); + + b.HasIndex("Name") + .IsUnique() + .HasDatabaseName("ix_stack_templates_name"); + + b.HasIndex("ProductId") + .HasDatabaseName("ix_stack_templates_product_id"); + + b.HasIndex("RealmId") + .HasDatabaseName("ix_stack_templates_realm_id"); + + b.ToTable("stack_templates", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackTemplateEnvVar", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Key") + .IsRequired() + .HasColumnType("text") + .HasColumnName("key"); + + b.Property("TemplateId") + .HasColumnType("integer") + .HasColumnName("template_id"); + + b.Property("Value") + .IsRequired() + .HasColumnType("text") + .HasColumnName("value"); + + b.HasKey("Id") + .HasName("pk_stack_template_env_vars"); + + b.HasIndex("TemplateId", "Key") + .IsUnique() + .HasDatabaseName("ix_stack_template_env_vars_template_id_key"); + + b.ToTable("stack_template_env_vars", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackUpdateCheck", b => + { + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("AvailableReleaseId") + .HasColumnType("integer") + .HasColumnName("available_release_id"); + + b.Property("AvailableReleaseVersion") + .HasColumnType("text") + .HasColumnName("available_release_version"); + + b.Property("CheckedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("checked_at"); + + b.Property("DriftedContainers") + .IsRequired() + .HasColumnType("text") + .HasColumnName("drifted_containers"); + + b.Property("HasUpdates") + .HasColumnType("boolean") + .HasColumnName("has_updates"); + + b.Property("NewCommitSha") + .HasColumnType("text") + .HasColumnName("new_commit_sha"); + + b.Property("OutdatedImageDigests") + .IsRequired() + .HasColumnType("text") + .HasColumnName("outdated_image_digests"); + + b.Property("OutdatedImages") + .IsRequired() + .HasColumnType("text") + .HasColumnName("outdated_images"); + + b.HasKey("StackId") + .HasName("pk_stack_update_checks"); + + b.ToTable("stack_update_checks", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.TemplateBackupServiceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Dump") + .HasColumnType("text") + .HasColumnName("dump"); + + b.Property("Exclude") + .HasColumnType("boolean") + .HasColumnName("exclude"); + + b.Property("Service") + .IsRequired() + .HasColumnType("text") + .HasColumnName("service"); + + b.Property("Stop") + .HasColumnType("text") + .HasColumnName("stop"); + + b.Property("TemplateId") + .HasColumnType("integer") + .HasColumnName("template_id"); + + b.HasKey("Id") + .HasName("pk_template_backup_service_overrides"); + + b.HasIndex("TemplateId", "Service") + .IsUnique() + .HasDatabaseName("ix_template_backup_service_overrides_template_id_service"); + + b.ToTable("template_backup_service_overrides", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.TemplateManagementGrant", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AllowDelete") + .HasColumnType("boolean") + .HasColumnName("allow_delete"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("StackId") + .HasColumnType("integer") + .HasColumnName("stack_id"); + + b.Property("TemplateId") + .HasColumnType("integer") + .HasColumnName("template_id"); + + b.HasKey("Id") + .HasName("pk_template_management_grants"); + + b.HasIndex("TemplateId") + .HasDatabaseName("ix_template_management_grants_template_id"); + + b.HasIndex("StackId", "TemplateId") + .IsUnique() + .HasDatabaseName("ix_template_management_grants_stack_id_template_id"); + + b.ToTable("template_management_grants", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.User", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("AccessFailedCount") + .HasColumnType("integer") + .HasColumnName("access_failed_count"); + + b.Property("AuthenticatorKey") + .HasColumnType("text") + .HasColumnName("authenticator_key"); + + b.Property("ConcurrencyStamp") + .IsConcurrencyToken() + .IsRequired() + .HasColumnType("text") + .HasColumnName("concurrency_stamp"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Disabled") + .HasColumnType("boolean") + .HasColumnName("disabled"); + + b.Property("Email") + .HasColumnType("text") + .HasColumnName("email"); + + b.Property("IsAdmin") + .HasColumnType("boolean") + .HasColumnName("is_admin"); + + b.Property("LockoutEnd") + .HasColumnType("timestamp with time zone") + .HasColumnName("lockout_end"); + + b.Property("NormalizedUserName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("normalized_user_name"); + + b.Property("PasswordHash") + .IsRequired() + .HasColumnType("text") + .HasColumnName("password_hash"); + + b.Property("RealmId") + .HasColumnType("integer") + .HasColumnName("realm_id"); + + b.Property("SecurityStamp") + .IsRequired() + .HasColumnType("text") + .HasColumnName("security_stamp"); + + b.Property("TwoFactorEnabled") + .HasColumnType("boolean") + .HasColumnName("two_factor_enabled"); + + b.Property("UserName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("user_name"); + + b.HasKey("Id") + .HasName("pk_users"); + + b.HasIndex("RealmId", "NormalizedUserName") + .IsUnique() + .HasDatabaseName("ix_users_realm_id_normalized_user_name"); + + b.ToTable("users", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.UserRecoveryCode", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CodeHash") + .IsRequired() + .HasColumnType("text") + .HasColumnName("code_hash"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("UserId") + .HasColumnType("integer") + .HasColumnName("user_id"); + + b.HasKey("Id") + .HasName("pk_user_recovery_codes"); + + b.HasIndex("UserId", "CodeHash") + .IsUnique() + .HasDatabaseName("ix_user_recovery_codes_user_id_code_hash"); + + b.ToTable("user_recovery_codes", (string)null); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.AuthSession", b => + { + b.HasOne("Watchtower.Application.Entities.Route", "Route") + .WithMany() + .HasForeignKey("RouteId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_auth_sessions_routes_route_id"); + + b.HasOne("Watchtower.Application.Entities.User", "User") + .WithMany() + .HasForeignKey("UserId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_auth_sessions_users_user_id"); + + b.Navigation("Route"); + + b.Navigation("User"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.BackupEvent", b => + { + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany() + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_backup_events_stacks_stack_id"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.CiRepo", b => + { + b.HasOne("Watchtower.Application.Entities.Credential", "Credential") + .WithMany() + .HasForeignKey("CredentialId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_ci_repos_credentials_credential_id"); + + b.Navigation("Credential"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.DeployEvent", b => + { + b.HasOne("Watchtower.Application.Entities.Release", "Release") + .WithMany() + .HasForeignKey("ReleaseId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_deploy_events_releases_release_id"); + + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany("DeployEvents") + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_deploy_events_stacks_stack_id"); + + b.Navigation("Release"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Group", b => + { + b.HasOne("Watchtower.Application.Entities.Realm", "Realm") + .WithMany() + .HasForeignKey("RealmId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_groups_realms_realm_id"); + + b.Navigation("Realm"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.GroupMember", b => + { + b.HasOne("Watchtower.Application.Entities.Group", "Group") + .WithMany() + .HasForeignKey("GroupId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_group_members_groups_group_id"); + + b.HasOne("Watchtower.Application.Entities.User", "User") + .WithMany() + .HasForeignKey("UserId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_group_members_users_user_id"); + + b.Navigation("Group"); + + b.Navigation("User"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.LoginCode", b => + { + b.HasOne("Watchtower.Application.Entities.Route", "Route") + .WithMany() + .HasForeignKey("RouteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_login_codes_routes_route_id"); + + b.HasOne("Watchtower.Application.Entities.User", "User") + .WithMany() + .HasForeignKey("UserId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_login_codes_users_user_id"); + + b.Navigation("Route"); + + b.Navigation("User"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Product", b => + { + b.HasOne("Watchtower.Application.Entities.CiRepo", "CiRepo") + .WithMany() + .HasForeignKey("CiRepoId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_products_ci_repos_ci_repo_id"); + + b.HasOne("Watchtower.Application.Entities.Credential", "Credential") + .WithMany() + .HasForeignKey("CredentialId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_products_credentials_credential_id"); + + b.Navigation("CiRepo"); + + b.Navigation("Credential"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Realm", b => + { + b.HasOne("Watchtower.Application.Entities.Route", "LoginRoute") + .WithMany() + .HasForeignKey("LoginRouteId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_realms_routes_login_route_id"); + + b.Navigation("LoginRoute"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Registry", b => + { + b.HasOne("Watchtower.Application.Entities.Credential", "Credential") + .WithMany() + .HasForeignKey("CredentialId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_registries_credentials_credential_id"); + + b.Navigation("Credential"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Release", b => + { + b.HasOne("Watchtower.Application.Entities.Product", "Product") + .WithMany("Releases") + .HasForeignKey("ProductId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_releases_products_product_id"); + + b.Navigation("Product"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.ReleaseImage", b => + { + b.HasOne("Watchtower.Application.Entities.Release", "Release") + .WithMany("Images") + .HasForeignKey("ReleaseId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_release_images_releases_release_id"); + + b.Navigation("Release"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Route", b => + { + b.HasOne("Watchtower.Application.Entities.Realm", "Realm") + .WithMany() + .HasForeignKey("RealmId") + .OnDelete(DeleteBehavior.Restrict) + .HasConstraintName("fk_routes_realms_realm_id"); + + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany() + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_routes_stacks_stack_id"); + + b.Navigation("Realm"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.RouteAccessGrant", b => + { + b.HasOne("Watchtower.Application.Entities.Group", "Group") + .WithMany() + .HasForeignKey("GroupId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_route_access_grants_groups_group_id"); + + b.HasOne("Watchtower.Application.Entities.Route", "Route") + .WithMany() + .HasForeignKey("RouteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_route_access_grants_routes_route_id"); + + b.HasOne("Watchtower.Application.Entities.User", "User") + .WithMany() + .HasForeignKey("UserId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_route_access_grants_users_user_id"); + + b.Navigation("Group"); + + b.Navigation("Route"); + + b.Navigation("User"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Stack", b => + { + b.HasOne("Watchtower.Application.Entities.Release", "LastDeployedRelease") + .WithMany() + .HasForeignKey("LastDeployedReleaseId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_stacks_releases_last_deployed_release_id"); + + b.HasOne("Watchtower.Application.Entities.Release", "PinnedRelease") + .WithMany() + .HasForeignKey("PinnedReleaseId") + .OnDelete(DeleteBehavior.Restrict) + .HasConstraintName("fk_stacks_releases_pinned_release_id"); + + b.HasOne("Watchtower.Application.Entities.Product", "Product") + .WithMany("Stacks") + .HasForeignKey("ProductId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_stacks_products_product_id"); + + b.HasOne("Watchtower.Application.Entities.StackTemplate", "Template") + .WithMany("Instances") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_stacks_stack_templates_template_id"); + + b.Navigation("LastDeployedRelease"); + + b.Navigation("PinnedRelease"); + + b.Navigation("Product"); + + b.Navigation("Template"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackBackupServiceOverride", b => + { + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany() + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_stack_backup_service_overrides_stacks_stack_id"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackEnvVar", b => + { + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany("EnvVars") + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_stack_env_vars_stacks_stack_id"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackTemplate", b => + { + b.HasOne("Watchtower.Application.Entities.Release", "DefaultPinnedRelease") + .WithMany() + .HasForeignKey("DefaultPinnedReleaseId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_stack_templates_releases_default_pinned_release_id"); + + b.HasOne("Watchtower.Application.Entities.Product", "Product") + .WithMany("Templates") + .HasForeignKey("ProductId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_stack_templates_products_product_id"); + + b.HasOne("Watchtower.Application.Entities.Realm", "Realm") + .WithMany() + .HasForeignKey("RealmId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_stack_templates_realms_realm_id"); + + b.Navigation("DefaultPinnedRelease"); + + b.Navigation("Product"); + + b.Navigation("Realm"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackTemplateEnvVar", b => + { + b.HasOne("Watchtower.Application.Entities.StackTemplate", "Template") + .WithMany("BaseEnvVars") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_stack_template_env_vars_stack_templates_template_id"); + + b.Navigation("Template"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackUpdateCheck", b => + { + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithOne("UpdateCheck") + .HasForeignKey("Watchtower.Application.Entities.StackUpdateCheck", "StackId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_stack_update_checks_stacks_stack_id"); + + b.Navigation("Stack"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.TemplateBackupServiceOverride", b => + { + b.HasOne("Watchtower.Application.Entities.StackTemplate", "Template") + .WithMany("BackupServiceOverrides") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_template_backup_service_overrides_stack_templates_template_"); + + b.Navigation("Template"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.TemplateManagementGrant", b => + { + b.HasOne("Watchtower.Application.Entities.Stack", "Stack") + .WithMany() + .HasForeignKey("StackId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_template_management_grants_stacks_stack_id"); + + b.HasOne("Watchtower.Application.Entities.StackTemplate", "Template") + .WithMany() + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_template_management_grants_stack_templates_template_id"); + + b.Navigation("Stack"); + + b.Navigation("Template"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.User", b => + { + b.HasOne("Watchtower.Application.Entities.Realm", "Realm") + .WithMany() + .HasForeignKey("RealmId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_users_realms_realm_id"); + + b.Navigation("Realm"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.UserRecoveryCode", b => + { + b.HasOne("Watchtower.Application.Entities.User", "User") + .WithMany() + .HasForeignKey("UserId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_user_recovery_codes_users_user_id"); + + b.Navigation("User"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Product", b => + { + b.Navigation("Releases"); + + b.Navigation("Stacks"); + + b.Navigation("Templates"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Release", b => + { + b.Navigation("Images"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.Stack", b => + { + b.Navigation("DeployEvents"); + + b.Navigation("EnvVars"); + + b.Navigation("UpdateCheck"); + }); + + modelBuilder.Entity("Watchtower.Application.Entities.StackTemplate", b => + { + b.Navigation("BackupServiceOverrides"); + + b.Navigation("BaseEnvVars"); + + b.Navigation("Instances"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.cs b/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.cs new file mode 100644 index 0000000..76528bd --- /dev/null +++ b/src/Watchtower.Application/Persistence/Migrations/20260826192544_AddInstanceBackupEvents.cs @@ -0,0 +1,36 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace Watchtower.Application.Persistence.Migrations +{ + /// + public partial class AddInstanceBackupEvents : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.AlterColumn( + name: "stack_id", + table: "backup_events", + type: "integer", + nullable: true, + oldClrType: typeof(int), + oldType: "integer"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.AlterColumn( + name: "stack_id", + table: "backup_events", + type: "integer", + nullable: false, + defaultValue: 0, + oldClrType: typeof(int), + oldType: "integer", + oldNullable: true); + } + } +} diff --git a/src/Watchtower.Application/Persistence/Migrations/WatchtowerDbContextModelSnapshot.cs b/src/Watchtower.Application/Persistence/Migrations/WatchtowerDbContextModelSnapshot.cs index cf9e6ed..cc80f27 100644 --- a/src/Watchtower.Application/Persistence/Migrations/WatchtowerDbContextModelSnapshot.cs +++ b/src/Watchtower.Application/Persistence/Migrations/WatchtowerDbContextModelSnapshot.cs @@ -339,7 +339,7 @@ protected override void BuildModel(ModelBuilder modelBuilder) .HasColumnType("bigint") .HasColumnName("size_bytes"); - b.Property("StackId") + b.Property("StackId") .HasColumnType("integer") .HasColumnName("stack_id"); @@ -1998,7 +1998,6 @@ protected override void BuildModel(ModelBuilder modelBuilder) .WithMany() .HasForeignKey("StackId") .OnDelete(DeleteBehavior.Cascade) - .IsRequired() .HasConstraintName("fk_backup_events_stacks_stack_id"); b.Navigation("Stack"); diff --git a/src/Watchtower.Application/Services/BackupArchiveReader.cs b/src/Watchtower.Application/Services/BackupArchiveReader.cs new file mode 100644 index 0000000..59c2fab --- /dev/null +++ b/src/Watchtower.Application/Services/BackupArchiveReader.cs @@ -0,0 +1,89 @@ +using System.Formats.Tar; +using System.IO.Compression; +using System.Security.Cryptography; + +namespace Watchtower.Application.Services; + +/// +/// Opens a spooled backup archive for reading: file → optional decrypt → gunzip → tar. Shared by the +/// stack restore (ADR-0016) and the instance restore (ADR-0027), which read the same format for +/// different reasons and must not drift on how a damaged or wrongly-keyed archive is reported. +/// +/// +/// Takes the passphrase directly rather than the options: an instance restore decrypts with the +/// passphrase from the bundle it was handed, which is by definition not the one this instance +/// is configured with. +/// +public static class BackupArchiveReader { + /// + /// Runs over the archive as an uncompressed tar stream, disposing every + /// layer afterwards — including the file handle, which a caller's delete depends on. + /// + /// The spooled archive on disk. + /// The passphrase it was encrypted with, or null when it is not encrypted. + /// What to do with the tar stream. + public static async Task WithArchiveAsync( + string archivePath, string? passphrase, Func> action) { + await using var file = File.OpenRead(archivePath); + var inner = passphrase is { Length: > 0 } + ? BackupEncryption.CreateDecryptingStream(file, passphrase) + : file; + try { + await using var tar = new GZipStream(inner, CompressionMode.Decompress, leaveOpen: true); + return await action(tar); + } finally { + if (!ReferenceEquals(inner, file)) await inner.DisposeAsync(); + } + } + + /// + /// Scans the archive's table of contents, translating a decode failure on an encrypted archive into + /// the question an operator can actually answer. + /// + /// The spooled archive on disk. + /// The passphrase it was encrypted with, or null when it is not encrypted. + /// Cancellation token. + /// The archive could not be decoded. + public static Task ReadContentsAsync( + string archivePath, string? passphrase, CancellationToken ct) => + WithArchiveAsync(archivePath, passphrase, async tar => { + try { + return await BackupArchiveInspector.InspectAsync(tar, ct); + } catch (Exception ex) when (passphrase is { Length: > 0 } + && ex is InvalidDataException or CryptographicException) { + throw new InvalidOperationException( + "Could not read the encrypted archive — is the encryption passphrase the one it was " + + $"written with? ({ex.Message})"); + } + }); + + /// + /// Copies one file out of the archive into . Read straight from the + /// archive rather than kept from an earlier pass: a dump is arbitrarily large and has no business + /// in memory. + /// + /// The spooled archive on disk. + /// The passphrase it was encrypted with, or null when it is not encrypted. + /// The path inside the archive's backup/ root, e.g. _dumps/db.sql. + /// Host path to write; created and overwritten. + /// Cancellation token. + /// The size of the extracted file. + /// The archive does not contain that file. + public static Task ExtractAsync( + string archivePath, string? passphrase, string relativeFile, string destination, CancellationToken ct) => + WithArchiveAsync(archivePath, passphrase, async tar => { + var wanted = $"backup/{relativeFile}"; + await using var reader = new TarReader(tar, leaveOpen: true); + while (await reader.GetNextEntryAsync(cancellationToken: ct) is { } entry) { + if (!string.Equals(entry.Name.TrimStart('.', '/'), wanted, StringComparison.Ordinal)) continue; + if (entry.DataStream is not { } data) break; + await using var file = File.Create(destination); + await data.CopyToAsync(file, ct); + return file.Length; + } + // The table-of-contents scan found it a moment ago, so this means the archive changed under + // us or is damaged — either way, not something to restore from. + throw new InvalidOperationException( + $"The archive does not contain '{wanted}', although its table of contents lists it."); + }); +} diff --git a/src/Watchtower.Application/Services/BackupBundle.cs b/src/Watchtower.Application/Services/BackupBundle.cs new file mode 100644 index 0000000..18af8fd --- /dev/null +++ b/src/Watchtower.Application/Services/BackupBundle.cs @@ -0,0 +1,144 @@ +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace Watchtower.Application.Services; + +/// +/// The layout of a full backup bundle (ADR-0027 §4): one plain tar — its members are already +/// gzipped and encrypted, and its purpose is to be handed to the import on the other side, not to be +/// small — holding a fresh archive of Watchtower's own database, the newest archive of every stack, and +/// the two JSON files that describe them. +/// +/// +/// +/// bundle-manifest.json +/// secrets.json +/// watchtower/watchtower_20260826T033000Z.tar.gz.enc +/// stacks/prod/blog/blog_20260826T033000Z.tar.gz.enc +/// stacks/prod/shop/globex/shop-globex_20260826T033100Z.tar.gz.enc +/// +/// A stack archive keeps its storage-relative path under stacks/, so an import can put +/// it back byte for byte where the restored database already expects to find it +/// () instead of having to rewrite paths it cannot verify. +/// +public static class BackupBundle { + /// The manifest's entry name inside the tar. + public const string ManifestEntry = "bundle-manifest.json"; + + /// The secrets file's entry name inside the tar. + public const string SecretsEntry = "secrets.json"; + + /// Directory holding the instance's own archive. + public const string InstanceDirectory = "watchtower"; + + /// Directory the stack archives keep their storage-relative paths under. + public const string StacksDirectory = "stacks"; + + /// + /// The bundle format this build writes and reads. Bumped only for a change a previous reader could + /// not survive; additive keys do not move it. + /// + public const int FormatVersion = 1; + + /// The file name offered for download, e.g. watchtower-bundle_prod_20260826T033000Z.tar. + public static string FileName(string instanceName, DateTimeOffset createdAt) => + $"watchtower-bundle_{BackupNaming.Sanitize(instanceName)}_" + + $"{createdAt.UtcDateTime:yyyyMMdd'T'HHmmss'Z'}.tar"; + + /// How the bundle's JSON is written and read: camelCase, nulls kept, indented for a human. + public static readonly JsonSerializerOptions JsonOptions = new() { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + PropertyNameCaseInsensitive = true, + WriteIndented = true, + DefaultIgnoreCondition = JsonIgnoreCondition.Never, + }; +} + +/// One archive inside the bundle, and enough about it to verify and place it. +/// Its path inside the tar. +/// Where it belongs on the backup storage, relative to the provider base. +/// Its size. +/// Lowercase hex digest of the bytes, so an import can prove the tar is intact. +/// When the archive was made. +/// Whether it needs the passphrase in secrets.json. +public sealed record BundleArchive( + string Entry, string StoragePath, long SizeBytes, string Sha256, DateTimeOffset TakenAtUtc, bool Encrypted); + +/// One stack's entry in the bundle manifest: what it is, and which archive belongs to it. +/// The stack's id in the exporting instance's database. +/// The stack's name, for the operator-facing checklist. +/// Its compose project — the identity its volumes carry. +/// The archive, or null when the stack had none on the storage. +/// Why is null; null when it is not. +public sealed record BundleStack( + int StackId, string Name, string ComposeProject, BundleArchive? Archive, string? Reason); + +/// +/// bundle-manifest.json: what the bundle holds and which Watchtower wrote it. +/// +/// See . +/// Always watchtower, so a stray tar identifies itself. +/// When the export ran. +/// The exporting instance's backup name. +/// Its build, for the operator-facing half of a version refusal. +/// +/// Its schema. This is what an import decides on: migrations only roll forward, so a bundle +/// whose last migration the target binary has never heard of cannot be replayed into it. +/// +/// +/// Whether the exporting instance encrypted its stored private keys at all — so an import can tell +/// "no secret was in use" from "the secret is missing from this bundle". +/// +/// The archive of Watchtower's own database. +/// Every stack, including those with no archive to carry. +public sealed record BundleManifest( + int BundleFormatVersion, + string Tool, + DateTimeOffset CreatedAtUtc, + string InstanceName, + string AppVersion, + string? LastMigrationId, + bool KeyProtectionSecretConfigured, + BundleArchive Instance, + IReadOnlyList Stacks); + +/// +/// secrets.json: the material that lives outside the database, without which a restored +/// instance is inert (ADR-0027 §4). +/// +/// +/// Plain text, deliberately. The alternative — a bundle that restores into an instance whose every +/// certificate and key throws because a secret the operator never knew about stayed behind on a machine +/// that no longer exists — is the failure this file exists to prevent. It is why the export is +/// admin-only and audited, and why the UI says what the file is. +/// +/// Versioned separately from the manifest; both are read together. +/// +/// Watchtower:Auth:KeyProtectionSecret. The stored certificates, ACME account key and signing key +/// are AES-GCM under it, and it cannot be changed at runtime — a restore checks it before replaying. +/// +/// What decrypts every archive in this bundle. +/// The instance name the storage layout was written under. +/// Where the archives came from, so a restored instance keeps backing up. +public sealed record BundleSecrets( + int SecretsFormatVersion, + string? KeyProtectionSecret, + string? BackupEncryptionPassphrase, + string? BackupInstanceName, + BundleStorageSecrets Storage); + +/// The backup storage credentials, as the exporting instance held them. +public sealed record BundleStorageSecrets( + string Provider, + BundleSftpSecrets Sftp, + string LocalBasePath); + +/// SFTP credentials, whole — a restore that cannot reach the storage cannot revive the stacks. +public sealed record BundleSftpSecrets( + string? Host, + int Port, + string? Username, + string? Password, + string? PrivateKey, + string? PrivateKeyPassphrase, + string BasePath); diff --git a/src/Watchtower.Application/Services/BackupBundleService.cs b/src/Watchtower.Application/Services/BackupBundleService.cs new file mode 100644 index 0000000..1232fdc --- /dev/null +++ b/src/Watchtower.Application/Services/BackupBundleService.cs @@ -0,0 +1,352 @@ +using System.Formats.Tar; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// A finished bundle waiting on disk for its download. +/// Host path of the tar. +/// The name it is offered under. +/// Its size. +/// When the export finished. +/// How many stack archives it carries. +/// How many stacks had no archive to carry. +public sealed record StagedBundle( + string Path, string FileName, long SizeBytes, DateTimeOffset CreatedAtUtc, + int StackCount, int MissingStackCount); + +/// +/// Holds the one bundle that is ready to download. Process-local and deliberately not persisted: the +/// tar lives in the container's own filesystem, so a restart loses both halves together and the export +/// is cheap to repeat. +/// +public sealed class BundleExportState { + private readonly Lock _gate = new(); + private StagedBundle? _staged; + + /// The staged bundle, or null when there is none (or its file has since gone). + public StagedBundle? Current { + get { + lock (_gate) { + if (_staged is { } staged && !File.Exists(staged.Path)) _staged = null; + return _staged; + } + } + } + + /// Publishes a new bundle and deletes the one it replaces. + public void Replace(StagedBundle staged) { + StagedBundle? previous; + lock (_gate) { + previous = _staged; + _staged = staged; + } + Delete(previous); + } + + /// Drops and deletes the staged bundle, if any. + public void Clear() { + StagedBundle? previous; + lock (_gate) { + previous = _staged; + _staged = null; + } + Delete(previous); + } + + /// + /// Removes a replaced bundle, directory and all. Each export stages into a directory of its own, so + /// this can never reach the bundle that replaced it — two exports within the same second would + /// otherwise agree on a file name, and deleting "the old one" would delete the new one's bytes. + /// + private static void Delete(StagedBundle? staged) { + if (staged is null) return; + try { + var directory = Path.GetDirectoryName(staged.Path); + if (directory is not null && Directory.Exists(directory)) + Directory.Delete(directory, recursive: true); + } catch (IOException) { + // A download may still be reading it. It is in the container's temp directory, so the + // worst case is a file that outlives the process rather than one that is never freed. + } catch (UnauthorizedAccessException) { + // Same. + } + } +} + +/// +/// Builds the exportable full backup bundle (ADR-0027 §4): a fresh dump of Watchtower's own database +/// plus the newest archive of every stack, in one plain tar with a manifest and the out-of-database +/// secrets, staged on disk for an admin to download. +/// +/// +/// Runs as a job on the single-flight backup queue rather than inside the request that asked for it: it +/// takes a dump and downloads every stack's newest archive, which is minutes of work against the same +/// spool disk and storage connection a stack backup uses. +/// +public sealed class BackupBundleService( + InstanceBackupService instanceBackup, + BackupStorageFactory storageFactory, + BundleExportState state, + IServiceScopeFactory scopeFactory, + IOptionsMonitor options, + AuditLog audit, + ILogger logger) { + /// How the bundle export names itself in the audit trail and the run log. + internal const string AuditTarget = "watchtower (bundle)"; + + /// Where staged bundles are written, under the process temp directory. + private static string StagingDirectory => Path.Combine(Path.GetTempPath(), "watchtower-bundle"); + + /// + /// Deletes bundles left by a previous process. Called at startup: the state that knew about them + /// did not survive, so nothing will ever hand them out — they would just occupy the disk. + /// + public void CleanStagingDirectory() { + try { + if (Directory.Exists(StagingDirectory)) Directory.Delete(StagingDirectory, recursive: true); + } catch (Exception ex) { + logger.LogWarning(ex, "Could not clear the bundle staging directory {Directory}", StagingDirectory); + } + } + + /// Builds the bundle for an event created by . + /// The stackless event tracking this export. + /// The worker's token. + public async Task ExecuteExportAsync(int backupEventId, CancellationToken ct) { + var output = new StringBuilder(); + void Log(string line) { lock (output) output.AppendLine(line); } + + string triggeredBy; + using (var scope = scopeFactory.CreateScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, ct); + if (evt is null) return; + triggeredBy = evt.TriggeredBy; + evt.Status = BackupStatuses.Running; + evt.StartedAt = DateTimeOffset.UtcNow; + await db.SaveChangesAsync(ct); + } + + var backup = options.CurrentValue.Backup; + try { + var staged = await BuildAsync(backup, Log, ct); + state.Replace(staged); + await FinishAsync(backupEventId, success: true, output.ToString(), staged.FileName, staged.SizeBytes); + await audit.RecordAsync(BackupService.AuditCategory, "bundle.export", AuditTarget, + $"{triggeredBy} · {staged.StackCount} stack archive(s)" + + (staged.MissingStackCount > 0 ? $" · {staged.MissingStackCount} stack(s) with no archive" : "") + + $" · {staged.SizeBytes} bytes → {staged.FileName}", + ct: CancellationToken.None); + logger.LogInformation( + "Backup bundle staged: {FileName} ({SizeBytes} bytes)", staged.FileName, staged.SizeBytes); + } catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) { + Log($"FAILED: {ex.Message}"); + await FinishAsync(backupEventId, success: false, output.ToString(), remotePath: null, sizeBytes: null); + await audit.RecordAsync(BackupService.AuditCategory, "bundle.export", AuditTarget, + InstanceBackupService.Summary(triggeredBy, backup), + success: false, error: ex.Message, ct: CancellationToken.None); + logger.LogWarning(ex, "Backup bundle export failed"); + } + } + + /// Takes the dump, collects the stack archives and writes the tar. + private async Task BuildAsync( + BackupOptions backup, Action log, CancellationToken ct) { + // The instance archive is taken fresh rather than read back off the storage: a bundle is a + // point-in-time copy of *this* instance, and the newest stored dump could be a day old. + log("Backing up Watchtower's own database for the bundle…"); + var instance = await instanceBackup.RunAsync(backup, log, ct); + + var createdAt = DateTimeOffset.UtcNow; + var fileName = BackupBundle.FileName(backup.ResolveInstanceName(), createdAt); + // A directory per export, so the path is unique even when two exports agree on the file name — + // the name is second-resolution, and it is what the operator downloads, not what identifies it. + var stagingDirectory = Path.Combine(StagingDirectory, Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(stagingDirectory); + var bundlePath = Path.Combine(stagingDirectory, fileName); + var spools = new List(); + + try { + using var storage = storageFactory.Create(backup); + + // Downloaded to disk first, because a tar entry has to declare its length before its bytes. + var instanceSpool = await SpoolAsync(storage, instance.RelativePath, spools, ct); + var instanceEntry = + $"{BackupBundle.InstanceDirectory}/{instance.FileName}"; + var instanceArchive = new BundleArchive( + instanceEntry, instance.RelativePath, new FileInfo(instanceSpool).Length, + await Sha256Async(instanceSpool, ct), instance.TakenAt, Encrypted: true); + + var stacks = new List<(BundleStack Stack, string? Spool)>(); + foreach (var stack in await LoadStacksAsync(ct)) { + var directory = BackupNaming.ResolveDirectory(stack, backup.ResolveInstanceName()); + var newest = await NewestArchiveAsync(storage, directory, ct); + if (newest is not { } found) { + log($"WARNING: stack '{stack.Name}' has no archive on the storage — the bundle " + + "carries its definition but not its data."); + stacks.Add(( + new BundleStack(stack.Id, stack.Name, stack.ComposeProjectName, null, + "no archive on the backup storage"), + null)); + continue; + } + + var storagePath = $"{directory}/{found.File.Name}"; + var spool = await SpoolAsync(storage, storagePath, spools, ct); + log($"Collected '{stack.Name}': {found.File.Name} ({found.File.SizeBytes} bytes)"); + stacks.Add(( + new BundleStack(stack.Id, stack.Name, stack.ComposeProjectName, + new BundleArchive( + $"{BackupBundle.StacksDirectory}/{storagePath}", storagePath, + new FileInfo(spool).Length, await Sha256Async(spool, ct), found.TakenAt, + found.File.Name.EndsWith(".enc", StringComparison.Ordinal)), + Reason: null), + spool)); + } + + var manifest = new BundleManifest( + BackupBundle.FormatVersion, "watchtower", createdAt, backup.ResolveInstanceName(), + InstanceVersion.App, await LastMigrationAsync(ct), + KeyProtectionSecretConfigured: !string.IsNullOrEmpty(options.CurrentValue.Auth.KeyProtectionSecret), + instanceArchive, [.. stacks.Select(s => s.Stack)]); + + await using (var tar = File.Create(bundlePath)) + await using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true)) { + await WriteJsonAsync(writer, BackupBundle.ManifestEntry, manifest, ct); + // 0600 on the secrets, matching how the dumps are written into the helper container: + // the mode is not protection on its own, but a file this sensitive should not be + // world-readable the moment someone untars it as root. + await WriteJsonAsync( + writer, BackupBundle.SecretsEntry, BuildSecrets(options.CurrentValue), ct, + UnixFileMode.UserRead | UnixFileMode.UserWrite); + await WriteFileAsync(writer, instanceEntry, instanceSpool, ct); + foreach (var (stack, spool) in stacks) + if (stack.Archive is { } archive && spool is not null) + await WriteFileAsync(writer, archive.Entry, spool, ct); + } + + var sizeBytes = new FileInfo(bundlePath).Length; + var carried = stacks.Count(s => s.Stack.Archive is not null); + log($"Bundle complete: {sizeBytes} bytes, {carried} stack archive(s)."); + return new StagedBundle( + bundlePath, fileName, sizeBytes, createdAt, carried, stacks.Count - carried); + } catch { + try { + Directory.Delete(stagingDirectory, recursive: true); + } catch (Exception ex) { + logger.LogWarning(ex, "Could not delete the partial bundle {BundlePath}", bundlePath); + } + throw; + } finally { + foreach (var spool in spools) { + try { + if (File.Exists(spool)) File.Delete(spool); + } catch (Exception ex) { + logger.LogWarning(ex, "Failed to delete bundle spool file {SpoolPath}", spool); + } + } + } + } + + /// The stacks a bundle describes, in a stable order so two exports read the same. + private async Task> LoadStacksAsync(CancellationToken ct) { + using var scope = scopeFactory.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + return await db.Stacks.AsNoTracking().OrderBy(s => s.Name).ToListAsync(ct); + } + + private async Task LastMigrationAsync(CancellationToken ct) { + using var scope = scopeFactory.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + return await InstanceVersion.LastMigrationAsync(db, ct); + } + + /// The newest Watchtower-named archive in one directory, or null when there is none. + private static async Task<(BackupStorageFile File, DateTimeOffset TakenAt)?> NewestArchiveAsync( + IBackupStorage storage, string directory, CancellationToken ct) => + (await storage.ListFilesAsync(directory, ct)) + .Select(f => (File: f, TakenAt: BackupNaming.ParseTimestamp(f.Name))) + .Where(x => x.TakenAt is not null) + .OrderByDescending(x => x.TakenAt) + .Select(x => ((BackupStorageFile, DateTimeOffset)?)(x.File, x.TakenAt!.Value)) + .FirstOrDefault(); + + /// Downloads one archive to a temp file and registers it for cleanup. + private static async Task SpoolAsync( + IBackupStorage storage, string relativePath, List spools, CancellationToken ct) { + var spool = Path.Combine(Path.GetTempPath(), $"watchtower-bundle-{Guid.NewGuid():N}.spool"); + spools.Add(spool); + await using var file = File.Create(spool); + await storage.DownloadAsync(relativePath, file, ct); + return spool; + } + + /// + /// The out-of-database material, exactly as this instance holds it (ADR-0027 §4). Plain text: see + /// for why that is the decision rather than an oversight. + /// + internal static BundleSecrets BuildSecrets(WatchtowerOptions options) { + var backup = options.Backup; + return new BundleSecrets( + SecretsFormatVersion: 1, + KeyProtectionSecret: options.Auth.KeyProtectionSecret, + BackupEncryptionPassphrase: backup.EncryptionPassphrase, + BackupInstanceName: backup.ResolveInstanceName(), + Storage: new BundleStorageSecrets( + Provider: backup.ResolveProvider() == BackupProviderKind.Local ? "local" : "sftp", + Sftp: new BundleSftpSecrets( + backup.Sftp.Host, backup.Sftp.Port, backup.Sftp.Username, backup.Sftp.Password, + backup.Sftp.PrivateKey, backup.Sftp.PrivateKeyPassphrase, backup.Sftp.BasePath), + LocalBasePath: backup.Local.BasePath)); + } + + private static async Task WriteJsonAsync( + TarWriter writer, string entryName, T value, CancellationToken ct, + UnixFileMode mode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead + | UnixFileMode.OtherRead) { + using var content = new MemoryStream( + JsonSerializer.SerializeToUtf8Bytes(value, BackupBundle.JsonOptions)); + await writer.WriteEntryAsync( + new PaxTarEntry(TarEntryType.RegularFile, entryName) { Mode = mode, DataStream = content }, ct); + } + + private static async Task WriteFileAsync( + TarWriter writer, string entryName, string sourcePath, CancellationToken ct) { + await using var content = File.OpenRead(sourcePath); + await writer.WriteEntryAsync( + new PaxTarEntry(TarEntryType.RegularFile, entryName) { + Mode = UnixFileMode.UserRead | UnixFileMode.UserWrite, + DataStream = content, + }, ct); + } + + /// Lowercase hex SHA-256 of a file, so an import can prove the tar arrived intact. + private static async Task Sha256Async(string path, CancellationToken ct) { + await using var file = File.OpenRead(path); + return Convert.ToHexStringLower(await SHA256.HashDataAsync(file, ct)); + } + + private async Task FinishAsync( + int backupEventId, bool success, string output, string? remotePath, long? sizeBytes) { + using var scope = scopeFactory.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, CancellationToken.None); + if (evt is null) return; + evt.Status = success ? BackupStatuses.Success : BackupStatuses.Failed; + evt.Output = output.Replace("\0", ""); + // Not a storage path: the bundle never leaves this host. The file name is what the UI shows. + evt.RemotePath = remotePath; + evt.SizeBytes = sizeBytes; + evt.FinishedAt = DateTimeOffset.UtcNow; + await db.SaveChangesAsync(CancellationToken.None); + } +} diff --git a/src/Watchtower.Application/Services/BackupNaming.cs b/src/Watchtower.Application/Services/BackupNaming.cs index 242f3d9..c70c968 100644 --- a/src/Watchtower.Application/Services/BackupNaming.cs +++ b/src/Watchtower.Application/Services/BackupNaming.cs @@ -38,6 +38,37 @@ public static string StackDirectory(string instanceName, string stackName) => public static string TenantDirectory(string instanceName, string productName, string tenantSlug) => $"{Sanitize(instanceName)}/{Sanitize(productName)}/{Sanitize(tenantSlug)}"; + /// + /// Where the instance's own archives live: {instance}/_watchtower, a sibling of the stack + /// directories under the same instance root (ADR-0027). One storage folder per instance therefore + /// holds everything needed to rebuild it — Watchtower's database next to the stacks' volumes. + /// + /// + /// The leading underscore is what keeps it out of the stacks' way. preserves + /// underscores, so a stack literally named _watchtower would collide — + /// is where that is refused, at stack-create time, rather than discovered here at backup time. + /// + /// The resolved Watchtower instance name. + public static string InstanceDirectory(string instanceName) => + $"{Sanitize(instanceName)}/{InstanceDirectorySegment}"; + + /// The directory segment the instance's own archives live in, under the instance root. + public const string InstanceDirectorySegment = "_watchtower"; + + /// + /// The file-name stem of an instance archive, standing where a stack archive carries its compose + /// project. anchors on the timestamp suffix alone, so retention and the + /// remote listing treat these exactly like any other archive. + /// + public const string InstanceFileStem = "watchtower"; + + /// + /// Whether would sanitize onto the instance's own directory — the one + /// name a stack may not have, since its archives would then be written among Watchtower's own. + /// + public static bool IsReserved(string stackName) => + string.Equals(Sanitize(stackName), InstanceDirectorySegment, StringComparison.OrdinalIgnoreCase); + /// /// Where 's archives live: the directory stamped on the row, or — for a /// stack created before the column existed — the value that has always been computed from the live diff --git a/src/Watchtower.Application/Services/BackupQueueService.cs b/src/Watchtower.Application/Services/BackupQueueService.cs index cb43f07..b6efa65 100644 --- a/src/Watchtower.Application/Services/BackupQueueService.cs +++ b/src/Watchtower.Application/Services/BackupQueueService.cs @@ -26,13 +26,16 @@ public sealed record BackupEnqueueResult(int BackupEventId, string Status); /// public class BackupQueueService( BackupService backupService, + InstanceBackupService instanceBackupService, + BackupBundleService bundleService, BackupChainCoordinator chain, IServiceScopeFactory scopeFactory, ILogger logger) : BackgroundService { - private enum JobKind { Backup, Restore } + private enum JobKind { Backup, Restore, InstanceBackup, BundleExport } - private sealed record Job(int EventId, int StackId, JobKind Kind, string? FileName); + /// is null for the jobs that back up Watchtower itself (ADR-0027). + private sealed record Job(int EventId, int? StackId, JobKind Kind, string? FileName); private readonly Channel _channel = Channel.CreateUnbounded(new UnboundedChannelOptions { SingleReader = true }); @@ -41,6 +44,20 @@ private sealed record Job(int EventId, int StackId, JobKind Kind, string? FileNa private readonly HashSet _queuedRestoreStacks = []; private int? _runningStackId; + /// + /// The instance self-backup waiting on the queue, if any — the stackless counterpart of + /// , coalescing for the same reason: a second request while one is + /// still waiting wants the backup that is about to happen, not two of them. + /// + private int? _queuedInstanceEventId; + + /// + /// The bundle export waiting on the queue, if any. Coalesced like the others: an export takes a + /// fresh dump and downloads every stack's newest archive, so two of them are minutes of duplicated + /// work for one file that only the second would keep. + /// + private int? _queuedBundleEventId; + /// /// Enqueues a backup for . Returns the tracking event — a fresh /// queued one, or the stack's already-waiting backup event (coalesced). @@ -89,6 +106,45 @@ public virtual BackupEnqueueResult Enqueue( } } + /// + /// Enqueues a backup of Watchtower's own database (ADR-0027). Returns the tracking event — a fresh + /// stackless queued one, or the already-waiting instance backup (coalesced). + /// + /// + /// Deliberately on the same single-flight queue as the stack runs rather than beside it: they compete + /// for the same spool disk, the same storage connection and the same daemon, and the instance dump is + /// small and infrequent. The cost is that it waits behind a large stack backup, which is the right way + /// round — a queued dump is a delayed dump, whereas two runs racing for the disk is a failed one. + /// + /// What to record on the event — see . + public virtual BackupEnqueueResult EnqueueInstance(string triggeredBy) { + lock (_lock) { + if (_queuedInstanceEventId is { } pending) return new BackupEnqueueResult(pending, "queued"); + + var eventId = CreateEvent(stackId: null, triggeredBy); + _queuedInstanceEventId = eventId; + _channel.Writer.TryWrite(new Job(eventId, StackId: null, JobKind.InstanceBackup, FileName: null)); + return new BackupEnqueueResult(eventId, "queued"); + } + } + + /// + /// Enqueues a full backup bundle export (ADR-0027 §4) — a fresh instance dump plus every stack's + /// newest archive, staged on disk for download. Returns the tracking event, coalescing onto an + /// export that is already waiting. + /// + /// What to record on the event — see . + public virtual BackupEnqueueResult EnqueueBundleExport(string triggeredBy) { + lock (_lock) { + if (_queuedBundleEventId is { } pending) return new BackupEnqueueResult(pending, "queued"); + + var eventId = CreateEvent(stackId: null, triggeredBy); + _queuedBundleEventId = eventId; + _channel.Writer.TryWrite(new Job(eventId, StackId: null, JobKind.BundleExport, FileName: null)); + return new BackupEnqueueResult(eventId, "queued"); + } + } + /// How often the startup reconcile retries while the daemon is not answering. internal static readonly TimeSpan ReconcileRetryDelay = TimeSpan.FromSeconds(15); @@ -109,16 +165,24 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken) { lock (_lock) { // Only remove the backup mapping if it still points at this event (a newer // request may have been queued for the same stack after this one started). - if (job.Kind == JobKind.Backup - && _queuedBackupByStack.TryGetValue(job.StackId, out var current) + if (job.Kind == JobKind.Backup && job.StackId is { } backupStack + && _queuedBackupByStack.TryGetValue(backupStack, out var current) && current == job.EventId) - _queuedBackupByStack.Remove(job.StackId); - if (job.Kind == JobKind.Restore) - _queuedRestoreStacks.Remove(job.StackId); + _queuedBackupByStack.Remove(backupStack); + if (job.Kind == JobKind.Restore && job.StackId is { } restoreStack) + _queuedRestoreStacks.Remove(restoreStack); + if (job.Kind == JobKind.InstanceBackup && _queuedInstanceEventId == job.EventId) + _queuedInstanceEventId = null; + if (job.Kind == JobKind.BundleExport && _queuedBundleEventId == job.EventId) + _queuedBundleEventId = null; _runningStackId = job.StackId; } try { - if (job.Kind == JobKind.Backup) + if (job.Kind == JobKind.BundleExport) + await bundleService.ExecuteExportAsync(job.EventId, stoppingToken); + else if (job.Kind == JobKind.InstanceBackup) + await instanceBackupService.ExecuteInstanceBackupAsync(job.EventId, stoppingToken); + else if (job.Kind == JobKind.Backup) await backupService.ExecuteBackupAsync(job.EventId, stoppingToken); else await backupService.ExecuteRestoreAsync(job.EventId, job.FileName!, stoppingToken); @@ -187,7 +251,9 @@ private async Task NotifyChainAsync(int eventId, CancellationToken ct) { } } - private int CreateEvent(int stackId, string triggeredBy) { + /// The stack the run belongs to, or null for an instance self-backup. + /// What to record on the event — see . + private int CreateEvent(int? stackId, string triggeredBy) { using var scope = scopeFactory.CreateScope(); var db = scope.ServiceProvider.GetRequiredService(); var evt = new BackupEvent { diff --git a/src/Watchtower.Application/Services/BackupRetentionRunner.cs b/src/Watchtower.Application/Services/BackupRetentionRunner.cs new file mode 100644 index 0000000..25eac4d --- /dev/null +++ b/src/Watchtower.Application/Services/BackupRetentionRunner.cs @@ -0,0 +1,56 @@ +using Microsoft.Extensions.Logging; +using Watchtower.Application.Config; + +namespace Watchtower.Application.Services; + +/// +/// Prunes one remote directory to the configured retention after a successful run (ADR-0016 §4), and +/// records what it deleted. Shared by the stack runs and the instance self-backup (ADR-0027), so both +/// prune by the same rule and leave the same trail. +/// +/// +/// Never allowed to fail the run that called it: the archive is already on the storage by this point, +/// so an unreachable prune is a warning on the run and a failed audit row, and the next successful run +/// tries again. decides — it only ever considers names +/// that parse as Watchtower archives, and never the newest one. +/// +public sealed class BackupRetentionRunner(AuditLog audit, ILogger logger) { + /// How many deleted names one audit row lists before it is elided. + private const int AuditListLimit = 10; + + /// Applies the retention policy to . + /// The storage the run uploaded to, already open. + /// The provider-relative directory to prune. + /// The options the run operated under. + /// Receives operator-facing lines, WARNING: prefix included. + /// The run's token. + public async Task ApplyAsync( + IBackupStorage storage, string directory, BackupOptions backup, Action log, CancellationToken ct) { + if (backup.RetentionDays <= 0 && backup.RetentionMaxCount <= 0) return; + try { + var names = (await storage.ListFilesAsync(directory, ct)).Select(f => f.Name).ToList(); + var deletions = BackupRetention.SelectDeletions( + names, DateTimeOffset.UtcNow, backup.RetentionDays, backup.RetentionMaxCount); + foreach (var name in deletions) { + await storage.DeleteFileAsync($"{directory}/{name}", ct); + log($"Retention: deleted {name}"); + } + if (deletions.Count > 0) { + // A retention change can prune a large backlog at once — cap the listing so one + // pathological pass cannot bloat the audit row. + var listed = string.Join(", ", deletions.Take(AuditListLimit)); + await audit.RecordAsync(BackupService.AuditCategory, "retention.prune", directory, + $"{BackupService.RetentionSummary(backup)} · deleted {deletions.Count} archive(s): {listed}" + + (deletions.Count > AuditListLimit ? ", …" : ""), + ct: CancellationToken.None); + } + } catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) { + // The backup itself succeeded — an unreachable prune retries on the next run. + log($"WARNING: retention pruning failed: {ex.Message}"); + await audit.RecordAsync(BackupService.AuditCategory, "retention.prune", directory, + BackupService.RetentionSummary(backup), success: false, error: ex.Message, + ct: CancellationToken.None); + logger.LogWarning(ex, "Retention pruning failed for {Directory}", directory); + } + } +} diff --git a/src/Watchtower.Application/Services/BackupService.cs b/src/Watchtower.Application/Services/BackupService.cs index 0a7482e..80efcd7 100644 --- a/src/Watchtower.Application/Services/BackupService.cs +++ b/src/Watchtower.Application/Services/BackupService.cs @@ -38,6 +38,7 @@ public sealed class BackupService( BackupArchiveService archiveService, PostgresDumpService postgres, BackupStorageFactory storageFactory, + BackupRetentionRunner retention, IServiceScopeFactory scopeFactory, IOptionsMonitor options, AuditLog audit, @@ -49,29 +50,37 @@ public sealed class BackupService( /// The compose label a stack's volumes carry. private const string ComposeProjectLabel = "com.docker.compose.project"; + /// + /// Why a run on this path had no stack to work with. Covers both ways that happens: the stack was + /// deleted while the run waited (the ordinary one), and a stackless instance event reached the stack + /// path at all (ADR-0027 gave a null case, and the queue routes + /// those elsewhere — so this half is a bug rather than a state, and says as much). + /// + private const string StacklessMessage = + "This run has no stack: it was either deleted while the run was queued, or the event belongs to " + + "an instance backup and reached the stack path by mistake."; + /// Runs the backup for an event created by . public async Task ExecuteBackupAsync(int backupEventId, CancellationToken ct) { var output = new StringBuilder(); // Locked: a dependency level is quiesced concurrently, and its tasks all log. void Log(string line) { lock (output) output.AppendLine(line); } - int stackId; string triggeredBy; Stack? stack; using (var scope = scopeFactory.CreateScope()) { var db = scope.ServiceProvider.GetRequiredService(); var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, ct); if (evt is null) return; // stack (and its events) deleted while queued - stackId = evt.StackId; triggeredBy = evt.TriggeredBy; - stack = await LoadStackAsync(db, stackId, ct); + stack = evt.StackId is { } id ? await LoadStackAsync(db, id, ct) : null; evt.Status = "running"; evt.StartedAt = DateTimeOffset.UtcNow; await db.SaveChangesAsync(ct); } if (stack is null) { - await FinishAsync(backupEventId, success: false, "Stack no longer exists.", null, null, ct); + await FinishAsync(backupEventId, success: false, StacklessMessage, null, null, ct); return; } @@ -201,14 +210,14 @@ public async Task ExecuteRestoreAsync(int backupEventId, string fileName, Cancel var db = scope.ServiceProvider.GetRequiredService(); var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, ct); if (evt is null) return; - stack = await LoadStackAsync(db, evt.StackId, ct); + stack = evt.StackId is { } id ? await LoadStackAsync(db, id, ct) : null; evt.Status = "running"; evt.StartedAt = DateTimeOffset.UtcNow; await db.SaveChangesAsync(ct); } if (stack is null) { - await FinishAsync(backupEventId, success: false, "Stack no longer exists.", null, null, ct); + await FinishAsync(backupEventId, success: false, StacklessMessage, null, null, ct); return; } @@ -253,8 +262,8 @@ private async Task RunRestoreAsync( log($"Downloaded {sizeBytes} bytes{(encrypted ? " (encrypted)" : "")}"); // 2. Scan the table of contents and match it against the host's volumes. - var contents = await WithSpoolTarAsync(spoolPath, encrypted, backup, - tar => ReadContentsAsync(tar, encrypted, ct)); + var contents = await BackupArchiveReader.ReadContentsAsync( + spoolPath, Passphrase(encrypted, backup), ct); if (contents.ManifestJson is null) log("Note: the archive carries no manifest (single-volume download?) — proceeding by its directory layout."); @@ -349,10 +358,11 @@ private async Task RunRestoreAsync( try { // 9. Wipe + extract, unless the archive is dumps only. if (plan.Volumes.Count > 0) { - await WithSpoolTarAsync(spoolPath, encrypted, backup, async tar => { - await archiveService.RestoreArchiveAsync(plan.Volumes, tar, backup.HelperImage, ct); - return null; - }); + await BackupArchiveReader.WithArchiveAsync( + spoolPath, Passphrase(encrypted, backup), async tar => { + await archiveService.RestoreArchiveAsync(plan.Volumes, tar, backup.HelperImage, ct); + return null; + }); log("Archive extracted into the volumes."); } @@ -435,52 +445,12 @@ private async Task ReplayDumpsAsync( private static Task ExtractDumpAsync( string spoolPath, bool encrypted, BackupOptions backup, string relativeFile, string destination, CancellationToken ct) => - WithSpoolTarAsync(spoolPath, encrypted, backup, async tar => { - var wanted = $"backup/{relativeFile}"; - await using var reader = new TarReader(tar, leaveOpen: true); - while (await reader.GetNextEntryAsync(cancellationToken: ct) is { } entry) { - if (!string.Equals(entry.Name.TrimStart('.', '/'), wanted, StringComparison.Ordinal)) continue; - if (entry.DataStream is not { } data) break; - await using var file = File.Create(destination); - await data.CopyToAsync(file, ct); - return file.Length; - } - // The table-of-contents scan found it a moment ago, so this means the archive changed - // under us or is damaged — either way, not something to restore from. - throw new InvalidOperationException( - $"The archive does not contain '{wanted}', although its table of contents lists it."); - }); + BackupArchiveReader.ExtractAsync( + spoolPath, Passphrase(encrypted, backup), relativeFile, destination, ct); - /// - /// Runs over the spool opened as an uncompressed tar stream - /// (file → optional decrypt → gunzip), disposing every layer afterwards — including the file - /// handle, which the delete in the caller's finally depends on. - /// - private static async Task WithSpoolTarAsync( - string spoolPath, bool encrypted, BackupOptions backup, Func> action) { - await using var file = File.OpenRead(spoolPath); - var inner = encrypted - ? BackupEncryption.CreateDecryptingStream(file, backup.EncryptionPassphrase!) - : file; - try { - await using var tar = new GZipStream(inner, CompressionMode.Decompress, leaveOpen: true); - return await action(tar); - } finally { - if (!ReferenceEquals(inner, file)) await inner.DisposeAsync(); - } - } - - /// Scans the tar, translating decode failures on encrypted archives into a passphrase hint. - private static async Task ReadContentsAsync( - Stream tar, bool encrypted, CancellationToken ct) { - try { - return await BackupArchiveInspector.InspectAsync(tar, ct); - } catch (Exception ex) when (encrypted - && ex is InvalidDataException or System.Security.Cryptography.CryptographicException) { - throw new InvalidOperationException( - $"Could not read the encrypted archive — is the encryption passphrase the one it was written with? ({ex.Message})"); - } - } + /// The passphrase this archive was written with, or null when it was not encrypted. + private static string? Passphrase(bool encrypted, BackupOptions backup) => + encrypted ? backup.EncryptionPassphrase : null; /// Why a container that mounts a volume being restored was nevertheless left running. private static string RestoreKeepReason(BackupKeepReason reason) => reason switch { @@ -605,7 +575,7 @@ await storage.UploadAsync(relativePath, async (dest, uploadCt) => { await read.CopyToAsync(dest, uploadCt); }, ct); - await ApplyRetentionAsync(storage, directory, backup, log, ct); + await retention.ApplyAsync(storage, directory, backup, log, ct); return new RunResult( relativePath, sizeBytes, volumes.Count, stopped.StoppedCount, plan.Excluded.Count, dumps.Count, stopped.PausedCount, directory); @@ -1009,35 +979,6 @@ await audit.RecordAsync(AuditCategory, "reconcile.unpause", return unpaused.Count; } - private async Task ApplyRetentionAsync( - IBackupStorage storage, string directory, BackupOptions backup, Action log, CancellationToken ct) { - if (backup.RetentionDays <= 0 && backup.RetentionMaxCount <= 0) return; - try { - var names = (await storage.ListFilesAsync(directory, ct)).Select(f => f.Name).ToList(); - var deletions = BackupRetention.SelectDeletions( - names, DateTimeOffset.UtcNow, backup.RetentionDays, backup.RetentionMaxCount); - foreach (var name in deletions) { - await storage.DeleteFileAsync($"{directory}/{name}", ct); - log($"Retention: deleted {name}"); - } - if (deletions.Count > 0) { - // A retention change can prune a large backlog at once — cap the listing so one - // pathological pass cannot bloat the audit row. - var listed = string.Join(", ", deletions.Take(10)); - await audit.RecordAsync(AuditCategory, "retention.prune", directory, - $"{RetentionSummary(backup)} · deleted {deletions.Count} archive(s): {listed}" - + (deletions.Count > 10 ? ", …" : ""), - ct: CancellationToken.None); - } - } catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) { - // The backup itself succeeded — an unreachable prune retries on the next run. - log($"WARNING: retention pruning failed: {ex.Message}"); - await audit.RecordAsync(AuditCategory, "retention.prune", directory, RetentionSummary(backup), - success: false, error: ex.Message, ct: CancellationToken.None); - logger.LogWarning(ex, "Retention pruning failed for {Directory}", directory); - } - } - /// /// The archive's self-description, written to backup/backup-manifest.json. /// @@ -1099,8 +1040,11 @@ internal static string BuildManifest( /// internal const int ManifestFormatVersion = 3; - /// One entry of the manifest's dumps array. - private static JsonNode DumpNode(BackupDumpEntry dump) => new JsonObject { + /// + /// One entry of the manifest's dumps array. Shared with the instance manifest (ADR-0027), so + /// a reader that can parse a stack archive's dumps can parse Watchtower's own. + /// + internal static JsonNode DumpNode(BackupDumpEntry dump) => new JsonObject { ["service"] = dump.Service, ["engine"] = dump.Engine.ToString().ToLowerInvariant(), ["file"] = dump.File, diff --git a/src/Watchtower.Application/Services/BackupTriggers.cs b/src/Watchtower.Application/Services/BackupTriggers.cs index cd464bf..15f6b82 100644 --- a/src/Watchtower.Application/Services/BackupTriggers.cs +++ b/src/Watchtower.Application/Services/BackupTriggers.cs @@ -16,6 +16,12 @@ public static class BackupTriggers { /// A restore run, which shares the queue and the event table with backups. public const string Restore = "restore"; + /// + /// A full backup bundle export (ADR-0027 §4), which shares the queue and the event table too — it + /// takes a dump and downloads every stack's newest archive. + /// + public const string BundleExport = "bundle-export"; + /// templates.backupAll fanned a backup out to every tenant of a template. public const string TemplateAll = "template-backup-all"; diff --git a/src/Watchtower.Application/Services/InstanceBackupService.cs b/src/Watchtower.Application/Services/InstanceBackupService.cs new file mode 100644 index 0000000..a39c744 --- /dev/null +++ b/src/Watchtower.Application/Services/InstanceBackupService.cs @@ -0,0 +1,252 @@ +using System.IO.Compression; +using System.Text; +using System.Text.Json.Nodes; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// What one finished instance archive is, for the caller that has to name or ship it. +/// Provider-relative path the archive was uploaded to. +/// The archive's file name alone. +/// The provider-relative directory it lives in. +/// Size of the uploaded archive, after compression and encryption. +/// When the run started — the timestamp in the file name. +/// The databases the dump covers. +public sealed record InstanceArchiveResult( + string RelativePath, string FileName, string Directory, long SizeBytes, + DateTimeOffset TakenAt, IReadOnlyList Databases); + +/// +/// Backs up Watchtower's own PostgreSQL (ADR-0027): a logical pg_dumpall of the database every +/// piece of Watchtower state lives in since ADR-0024, wrapped in the same archive format, encryption and +/// storage the stack backups use, and written to beside +/// them. One storage folder per instance therefore holds everything a rebuild needs. +/// +/// +/// +/// Nothing is stopped or paused. pg_dumpall is consistent by construction, so Watchtower keeps +/// serving through its own backup — which it must, since it is the thing running the backup. The archive +/// carries no volumes at all: since ADR-0024 the container's /data holds nothing Watchtower needs, +/// and the dump is the whole of the state. +/// +/// +/// Encryption is required here, unlike for a stack. pg_dumpall writes every database +/// role's password hash into the SQL, and the tables it carries include the data-protection key ring, +/// the identity signing key and every certificate's private key. An unencrypted copy of that on a backup +/// target is a worse outcome than no backup at all, so a run without a passphrase is refused rather than +/// quietly downgraded. +/// +/// +/// Singleton driven by 's worker, reaching the scoped DbContext through +/// (ADR-0004) — the same shape as , whose +/// event shell, spool-then-upload ordering and audit vocabulary it deliberately mirrors. +/// +/// +/// Not sealed, and virtual, for the reason 's +/// enqueues are: the bundle export composes this service, and a test of what a bundle contains +/// should not need a Docker daemon and a live PostgreSQL to produce the one archive it wraps. +/// +/// +public class InstanceBackupService( + BackupArchiveService archiveService, + PostgresDumpService postgres, + SelfPostgresLocator locator, + BackupStorageFactory storageFactory, + BackupRetentionRunner retention, + IServiceScopeFactory scopeFactory, + IOptionsMonitor options, + AuditLog audit, + ILogger logger) { + /// How the instance backup names itself in the audit trail and the run log. + internal const string AuditTarget = "watchtower (instance)"; + + /// + /// The formatVersion of an instance manifest. Its own sequence, independent of the stack + /// manifest's: the two describe different things and will not move together. + /// + internal const int ManifestFormatVersion = 1; + + /// The kind an instance manifest declares, so a reader can tell the two apart. + internal const string ManifestKind = "watchtower-instance"; + + /// Runs the self-backup for an event created by . + /// The stackless event tracking this run. + /// The worker's token. + public async Task ExecuteInstanceBackupAsync(int backupEventId, CancellationToken ct) { + var output = new StringBuilder(); + void Log(string line) { lock (output) output.AppendLine(line); } + + string triggeredBy; + using (var scope = scopeFactory.CreateScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, ct); + if (evt is null) return; + triggeredBy = evt.TriggeredBy; + evt.Status = BackupStatuses.Running; + evt.StartedAt = DateTimeOffset.UtcNow; + await db.SaveChangesAsync(ct); + } + + var backup = options.CurrentValue.Backup; + try { + var result = await RunAsync(backup, Log, ct); + await FinishAsync(backupEventId, success: true, output.ToString(), result.RelativePath, result.SizeBytes); + await audit.RecordAsync(BackupService.AuditCategory, "run", AuditTarget, + $"{Summary(triggeredBy, backup)} · {result.Databases.Count} database(s)" + + $", {result.SizeBytes} bytes → {result.RelativePath}", + ct: CancellationToken.None); + logger.LogInformation( + "Instance backup completed: {RemotePath} ({SizeBytes} bytes)", result.RelativePath, result.SizeBytes); + } catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) { + Log($"FAILED: {ex.Message}"); + await FinishAsync(backupEventId, success: false, output.ToString(), remotePath: null, sizeBytes: null); + await audit.RecordAsync(BackupService.AuditCategory, "run", AuditTarget, Summary(triggeredBy, backup), + success: false, error: ex.Message, ct: CancellationToken.None); + logger.LogWarning(ex, "Instance backup failed"); + } + } + + /// + /// Takes the dump, builds the archive, uploads it and prunes — the whole run, without the event + /// bookkeeping. Also the bundle export's first step (ADR-0027 stage 2), which needs the archive but + /// writes its own event. + /// + /// The options the run operates under. + /// Receives operator-facing lines, WARNING: prefix included. + /// The run's token. + /// No passphrase, or the database is not reachable as a container. + public virtual async Task RunAsync( + BackupOptions backup, Action log, CancellationToken ct) { + if (string.IsNullOrEmpty(backup.EncryptionPassphrase)) + throw new InvalidOperationException( + "Backing up Watchtower itself needs an encryption passphrase: the dump carries every " + + "database role's password hash, the data-protection key ring and every certificate's " + + "private key. Set one under Settings → Backups first."); + + var takenAt = DateTimeOffset.UtcNow; + var target = await locator.LocateAsync(log, ct); + var dumpTarget = target.ToDumpTarget(); + + // Proven before anything is written: a database we cannot reach has to fail the run here. + var connection = await postgres.PreflightAsync(dumpTarget, log, ct); + + var instance = backup.ResolveInstanceName(); + var directory = BackupNaming.InstanceDirectory(instance); + var fileName = BackupNaming.FileName(BackupNaming.InstanceFileStem, takenAt, encrypted: true); + var relativePath = $"{directory}/{fileName}"; + + var spoolPath = Path.Combine(Path.GetTempPath(), $"watchtower-{fileName}.spool"); + var dumpSpool = Path.Combine(Path.GetTempPath(), $"watchtower-dump-{Guid.NewGuid():N}.sql"); + try { + var dumped = await postgres.DumpAsync(dumpTarget, connection, dumpSpool, log, ct); + var file = $"{PostgresDumpService.DumpDirectory}/{SelfPostgresLocator.ServiceName}.sql"; + var dump = new BackupService.BackupDumpEntry( + SelfPostgresLocator.ServiceName, DumpEngine.Postgres, file, target.Image, connection.User, + target.ContainerName, Volumes: [], dumped.Databases, dumped.SizeBytes); + + var manifest = await BuildManifestAsync(instance, target, takenAt, dump, ct); + await using (var spool = File.Create(spoolPath)) { + var sink = BackupEncryption.CreateEncryptingStream(spool, backup.EncryptionPassphrase!); + try { + await using (var gzip = new GZipStream(sink, CompressionLevel.Optimal, leaveOpen: true)) + // No volumes: the archive is the dump and the manifest. WriteArchiveAsync writes + // the `backup/` directory entry itself, which is what makes that legal. + await archiveService.WriteArchiveAsync( + [], manifest, [new BackupExtraFile(file, dumpSpool)], gzip, backup.HelperImage, ct); + } finally { + await sink.DisposeAsync(); // flushes the final cipher block + } + } + + var sizeBytes = new FileInfo(spoolPath).Length; + log($"Snapshot complete: {sizeBytes} bytes (encrypted)"); + + using var storage = storageFactory.Create(backup); + log($"Uploading to {storage.Description}: {relativePath}"); + await storage.UploadAsync(relativePath, async (dest, uploadCt) => { + await using var read = File.OpenRead(spoolPath); + await read.CopyToAsync(dest, uploadCt); + }, ct); + + await retention.ApplyAsync(storage, directory, backup, log, ct); + return new InstanceArchiveResult( + relativePath, fileName, directory, sizeBytes, takenAt, dumped.Databases); + } finally { + foreach (var path in new[] { dumpSpool, spoolPath }) { + try { + if (File.Exists(path)) File.Delete(path); + } catch (Exception ex) { + logger.LogWarning(ex, "Failed to delete instance backup spool file {SpoolPath}", path); + } + } + } + } + + /// + /// The instance archive's self-description, written to backup/backup-manifest.json — the same + /// path a stack archive uses, with kind telling a reader which of the two it is holding. + /// + /// + /// appVersion and lastMigrationId are what make the archive restorable rather than + /// merely readable: schema migrations only roll forward, so a restore has to be able to refuse a dump + /// this binary has never known a schema for (). + /// + internal async Task BuildManifestAsync( + string instance, SelfPostgresTarget target, DateTimeOffset takenAt, + BackupService.BackupDumpEntry dump, CancellationToken ct) { + string? lastMigration; + using (var scope = scopeFactory.CreateScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + lastMigration = await InstanceVersion.LastMigrationAsync(db, ct); + } + return BuildManifest(instance, target, takenAt, dump, lastMigration); + } + + /// The manifest as a pure function of the run's facts, so its shape is testable on its own. + internal static string BuildManifest( + string instance, SelfPostgresTarget target, DateTimeOffset takenAt, + BackupService.BackupDumpEntry dump, string? lastMigrationId) { + var manifest = new JsonObject { + ["formatVersion"] = ManifestFormatVersion, + ["kind"] = ManifestKind, + ["tool"] = "watchtower", + ["instance"] = instance, + ["appVersion"] = InstanceVersion.App, + ["lastMigrationId"] = lastMigrationId, + ["database"] = target.Database, + ["createdAtUtc"] = takenAt.UtcDateTime.ToString("O"), + ["encrypted"] = true, + ["dumps"] = new JsonArray(BackupService.DumpNode(dump)), + }; + return manifest.ToJsonString(); + } + + /// The effective settings a run operated under, for its audit row. Never includes secrets. + internal static string Summary(string trigger, BackupOptions backup) { + var provider = backup.ResolveProvider() == BackupProviderKind.Local ? "local" : "sftp"; + return $"{trigger} · {provider} · encrypted · {BackupService.RetentionSummary(backup)}"; + } + + private async Task FinishAsync( + int backupEventId, bool success, string output, string? remotePath, long? sizeBytes) { + // Not the caller's token: the terminal state must be written even when the run was cancelled. + using var scope = scopeFactory.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var evt = await db.BackupEvents.FirstOrDefaultAsync(e => e.Id == backupEventId, CancellationToken.None); + if (evt is null) return; + evt.Status = success ? BackupStatuses.Success : BackupStatuses.Failed; + // PostgreSQL text cannot hold NUL (22021), and the log may carry one — exec stderr is raw + // process output. An unsaveable outcome would leave the event stuck as "running" forever. + evt.Output = output.Replace("\0", ""); + evt.RemotePath = remotePath; + evt.SizeBytes = sizeBytes; + evt.FinishedAt = DateTimeOffset.UtcNow; + await db.SaveChangesAsync(CancellationToken.None); + } +} diff --git a/src/Watchtower.Application/Services/InstanceRestoreModels.cs b/src/Watchtower.Application/Services/InstanceRestoreModels.cs new file mode 100644 index 0000000..d58bee1 --- /dev/null +++ b/src/Watchtower.Application/Services/InstanceRestoreModels.cs @@ -0,0 +1,69 @@ +using System.Text.Json.Serialization; + +namespace Watchtower.Application.Services; + +/// Why a staged bundle cannot be restored into this instance, or a caveat about doing so. +/// A stable key the UI can branch on, e.g. key-protection-secret. +/// The operator-facing sentence, which always names what to do about it. +public sealed record RestoreFinding(string Code, string Message); + +/// +/// What a staged bundle is, and whether this instance can restore it (ADR-0027 §5). Produced before +/// anything is touched — the whole point is that the refusals happen while the instance is still intact. +/// +/// False when is non-empty. +/// Reasons the restore is refused outright. +/// Things worth knowing that do not stop it. +/// The instance the bundle came from. +/// The Watchtower build that wrote it. +/// When it was written. +/// How many stacks it carries archives for. +/// How many stacks it describes but has no archive for. +/// Their names, so the confirmation dialog can say what is about to arrive. +public sealed record RestoreValidation( + bool CanRestore, + IReadOnlyList Blocking, + IReadOnlyList Warnings, + string InstanceName, + string AppVersion, + DateTimeOffset CreatedAtUtc, + int StackCount, + int MissingStackCount, + IReadOnlyList StackNames); + +/// +/// The marker an in-progress restore leaves in Watchtower's own container, so the process that comes +/// back after the coordinator has stopped and started it can say what happened (ADR-0027 §5). +/// +/// +/// It lives in the container's filesystem rather than the database precisely because the database is +/// what is being replaced. The container is stopped and started, never recreated, so the file survives — +/// that is the reason the restore coordinator does not use the self-update coordinator's recreate. +/// +/// +/// A random value also written into the database being replaced. After the restart its absence +/// from the database is the proof that the replay committed: no other event can remove it. +/// +/// When the restore was kicked off. +/// The instance name the bundle came from, for the audit row. +/// The coordinator container, so its exit code and logs can be read back. +/// The stacks the bundle carried, for the recovery checklist. +public sealed record RestoreProgress( + string Nonce, + DateTimeOffset StartedAtUtc, + string SourceInstance, + string? CoordinatorId, + IReadOnlyList StackNames); + +/// How the last restore this instance attempted ended. +[JsonConverter(typeof(JsonStringEnumConverter))] +public enum RestoreOutcome { + /// No restore has been attempted, or the result has been cleared. + None, + + /// The replay committed and this process is running on the restored database. + Succeeded, + + /// The coordinator did not replace the database; the instance is as it was. + Failed, +} diff --git a/src/Watchtower.Application/Services/InstanceRestoreService.cs b/src/Watchtower.Application/Services/InstanceRestoreService.cs new file mode 100644 index 0000000..913bb97 --- /dev/null +++ b/src/Watchtower.Application/Services/InstanceRestoreService.cs @@ -0,0 +1,388 @@ +using System.Formats.Tar; +using System.Security.Cryptography; +using System.Text.Json; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// +/// Takes an uploaded full backup bundle, decides whether this instance can restore it, and — once an +/// admin confirms — replaces this instance's database with the one inside it (ADR-0027 §5). +/// +/// +/// +/// The replay cannot be done by this process. pg_dumpall --clean terminates every session and +/// drops every database, and Watchtower's own connection pool would reconnect straight into the middle +/// of that. So everything that can be done from here is done first — validating, re-uploading +/// the stack archives to storage, pushing the SQL into the database container, writing the marker — and +/// then a sibling coordinator container stops Watchtower, replays, and starts it again. +/// +/// +/// Every refusal happens before any of that. An instance that cannot read the bundle it was given must +/// still be the instance it was. +/// +/// +public sealed class InstanceRestoreService( + DockerEngineClient docker, + PostgresDumpService postgres, + SelfPostgresLocator locator, + BackupStorageFactory storageFactory, + InstanceRestoreStaging staging, + SelfUpdateService selfUpdate, + IServiceScopeFactory scopeFactory, + IOptionsMonitor options, + AuditLog audit, + ILogger logger) { + /// How the restore names itself in the audit trail. + internal const string AuditTarget = "watchtower (instance)"; + + /// + /// Unpacks and checks an uploaded bundle, publishing it as the staged restore when it is at least + /// readable. A bundle with blocking findings is still staged, so the UI can show what is wrong + /// rather than only that something was. + /// + /// The uploaded bundle. + /// Cancellation token. + /// The upload is not a Watchtower bundle at all. + public async Task StageAsync(Stream tar, CancellationToken ct) { + var directory = staging.NewUploadDirectory(); + Dictionary digests; + try { + digests = await InstanceRestoreStaging.ExtractAsync(tar, directory, ct); + } catch { + try { + Directory.Delete(directory, recursive: true); + } catch (Exception ex) { + logger.LogWarning(ex, "Could not delete a partially extracted bundle at {Directory}", directory); + } + throw; + } + + var manifest = ReadJson(directory, BackupBundle.ManifestEntry) + ?? throw new InvalidOperationException( + $"The upload has no {BackupBundle.ManifestEntry} — it is not a Watchtower backup bundle."); + var secrets = ReadJson(directory, BackupBundle.SecretsEntry) + ?? throw new InvalidOperationException( + $"The bundle has no {BackupBundle.SecretsEntry}."); + + var staged = new StagedRestore(directory, manifest, secrets, DateTimeOffset.UtcNow); + staging.Replace(staged); + return await ValidateAsync(staged, digests, ct); + } + + /// Re-checks the staged bundle — the UI reads this on load, when nothing was just uploaded. + public Task ValidateAsync(StagedRestore staged, CancellationToken ct) => + ValidateAsync(staged, digests: null, ct); + + /// + /// SHA-256 by entry name from the extraction, when this call follows one. Null on a re-check, where + /// re-hashing a multi-gigabyte bundle to answer a page load would be the wrong trade — the digests + /// were checked when it arrived. + /// + private async Task ValidateAsync( + StagedRestore staged, Dictionary? digests, CancellationToken ct) { + var manifest = staged.Manifest; + var blocking = new List(); + var warnings = new List(); + + if (manifest.BundleFormatVersion != BackupBundle.FormatVersion) + blocking.Add(new RestoreFinding("bundle-format", + $"This bundle is format version {manifest.BundleFormatVersion}, and this Watchtower reads " + + $"version {BackupBundle.FormatVersion}.")); + + // Migrations only roll forward, so the question is not "which version is newer" but "does this + // binary know that schema" — which is exact. + using (var scope = scopeFactory.CreateScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + if (!InstanceVersion.Knows(db, manifest.LastMigrationId)) + blocking.Add(new RestoreFinding("newer-schema", + $"The bundle was written by Watchtower {manifest.AppVersion}, whose database schema " + + "this build does not know. Update this Watchtower to that version or newer, then " + + "restore — a database only ever migrates forward.")); + } + + // The sharpest edge in the whole feature: the stored certificates, ACME account key and signing + // key are AES-GCM under this secret, and it cannot be changed at runtime. + var current = options.CurrentValue.Auth.KeyProtectionSecret; + var fromBundle = staged.Secrets.KeyProtectionSecret; + if (!string.IsNullOrEmpty(fromBundle) && !string.Equals(fromBundle, current, StringComparison.Ordinal)) + blocking.Add(new RestoreFinding("key-protection-secret", + "The bundle's private keys are encrypted with a key-protection secret this instance does " + + "not have, so every certificate and signing key in it would be unreadable here. Set " + + "WATCHTOWER__AUTH__KEYPROTECTIONSECRET to the value in the bundle's secrets.json and " + + "restart Watchtower, then restore — it cannot be changed while running.")); + else if (string.IsNullOrEmpty(fromBundle) && !string.IsNullOrEmpty(current)) + warnings.Add(new RestoreFinding("key-protection-secret-new", + "The bundle's keys are stored unencrypted, and this instance has a key-protection secret " + + "configured. The restored keys stay readable, and anything written afterwards is " + + "encrypted — nothing is lost, but the two halves of the database differ.")); + + // Every archive present and intact, checked before rather than after the database is gone. + foreach (var archive in Archives(manifest)) { + var path = staged.PathOf(archive); + if (!File.Exists(path)) { + blocking.Add(new RestoreFinding("missing-archive", + $"The bundle's manifest lists '{archive.Entry}', which is not in the file. It is " + + "incomplete or was repacked.")); + continue; + } + if (digests is not null + && digests.TryGetValue(archive.Entry, out var actual) + && !string.Equals(actual, archive.Sha256, StringComparison.OrdinalIgnoreCase)) + blocking.Add(new RestoreFinding("corrupt-archive", + $"'{archive.Entry}' does not match the checksum in the manifest — the bundle was " + + "damaged in transit or altered.")); + } + + // Proves the passphrase and the archive together, without touching anything. + if (blocking.Count == 0) { + try { + var contents = await BackupArchiveReader.ReadContentsAsync( + staged.PathOf(manifest.Instance), + manifest.Instance.Encrypted ? staged.Secrets.BackupEncryptionPassphrase : null, ct); + if (contents.DumpFiles.Count == 0) + blocking.Add(new RestoreFinding("no-dump", + "The bundle's Watchtower archive carries no database dump, so there is nothing " + + "to restore from.")); + } catch (Exception ex) when (ex is not OperationCanceledException) { + blocking.Add(new RestoreFinding("unreadable-archive", + $"The bundle's Watchtower archive could not be read: {ex.Message}")); + } + } + + // A restore needs the database as a container to exec into, exactly as a self-backup does. + try { + await locator.LocateAsync(_ => { }, ct); + } catch (Exception ex) when (ex is not OperationCanceledException) { + blocking.Add(new RestoreFinding("no-database-container", ex.Message)); + } + + if (!await IsFreshAsync(ct)) + warnings.Add(new RestoreFinding("not-fresh", + "This Watchtower already manages stacks. Restoring replaces its entire database — the " + + "stacks, accounts and settings it has now are gone, and the containers they deployed " + + "keep running unmanaged.")); + + var stacks = manifest.Stacks; + return new RestoreValidation( + CanRestore: blocking.Count == 0, + Blocking: blocking, + Warnings: warnings, + InstanceName: manifest.InstanceName, + AppVersion: manifest.AppVersion, + CreatedAtUtc: manifest.CreatedAtUtc, + StackCount: stacks.Count(s => s.Archive is not null), + MissingStackCount: stacks.Count(s => s.Archive is null), + StackNames: [.. stacks.Select(s => s.Name)]); + } + + /// + /// Puts everything in place and hands over to the coordinator. Returns once the coordinator has been + /// started — from the caller's point of view Watchtower is about to stop answering. + /// + /// Who asked, for the audit row written before the database goes. + /// Cancellation token. + /// Nothing staged, validation refuses, or the pre-stage fails. + public async Task StartAsync(string? actor, CancellationToken ct) { + var staged = staging.Current + ?? throw new InvalidOperationException("No bundle has been uploaded."); + var validation = await ValidateAsync(staged, ct); + if (!validation.CanRestore) + throw new InvalidOperationException( + "This bundle cannot be restored into this instance: " + + string.Join(" ", validation.Blocking.Select(b => b.Message))); + + var self = await selfUpdate.DetectSelfAsync(ct); + if (self.ContainerId is not { Length: > 0 } selfContainerId) + throw new InvalidOperationException( + "Watchtower is not running as a container on this Docker daemon, so it cannot be stopped " + + "and restarted around the replay. Restore the dump by hand — see docs/backups.md."); + + var manifest = staged.Manifest; + var passphrase = staged.Secrets.BackupEncryptionPassphrase; + + // 1. The stack archives go back to the storage first, at the paths the restored database will + // look for them at, so the recovery checklist has something to restore from afterwards. + var backup = options.CurrentValue.Backup; + using (var storage = storageFactory.Create(backup)) { + foreach (var stack in manifest.Stacks) { + if (stack.Archive is not { } archive) continue; + var source = staged.PathOf(archive); + await storage.UploadAsync(archive.StoragePath, async (destination, token) => { + await using var file = File.OpenRead(source); + await file.CopyToAsync(destination, token); + }, ct); + } + } + + // 2. The SQL is pushed into the database container now, while this process still has a working + // Docker client and the archive's passphrase — the coordinator has neither. + var target = await locator.LocateAsync(_ => { }, ct); + var connection = await postgres.PreflightAsync(target.ToDumpTarget(), _ => { }, ct); + var sqlSpool = Path.Combine(Path.GetTempPath(), $"watchtower-restore-{Guid.NewGuid():N}.sql"); + try { + var dumpFile = await ResolveDumpFileAsync(staged, passphrase, ct); + await BackupArchiveReader.ExtractAsync( + staged.PathOf(manifest.Instance), + manifest.Instance.Encrypted ? passphrase : null, dumpFile, sqlSpool, ct); + await PushSqlAsync(target.ContainerId, sqlSpool, ct); + } finally { + try { + if (File.Exists(sqlSpool)) File.Delete(sqlSpool); + } catch (Exception ex) { + logger.LogWarning(ex, "Failed to delete the restore SQL spool {SpoolPath}", sqlSpool); + } + } + + // 3. The nonce goes into the database that is about to be replaced. Its absence afterwards is + // what proves the replay committed; nothing else can remove it. + var nonce = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16)); + using (var scope = scopeFactory.CreateScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + await db.Database.ExecuteSqlRawAsync( + """ + INSERT INTO elarion_settings (kind, owner, key, value, version, updated_at) + VALUES ('global', '', {0}, {1}, 1, now()) + ON CONFLICT (kind, owner, key) DO UPDATE SET value = EXCLUDED.value + """, + [WatchtowerSettingPaths.RestorePendingNonce, nonce], ct); + } + + await staging.WriteProgressAsync( + new RestoreProgress( + nonce, DateTimeOffset.UtcNow, manifest.InstanceName, CoordinatorId: null, + [.. manifest.Stacks.Select(s => s.Name)]), + ct); + + // Written before the coordinator starts: after this point the database that would hold the row + // is replaced, so an audit row written later would be written into a database that never saw + // the decision. + await audit.RecordAsync( + BackupService.AuditCategory, "instance.restore", AuditTarget, + $"restoring from a bundle taken from '{manifest.InstanceName}' ({manifest.AppVersion}, " + + $"{manifest.CreatedAtUtc:u}) — {validation.StackCount} stack archive(s)", + actor: actor, ct: CancellationToken.None); + + // 4. Hand over. From here the coordinator owns the outcome. + var coordinatorId = await SpawnCoordinatorAsync( + self.ImageName ?? throw new InvalidOperationException( + "Watchtower's own image could not be determined, so no coordinator can be started from it."), + selfContainerId, target.ContainerId, connection, ct); + + await staging.WriteProgressAsync( + new RestoreProgress( + nonce, DateTimeOffset.UtcNow, manifest.InstanceName, coordinatorId, + [.. manifest.Stacks.Select(s => s.Name)]), + ct); + logger.LogWarning( + "Instance restore handed to coordinator {CoordinatorId}; this process will be stopped shortly", + coordinatorId); + } + + /// The dump's path inside the archive, from the archive's own manifest. + private static async Task ResolveDumpFileAsync( + StagedRestore staged, string? passphrase, CancellationToken ct) { + var contents = await BackupArchiveReader.ReadContentsAsync( + staged.PathOf(staged.Manifest.Instance), + staged.Manifest.Instance.Encrypted ? passphrase : null, ct); + return contents.DumpFiles.FirstOrDefault() + ?? throw new InvalidOperationException( + "The bundle's Watchtower archive carries no database dump."); + } + + /// + /// Copies the SQL into the database container at , + /// 0600 — it carries every role's password hash. + /// + private async Task PushSqlAsync(string containerId, string sqlPath, CancellationToken ct) { + var directory = Path.GetDirectoryName(InstanceRestoreStaging.RemoteSqlPath)!.Replace('\\', '/'); + var name = Path.GetFileName(InstanceRestoreStaging.RemoteSqlPath); + // PutContainerArchive will not create the parent, so it is made first. + var mkdir = await docker.ExecAsync(containerId, ["mkdir", "-p", directory], ct: ct); + if (!mkdir.Success) + throw new InvalidOperationException( + $"Could not create {directory} inside the database container " + + $"(exit code {mkdir.ExitCode}): {PostgresDumpService.Tail(mkdir.Stderr)}"); + + await docker.PutContainerArchiveAsync(containerId, directory, async (stream, token) => { + await using var writer = new TarWriter(stream, TarEntryFormat.Pax, leaveOpen: true); + await using var content = File.OpenRead(sqlPath); + await writer.WriteEntryAsync( + new PaxTarEntry(TarEntryType.RegularFile, name) { + Mode = UnixFileMode.UserRead | UnixFileMode.UserWrite, + DataStream = content, + }, token); + }, ct); + } + + /// + /// Starts the sibling container that does the replay. Same shape as the self-update coordinator: + /// Watchtower's own image so it runs the code it was built with, the Docker socket, no network, and + /// the process's supplementary groups () so it can use the + /// socket at all — the third consumer of that rule, after the self-update coordinator and the CI + /// runner containers. + /// + private async Task SpawnCoordinatorAsync( + string imageName, string selfContainerId, string postgresContainerId, + PostgresConnection connection, CancellationToken ct) { + var name = $"watchtower-restore-{DateTimeOffset.UtcNow.ToUnixTimeSeconds()}"; + string[] command = [ + "--restore-self", + "--container-id", selfContainerId, + "--postgres-id", postgresContainerId, + "--sql", InstanceRestoreStaging.RemoteSqlPath, + "--db-user", connection.User, + .. connection.ExecUser is { Length: > 0 } execUser ? (string[])["--db-exec-user", execUser] : [], + .. connection.Databases.SelectMany(d => new[] { "--expect-db", d }), + ]; + List env = [$"WATCHTOWER__DOCKERAPIVERSION={options.CurrentValue.DockerApiVersion}"]; + // Only when there is one. It is visible in `docker inspect` of the coordinator, which is + // accepted: reading that needs the Docker socket, and anyone holding it owns the host anyway. + if (connection.Password is { Length: > 0 } password) + env.Add($"{RestoreCoordinatorEnvironment.PostgresPassword}={password}"); + + var coordinatorId = await docker.CreateContainerAsync(new DockerCreateContainerBody { + Image = imageName, + Cmd = command, + Env = [.. env], + HostConfig = new DockerCreateHostConfig { + Binds = ["/var/run/docker.sock:/var/run/docker.sock"], + NetworkMode = "none", + GroupAdd = HostSupplementaryGroups.Current(), + }, + }, name, ct); + await docker.StartContainerAsync(coordinatorId, ct); + return coordinatorId; + } + + /// + /// Whether this looks like a Watchtower nobody has used yet: no stacks, no deploys, and one account. + /// A heuristic for a warning, never for a permission — the restore is gated on being an admin. + /// + public async Task IsFreshAsync(CancellationToken ct) { + using var scope = scopeFactory.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + return !await db.Stacks.AnyAsync(ct) + && !await db.DeployEvents.AnyAsync(ct) + && await db.Users.CountAsync(u => u.RealmId == Entities.Realm.SystemRealmId, ct) <= 1; + } + + /// Every archive the manifest promises, instance and stacks alike. + private static IEnumerable Archives(BundleManifest manifest) => + [manifest.Instance, .. manifest.Stacks.Select(s => s.Archive).OfType()]; + + private static T? ReadJson(string directory, string entry) { + var path = Path.Combine(directory, entry); + if (!File.Exists(path)) return default; + try { + return JsonSerializer.Deserialize(File.ReadAllText(path), BackupBundle.JsonOptions); + } catch (JsonException ex) { + throw new InvalidOperationException($"The bundle's {entry} could not be read: {ex.Message}"); + } + } +} diff --git a/src/Watchtower.Application/Services/InstanceRestoreStaging.cs b/src/Watchtower.Application/Services/InstanceRestoreStaging.cs new file mode 100644 index 0000000..176e260 --- /dev/null +++ b/src/Watchtower.Application/Services/InstanceRestoreStaging.cs @@ -0,0 +1,171 @@ +using System.Formats.Tar; +using System.Security.Cryptography; +using System.Text.Json; +using Microsoft.Extensions.Logging; + +namespace Watchtower.Application.Services; + +/// A bundle unpacked on disk, waiting for its restore to be confirmed. +/// Where its members were extracted. +/// Its manifest. +/// Its secrets file. +/// When it arrived. +public sealed record StagedRestore( + string Directory, BundleManifest Manifest, BundleSecrets Secrets, DateTimeOffset UploadedAtUtc) { + /// The host path of one of the bundle's members. + public string PathOf(BundleArchive archive) => + Path.Combine(Directory, archive.Entry.Replace('/', Path.DirectorySeparatorChar)); +} + +/// +/// Holds the one uploaded bundle awaiting a restore, and the marker file that outlives the process a +/// restore stops (ADR-0027 §5). +/// +/// +/// The directory is fixed rather than random because the completion pass has to find the marker after a +/// restart with no memory of having written it. Watchtower's container is stopped and started by the +/// coordinator, never recreated, so its filesystem is still there. +/// +/// Logger. +/// +/// Where uploads and the marker live. Defaults to the container's temp directory, which is what a +/// deployment wants; a test overrides it so two of them cannot meet in one path. +/// +public sealed class InstanceRestoreStaging( + ILogger logger, string? rootDirectory = null) { + /// The default root: one directory inside Watchtower's own container. + public static string DefaultRootDirectory => Path.Combine(Path.GetTempPath(), "watchtower-restore"); + + /// Where an uploaded bundle is unpacked and the progress marker is kept. + public string RootDirectory { get; } = rootDirectory ?? DefaultRootDirectory; + + /// The marker file naming an in-flight restore. + private string ProgressPath => Path.Combine(RootDirectory, "restore-progress.json"); + + /// Where the SQL is placed inside the database container for the replay. + public const string RemoteSqlPath = "/tmp/watchtower-restore/restore.sql"; + + private readonly Lock _gate = new(); + private StagedRestore? _staged; + + /// The uploaded bundle, or null when there is none (or its directory has gone). + public StagedRestore? Current { + get { + lock (_gate) { + if (_staged is { } staged && !Directory.Exists(staged.Directory)) _staged = null; + return _staged; + } + } + } + + /// Publishes a newly unpacked bundle, discarding whatever it replaces. + public void Replace(StagedRestore staged) { + StagedRestore? previous; + lock (_gate) { + previous = _staged; + _staged = staged; + } + if (previous is not null) DeleteDirectory(previous.Directory); + } + + /// Discards the staged bundle and its files. + public void Clear() { + StagedRestore? previous; + lock (_gate) { + previous = _staged; + _staged = null; + } + if (previous is not null) DeleteDirectory(previous.Directory); + } + + /// A fresh directory for one upload, under . + public string NewUploadDirectory() { + var path = Path.Combine(RootDirectory, $"bundle-{Guid.NewGuid():N}"); + Directory.CreateDirectory(path); + return path; + } + + /// Writes the in-flight marker. Read back by the process that comes up after the restart. + public async Task WriteProgressAsync(RestoreProgress progress, CancellationToken ct) { + Directory.CreateDirectory(RootDirectory); + await File.WriteAllTextAsync( + ProgressPath, JsonSerializer.Serialize(progress, BackupBundle.JsonOptions), ct); + } + + /// The in-flight marker, or null when no restore was under way. + public RestoreProgress? ReadProgress() { + try { + if (!File.Exists(ProgressPath)) return null; + return JsonSerializer.Deserialize( + File.ReadAllText(ProgressPath), BackupBundle.JsonOptions); + } catch (Exception ex) { + // A marker we cannot read is a marker we cannot act on. Reported, not thrown: the instance + // is up, and refusing to finish starting over a stale temp file would be the worse outcome. + logger.LogWarning(ex, "Could not read the restore progress marker at {Path}", ProgressPath); + return null; + } + } + + /// Removes the in-flight marker, once its outcome has been recorded. + public void ClearProgress() { + try { + if (File.Exists(ProgressPath)) File.Delete(ProgressPath); + } catch (Exception ex) { + logger.LogWarning(ex, "Could not delete the restore progress marker at {Path}", ProgressPath); + } + } + + /// + /// Unpacks a bundle tar into a fresh directory, refusing any entry that would escape it. + /// + /// + /// The entry names come from a file an operator uploaded, so ../ and absolute paths are + /// exactly what has to be refused — a tar that writes outside its directory is writing wherever the + /// Watchtower process can write. + /// + /// The uploaded tar. + /// The (already created) directory to unpack into. + /// Cancellation token. + /// The SHA-256 of every extracted member, by entry name. + public static async Task> ExtractAsync( + Stream tar, string directory, CancellationToken ct) { + var digests = new Dictionary(StringComparer.Ordinal); + var root = Path.GetFullPath(directory) + Path.DirectorySeparatorChar; + + await using var reader = new TarReader(tar); + while (await reader.GetNextEntryAsync(cancellationToken: ct) is { } entry) { + if (entry.EntryType is not (TarEntryType.RegularFile or TarEntryType.V7RegularFile)) continue; + + // Only a leading "./" is stripped — some tar writers emit it. Everything else is left as it + // is and put to the containment check below, so a name that tries to escape is *refused* + // rather than quietly rewritten into a benign one. + var name = entry.Name.StartsWith("./", StringComparison.Ordinal) ? entry.Name[2..] : entry.Name; + var destination = Path.GetFullPath( + Path.Combine(directory, name.Replace('/', Path.DirectorySeparatorChar))); + if (!destination.StartsWith(root, StringComparison.Ordinal)) + throw new InvalidOperationException( + $"The bundle contains an entry that would be written outside it ('{entry.Name}'). " + + "It was not produced by Watchtower, or it has been tampered with."); + + Directory.CreateDirectory(Path.GetDirectoryName(destination)!); + if (entry.DataStream is not { } data) continue; + await using (var file = File.Create(destination)) + await data.CopyToAsync(file, ct); + + await using var written = File.OpenRead(destination); + digests[name] = Convert.ToHexStringLower(await SHA256.HashDataAsync(written, ct)); + } + return digests; + } + + /// Deletes everything under , marker included. + public void DeleteAll() => DeleteDirectory(RootDirectory); + + private void DeleteDirectory(string path) { + try { + if (Directory.Exists(path)) Directory.Delete(path, recursive: true); + } catch (Exception ex) { + logger.LogWarning(ex, "Could not delete the restore staging directory {Directory}", path); + } + } +} diff --git a/src/Watchtower.Application/Services/InstanceVersion.cs b/src/Watchtower.Application/Services/InstanceVersion.cs new file mode 100644 index 0000000..8bd1a48 --- /dev/null +++ b/src/Watchtower.Application/Services/InstanceVersion.cs @@ -0,0 +1,52 @@ +using System.Reflection; +using Microsoft.EntityFrameworkCore; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// +/// What build and what schema this instance is running — the two facts an instance archive records so a +/// restore can refuse a backup the target cannot read (ADR-0027). +/// +/// +/// The pair is deliberate. The version string is for the operator ("this bundle came from 1.4.2"); the +/// migration id is what the decision is actually made on, because it is exact: a binary either knows a +/// migration or it does not, where comparing version strings only guesses at what that means. Migrations +/// roll forward only, so an archive whose last migration this binary has never heard of was written by a +/// newer Watchtower and must not be replayed into this one. +/// +public static class InstanceVersion { + /// + /// The informational version of the Watchtower build, e.g. 1.4.2+abc1234. Falls back to the + /// assembly version, and finally to unknown — a manifest key that is never absent is easier to + /// read than one that sometimes is. + /// + public static string App { get; } = Resolve(); + + /// + /// The last migration applied to , or null when the database has none yet. + /// + /// The context to ask. + /// Cancellation token. + public static async Task LastMigrationAsync(WatchtowerDbContext db, CancellationToken ct) => + (await db.Database.GetAppliedMigrationsAsync(ct)).LastOrDefault(); + + /// + /// Whether this binary knows — i.e. whether an archive stamped with it + /// can be replayed here. A null id (an archive from a database with no migrations, or a manifest + /// written before the key existed) is accepted: there is nothing to be newer than. + /// + /// The context whose model carries the known migrations. + /// The migration id recorded in the archive. + public static bool Knows(WatchtowerDbContext db, string? migrationId) => + string.IsNullOrEmpty(migrationId) + || db.Database.GetMigrations().Contains(migrationId, StringComparer.Ordinal); + + private static string Resolve() { + var assembly = typeof(InstanceVersion).Assembly; + var informational = assembly.GetCustomAttribute() + ?.InformationalVersion; + if (!string.IsNullOrWhiteSpace(informational)) return informational; + return assembly.GetName().Version?.ToString() ?? "unknown"; + } +} diff --git a/src/Watchtower.Application/Services/PostgresDumpService.cs b/src/Watchtower.Application/Services/PostgresDumpService.cs index 514d4e4..be73195 100644 --- a/src/Watchtower.Application/Services/PostgresDumpService.cs +++ b/src/Watchtower.Application/Services/PostgresDumpService.cs @@ -376,9 +376,6 @@ public async Task WaitReadyAsync( public async Task ReplayAsync( string containerId, PostgresConnection connection, string service, string sqlPath, IReadOnlyList expectedDatabases, Action log, CancellationToken ct) { - var execEnv = ExecEnv(connection.Password); - await TerminateSessionsAsync(containerId, connection, service, log, ct); - // Sanitized, so a service name can never steer the path the file lands on. var remotePath = $"/tmp/{BackupNaming.Sanitize(service)}.sql"; try { @@ -393,28 +390,87 @@ await writer.WriteEntryAsync( }, token); }, ct); - var replay = await _docker.ExecAsync( - containerId, - ["psql", "-U", connection.User, "-d", "postgres", "-w", "-v", "ON_ERROR_STOP=0", "-f", remotePath], - stdout: null, execEnv, connection.ExecUser, ct); - - // Asked even after a non-zero exit: which databases actually exist is the verdict, and it - // is also the most useful thing to put in the failure message. - var (listing, present) = await ListDatabasesAsync(containerId, connection, ct); - var outcome = PostgresReplayOutcome.Classify( - replay.ExitCode, replay.Stderr, expectedDatabases, listing.Success ? present : []); - if (outcome.Failure is { } failure) - throw new InvalidOperationException($"Replaying the '{service}' dump failed: {failure}"); - _logger.LogInformation( - "Replayed a dump into container {ContainerId}: {DatabaseCount} database(s) present, " - + "{DiagnosticCount} psql diagnostic(s)", - containerId, present.Count, outcome.ErrorLineCount); - return outcome; + return await ReplayRemoteAsync(containerId, connection, service, remotePath, expectedDatabases, log, ct); } finally { await RemoveRemoteFileAsync(containerId, connection, service, remotePath, log); } } + /// + /// Replays a dump that is already inside the container — the second half of + /// , and the whole of what the instance-restore coordinator does (ADR-0027 + /// §5): it is a bare process with the Docker socket and no filesystem in common with the Watchtower + /// that staged the SQL, so it can only ever replay a path, never push one. + /// + /// The database container, already running and ready. + /// What established. + /// The compose service, for the run output. + /// Path of the SQL inside the container. + /// The databases the dump promises; empty skips that check. + /// Receives operator-facing lines, WARNING: prefix included. + /// The run's token. + /// psql failed, or a database did not come back. + public async Task ReplayRemoteAsync( + string containerId, PostgresConnection connection, string service, string remoteSqlPath, + IReadOnlyList expectedDatabases, Action log, CancellationToken ct) { + // Terminated immediately before the script runs rather than before the file is staged, so the + // gap in which something could reconnect and block a DROP DATABASE is as short as it can be. + await TerminateSessionsAsync(containerId, connection, service, log, ct); + + var replay = await _docker.ExecAsync( + containerId, + ["psql", "-U", connection.User, "-d", "postgres", "-w", "-v", "ON_ERROR_STOP=0", "-f", remoteSqlPath], + stdout: null, ExecEnv(connection.Password), connection.ExecUser, ct); + + // Asked even after a non-zero exit: which databases actually exist is the verdict, and it + // is also the most useful thing to put in the failure message. + var (listing, present) = await ListDatabasesAsync(containerId, connection, ct); + var outcome = PostgresReplayOutcome.Classify( + replay.ExitCode, replay.Stderr, expectedDatabases, listing.Success ? present : []); + if (outcome.Failure is { } failure) + throw new InvalidOperationException($"Replaying the '{service}' dump failed: {failure}"); + _logger.LogInformation( + "Replayed a dump into container {ContainerId}: {DatabaseCount} database(s) present, " + + "{DiagnosticCount} psql diagnostic(s)", + containerId, present.Count, outcome.ErrorLineCount); + return outcome; + } + + /// + /// Dumps straight to a file inside the container, for the safety copy the instance-restore + /// coordinator takes before it replaces anything (ADR-0027 §5). The coordinator has nowhere else to + /// put it: it shares no filesystem with the database container or with Watchtower. + /// + /// + /// The role and password travel as PGUSER/PGPASSWORD rather than as arguments, so + /// nothing derived from configuration is interpolated into the shell command — the only strings in + /// it are compile-time constants. + /// + /// The database container. + /// What established. + /// Where to write, inside the container. Its directory is created. + /// Cancellation token. + /// The dump could not be taken. + public async Task DumpToContainerFileAsync( + string containerId, PostgresConnection connection, string remotePath, CancellationToken ct) { + var directory = remotePath[..remotePath.LastIndexOf('/')]; + string[] env = [ + $"PGUSER={connection.User}", + .. connection.Password is { Length: > 0 } password ? (string[])[$"PGPASSWORD={password}"] : [], + $"WT_DUMP_DIR={directory}", + $"WT_DUMP_FILE={remotePath}", + ]; + var result = await _docker.ExecAsync( + containerId, + ["sh", "-c", + "mkdir -p \"$WT_DUMP_DIR\" && umask 077 && " + + "pg_dumpall --clean --if-exists --no-password > \"$WT_DUMP_FILE\""], + stdout: null, env, connection.ExecUser, ct); + if (!result.Success) + throw new InvalidOperationException( + $"pg_dumpall to {remotePath} failed with exit code {result.ExitCode}: {Tail(result.Stderr)}"); + } + /// /// Disconnects everything else from the server so --clean can drop the databases. Never /// fatal on its own: the replay itself reports what actually went wrong. diff --git a/src/Watchtower.Application/Services/RestoreCompletionService.cs b/src/Watchtower.Application/Services/RestoreCompletionService.cs new file mode 100644 index 0000000..1b64f13 --- /dev/null +++ b/src/Watchtower.Application/Services/RestoreCompletionService.cs @@ -0,0 +1,146 @@ +using Elarion.Settings; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Watchtower.Application.Config; +using Watchtower.Application.Persistence; +using Watchtower.Application.Services.Yarp; + +namespace Watchtower.Application.Services; + +/// +/// Works out what happened to an instance restore, on the first start after the coordinator stopped and +/// started this container (ADR-0027 §5) — and repairs the few things a replayed database gets wrong +/// about the present. +/// +/// +/// +/// The verdict comes from the nonce the restore wrote into the database it was about to replace. If it +/// is gone, the replay committed: nothing else knows that value, so nothing else could have removed it. +/// If it is still there, the coordinator never replaced the database and this instance is exactly as it +/// was — which is a failure worth reporting, not a silent no-op. +/// +/// +/// Runs as a hosted service rather than inline in the startup path because it is only ever relevant +/// after a restore, and a restore is rare: an instance that has never had one does one file check. +/// +/// +public sealed class RestoreCompletionService( + InstanceRestoreStaging staging, + DockerEngineClient docker, + ProxyChangeSignal proxySignal, + IServiceScopeFactory scopeFactory, + ILogger logger) : IHostedService { + /// How the last restore ended, for backups.getRestoreStatus. + public RestoreOutcome LastOutcome { get; private set; } = RestoreOutcome.None; + + /// What went wrong, when is a failure. + public string? LastError { get; private set; } + + /// How many lines of the coordinator's output are kept for the audit row. + private const int CoordinatorLogTailLines = 40; + + public async Task StartAsync(CancellationToken ct) { + if (staging.ReadProgress() is not { } progress) return; + try { + await CompleteAsync(progress, ct); + } catch (Exception ex) { + // Never allowed to stop the host coming up: an instance that will not start is strictly + // worse than one whose restore outcome went unrecorded. + logger.LogError(ex, "Could not complete the instance restore recorded in the staging directory"); + staging.ClearProgress(); + } + } + + public Task StopAsync(CancellationToken ct) => Task.CompletedTask; + + private async Task CompleteAsync(RestoreProgress progress, CancellationToken ct) { + await using var scope = scopeFactory.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var settings = scope.ServiceProvider.GetRequiredService(); + var audit = scope.ServiceProvider.GetRequiredService(); + + var pending = await settings.GetStringAsync( + WatchtowerSettingPaths.RestorePendingNonce, SettingsScope.Global, ct); + var replayed = !string.Equals(pending, progress.Nonce, StringComparison.Ordinal); + var coordinatorLog = await ReadCoordinatorLogAsync(progress.CoordinatorId, ct); + + if (!replayed) { + LastOutcome = RestoreOutcome.Failed; + LastError = "The database was not replaced — this instance is running on the database it had."; + logger.LogError( + "The instance restore from '{SourceInstance}' did not complete; the database is unchanged. " + + "Coordinator output:\n{CoordinatorLog}", progress.SourceInstance, coordinatorLog); + await audit.RecordAsync( + BackupService.AuditCategory, "instance.restore", InstanceRestoreService.AuditTarget, + $"restore from '{progress.SourceInstance}' did not complete{Tail(coordinatorLog)}", + success: false, error: LastError, ct: CancellationToken.None); + // Our own litter, in a database that is staying: the marker row means nothing now. + await settings.RemoveAsync( + WatchtowerSettingPaths.RestorePendingNonce, SettingsScope.Global, expectedVersion: null, ct); + // The upload survives, so the operator can look at the log and try again. + staging.ClearProgress(); + return; + } + + LastOutcome = RestoreOutcome.Succeeded; + LastError = null; + logger.LogWarning( + "This instance was restored from a backup of '{SourceInstance}'; its previous database is gone", + progress.SourceInstance); + + // ── What a replayed database is wrong about ────────────────────────── + // The restored rows describe the source instance at the moment of its dump, so anything that + // tracks *this* instance's present has to be corrected before it acts on stale beliefs. + + // The schedule cursors rolled back with everything else. Left alone, every window between the + // dump and now looks missed, and the misfire grace would fire a backup of every stack at once — + // against volumes that have not been redeployed yet. + var now = DateTimeOffset.UtcNow; + await db.Stacks + .Where(s => s.LastScheduledBackupAt == null || s.LastScheduledBackupAt < now) + .ExecuteUpdateAsync(s => s.SetProperty(x => x.LastScheduledBackupAt, now), ct); + await settings.SetStringAsync( + WatchtowerSettingPaths.BackupSelfLastScheduledAt, now.UtcDateTime.ToString("O"), + SettingsScope.Global, expectedVersion: null, ct); + + // The routes table arrived wholesale; the proxy plane has to re-project it rather than keep + // serving what this instance had before. + await proxySignal.BumpAsync("instance restored from a backup bundle", ct); + + // The checklist the operator works through next: redeploy each stack, then restore its volumes. + await StackRevivalState.SeedAsync(settings, progress, db, ct); + + await audit.RecordAsync( + BackupService.AuditCategory, "instance.restore", InstanceRestoreService.AuditTarget, + $"restored from a bundle taken from '{progress.SourceInstance}' " + + $"({progress.StackNames.Count} stack(s) to revive){Tail(coordinatorLog)}", + ct: CancellationToken.None); + + // The bundle has served its purpose, and it holds every secret the source instance had. + staging.Clear(); + staging.ClearProgress(); + } + + /// + /// The coordinator's output, so the audit row can say what it actually did. Best effort — it is a + /// stopped container that may already have been reaped, and its absence must not change the verdict. + /// + private async Task ReadCoordinatorLogAsync(string? coordinatorId, CancellationToken ct) { + if (coordinatorId is not { Length: > 0 }) return null; + try { + var lines = new List(); + await foreach (var line in docker.StreamLogsAsync( + coordinatorId, CoordinatorLogTailLines, follow: false, ct)) + lines.Add(line); + return lines.Count == 0 ? null : string.Join(" | ", lines); + } catch (Exception ex) { + logger.LogDebug(ex, "Could not read the restore coordinator's log"); + return null; + } + } + + private static string Tail(string? coordinatorLog) => + coordinatorLog is { Length: > 0 } ? $" · coordinator: {coordinatorLog}" : ""; +} diff --git a/src/Watchtower.Application/Services/RestoreCoordinatorEnvironment.cs b/src/Watchtower.Application/Services/RestoreCoordinatorEnvironment.cs new file mode 100644 index 0000000..0c9b783 --- /dev/null +++ b/src/Watchtower.Application/Services/RestoreCoordinatorEnvironment.cs @@ -0,0 +1,34 @@ +namespace Watchtower.Application.Services; + +/// +/// The contract between the Watchtower process that starts a restore and the coordinator container that +/// carries it out (ADR-0027 §5). Named in one place so the two halves cannot drift: they are compiled +/// into the same image but never run in the same process. +/// +/// +/// +/// --restore-self +/// --container-id <watchtower container> the container to stop, replay behind, and start again +/// --postgres-id <database container> the container psql is exec'd in +/// --sql <path inside it> the dump, already placed there by the starting process +/// --db-user <role> the role that answered the preflight +/// [--db-exec-user <os user>] the OS user psql must run as, when it is not the default +/// [--expect-db <name>]… the databases the dump promises; the success check +/// +/// The password, when the image needs one, travels as an environment variable on the coordinator's +/// create body — visible in docker inspect, which is accepted because reading it requires the +/// Docker socket, and holding that already means owning the host. +/// +public static class RestoreCoordinatorEnvironment { + /// The CLI flag that puts the process into restore-coordinator mode. + public const string Flag = "--restore-self"; + + /// Environment variable carrying PGPASSWORD to the coordinator. + public const string PostgresPassword = "WATCHTOWER_RESTORE_PGPASSWORD"; + + /// + /// Where the coordinator writes its safety dump inside the database container, before it replaces + /// anything. Replayed back if the restore's own replay fails. + /// + public const string SafetyDumpPath = "/tmp/watchtower-restore/pre-restore.sql"; +} diff --git a/src/Watchtower.Application/Services/SelfPostgresLocator.cs b/src/Watchtower.Application/Services/SelfPostgresLocator.cs new file mode 100644 index 0000000..2e48768 --- /dev/null +++ b/src/Watchtower.Application/Services/SelfPostgresLocator.cs @@ -0,0 +1,244 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Npgsql; +using Watchtower.Application.Config; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// +/// Watchtower's own database, as a container the dump machinery can exec into. +/// +/// Engine id — what every exec of the dump is addressed to. +/// Operator-facing container name, for the run log and the manifest. +/// The image the container runs, recorded in the manifest. +/// The host the connection string names — how the container was recognized. +/// The database the connection string names. +/// The role the connection string connects as. +public sealed record SelfPostgresTarget( + string ContainerId, string ContainerName, string Image, string Host, string Database, string Username) { + /// + /// The dump target for this container. watchtower is its service identity inside the archive, + /// so the SQL always lands at backup/_dumps/watchtower.sql whatever the container is called. + /// No data volume and no mounted volumes: an instance archive carries the dump alone, so there is + /// nothing for the exclusion to exclude it from. + /// + public DumpTarget ToDumpTarget() => new( + ContainerId, ContainerName, SelfPostgresLocator.ServiceName, Image, DumpEngine.Postgres, + DataVolume: null, MountedVolumes: []); +} + +/// +/// Finds the container running Watchtower's own PostgreSQL, so the instance self-backup (ADR-0027) can +/// dump it with the same pg_dumpall machinery a stack's database goes through (ADR-0017). +/// +/// +/// +/// Watchtower cannot register itself as a stack — reserves its own +/// compose project precisely so nothing can — so the container has to be found rather than configured. +/// The search is: the explicit setting if there is one; else the postgres-imaged containers of +/// Watchtower's own compose project; else, for an install that is not under Compose at all, every +/// running postgres-imaged container. In each case the connection string's Host is what picks the +/// winner out of the candidates, since that is the name Watchtower's own connections resolve. +/// +/// +/// Every failure throws with a message an operator can act on. A self-backup that quietly does nothing +/// because the database turned out to be managed, or because the daemon blipped, is worse than one that +/// fails loudly — the whole point of the feature is that the archive is there when the instance is not. +/// +/// +/// Not sealed, and virtual, for the reason the backup queue's enqueues are: a +/// test of what the restore decides should not need a Docker daemon to answer the one question +/// this class asks it. +/// +/// +public class SelfPostgresLocator( + DockerEngineClient docker, + SelfProjectNameProvider selfProjects, + IConfiguration configuration, + IOptionsMonitor options, + ILogger logger) { + /// The compose label a container's project is stamped with. + private const string ComposeProjectLabel = "com.docker.compose.project"; + + /// The compose label a container's service name is stamped with. + private const string ComposeServiceLabel = "com.docker.compose.service"; + + /// The service identity Watchtower's own dump carries inside an archive. + internal const string ServiceName = "watchtower"; + + /// + /// Locates the container, or throws explaining what to do about it. + /// + /// Receives operator-facing lines for the run output. + /// The run's token. + /// + /// No connection string, no candidate container, or more than one candidate and nothing to choose by. + /// + public virtual async Task LocateAsync(Action log, CancellationToken ct) { + var connectionString = WatchtowerConnectionString.Find(configuration) + ?? throw new InvalidOperationException( + "No PostgreSQL connection string is configured, so there is no database to back up. " + + $"Set '{WatchtowerConnectionString.ConfigurationKey}'."); + + NpgsqlConnectionStringBuilder parsed; + try { + parsed = new NpgsqlConnectionStringBuilder(connectionString); + } catch (Exception ex) when (ex is ArgumentException or FormatException) { + throw new InvalidOperationException( + $"Watchtower's connection string could not be parsed ({ex.Message}), so its database " + + "container cannot be identified."); + } + + var host = parsed.Host ?? ""; + var database = string.IsNullOrEmpty(parsed.Database) ? "postgres" : parsed.Database; + var username = parsed.Username ?? "postgres"; + + if (options.CurrentValue.Backup.SelfPostgresContainer is { } configured + && !string.IsNullOrWhiteSpace(configured)) + return await LocateConfiguredAsync(configured.Trim(), host, database, username, log, ct); + + var chosen = Choose(await CandidatesAsync(log, ct), host); + + var target = new SelfPostgresTarget( + chosen.Id, DisplayName(chosen), chosen.Image, host, database, username); + log($"Watchtower's database is container '{target.ContainerName}' ({target.Image}) " + + $"— host '{host}', database '{database}'."); + logger.LogInformation( + "Located Watchtower's own database in container {ContainerId} ({Image})", chosen.Id, chosen.Image); + return target; + } + + /// + /// Resolves the explicitly configured container. An operator who named one gets that one or an + /// error — never a different container the detection happened to like better. + /// + private async Task LocateConfiguredAsync( + string configured, string host, string database, string username, Action log, CancellationToken ct) { + DockerContainerDetails details; + try { + details = await docker.InspectContainerAsync(configured, ct); + } catch (Exception ex) when (ex is not OperationCanceledException) { + throw new InvalidOperationException( + $"The container '{configured}' named in {WatchtowerSettingPaths.BackupSelfPostgresContainer} " + + $"could not be inspected: {ex.Message}"); + } + if (!string.Equals(details.State?.Status, "running", StringComparison.OrdinalIgnoreCase)) + throw new InvalidOperationException( + $"The container '{configured}' named in {WatchtowerSettingPaths.BackupSelfPostgresContainer} " + + $"is {details.State?.Status ?? "in an unknown state"} — pg_dumpall needs a live server."); + + var name = configured; + if (!DatabaseDumpTargets.IsPostgresImage(details.Config.Image)) + // Not fatal: the list of known repositories is not the list of images that run a server, and + // an operator who named this container meant it. The preflight proves it either way. + log($"WARNING: '{name}' runs {details.Config.Image}, which is not a recognized PostgreSQL " + + "image — trying it anyway, since it is the configured container."); + + log($"Watchtower's database is container '{name}' ({details.Config.Image}), from " + + $"{WatchtowerSettingPaths.BackupSelfPostgresContainer} — database '{database}'."); + return new SelfPostgresTarget(details.Id, name, details.Config.Image, host, database, username); + } + + /// + /// The running PostgreSQL containers worth considering: Watchtower's own compose project when it has + /// one, else every running container on the daemon (a docker run install has no project to + /// narrow by, and narrowing by nothing is better than finding nothing). + /// + private async Task> CandidatesAsync(Action log, CancellationToken ct) { + var project = await selfProjects.GetAsync(ct); + var containers = project is { Length: > 0 } + ? await docker.ListContainersByLabelsAsync([$"{ComposeProjectLabel}={project}"], ct) + : await docker.ListContainersAsync(ct); + if (project is not { Length: > 0 }) + log("Watchtower is not running under a Compose project, so every running PostgreSQL " + + "container on this daemon is a candidate for its database."); + return [ + .. containers.Where(c => + string.Equals(c.State, "running", StringComparison.OrdinalIgnoreCase) + && DatabaseDumpTargets.IsPostgresImage(c.Image)), + ]; + } + + /// + /// Picks the one container that holds Watchtower's database out of the PostgreSQL containers on the + /// daemon, or throws saying why it could not. Pure, so the whole rule is testable without a daemon. + /// + /// + /// The connection string's host is the name Watchtower's own connection pool resolves, so a container + /// that answers to it is the database rather than merely a database. When nothing + /// answers to it, a single candidate is still unambiguous — a Compose install whose service is + /// aliased differently from the host is ordinary — but several are not, and guessing which one holds + /// the instance's own state is precisely the wrong thing to guess: the loser would be dumped, and the + /// dump would look like a good backup. + /// + /// The running PostgreSQL containers to choose from. + /// The host Watchtower's connection string names. + /// No candidate, or no way to tell them apart. + internal static DockerContainerInfo Choose(IReadOnlyList candidates, string host) { + if (candidates.Count == 0) throw new InvalidOperationException(NoCandidateMessage(host)); + + var matched = candidates.Where(c => AnswersTo(c, host)).ToList(); + return matched switch { + [var only] => only, + { Count: > 1 } => throw new InvalidOperationException( + $"More than one PostgreSQL container answers to the host '{host}' that Watchtower's " + + $"connection string names ({string.Join(", ", matched.Select(DisplayName))}). " + + $"Name the right one in {WatchtowerSettingPaths.BackupSelfPostgresContainer}."), + _ => candidates switch { + [var only] => only, + _ => throw new InvalidOperationException( + $"None of the PostgreSQL containers on this daemon answers to the host '{host}' that " + + "Watchtower's connection string names, and there is more than one to choose from " + + $"({string.Join(", ", candidates.Select(DisplayName))}). " + + $"Name the right one in {WatchtowerSettingPaths.BackupSelfPostgresContainer}."), + }, + }; + } + + /// + /// Whether is reachable under : its compose + /// service (what a container resolves a sibling by), its container name, or the + /// {project}-{service}-{n} name Compose generates from that service. + /// + private static bool AnswersTo(DockerContainerInfo container, string host) { + if (string.IsNullOrEmpty(host)) return false; + if (Same(container.Labels.GetValueOrDefault(ComposeServiceLabel), host)) return true; + foreach (var raw in container.Names) { + // Compose names a container "{project}-{service}-{replica}", so both ends have to come off + // before what is left can be compared to the service the connection string names. + var name = StripReplicaIndex(raw.TrimStart('/')); + if (Same(name, host) || name.EndsWith($"-{host}", StringComparison.OrdinalIgnoreCase)) + return true; + } + return false; + } + + /// + /// watchtower-postgres-1 → watchtower-postgres; anything not ending in a numeric + /// segment is returned as it is (a container the operator named by hand keeps its name). + /// + private static string StripReplicaIndex(string name) { + var lastDash = name.LastIndexOf('-'); + return lastDash > 0 && int.TryParse(name.AsSpan(lastDash + 1), out _) ? name[..lastDash] : name; + } + + private static bool Same(string? left, string right) => + left is not null && string.Equals(left, right, StringComparison.OrdinalIgnoreCase); + + private static string DisplayName(DockerContainerInfo container) => + container.Names.FirstOrDefault()?.TrimStart('/') ?? container.Id; + + /// + /// The message for "there is no container to dump". Names both reasons it can be true, because from + /// here they look identical: the database really is somewhere else, or the daemon did not answer. + /// + private static string NoCandidateMessage(string host) => + $"No running PostgreSQL container was found for the host '{host}' in Watchtower's connection " + + "string. Watchtower can only back up its own database when that database runs as a container " + + "on this Docker daemon — a managed or host-installed PostgreSQL has to be backed up by " + + "whoever operates it. If it is a container, name it in " + + $"{WatchtowerSettingPaths.BackupSelfPostgresContainer}. " + + "(This is also what you see when the Docker daemon could not be reached at all.)"; +} diff --git a/src/Watchtower.Application/Services/SelfUpdateService.cs b/src/Watchtower.Application/Services/SelfUpdateService.cs index ebf3593..da32488 100644 --- a/src/Watchtower.Application/Services/SelfUpdateService.cs +++ b/src/Watchtower.Application/Services/SelfUpdateService.cs @@ -1,4 +1,4 @@ -using Elarion.Settings; +using Elarion.Settings; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; @@ -515,9 +515,25 @@ private async Task UpdateRuntimeAsync( .FirstOrDefaultAsync(ct); } - private sealed record DetectedSelfInfo { + /// + /// Which container and image Watchtower is running as, or empty when it is not containerised. + /// + /// + /// Also what the instance restore needs (ADR-0027 §5): the coordinator it spawns has to be built + /// from this image and told which container to stop, and both answers come from the same + /// HOSTNAME → inspect this service already does for the self-update. + /// + public sealed record DetectedSelfInfo { public string? ContainerId { get; init; } public string? ImageName { get; init; } public bool IsRunningInContainer { get; init; } } + + /// + /// Never throws: an undetectable self is + /// reported as an empty record, and the caller says what that means for what it was about to do. + /// + /// Cancellation token. + public Task DetectSelfAsync(CancellationToken ct = default) => + TryInspectSelfAsync(ct); } diff --git a/src/Watchtower.Application/Services/StackProjectNames.cs b/src/Watchtower.Application/Services/StackProjectNames.cs index 09cde44..0754086 100644 --- a/src/Watchtower.Application/Services/StackProjectNames.cs +++ b/src/Watchtower.Application/Services/StackProjectNames.cs @@ -33,10 +33,22 @@ public static class StackProjectNames { /// Resolved compose project name to test. /// Stack to exclude from the check (the one being updated); null when creating. /// Cancellation token. + /// + /// The stack's display name, when the caller has one. A plain stack's backup directory is derived + /// from it (), so the one name that has to be refused here + /// as well is the one that would land its archives in Watchtower's own backup directory — where + /// retention would then prune the instance's dumps and the stack's archives as one set (ADR-0027). + /// Tenant stacks are unaffected: their directory is a level deeper, under the product. + /// /// An operator-facing error message, or null when the name is free to use. public static async Task ValidateAsync( WatchtowerDbContext db, SelfProjectNameProvider selfProjects, - string projectName, int? excludeStackId, CancellationToken ct) { + string projectName, int? excludeStackId, CancellationToken ct, string? stackName = null) { + if (stackName is not null && BackupNaming.IsReserved(stackName)) + return $"Stack name '{stackName.Trim()}' is reserved: it is where Watchtower keeps the " + + "backups of its own database, and a stack backing up alongside them would share " + + "their retention. Choose a different stack name."; + if (await selfProjects.IsReservedAsync(projectName, ct)) return $"Compose project name '{projectName}' is reserved: it is the project Watchtower " + "itself runs under. A stack sharing it would expose Watchtower's own containers " diff --git a/src/Watchtower.Application/Services/StackRevivalCoordinator.cs b/src/Watchtower.Application/Services/StackRevivalCoordinator.cs new file mode 100644 index 0000000..f901bce --- /dev/null +++ b/src/Watchtower.Application/Services/StackRevivalCoordinator.cs @@ -0,0 +1,265 @@ +using Elarion.Settings; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Watchtower.Application.Config; +using Watchtower.Application.Entities; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// +/// Brings one stack back after an instance restore (ADR-0027 §6): deploy it from git — the definition +/// arrived with the restored database — and then restore its newest archive into the volumes that deploy +/// created. +/// +/// +/// +/// The order is the whole point. A restore needs the volumes to exist, and only a deploy creates them; +/// a deploy on its own leaves the stack running with empty ones. Doing it by hand means remembering +/// that for every stack, in the middle of a disaster. +/// +/// +/// Progress is followed by reading each run's own event rather than by holding the work: both queues +/// already own their runs, both persist their outcome, and a coordinator that tried to own them too +/// would be a second opinion about what happened. It also means a restart mid-revival loses nothing +/// but the polling — the checklist row says what it was doing, and the operator can press it again. +/// +/// +public sealed class StackRevivalCoordinator( + DeployQueueService deploys, + BackupQueueService backups, + BackupStorageFactory storageFactory, + IServiceScopeFactory scopeFactory, + IOptionsMonitor options, + TimeProvider timeProvider, + ILogger logger) { + /// How long one stack's deploy or restore may run before the revival gives up watching. + internal static readonly TimeSpan DefaultStepTimeout = TimeSpan.FromHours(2); + + /// Gap between reads of a running event's status. + internal static readonly TimeSpan DefaultPollInterval = TimeSpan.FromSeconds(2); + + private readonly SemaphoreSlim _gate = new(1, 1); + private readonly TimeSpan _stepTimeout = DefaultStepTimeout; + private readonly TimeSpan _pollInterval = DefaultPollInterval; + + /// + /// Test seam for the wait: a test cannot spend two real hours proving the ceiling works, and a + /// two-second poll would make every revival test that slow. Same shape as + /// 's injected readiness wait; the parameters are not resolvable + /// from the container, so DI keeps picking the public constructor. + /// + internal StackRevivalCoordinator( + DeployQueueService deploys, BackupQueueService backups, BackupStorageFactory storageFactory, + IServiceScopeFactory scopeFactory, IOptionsMonitor options, + TimeProvider timeProvider, ILogger logger, + TimeSpan stepTimeout, TimeSpan pollInterval) + : this(deploys, backups, storageFactory, scopeFactory, options, timeProvider, logger) { + _stepTimeout = stepTimeout; + _pollInterval = pollInterval; + } + + /// + /// Revives one stack, updating its row on the checklist as it goes. Serialized process-wide: the two + /// queues below are single-flight anyway, and running several revivals at once would only interleave + /// their waiting. + /// + /// The stack to revive, as the restored database numbers it. + /// Cancellation token. + /// The stack's row as it ended up, or null when the checklist does not list it. + public async Task ReviveAsync(int stackId, CancellationToken ct) { + await _gate.WaitAsync(ct); + try { + return await ReviveOneAsync(stackId, ct); + } finally { + _gate.Release(); + } + } + + /// + /// Revives every stack still pending or failed, one after another. A failure does not stop the rest: + /// the checklist is a list of independent stacks, and stopping at the first would leave the operator + /// to work out which of the others had been tried. + /// + /// Cancellation token. + /// How many stacks ended up done. + public async Task ReviveAllAsync(CancellationToken ct) { + await _gate.WaitAsync(ct); + try { + var checklist = await LoadAsync(ct); + if (checklist is null) return 0; + + var revived = 0; + foreach (var stack in checklist.Stacks + .Where(s => s.Status is RevivalStatus.Pending or RevivalStatus.Failed) + .ToList()) { + var result = await ReviveOneAsync(stack.StackId, ct); + if (result?.Status == RevivalStatus.Done) revived++; + } + return revived; + } finally { + _gate.Release(); + } + } + + private async Task ReviveOneAsync(int stackId, CancellationToken ct) { + var checklist = await LoadAsync(ct); + if (checklist?.Stacks.FirstOrDefault(s => s.StackId == stackId) is not { } entry) return null; + + try { + var deployEventId = deploys.Enqueue(stackId, DeployTriggers.Manual).DeployEventId; + entry = await SaveAsync( + entry with { + Status = RevivalStatus.Deploying, Detail = "Deploying from git…", + DeployEventId = deployEventId, BackupEventId = null, + }, ct); + + var deployed = await WaitForDeployAsync(deployEventId, ct); + if (!deployed.Success) + return await SaveAsync( + entry with { Status = RevivalStatus.Failed, Detail = $"The deploy {deployed.Detail}." }, ct); + + // The archive is looked for only now: a stack whose volumes were just created has somewhere + // to put one, and the newest archive is whatever the storage holds at this moment. + var archive = await NewestArchiveAsync(stackId, ct); + if (archive is null) + return await SaveAsync( + entry with { + Status = RevivalStatus.Done, + Detail = "Deployed. No archive on the backup storage, so nothing was restored.", + }, ct); + + if (backups.TryEnqueueRestore(stackId, archive) is not { } restore) + return await SaveAsync( + entry with { + Status = RevivalStatus.Failed, + Detail = "Deployed, but a backup or restore was already running for this stack.", + }, ct); + + entry = await SaveAsync( + entry with { + Status = RevivalStatus.Restoring, Detail = $"Restoring {archive}…", + BackupEventId = restore.BackupEventId, + }, ct); + + var restored = await WaitForBackupAsync(restore.BackupEventId, ct); + return await SaveAsync( + restored.Success + ? entry with { Status = RevivalStatus.Done, Detail = $"Deployed and restored from {archive}." } + : entry with { Status = RevivalStatus.Failed, Detail = $"The restore {restored.Detail}." }, + ct); + } catch (Exception ex) when (ex is not OperationCanceledException) { + logger.LogWarning(ex, "Reviving stack {StackId} after a restore failed", stackId); + return await SaveAsync(entry with { Status = RevivalStatus.Failed, Detail = ex.Message }, ct); + } + } + + /// Marks one stack as handled by the operator, so "revive all" leaves it alone. + public async Task SkipAsync(int stackId, CancellationToken ct) { + var checklist = await LoadAsync(ct); + if (checklist?.Stacks.FirstOrDefault(s => s.StackId == stackId) is not { } entry) return null; + return await SaveAsync( + entry with { Status = RevivalStatus.Skipped, Detail = "Skipped — handled outside Watchtower." }, + ct); + } + + /// The checklist as it stands, or null when there is none. + public async Task LoadAsync(CancellationToken ct) { + await using var scope = scopeFactory.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + return await StackRevivalState.LoadAsync(settings, ct); + } + + /// Puts the checklist away. It is a prompt, not a record — the audit trail is the record. + public async Task DismissAsync(CancellationToken ct) { + await using var scope = scopeFactory.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + if (await StackRevivalState.LoadAsync(settings, ct) is not { } checklist) return; + await (checklist with { Dismissed = true }).SaveAsync(settings, ct); + } + + /// + /// Writes one row back, re-reading the checklist first so a concurrent change to another stack is + /// not overwritten by this one's stale copy. + /// + private async Task SaveAsync(RevivalStack stack, CancellationToken ct) { + await using var scope = scopeFactory.CreateAsyncScope(); + var settings = scope.ServiceProvider.GetRequiredService(); + if (await StackRevivalState.LoadAsync(settings, ct) is { } checklist) + await checklist.With(stack).SaveAsync(settings, ct); + return stack; + } + + /// The newest archive on the storage for one stack, or null when it has none. + private async Task NewestArchiveAsync(int stackId, CancellationToken ct) { + await using var scope = scopeFactory.CreateAsyncScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var stack = await db.Stacks.AsNoTracking().FirstOrDefaultAsync(s => s.Id == stackId, ct); + if (stack is null) return null; + + var backup = options.CurrentValue.Backup; + try { + using var storage = storageFactory.Create(backup); + var directory = BackupNaming.ResolveDirectory(stack, backup.ResolveInstanceName()); + return (await storage.ListFilesAsync(directory, ct)) + .Select(f => (f.Name, TakenAt: BackupNaming.ParseTimestamp(f.Name))) + .Where(x => x.TakenAt is not null) + .OrderByDescending(x => x.TakenAt) + .Select(x => x.Name) + .FirstOrDefault(); + } catch (Exception ex) when (ex is not OperationCanceledException) { + // Reported as "no archive" rather than as a failure: the deploy worked, which is most of the + // value, and the operator can restore by hand from the stack's own Backups tab. + logger.LogWarning(ex, "Could not list the backup storage while reviving stack {StackId}", stackId); + return null; + } + } + + /// Waits for one deploy to reach a terminal state. + private Task<(bool Success, string Detail)> WaitForDeployAsync(int deployEventId, CancellationToken ct) => + WaitAsync(async db => { + var status = await db.DeployEvents.AsNoTracking() + .Where(e => e.Id == deployEventId).Select(e => e.Status).FirstOrDefaultAsync(ct); + return status; + }, ct); + + /// Waits for one backup or restore to reach a terminal state. + private Task<(bool Success, string Detail)> WaitForBackupAsync(int backupEventId, CancellationToken ct) => + WaitAsync(async db => { + var status = await db.BackupEvents.AsNoTracking() + .Where(e => e.Id == backupEventId).Select(e => e.Status).FirstOrDefaultAsync(ct); + return status; + }, ct); + + /// + /// Polls one run's status until it stops being queued or running. Both event tables spell the four + /// states the same way, which is what lets one loop follow either. + /// + private async Task<(bool Success, string Detail)> WaitAsync( + Func> read, CancellationToken ct) { + var deadline = timeProvider.GetUtcNow() + _stepTimeout; + while (true) { + string? status; + await using (var scope = scopeFactory.CreateAsyncScope()) { + var db = scope.ServiceProvider.GetRequiredService(); + status = await read(db); + } + + switch (status) { + case BackupStatuses.Success: + return (true, "succeeded"); + case BackupStatuses.Failed: + return (false, "failed — its own log says why"); + case null: + // The row is gone: the stack was deleted under us, so there is nothing to revive. + return (false, "left no record — the stack no longer exists"); + } + + if (timeProvider.GetUtcNow() >= deadline) + return (false, $"was still running after {_stepTimeout.TotalHours:0} hours"); + await Task.Delay(_pollInterval, timeProvider, ct); + } + } +} diff --git a/src/Watchtower.Application/Services/StackRevivalState.cs b/src/Watchtower.Application/Services/StackRevivalState.cs new file mode 100644 index 0000000..0473b25 --- /dev/null +++ b/src/Watchtower.Application/Services/StackRevivalState.cs @@ -0,0 +1,112 @@ +using System.Text.Json; +using System.Text.Json.Serialization; +using Elarion.Settings; +using Microsoft.EntityFrameworkCore; +using Watchtower.Application.Config; +using Watchtower.Application.Persistence; + +namespace Watchtower.Application.Services; + +/// Where one stack has got to in the post-restore revival (ADR-0027 §6). +[JsonConverter(typeof(JsonStringEnumConverter))] +public enum RevivalStatus { + /// Nothing has been done for it yet. + Pending, + + /// Being deployed from git. + Deploying, + + /// Deployed; its newest archive is being restored into its volumes. + Restoring, + + /// Deployed and, where there was an archive, restored. + Done, + + /// The deploy or the restore failed; the run's own event says why. + Failed, + + /// Dismissed by the operator, who is handling this one themselves. + Skipped, +} + +/// One stack on the recovery checklist. +/// Its id in the restored database. +/// Its name, as the checklist shows it. +/// Where it has got to. +/// What last happened to it, in a sentence. +/// The deploy this revival started, once it has one. +/// The restore this revival started, once it has one. +public sealed record RevivalStack( + int StackId, + string Name, + RevivalStatus Status, + string? Detail = null, + int? DeployEventId = null, + int? BackupEventId = null); + +/// +/// The checklist an operator works through after an instance restore: every stack the restored database +/// knows about, to be redeployed from git and then restored from its newest archive (ADR-0027 §6). +/// +/// When the restore completed. +/// The instance the bundle came from. +/// Whether the operator has put the checklist away. +/// The stacks, in the order the checklist shows them. +public sealed record StackRevivalState( + DateTimeOffset RestoredAtUtc, + string SourceInstance, + bool Dismissed, + IReadOnlyList Stacks) { + /// + /// A Global settings row rather than a table: there is at most one of these, it has to survive the + /// restart the restore itself causes, and giving it a migration would be a schema change carried by + /// every instance that never restores anything. + /// + public const string SettingPath = WatchtowerSettingPaths.RestoreRecovery; + + /// Reads the checklist, or null when there is none. + public static async Task LoadAsync( + ISettingsManager settings, CancellationToken ct) { + var stored = await settings.GetStringAsync(SettingPath, SettingsScope.Global, ct); + if (string.IsNullOrWhiteSpace(stored)) return null; + try { + return JsonSerializer.Deserialize(stored, BackupBundle.JsonOptions); + } catch (JsonException) { + // Written by this build alone; unreadable means a hand-edit or a downgrade. Treated as + // absent rather than fatal — the checklist is a convenience, not a source of truth. + return null; + } + } + + /// Writes the checklist back. + public Task SaveAsync(ISettingsManager settings, CancellationToken ct) => + settings.SetStringAsync( + SettingPath, JsonSerializer.Serialize(this, BackupBundle.JsonOptions), + SettingsScope.Global, expectedVersion: null, ct).AsTask(); + + /// Removes it, once the operator is done with it. + public static Task ClearAsync(ISettingsManager settings, CancellationToken ct) => + settings.RemoveAsync(SettingPath, SettingsScope.Global, expectedVersion: null, ct).AsTask(); + + /// + /// Seeds the checklist from the database a restore has just brought in. The stacks come from that + /// database rather than from the bundle's manifest, because the ids the checklist has to act on are + /// the restored ones. + /// + internal static async Task SeedAsync( + ISettingsManager settings, RestoreProgress progress, WatchtowerDbContext db, CancellationToken ct) { + var stacks = await db.Stacks.AsNoTracking() + .OrderBy(s => s.Name) + .Select(s => new { s.Id, s.Name }) + .ToListAsync(ct); + var state = new StackRevivalState( + DateTimeOffset.UtcNow, progress.SourceInstance, Dismissed: false, + [.. stacks.Select(s => new RevivalStack(s.Id, s.Name, RevivalStatus.Pending))]); + await state.SaveAsync(settings, ct); + } + + /// The checklist with one stack replaced, leaving the rest as they were. + public StackRevivalState With(RevivalStack stack) => this with { + Stacks = [.. Stacks.Select(s => s.StackId == stack.StackId ? stack : s)], + }; +} diff --git a/src/Watchtower.Application/WatchtowerServiceCollectionExtensions.cs b/src/Watchtower.Application/WatchtowerServiceCollectionExtensions.cs index 9bf195b..91a131c 100644 --- a/src/Watchtower.Application/WatchtowerServiceCollectionExtensions.cs +++ b/src/Watchtower.Application/WatchtowerServiceCollectionExtensions.cs @@ -383,7 +383,25 @@ public static IServiceCollection AddWatchtowerServices(this IServiceCollection s // Database-aware dumps (ADR-0017): stateless over the engine's exec API, so a singleton. services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); + // Watchtower's own database (ADR-0027): the same dump/archive/storage path a stack's database + // takes, pointed at the container Watchtower itself connects to. + services.AddSingleton(); + services.AddSingleton(); + // The exportable bundle and the one staged file it produces (ADR-0027 §4). The state is a + // singleton because the tar outlives the request that built it and is served by another. + services.AddSingleton(); + services.AddSingleton(); + // Restoring an instance from a bundle (ADR-0027 §5). The staging holds the uploaded bundle and + // the marker that outlives the restart the restore itself causes; the completion service reads + // that marker on the way back up and says what happened. + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddHostedService(sp => sp.GetRequiredService()); + // Bringing the stacks back afterwards: deploy from git, then restore each one's newest archive. + services.AddSingleton(); // The two queues are separate by design, so what happens *after* a backup succeeds is its own // object: singleton, because the pending chains have to outlive the request that registered // them and the backup worker is the thing that releases them (design.md §"Backups across diff --git a/src/watchtower-web/src/lib/api.ts b/src/watchtower-web/src/lib/api.ts index 7dfd9b9..640eec2 100644 --- a/src/watchtower-web/src/lib/api.ts +++ b/src/watchtower-web/src/lib/api.ts @@ -3,6 +3,7 @@ // Nullable params are built explicitly (`?? null`) because the generated param types require // every key to be present. import { rpc } from './rpc-client' +import { apiBase } from './config' import type { AdoptStackResult, HostRegistry, @@ -13,9 +14,15 @@ import type { AuditEventPage, AuthConfig, AutomationConfig, + BackupBundle, BackupConfig, BackupEvent, + BackupEventKind, BackupRemoteFile, + InstanceRestoreStatus, + RecoveryChecklist, + RecoveryStack, + RestoreValidation, BackupRunAccepted, BackupPlanPreview, BackupQuiesceMode, @@ -97,6 +104,44 @@ import type { VolumeSize, } from './types' +/** + * Where a staged full backup bundle is fetched from (ADR-0027). A plain link rather than an RPC call: + * it streams a tar of arbitrary size, and the browser's own download handling is what should own it. + * Admin-only, and authenticated by the same session cookie every other request carries. + */ +export const BUNDLE_DOWNLOAD_URL = `${apiBase}/api/instance/bundle` + +/** Where an operator's bundle is uploaded for a restore. Admin-only; see {@link uploadRestoreBundle}. */ +const BUNDLE_UPLOAD_URL = `${apiBase}/api/instance/restore/bundle` + +/** + * Uploads a bundle and returns this instance's verdict on restoring it (ADR-0027). Nothing is replaced + * here — the upload is staged, and `backups.startInstanceRestore` is what acts on it. + * + * A plain fetch rather than an RPC call: the body is a tar of arbitrary size, streamed straight to disk + * on the other end. + */ +export async function uploadRestoreBundle( + file: File, + signal?: AbortSignal, +): Promise { + const response = await fetch(BUNDLE_UPLOAD_URL, { + method: 'POST', + credentials: 'include', + headers: { 'Content-Type': 'application/x-tar' }, + body: file, + signal, + }) + if (!response.ok) { + // The endpoint answers a rejected upload with a problem document whose `detail` says why. + const problem = await response.json().catch(() => null) + throw new Error( + problem?.detail ?? problem?.title ?? `The upload failed (HTTP ${response.status}).`, + ) + } + return (await response.json()) as RestoreValidation +} + export const api = { registries: { list: async () => (await rpc('registries.list', {})).registries as Registry[], @@ -476,16 +521,69 @@ export const api = { sftpPrivateKeyPassphrase: data.sftpPrivateKeyPassphrase ?? null, sftpBasePath: data.sftpBasePath ?? null, localBasePath: data.localBasePath ?? null, + includeSelf: data.includeSelf ?? null, + selfPostgresContainer: data.selfPostgresContainer ?? null, })).config as BackupConfig, testStorage: async () => (await rpc('backups.testStorage', {})).description as string, - events: async (stackId?: number, limit?: number, productId?: number) => + events: async (stackId?: number, limit?: number, productId?: number, kind?: BackupEventKind) => (await rpc('backups.events', { stackId: stackId ?? null, limit: limit ?? 50, // The fleet history: every deployment of one product. Additive — omitting it is the old call. productId: productId ?? null, + // 'instance' for Watchtower's own runs, 'stack' for the rest; omitted returns both (ADR-0027). + kind: kind ?? null, })).events as BackupEvent[], run: async (stackId: number) => (await rpc('backups.run', { stackId })).backup as BackupRunAccepted, + + /** Backs up Watchtower's own database (ADR-0027). Admin-only; needs an encryption passphrase. */ + runInstance: async () => (await rpc('backups.runInstance', {})).backup as BackupRunAccepted, + /** The instance's own archives present on the storage, newest first. Admin-only. */ + listInstance: async () => + (await rpc('backups.listInstance', {})) as { files: BackupRemoteFile[]; directory: string }, + + /** + * Starts building a full backup bundle (ADR-0027): a fresh instance dump plus every stack's newest + * archive, staged for download at {@link BUNDLE_DOWNLOAD_URL}. Admin-only, and slow — poll + * {@link getBundleStatus}. Returns the tracking event. + */ + exportBundle: async () => + (await rpc('backups.exportBundle', {})).export as BackupRunAccepted, + /** The staged bundle, or null when there is none. Admin-only. */ + getBundleStatus: async () => + (await rpc('backups.getBundleStatus', {})).bundle as BackupBundle | null, + + // ── Restoring this instance from a bundle (ADR-0027) ─────────────────── + /** Whether this instance looks fresh, what bundle is staged, and how the last restore ended. */ + getRestoreStatus: async () => + (await rpc('backups.getRestoreStatus', {})) as unknown as InstanceRestoreStatus, + /** + * Replaces this instance's database with the uploaded bundle's. Returns once the coordinator has + * been started — Watchtower stops answering a few seconds later and comes back on the restored + * database, where the caller's session no longer exists. + */ + startInstanceRestore: async () => + (await rpc('backups.startInstanceRestore', {})).sourceInstance as string, + + /** The post-restore checklist, or null when there is nothing to recover. */ + getRecoveryChecklist: async () => + (await rpc('backups.getRecoveryChecklist', {})).checklist as RecoveryChecklist | null, + /** Deploys one stack from git, then restores its newest archive. Runs to completion. */ + reviveStack: async (stackId: number) => + (await rpc('backups.reviveStack', { stackId })).stack as RecoveryStack, + /** The same for every stack still pending or failed, one after another. */ + reviveAll: async () => + (await rpc('backups.reviveAll', {})) as unknown as { + revived: number + checklist: RecoveryChecklist | null + }, + /** Marks one stack as handled outside Watchtower, so "revive all" leaves it alone. */ + skipRecoveryStack: async (stackId: number) => + (await rpc('backups.skipRecoveryStack', { stackId })).stack as RecoveryStack, + /** Puts the checklist away. What happened stays in the audit trail. */ + dismissRecovery: async () => { + await rpc('backups.dismissRecovery', {}) + }, listRemote: async (stackId: number) => (await rpc('backups.listRemote', { stackId })).files as BackupRemoteFile[], restore: async (stackId: number, fileName: string) => diff --git a/src/watchtower-web/src/lib/types.ts b/src/watchtower-web/src/lib/types.ts index b7cef2e..f6b729d 100644 --- a/src/watchtower-web/src/lib/types.ts +++ b/src/watchtower-web/src/lib/types.ts @@ -1356,6 +1356,12 @@ export interface BackupConfig { localBasePath: string /** Config paths pinned by `WATCHTOWER__*` env vars (env wins) — those fields are read-only. */ pinnedPaths: string[] + /** Whether the schedule also dumps Watchtower's own database (ADR-0027). Needs a passphrase. */ + includeSelf: boolean + /** Explicit container for Watchtower's own PostgreSQL, when detection cannot pick one. */ + selfPostgresContainer: string | null + /** Where the instance's own archives are written, e.g. `prod/_watchtower`. Derived, not settable. */ + instanceDirectory: string } /** `backups.updateConfig` request. Null secret fields keep the stored values; empty string clears. */ @@ -1377,13 +1383,97 @@ export interface UpdateBackupConfigRequest { sftpPrivateKeyPassphrase?: string | null sftpBasePath?: string | null localBasePath?: string | null + includeSelf?: boolean | null + selfPostgresContainer?: string | null +} + +/** What a backup run covered: one stack, or Watchtower's own database (ADR-0027). */ +export type BackupEventKind = 'stack' | 'instance' + +/** + * The full backup bundle staged for download (ADR-0027). Fetched from `/api/instance/bundle`, which is + * admin-only: the tar carries the key-protection secret, the backup passphrase and the storage + * credentials in plain text. + */ +export interface BackupBundle { + fileName: string + sizeBytes: number + createdAtUtc: string + /** How many stack archives it carries. */ + stackCount: number + /** How many stacks it describes but has no archive for — never backed up, so only the definition returns. */ + missingStackCount: number +} + +/** One reason a bundle cannot be restored here, or one caveat about doing so (ADR-0027). */ +export interface RestoreFinding { + /** A stable key to branch on, e.g. `key-protection-secret`. */ + code: string + /** The operator-facing sentence, which always names what to do about it. */ + message: string +} + +/** An uploaded bundle and this instance's verdict on restoring it (ADR-0027). */ +export interface RestoreValidation { + canRestore: boolean + /** Reasons the restore is refused outright. Non-empty means `canRestore` is false. */ + blocking: RestoreFinding[] + /** Things worth knowing that do not stop it. */ + warnings: RestoreFinding[] + /** The instance the bundle came from. */ + instanceName: string + /** The Watchtower build that wrote it. */ + appVersion: string + createdAtUtc: string + stackCount: number + missingStackCount: number + stackNames: string[] +} + +/** How the last instance restore ended. */ +export type RestoreOutcome = 'none' | 'succeeded' | 'failed' + +/** What the restore wizard needs to decide what to show (ADR-0027). */ +export interface InstanceRestoreStatus { + /** No stacks, no deploys and one account — a Watchtower nobody has used yet. */ + freshInstance: boolean + /** The uploaded bundle, checked against this instance, or null. */ + staged: RestoreValidation | null + lastOutcome: RestoreOutcome + lastError: string | null + /** Whether a post-restore recovery checklist is still open. */ + recoveryPending: boolean +} + +/** Where one stack has got to on the post-restore recovery checklist (ADR-0027). */ +export type RevivalStatus = 'pending' | 'deploying' | 'restoring' | 'done' | 'failed' | 'skipped' + +/** One stack on the recovery checklist. */ +export interface RecoveryStack { + stackId: number + name: string + status: RevivalStatus + detail: string | null + deployEventId: number | null + backupEventId: number | null +} + +/** The checklist an operator works through after an instance restore (ADR-0027). */ +export interface RecoveryChecklist { + restoredAtUtc: string + sourceInstance: string + dismissed: boolean + stacks: RecoveryStack[] } /** One backup run, for the history views. */ export interface BackupEvent { id: number - stackId: number - stackName: string + /** Null for an instance run — it backs up Watchtower itself, so there is no stack. */ + stackId: number | null + /** Null for an instance run; see `kind`. */ + stackName: string | null + kind: BackupEventKind triggeredBy: string status: 'queued' | 'running' | 'success' | 'failed' /** Provider-relative path of the uploaded archive (null until upload, and on failure). */ diff --git a/src/watchtower-web/src/modules/backups/ProductBackupsTab.tsx b/src/watchtower-web/src/modules/backups/ProductBackupsTab.tsx index af412c1..c4fe3e5 100644 --- a/src/watchtower-web/src/modules/backups/ProductBackupsTab.tsx +++ b/src/watchtower-web/src/modules/backups/ProductBackupsTab.tsx @@ -684,8 +684,11 @@ function FleetHistoryRow({ }: { event: { id: number - stackId: number - stackName: string + // Nullable since ADR-0027 gave the history stackless (instance) rows. They cannot reach this list — + // it is filtered by product, and an instance run has no stack — but the row still has to say what it + // would render rather than trusting a filter two layers away. + stackId: number | null + stackName: string | null triggeredBy: string status: string remotePath: string | null @@ -717,7 +720,9 @@ function FleetHistoryRow({ {/* Which instance — the column a per-stack history has no need of and a fleet history cannot do without. */} - {event.stackName} + + {event.stackName ?? 'Watchtower'} + {event.triggeredBy} diff --git a/src/watchtower-web/src/modules/settings/RecoveryChecklist.tsx b/src/watchtower-web/src/modules/settings/RecoveryChecklist.tsx new file mode 100644 index 0000000..7ed3b58 --- /dev/null +++ b/src/watchtower-web/src/modules/settings/RecoveryChecklist.tsx @@ -0,0 +1,180 @@ +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' +import { useRouteContext } from '@tanstack/react-router' +import { api } from '@/lib/api' +import type { RecoveryStack, RevivalStatus } from '@/lib/types' +import { timeAgo, absoluteTitle } from '@/lib/format' +import { Badge } from '@/components/ui/badge' +import { Button } from '@/components/ui/button' +import { Card, CardContent } from '@/components/ui/card' +import { toast } from '@/components/ui/use-toast' + +/** How each revival state reads, and which badge tone carries it. */ +const STATUS: Record = { + pending: { label: 'waiting', tone: 'neutral' }, + deploying: { label: 'deploying', tone: 'run' }, + restoring: { label: 'restoring', tone: 'run' }, + done: { label: 'back', tone: 'ok' }, + failed: { label: 'failed', tone: 'danger' }, + skipped: { label: 'skipped', tone: 'neutral' }, +} + +/** + * The checklist after an instance restore (ADR-0027): each stack is deployed from git — its definition + * arrived with the restored database — and then restored from its newest archive. + * + * The two steps are one action because the order matters and is easy to get wrong by hand: only a + * deploy creates the volumes, and a deploy on its own leaves the stack running with empty ones. + */ +export function RecoveryChecklistCard() { + const qc = useQueryClient() + // Reviving a stack deploys and restores it, which is admin-only on the server. + const { caps } = useRouteContext({ from: '__root__' }) + const isAdmin = caps.hasRole('Admin') + + const { data: checklist } = useQuery({ + queryKey: ['backups', 'recovery'], + queryFn: api.backups.getRecoveryChecklist, + enabled: isAdmin, + }) + + const invalidate = () => { + void qc.invalidateQueries({ queryKey: ['backups', 'recovery'] }) + void qc.invalidateQueries({ queryKey: ['backups', 'restoreStatus'] }) + void qc.invalidateQueries({ queryKey: ['stacks'] }) + } + + const revive = useMutation({ + mutationFn: api.backups.reviveStack, + onSuccess: stack => { + invalidate() + if (stack.status === 'failed') toast.error(`${stack.name}: ${stack.detail}`) + else toast.success(`${stack.name}: ${stack.detail}`) + }, + onError: err => toast.error(err instanceof Error ? err.message : 'The stack could not be revived.'), + }) + + const reviveAll = useMutation({ + mutationFn: api.backups.reviveAll, + onSuccess: result => { + invalidate() + toast.success(`${result.revived} stack${result.revived === 1 ? '' : 's'} deployed and restored.`) + }, + onError: err => toast.error(err instanceof Error ? err.message : 'The stacks could not be revived.'), + }) + + const skip = useMutation({ + mutationFn: api.backups.skipRecoveryStack, + onSuccess: invalidate, + onError: err => toast.error(err instanceof Error ? err.message : 'The stack could not be skipped.'), + }) + + const dismiss = useMutation({ + mutationFn: api.backups.dismissRecovery, + onSuccess: () => { + invalidate() + toast.success('Checklist dismissed.') + }, + onError: err => toast.error(err instanceof Error ? err.message : 'The checklist could not be dismissed.'), + }) + + if (!isAdmin || !checklist || checklist.dismissed) return null + + const busy = revive.isPending || reviveAll.isPending + const outstanding = checklist.stacks.filter(s => s.status === 'pending' || s.status === 'failed') + const revivingId = revive.variables + + return ( +
+
+

Bring the stacks back

+

+ This Watchtower was restored from a backup of{' '} + {checklist.sourceInstance}{' '} + {timeAgo(checklist.restoredAtUtc)}. + Each stack is deployed from git and then restored from its newest archive — in that order, + because only the deploy creates the volumes the restore needs. +

+
+ + + +
    + {checklist.stacks.map(stack => ( + revive.mutate(stack.stackId)} + onSkip={() => skip.mutate(stack.stackId)} + /> + ))} +
+ +
+ + + {busy && ( + + Deploying and restoring — this takes as long as the deploys do. + + )} +
+ + +
+ ) +} + +function ChecklistRow({ + stack, + busy, + working, + onRevive, + onSkip, +}: { + stack: RecoveryStack + busy: boolean + working: boolean + onRevive: () => void + onSkip: () => void +}) { + const status = STATUS[stack.status] + const outstanding = stack.status === 'pending' || stack.status === 'failed' + + return ( +
  • + + {status.label} + + {stack.name} + {stack.detail && {stack.detail}} + {outstanding && ( + + + + + )} +
    • + ) +} diff --git a/src/watchtower-web/src/modules/settings/RestoreInstancePage.tsx b/src/watchtower-web/src/modules/settings/RestoreInstancePage.tsx new file mode 100644 index 0000000..8bf50fc --- /dev/null +++ b/src/watchtower-web/src/modules/settings/RestoreInstancePage.tsx @@ -0,0 +1,329 @@ +import { useEffect, useRef, useState } from 'react' +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' +import { Link } from '@tanstack/react-router' +import { ArrowLeft, Upload } from 'lucide-react' +import { api, uploadRestoreBundle } from '@/lib/api' +import type { RestoreFinding, RestoreValidation } from '@/lib/types' +import { absoluteTitle } from '@/lib/format' +import { Banner } from '@/components/ui/banner' +import { Button } from '@/components/ui/button' +import { Card, CardContent } from '@/components/ui/card' +import { ConfirmDialog } from '@/components/ui/confirm-dialog' +import { Skeleton } from '@/components/ui/skeleton' +import { toast } from '@/components/ui/use-toast' +import { RecoveryChecklistCard } from './RecoveryChecklist' + +/** How long to wait for Watchtower to answer again before saying so. */ +const RESTART_TIMEOUT_MS = 5 * 60 * 1000 + +/** Gap between health probes while the instance is restarting. */ +const RESTART_POLL_MS = 2000 + +/** + * Restoring this Watchtower from a full backup bundle (ADR-0027). + * + * Behind the admin gate and behind a login, deliberately: an unauthenticated restore endpoint would be + * an unauthenticated way to replace an instance, and "the instance looked empty" is not an + * authorization decision. A fresh install signs in with the bootstrap admin first, then comes here. + */ +export function RestoreInstancePage() { + const qc = useQueryClient() + const fileInput = useRef(null) + const [restarting, setRestarting] = useState(false) + + const { data: status, isLoading } = useQuery({ + queryKey: ['backups', 'restoreStatus'], + queryFn: api.backups.getRestoreStatus, + }) + + const upload = useMutation({ + mutationFn: (file: File) => uploadRestoreBundle(file), + onSuccess: validation => { + qc.setQueryData(['backups', 'restoreStatus'], { + ...(status ?? { freshInstance: false, lastOutcome: 'none', lastError: null, recoveryPending: false }), + staged: validation, + }) + void qc.invalidateQueries({ queryKey: ['backups', 'restoreStatus'] }) + toast.success( + validation.canRestore + ? 'Bundle read — review what it holds before restoring.' + : 'Bundle read, but it cannot be restored here yet.', + ) + }, + onError: err => toast.error(err instanceof Error ? err.message : 'The upload failed.'), + }) + + const start = useMutation({ + mutationFn: api.backups.startInstanceRestore, + onSuccess: () => setRestarting(true), + onError: err => toast.error(err instanceof Error ? err.message : 'The restore could not be started.'), + }) + + const staged = status?.staged ?? null + + if (restarting) return + + return ( +
    +
    + + + Settings + +

    Restore this Watchtower

    +

    + Replaces everything this Watchtower knows — its stacks, accounts, routes, settings and keys — + with what is in a full backup bundle. Afterwards a checklist walks you through redeploying + each stack and restoring its data. +

    +
    + + {isLoading ? ( + + + + + + + ) : ( + <> + {status?.lastOutcome === 'failed' && ( + + {status.lastError ?? 'The database was not replaced.'} Nothing was changed — this + Watchtower is running on the database it had. The bundle is still here, so you can try + again. + + )} + + {status?.recoveryPending && } + + + +
    +

    1. Upload the bundle

    +

    + The .tar file built by “Build bundle” on the + instance you are restoring from. +

    +
    + + { + const file = e.target.files?.[0] + if (file) upload.mutate(file) + // Cleared so choosing the same file twice fires the change event again. + e.target.value = '' + }} + /> +
    + + {upload.isPending && ( + + Uploading and checking it — a large bundle takes a while. + + )} +
    +     +     + + {staged && ( + start.mutate()} + /> + )} + + )} +
    + ) +} + +/** What the uploaded bundle holds, what stops it, and the confirmation gate. */ +function StagedBundleCard({ + validation, + starting, + onRestore, +}: { + validation: RestoreValidation + starting: boolean + onRestore: () => void +}) { + const [confirming, setConfirming] = useState(false) + + return ( + + +
    +

    2. Check what it holds

    +
    +
    From
    +
    {validation.instanceName}
    +
    Built by
    +
    Watchtower {validation.appVersion}
    +
    Taken
    +
    + {new Date(validation.createdAtUtc).toLocaleString()} +
    +
    Stacks
    +
    + {validation.stackCount} with data + {validation.missingStackCount > 0 && + `, ${validation.missingStackCount} without an archive`} +
    +
    + {validation.stackNames.length > 0 && ( +

    {validation.stackNames.join(', ')}

    + )} +
    + + {validation.blocking.map(finding => ( + + ))} + {validation.warnings.map(finding => ( + + ))} + +
    +

    3. Restore

    +

    + Watchtower stops for a few seconds while its database is replaced, then comes back. You + will be signed out — sign in again with an account from{' '} + {validation.instanceName}. +

    +
    + +
    + +
    + + + Everything this Watchtower knows now is replaced by the backup of{' '} + {validation.instanceName}. Containers it deployed keep + running, unmanaged, until you redeploy them from the checklist afterwards. This cannot be + undone. + + } + confirmLabel="Restore" + requireText={validation.instanceName} + loading={starting} + onConfirm={() => { + setConfirming(false) + onRestore() + }} + /> +     +     + ) +} + +function FindingBanner({ finding, tone }: { finding: RestoreFinding; tone: 'danger' | 'warn' }) { + return ( + + {finding.message} + + ) +} + +/** + * The wait while the coordinator stops Watchtower, replays and starts it again. The session dies with + * the restart, so this polls the unauthenticated health endpoint rather than any API the old session + * could still reach, and sends the operator to the login page once it answers. + */ +function RestartingCard({ sourceInstance }: { sourceInstance: string }) { + const [tooLong, setTooLong] = useState(false) + + useEffect(() => { + const startedAt = Date.now() + // Watchtower is still answering right now — the coordinator waits a moment so this very request can + // return before it stops the container. So a health check that succeeds proves nothing until one + // has *failed* first: going away is what says the restore actually started. + let wentDown = false + let cancelled = false + + const probe = async () => { + const ok = await fetch('/health', { cache: 'no-store' }) + .then(r => r.ok) + .catch(() => false) + if (cancelled) return + if (!ok) { + wentDown = true + } else if (wentDown) { + // Down and back up: the restore is over, whatever its outcome, and the session went with it. + window.location.assign('/login') + return + } + if (Date.now() - startedAt > RESTART_TIMEOUT_MS) setTooLong(true) + } + + void probe() + const timer = window.setInterval(() => void probe(), RESTART_POLL_MS) + return () => { + cancelled = true + window.clearInterval(timer) + } + }, []) + + return ( +
    + + +
    + +

    Restoring…

    +
    +

    + Watchtower is being stopped, its database replaced with the backup of{' '} + {sourceInstance}, and started again. This page reconnects + on its own and sends you to the sign-in form. +

    +

    + Sign in with an account from {sourceInstance} — the + accounts this Watchtower had are gone. +

    + {tooLong && ( + + Watchtower has not answered for a few minutes. Check the{' '} + watchtower-restore-* container’s log on the host — it + always restarts Watchtower, whatever the outcome of the replay. + + )} +     +     +
    + ) +} diff --git a/src/watchtower-web/src/modules/settings/SettingsPage.tsx b/src/watchtower-web/src/modules/settings/SettingsPage.tsx index 0b33abb..6413f4e 100644 --- a/src/watchtower-web/src/modules/settings/SettingsPage.tsx +++ b/src/watchtower-web/src/modules/settings/SettingsPage.tsx @@ -1,5 +1,6 @@ -import { useState } from 'react' +import { useEffect, useState } from 'react' import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query' +import { Link, useRouteContext } from '@tanstack/react-router' import { AlertTriangle, CheckCircle2, @@ -8,11 +9,12 @@ import { RotateCcw, Timer, } from 'lucide-react' -import { api } from '@/lib/api' +import { api, BUNDLE_DOWNLOAD_URL } from '@/lib/api' import type { AuthConfig, AutomationConfig, BackupConfig, + BackupEvent, BackupProvider, MetricsBackend, MetricsConfig, @@ -22,7 +24,7 @@ import type { UpdateSelfConfigRequest, } from '@/lib/types' import { describeCron } from '@/lib/cron' -import { absoluteTitle, formatUptime, shortDigest, timeAgo } from '@/lib/format' +import { absoluteTitle, formatBytes, formatUptime, shortDigest, timeAgo } from '@/lib/format' import { ContainerLogs } from '@/components/container-logs' import { Badge } from '@/components/ui/badge' import { Banner } from '@/components/ui/banner' @@ -32,6 +34,7 @@ import { Field } from '@/components/ui/field' import { Input, Textarea } from '@/components/ui/input' import { SecretField } from '@/components/ui/secret-field' import { Skeleton } from '@/components/ui/skeleton' +import { StatusBadge } from '@/components/ui/status-badge' import { Select, SelectContent, @@ -41,6 +44,7 @@ import { } from '@/components/ui/select' import { Switch } from '@/components/ui/switch' import { toast } from '@/components/ui/use-toast' +import { RecoveryChecklistCard } from './RecoveryChecklist' const NO_CREDENTIAL = 'none' // Radix Select has no empty-string value. @@ -153,6 +157,11 @@ export function SettingsPage() { + + + {/* Self-hiding: renders only while a restore has left stacks to bring back (ADR-0027). */} + + ) @@ -1933,6 +1942,317 @@ function BackupsCard() { ) } +// ── Watchtower's own database (ADR-0027) ───────────────────────────────────── +// Its own card rather than a section of the backups card: the stack backups are a policy an operator +// configures, this is a thing they *do* — and it carries the run action, the archive list, and (from +// stage 2) the bundle export. + +/** How an instance archive's own run is summarised in the card's history list. */ +function InstanceRunRow({ event }: { event: BackupEvent }) { + const finished = event.finishedAt ?? event.startedAt + return ( +
  • + + + {event.triggeredBy} + + + {event.sizeBytes != null && `${formatBytes(event.sizeBytes)} · `} + {timeAgo(finished)} + +
    • + ) +} + +function InstanceBackupCard() { + const qc = useQueryClient() + // Every action in this card is admin-only on the server (the archive it produces is the instance), so + // an operator without the role is shown nothing rather than buttons that answer 403. + const { caps } = useRouteContext({ from: '__root__' }) + const isAdmin = caps.hasRole('Admin') + + const { data, isLoading } = useQuery({ + queryKey: ['backups', 'config'], + queryFn: api.backups.getConfig, + staleTime: 60_000, + enabled: isAdmin, + }) + + // A run in flight is worth watching: it is one dump, so it finishes in seconds to minutes. + const { data: events } = useQuery({ + queryKey: ['backups', 'events', 'instance'], + queryFn: () => api.backups.events(undefined, 10, undefined, 'instance'), + enabled: isAdmin, + refetchInterval: query => { + const runs = query.state.data ?? [] + return runs.some(e => e.status === 'queued' || e.status === 'running') ? 2000 : false + }, + }) + + const [container, setContainer] = useState(null) + const containerValue = container ?? data?.selfPostgresContainer ?? '' + const containerDirty = + container != null && container.trim() !== (data?.selfPostgresContainer ?? '').trim() + + // Both writes go through the whole-config handler, so the stored values for every other field are + // resent unchanged — the same shape the backups card posts. + const save = useMutation({ + mutationFn: (patch: { includeSelf?: boolean; selfPostgresContainer?: string }) => { + if (!data) throw new Error('Backup settings are still loading.') + return api.backups.updateConfig({ + enabled: data.enabled, + cron: data.cron, + instanceName: data.instanceName, + retentionDays: data.retentionDays, + retentionMaxCount: data.retentionMaxCount, + helperImage: data.helperImage, + provider: data.provider, + ...patch, + }) + }, + onSuccess: next => { + qc.setQueryData(['backups', 'config'], next) + setContainer(null) + toast.success('Saved — changes apply immediately.') + }, + onError: err => toast.error(err instanceof Error ? err.message : 'Failed to save.'), + }) + + const run = useMutation({ + mutationFn: api.backups.runInstance, + onSuccess: () => { + void qc.invalidateQueries({ queryKey: ['backups', 'events', 'instance'] }) + toast.success('Backing up Watchtower’s database — the run appears below.') + }, + onError: err => toast.error(err instanceof Error ? err.message : 'Could not start the backup.'), + }) + + // An export dumps the database and downloads every stack's newest archive, so it is minutes of work + // and the only honest progress signal is the run itself. + const exporting = (events ?? []).some( + e => e.triggeredBy === 'bundle-export' && (e.status === 'queued' || e.status === 'running'), + ) + const { data: bundle } = useQuery({ + queryKey: ['backups', 'bundle'], + queryFn: api.backups.getBundleStatus, + enabled: isAdmin, + refetchInterval: () => (exporting ? 2000 : false), + }) + + // The polling above stops the moment the run leaves 'running', which is one tick before the finished + // bundle would have been read — so the transition itself is what asks for it. + useEffect(() => { + if (!exporting) void qc.invalidateQueries({ queryKey: ['backups', 'bundle'] }) + }, [exporting, qc]) + + const exportBundle = useMutation({ + mutationFn: api.backups.exportBundle, + onSuccess: () => { + void qc.invalidateQueries({ queryKey: ['backups', 'events', 'instance'] }) + toast.success('Building the bundle — this takes a while for a large estate.') + }, + onError: err => toast.error(err instanceof Error ? err.message : 'Could not start the export.'), + }) + + const isPinned = (path: string) => data?.pinnedPaths.includes(path) ?? false + const noPassphrase = data != null && !data.hasEncryptionPassphrase + + if (!isAdmin) return null + + return ( +
    +
    +

    Watchtower’s own database

    +

    + Everything Watchtower knows — stacks, environment variables, products and releases, routes, + accounts, certificates and keys — lives in its PostgreSQL database. Backing up your stacks + without it restores their data but nothing that deploys them. +

    +
    + + {isLoading || !data ? ( + + + + + + + ) : ( + + + {noPassphrase && ( + + The dump carries every database role’s password hash, the data-protection key ring and + every certificate’s private key. Set a passphrase under Backups above and this turns on. + + )} + + + + + {({ id }) => ( + <> +
    + setContainer(e.target.value)} + disabled={isPinned('Watchtower:Backup:SelfPostgresContainer')} + /> + {containerDirty && ( + + )} +
    + {isPinned('Watchtower:Backup:SelfPostgresContainer') && ( + + )} + + )} +     + +
    + +
    + +
    +

    Full backup bundle

    +

    + One file holding a fresh dump of this database, the newest archive of every stack, and + the secrets that live outside the database — everything a new Watchtower needs to become + this one. Take one before a migration, and keep it somewhere you would keep a password. +

    + + + The key-protection secret, the backup passphrase and your storage credentials are inside + it. Anyone who has the file can stand this instance up elsewhere. + + +
    + + + {exporting ? ( + + Dumping the database and collecting each stack’s newest archive… + + ) : bundle ? ( + <> + + Download {bundle.fileName} + + + {formatBytes(bundle.sizeBytes)} · {bundle.stackCount} stack archive + {bundle.stackCount === 1 ? '' : 's'} · {timeAgo(bundle.createdAtUtc)} + + + ) : null} +
    + + {!exporting && bundle != null && bundle.missingStackCount > 0 && ( +

    + {bundle.missingStackCount} stack + {bundle.missingStackCount === 1 ? ' has' : 's have'} no archive on the storage — the + bundle carries {bundle.missingStackCount === 1 ? 'its' : 'their'} definition but not{' '} + {bundle.missingStackCount === 1 ? 'its' : 'their'} data. Back{' '} + {bundle.missingStackCount === 1 ? 'it' : 'them'} up and build again. +

    + )} + + {bundle != null && ( +

    + The bundle is kept in this container, so it is lost when Watchtower restarts — download + it now, or build a fresh one later. +

    + )} +
    + +
    +

    Restore from a bundle

    +

    + Replaces everything this Watchtower knows with what is in a bundle from another + instance, then walks you through bringing its stacks back. +

    + + Restore this Watchtower… + +
    + + {events != null && events.length > 0 && ( +
    +

    Recent runs

    +
      + {events.map(e => ( + + ))} +
    +
    + )} +     +     + )} +
    + ) +} + // ── Auth card (restart-required toggle) ─────────────────────────────────────── interface AuthDraft { diff --git a/src/watchtower-web/src/modules/settings/index.ts b/src/watchtower-web/src/modules/settings/index.ts index ba20d98..a912f12 100644 --- a/src/watchtower-web/src/modules/settings/index.ts +++ b/src/watchtower-web/src/modules/settings/index.ts @@ -1,9 +1,9 @@ import type { AppModule } from '@/platform/app-module' -import { settingsManifest, settingsRoute } from './module' +import { restoreInstanceRoute, settingsManifest, settingsRoute } from './module' const settingsModule = { manifest: settingsManifest, - routes: [settingsRoute], + routes: [settingsRoute, restoreInstanceRoute], } satisfies AppModule export default settingsModule diff --git a/src/watchtower-web/src/modules/settings/module.tsx b/src/watchtower-web/src/modules/settings/module.tsx index ef7e7a9..48453c4 100644 --- a/src/watchtower-web/src/modules/settings/module.tsx +++ b/src/watchtower-web/src/modules/settings/module.tsx @@ -30,3 +30,15 @@ export const settingsRoute = createRoute({ beforeLoad: redirectUnless({ module: 'System' }, '/'), component: lazyRouteComponent(() => import('./SettingsPage'), 'SettingsPage'), }) + +/** + * Restoring this Watchtower from a full backup bundle (ADR-0027). Its own route rather than a card: + * it is a multi-step flow that ends by taking the instance down, and it has to survive being the only + * thing an operator does on a freshly installed box. + */ +export const restoreInstanceRoute = createRoute({ + getParentRoute: () => rootRoute, + path: '/settings/restore', + beforeLoad: redirectUnless({ module: 'System' }, '/'), + component: lazyRouteComponent(() => import('./RestoreInstancePage'), 'RestoreInstancePage'), +})