Repository navigation
Workspaces: 7-day backup add-on (paid) with verified restore #130
Description
Context
Workspace (Firecracker VM) snapshots are currently broken and, even when they ran, they protected nothing. We want to turn backups into an explicit, paid, opt-in add-on with a restore path that is actually tested.
Background: handoff-workspace-tiers-2026-09-18.md (workspace tiers and slot purchase flow).
Findings (current state)
- Every nan-vmd restic snapshot fails: 226/226 since 2026-09-18. Error:
unable to open config file ... Is there a repository. The per-agent restic repository is never initialised. - The scheduler snapshots the wrong path. It targets
instances/<uuid>/rootfs.ext4, which is not modified after create. The live disk isjails/firecracker/<uuid>/root/rootfs.ext4. Seesnapshot/scheduler.go:362andcmd/nan-vmd/main.go:59-63. - Interim mitigation: the snapshot scheduler is being disabled until this issue lands. So today no workspace has a usable backup.
Product proposal
-
Opt-in backup add-on with 7-day retention.
-
Billed as a Stripe subscription item on the slot's own subscription (same EUR currency; it cancels automatically with the slot, no separate subscription to reconcile).
-
Proposed price per tier (EUR/month):
Tier Disk Backup add-on micro 10 GiB €0.99 nano 20 GiB €1.49 basic 40 GiB €2.49 medium 80 GiB €3.99 large 160 GiB €6.99 Cost basis: R2 at about $0.015/GB-month. Worst case for large is about 200 GB stored (full disk plus 7 days of incrementals), roughly €3/month, so the price covers storage plus egress on restore and operational margin.
-
Only available while the VM slot is active. Cancelling the slot cancels the backups.
-
The UI warns clearly that disabling backups or cancelling the slot deletes the snapshots.
-
Open decision: is the free workspace (premium membership benefit) eligible for the add-on, and at which price (micro)?
Acceptance criteria
- restic repository initialised per agent (idempotent init on first snapshot; covered by a test).
- Snapshots read the live disk (
jails/firecracker/<uuid>/root/rootfs.ext4), not the create-time image. - Snapshots are consistent: either guest freeze/fsync before the snapshot, or snapshot a reflink (XFS CoW) copy taken first. Document which one and why.
- Retention enforced at 7 days (
forget --prunepolicy), verified by a test. - End-to-end restore drill: a documented runbook in
platform/devops/docs/runbooks/plus an automated restore test onnan-vmd-dev(write a marker file, snapshot, destroy, restore, check the marker). - Metrics for snapshot success/failure/age, with a Slack alert on failures and on stale snapshots (last success older than 36h).
- UI toggle per workspace in cloud-ui, plus the add-on shown as its own line on Stripe invoices.
- Terms/docs state explicitly: no backups unless the add-on is active.
- Disabling the add-on or cancelling the slot removes the item from the subscription and deletes the restic repository (after the warning).
Out of scope
- Backups for workspaces without the add-on.
- Retention longer than 7 days (can come later as a separate tier).
Activity
docs(terms): workspace backup add-on (#144)
sre-helmcode commentedon Oct 4, 2026
Status 2026-10-04: the backup add-on is LIVE for the beta (owner account only, behind the workspace beta gate).
Next: owner end-to-end test, then remove the beta gate (#139). Non-blocking hardening tracked separately.
🤖 Generated with Claude Code