Skip to content

Workspaces: 7-day backup add-on (paid) with verified restore #130

Open

Description

@sre-helmcode

Context

Workspace (Firecracker VM) snapshots are currently broken and, even when they ran, they protected nothing. We want to turn backups into an explicit, paid, opt-in add-on with a restore path that is actually tested.

Background: handoff-workspace-tiers-2026-09-18.md (workspace tiers and slot purchase flow).

Findings (current state)

  • Every nan-vmd restic snapshot fails: 226/226 since 2026-09-18. Error: unable to open config file ... Is there a repository. The per-agent restic repository is never initialised.
  • The scheduler snapshots the wrong path. It targets instances/<uuid>/rootfs.ext4, which is not modified after create. The live disk is jails/firecracker/<uuid>/root/rootfs.ext4. See snapshot/scheduler.go:362 and cmd/nan-vmd/main.go:59-63.
  • Interim mitigation: the snapshot scheduler is being disabled until this issue lands. So today no workspace has a usable backup.

Product proposal

  • Opt-in backup add-on with 7-day retention.

  • Billed as a Stripe subscription item on the slot's own subscription (same EUR currency; it cancels automatically with the slot, no separate subscription to reconcile).

  • Proposed price per tier (EUR/month):

    Tier Disk Backup add-on
    micro 10 GiB €0.99
    nano 20 GiB €1.49
    basic 40 GiB €2.49
    medium 80 GiB €3.99
    large 160 GiB €6.99

    Cost basis: R2 at about $0.015/GB-month. Worst case for large is about 200 GB stored (full disk plus 7 days of incrementals), roughly €3/month, so the price covers storage plus egress on restore and operational margin.

  • Only available while the VM slot is active. Cancelling the slot cancels the backups.

  • The UI warns clearly that disabling backups or cancelling the slot deletes the snapshots.

  • Open decision: is the free workspace (premium membership benefit) eligible for the add-on, and at which price (micro)?

Acceptance criteria

  • restic repository initialised per agent (idempotent init on first snapshot; covered by a test).
  • Snapshots read the live disk (jails/firecracker/<uuid>/root/rootfs.ext4), not the create-time image.
  • Snapshots are consistent: either guest freeze/fsync before the snapshot, or snapshot a reflink (XFS CoW) copy taken first. Document which one and why.
  • Retention enforced at 7 days (forget --prune policy), verified by a test.
  • End-to-end restore drill: a documented runbook in platform/devops/docs/runbooks/ plus an automated restore test on nan-vmd-dev (write a marker file, snapshot, destroy, restore, check the marker).
  • Metrics for snapshot success/failure/age, with a Slack alert on failures and on stale snapshots (last success older than 36h).
  • UI toggle per workspace in cloud-ui, plus the add-on shown as its own line on Stripe invoices.
  • Terms/docs state explicitly: no backups unless the add-on is active.
  • Disabling the add-on or cancelling the slot removes the item from the subscription and deletes the restic repository (after the warning).

Out of scope

  • Backups for workspaces without the add-on.
  • Retention longer than 7 days (can come later as a separate tier).

Activity

  1. sre-helmcode commented on Oct 4, 2026

    @sre-helmcode
    ContributorAuthor

    Status 2026-10-04: the backup add-on is LIVE for the beta (owner account only, behind the workspace beta gate).

    • nan-vmd v0.0.62-alpha (backup plane, sandboxed e2fsprogs for restores), operator v0.0.45-alpha (+ drill fixes v0.0.46/v0.0.47), cloud-api 0.8.211 (migration 096, API, Stripe), cloud-ui 0.8.95, website terms, devops alerts/runbook/dev drill.
    • Dev drill PASSED (snapshot, in-place restore, destroy+recreate restore, cleanup) on 2026-10-04; daily at 04:30 UTC with alerts.
    • Prod: snapshots enabled (devops#741), live Stripe prices (prod_VNZIma3VlhLcS9) in platform-api-secrets; R2 permissions and leftovers checked.
    • Owner decisions recorded in the design: free workspace eligible at EUR 0.99; self-service restore; snapshots kept (no pruning) until workspace deletion after billing ends; restore needs the add-on active; re-enable window 14 days (configurable); 3 restores/24h; no manual backup.

    Next: owner end-to-end test, then remove the beta gate (#139). Non-blocking hardening tracked separately.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions