Skip to content

Move media staging off local disk before multi-replica deployment #1336

Description

@njbrake

Note: this issue was drafted by Claude via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

Context

#1333 moved the inbound media staging cache from an in-process Python dict to a disk + Postgres layout: bytes live on the deployment's persistent volume at `MEDIA_STAGING_BASE_DIR` (default `data/staged_media`), metadata lives in the `staged_media` table. This gives us 7-day retention that survives process restarts on a single-instance deployment.

It does not survive horizontal scaling. If we ever run more than one Clawbolt replica behind a load balancer, a webhook delivered to replica A writes bytes to A's local volume; a follow-up turn routed to replica B finds the DB row but cannot read the bytes (different volume).

When this becomes a problem

When we deploy to a multi-replica topology. Today Railway runs one instance so this is a deferred concern, not an active bug.

Options for the fix

  1. Postgres `BYTEA` on the existing `staged_media` row. Simplest: single source of truth, no new infrastructure. Adds DB load: a `stage()` call uploads several MB into a transaction, and reads pull them back. With the 50-per-user cap and 7-day TTL the table can grow substantially. Probably fine up to a few hundred active users.
  2. Object store (S3, GCS, R2). Bytes live in a bucket keyed by `(user_id, handle)`; DB row keeps the object URL instead of `disk_path`. Scales further, costs a small amount per request, adds a new dependency and credentials surface.
  3. Hybrid: keep disk-backed staging but co-locate the agent loop with the channel webhook (sticky routing by user_id). Avoids object-store cost but adds infra complexity (sharded sessions, drained-host handling).

(2) is the conventional answer; (1) buys time and is cheap.

Where the warning lives in code

  • `backend/app/agent/media_staging.py` module docstring ("MULTI-REPLICA WARNING")
  • `backend/app/config.py` next to the `media_staging_base_dir` setting
  • `docs/self-host/configuration.md` in the "Inbound media staging" section

When this issue is fixed, those references should be updated.

Definition of done

  • Bytes are reachable from any replica without depending on local disk.
  • Migration plan: existing on-disk entries either drained (TTL is 7 days, so a window-deferred deploy works) or copied into the new backing store at boot.
  • The warnings in the three locations above are removed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions