diff --git a/docs/plans/hosted-product-railway.md b/docs/plans/hosted-product-railway.md new file mode 100644 index 0000000..fb5a093 --- /dev/null +++ b/docs/plans/hosted-product-railway.md @@ -0,0 +1,115 @@ +# adrive hosted on Railway: comparison + +Companion to `hosted-product.md`. Same product, same external services (WorkOS, Autumn), built as Railway services on private networking instead of Cloudflare Workers. Written 2026-09-08 from Railway docs and pricing pages as served that day. + +## Service layout + +``` +railway project: adrive +├─ web SvelteKit Node server, 2-3 replicas, public domain + wildcard content domain +├─ worker queue consumer: indexing, purges, scans (pg-boss on Postgres, or BullMQ on Redis) +├─ screenshots Playwright, serverless sleep, called over private network +├─ postgres pgvector template image on a volume, pg_trgm included +├─ redis only if BullMQ is chosen over pg-boss +└─ bucket Railway Buckets (S3 on Tigris), file bytes +external: Workers AI over HTTP for embeddings, WorkOS, Autumn, Stripe +``` + +Everything except the bucket talks over `.railway.internal`. Internal traffic is free. Buckets are on the public network, so uploads from the service to the bucket count as service egress. + +## What gets simpler + +- **One runtime.** Plain Node. No adapter facade, no `nodejs_compat` flags, no per-request layer construction. Effect's Postgres client and `pg` pool are module-scope singletons. This deletes the trickiest parts of stack A in the Cloudflare plan. +- **One database for everything.** Same Postgres plan as the Cloudflare version: `tsvector`, `pg_trgm`, `pgvector`. Nothing changes in stacks A and B except the connection layer. +- **Queue is just Postgres.** pg-boss on the same database replaces Cloudflare Queues. Transactional enqueue: the index job is inserted in the same transaction as the file row, so the "never send inside waitUntil" class of bugs cannot happen. Cron is a Railway cron on the worker service. +- **Wildcards without a second product.** Railway issues Let's Encrypt wildcard certs itself. Per-tenant subdomains are a wildcard CNAME plus two ACME records. +- **PR environments.** Every PR gets a full stack including its own bucket instance. Good for the stacked PR workflow, though you pay for each one while it is up. +- **Config as code** via `.railway/railway.ts` with plan/apply and drift detection. The old `railway.json` is deprecated with a hard cutoff on 2026-12-01. + +## What gets harder + +- **Public file serving.** Railway Buckets are private only. No public buckets, no custom domain on the bucket. Two options: + - 302 from `/f/` to a presigned `*.storageapi.dev` URL. Egress is free. Users see a Tigris URL, range requests work, but every download is a redirect hop and the URL leaks the storage provider. + - Proxy bytes through the web service. Clean URLs, but $0.05 per GB egress. The free Fastly-backed Railway CDN caches objects under 512 MB and cache hits cost nothing, so real cost is roughly $0.05 per GB times the miss rate. + - Third option: keep R2 with a custom domain for the content origin, even though the app runs on Railway. R2 supports public buckets and free egress. This is the best fit for a clean public content domain and is what I would do. +- **Uploads.** Railway's edge caps request bodies at 5 minutes and requests at 15 minutes. Large uploads must go direct to the bucket via presigned PUT from the browser and the CLI. The current streaming upload path through the server would need to become a presign-then-confirm flow. This is a real change to `services/files/upload.ts` and the CLI. +- **Postgres is a container, not a service.** It runs on a volume. Daily snapshots, PITR via pgBackRest to a bucket, and one-click HA all exist in 2026, but the pgvector template image is not eligible for the one-click HA conversion. HA with pgvector means building the Patroni cluster yourself or picking PlanetScale from Railway. +- **No GPUs.** Embeddings stay on Workers AI over HTTP at $0.02 per million tokens, or an equivalent API. This is fine, just noted. +- **No scale to zero for the web tier.** Serverless sleep triggers on outbound silence, and a Postgres pool keeps it awake. Idle web replicas cost roughly $3 to $4 a month each, so this is a rounding error, but there is no "pay nothing at zero users" like Workers. +- **Reliability.** Five or more public incidents from November 2025 to July 2026, including an 8 hour platform-wide outage on 2026-05-19 when Google Cloud suspended Railway's account, and a 52 minute CDN bug on 2026-03-30 that served authenticated responses to the wrong users. Status page shows about 99.8 percent over 90 days for US regions. Cloudflare has its own incidents, but nothing in that class recently. + +## Latency and feel + +- **Dashboard and API.** Single region, so a user in Europe hitting US East sees 80 to 120 ms per request before any work. Workers plus Hyperdrive puts the compute at the edge but still crosses to the database region for every query, so the difference on database-bound requests is smaller than it looks. Multi-region replicas exist on Railway but volumes do not replicate, so the database stays in one region either way. +- **Public content.** Presigned bucket URLs are served from Tigris, which is globally distributed. Proxied content through the Railway CDN is served from Fastly POPs. R2 with a custom domain is served from Cloudflare's edge. All three are fine. The redirect hop on the presigned option is the only thing a user might notice. +- **Background work.** Always-on worker with a pool. No 30 second CPU limit, no 128 KB message cap, no consumer wall clock. Indexing a 50 MB text file in one job is fine on Railway and needs chunked jobs on Workers. +- **Local dev.** `docker compose` with Postgres and MinIO mirrors production almost exactly. On Cloudflare, local dev runs through miniflare and the Hyperdrive local connection string, which works but is a second world. + +## Cost + +Railway list prices, Pro plan, single region. Storage line assumes Railway Buckets. R2 would be the same $0.015 per GB plus a few dollars in request fees. + +| Line | 100 users | 1,000 users | +| ---------------------------------- | ------------- | ----------------- | +| Pro subscription | $20 | $20 | +| Web replicas | $40 | $120 | +| Workers | $40 | $160 | +| Screenshots | $10 | $30 | +| Postgres single node with pgvector | $130 | $270 to $430 | +| Postgres HA instead | n/a | $900 to $1,350 | +| Embeddings via API | $1 | $10 | +| Bucket storage | $24 | $240 | +| Egress, presigned | $0 | $0 | +| Egress, proxied no cache | $100 | $1,000 | +| Egress, proxied with 70% CDN hit | $30 | $300 | +| Total, presigned, single node DB | ~$265 | ~$850 to $1,010 | +| Total, proxied no cache, HA DB | not estimated | ~$2,480 to $2,930 | + +Same scenarios on the Cloudflare plan from `hosted-product.md`: + +| Line | 100 users | 1,000 users | +| --------------------------------------- | --------- | ---------------------------------- | +| Workers Paid | $5 | $5 | +| PlanetScale | $30 | $30 to $190, more storage at scale | +| R2 storage | $24 | $246 | +| Egress | $0 | $0 | +| Workers AI, Queues, Browser Run, Images | ~$1 | ~$50 | +| Total | ~$60 | ~$350 to $550 | + +Where the gap comes from: + +- **Compute.** Workers charge per request and are nearly free at this scale. Railway charges for always-on replicas and a worker. Roughly $100 to $300 a month of the gap. +- **Postgres.** Railway's container at 4 vCPU and 32 GB is about $400. PlanetScale PS-320 single node is $190 and HA is $570, and it is managed. At low load, PlanetScale PS-10 at $30 has no Railway equivalent since a pgvector container with enough RAM for the index is the floor. +- **Egress.** Identical at $0 only if you accept the presigned redirect or keep R2. Proxying through Railway is the single largest possible line at scale. + +The presigned, single-node Railway estimate is about $265 versus $60 a month at 100 users, and $850 to $1,010 versus $350 to $550 at 1,000 users. These estimates imply a gap of about $205 and $300 to $660 a month, respectively. + +## Recommendation + +Cloudflare for the product as scoped. The reasons in order: + +1. Public file serving with clean URLs, range requests, and free egress is the core of the product. R2 with a custom domain does this natively. Railway needs R2 anyway or a redirect hop. +2. At 100 users, the estimated monthly cost is about $60 versus $265 for Railway's presigned, single-node setup, which matters during experimentation. +3. The reliability record over the last ten months is materially worse on Railway, and the May 2026 outage was the kind that no architecture on their platform could route around. + +Railway wins if the shape changes toward: + +- Heavy server-side processing per file, such as video transcoding, OCR, or running larger models, where the Workers CPU and memory limits bite. +- Wanting a boring Node process with a pool and a debugger rather than the Workers runtime. +- Multi-region being off the table anyway and local dev fidelity mattering more than edge latency. + +A hybrid is reasonable and not weird: Railway for web and workers, R2 for bytes, PlanetScale for the database. That keeps the Node runtime and PR environments, drops the egress and public-bucket problems, and drops the unmanaged Postgres problem. Its bill lands around $150 at 100 users and $500 to $700 at 1,000, so between the two pure options. + +## Sources + +- Railway [pricing](https://railway.com/pricing), [plans](https://docs.railway.com/pricing/plans), [pricing FAQs](https://docs.railway.com/pricing/faqs), [cost control](https://docs.railway.com/pricing/cost-control) +- Railway [storage buckets](https://docs.railway.com/storage-buckets), [bucket billing](https://docs.railway.com/storage-buckets/billing), [uploading and serving files](https://docs.railway.com/storage-buckets/uploading-serving) +- Railway [PostgreSQL](https://docs.railway.com/databases/postgresql), [PostgreSQL HA](https://docs.railway.com/databases/postgresql-ha), [volumes](https://docs.railway.com/volumes/reference), [point-in-time recovery](https://docs.railway.com/volumes/point-in-time-recovery) +- Railway [scaling](https://docs.railway.com/deployments/scaling), [regions](https://docs.railway.com/deployments/regions), [serverless](https://docs.railway.com/deployments/serverless) +- Railway [private networking](https://docs.railway.com/networking/private-networking/how-it-works), [domains](https://docs.railway.com/networking/domains/working-with-domains), [public networking limits](https://docs.railway.com/networking/public-networking/specs-and-limits), [CDN](https://docs.railway.com/networking/cdn), [WAF](https://docs.railway.com/networking/waf) +- Railway [cron jobs, workers, and queues](https://docs.railway.com/guides/cron-workers-queues), [Playwright](https://docs.railway.com/guides/playwright), [cron jobs](https://docs.railway.com/cron-jobs) +- Railway [infrastructure as code](https://docs.railway.com/infrastructure-as-code), [environments](https://docs.railway.com/environments), [observability](https://docs.railway.com/observability) +- Railway incident reports: [2026-05-19 GCP account outage](https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage), [2026-03-30 cached authenticated responses](https://blog.railway.com/p/incident-report-march-30-2026-authenticated-user-data-cached), [2026-02-11](https://blog.railway.com/p/incident-report-february-11-2026), [2026-07-02](https://blog.railway.com/p/incident-report-july-2-2026-us-east-services-outage) +- [Railway status](https://status.railway.com) +- Cloudflare [Workers AI pricing](https://developers.cloudflare.com/workers-ai/platform/pricing/), [R2 pricing](https://developers.cloudflare.com/r2/pricing/) +- [PlanetScale Postgres pricing](https://planetscale.com/docs/postgres/pricing) diff --git a/docs/plans/hosted-product.md b/docs/plans/hosted-product.md new file mode 100644 index 0000000..ffbf64d --- /dev/null +++ b/docs/plans/hosted-product.md @@ -0,0 +1,545 @@ +# adrive hosted: phased plan + +Status: exploration. Nothing here is committed to. Written 2026-09-08. + +Decisions already made: + +- The hosted product replaces the self-hosted passcode + D1 + Vectorize mode. One codebase, one data model. +- Tenant unit is the org. Every tenant row carries `org_id`. Signup creates a personal org. +- All search moves into Postgres: `tsvector` + `pg_trgm` for keyword, `pgvector` for semantic. Vectorize is removed. Workers AI still produces embeddings. +- Public content is served from `.` on a throwaway domain, one wildcard route. +- Postgres port lands first while still single-tenant on passcode auth, so parity can be checked on the real drive before tenancy. +- Stacked PRs via GitHub native `gh stack`. +- Indexing and other background work moves to Cloudflare Queues. Cron shrinks to reconciliation sweeps. +- Abuse at launch: hash lists + Cloudflare URL Scanner + trust levels + rate limit bindings + kill switch. ClamAV in Containers is a later phase. + +Stack: WorkOS AuthKit, PlanetScale Postgres via Hyperdrive, `@effect/sql-pg`, Autumn on your Stripe, Cloudflare Queues, Workers rate limit bindings. + +## Where the code is today + +Facts from the current tree that shape the plan. Paths are under `apps/web/src`. + +| Area | Current state | What it means for the port | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| SQL client | `@effect/sql-d1` imported in exactly one file, `lib/server/layer.ts:17`. But most data code uses the raw D1 API: `db.prepare().bind().run()`, 19 `db.batch()` sites, and `result.meta.changes` for optimistic concurrency | Swapping the layer is one line. Rewriting ~190 statements across 22 files to the `sql` template, `withTransaction`, and `RETURNING` is the real work | +| Schema | 10 migrations, 18 tables + 2 FTS5 virtual tables. No `user_id` or `org_id` anywhere. Three singleton rows keyed `id = 1`: `credential_state`, `instance_secrets`, and the implicit global quota | Tenancy is additive columns plus scoping every query, not a redesign | +| SQLite-only syntax | FTS5 `MATCH` + `bm25()` in `search-candidates.ts:29-39`, trigram table, `json_each` (7 sites, 0-based ordinal at `search-candidates.ts:49`), `json_extract`, `json_group_array`, `group_concat`, `INSERT OR IGNORE`, booleans as 0/1, timestamps as ISO strings | Keyword search is a rewrite. Everything else is mechanical | +| Auth | No `App.Locals`. `hooks.server.ts` does origin gating only. ~30 `authorizeRequest` calls across 19 route files. Passcode compared in `services/auth.ts:408` | Introduce `locals.auth` and a `handle` hook once, then thin the route files | +| Vectorize | 4 calls, all in `services/semantic.ts:145-187`, behind `VectorIndexShape` with a Null impl. Metadata is written but never read at query time | Easiest swap in the whole plan | +| Cron | Facade generated by `apps/web/scripts/cloudflare-adapter.mjs`, HMAC-signed self-fetch to `/api/internal/maintenance`, five sequential sweeps with global caps in `services/lifecycle.ts:71-95` | Same facade grows a `queue()` export. Sweeps become per-org reconciliation | +| Quota | `storage-quota.ts` sums every version in the deployment, called from two upload paths | Becomes a per-org counter row updated in the upload transaction | +| Tests | Real workerd via `getPlatformProxy`, migrations applied in `routes/global-setup.ts`, 55 test files | Hyperdrive has a local connection string override, so the same harness works against a local Postgres | +| CLI | Device code flow, never sees the passcode. Hits `/api/auth/device`, polls `/api/auth/device/token`, stores an `adr_` key | Wire format is unchanged by tenancy. The approve page just gains an org name | + +## Stack layout + +Seven stacks. Each stack is a chain of PRs where each PR targets the one below it. Stacks A through D are strictly sequential. E and F can start once B has merged. G is ops and docs. + +``` +main + └─ A Postgres port (single-tenant) 8 PRs + └─ B Tenancy + WorkOS 8 PRs + └─ C Content origin + subdomains 3 PRs + └─ D Queues 4 PRs + ├─ E Abuse 6 PRs + └─ F Billing (Autumn) 4 PRs +G Ops, backups, launch checklist 3 PRs, off main +``` + +Rough effort at one focused person: + +| Stack | Weeks | Notes | +| ----- | ----- | ------------------------------------------------------------- | +| A | 3 | Half of it is the keyword search rewrite and the test harness | +| B | 3 | Scoping every query is tedious, not hard | +| C | 1 | | +| D | 1 | | +| E | 2 | The kill switch and report queue are the parts that matter | +| F | 1 | Autumn does most of it | +| G | 1 | | + +Total is roughly 12 weeks. The first shippable point is the top of stack A, at week 3, on your own drive. + +## Stacked PR workflow + +GitHub native stacks, public preview since July 2026. + +```sh +gh extension install github/gh-stack + +# start a stack +git switch -c a1-pg-infra main +# ... commit ... +git switch -c a2-auth-tables +# ... commit ... +gh stack init --base main a1-pg-infra a2-auth-tables +gh stack submit --auto # opens draft PRs with the matching base chain + +# after review changes low in the stack +gh stack sync # rebases everything above the changed layer + +# merge bottom-up from the PR UI. GitHub retargets the upper PRs to main automatically. +``` + +Preserve the repository's review protections. If stack tooling cannot merge while stale approvals are dismissed, refresh the affected reviews and merge bottom-up through the PR UI rather than disabling the rule for the repository. + +Keep each PR under about 600 lines of diff. Where a PR below is called out as "not independently deployable," that is fine. Deploy from the top of the stack. + +## Stack A: Postgres port + +Goal: the app runs on PlanetScale Postgres with the passcode, single tenant, and reaches feature parity on your drive. D1 and Vectorize are removed at the end of this stack. + +### A1 Infrastructure + +- Add `nodejs_compat` to `wrangler.jsonc` compatibility flags. `@effect/sql-pg` is built on `pg`, which needs it. +- Add a Hyperdrive binding pointed at PlanetScale port 5432, not the PgBouncer port. Hyperdrive is already a transaction pooler. Set placement to the database region. + +```jsonc +"hyperdrive": [{ "binding": "HYPERDRIVE", "id": "", "localConnectionString": "postgres://adrive:adrive@localhost:5432/adrive" }] +``` + +- Add a second Hyperdrive binding with caching disabled for read-after-write paths, or turn caching off entirely. Recommendation: caching off. The app already has an edge cache for the paths that matter. +- New `Pg` tag in `services/bindings.ts` for the Hyperdrive binding. New layer next to `SqlLive` in `layer.ts`: + +```ts +import { PgClient } from '@effect/sql-pg'; +import { Redacted } from 'effect'; + +const PgLive = Layer.unwrap( + Effect.map(Pg, (hyperdrive) => + PgClient.layer({ + url: Redacted.make(hyperdrive.connectionString), + maxConnections: 5, + applicationName: 'adrive' + }) + ) +); +``` + +The connection string is only available at request time, so the per-request `requestLayer` shape in `layer.ts:19` stays. `PgClient.layer` runs `SELECT 1` on acquire. If that shows up in latency, use `PgClient.fromPool` with a request-scoped `pg.Pool` that closes when the request layer is released. Hyperdrive owns pooling across requests; connection-bearing clients must not outlive their Worker request. + +- Migrations: plain SQL files in `apps/web/migrations-pg/`, run with dbmate against the direct 5432 port using an admin role. PlanetScale has no deploy requests for Postgres, so this is the only option. Migration 0001 enables extensions: + +```sql +CREATE EXTENSION IF NOT EXISTS vector; +CREATE EXTENSION IF NOT EXISTS pg_trgm; +``` + +- Tests: `docker compose` with `pgvector/pgvector:pg17`, `global-setup.ts` runs dbmate up against it, `WRANGLER_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE` points the proxy at it. Route suites keep sharing state exactly as they do now. +- `check-wrangler-drift.mjs` learns the new bindings. + +No callers change in this PR. Deployable. + +### A2 Auth tables + +`services/auth.ts`, 33 statements, tables `api_keys`, `dashboard_sessions`, `device_codes`, `credential_state`. Isolated from everything else, so it goes first. + +- Port each statement to the `sql` template from `SqlClient`. Every `meta.changes` check becomes `RETURNING id` and a length check. Every `db.batch` becomes `sql.withTransaction`. +- The device token poll at `auth.ts:676-703` uses a guarded `INSERT ... SELECT ... WHERE EXISTS` inside a batch. Same statement works in Postgres inside a transaction. +- Passcode rotation at `auth.ts:735-793` relied on batch atomicity. `withTransaction` gives the same guarantee. +- Column types: `expires_at`, `created_at`, `last_used_at` become `timestamptz`. Comparisons against `new Date().toISOString()` still work because `pg` parses the string. + +Not independently deployable until A8, since D1 and Postgres run side by side with different tables. That is fine, both bindings exist from A1. + +### A3 Grant secrets and quota + +- `services/grant-secrets.ts`: `INSERT OR IGNORE` becomes `ON CONFLICT (id) DO NOTHING`. +- `storage-quota.ts`: the three-way sum ports as is. Its `StorageQuotaDatabase` structural type becomes the `sql` client. Leave the aggregate for now, A6 in stack B replaces it with a counter. + +### A4 Files core + +Tables `files`, `file_versions`, `file_tags`, `tags`. Files `services/files/*`, `services/tags.ts`, `file-rows.ts`, `thumbnail-storage.ts`. Roughly 45 statements. + +- Booleans: `public`, `is_site`, `deleted` become real `boolean`. Every `= 1` and `? 1 : 0` in the TypeScript goes away. This touches the row schemas in `services/files/types.ts` and `packages/shared`. +- `json_group_array(json_object(...))` at `file-rows.ts:56` becomes `jsonb_agg(jsonb_build_object(...))`. +- `ON CONFLICT(normalized_name)` at `tags.ts:167,205` keeps its shape. The unique constraint changes in stack B. +- The dynamic `sql.unsafe` clauses in `files/queries.ts:36-45` stay dynamic. Values interpolated there are internal enums, not user input. Postgres placeholders are `$1`, not `?`, so the builder changes. + +### A5 Keyword search + +This is the only real rewrite. Files `search-candidates.ts`, `search-index.ts`, `indexing-sql.ts`, `services/search.ts`. + +Replace `files_fts` and `files_trgm` with one table: + +```sql +CREATE TABLE search_documents ( + file_id text NOT NULL REFERENCES files(id) ON DELETE CASCADE, + chunk_no integer NOT NULL, + name text NOT NULL, + tags text NOT NULL DEFAULT '', + body text NOT NULL DEFAULT '', + tsv tsvector GENERATED ALWAYS AS ( + setweight(to_tsvector('simple', name), 'A') || + setweight(to_tsvector('simple', tags), 'B') || + setweight(to_tsvector('english', body), 'C')) STORED, + PRIMARY KEY (file_id, chunk_no) +); +CREATE INDEX search_documents_tsv ON search_documents USING gin (tsv); +CREATE INDEX search_documents_name_trgm ON search_documents USING gin (name gin_trgm_ops) WHERE chunk_no = 0; +``` + +- `bm25(files_fts, 10.0, 5.0, 1.0)` becomes `ts_rank_cd('{0.1, 0.2, 0.4, 1.0}', tsv, query)`. Note the sign flip: bm25 is lower-is-better and the code orders `ASC` at `search-candidates.ts:39`. `ts_rank_cd` is higher-is-better. +- `sanitizeMatchQuery` at `services/search.ts:204` becomes `websearch_to_tsquery('english', $1)`, which handles quoting and operators safely. Delete the FTS5 escaping code. +- `sanitizeTrigramQuery` at `search.ts:220` becomes `name ILIKE '%' || $1 || '%'` or `similarity(name, $1) > 0.3` over the trigram index. +- `json_each($1)` becomes `unnest($1::text[])`. The ordinal use at `search-candidates.ts:49` needs `WITH ORDINALITY` and a `- 1`. +- `group_concat(t.name, ' ')` at three sites becomes `string_agg(t.name, ' ')`. +- The fusion ranking in `services/search.ts` is pure TypeScript and does not change. + +Snapshot the current search test fixtures before this PR and assert the same top-5 results on both engines for a dozen queries. Exact scores will differ, order for obvious queries should not. + +### A6 Semantic search on pgvector + +- `file_chunks` gains `embedding vector(384)`. HNSW index with cosine ops: + +```sql +ALTER TABLE file_chunks ADD COLUMN embedding vector(384); +CREATE INDEX file_chunks_embedding ON file_chunks USING hnsw (embedding vector_cosine_ops); +``` + +- New `VectorIndexShape` implementation in `services/semantic.ts` that upserts into `file_chunks` and queries with `ORDER BY embedding <=> $1 LIMIT 100`. The Null implementation stays for `SEMANTIC_SEARCH=off`. +- `eligibleSemanticCommand` at `search-candidates.ts:44` currently re-joins returned ids against `files` to filter visibility. Fold that join into the vector query itself. One query, one round trip. +- Drop `pending_vector_deletes`, `retryVectorDeletes`, and `vectorIdForChunk`. Deleting the chunk row deletes the vector. The lease-token-scoped id scheme in `semantic-policy.ts:53` becomes just `(file_id, version, ordinal)`. +- Remove the `VECTORIZE` binding and `SemanticBindingsLive` branching for it. `AI` stays. + +Storage math: 1,544 bytes per row plus a comparable amount for the HNSW index. Ten million chunks is roughly 30 GB and wants a cluster with enough RAM to hold the index. At that point move to `halfvec(384)` to halve it. Not a launch concern. + +### A7 Sites + +`services/sites/*`, 44 statements. Same mechanical port as A4. `ON CONFLICT(r2_key)` sites keep their shape. + +### A8 Cut over and remove D1 + +- One-time data move: `wrangler d1 export` to SQL, a script that transforms 0/1 to booleans and loads into Postgres via `psql`. R2 does not move. Run it against your drive, verify counts, flip. +- Delete `@effect/sql-d1`, the `DB` binding, `apps/web/migrations/`, the D1 steps in `docs/release.md` and `docs/backup-restore.md`. +- `docs/backup-restore.md` shrinks to R2 mirroring. PlanetScale keeps its own backups. + +Deployable. This is the first checkpoint: same product, new database. + +## Stack B: Tenancy and WorkOS + +Goal: real accounts, every row scoped to an org, passcode gone. + +### B1 Schema + +```sql +CREATE TABLE orgs ( + id text PRIMARY KEY, -- WorkOS org id + slug text NOT NULL UNIQUE, + name text NOT NULL, + trust text NOT NULL DEFAULT 'new', -- new | verified | established | suspended + plan text NOT NULL DEFAULT 'free', + created_at timestamptz NOT NULL DEFAULT now() +); +CREATE TABLE users ( + id text PRIMARY KEY, -- WorkOS user id + email text NOT NULL, + email_verified boolean NOT NULL DEFAULT false, + created_at timestamptz NOT NULL DEFAULT now() +); +CREATE TABLE memberships ( + org_id text NOT NULL REFERENCES orgs(id), + user_id text NOT NULL REFERENCES users(id), + role text NOT NULL DEFAULT 'owner', + PRIMARY KEY (org_id, user_id) +); +CREATE TABLE org_usage ( + org_id text PRIMARY KEY REFERENCES orgs(id), + stored_bytes bigint NOT NULL DEFAULT 0, + file_count integer NOT NULL DEFAULT 0 +); +``` + +- Add `org_id text NOT NULL REFERENCES orgs(id)` to `files`, `tags`, `api_keys`, `device_codes`, `site_upload_sessions`. Child tables reach the org through their parent, but add `org_id` to `file_versions` and `file_chunks` too so the hot queries do not need a join. +- Unique constraints gain the org: `tags (org_id, normalized_name)`, `api_keys (org_id, prefix)` if you want prefixes per org, though a global prefix is fine. +- Backfill: create one org for the existing drive, set `org_id` on every row, then add `NOT NULL`. +- Row level security as a safety net, policy keyed on a transaction-local setting. PlanetScale supports it and also advises against relying on it alone. Use it as the second layer, never the first: + +```sql +ALTER TABLE files ENABLE ROW LEVEL SECURITY; +ALTER TABLE files FORCE ROW LEVEL SECURITY; +CREATE POLICY org_isolation ON files + USING (org_id = (SELECT current_setting('app.current_org', true))); +``` + +The app connects as a non-owner role. Migrations run as admin and bypass RLS. + +### B2 Request identity + +- Declare `App.Locals` in `app.d.ts`: + +```ts +interface Locals { + auth: null | { + userId: string; + orgId: string; + role: string; + via: 'session' | 'api-key'; + scope: 'read-only' | 'read-write'; + }; +} +``` + +- `hooks.server.ts` gains the identity step after origin gating. Session cookies and `adr_` bearer keys both resolve to the same `locals.auth` shape. +- New `CurrentOrg` service in `requestLayer`, provided from `locals.auth`. Every service that runs SQL yields it and adds `WHERE org_id = ${org.id}`. With RLS, the request layer also runs `SELECT set_config('app.current_org', $1, true)` at the start of each transaction. +- Refactor the ~30 `authorizeRequest` and `authorizeWriteRequest` calls to read `locals.auth` and check scope only. The origin-pin logic at `services/auth.ts:297` moves into the hook. + +### B3 WorkOS + +Use `@workos-inc/node` v10 directly. It runs on workerd without polyfills. The official `@workos/authkit-sveltekit` is 0.3.0 on a prerelease core, so hand-roll the roughly 60 lines instead and revisit when it stabilizes. + +- Routes: `/auth/sign-in` redirects to `getAuthorizationUrl({ provider: 'authkit' })`, `/auth/callback` exchanges the code, `/auth/sign-out` clears the cookie and redirects to the WorkOS logout URL. +- Callback: + +```ts +const { sealedSession, user, organizationId } = + await workos.userManagement.authenticateWithCode({ + clientId, + code, + session: { sealSession: true, cookiePassword: env.WORKOS_COOKIE_PASSWORD } + }); +``` + +- WorkOS does not create a personal org automatically. On first callback with no `organizationId`, create one, create the membership with role `owner`, refresh the session with that org, upsert `users` and `orgs`, and insert `org_usage`. +- Per-request validation in the hook: `loadSealedSession(...).authenticate()` verifies the JWT locally against cached JWKS, no API call. Only on `invalid_jwt` call `refresh()`, which is one API call and rotates the refresh token. Access tokens default to 5 minutes. +- Cookie is 2 to 3.5 KB. Under the 4 KB limit. Do not add JWT template claims without checking. +- Delete: `PASSCODE`, `credential_state`, `dashboard_sessions`, `enforcePasscodeRotation`, the login page. The cron HMAC in `cloudflare-adapter.mjs:44` gets its own `MAINTENANCE_SECRET`. +- Webhooks: `user.deleted` and `organization_membership.deleted` to `/api/webhooks/workos`, verified with `workos.webhooks.constructEvent`. Everything else is upserted in the callback. + +WorkOS is free to 1M monthly active users. + +### B4 API keys and device codes per org + +- `api_keys` and `device_codes` carry `org_id` and `user_id`. Key lookup by prefix is unchanged. +- The device approve page shows which org the key will land in. If a user has several orgs, it offers a choice. +- MCP auth in `mcp/auth.ts` reads `locals.auth` like everything else. +- The CLI does not change. The `adr_` format, the device flow, and every endpoint keep their shape. + +### B5 through B7 Scope every query + +Three PRs, one per cluster, each adding the org predicate and updating tests: + +- B5 files, tags, search, indexing +- B6 sites and the content read path +- B7 lifecycle sweeps, which become `... WHERE org_id IN (SELECT id FROM orgs WHERE trust <> 'suspended') ORDER BY random() LIMIT n` so no org starves another. Stack D replaces most of this anyway. + +### B8 Per-org quota + +- `storage-quota.ts` reads `org_usage.stored_bytes` instead of summing every version. +- The upload transaction does `UPDATE org_usage SET stored_bytes = stored_bytes + $1 WHERE org_id = $2 AND stored_bytes + $1 <= $3 RETURNING stored_bytes`. Zero rows means over quota. That closes the concurrent-upload race the current comment accepts. +- Deletes, purges, and site commits decrement in their own transactions. +- Limit comes from `orgs.plan` for now. Stack F replaces that with an Autumn check. +- Grant signing stays on one global key in `instance_secrets`. Add `orgId` to the signed payload so a grant for one org cannot be replayed against another. + +Deployable at the top. Second checkpoint: you can sign up. + +## Stack C: Content origin and subdomains + +Goal: `./f/`. + +### C1 Wildcard content domain + +- Buy a throwaway domain for content. Never serve tenant content from the brand domain. +- DNS: one proxied `AAAA *. 100::` record. Universal SSL covers first-level wildcards, so keep slugs at one level. Route `*./*` to the worker. +- `CONTENT_ORIGIN` becomes `CONTENT_DOMAIN`. `host-gate.ts` reads the slug from `Host`, lowercases it, resolves the org id from a small KV cache in front of `orgs`, and puts it on `locals`. +- `/f`, `/s`, `/t` handlers gain `AND org_id = $slugOrg`. A file id from another org on this host is a 404. +- `content-cache.ts` cache keys already include the full URL, so per-host caching is free. +- Dashboard link generation and the CLI's `discoverContentOrigin` in `packages/cli/src/config.ts:32` receive the org's content origin from `/api/files`, which is how the CLI already learns it. + +### C2 Slugs + +- Slug chosen at signup, editable once per 30 days, reserved-word list, 3 to 32 chars, lowercase alphanumeric plus hyphen. +- Old slugs are retained for 30 days as redirects. + +### C3 Custom domains + +Later. Cloudflare for SaaS gives 100 hostnames free then $0.10 each. One Worker route `*/*` covers all of them. Tenant resolution from `Host` already exists after C1, so this PR is the custom hostname API calls and a settings page. + +## Stack D: Queues + +Goal: indexing and cleanup are event-driven. Cron only reconciles. + +### D1 Queue binding and consumer export + +```jsonc +"queues": { + "producers": [{ "queue": "adrive-jobs", "binding": "JOBS" }], + "consumers": [{ + "queue": "adrive-jobs", + "max_batch_size": 5, + "max_batch_timeout": 5, + "max_retries": 5, + "max_concurrency": 10, + "dead_letter_queue": "adrive-jobs-dlq" + }] +} +``` + +- `cloudflare-adapter.mjs` facade adds `queue(batch, env, ctx)` next to `scheduled`. It calls `runWorkerProgram(env, ...)` from `edge.ts:163` directly, no self-fetch. +- Job envelope is a discriminated union in `packages/shared`: + +```ts +type Job = + | { kind: 'index'; orgId: string; fileId: string; version: number } + | { kind: 'purge'; orgId: string; fileId: string } + | { kind: 'scan'; orgId: string; fileId: string; version: number } + | { kind: 'site-cleanup'; orgId: string; sessionId: string }; +``` + +### D2 Indexing via queue + +- Upload commit sends `{ kind: 'index', ... }` after the transaction commits, never inside `waitUntil`. +- Consumer calls `indexing.runOne(job)`. Idempotency key is `(file_id, version)`. If the version is no longer current, ack and skip. +- Transient failures call `msg.retry({ delaySeconds: 30 ** attempts })`. Permanent ones ack and record the failure on the row, replacing `MAX_INDEX_ATTEMPTS`. +- Cron `indexing.runDue` becomes a reconciliation sweep that re-enqueues rows stuck in `pending` for more than 15 minutes. + +### D3 Purges and cleanup via queue + +- Trash sends a `purge` job with `delaySeconds` equal to the retention window, up to the 24 hour cap. Longer retentions re-enqueue. +- Site session expiry sends `site-cleanup` with the session TTL as delay. +- Cron shrinks to: reconcile stuck indexes, reconcile stuck purges, expire sessions the queue missed. + +### D4 Dead letter handling + +- A tiny DLQ consumer that writes the job and error to a `failed_jobs` table and pings a webhook. Admin surface in stack E lists it. + +Queues cost $0.40 per million operations after the first million. A delivery is three operations. At 10k users this is a few dollars a month. + +## Stack E: Abuse + +Goal: one bad tenant cannot take the domain down, and you can stop them in under a minute. + +### E1 Rate limit bindings + +```jsonc +"ratelimits": [ + { "name": "RL_UPLOAD", "namespace_id": "1001", "simple": { "limit": 60, "period": 60 } }, + { "name": "RL_PUBLISH", "namespace_id": "1002", "simple": { "limit": 10, "period": 60 } }, + { "name": "RL_AUTH", "namespace_id": "1003", "simple": { "limit": 10, "period": 60 } }, + { "name": "RL_ANON", "namespace_id": "1004", "simple": { "limit": 300, "period": 60 } } +] +``` + +- Keyed by `orgId` for upload and publish, by `userId` for auth, by client IP for anonymous content fetches. +- Replaces the KV-based `auth-guard.ts` counters. Passcode lockout goes away with the passcode. +- Bindings are per-colo and approximate. That is fine for abuse, not for billing. + +### E2 Trust levels + +- `orgs.trust`: `new` at signup, `verified` after email verification, `established` after 14 days plus a paid plan or an admin bump, `suspended`. +- Gates: `new` cannot make anything public or publish sites. `verified` can, with lower publish limits and every publish scanned before it goes live. `established` publishes immediately and scans after. +- The free plan is private-only by default. Public sharing is a feature flag on paid plans in stack F, or an admin grant. + +### E3 Scan pipeline + +- On visibility change to public, and on site commit, send a `scan` job. Content stays private until the verdict arrives for `new` and `verified` orgs. +- Scanner does, in order: sha256 against a known-bad hash list mirrored to R2 nightly, MIME sniff versus declared type, and for HTML and sites a Cloudflare URL Scanner submission with the verdict polled by a delayed re-enqueue. +- `scan_verdicts (file_id, version, verdict, source, details, created_at)`. `malicious` flips visibility back to private, sets `files.quarantined`, and notifies the owner. Never silently drop. +- Web Risk lookup for outbound links inside published HTML is optional. First 100k calls a month are free, then $0.50 per thousand. +- ClamAV in a Cloudflare Container is a later PR on this stack. The pattern is a Container running clamd fed by a presigned R2 URL, standard-1 instance, signatures baked into the image. + +### E4 Report and kill switch + +- Report button on every public file page and site 404 page posts to `/report` on the content origin, rate limited by IP, stored in `reports`. +- Kill switch: `UPDATE orgs SET trust = 'suspended'`. The host gate returns 404 for every path on that slug, the KV org cache is purged, API keys stop resolving, and the edge cache for that host is purged via the Cloudflare API. +- Reversible from the same admin page. + +### E5 Admin surface + +- `/admin` on the dashboard origin, gated by a WorkOS role `admin` on a dedicated admin org, or an allowlist of user ids in a secret. +- Pages: report queue, failed jobs, org lookup with usage and trust, suspend and restore. + +### E6 Zone rules + +Not code. A checklist in `docs/abuse.md`: WAF managed rules on the content zone, bot fight mode, hotlink protection off since that is the point, and the Netcraft hosting provider abuse feed registration. + +## Stack F: Billing with Autumn + +Goal: plans, Stripe checkout, and usage gates without writing billing code. + +### F1 Config as code + +```ts +// autumn.config.ts +export const storageBytes = feature({ + id: 'storage_bytes', + type: 'metered', + consumable: false +}); +export const aiOps = feature({ + id: 'ai_ops', + type: 'metered', + consumable: true +}); +export const publicShare = feature({ id: 'public_sharing', type: 'boolean' }); + +export const free = plan({ + id: 'free', + autoEnable: true, + items: [ + featureItem({ featureId: storageBytes.id, includedUsage: 2 * 2 ** 30 }), + featureItem({ featureId: aiOps.id, includedUsage: 500, interval: 'month' }) + ] +}); +export const pro = plan({ + id: 'pro', + items: [ + priceItem({ price: 8, interval: 'month' }), + featureItem({ featureId: storageBytes.id, includedUsage: 100 * 2 ** 30 }), + featureItem({ + featureId: aiOps.id, + includedUsage: 10_000, + interval: 'month' + }), + featureItem({ featureId: publicShare.id }) + ] +}); +``` + +- `atmn push` to sandbox, `atmn push -p` to production. +- `customers.getOrCreate({ customerId: orgId })` in the org creation path from B3. Customer id is the org id. + +### F2 Gates + +- `storage-quota.ts` calls `autumn.check({ customerId, featureId: 'storage_bytes', requiredBalance: incomingBytes })` and after the transaction `balances.update({ usage: newTotal })`. The local `org_usage` row stays as the fast path and the source of truth. Autumn is synced, not queried, on the hot path. +- Indexing tracks `ai_ops` per chunk batch. +- Publishing checks `public_sharing`. +- `failOpen` is on by default in `autumn-js`. Keep it on for `check`, and rely on the local quota as the hard stop. + +### F3 Billing page + +- `billing.attach({ customerId, planId: 'pro', redirectMode: 'if_required' })` returns a checkout URL. +- `billing.openCustomerPortal` for cards, invoices, and cancellation. +- Usage bars from the local `org_usage` row. + +### F4 Webhook + +- `billing.updated` via Svix to `/api/webhooks/autumn`. Updates `orgs.plan` so trust and abuse gates can read it without an API call. + +Autumn is free under $8k monthly billing volume. Stripe fees apply as usual. + +## Stack G: Ops + +- G1 Backups: PlanetScale automatic backups are on by default. Enable R2 object versioning or a nightly copy to a second bucket. Delete the home-machine rclone docs. +- G2 Observability: Workers Logs with `orgId` on every log line, an Analytics Engine dataset for per-org request and byte counts, alerts on DLQ depth and 5xx rate. +- G3 Launch checklist: status page, terms and privacy, DMCA agent registration, abuse contact on the content domain's WHOIS, and the `docs/release.md` rewrite. + +## What changes in the cost model + +From the earlier estimate, the deltas at 1,000 users: + +| Line | Before | After | +| ----------- | ------ | ----------------------------------------------- | +| Vectorize | $130 | $0 | +| D1 | $34 | $0 | +| PlanetScale | $0 | $30 for PS-10 HA, plus $0.125 per GB past 10 GB | +| Queues | $0 | roughly $3 | +| Autumn | $0 | $0 under $8k monthly billing | +| WorkOS | $0 | $0 under 1M MAU | + +Postgres storage is the thing to watch. Extracted text plus vectors plus the HNSW index for 1,000 users is on the order of 50 GB, or about $5 a month at the overage rate. Fine. + +## Open questions to settle before stack A starts + +- Which PlanetScale region. Pick the one closest to most users, then set Hyperdrive placement to match. +- Whether to keep the `apps/site` marketing app in this repo or move it. It is untouched by every stack above. +- Whether the free plan gets any public sharing at all, or whether that is strictly the paid feature. It changes E2 and F1 together.