diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md new file mode 100644 index 00000000..e0aa2dce --- /dev/null +++ b/.claude/skills/verify/SKILL.md @@ -0,0 +1,121 @@ +--- +name: verify +description: Prove a change to sql-fs works the way an API, MCP, or SDK client sees it. Picks the gates a change needs (pnpm check, verify:local, verify:architecture, single suites) and drives a persistent disposable Postgres/Redis/API stack with curl-level HTTP, MCP tool calls, and SQL inspection, keeping evidence. Use before handing back any change, when asked to verify/prove/test behavior end to end, or when a full verify:local run is too coarse to debug a failure. +--- + +# Verify sql-fs + +sql-fs is an HTTP + MCP service. Its users are API clients, MCP clients, and the TypeScript/Python SDKs, so proof means real requests against a real server backed by real Postgres and Redis. This skill adds two things on top of the repo's gates: + +1. A **change → gate map** so you run the right checks without asking. +2. A **persistent disposable stack** (`scripts/stack.sh`) so you can drive one endpoint, rerun one suite, or inspect the database while you iterate, instead of paying for a full `pnpm verify:local` on every attempt. + +All paths below are relative to the repo root. `S=.claude/skills/verify/scripts/stack.sh`. + +## Choose the gates + +Run every row your diff touches. `pnpm check` is always required; the full verify command named on a row is the completion evidence, and the stack is how you iterate toward it. For any runtime change, the default loop is: `$S up` → edit → `$S restart` → `$S doctor` → the row's drive/suite → `journeys.sh` → the completion gate → `$S down`. + +| Diff touches | Iterate with | Completion evidence | +| --- | --- | --- | +| Docs, comments, `.changeset/` only | — | `pnpm check` | +| `src/sql-fs/` (fs, transactions, cache, RLS, symlinks) | targeted `pnpm exec vitest run `, `$S suite integration `, [files](features/files.md) / [exec](features/exec.md) drive | `pnpm check` + `pnpm verify:local` | +| `src/api/routes/`, `auth.ts`, `validation.ts`, `openapi-spec.ts` | the matching feature drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| `src/api/mcp/` | [mcp](features/mcp.md) drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| `src/api/worker-owner/`, `ingress-admission.ts`, `session-manager.ts`, `src/execution/`, `script-scope.ts`, budgets, cancellation, deadlines, body limits | `$S suite worker-failures`, `sse-finalization`, `input-budgets`, `output-cap` | `pnpm check` + `pnpm verify:architecture` | +| `src/api/server.ts` startup/shutdown, `src/api/mcp/server.ts` listener lifecycle, ingress waits | `$S suite shutdown`, `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| Tenancy, `TENANT_DATABASES`, owner isolation | `$S suite tenants`, [sandboxes](features/sandboxes.md) isolation steps | `pnpm check` + `pnpm verify:architecture` | +| `src/redis/`, `redis-blob-cache.ts` | `$S suite integration src/redis` | `pnpm check` + `pnpm verify:local` | +| `src/sql-fs/migrations/postgres/` | `$S down && $S up` (fresh DB runs every migration), `$S psql '\d '` | `pnpm check` + `pnpm verify:local` | +| `clients/typescript/` or `clients/python/` only | `pnpm check:typescript` / `pnpm check:python`, `$S suite sdk-clients` | `pnpm check` + `pnpm verify:local` | +| `package.json`, lockfiles, `patches/` | — | `pnpm check` + `pnpm verify:local` + `pnpm security:audit` | + +Also, from CLAUDE.md: add `.changeset/*.md` for behavior changes, update `src/api/openapi-spec.ts` when an API contract changes, and report every command with its result. A skipped test is not proof. + +## Launch + +Preconditions: Docker daemon running, `pnpm setup:dev` done (needs `node_modules` and `clients/python/.venv`), and `jq` on `PATH`. `journeys.sh` and the feature recipes need `jq`. + +```bash +$S up # compose stack + pnpm build + API server; ~10s +$S up --no-build # reuse the current dist/ +``` + +`up` starts `infra/docker-compose.verify.yml` under a unique Compose project with tmpfs Postgres and random loopback ports. It builds, then starts `node dist/api/server.js` with a clean environment (`env -i`, so `.env` and your shell's deployment variables are ignored) on a random port. The server applies every migration at startup. Ready means `GET /readyz` returns `{"status":"ok"}`, and the server only gets there after migrations and Redis connect. If the server exits first, `up` prints the tail of `server.log` and fails. + +`up` refuses while a stack is recorded for this checkout. Reuse it (`$S doctor`) or `$S down` first. After a failed `up`, run `$S down` too, because the containers may be running. + +Server environment overrides: `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false SQL_FS_WORKER_OUTPUT_BYTES=1048576" $S restart`. Overrides persist across later restarts until you pass `SERVER_ENV= $S restart`. Values are split on spaces, so use compact JSON with no spaces. `server.log` and `doctor` show only the override names; the values stay in the owner-only `state.env`. + +## Doctor + +Run `$S doctor` first, and again whenever anything looks off. It is read-only and exits non-zero on any `FAIL`: + +- all three services `healthy` +- the recorded PID is alive and is `node dist/api/server.js` +- that PID owns the recorded port (via `lsof`) +- `/readyz` ok and a freshly minted token gets `200` on `GET /v1/sandboxes` +- `node_modules` matches `pnpm-lock.yaml`, compared byte-for-byte with `node_modules/.pnpm/lock.yaml`. After a pull that changed the lockfile, the old install keeps running and unit tests fail for reasons unrelated to your diff. Fix it with `pnpm install --frozen-lockfile && $S restart`. `up` refuses to start on drift. +- `dist/` is not older than any non-test `src/**/*.ts` or `src/**/*.sql`, so a migration edit also fails until you restart. **A stale build is the most common false result**: you edited source, but the server and every suite still run the old `dist/`. Fix it with `$S restart`. + +## Drive + +After every source edit: `$S restart` (rebuild + restart, same port, data kept), then `$S doctor`. + +**Regression sweep:** `.claude/skills/verify/scripts/journeys.sh [auth|sandboxes|exec|files|ingest|mcp ...]` runs every journey in the feature map against the stack and asserts the documented result. With no arguments it runs all six, about 40 assertions in roughly 10 seconds. It exits non-zero at the first mismatch and prints the expected and actual values. Run it after any change to `src/api/` or `src/sql-fs/` before the full gate. It is the executable form of `features/*.md`: when you add or change a journey there, change the matching `journey_` function in the same commit. It is iteration evidence, not a gate. CI and `verify:local` do not run it. + +| Command | What it does | +| --- | --- | +| `$S api METHOD PATH [curl args]` | Authenticated request. Body goes to stdout and `HTTP ` to stderr, and a record is appended to `transcript.jsonl`. Exits non-zero unless the status is 2xx, or equals `EXPECT=`. `AS=` picks the caller (default `verify-agent`), `AS=none` sends no token, `TENANT=` adds a tenant claim. | +| `$S mcp list` / `$S mcp TOOL ''` | MCP over Streamable HTTP with the same `AS`/`TENANT`. Exits non-zero on `isError` or a `{"ok":false}` payload. Use `EXPECT=error` to assert one of those. `bash_exec` denials are neither; assert their `exitCode`/`stderr`. | +| `$S token [sub] [tenant]` | Print a 24h JWT signed with the stack's `AUTH_SECRET`. | +| `$S psql 'SQL'` | Run SQL as the `postgres` superuser on the stack DB. This **bypasses RLS** and is only for inspecting side effects; never use it to prove isolation. Tables: `sandboxes`, `inodes`, `dirents`, `blobs`, `sandbox_epochs`. | +| `$S suite NAME [args]` | Run the suite entry point `scripts/verify/NAME.mjs` against the stack's Postgres and Redis. Suites start their own API servers from `dist/`, so a passing suite does not prove the `$S up` server; restart (build) first. An entry point is a non-test file that creates a `LocalApiRun`; helper modules such as `local-api.mjs` or `measurements.mjs` are rejected with the list of entry points. `NAME=integration [file filters]` runs vitest integration and writes a JSON report. | +| `$S env` | Print the recorded state (URLs, port, PID, run dir). | + +Use `curl --json ''` for JSON (sets both headers). Use `--data-binary @file -H 'Content-Type: application/octet-stream'` for raw file bytes. + +The feature map in [features/](features/README.md) has the exact journeys. Always assert the observable result (status, body, a second read through another path). Don't settle for "it returned 200". + +## Evidence + +Everything lands in `scripts/verify/.run/stack-/` (gitignored) and survives `down`: + +- `transcript.jsonl`: every `api` call with caller, method, path, curl args, and HTTP status, and every `mcp` call (including `list`) with caller, tool, args, and whether it failed. Each record keeps the response body up to 4 KiB; a longer body is cut, marked `truncated: true`, and `bytes` keeps the full size. The stack secret, JWTs, values of token/secret/password/authorization keys (also inside JSON strings such as `--json` args and MCP text parts), and sensitive header values are redacted. +- `server.log`: server stdout/stderr across restarts, with a `--- restart` line at each one. +- `services.log`: Postgres/Redis logs, written at `down`. +- `suites/-/proof.json` and `requests.jsonl`, plus `suites/integration-*.json`. +- `state.env`: the stack's coordinates. It contains the local-only verification secret and any `SERVER_ENV` values, so the run directory is owner-only. + +`pnpm verify:local` writes its own `scripts/verify/.run/verify-*/` directory with `proof.json` per suite and `integration.json`. + +Proof standards: + +- Drive the public path a client uses (`/v1/*`, `/mcp`, the SDKs). Never call internals or test-only hooks to make a journey pass. +- For mutations, capture the action and an independent read: write via REST, read via bash or MCP, or the reverse. Then confirm the side effect with `$S psql` where it matters. +- For isolation, prove the denial (`AS=intruder EXPECT=403`) through the API, not through SQL. +- If a precondition is missing, report the journey as failed with the reason. Never weaken the journey to make it pass. +- Report what you ran and what you saw: commands, statuses, evidence paths. Name anything you did not verify. + +## Cleanup + +```bash +$S down +``` + +`down` stops only the recorded PID, and only if it still runs `dist/api/server.js`. It saves service logs, removes this run's containers and volumes (`down --volumes`), and clears the pointer. Evidence stays on disk. Never `pkill node` or `docker rm` by pattern, because other worktrees and the user's own stacks may be running. Delete old `scripts/verify/.run/stack-*` directories only when the user asks. + +## Isolation + +- One recorded stack per checkout (pointer: `scripts/verify/.run/current-stack`). Separate worktrees get separate stacks and never collide, since project names and ports are unique. +- `STACK_DIR=scripts/verify/.run/stack- $S doctor` targets a specific recorded stack. +- `pnpm verify:local` and `verify:architecture` start their own fresh stack and can run while yours is up. +- Never point the helper or suites at a shared or non-disposable database. `tenants`, `worker-failures`, and the integration suite create databases and inject faults. + +## Gotchas + +- zsh has no `PIPESTATUS`; use `${pipestatus[1]}`, or don't pipe when you need the exit code. +- `exec-sync` timeouts return `408 EXEC_TIMEOUT` but commit the completed prefix. Assert both. +- MCP tool errors use three shapes (`src/api/mcp/tool-errors.ts`): worker errors set `isError:true`; file and sandbox tools return `{"ok":false,"error":...}` with `isError:false`; `bash_exec` returns `{"exitCode":1,"stderr":"forbidden"}`. The helper catches the first two. For bash tools, assert the fields. +- `PATCH /v1/sandboxes/:id` accepts only `python`, `javascript`, and `network`. A `name` is rejected with 400. +- The integration suite takes about 2.5 minutes. Narrow it with a path filter while iterating, then run it whole. diff --git a/.claude/skills/verify/features/README.md b/.claude/skills/verify/features/README.md new file mode 100644 index 00000000..ff77b8b4 --- /dev/null +++ b/.claude/skills/verify/features/README.md @@ -0,0 +1,51 @@ +# sql-fs verification map + +This directory is the maintained source for verifying what sql-fs clients can observe. Read this index, then use the matching feature file as the recipe. When a change adds or alters a client-visible path, update its feature file in the same change. + +## Baseline preconditions + +- `S=.claude/skills/verify/scripts/stack.sh` +- `$S up` has completed and `$S doctor` exits 0 with no `FAIL` lines. +- Run all commands from the repo root. +- Never drive a server or database that this checkout's `$S up` did not start. + +## Driving conventions + +- `.claude/skills/verify/scripts/journeys.sh ` runs a feature's journey with assertions. Keep it in step with these files. +- Create the journey's own sandbox and capture its ID: `ID=$($S api POST /v1/sandboxes --json '{"name":"verify"}' | jq -r .id)`. +- The default caller is `verify-agent`. Use `AS=intruder` for a second owner and `AS=none` for no token. +- Assert with `EXPECT=` (the helper exits non-zero on mismatch) and inspect the body with `jq`. +- Prove a mutation with a second, independent read: a REST read, a bash `cat`, an MCP `file_read`, or `$S psql`. +- Delete the sandbox at the end with `$S api DELETE /v1/sandboxes/$ID`. Evidence stays in the run directory. + +## Proof and skip reporting + +- Cite `transcript.jsonl` lines or suite `proof.json` paths for every claim. +- Report an unreachable journey with the command and the unmet precondition. +- Proving one entry point (REST) does not prove another (MCP, SDK). List which ones you drove. + +## Feature entry contract + +Each feature file has an H1, one paragraph of client-visible behavior, then exactly four H2s: `Sub-features`, `How to get to it (user POV)`, `Driving it with stack.sh`, and `Gotchas`. + +## Features + +- [Auth](./auth.md) covers bearer tokens, bootstrap with `X-Auth-Secret`, and unauthenticated rejection. +- [Sandboxes](./sandboxes.md) covers create with seed files, get, list, runtime update, owner isolation, and delete. +- [Bash execution](./exec.md) covers `exec-sync`, SSE `exec`, batches, read-only mode, and cooperative timeouts. +- [Files](./files.md) covers raw read and write, edit, bulk write, mkdir, tree, and delete. +- [Ingest](./ingest.md) covers base64 manifest upload and archive export through bash. +- [MCP](./mcp.md) covers tool listing, file and bash tools, and cross-transport visibility. + +## Covered only by suites + +These have no hand-driven journey yet. Run the suite through `$S suite `, or as part of `pnpm verify:architecture`. + +- SDKs (TypeScript and Python, live): `sdk-clients` +- Multi-tenant routing and isolation: `tenants` +- Worker crash, lease loss, and unknown-outcome errors: `worker-failures` +- SSE stream finalization: `sse-finalization` +- Request and input budgets: `input-budgets` +- Output caps: `output-cap` +- Graceful shutdown with open MCP streams: `shutdown` +- Full HTTP/MCP contract against `/openapi.json`: `contracts` diff --git a/.claude/skills/verify/features/auth.md b/.claude/skills/verify/features/auth.md new file mode 100644 index 00000000..322de036 --- /dev/null +++ b/.claude/skills/verify/features/auth.md @@ -0,0 +1,33 @@ +# Auth + +Every `/v1/*` and `/mcp` call carries a bearer JWT signed with `AUTH_SECRET`, with two exceptions. `POST /v1/auth/bootstrap` takes no bearer token: a caller proves it holds the secret with `X-Auth-Secret` and gets a token back. When `MCP_API_KEY` is set, `/mcp` also accepts that static key as the bearer value and takes the owner from an identity header. Requests without a valid token are rejected before they touch any sandbox. + +## Sub-features + +- `auth-required` rejects `/v1` requests without a bearer token with `401 AUTH_REQUIRED`. `POST /v1/auth/bootstrap` is exempt. +- `auth-bootstrap` issues a token for a `sub` when `X-Auth-Secret` matches. +- `auth-bootstrap-denied` rejects a wrong secret with `403 FORBIDDEN` before parsing the body. +- `auth-token-cli` mints tokens offline with `pnpm token:create`. + +## How to get to it (user POV) + +- Send `Authorization: Bearer ` on any `/v1` or `/mcp` request. +- `POST /v1/auth/bootstrap` with header `X-Auth-Secret` and body `{"sub":"...","expiresIn":"24h"}`. +- `AUTH_SECRET=... pnpm token:create -- --sub agent-1 [--tenant t] --expires 30d`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md). `$S env` shows `AUTH_SECRET`. + +- **No token.** Run `EXPECT=401 AS=none $S api GET /v1/sandboxes`. The body is `{"error":"unauthorized","code":"AUTH_REQUIRED"}`. +- **Wrong secret.** Run `EXPECT=403 AS=none $S api POST /v1/auth/bootstrap -H 'X-Auth-Secret: wrong' --json '{"sub":"boot"}'`. The body has `code` `FORBIDDEN`. +- **Bootstrap.** Run `eval "$($S env | grep AUTH_SECRET)"` and then `TOKEN=$(EXPECT=201 AS=none $S api POST /v1/auth/bootstrap -H "X-Auth-Secret: $AUTH_SECRET" --json '{"sub":"boot","expiresIn":"24h"}' | jq -r .token)`. `TOKEN` is a three-part JWT. +- **Use the issued token.** Run `EXPECT=200 AS=none $S api GET /v1/sandboxes -H "Authorization: Bearer $TOKEN"`. The body is `{"sandboxes":[]}` for the fresh `boot` owner. + +## Gotchas + +- Every `POST /v1/auth/bootstrap` counts toward a per-IP limit, successful mints included: 5 per 60 seconds by default (`BOOTSTRAP_RATE_LIMIT_MAX`, `BOOTSTRAP_RATE_LIMIT_WINDOW_MS`). The `auth` journey makes two calls, so a third run within a minute gets `429`. +- `POST /v1/auth/admin` needs `ADMIN_SECRET`, which the stack does not set. It returns `500 ADMIN_NOT_CONFIGURED` unless you pass `SERVER_ENV="ADMIN_SECRET=..." $S restart`. +- A token without a `tenant` claim resolves to the `default` tenant. diff --git a/.claude/skills/verify/features/exec.md b/.claude/skills/verify/features/exec.md new file mode 100644 index 00000000..5b7c1b18 --- /dev/null +++ b/.claude/skills/verify/features/exec.md @@ -0,0 +1,36 @@ +# Bash execution + +Clients run bash scripts inside a sandbox and get stdout, stderr, and the exit code back. They can do this synchronously, as a Server-Sent Events stream, or as a batch. A script's file writes persist. Read-only mode rejects mutations, and a timeout stops the script while keeping the work it already committed. + +## Sub-features + +- `exec-sync` returns `{exitCode, stdout, stderr}` and persists writes. +- `exec-sse` streams `stdout` events and exactly one `exit` event. +- `exec-batch` runs ordered scripts and returns results keyed by `id`. +- `exec-readonly` rejects mutation with `422 EREADONLY_VIOLATION` and leaves no file. +- `exec-timeout` returns `408 EXEC_TIMEOUT` and keeps the completed prefix. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes/:id/exec-sync`, `/exec` (SSE), `/exec-sync-batch`. +- MCP: `bash_exec`, `bash_exec_batch`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Sync run and persistence.** Run `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo hi > /home/user/a.txt; cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}'`. The output is `{"exitCode":0,"stdout":"hi\n"}`. Then `$S api GET /v1/sandboxes/$ID/files/home/user/a.txt` prints `hi`. +- **Read-only rejection.** Run `EXPECT=422 $S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo x > /home/user/ro.txt","readOnly":true}'`. The `code` is `EREADONLY_VIOLATION`. Then `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/ro.txt` confirms no file was written. +- **SSE stream.** Run `$S api POST /v1/sandboxes/$ID/exec --json '{"script":"printf stream-proof"}'`. The output contains `event: stdout` with `"data":"stream-proof"` and exactly one `event: exit` with `"exitCode":0`. +- **Batch.** Run `$S api POST /v1/sandboxes/$ID/exec-sync-batch --json '{"scripts":[{"id":"a","script":"echo one"},{"id":"b","script":"echo two"}]}' | jq -c '[.results[]|[.id,.exitCode,.stdout]]'`. The output is `[["a",0,"one\n"],["b",0,"two\n"]]`. +- **Timeout prefix.** Run `EXPECT=408 $S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo p > /home/user/p.txt; sleep 2; echo s > /home/user/s.txt","timeoutMs":100}'`. The `code` is `EXEC_TIMEOUT`. Then `$S api GET /v1/sandboxes/$ID/files/home/user/p.txt` prints `p` and `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/s.txt` succeeds. +- **Worker modes.** Repeat the sync run after `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false" $S restart --no-build`, then restore with `SERVER_ENV= $S restart --no-build`. Both modes must give identical results. + +## Gotchas + +- A non-zero script exit is still HTTP `200`. Assert `exitCode`, not just the status. +- Python and JavaScript commands need the runtime enabled (`PATCH {"python":true}`). The first `python3` call can take tens of seconds. +- Output above `SQL_FS_WORKER_OUTPUT_BYTES` fails with exit `126`. `$S suite output-cap` covers both worker modes. +- The outcome of an `exec` or `execBatch` is unknown after `503` with code `EEXEC_OUTCOME_UNKNOWN`, `EDRIVERFAULT`, or `ECOHERENCE` (writes may have been applied), or after an SDK `TransportError` starting with `network error after` or `script outcome unknown`. Never retry those automatically. Read the sandbox state before running anything else. `worker-failures` covers them. diff --git a/.claude/skills/verify/features/files.md b/.claude/skills/verify/features/files.md new file mode 100644 index 00000000..aed9cd8a --- /dev/null +++ b/.claude/skills/verify/features/files.md @@ -0,0 +1,37 @@ +# Files + +Clients read and write sandbox files directly, without bash. They can write raw bytes, edit a string in place, write many files at once, create directories, list a tree, and delete. Every change is visible to bash and to other replicas. + +## Sub-features + +- `files-put-get` writes raw bytes (`204`) and reads them back exactly. +- `files-edit` replaces `oldString` with `newString` and reports `replacements`. +- `files-bulk` writes several files with `writeFiles` (`204`). +- `files-mkdir` creates directories recursively (`204`). +- `files-tree` lists entries under a prefix to a given depth. +- `files-delete` removes a file (`204`), after which reads return `404 ENOENT`. + +## How to get to it (user POV) + +- REST: `GET|PUT|PATCH|DELETE /v1/sandboxes/:id/files/`, `POST /v1/sandboxes/:id/writeFiles`, `POST /v1/sandboxes/:id/mkdir`, `GET /v1/sandboxes/:id/tree?prefix=&depth=`. +- MCP: `file_read`, `file_write`, `file_edit`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Mkdir.** Run `EXPECT=204 $S api POST /v1/sandboxes/$ID/mkdir --json '{"path":"/home/user/v","recursive":true}'`. +- **Raw write and read.** Run `EXPECT=204 $S api PUT /v1/sandboxes/$ID/files/home/user/v/note.txt --data-binary $'before\n' -H 'Content-Type: application/octet-stream'`, then `$S api GET /v1/sandboxes/$ID/files/home/user/v/note.txt`. It prints `before`. +- **Edit.** Run `$S api PATCH /v1/sandboxes/$ID/files/home/user/v/note.txt --json '{"oldString":"before","newString":"after"}' | jq .replacements`. It prints `1`, and a second GET prints `after`. +- **Bulk write, seen by bash.** Run `EXPECT=204 $S api POST /v1/sandboxes/$ID/writeFiles --json '{"files":{"/home/user/v/one.txt":"one\n","/home/user/v/two.txt":"two\n"}}'`, then `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"cat /home/user/v/one.txt /home/user/v/two.txt","readOnly":true}' | jq -r .stdout`. It prints `one` and `two`. +- **Tree.** Run `$S api GET "/v1/sandboxes/$ID/tree?prefix=/home/user/v&depth=3" | jq -c '[.[].path]'`. The output lists `note.txt`, `one.txt`, and `two.txt` under `/home/user/v/`. +- **Delete.** Run `EXPECT=204 $S api DELETE /v1/sandboxes/$ID/files/home/user/v/note.txt`, then `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/v/note.txt`. The body is `{"error":"not_found","code":"ENOENT"}`. + +## Gotchas + +- The path in `/files/` has no leading slash after `files/`. The server treats it as absolute. +- Quote URLs that contain `?` or `&`, or the shell will mangle the request. +- Null bytes and path traversal are rejected. Symlinks are allowed by default, with bounded resolution. +- Write bodies have size limits (`MAX_BULK_WRITE_BYTES` and others). See `docs/CONFIGURATION.md` before writing large-file journeys, and `scripts/verify/large-files.mjs` for profiling. diff --git a/.claude/skills/verify/features/ingest.md b/.claude/skills/verify/features/ingest.md new file mode 100644 index 00000000..5e75f25d --- /dev/null +++ b/.claude/skills/verify/features/ingest.md @@ -0,0 +1,30 @@ +# Ingest + +Clients upload a tree of files in one request as a JSON manifest of base64 contents under a `basePath`. They can get the tree back out as an archive by running `tar` in bash. + +## Sub-features + +- `ingest-manifest` writes every manifest entry under `basePath` and reports `fileCount`. +- `ingest-nested` creates nested directories from manifest keys. +- `ingest-export` round-trips the tree as a `tar.gz` produced by bash. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes/:id/ingest-files` with `{"basePath":"...","files":{"rel/path":""}}`. +- MCP: `fs_ingest`, `fs_export`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Upload.** Run `$S api POST /v1/sandboxes/$ID/ingest-files --json "{\"basePath\":\"/home/user/in\",\"files\":{\"g.txt\":\"$(printf 'ingested\n' | base64)\",\"nested/two.txt\":\"$(printf 'two\n' | base64)\"}}"`. The body is `{"status":"ok","fileCount":2}`. +- **Bytes landed.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/in/g.txt` and `$S api GET /v1/sandboxes/$ID/files/home/user/in/nested/two.txt`. They print `ingested` and `two`. +- **Export round-trip.** Run `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"tar -czf - -C /home/user/in . | base64 -w 0","readOnly":true}' | jq -r .stdout | base64 -d | tar -tzf -`. The listing contains `./g.txt` and `./nested/two.txt`. + +## Gotchas + +- HTTP ingest does not accept the host-filesystem `paths` form. That form is MCP-only. +- A manifest for a sandbox that does not exist returns `404 ENOENT`. +- Ingest has its own admission and byte budgets. `scripts/verify/ingest-admission.mjs` exercises them. diff --git a/.claude/skills/verify/features/mcp.md b/.claude/skills/verify/features/mcp.md new file mode 100644 index 00000000..92038cf7 --- /dev/null +++ b/.claude/skills/verify/features/mcp.md @@ -0,0 +1,35 @@ +# MCP + +MCP clients connect to `/mcp` over Streamable HTTP with the same bearer token as REST. They get tools for sandboxes, files, bash, and ingest. MCP and REST share one filesystem, so a write through one transport is visible through the other, and owner isolation applies to both. + +## Sub-features + +- `mcp-list` lists 11 tools: `sandbox_create`, `sandbox_list`, `sandbox_update`, `sandbox_delete`, `file_read`, `file_write`, `file_edit`, `bash_exec`, `bash_exec_batch`, `fs_ingest`, `fs_export`. +- `mcp-file` writes and reads files through tools. +- `mcp-bash` runs bash through tools, including `readOnly`. +- `mcp-cross-transport` makes MCP writes visible to REST, and the reverse. +- `mcp-isolation` returns `{"ok":false,"error":"forbidden"}` to other owners. + +## How to get to it (user POV) + +- Connect an MCP client to `/mcp` with `Authorization: Bearer `. +- With `MCP_API_KEY` set, a trusted proxy may authenticate with the static key plus an identity header. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **List tools.** Run `$S mcp list`. It prints the 11 tools with their argument names. +- **Write via MCP.** Run `$S mcp file_write "{\"id\":\"$ID\",\"path\":\"/home/user/mcp.txt\",\"content\":\"mcp-proof\\n\"}"`. The output is `{"ok":true,...,"size":10}`. +- **Read via REST.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/mcp.txt`. It prints `mcp-proof`. +- **Bash via MCP.** Run `$S mcp bash_exec "{\"id\":\"$ID\",\"script\":\"cat /home/user/mcp.txt\",\"readOnly\":true}"`. The output has `"stdout":"mcp-proof\n"` and `"exitCode":0`. +- **Isolation.** Run `EXPECT=error AS=intruder $S mcp file_read "{\"id\":\"$ID\",\"path\":\"/home/user/mcp.txt\"}"`. It prints `{"ok":false,"error":"forbidden"}` and exits 0 because the denial was expected. Then run `AS=intruder $S mcp bash_exec "{\"id\":\"$ID\",\"script\":\"cat /home/user/mcp.txt\"}"`. The output has `"exitCode":1` and `"stderr":"forbidden"`. + +## Gotchas + +- File and sandbox tool failures are `{"ok":false}` with `isError:false`. Without `EXPECT=error`, the helper exits non-zero on them. +- `bash_exec` reports denial as `exitCode:1` with `stderr:"forbidden"`. The helper exits 0, so assert the fields yourself. +- `bash_exec` takes `timeout`, not REST's `timeoutMs`. +- Static-key mode (`MCP_API_KEY`) is off on the stack. To enable it, restart with `SERVER_ENV="MCP_API_KEY=..."`, then read `docs/CONFIGURATION.md` for the identity header. diff --git a/.claude/skills/verify/features/sandboxes.md b/.claude/skills/verify/features/sandboxes.md new file mode 100644 index 00000000..340d24a3 --- /dev/null +++ b/.claude/skills/verify/features/sandboxes.md @@ -0,0 +1,39 @@ +# Sandboxes + +A sandbox is a persistent, owner-scoped filesystem with optional Python and JavaScript runtimes. Owners can create one with seed files, read and list their own sandboxes, toggle runtimes, and delete them. Other owners cannot see or touch them. + +## Sub-features + +- `sandbox-create` creates a sandbox, optionally with seed `files`, and returns `201` with `id`. +- `sandbox-get` returns metadata including `owner`. +- `sandbox-list` lists only the caller's sandboxes. +- `sandbox-update` sets `python`, `javascript`, and `network` to `true` or `false`. An omitted flag keeps its current value. +- `sandbox-isolation` denies other owners with `403 FORBIDDEN`. +- `sandbox-delete` removes the sandbox and its files. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes`, `GET /v1/sandboxes`, `GET|PATCH|DELETE /v1/sandboxes/:id`. +- MCP: `sandbox_create`, `sandbox_list`, `sandbox_update`, `sandbox_delete`. +- SDKs: `Client` from `sql-fs-sdk` (`clients/typescript`) and `Client` from `sqlfs` (`clients/python`). + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md). + +- **Create with seed.** Run `ID=$(EXPECT=201 $S api POST /v1/sandboxes --json '{"name":"verify","files":{"/home/user/seed.txt":"seed\n"}}' | jq -r .id)`. `ID` is a UUID. +- **Seed persisted.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/seed.txt`. The body is `seed`. +- **Metadata.** Run `$S api GET /v1/sandboxes/$ID | jq -c '{owner,name}'`. The output is `{"owner":"verify-agent","name":"verify"}`. +- **List.** Run `$S api GET /v1/sandboxes | jq '[.sandboxes[].id]'`. The output contains `$ID`. +- **Update runtime.** Run `$S api PATCH /v1/sandboxes/$ID --json '{"python":true}' | jq -c '{python,javascript,network}'`. The output is `{"python":true,"javascript":false,"network":false}`. +- **Isolation.** Run `EXPECT=403 AS=intruder $S api GET /v1/sandboxes/$ID` and `AS=intruder $S api GET /v1/sandboxes | jq '.sandboxes | length'`. The first returns `FORBIDDEN` and the second prints `0`. +- **Row stored.** Run `$S psql "select owner, python from sandboxes where id = '$ID'"`. One row reads `verify-agent | t`. +- **Delete.** Run `EXPECT=204 $S api DELETE /v1/sandboxes/$ID`, then `EXPECT=404 $S api GET /v1/sandboxes/$ID`. Then run `$S psql "select count(*) from dirents where sandbox_id = '$ID'"`, which prints `0`. + +## Gotchas + +- `PATCH` is `.strict()` and needs at least one of `python`, `javascript`, or `network`. A `name` is rejected with `400 INVALID_INPUT`. +- Seed `files` share the `writeFiles` count and byte caps. +- `psql` runs as superuser and bypasses RLS. Prove isolation through the API only. diff --git a/.claude/skills/verify/scripts/journeys.sh b/.claude/skills/verify/scripts/journeys.sh new file mode 100755 index 00000000..ecd2045d --- /dev/null +++ b/.claude/skills/verify/scripts/journeys.sh @@ -0,0 +1,119 @@ +#!/usr/bin/env bash +set -euo pipefail +repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) +cd "$repo" +S=.claude/skills/verify/scripts/stack.sh +command -v jq >/dev/null || { echo "journeys.sh needs jq on PATH" >&2; exit 2; } +doctor=$($S doctor) || { echo "$doctor"; exit 1; } +eval "$($S env)" + +passed=0 +eq() { + (($# == 3)) || { echo " FAIL eq got $# args (quoting bug): $*" >&2; exit 1; } + if [[ "$1" == "$2" ]]; then passed=$((passed + 1)); echo " ok $3" + else printf ' FAIL %s\n expected: %q\n actual: %q\n' "$3" "$2" "$1" >&2; exit 1; fi +} +has() { [[ "$1" == *"$2"* ]] && eq yes yes "$3" || eq "$1" "*$2*" "$3"; } +api() { $S api "$@" 2>/dev/null; } +mcp() { $S mcp "$@" 2>/dev/null; } +absent() { EXPECT=404 api GET "$1" >/dev/null && echo absent; } +sql_value() { $S psql "$1" | sed -n 3p | tr -d ' '; } +new_sandbox() { api POST /v1/sandboxes --json '{"name":"verify"}' | jq -r .id; } +drop() { api DELETE "/v1/sandboxes/$1" >/dev/null; } + +journey_auth() { + eq "$(EXPECT=401 AS=none api GET /v1/sandboxes | jq -r .code)" AUTH_REQUIRED "no token -> 401 AUTH_REQUIRED" + eq "$(EXPECT=403 AS=none api POST /v1/auth/bootstrap -H 'X-Auth-Secret: wrong' --json '{"sub":"boot"}' | jq -r .code)" FORBIDDEN "wrong secret -> 403" + local token + token=$(EXPECT=201 AS=none api POST /v1/auth/bootstrap -H "X-Auth-Secret: $AUTH_SECRET" --json '{"sub":"boot","expiresIn":"24h"}' | jq -r .token) + eq "$(tr -cd . <<<"$token" | wc -c | tr -d ' ')" 2 "bootstrap issues a JWT" + eq "$(EXPECT=200 AS=none api GET /v1/sandboxes -H "Authorization: Bearer $token" | jq -c .)" '{"sandboxes":[]}' "issued token is accepted" +} + +journey_sandboxes() { + local id + id=$(EXPECT=201 api POST /v1/sandboxes --json '{"name":"verify","files":{"/home/user/seed.txt":"seed\n"}}' | jq -r .id) + eq "$(api GET "/v1/sandboxes/$id/files/home/user/seed.txt")" seed "seed file persisted" + eq "$(api GET "/v1/sandboxes/$id" | jq -c '{owner,name}')" '{"owner":"verify-agent","name":"verify"}' "metadata owner and name" + eq "$(api GET /v1/sandboxes | jq --arg id "$id" '[.sandboxes[].id] | index($id) != null')" true "list contains sandbox" + eq "$(api PATCH "/v1/sandboxes/$id" --json '{"python":true}' | jq -c '{python,javascript,network}')" '{"python":true,"javascript":false,"network":false}' "runtime update" + eq "$(EXPECT=403 AS=intruder api GET "/v1/sandboxes/$id" | jq -r .code)" FORBIDDEN "other owner -> 403" + eq "$(AS=intruder api GET /v1/sandboxes | jq '.sandboxes | length')" 0 "other owner lists nothing" + has "$($S psql "select owner, python from sandboxes where id = '$id'")" "verify-agent | t" "row stored with owner and python" + EXPECT=204 api DELETE "/v1/sandboxes/$id" >/dev/null + eq "$(absent "/v1/sandboxes/$id")" absent "deleted sandbox -> 404" + eq "$(sql_value "select count(*) from dirents where sandbox_id = '$id'")" 0 "no dirents remain" +} + +journey_exec() { + local id out + id=$(new_sandbox) + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo hi > /home/user/a.txt; cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"hi\n"}' "exec-sync stdout and exit" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/a.txt")" hi "exec write persisted" + eq "$(EXPECT=422 api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo x > /home/user/ro.txt","readOnly":true}' | jq -r .code)" EREADONLY_VIOLATION "read-only rejects mutation" + eq "$(absent "/v1/sandboxes/$id/files/home/user/ro.txt")" absent "read-only left no file" + out=$(api POST "/v1/sandboxes/$id/exec" --json '{"script":"printf stream-proof"}') + has "$out" '"data":"stream-proof"' "SSE stdout event" + eq "$(grep -c '^event: exit' <<<"$out")" 1 "exactly one SSE exit" + has "$out" '"exitCode":0' "SSE exit code 0" + eq "$(api POST "/v1/sandboxes/$id/exec-sync-batch" --json '{"scripts":[{"id":"a","script":"echo one"},{"id":"b","script":"echo two"}]}' | jq -c '[.results[]|[.id,.exitCode,.stdout]]')" '[["a",0,"one\n"],["b",0,"two\n"]]' "batch results" + eq "$(EXPECT=408 api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo p > /home/user/p.txt; sleep 2; echo s > /home/user/s.txt","timeoutMs":100}' | jq -r .code)" EXEC_TIMEOUT "timeout -> 408" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/p.txt")" p "timeout kept prefix" + eq "$(absent "/v1/sandboxes/$id/files/home/user/s.txt")" absent "timeout dropped suffix" + trap 'SERVER_ENV= $S restart --no-build >/dev/null' EXIT + SERVER_ENV="SQL_FS_WORKERS_ENABLED=false" $S restart --no-build >/dev/null + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"hi\n"}' "workers disabled: same result" + SERVER_ENV= $S restart --no-build >/dev/null + trap - EXIT + drop "$id" +} + +journey_files() { + local id + id=$(new_sandbox) + EXPECT=204 api POST "/v1/sandboxes/$id/mkdir" --json '{"path":"/home/user/v","recursive":true}' >/dev/null + EXPECT=204 api PUT "/v1/sandboxes/$id/files/home/user/v/note.txt" --data-binary $'before\n' -H 'Content-Type: application/octet-stream' >/dev/null + eq "$(api GET "/v1/sandboxes/$id/files/home/user/v/note.txt")" before "raw write read back" + eq "$(api PATCH "/v1/sandboxes/$id/files/home/user/v/note.txt" --json '{"oldString":"before","newString":"after"}' | jq .replacements)" 1 "edit replacement count" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/v/note.txt")" after "edited bytes visible" + EXPECT=204 api POST "/v1/sandboxes/$id/writeFiles" --json '{"files":{"/home/user/v/one.txt":"one\n","/home/user/v/two.txt":"two\n"}}' >/dev/null + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"cat /home/user/v/one.txt /home/user/v/two.txt","readOnly":true}' | jq -r .stdout)" $'one\ntwo' "bulk write visible to bash" + eq "$(api GET "/v1/sandboxes/$id/tree?prefix=/home/user/v&depth=3" | jq -c '[.[].path] | sort')" '["/home/user/v/note.txt","/home/user/v/one.txt","/home/user/v/two.txt"]' "tree lists files" + EXPECT=204 api DELETE "/v1/sandboxes/$id/files/home/user/v/note.txt" >/dev/null + eq "$(EXPECT=404 api GET "/v1/sandboxes/$id/files/home/user/v/note.txt" | jq -r .code)" ENOENT "deleted file -> ENOENT" + drop "$id" +} + +journey_ingest() { + local id body + id=$(new_sandbox) + body="{\"basePath\":\"/home/user/in\",\"files\":{\"g.txt\":\"$(printf 'ingested\n' | base64)\",\"nested/two.txt\":\"$(printf 'two\n' | base64)\"}}" + eq "$(api POST "/v1/sandboxes/$id/ingest-files" --json "$body" | jq -c .)" '{"status":"ok","fileCount":2}' "ingest reports fileCount" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/in/g.txt")" ingested "ingested top-level bytes" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/in/nested/two.txt")" two "ingested nested bytes" + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"tar -czf - -C /home/user/in . | base64 -w 0","readOnly":true}' | jq -r .stdout | base64 -d | tar -tzf - | grep -E 'g.txt|two.txt' | sort | tr '\n' ' ')" "./g.txt ./nested/two.txt " "export archive round-trips" + drop "$id" +} + +journey_mcp() { + local id write exec read + id=$(new_sandbox) + write=$(jq -nc --arg id "$id" '{id:$id,path:"/home/user/mcp.txt",content:"mcp-proof\n"}') + exec=$(jq -nc --arg id "$id" '{id:$id,script:"cat /home/user/mcp.txt",readOnly:true}') + read=$(jq -nc --arg id "$id" '{id:$id,path:"/home/user/mcp.txt"}') + eq "$(mcp list | cut -f1 | sort | tr '\n' ' ')" "bash_exec bash_exec_batch file_edit file_read file_write fs_export fs_ingest sandbox_create sandbox_delete sandbox_list sandbox_update " "11 MCP tools" + eq "$(mcp file_write "$write" | jq -c '{ok,size}')" '{"ok":true,"size":10}' "MCP file_write" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/mcp.txt")" mcp-proof "MCP write visible via REST" + eq "$(mcp bash_exec "$exec" | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"mcp-proof\n"}' "MCP bash reads it" + eq "$(EXPECT=error AS=intruder mcp file_read "$read")" '{"ok":false,"error":"forbidden"}' "MCP file tool: other owner forbidden" + eq "$(AS=intruder mcp bash_exec "$exec" | jq -c '{exitCode,stderr}')" '{"exitCode":1,"stderr":"forbidden"}' "MCP bash tool: other owner forbidden" + drop "$id" +} + +all=(auth sandboxes exec files ingest mcp) +for name in "${@:-${all[@]}}"; do + declare -F "journey_$name" >/dev/null || { echo "unknown journey: $name (have: ${all[*]})" >&2; exit 2; } + echo "== $name" + "journey_$name" +done +echo "journeys passed: $passed assertions; transcript: $DIR/transcript.jsonl" diff --git a/.claude/skills/verify/scripts/mcp.mjs b/.claude/skills/verify/scripts/mcp.mjs new file mode 100755 index 00000000..d8398c81 --- /dev/null +++ b/.claude/skills/verify/scripts/mcp.mjs @@ -0,0 +1,41 @@ +#!/usr/bin/env node +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; +import { record } from "./record.mjs"; + +function isToolFailure(part) { + if (part.type !== "text") return false; + try { + return JSON.parse(part.text)?.ok === false; + } catch { + return false; + } +} + +const { STACK_BASE, STACK_TOKEN, STACK_TRANSCRIPT, STACK_SECRET, STACK_AS } = process.env; +const [tool, rawArgs = "{}"] = process.argv.slice(2); +const client = new Client({ name: "verify-skill", version: "1.0.0" }); +await client.connect( + new StreamableHTTPClientTransport(new URL(`${STACK_BASE}/mcp`), { + requestInit: { headers: { Authorization: `Bearer ${STACK_TOKEN}` } }, + }), +); +try { + if (tool === "list") { + const { tools } = await client.listTools(); + record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", as: STACK_AS, tool }, tools.map((t) => t.name)); + for (const t of tools) console.log(`${t.name}\t${Object.keys(t.inputSchema.properties ?? {}).join(",")}`); + } else { + const args = JSON.parse(rawArgs); + const response = await client.callTool({ name: tool, arguments: args }); + const parts = response.content; + const failed = response.isError === true || parts.some(isToolFailure); + record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", as: STACK_AS, tool, args, failed }, parts); + for (const part of parts) console.log(part.type === "text" ? part.text : JSON.stringify(part)); + console.error(`MCP ${tool}: ${failed ? "error" : "ok"}`); + const expectError = process.env.EXPECT === "error"; + if (failed !== expectError) process.exitCode = 1; + } +} finally { + await client.close(); +} diff --git a/.claude/skills/verify/scripts/record.mjs b/.claude/skills/verify/scripts/record.mjs new file mode 100755 index 00000000..a100bb74 --- /dev/null +++ b/.claude/skills/verify/scripts/record.mjs @@ -0,0 +1,59 @@ +#!/usr/bin/env node +import { appendFileSync, readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +const JWT = /eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g; +const SENSITIVE_KEY = /token|secret|password|authorization/i; +const HEADER = /^([A-Za-z0-9-]+):/; +const BODY_LIMIT = 4096; + +function parseJson(text) { + if (!/^\s*[[{]/.test(text)) return undefined; + try { + return JSON.parse(text); + } catch { + return undefined; + } +} + +function redactString(value, secret) { + const parsed = parseJson(value); + if (parsed !== undefined) return JSON.stringify(redact(parsed, secret)); + const header = HEADER.exec(value); + if (header && SENSITIVE_KEY.test(header[1])) return `${header[1]}: [REDACTED]`; + return value.replaceAll(secret, "[REDACTED]").replace(JWT, "[REDACTED JWT]"); +} + +function redact(value, secret) { + if (typeof value === "string") return redactString(value, secret); + if (Array.isArray(value)) return value.map((item) => redact(item, secret)); + if (value && typeof value === "object") { + return Object.fromEntries( + Object.entries(value).map(([key, item]) => [key, SENSITIVE_KEY.test(key) ? "[REDACTED]" : redact(item, secret)]), + ); + } + return value; +} + +function serialize(value) { + return typeof value === "string" ? value : JSON.stringify(value); +} + +export function record(file, secret, event, body, bytes = Buffer.byteLength(serialize(body))) { + const redacted = redact(body, secret); + const text = serialize(redacted); + const captured = + Buffer.byteLength(text) <= BODY_LIMIT + ? { body: redacted } + : { truncated: true, body: Buffer.from(text).subarray(0, BODY_LIMIT).toString("utf8") }; + const entry = { at: new Date().toISOString(), ...redact(event, secret), bytes, ...captured }; + appendFileSync(file, `${JSON.stringify(entry)}\n`); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const [file, method, path, status, as, bodyFile, ...args] = process.argv.slice(2); + const raw = readFileSync(bodyFile); + const text = raw.toString("utf8"); + const event = { as, method, path, status: Number(status), args }; + record(file, process.env.STACK_SECRET, event, parseJson(text) ?? text, raw.length); +} diff --git a/.claude/skills/verify/scripts/stack.sh b/.claude/skills/verify/scripts/stack.sh new file mode 100755 index 00000000..c943dc4e --- /dev/null +++ b/.claude/skills/verify/scripts/stack.sh @@ -0,0 +1,243 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) +cd "$repo" +here="$repo/.claude/skills/verify/scripts" +runs="$repo/scripts/verify/.run" +pointer="$runs/current-stack" +secret=local-verification-secret-at-least-32-bytes + +die() { echo "stack: $*" >&2; exit 1; } + +set_compose() { + read -r -a compose <<<"$COMPOSE" + compose+=(--env-file /dev/null -p "$PROJECT" -f infra/docker-compose.verify.yml) +} + +load_state() { + local dir=${STACK_DIR:-} + if [[ -z "$dir" ]]; then + [[ -f "$pointer" ]] || die "no stack is up; run: $0 up" + dir=$(cat "$pointer") + fi + [[ -f "$dir/state.env" ]] || die "missing $dir/state.env" + # shellcheck disable=SC1091 + source "$dir/state.env" + set_compose +} + +deps_current() { cmp -s pnpm-lock.yaml node_modules/.pnpm/lock.yaml; } + +ready() { curl -fsS "$BASE/readyz" 2>/dev/null | grep -q '"status":"ok"'; } + +maybe_build() { [[ "${1:-}" == "--no-build" ]] || pnpm build >&2; } + +server_alive() { + [[ -n "${PID:-}" ]] && kill -0 "$PID" 2>/dev/null && ps -p "$PID" -o command= | grep -q "dist/api/server.js" +} + +exec_in_stack_env() { + exec env -i "PATH=$PATH" "HOME=$HOME" "DATABASE_URL=$DATABASE_URL" "REDIS_URL=$REDIS_URL" \ + "REDIS_DATA_URL=$REDIS_DATA_URL" "AUTH_SECRET=$AUTH_SECRET" "$@" +} + +override_names() { + local -a overrides=() + read -r -a overrides <<<"${SERVER_ENV:-}" + local kv + for kv in ${overrides[@]+"${overrides[@]}"}; do printf '%s ' "${kv%%=*}"; done +} + +start_server() { + local -a overrides=() + read -r -a overrides <<<"${SERVER_ENV:-}" + (exec_in_stack_env "PORT=$PORT" ${overrides[@]+"${overrides[@]}"} node dist/api/server.js >>"$DIR/server.log" 2>&1) & + PID=$! + write_state + local deadline=$((SECONDS + 60)) + until ready; do + if ! kill -0 "$PID" 2>/dev/null; then tail -40 "$DIR/server.log" >&2; die "server exited before ready"; fi + ((SECONDS < deadline)) || { tail -40 "$DIR/server.log" >&2; die "server not ready in 60s"; } + sleep 0.3 + done +} + +write_state() { + local built_head started_at + built_head=$(git rev-parse --short HEAD) + started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ) + ( + umask 077 + printf '%s=%q\n' DIR "$DIR" PROJECT "$PROJECT" COMPOSE "$COMPOSE" DATABASE_URL "$DATABASE_URL" \ + REDIS_URL "$REDIS_URL" REDIS_DATA_URL "$REDIS_DATA_URL" AUTH_SECRET "$AUTH_SECRET" PORT "$PORT" BASE "$BASE" \ + PID "$PID" SERVER_ENV "${SERVER_ENV:-}" BUILT_HEAD "$built_head" STARTED_AT "$started_at" >"$DIR/state.env" + ) +} + +register_for_teardown() { + DATABASE_URL="" REDIS_URL="" REDIS_DATA_URL="" AUTH_SECRET=$secret PORT="" BASE="" PID="" + write_state + echo "$DIR" >"$pointer" +} + +stop_server() { + server_alive || return 0 + kill -TERM "$PID" + for _ in $(seq 1 100); do kill -0 "$PID" 2>/dev/null || return 0; sleep 0.1; done + kill -KILL "$PID" 2>/dev/null || true +} + +mint() { + AUTH_SECRET=$AUTH_SECRET node dist/api/cli/token.js --sub "${1:-verify-agent}" ${2:+--tenant "$2"} --expires 24h +} + +cmd_up() { + [[ -f "$pointer" ]] && die "a stack is recorded at $(cat "$pointer"); reuse it ($0 doctor) or run: $0 down" + command -v docker >/dev/null && docker info >/dev/null 2>&1 || die "Docker daemon is not running" + [[ -d node_modules ]] || die "run pnpm setup:dev first" + deps_current || die "node_modules does not match pnpm-lock.yaml; run: pnpm install --frozen-lockfile" + local id="stack-$(date +%s)-$$" + DIR="$runs/$id" + PROJECT="sqlfs-$id" + if docker compose version >/dev/null 2>&1; then COMPOSE="docker compose"; else COMPOSE="docker-compose"; fi + mkdir -p "$runs" + mkdir -m 700 "$DIR" + set_compose + register_for_teardown + "${compose[@]}" up -d --wait --wait-timeout 120 >&2 + DATABASE_URL="postgres://sqlfs_app:sqlfs_app@$("${compose[@]}" port postgres 5432)/sqlfs" + REDIS_URL="redis://$("${compose[@]}" port redis 6379)" + REDIS_DATA_URL="redis://$("${compose[@]}" port redis-data 6379)" + maybe_build "${1:-}" + PORT=$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})') + BASE="http://127.0.0.1:$PORT" + start_server + echo "stack up: $BASE (pid $PID)" + echo "evidence: $DIR" +} + +cmd_restart() { + local requested=${SERVER_ENV-__keep__} + load_state + [[ "$requested" == __keep__ ]] || SERVER_ENV=$requested + maybe_build "${1:-}" + stop_server + echo "--- restart $(date -u +%Y-%m-%dT%H:%M:%SZ) overrides: $(override_names)" >>"$DIR/server.log" + start_server + echo "restarted: $BASE (pid $PID)" +} + +cmd_doctor() { + load_state + local bad=0 + ok() { echo "ok $*"; } + fail() { echo "FAIL $*"; bad=1; } + warn() { echo "warn $*"; } + local states + states=$("${compose[@]}" ps --format '{{.Service}}={{.Health}}' 2>/dev/null | sort | tr '\n' ' ') + [[ "$states" == "postgres=healthy redis-data=healthy redis=healthy " ]] && ok "services: $states" || fail "services: ${states:-none}" + server_alive && ok "server pid $PID runs dist/api/server.js" || fail "server pid ${PID:-?} not running (see $DIR/server.log)" + if command -v lsof >/dev/null; then + local owner + owner=$(lsof -nP -iTCP:"$PORT" -sTCP:LISTEN -t 2>/dev/null | head -1) || true + [[ "$owner" == "$PID" ]] && ok "port $PORT owned by pid $PID" || fail "port $PORT listener is '${owner:-none}', expected $PID" + fi + ready && ok "GET /readyz ok" || fail "GET /readyz not ok" + local code + code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(mint doctor)" "$BASE/v1/sandboxes") || true + [[ "$code" == 200 ]] && ok "minted token accepted (GET /v1/sandboxes 200)" || fail "GET /v1/sandboxes returned $code" + deps_current && ok "node_modules matches pnpm-lock.yaml" || + fail "node_modules does not match pnpm-lock.yaml -> run: pnpm install --frozen-lockfile && $0 restart" + local newer + newer=$(find src \( -name '*.ts' -o -name '*.sql' \) -newer dist/api/server.js -not -path '*/tests/*' 2>/dev/null | head -3) + if [[ -n "$newer" ]]; then + fail "dist is stale; source changed after build: $(echo "$newer" | tr '\n' ' ') -> run: $0 restart" + elif [[ dist/api/server.js -nt "$DIR/state.env" ]]; then + warn "dist rebuilt after the server started; run: $0 restart --no-build" + else + ok "dist is current for src (server started $STARTED_AT, HEAD $BUILT_HEAD, now $(git rev-parse --short HEAD))" + fi + [[ -n "${SERVER_ENV:-}" ]] && echo "info server env overrides (values hidden): $(override_names)" + echo "base $BASE" + echo "dir $DIR" + return "$bad" +} + +cmd_token() { + load_state + mint "$@" +} + +cmd_api() { + load_state + (($# >= 2)) || die "usage: $0 api METHOD PATH [curl args...] (env: AS=sub|none, TENANT=, EXPECT=status)" + local method=$1 path=$2 as=${AS:-verify-agent} + shift 2 + local -a auth=() + [[ "$as" == none ]] || auth=(-H "Authorization: Bearer $(mint "$as" "${TENANT:-}")") + local body status + body=$(mktemp) + status=$(curl -sS -X "$method" -o "$body" -w '%{http_code}' ${auth[@]+"${auth[@]}"} "$@" "$BASE$path") + STACK_SECRET=$AUTH_SECRET node "$here/record.mjs" "$DIR/transcript.jsonl" "$method" "$path" "$status" "$as" "$body" "$@" + cat "$body" + [[ -s "$body" && "$(tail -c1 "$body" | od -An -c | tr -d ' ')" != '\n' ]] && echo + rm -f "$body" + echo "HTTP $status $method $path" >&2 + local expect=${EXPECT:-2xx} + if [[ "$expect" == 2xx ]]; then [[ "$status" == 2* ]] || die "expected 2xx, got $status" + else [[ "$status" == "$expect" ]] || die "expected $expect, got $status"; fi +} + +cmd_mcp() { + load_state + (($# >= 1)) || die "usage: $0 mcp list | $0 mcp TOOL '{json args}' (env: AS=sub, EXPECT=error)" + STACK_BASE=$BASE STACK_TOKEN=$(mint "${AS:-verify-agent}" "${TENANT:-}") STACK_AS=${AS:-verify-agent} STACK_TRANSCRIPT=$DIR/transcript.jsonl \ + STACK_SECRET=$AUTH_SECRET node "$here/mcp.mjs" "$@" +} + +cmd_psql() { + load_state + (($# == 1)) || die "usage: $0 psql 'SQL' (runs as the postgres superuser; bypasses RLS for inspection)" + "${compose[@]}" exec -T postgres psql -U postgres -d sqlfs -v ON_ERROR_STOP=1 -c "$1" +} + +suite_names() { + echo integration + grep -l "new LocalApiRun(" scripts/verify/*.mjs | grep -v '\.test\.mjs$' | xargs -n1 basename | sed 's/\.mjs$//' +} + +cmd_suite() { + load_state + (($# >= 1)) || die "usage: $0 suite NAME [args] (NAME: integration or a suite entry point in scripts/verify/)" + local name=$1 + shift + suite_names | grep -qx "$name" || die "$name is not a suite entry point (have: $(suite_names | tr '\n' ' '))" + local -a suite_env=("VERIFY_ENV_FILE=none" "VERIFY_EVIDENCE_DIR=$DIR/suites" "VERIFY_PYTHON=$repo/clients/python/.venv/bin/python") + mkdir -p "$DIR/suites" + if [[ "$name" == integration ]]; then + local report + report="$DIR/suites/integration-$(date +%s).json" + (exec_in_stack_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@") + node scripts/verify/assert-integration-report.mjs "$report" + else + (exec_in_stack_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@") + fi +} + +cmd_down() { + load_state + stop_server + "${compose[@]}" logs --no-color >"$DIR/services.log" 2>&1 || true + "${compose[@]}" down --volumes --remove-orphans >&2 + [[ -f "$pointer" && "$(cat "$pointer")" == "$DIR" ]] && rm -f "$pointer" + echo "stack down; evidence kept at $DIR" +} + +cmd=${1:-} +shift || true +case "$cmd" in +up | restart | doctor | token | api | mcp | psql | suite | down) "cmd_$cmd" "$@" ;; +env) load_state && cat "$DIR/state.env" ;; +*) die "usage: $0 {up [--no-build]|doctor|token [sub] [tenant]|api METHOD PATH ...|mcp TOOL JSON|psql SQL|suite NAME|restart [--no-build]|env|down}" ;; +esac diff --git a/AGENTS.md b/AGENTS.md index c3efe609..e9e06b12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,39 @@ pnpm verify:architecture # also worker failure, stream finalization, input budge Use `pnpm check:typescript` or `pnpm check:python` for SDK-only changes. Run a targeted core test with `pnpm exec vitest run src/sql-fs/tests/unit/sql-fs.cache.test.ts`. A nonexistent test selection must fail. `pnpm test:integration` is the lower-level command for an already migrated disposable database with `DATABASE_URL` and `REDIS_URL`; it can skip without those variables. Prefer `pnpm verify:local` for completion evidence. +## Verify while you work + +Prove every runtime change against a real server before handing back. The full gates above are completion evidence. While iterating, use the verification skill in [`.claude/skills/verify/`](.claude/skills/verify/SKILL.md). In Claude Code, invoke it as `verify`. Other agents read `SKILL.md` and run its scripts directly. The skill maps each kind of change to its required gates, and [`features/`](.claude/skills/verify/features/README.md) holds the client journeys for auth, sandboxes, exec, files, ingest, and MCP. + +```bash +S=.claude/skills/verify/scripts/stack.sh +$S up # disposable Postgres/Redis + built API server; keeps running +$S doctor # read-only health check; FAILs when dist/ is older than src/ +$S restart # rebuild and restart after each source edit (data kept) +$S api POST /v1/sandboxes --json '{"name":"x"}' # authed request as verify-agent; EXPECT= asserts the status +EXPECT=403 AS=intruder $S api GET /v1/sandboxes/$ID # another subject is denied a sandbox it does not own +$S mcp list # MCP over Streamable HTTP; $S mcp TOOL '' +$S psql 'select ...' # inspect side effects (superuser, bypasses RLS: never proof of isolation) +$S suite contracts # a suite from the table below, or `integration [filter]`, on the stack's databases +.claude/skills/verify/scripts/journeys.sh # every mapped journey, ~40 assertions in ~10s; needs jq +$S down # remove only this run's server, containers, and volumes; evidence stays +``` + +Evidence lands in `scripts/verify/.run/stack-/` (`transcript.jsonl`, `server.log`, `suites/`). The stack can run alongside `pnpm verify:local`, which starts its own. + +`scripts/verify/` holds the live suites. Every suite except `large-files` runs through `$S suite `. It reuses the stack's Postgres and Redis but starts its own API server from `dist/`, so a passing suite does not prove the `$S up` server. The full gates run them as listed: + +| Suite | Proves | Run by | +| --- | --- | --- | +| `contracts` | HTTP and MCP behavior across replicas, validated against `/openapi.json` | `verify:local` | +| `output-cap` | worker output limits in both worker modes | `verify:local` | +| `shutdown` | graceful shutdown ends MCP listener streams and refuses new connections | `verify:local` | +| `sdk-clients` (+ `sdk-python.py`) | live TypeScript and Python SDKs | `verify:local` | +| `worker-failures`, `sse-finalization`, `input-budgets`, `tenants` | worker loss, stream finalization, input budgets, tenant isolation | `verify:architecture` | +| `ingest-admission`, `large-files` | ingest budgets, large-file profiling | manual; large files need cgroup v2 | + +`local-api.mjs` is the shared harness: owned servers, tokens, schema validation, `proof.json`. `assert-integration-report.mjs` is the shared check that an integration run executed tests with none failed or skipped. `*-faults.mjs` files are fault injectors loaded by the suites, not entry points. When you add an endpoint or client-visible behavior, add or extend a suite here, add its journey to `features/` and `journeys.sh`, and update `openapi-spec.ts`. + See [READINESS.md](docs/READINESS.md) for verified scope and hosted follow-up. Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup and contribution steps, [DEVELOPER.md](docs/DEVELOPER.md) before changing consistency or session ownership, [README.md](README.md) for API behavior, [CONFIGURATION.md](docs/CONFIGURATION.md) for runtime settings, and [AUTOSCALING.md](docs/AUTOSCALING.md) for probes, metrics and replica scaling. Python type checking targets the oldest supported interpreter. `pnpm check:python:types` uses uv to provision Python 3.9 in `clients/python/.venv-py39`, separate from the Python 3.12 development environment. This keeps the dependency syntax aligned with the declared target. Mypy stays below version 2 while Python 3.9 is supported. @@ -70,6 +103,6 @@ Migrations are handwritten numbered SQL files. Add the next migration without ed Work autonomously on local implementation and reversible verification. Use a separate checkout for concurrent writers. Do not overwrite another agent's changes. Keep secrets out of commits and evidence. Use the disposable verification stack for tests that create databases or inject faults. -Before handing back, run `pnpm check`, then `pnpm verify:local` for runtime, database, dependency, or SDK behavior changes. Run `pnpm security:audit` after dependency changes. Report the commands, results, and any unverified claims. A skipped test is not proof. +Before handing back, run `pnpm check`, then `pnpm verify:local` for runtime, database, dependency, or SDK behavior changes. For runtime changes, also drive the changed behavior on the verify stack and run `journeys.sh` (see [Verify while you work](#verify-while-you-work)). Cite the evidence paths. Run `pnpm security:audit` after dependency changes. Report the commands, results, and any unverified claims. A skipped test is not proof. Add a `.changeset/*.md` entry for behavior changes. Do not bump versions or edit `CHANGELOG.md` in feature work. Deployment, release, merging, production data changes, and shared-branch force pushes need explicit task authorization. Repository changes do not prove that hosted CI or branch protection is active. diff --git a/scripts/verify-local.sh b/scripts/verify-local.sh index f28f283e..3db8e900 100644 --- a/scripts/verify-local.sh +++ b/scripts/verify-local.sh @@ -52,14 +52,7 @@ pnpm --dir clients/typescript build "${verify_env[@]}" node scripts/verify/output-cap.mjs "${verify_env[@]}" node scripts/verify/shutdown.mjs "${verify_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$evidence/integration.json" -node --input-type=module - "$evidence/integration.json" <<'JS' -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; -const report = JSON.parse(readFileSync(process.argv[2], "utf8")); -assert.ok(report.numPassedTests > 0, "Integration tests must execute"); -assert.equal(report.numFailedTests, 0, "Integration tests must pass"); -assert.equal(report.numPendingTests, 0, "Integration tests must not silently skip"); -JS +node scripts/verify/assert-integration-report.mjs "$evidence/integration.json" "${verify_env[@]}" node scripts/verify/sdk-clients.mjs if [[ "$profile" == architecture ]]; then diff --git a/scripts/verify/assert-integration-report.mjs b/scripts/verify/assert-integration-report.mjs new file mode 100644 index 00000000..b26d7a7e --- /dev/null +++ b/scripts/verify/assert-integration-report.mjs @@ -0,0 +1,14 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +export function assertIntegrationReport(report) { + assert.ok(report.numPassedTests > 0, "Integration tests must execute"); + assert.equal(report.numFailedTests, 0, "Integration tests must pass"); + assert.equal(report.numPendingTests, 0, "Integration tests must not silently skip"); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + assertIntegrationReport(JSON.parse(readFileSync(process.argv[2], "utf8"))); +} diff --git a/scripts/verify/assert-integration-report.test.mjs b/scripts/verify/assert-integration-report.test.mjs new file mode 100644 index 00000000..e4fee2ea --- /dev/null +++ b/scripts/verify/assert-integration-report.test.mjs @@ -0,0 +1,37 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import { assertIntegrationReport } from "./assert-integration-report.mjs"; + +const passing = { numPassedTests: 3, numFailedTests: 0, numPendingTests: 0 }; + +test("integration report proof rejects empty, failed or skipped runs", () => { + assertIntegrationReport(passing); + for (const report of [ + { ...passing, numPassedTests: 0 }, + { ...passing, numFailedTests: 1 }, + { ...passing, numPendingTests: 1 }, + {}, + ]) { + assert.throws(() => assertIntegrationReport(report)); + } +}); + +test("integration report CLI exits non-zero for a skipped run", () => { + const script = fileURLToPath(new URL("./assert-integration-report.mjs", import.meta.url)); + const directory = mkdtempSync(join(tmpdir(), "integration-report-")); + const write = (name, report) => { + const file = join(directory, name); + writeFileSync(file, JSON.stringify(report)); + return file; + }; + execFileSync(process.execPath, [script, write("pass.json", passing)]); + assert.throws( + () => execFileSync(process.execPath, [script, write("skip.json", { ...passing, numPendingTests: 2 })], { stdio: "pipe" }), + /must not silently skip/, + ); +});