From 4c26cfa5bf2455bf010a62b4123419aa9ca1b275 Mon Sep 17 00:00:00 2001 From: QuangNguyen2609 Date: Sat, 17 Oct 2026 15:09:49 +1030 Subject: [PATCH 1/4] test(verify): add a project verify skill and document it in AGENTS.md Agents could only prove a change with the all-or-nothing verify:local run. The skill keeps a disposable Postgres/Redis/API stack up for iteration, maps each kind of change to its required gates, and records redacted evidence. journeys.sh asserts the feature map's client journeys against it. Co-Authored-By: Claude Opus 4.6 --- .claude/skills/verify/SKILL.md | 121 ++++++++++ .claude/skills/verify/features/README.md | 51 +++++ .claude/skills/verify/features/auth.md | 33 +++ .claude/skills/verify/features/exec.md | 36 +++ .claude/skills/verify/features/files.md | 37 +++ .claude/skills/verify/features/ingest.md | 30 +++ .claude/skills/verify/features/mcp.md | 35 +++ .claude/skills/verify/features/sandboxes.md | 39 ++++ .claude/skills/verify/scripts/journeys.sh | 118 ++++++++++ .claude/skills/verify/scripts/mcp.mjs | 37 +++ .claude/skills/verify/scripts/record.mjs | 44 ++++ .claude/skills/verify/scripts/stack.sh | 242 ++++++++++++++++++++ AGENTS.md | 34 ++- 13 files changed, 856 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/verify/SKILL.md create mode 100644 .claude/skills/verify/features/README.md create mode 100644 .claude/skills/verify/features/auth.md create mode 100644 .claude/skills/verify/features/exec.md create mode 100644 .claude/skills/verify/features/files.md create mode 100644 .claude/skills/verify/features/ingest.md create mode 100644 .claude/skills/verify/features/mcp.md create mode 100644 .claude/skills/verify/features/sandboxes.md create mode 100755 .claude/skills/verify/scripts/journeys.sh create mode 100755 .claude/skills/verify/scripts/mcp.mjs create mode 100755 .claude/skills/verify/scripts/record.mjs create mode 100755 .claude/skills/verify/scripts/stack.sh diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md new file mode 100644 index 00000000..400ed398 --- /dev/null +++ b/.claude/skills/verify/SKILL.md @@ -0,0 +1,121 @@ +--- +name: verify +description: Prove a change to sql-fs works the way an API, MCP, or SDK client sees it. Picks the gates a change needs (pnpm check, verify:local, verify:architecture, single suites) and drives a persistent disposable Postgres/Redis/API stack with curl-level HTTP, MCP tool calls, and SQL inspection, keeping evidence. Use before handing back any change, when asked to verify/prove/test behavior end to end, or when a full verify:local run is too coarse to debug a failure. +--- + +# Verify sql-fs + +sql-fs is an HTTP + MCP service. Its users are API clients, MCP clients, and the TypeScript/Python SDKs, so proof means real requests against a real server backed by real Postgres and Redis. This skill adds two things on top of the repo's gates: + +1. A **change → gate map** so you run the right checks without asking. +2. A **persistent disposable stack** (`scripts/stack.sh`) so you can drive one endpoint, rerun one suite, or inspect the database while you iterate, instead of paying for a full `pnpm verify:local` on every attempt. + +All paths below are relative to the repo root. `S=.claude/skills/verify/scripts/stack.sh`. + +## Choose the gates + +Run every row your diff touches. `pnpm check` is always required; the full verify command named on a row is the completion evidence, and the stack is how you iterate toward it. For any runtime change, the default loop is: `$S up` → edit → `$S restart` → `$S doctor` → the row's drive/suite → `journeys.sh` → the completion gate → `$S down`. + +| Diff touches | Iterate with | Completion evidence | +| --- | --- | --- | +| Docs, comments, `.changeset/` only | — | `pnpm check` | +| `src/sql-fs/` (fs, transactions, cache, RLS, symlinks) | targeted `pnpm exec vitest run `, `$S suite integration `, [files](features/files.md) / [exec](features/exec.md) drive | `pnpm check` + `pnpm verify:local` | +| `src/api/routes/`, `auth.ts`, `validation.ts`, `openapi-spec.ts` | the matching feature drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| `src/api/mcp/` | [mcp](features/mcp.md) drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| `src/api/worker-owner/`, `ingress-admission.ts`, `session-manager.ts`, `src/execution/`, `script-scope.ts`, budgets, cancellation, deadlines, body limits | `$S suite worker-failures`, `sse-finalization`, `input-budgets`, `output-cap` | `pnpm check` + `pnpm verify:architecture` | +| `src/api/server.ts` startup/shutdown, `src/api/mcp/server.ts` listener lifecycle, ingress waits | `$S suite shutdown`, `$S suite contracts` | `pnpm check` + `pnpm verify:local` | +| Tenancy, `TENANT_DATABASES`, owner isolation | `$S suite tenants`, [sandboxes](features/sandboxes.md) isolation steps | `pnpm check` + `pnpm verify:architecture` | +| `src/redis/`, `redis-blob-cache.ts` | `$S suite integration src/redis` | `pnpm check` + `pnpm verify:local` | +| `src/sql-fs/migrations/postgres/` | `$S down && $S up` (fresh DB runs every migration), `$S psql '\d '` | `pnpm check` + `pnpm verify:local` | +| `clients/typescript/` or `clients/python/` only | `pnpm check:typescript` / `pnpm check:python`, `$S suite sdk-clients` | `pnpm check` + `pnpm verify:local` | +| `package.json`, lockfiles, `patches/` | — | `pnpm check` + `pnpm verify:local` + `pnpm security:audit` | + +Also, from CLAUDE.md: add `.changeset/*.md` for behavior changes, update `src/api/openapi-spec.ts` when an API contract changes, and report every command with its result. A skipped test is not proof. + +## Launch + +Preconditions: Docker daemon running, `pnpm setup:dev` done (needs `node_modules` and `clients/python/.venv`). `jq` makes JSON assertions easier but is optional. + +```bash +$S up # compose stack + pnpm build + API server; ~10s +$S up --no-build # reuse the current dist/ +``` + +`up` starts `infra/docker-compose.verify.yml` under a unique Compose project with tmpfs Postgres and random loopback ports. It builds, then starts `node dist/api/server.js` with a clean environment (`env -i`, so `.env` and your shell's deployment variables are ignored) on a random port. The server applies every migration at startup. Ready means `GET /readyz` returns `{"status":"ok"}`, and the server only gets there after migrations and Redis connect. If the server exits first, `up` prints the tail of `server.log` and fails. + +`up` refuses while a stack is recorded for this checkout. Reuse it (`$S doctor`) or `$S down` first. After a failed `up`, run `$S down` too, because the containers may be running. + +Server environment overrides: `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false SQL_FS_WORKER_OUTPUT_BYTES=1048576" $S restart`. Overrides persist across later restarts until you pass `SERVER_ENV= $S restart`. Values are split on spaces, so use compact JSON with no spaces. + +## Doctor + +Run `$S doctor` first, and again whenever anything looks off. It is read-only and exits non-zero on any `FAIL`: + +- all three services `healthy` +- the recorded PID is alive and is `node dist/api/server.js` +- that PID owns the recorded port (via `lsof`) +- `/readyz` ok and a freshly minted token gets `200` on `GET /v1/sandboxes` +- `node_modules` matches `pnpm-lock.yaml`, compared byte-for-byte with `node_modules/.pnpm/lock.yaml`. After a pull that changed the lockfile, the old install keeps running and unit tests fail for reasons unrelated to your diff. Fix it with `pnpm install --frozen-lockfile && $S restart`. `up` refuses to start on drift. +- `dist/` is not older than any non-test `src/**/*.ts`. **A stale build is the most common false result**: you edited source, but the server and every suite still run the old `dist/`. Fix it with `$S restart`. + +## Drive + +After every source edit: `$S restart` (rebuild + restart, same port, data kept), then `$S doctor`. + +**Regression sweep:** `.claude/skills/verify/scripts/journeys.sh [auth|sandboxes|exec|files|ingest|mcp ...]` runs every journey in the feature map against the stack and asserts the documented result. With no arguments it runs all six, about 40 assertions in roughly 10 seconds. It exits non-zero at the first mismatch and prints the expected and actual values. Run it after any change to `src/api/` or `src/sql-fs/` before the full gate. It is the executable form of `features/*.md`: when you add or change a journey there, change the matching `journey_` function in the same commit. It is iteration evidence, not a gate. CI and `verify:local` do not run it. + +| Command | What it does | +| --- | --- | +| `$S api METHOD PATH [curl args]` | Authenticated request. Body goes to stdout and `HTTP ` to stderr, and a record is appended to `transcript.jsonl`. Exits non-zero unless the status is 2xx, or equals `EXPECT=`. `AS=` picks the caller (default `verify-agent`), `AS=none` sends no token, `TENANT=` adds a tenant claim. | +| `$S mcp list` / `$S mcp TOOL ''` | MCP over Streamable HTTP with the same `AS`/`TENANT`. Exits non-zero on `isError` or a `{"ok":false}` payload. Use `EXPECT=error` to assert one of those. `bash_exec` denials are neither; assert their `exitCode`/`stderr`. | +| `$S token [sub] [tenant]` | Print a 24h JWT signed with the stack's `AUTH_SECRET`. | +| `$S psql 'SQL'` | Run SQL as the `postgres` superuser on the stack DB. This **bypasses RLS** and is only for inspecting side effects; never use it to prove isolation. Tables: `sandboxes`, `inodes`, `dirents`, `blobs`, `sandbox_epochs`. | +| `$S suite NAME [args]` | Run `scripts/verify/NAME.mjs` with the stack's DB/Redis. Suites start their own servers from `dist/`, so restart (build) first. `NAME=integration [file filters]` runs vitest integration and writes a JSON report. | +| `$S env` | Print the recorded state (URLs, port, PID, run dir). | + +Use `curl --json ''` for JSON (sets both headers). Use `--data-binary @file -H 'Content-Type: application/octet-stream'` for raw file bytes. + +The feature map in [features/](features/README.md) has the exact journeys. Always assert the observable result (status, body, a second read through another path). Don't settle for "it returned 200". + +## Evidence + +Everything lands in `scripts/verify/.run/stack-/` (gitignored) and survives `down`: + +- `transcript.jsonl`: every `api`/`mcp` call with caller, status, and body (≤4 KiB). The stack secret, JWTs (in curl args or bodies), and token/secret/password/authorization keys are redacted. +- `server.log`: server stdout/stderr across restarts, with a `--- restart` line at each one. +- `services.log`: Postgres/Redis logs, written at `down`. +- `suites/-/proof.json` and `requests.jsonl`, plus `suites/integration-*.json`. +- `state.env`: the stack's coordinates. It contains the local-only verification secret. + +`pnpm verify:local` writes its own `scripts/verify/.run/verify-*/` directory with `proof.json` per suite and `integration.json`. + +Proof standards: + +- Drive the public path a client uses (`/v1/*`, `/mcp`, the SDKs). Never call internals or test-only hooks to make a journey pass. +- For mutations, capture the action and an independent read: write via REST, read via bash or MCP, or the reverse. Then confirm the side effect with `$S psql` where it matters. +- For isolation, prove the denial (`AS=intruder EXPECT=403`) through the API, not through SQL. +- If a precondition is missing, report the journey as failed with the reason. Never weaken the journey to make it pass. +- Report what you ran and what you saw: commands, statuses, evidence paths. Name anything you did not verify. + +## Cleanup + +```bash +$S down +``` + +`down` stops only the recorded PID, and only if it still runs `dist/api/server.js`. It saves service logs, removes this run's containers and volumes (`down --volumes`), and clears the pointer. Evidence stays on disk. Never `pkill node` or `docker rm` by pattern, because other worktrees and the user's own stacks may be running. Delete old `scripts/verify/.run/stack-*` directories only when the user asks. + +## Isolation + +- One recorded stack per checkout (pointer: `scripts/verify/.run/current-stack`). Separate worktrees get separate stacks and never collide, since project names and ports are unique. +- `STACK_DIR=scripts/verify/.run/stack- $S doctor` targets a specific recorded stack. +- `pnpm verify:local` and `verify:architecture` start their own fresh stack and can run while yours is up. +- Never point the helper or suites at a shared or non-disposable database. `tenants`, `worker-failures`, and the integration suite create databases and inject faults. + +## Gotchas + +- zsh has no `PIPESTATUS`; use `${pipestatus[1]}`, or don't pipe when you need the exit code. +- `exec-sync` timeouts return `408 EXEC_TIMEOUT` but commit the completed prefix. Assert both. +- MCP tool errors use three shapes (`src/api/mcp/tool-errors.ts`): worker errors set `isError:true`; file and sandbox tools return `{"ok":false,"error":...}` with `isError:false`; `bash_exec` returns `{"exitCode":1,"stderr":"forbidden"}`. The helper catches the first two. For bash tools, assert the fields. +- `PATCH /v1/sandboxes/:id` accepts only `python`, `javascript`, and `network`. A `name` is rejected with 400. +- The integration suite takes about 2.5 minutes. Narrow it with a path filter while iterating, then run it whole. diff --git a/.claude/skills/verify/features/README.md b/.claude/skills/verify/features/README.md new file mode 100644 index 00000000..ff77b8b4 --- /dev/null +++ b/.claude/skills/verify/features/README.md @@ -0,0 +1,51 @@ +# sql-fs verification map + +This directory is the maintained source for verifying what sql-fs clients can observe. Read this index, then use the matching feature file as the recipe. When a change adds or alters a client-visible path, update its feature file in the same change. + +## Baseline preconditions + +- `S=.claude/skills/verify/scripts/stack.sh` +- `$S up` has completed and `$S doctor` exits 0 with no `FAIL` lines. +- Run all commands from the repo root. +- Never drive a server or database that this checkout's `$S up` did not start. + +## Driving conventions + +- `.claude/skills/verify/scripts/journeys.sh ` runs a feature's journey with assertions. Keep it in step with these files. +- Create the journey's own sandbox and capture its ID: `ID=$($S api POST /v1/sandboxes --json '{"name":"verify"}' | jq -r .id)`. +- The default caller is `verify-agent`. Use `AS=intruder` for a second owner and `AS=none` for no token. +- Assert with `EXPECT=` (the helper exits non-zero on mismatch) and inspect the body with `jq`. +- Prove a mutation with a second, independent read: a REST read, a bash `cat`, an MCP `file_read`, or `$S psql`. +- Delete the sandbox at the end with `$S api DELETE /v1/sandboxes/$ID`. Evidence stays in the run directory. + +## Proof and skip reporting + +- Cite `transcript.jsonl` lines or suite `proof.json` paths for every claim. +- Report an unreachable journey with the command and the unmet precondition. +- Proving one entry point (REST) does not prove another (MCP, SDK). List which ones you drove. + +## Feature entry contract + +Each feature file has an H1, one paragraph of client-visible behavior, then exactly four H2s: `Sub-features`, `How to get to it (user POV)`, `Driving it with stack.sh`, and `Gotchas`. + +## Features + +- [Auth](./auth.md) covers bearer tokens, bootstrap with `X-Auth-Secret`, and unauthenticated rejection. +- [Sandboxes](./sandboxes.md) covers create with seed files, get, list, runtime update, owner isolation, and delete. +- [Bash execution](./exec.md) covers `exec-sync`, SSE `exec`, batches, read-only mode, and cooperative timeouts. +- [Files](./files.md) covers raw read and write, edit, bulk write, mkdir, tree, and delete. +- [Ingest](./ingest.md) covers base64 manifest upload and archive export through bash. +- [MCP](./mcp.md) covers tool listing, file and bash tools, and cross-transport visibility. + +## Covered only by suites + +These have no hand-driven journey yet. Run the suite through `$S suite `, or as part of `pnpm verify:architecture`. + +- SDKs (TypeScript and Python, live): `sdk-clients` +- Multi-tenant routing and isolation: `tenants` +- Worker crash, lease loss, and unknown-outcome errors: `worker-failures` +- SSE stream finalization: `sse-finalization` +- Request and input budgets: `input-budgets` +- Output caps: `output-cap` +- Graceful shutdown with open MCP streams: `shutdown` +- Full HTTP/MCP contract against `/openapi.json`: `contracts` diff --git a/.claude/skills/verify/features/auth.md b/.claude/skills/verify/features/auth.md new file mode 100644 index 00000000..395e4688 --- /dev/null +++ b/.claude/skills/verify/features/auth.md @@ -0,0 +1,33 @@ +# Auth + +Every `/v1/*` and `/mcp` call carries a bearer JWT signed with `AUTH_SECRET`. A caller with the secret can mint a token through `POST /v1/auth/bootstrap`. Requests without a valid token are rejected before they touch any sandbox. + +## Sub-features + +- `auth-required` rejects unauthenticated `/v1` requests with `401 AUTH_REQUIRED`. +- `auth-bootstrap` issues a token for a `sub` when `X-Auth-Secret` matches. +- `auth-bootstrap-denied` rejects a wrong secret with `403 FORBIDDEN` before parsing the body. +- `auth-token-cli` mints tokens offline with `pnpm token:create`. + +## How to get to it (user POV) + +- Send `Authorization: Bearer ` on any `/v1` or `/mcp` request. +- `POST /v1/auth/bootstrap` with header `X-Auth-Secret` and body `{"sub":"...","expiresIn":"24h"}`. +- `AUTH_SECRET=... pnpm token:create -- --sub agent-1 [--tenant t] --expires 30d`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md). `$S env` shows `AUTH_SECRET`. + +- **No token.** Run `EXPECT=401 AS=none $S api GET /v1/sandboxes`. The body is `{"error":"unauthorized","code":"AUTH_REQUIRED"}`. +- **Wrong secret.** Run `EXPECT=403 AS=none $S api POST /v1/auth/bootstrap -H 'X-Auth-Secret: wrong' --json '{"sub":"boot"}'`. The body has `code` `FORBIDDEN`. +- **Bootstrap.** Run `eval "$($S env | grep AUTH_SECRET)"` and then `TOKEN=$(EXPECT=201 AS=none $S api POST /v1/auth/bootstrap -H "X-Auth-Secret: $AUTH_SECRET" --json '{"sub":"boot","expiresIn":"24h"}' | jq -r .token)`. `TOKEN` is a three-part JWT. +- **Use the issued token.** Run `EXPECT=200 AS=none $S api GET /v1/sandboxes -H "Authorization: Bearer $TOKEN"`. The body is `{"sandboxes":[]}` for the fresh `boot` owner. + +## Gotchas + +- Bootstrap is rate-limited per IP. A burst of bad attempts returns `429` for a while. +- `POST /v1/auth/admin` needs `ADMIN_SECRET`, which the stack does not set. It returns `500 ADMIN_NOT_CONFIGURED` unless you pass `SERVER_ENV="ADMIN_SECRET=..." $S restart`. +- A token without a `tenant` claim resolves to the `default` tenant. diff --git a/.claude/skills/verify/features/exec.md b/.claude/skills/verify/features/exec.md new file mode 100644 index 00000000..60854605 --- /dev/null +++ b/.claude/skills/verify/features/exec.md @@ -0,0 +1,36 @@ +# Bash execution + +Clients run bash scripts inside a sandbox and get stdout, stderr, and the exit code back. They can do this synchronously, as a Server-Sent Events stream, or as a batch. A script's file writes persist. Read-only mode rejects mutations, and a timeout stops the script while keeping the work it already committed. + +## Sub-features + +- `exec-sync` returns `{exitCode, stdout, stderr}` and persists writes. +- `exec-sse` streams `stdout` events and exactly one `exit` event. +- `exec-batch` runs ordered scripts and returns results keyed by `id`. +- `exec-readonly` rejects mutation with `422 EREADONLY_VIOLATION` and leaves no file. +- `exec-timeout` returns `408 EXEC_TIMEOUT` and keeps the completed prefix. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes/:id/exec-sync`, `/exec` (SSE), `/exec-sync-batch`. +- MCP: `bash_exec`, `bash_exec_batch`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Sync run and persistence.** Run `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo hi > /home/user/a.txt; cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}'`. The output is `{"exitCode":0,"stdout":"hi\n"}`. Then `$S api GET /v1/sandboxes/$ID/files/home/user/a.txt` prints `hi`. +- **Read-only rejection.** Run `EXPECT=422 $S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo x > /home/user/ro.txt","readOnly":true}'`. The `code` is `EREADONLY_VIOLATION`. Then `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/ro.txt` confirms no file was written. +- **SSE stream.** Run `$S api POST /v1/sandboxes/$ID/exec --json '{"script":"printf stream-proof"}'`. The output contains `event: stdout` with `"data":"stream-proof"` and exactly one `event: exit` with `"exitCode":0`. +- **Batch.** Run `$S api POST /v1/sandboxes/$ID/exec-sync-batch --json '{"scripts":[{"id":"a","script":"echo one"},{"id":"b","script":"echo two"}]}' | jq -c '[.results[]|[.id,.exitCode,.stdout]]'`. The output is `[["a",0,"one\n"],["b",0,"two\n"]]`. +- **Timeout prefix.** Run `EXPECT=408 $S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"echo p > /home/user/p.txt; sleep 2; echo s > /home/user/s.txt","timeoutMs":100}'`. The `code` is `EXEC_TIMEOUT`. Then `$S api GET /v1/sandboxes/$ID/files/home/user/p.txt` prints `p` and `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/s.txt` succeeds. +- **Worker modes.** Repeat the sync run after `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false" $S restart --no-build`, then restore with `SERVER_ENV= $S restart --no-build`. Both modes must give identical results. + +## Gotchas + +- A non-zero script exit is still HTTP `200`. Assert `exitCode`, not just the status. +- Python and JavaScript commands need the runtime enabled (`PATCH {"python":true}`). The first `python3` call can take tens of seconds. +- Output above `SQL_FS_WORKER_OUTPUT_BYTES` fails with exit `126`. `$S suite output-cap` covers both worker modes. +- Never auto-retry an execution whose reply was lost. Unknown-outcome errors are typed on purpose; see `worker-failures`. diff --git a/.claude/skills/verify/features/files.md b/.claude/skills/verify/features/files.md new file mode 100644 index 00000000..aed9cd8a --- /dev/null +++ b/.claude/skills/verify/features/files.md @@ -0,0 +1,37 @@ +# Files + +Clients read and write sandbox files directly, without bash. They can write raw bytes, edit a string in place, write many files at once, create directories, list a tree, and delete. Every change is visible to bash and to other replicas. + +## Sub-features + +- `files-put-get` writes raw bytes (`204`) and reads them back exactly. +- `files-edit` replaces `oldString` with `newString` and reports `replacements`. +- `files-bulk` writes several files with `writeFiles` (`204`). +- `files-mkdir` creates directories recursively (`204`). +- `files-tree` lists entries under a prefix to a given depth. +- `files-delete` removes a file (`204`), after which reads return `404 ENOENT`. + +## How to get to it (user POV) + +- REST: `GET|PUT|PATCH|DELETE /v1/sandboxes/:id/files/`, `POST /v1/sandboxes/:id/writeFiles`, `POST /v1/sandboxes/:id/mkdir`, `GET /v1/sandboxes/:id/tree?prefix=&depth=`. +- MCP: `file_read`, `file_write`, `file_edit`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Mkdir.** Run `EXPECT=204 $S api POST /v1/sandboxes/$ID/mkdir --json '{"path":"/home/user/v","recursive":true}'`. +- **Raw write and read.** Run `EXPECT=204 $S api PUT /v1/sandboxes/$ID/files/home/user/v/note.txt --data-binary $'before\n' -H 'Content-Type: application/octet-stream'`, then `$S api GET /v1/sandboxes/$ID/files/home/user/v/note.txt`. It prints `before`. +- **Edit.** Run `$S api PATCH /v1/sandboxes/$ID/files/home/user/v/note.txt --json '{"oldString":"before","newString":"after"}' | jq .replacements`. It prints `1`, and a second GET prints `after`. +- **Bulk write, seen by bash.** Run `EXPECT=204 $S api POST /v1/sandboxes/$ID/writeFiles --json '{"files":{"/home/user/v/one.txt":"one\n","/home/user/v/two.txt":"two\n"}}'`, then `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"cat /home/user/v/one.txt /home/user/v/two.txt","readOnly":true}' | jq -r .stdout`. It prints `one` and `two`. +- **Tree.** Run `$S api GET "/v1/sandboxes/$ID/tree?prefix=/home/user/v&depth=3" | jq -c '[.[].path]'`. The output lists `note.txt`, `one.txt`, and `two.txt` under `/home/user/v/`. +- **Delete.** Run `EXPECT=204 $S api DELETE /v1/sandboxes/$ID/files/home/user/v/note.txt`, then `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/v/note.txt`. The body is `{"error":"not_found","code":"ENOENT"}`. + +## Gotchas + +- The path in `/files/` has no leading slash after `files/`. The server treats it as absolute. +- Quote URLs that contain `?` or `&`, or the shell will mangle the request. +- Null bytes and path traversal are rejected. Symlinks are allowed by default, with bounded resolution. +- Write bodies have size limits (`MAX_BULK_WRITE_BYTES` and others). See `docs/CONFIGURATION.md` before writing large-file journeys, and `scripts/verify/large-files.mjs` for profiling. diff --git a/.claude/skills/verify/features/ingest.md b/.claude/skills/verify/features/ingest.md new file mode 100644 index 00000000..5e75f25d --- /dev/null +++ b/.claude/skills/verify/features/ingest.md @@ -0,0 +1,30 @@ +# Ingest + +Clients upload a tree of files in one request as a JSON manifest of base64 contents under a `basePath`. They can get the tree back out as an archive by running `tar` in bash. + +## Sub-features + +- `ingest-manifest` writes every manifest entry under `basePath` and reports `fileCount`. +- `ingest-nested` creates nested directories from manifest keys. +- `ingest-export` round-trips the tree as a `tar.gz` produced by bash. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes/:id/ingest-files` with `{"basePath":"...","files":{"rel/path":""}}`. +- MCP: `fs_ingest`, `fs_export`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **Upload.** Run `$S api POST /v1/sandboxes/$ID/ingest-files --json "{\"basePath\":\"/home/user/in\",\"files\":{\"g.txt\":\"$(printf 'ingested\n' | base64)\",\"nested/two.txt\":\"$(printf 'two\n' | base64)\"}}"`. The body is `{"status":"ok","fileCount":2}`. +- **Bytes landed.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/in/g.txt` and `$S api GET /v1/sandboxes/$ID/files/home/user/in/nested/two.txt`. They print `ingested` and `two`. +- **Export round-trip.** Run `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"tar -czf - -C /home/user/in . | base64 -w 0","readOnly":true}' | jq -r .stdout | base64 -d | tar -tzf -`. The listing contains `./g.txt` and `./nested/two.txt`. + +## Gotchas + +- HTTP ingest does not accept the host-filesystem `paths` form. That form is MCP-only. +- A manifest for a sandbox that does not exist returns `404 ENOENT`. +- Ingest has its own admission and byte budgets. `scripts/verify/ingest-admission.mjs` exercises them. diff --git a/.claude/skills/verify/features/mcp.md b/.claude/skills/verify/features/mcp.md new file mode 100644 index 00000000..92038cf7 --- /dev/null +++ b/.claude/skills/verify/features/mcp.md @@ -0,0 +1,35 @@ +# MCP + +MCP clients connect to `/mcp` over Streamable HTTP with the same bearer token as REST. They get tools for sandboxes, files, bash, and ingest. MCP and REST share one filesystem, so a write through one transport is visible through the other, and owner isolation applies to both. + +## Sub-features + +- `mcp-list` lists 11 tools: `sandbox_create`, `sandbox_list`, `sandbox_update`, `sandbox_delete`, `file_read`, `file_write`, `file_edit`, `bash_exec`, `bash_exec_batch`, `fs_ingest`, `fs_export`. +- `mcp-file` writes and reads files through tools. +- `mcp-bash` runs bash through tools, including `readOnly`. +- `mcp-cross-transport` makes MCP writes visible to REST, and the reverse. +- `mcp-isolation` returns `{"ok":false,"error":"forbidden"}` to other owners. + +## How to get to it (user POV) + +- Connect an MCP client to `/mcp` with `Authorization: Bearer `. +- With `MCP_API_KEY` set, a trusted proxy may authenticate with the static key plus an identity header. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md), and `ID` from a fresh sandbox. + +- **List tools.** Run `$S mcp list`. It prints the 11 tools with their argument names. +- **Write via MCP.** Run `$S mcp file_write "{\"id\":\"$ID\",\"path\":\"/home/user/mcp.txt\",\"content\":\"mcp-proof\\n\"}"`. The output is `{"ok":true,...,"size":10}`. +- **Read via REST.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/mcp.txt`. It prints `mcp-proof`. +- **Bash via MCP.** Run `$S mcp bash_exec "{\"id\":\"$ID\",\"script\":\"cat /home/user/mcp.txt\",\"readOnly\":true}"`. The output has `"stdout":"mcp-proof\n"` and `"exitCode":0`. +- **Isolation.** Run `EXPECT=error AS=intruder $S mcp file_read "{\"id\":\"$ID\",\"path\":\"/home/user/mcp.txt\"}"`. It prints `{"ok":false,"error":"forbidden"}` and exits 0 because the denial was expected. Then run `AS=intruder $S mcp bash_exec "{\"id\":\"$ID\",\"script\":\"cat /home/user/mcp.txt\"}"`. The output has `"exitCode":1` and `"stderr":"forbidden"`. + +## Gotchas + +- File and sandbox tool failures are `{"ok":false}` with `isError:false`. Without `EXPECT=error`, the helper exits non-zero on them. +- `bash_exec` reports denial as `exitCode:1` with `stderr:"forbidden"`. The helper exits 0, so assert the fields yourself. +- `bash_exec` takes `timeout`, not REST's `timeoutMs`. +- Static-key mode (`MCP_API_KEY`) is off on the stack. To enable it, restart with `SERVER_ENV="MCP_API_KEY=..."`, then read `docs/CONFIGURATION.md` for the identity header. diff --git a/.claude/skills/verify/features/sandboxes.md b/.claude/skills/verify/features/sandboxes.md new file mode 100644 index 00000000..07ca4823 --- /dev/null +++ b/.claude/skills/verify/features/sandboxes.md @@ -0,0 +1,39 @@ +# Sandboxes + +A sandbox is a persistent, owner-scoped filesystem with optional Python and JavaScript runtimes. Owners can create one with seed files, read and list their own sandboxes, toggle runtimes, and delete them. Other owners cannot see or touch them. + +## Sub-features + +- `sandbox-create` creates a sandbox, optionally with seed `files`, and returns `201` with `id`. +- `sandbox-get` returns metadata including `owner`. +- `sandbox-list` lists only the caller's sandboxes. +- `sandbox-update` toggles `python`, `javascript`, and `network`. +- `sandbox-isolation` denies other owners with `403 FORBIDDEN`. +- `sandbox-delete` removes the sandbox and its files. + +## How to get to it (user POV) + +- REST: `POST /v1/sandboxes`, `GET /v1/sandboxes`, `GET|PATCH|DELETE /v1/sandboxes/:id`. +- MCP: `sandbox_create`, `sandbox_list`, `sandbox_update`, `sandbox_delete`. +- SDKs: `SqlFsClient` in `clients/typescript` and `sqlfs` in `clients/python`. + +## Driving it with stack.sh + +Preconditions: + +- Baseline from the [index](./README.md). + +- **Create with seed.** Run `ID=$(EXPECT=201 $S api POST /v1/sandboxes --json '{"name":"verify","files":{"/home/user/seed.txt":"seed\n"}}' | jq -r .id)`. `ID` is a UUID. +- **Seed persisted.** Run `$S api GET /v1/sandboxes/$ID/files/home/user/seed.txt`. The body is `seed`. +- **Metadata.** Run `$S api GET /v1/sandboxes/$ID | jq -c '{owner,name}'`. The output is `{"owner":"verify-agent","name":"verify"}`. +- **List.** Run `$S api GET /v1/sandboxes | jq '[.sandboxes[].id]'`. The output contains `$ID`. +- **Update runtime.** Run `$S api PATCH /v1/sandboxes/$ID --json '{"python":true}' | jq -c '{python,javascript,network}'`. The output is `{"python":true,"javascript":false,"network":false}`. +- **Isolation.** Run `EXPECT=403 AS=intruder $S api GET /v1/sandboxes/$ID` and `AS=intruder $S api GET /v1/sandboxes | jq '.sandboxes | length'`. The first returns `FORBIDDEN` and the second prints `0`. +- **Row stored.** Run `$S psql "select owner, python from sandboxes where id = '$ID'"`. One row reads `verify-agent | t`. +- **Delete.** Run `EXPECT=204 $S api DELETE /v1/sandboxes/$ID`, then `EXPECT=404 $S api GET /v1/sandboxes/$ID`. Then run `$S psql "select count(*) from dirents where sandbox_id = '$ID'"`, which prints `0`. + +## Gotchas + +- `PATCH` is `.strict()` and needs at least one of `python`, `javascript`, or `network`. A `name` is rejected with `400 INVALID_INPUT`. +- Seed `files` share the `writeFiles` count and byte caps. +- `psql` runs as superuser and bypasses RLS. Prove isolation through the API only. diff --git a/.claude/skills/verify/scripts/journeys.sh b/.claude/skills/verify/scripts/journeys.sh new file mode 100755 index 00000000..e8ddcea2 --- /dev/null +++ b/.claude/skills/verify/scripts/journeys.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash +# Runs every journey in ../features/ against the current stack and asserts the documented results. +# Usage: journeys.sh [auth|sandboxes|exec|files|ingest|mcp ...] (default: all) +set -euo pipefail +repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) +cd "$repo" +S=.claude/skills/verify/scripts/stack.sh +doctor=$($S doctor) || { echo "$doctor"; exit 1; } +eval "$($S env)" + +passed=0 +eq() { + (($# == 3)) || { echo " FAIL eq got $# args (quoting bug): $*" >&2; exit 1; } + if [[ "$1" == "$2" ]]; then passed=$((passed + 1)); echo " ok $3" + else printf ' FAIL %s\n expected: %q\n actual: %q\n' "$3" "$2" "$1" >&2; exit 1; fi +} +has() { [[ "$1" == *"$2"* ]] && eq yes yes "$3" || eq "$1" "*$2*" "$3"; } +api() { $S api "$@" 2>/dev/null; } +mcp() { $S mcp "$@" 2>/dev/null; } +absent() { EXPECT=404 api GET "$1" >/dev/null && echo absent; } +sql_value() { $S psql "$1" | sed -n 3p | tr -d ' '; } +new_sandbox() { api POST /v1/sandboxes --json '{"name":"verify"}' | jq -r .id; } +drop() { api DELETE "/v1/sandboxes/$1" >/dev/null; } + +journey_auth() { + eq "$(EXPECT=401 AS=none api GET /v1/sandboxes | jq -r .code)" AUTH_REQUIRED "no token -> 401 AUTH_REQUIRED" + eq "$(EXPECT=403 AS=none api POST /v1/auth/bootstrap -H 'X-Auth-Secret: wrong' --json '{"sub":"boot"}' | jq -r .code)" FORBIDDEN "wrong secret -> 403" + local token + token=$(EXPECT=201 AS=none api POST /v1/auth/bootstrap -H "X-Auth-Secret: $AUTH_SECRET" --json '{"sub":"boot","expiresIn":"24h"}' | jq -r .token) + eq "$(tr -cd . <<<"$token" | wc -c | tr -d ' ')" 2 "bootstrap issues a JWT" + eq "$(EXPECT=200 AS=none api GET /v1/sandboxes -H "Authorization: Bearer $token" | jq -c .)" '{"sandboxes":[]}' "issued token is accepted" +} + +journey_sandboxes() { + local id + id=$(EXPECT=201 api POST /v1/sandboxes --json '{"name":"verify","files":{"/home/user/seed.txt":"seed\n"}}' | jq -r .id) + eq "$(api GET "/v1/sandboxes/$id/files/home/user/seed.txt")" seed "seed file persisted" + eq "$(api GET "/v1/sandboxes/$id" | jq -c '{owner,name}')" '{"owner":"verify-agent","name":"verify"}' "metadata owner and name" + eq "$(api GET /v1/sandboxes | jq --arg id "$id" '[.sandboxes[].id] | index($id) != null')" true "list contains sandbox" + eq "$(api PATCH "/v1/sandboxes/$id" --json '{"python":true}' | jq -c '{python,javascript,network}')" '{"python":true,"javascript":false,"network":false}' "runtime update" + eq "$(EXPECT=403 AS=intruder api GET "/v1/sandboxes/$id" | jq -r .code)" FORBIDDEN "other owner -> 403" + eq "$(AS=intruder api GET /v1/sandboxes | jq '.sandboxes | length')" 0 "other owner lists nothing" + has "$($S psql "select owner, python from sandboxes where id = '$id'")" "verify-agent | t" "row stored with owner and python" + EXPECT=204 api DELETE "/v1/sandboxes/$id" >/dev/null + eq "$(absent "/v1/sandboxes/$id")" absent "deleted sandbox -> 404" + eq "$(sql_value "select count(*) from dirents where sandbox_id = '$id'")" 0 "no dirents remain" +} + +journey_exec() { + local id out + id=$(new_sandbox) + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo hi > /home/user/a.txt; cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"hi\n"}' "exec-sync stdout and exit" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/a.txt")" hi "exec write persisted" + eq "$(EXPECT=422 api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo x > /home/user/ro.txt","readOnly":true}' | jq -r .code)" EREADONLY_VIOLATION "read-only rejects mutation" + eq "$(absent "/v1/sandboxes/$id/files/home/user/ro.txt")" absent "read-only left no file" + out=$(api POST "/v1/sandboxes/$id/exec" --json '{"script":"printf stream-proof"}') + has "$out" '"data":"stream-proof"' "SSE stdout event" + eq "$(grep -c '^event: exit' <<<"$out")" 1 "exactly one SSE exit" + has "$out" '"exitCode":0' "SSE exit code 0" + eq "$(api POST "/v1/sandboxes/$id/exec-sync-batch" --json '{"scripts":[{"id":"a","script":"echo one"},{"id":"b","script":"echo two"}]}' | jq -c '[.results[]|[.id,.exitCode,.stdout]]')" '[["a",0,"one\n"],["b",0,"two\n"]]' "batch results" + eq "$(EXPECT=408 api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo p > /home/user/p.txt; sleep 2; echo s > /home/user/s.txt","timeoutMs":100}' | jq -r .code)" EXEC_TIMEOUT "timeout -> 408" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/p.txt")" p "timeout kept prefix" + eq "$(absent "/v1/sandboxes/$id/files/home/user/s.txt")" absent "timeout dropped suffix" + SERVER_ENV="SQL_FS_WORKERS_ENABLED=false" $S restart --no-build >/dev/null + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"hi\n"}' "workers disabled: same result" + SERVER_ENV= $S restart --no-build >/dev/null + drop "$id" +} + +journey_files() { + local id + id=$(new_sandbox) + EXPECT=204 api POST "/v1/sandboxes/$id/mkdir" --json '{"path":"/home/user/v","recursive":true}' >/dev/null + EXPECT=204 api PUT "/v1/sandboxes/$id/files/home/user/v/note.txt" --data-binary $'before\n' -H 'Content-Type: application/octet-stream' >/dev/null + eq "$(api GET "/v1/sandboxes/$id/files/home/user/v/note.txt")" before "raw write read back" + eq "$(api PATCH "/v1/sandboxes/$id/files/home/user/v/note.txt" --json '{"oldString":"before","newString":"after"}' | jq .replacements)" 1 "edit replacement count" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/v/note.txt")" after "edited bytes visible" + EXPECT=204 api POST "/v1/sandboxes/$id/writeFiles" --json '{"files":{"/home/user/v/one.txt":"one\n","/home/user/v/two.txt":"two\n"}}' >/dev/null + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"cat /home/user/v/one.txt /home/user/v/two.txt","readOnly":true}' | jq -r .stdout)" $'one\ntwo' "bulk write visible to bash" + eq "$(api GET "/v1/sandboxes/$id/tree?prefix=/home/user/v&depth=3" | jq -c '[.[].path] | sort')" '["/home/user/v/note.txt","/home/user/v/one.txt","/home/user/v/two.txt"]' "tree lists files" + EXPECT=204 api DELETE "/v1/sandboxes/$id/files/home/user/v/note.txt" >/dev/null + eq "$(EXPECT=404 api GET "/v1/sandboxes/$id/files/home/user/v/note.txt" | jq -r .code)" ENOENT "deleted file -> ENOENT" + drop "$id" +} + +journey_ingest() { + local id body + id=$(new_sandbox) + body="{\"basePath\":\"/home/user/in\",\"files\":{\"g.txt\":\"$(printf 'ingested\n' | base64)\",\"nested/two.txt\":\"$(printf 'two\n' | base64)\"}}" + eq "$(api POST "/v1/sandboxes/$id/ingest-files" --json "$body" | jq -c .)" '{"status":"ok","fileCount":2}' "ingest reports fileCount" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/in/g.txt")" ingested "ingested top-level bytes" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/in/nested/two.txt")" two "ingested nested bytes" + eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"tar -czf - -C /home/user/in . | base64 -w 0","readOnly":true}' | jq -r .stdout | base64 -d | tar -tzf - | grep -E 'g.txt|two.txt' | sort | tr '\n' ' ')" "./g.txt ./nested/two.txt " "export archive round-trips" + drop "$id" +} + +journey_mcp() { + local id write exec read + id=$(new_sandbox) + write=$(jq -nc --arg id "$id" '{id:$id,path:"/home/user/mcp.txt",content:"mcp-proof\n"}') + exec=$(jq -nc --arg id "$id" '{id:$id,script:"cat /home/user/mcp.txt",readOnly:true}') + read=$(jq -nc --arg id "$id" '{id:$id,path:"/home/user/mcp.txt"}') + eq "$(mcp list | cut -f1 | sort | tr '\n' ' ')" "bash_exec bash_exec_batch file_edit file_read file_write fs_export fs_ingest sandbox_create sandbox_delete sandbox_list sandbox_update " "11 MCP tools" + eq "$(mcp file_write "$write" | jq -c '{ok,size}')" '{"ok":true,"size":10}' "MCP file_write" + eq "$(api GET "/v1/sandboxes/$id/files/home/user/mcp.txt")" mcp-proof "MCP write visible via REST" + eq "$(mcp bash_exec "$exec" | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"mcp-proof\n"}' "MCP bash reads it" + eq "$(EXPECT=error AS=intruder mcp file_read "$read")" '{"ok":false,"error":"forbidden"}' "MCP file tool: other owner forbidden" + eq "$(AS=intruder mcp bash_exec "$exec" | jq -c '{exitCode,stderr}')" '{"exitCode":1,"stderr":"forbidden"}' "MCP bash tool: other owner forbidden" + drop "$id" +} + +all=(auth sandboxes exec files ingest mcp) +for name in "${@:-${all[@]}}"; do + declare -F "journey_$name" >/dev/null || { echo "unknown journey: $name (have: ${all[*]})" >&2; exit 2; } + echo "== $name" + "journey_$name" +done +echo "journeys passed: $passed assertions; transcript: $DIR/transcript.jsonl" diff --git a/.claude/skills/verify/scripts/mcp.mjs b/.claude/skills/verify/scripts/mcp.mjs new file mode 100755 index 00000000..b14d583e --- /dev/null +++ b/.claude/skills/verify/scripts/mcp.mjs @@ -0,0 +1,37 @@ +#!/usr/bin/env node +// Calls one MCP tool on the stack started by stack.sh. Invoked via `stack.sh mcp`. +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; +import { record } from "./record.mjs"; + +const { STACK_BASE, STACK_TOKEN, STACK_TRANSCRIPT, STACK_SECRET } = process.env; +const [tool, rawArgs = "{}"] = process.argv.slice(2); +const client = new Client({ name: "verify-skill", version: "1.0.0" }); +await client.connect( + new StreamableHTTPClientTransport(new URL(`${STACK_BASE}/mcp`), { + requestInit: { headers: { Authorization: `Bearer ${STACK_TOKEN}` } }, + }), +); +try { + if (tool === "list") { + const { tools } = await client.listTools(); + for (const t of tools) console.log(`${t.name}\t${Object.keys(t.inputSchema?.properties ?? {}).join(",")}`); + } else { + const args = JSON.parse(rawArgs); + const response = await client.callTool({ name: tool, arguments: args }); + record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", tool, args, response }); + let failed = response.isError === true; + for (const part of response.content ?? []) { + console.log(part.type === "text" ? part.text : JSON.stringify(part)); + // File and sandbox tools report failures as isError:false with {"ok":false} (src/api/mcp/tool-errors.ts). + try { + if (part.type === "text" && JSON.parse(part.text)?.ok === false) failed = true; + } catch {} + } + console.error(`MCP ${tool}: ${failed ? "error" : "ok"}`); + const expectError = process.env.EXPECT === "error"; + if (failed !== expectError) process.exitCode = 1; + } +} finally { + await client.close(); +} diff --git a/.claude/skills/verify/scripts/record.mjs b/.claude/skills/verify/scripts/record.mjs new file mode 100755 index 00000000..091d09f8 --- /dev/null +++ b/.claude/skills/verify/scripts/record.mjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +// Appends one redacted event to a stack transcript. CLI form is used by stack.sh `api`; mcp.mjs imports record(). +import { appendFileSync, readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +const JWT = /eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g; +const SENSITIVE_KEY = /token|secret|password|authorization/i; + +function redact(value, secret) { + if (typeof value === "string") { + const masked = secret ? value.replaceAll(secret, "[REDACTED]") : value; + return masked.replace(JWT, "[REDACTED JWT]"); + } + if (Array.isArray(value)) return value.map((item) => redact(item, secret)); + if (value && typeof value === "object") { + return Object.fromEntries( + Object.entries(value).map(([key, item]) => [key, SENSITIVE_KEY.test(key) ? "[REDACTED]" : redact(item, secret)]), + ); + } + return value; +} + +export function record(file, secret, event) { + appendFileSync(file, `${JSON.stringify(redact({ at: new Date().toISOString(), ...event }, secret))}\n`); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const [file, method, path, status, as, bodyFile, ...args] = process.argv.slice(2); + const bytes = readFileSync(bodyFile); + const text = bytes.toString("utf8"); + let json; + try { + json = JSON.parse(text); + } catch {} + record(file, process.env.STACK_SECRET, { + as, + method, + path, + status: Number(status), + args, + bytes: bytes.length, + ...(json !== undefined ? { json } : { text: text.slice(0, 4096) }), + }); +} diff --git a/.claude/skills/verify/scripts/stack.sh b/.claude/skills/verify/scripts/stack.sh new file mode 100755 index 00000000..3e1867a9 --- /dev/null +++ b/.claude/skills/verify/scripts/stack.sh @@ -0,0 +1,242 @@ +#!/usr/bin/env bash +# Persistent disposable sql-fs stack for driving the API by hand. See ../SKILL.md. +set -euo pipefail + +repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) +cd "$repo" +here="$repo/.claude/skills/verify/scripts" +runs="$repo/scripts/verify/.run" +pointer="$runs/current-stack" +secret=local-verification-secret-at-least-32-bytes + +die() { echo "stack: $*" >&2; exit 1; } + +set_compose() { + read -r -a compose <<<"$COMPOSE" + compose+=(--env-file /dev/null -p "$PROJECT" -f infra/docker-compose.verify.yml) +} + +load_state() { + local dir=${STACK_DIR:-} + if [[ -z "$dir" ]]; then + [[ -f "$pointer" ]] || die "no stack is up; run: $0 up" + dir=$(cat "$pointer") + fi + [[ -f "$dir/state.env" ]] || die "missing $dir/state.env" + # shellcheck disable=SC1091 + source "$dir/state.env" + set_compose +} + +deps_current() { cmp -s pnpm-lock.yaml node_modules/.pnpm/lock.yaml; } + +ready() { curl -fsS "$BASE/readyz" 2>/dev/null | grep -q '"status":"ok"'; } + +maybe_build() { [[ "${1:-}" == "--no-build" ]] || pnpm build >&2; } + +server_alive() { + [[ -n "${PID:-}" ]] && kill -0 "$PID" 2>/dev/null && ps -p "$PID" -o command= | grep -q "dist/api/server.js" +} + +# EXEC=1 replaces the calling (sub)shell, so a backgrounded call's $! is the server's own PID. +run_env() { + ${EXEC:+exec} env -i "PATH=$PATH" "HOME=$HOME" "DATABASE_URL=$DATABASE_URL" "REDIS_URL=$REDIS_URL" \ + "REDIS_DATA_URL=$REDIS_DATA_URL" "AUTH_SECRET=$AUTH_SECRET" "$@" +} + +start_server() { + # shellcheck disable=SC2086 + EXEC=1 run_env "PORT=$PORT" ${SERVER_ENV:-} node dist/api/server.js >>"$DIR/server.log" 2>&1 & + PID=$! + local deadline=$((SECONDS + 60)) + until ready; do + if ! kill -0 "$PID" 2>/dev/null; then tail -40 "$DIR/server.log" >&2; die "server exited before ready"; fi + ((SECONDS < deadline)) || { tail -40 "$DIR/server.log" >&2; die "server not ready in 60s"; } + sleep 0.3 + done +} + +write_state() { + cat >"$DIR/state.env" </dev/null || return 0; sleep 0.1; done + kill -KILL "$PID" 2>/dev/null || true +} + +mint() { + AUTH_SECRET=$AUTH_SECRET node dist/api/cli/token.js --sub "${1:-verify-agent}" ${2:+--tenant "$2"} --expires 24h +} + +cmd_up() { + [[ -f "$pointer" ]] && die "a stack is recorded at $(cat "$pointer"); reuse it ($0 doctor) or run: $0 down" + command -v docker >/dev/null && docker info >/dev/null 2>&1 || die "Docker daemon is not running" + [[ -d node_modules ]] || die "run pnpm setup:dev first" + deps_current || die "node_modules does not match pnpm-lock.yaml; run: pnpm install --frozen-lockfile" + local id="stack-$(date +%s)-$$" + DIR="$runs/$id" + PROJECT="sqlfs-$id" + if docker compose version >/dev/null 2>&1; then COMPOSE="docker compose"; else COMPOSE="docker-compose"; fi + mkdir -p "$DIR" + set_compose + # Record state before anything can fail so `down` can always clean up. + DATABASE_URL="" REDIS_URL="" REDIS_DATA_URL="" AUTH_SECRET=$secret PORT="" BASE="" PID="" + write_state + echo "$DIR" >"$pointer" + "${compose[@]}" up -d --wait --wait-timeout 120 >&2 + DATABASE_URL="postgres://sqlfs_app:sqlfs_app@$("${compose[@]}" port postgres 5432)/sqlfs" + REDIS_URL="redis://$("${compose[@]}" port redis 6379)" + REDIS_DATA_URL="redis://$("${compose[@]}" port redis-data 6379)" + maybe_build "${1:-}" + PORT=$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})') + BASE="http://127.0.0.1:$PORT" + write_state + start_server + write_state + echo "stack up: $BASE (pid $PID)" + echo "evidence: $DIR" +} + +cmd_restart() { + local requested=${SERVER_ENV-__keep__} + load_state + [[ "$requested" == __keep__ ]] || SERVER_ENV=$requested + maybe_build "${1:-}" + stop_server + echo "--- restart $(date -u +%Y-%m-%dT%H:%M:%SZ) SERVER_ENV=${SERVER_ENV:-}" >>"$DIR/server.log" + start_server + write_state + echo "restarted: $BASE (pid $PID)" +} + +cmd_doctor() { + load_state + local bad=0 + ok() { echo "ok $*"; } + fail() { echo "FAIL $*"; bad=1; } + warn() { echo "warn $*"; } + local states + states=$("${compose[@]}" ps --format '{{.Service}}={{.Health}}' 2>/dev/null | sort | tr '\n' ' ') + [[ "$states" == "postgres=healthy redis-data=healthy redis=healthy " ]] && ok "services: $states" || fail "services: ${states:-none}" + server_alive && ok "server pid $PID runs dist/api/server.js" || fail "server pid ${PID:-?} not running (see $DIR/server.log)" + if command -v lsof >/dev/null; then + local owner + owner=$(lsof -nP -iTCP:"$PORT" -sTCP:LISTEN -t 2>/dev/null | head -1) + [[ "$owner" == "$PID" ]] && ok "port $PORT owned by pid $PID" || fail "port $PORT listener is '${owner:-none}', expected $PID" + fi + ready && ok "GET /readyz ok" || fail "GET /readyz not ok" + local code + code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(mint doctor)" "$BASE/v1/sandboxes") + [[ "$code" == 200 ]] && ok "minted token accepted (GET /v1/sandboxes 200)" || fail "GET /v1/sandboxes returned $code" + deps_current && ok "node_modules matches pnpm-lock.yaml" || + fail "node_modules does not match pnpm-lock.yaml -> run: pnpm install --frozen-lockfile && $0 restart" + local newer + newer=$(find src \( -name '*.ts' -o -name '*.sql' \) -newer dist/api/server.js -not -path '*/tests/*' 2>/dev/null | head -3) + if [[ -n "$newer" ]]; then + fail "dist is stale; source changed after build: $(echo "$newer" | tr '\n' ' ') -> run: $0 restart" + elif [[ dist/api/server.js -nt "$DIR/state.env" ]]; then + warn "dist rebuilt after the server started; run: $0 restart --no-build" + else + ok "dist is current for src (server started $STARTED_AT, HEAD $BUILT_HEAD, now $(git rev-parse --short HEAD))" + fi + [[ -n "${SERVER_ENV:-}" ]] && echo "info server env overrides: $SERVER_ENV" + echo "base $BASE" + echo "dir $DIR" + return "$bad" +} + +cmd_token() { + load_state + mint "$@" +} + +cmd_api() { + load_state + (($# >= 2)) || die "usage: $0 api METHOD PATH [curl args...] (env: AS=sub|none, TENANT=, EXPECT=status)" + local method=$1 path=$2 as=${AS:-verify-agent} + shift 2 + local -a auth=() + [[ "$as" == none ]] || auth=(-H "Authorization: Bearer $(mint "$as" "${TENANT:-}")") + local body status + body=$(mktemp) + status=$(curl -sS -X "$method" -o "$body" -w '%{http_code}' ${auth[@]+"${auth[@]}"} "$@" "$BASE$path") + STACK_SECRET=$AUTH_SECRET node "$here/record.mjs" "$DIR/transcript.jsonl" "$method" "$path" "$status" "$as" "$body" "$@" + cat "$body" + [[ -s "$body" && "$(tail -c1 "$body" | od -An -c | tr -d ' ')" != '\n' ]] && echo + rm -f "$body" + echo "HTTP $status $method $path" >&2 + local expect=${EXPECT:-2xx} + if [[ "$expect" == 2xx ]]; then [[ "$status" == 2* ]] || die "expected 2xx, got $status" + else [[ "$status" == "$expect" ]] || die "expected $expect, got $status"; fi +} + +cmd_mcp() { + load_state + (($# >= 1)) || die "usage: $0 mcp list | $0 mcp TOOL '{json args}' (env: AS=sub, EXPECT=error)" + STACK_BASE=$BASE STACK_TOKEN=$(mint "${AS:-verify-agent}" "${TENANT:-}") STACK_TRANSCRIPT=$DIR/transcript.jsonl \ + STACK_SECRET=$AUTH_SECRET node "$here/mcp.mjs" "$@" +} + +cmd_psql() { + load_state + (($# == 1)) || die "usage: $0 psql 'SQL' (runs as the postgres superuser; bypasses RLS for inspection)" + "${compose[@]}" exec -T postgres psql -U postgres -d sqlfs -v ON_ERROR_STOP=1 -c "$1" +} + +cmd_suite() { + load_state + (($# >= 1)) || die "usage: $0 suite NAME [args] (NAME: integration or a scripts/verify/*.mjs basename)" + local name=$1 + shift + local -a suite_env=("VERIFY_ENV_FILE=none" "VERIFY_EVIDENCE_DIR=$DIR/suites" "VERIFY_PYTHON=$repo/clients/python/.venv/bin/python") + mkdir -p "$DIR/suites" + if [[ "$name" == integration ]]; then + local report + report="$DIR/suites/integration-$(date +%s).json" + run_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@" + # Same gate as verify-local.sh: a run that skips or executes nothing is not proof. + node -e ' + const r = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); + if (!(r.numPassedTests > 0) || r.numFailedTests || r.numPendingTests) { + console.error(`integration: passed=${r.numPassedTests} failed=${r.numFailedTests} skipped=${r.numPendingTests}`); + process.exit(1); + }' "$report" + else + [[ -f "scripts/verify/$name.mjs" ]] || die "no scripts/verify/$name.mjs" + run_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@" + fi +} + +cmd_down() { + load_state + stop_server + "${compose[@]}" logs --no-color >"$DIR/services.log" 2>&1 || true + "${compose[@]}" down --volumes --remove-orphans >&2 + [[ -f "$pointer" && "$(cat "$pointer")" == "$DIR" ]] && rm -f "$pointer" + echo "stack down; evidence kept at $DIR" +} + +cmd=${1:-} +shift || true +case "$cmd" in +up | restart | doctor | token | api | mcp | psql | suite | down) "cmd_$cmd" "$@" ;; +env) load_state && cat "$DIR/state.env" ;; +*) die "usage: $0 {up [--no-build]|doctor|token [sub] [tenant]|api METHOD PATH ...|mcp TOOL JSON|psql SQL|suite NAME|restart [--no-build]|env|down}" ;; +esac diff --git a/AGENTS.md b/AGENTS.md index c3efe609..6dbad365 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,38 @@ pnpm verify:architecture # also worker failure, stream finalization, input budge Use `pnpm check:typescript` or `pnpm check:python` for SDK-only changes. Run a targeted core test with `pnpm exec vitest run src/sql-fs/tests/unit/sql-fs.cache.test.ts`. A nonexistent test selection must fail. `pnpm test:integration` is the lower-level command for an already migrated disposable database with `DATABASE_URL` and `REDIS_URL`; it can skip without those variables. Prefer `pnpm verify:local` for completion evidence. +## Verify while you work + +Prove every runtime change against a real server before handing back. The full gates above are completion evidence. While iterating, use the verification skill in [`.claude/skills/verify/`](.claude/skills/verify/SKILL.md). In Claude Code, invoke it as `verify`. Other agents read `SKILL.md` and run its scripts directly. The skill maps each kind of change to its required gates, and [`features/`](.claude/skills/verify/features/README.md) holds the client journeys for auth, sandboxes, exec, files, ingest, and MCP. + +```bash +S=.claude/skills/verify/scripts/stack.sh +$S up # disposable Postgres/Redis + built API server; keeps running +$S doctor # read-only health check; FAILs when dist/ is older than src/ +$S restart # rebuild and restart after each source edit (data kept) +$S api POST /v1/sandboxes --json '{"name":"x"}' # authed request; EXPECT=403 AS=intruder to assert denial +$S mcp list # MCP over Streamable HTTP; $S mcp TOOL '' +$S psql 'select ...' # inspect side effects (superuser, bypasses RLS: never proof of isolation) +$S suite contracts # any scripts/verify/.mjs, or `integration [filter]`, against the stack +.claude/skills/verify/scripts/journeys.sh # every mapped journey, ~40 assertions in ~10s +$S down # remove only this run's server, containers, and volumes; evidence stays +``` + +Evidence lands in `scripts/verify/.run/stack-/` (`transcript.jsonl`, `server.log`, `suites/`). The stack can run alongside `pnpm verify:local`, which starts its own. + +`scripts/verify/` holds the live suites. Every suite except `large-files` runs against the stack through `$S suite `; the full gates run them as listed: + +| Suite | Proves | Run by | +| --- | --- | --- | +| `contracts` | HTTP and MCP behavior across replicas, validated against `/openapi.json` | `verify:local` | +| `output-cap` | worker output limits in both worker modes | `verify:local` | +| `shutdown` | graceful shutdown ends MCP listener streams and refuses new connections | `verify:local` | +| `sdk-clients` (+ `sdk-python.py`) | live TypeScript and Python SDKs | `verify:local` | +| `worker-failures`, `sse-finalization`, `input-budgets`, `tenants` | worker loss, stream finalization, input budgets, tenant isolation | `verify:architecture` | +| `ingest-admission`, `large-files` | ingest budgets, large-file profiling | manual; large files need cgroup v2 | + +`local-api.mjs` is the shared harness: owned servers, tokens, schema validation, `proof.json`. `*-faults.mjs` files are fault injectors loaded by the suites, not entry points. When you add an endpoint or client-visible behavior, add or extend a suite here, add its journey to `features/` and `journeys.sh`, and update `openapi-spec.ts`. + See [READINESS.md](docs/READINESS.md) for verified scope and hosted follow-up. Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup and contribution steps, [DEVELOPER.md](docs/DEVELOPER.md) before changing consistency or session ownership, [README.md](README.md) for API behavior, [CONFIGURATION.md](docs/CONFIGURATION.md) for runtime settings, and [AUTOSCALING.md](docs/AUTOSCALING.md) for probes, metrics and replica scaling. Python type checking targets the oldest supported interpreter. `pnpm check:python:types` uses uv to provision Python 3.9 in `clients/python/.venv-py39`, separate from the Python 3.12 development environment. This keeps the dependency syntax aligned with the declared target. Mypy stays below version 2 while Python 3.9 is supported. @@ -70,6 +102,6 @@ Migrations are handwritten numbered SQL files. Add the next migration without ed Work autonomously on local implementation and reversible verification. Use a separate checkout for concurrent writers. Do not overwrite another agent's changes. Keep secrets out of commits and evidence. Use the disposable verification stack for tests that create databases or inject faults. -Before handing back, run `pnpm check`, then `pnpm verify:local` for runtime, database, dependency, or SDK behavior changes. Run `pnpm security:audit` after dependency changes. Report the commands, results, and any unverified claims. A skipped test is not proof. +Before handing back, run `pnpm check`, then `pnpm verify:local` for runtime, database, dependency, or SDK behavior changes. For runtime changes, also drive the changed behavior on the verify stack and run `journeys.sh` (see [Verify while you work](#verify-while-you-work)). Cite the evidence paths. Run `pnpm security:audit` after dependency changes. Report the commands, results, and any unverified claims. A skipped test is not proof. Add a `.changeset/*.md` entry for behavior changes. Do not bump versions or edit `CHANGELOG.md` in feature work. Deployment, release, merging, production data changes, and shared-branch force pushes need explicit task authorization. Repository changes do not prove that hosted CI or branch protection is active. From bd55c2b0cc2a32cb2d17f3fd97944e8a3cc787ac Mon Sep 17 00:00:00 2001 From: QuangNguyen2609 Date: Sat, 17 Oct 2026 15:12:47 +1030 Subject: [PATCH 2/4] test(verify): drop narrating comments and defensive guards from the verify skill scripts Name the shapes those comments explained: exec_in_stack_env runs in a subshell so the server's PID is $!, SERVER_ENV is split into an array, register_for_teardown records the stack before anything can fail, and isToolFailure detects {"ok":false} MCP replies. Co-Authored-By: Claude Opus 4.6 --- .claude/skills/verify/scripts/journeys.sh | 2 -- .claude/skills/verify/scripts/mcp.mjs | 23 ++++++++++--------- .claude/skills/verify/scripts/record.mjs | 3 +-- .claude/skills/verify/scripts/stack.sh | 27 ++++++++++++----------- 4 files changed, 28 insertions(+), 27 deletions(-) diff --git a/.claude/skills/verify/scripts/journeys.sh b/.claude/skills/verify/scripts/journeys.sh index e8ddcea2..ccc10f96 100755 --- a/.claude/skills/verify/scripts/journeys.sh +++ b/.claude/skills/verify/scripts/journeys.sh @@ -1,6 +1,4 @@ #!/usr/bin/env bash -# Runs every journey in ../features/ against the current stack and asserts the documented results. -# Usage: journeys.sh [auth|sandboxes|exec|files|ingest|mcp ...] (default: all) set -euo pipefail repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) cd "$repo" diff --git a/.claude/skills/verify/scripts/mcp.mjs b/.claude/skills/verify/scripts/mcp.mjs index b14d583e..67b337dd 100755 --- a/.claude/skills/verify/scripts/mcp.mjs +++ b/.claude/skills/verify/scripts/mcp.mjs @@ -1,9 +1,17 @@ #!/usr/bin/env node -// Calls one MCP tool on the stack started by stack.sh. Invoked via `stack.sh mcp`. import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; import { record } from "./record.mjs"; +function isToolFailure(part) { + if (part.type !== "text") return false; + try { + return JSON.parse(part.text)?.ok === false; + } catch { + return false; + } +} + const { STACK_BASE, STACK_TOKEN, STACK_TRANSCRIPT, STACK_SECRET } = process.env; const [tool, rawArgs = "{}"] = process.argv.slice(2); const client = new Client({ name: "verify-skill", version: "1.0.0" }); @@ -15,19 +23,14 @@ await client.connect( try { if (tool === "list") { const { tools } = await client.listTools(); - for (const t of tools) console.log(`${t.name}\t${Object.keys(t.inputSchema?.properties ?? {}).join(",")}`); + for (const t of tools) console.log(`${t.name}\t${Object.keys(t.inputSchema.properties ?? {}).join(",")}`); } else { const args = JSON.parse(rawArgs); const response = await client.callTool({ name: tool, arguments: args }); record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", tool, args, response }); - let failed = response.isError === true; - for (const part of response.content ?? []) { - console.log(part.type === "text" ? part.text : JSON.stringify(part)); - // File and sandbox tools report failures as isError:false with {"ok":false} (src/api/mcp/tool-errors.ts). - try { - if (part.type === "text" && JSON.parse(part.text)?.ok === false) failed = true; - } catch {} - } + const parts = response.content; + for (const part of parts) console.log(part.type === "text" ? part.text : JSON.stringify(part)); + const failed = response.isError === true || parts.some(isToolFailure); console.error(`MCP ${tool}: ${failed ? "error" : "ok"}`); const expectError = process.env.EXPECT === "error"; if (failed !== expectError) process.exitCode = 1; diff --git a/.claude/skills/verify/scripts/record.mjs b/.claude/skills/verify/scripts/record.mjs index 091d09f8..4c90ad29 100755 --- a/.claude/skills/verify/scripts/record.mjs +++ b/.claude/skills/verify/scripts/record.mjs @@ -1,5 +1,4 @@ #!/usr/bin/env node -// Appends one redacted event to a stack transcript. CLI form is used by stack.sh `api`; mcp.mjs imports record(). import { appendFileSync, readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; @@ -8,7 +7,7 @@ const SENSITIVE_KEY = /token|secret|password|authorization/i; function redact(value, secret) { if (typeof value === "string") { - const masked = secret ? value.replaceAll(secret, "[REDACTED]") : value; + const masked = value.replaceAll(secret, "[REDACTED]"); return masked.replace(JWT, "[REDACTED JWT]"); } if (Array.isArray(value)) return value.map((item) => redact(item, secret)); diff --git a/.claude/skills/verify/scripts/stack.sh b/.claude/skills/verify/scripts/stack.sh index 3e1867a9..f1ead155 100755 --- a/.claude/skills/verify/scripts/stack.sh +++ b/.claude/skills/verify/scripts/stack.sh @@ -1,5 +1,4 @@ #!/usr/bin/env bash -# Persistent disposable sql-fs stack for driving the API by hand. See ../SKILL.md. set -euo pipefail repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) @@ -38,15 +37,15 @@ server_alive() { [[ -n "${PID:-}" ]] && kill -0 "$PID" 2>/dev/null && ps -p "$PID" -o command= | grep -q "dist/api/server.js" } -# EXEC=1 replaces the calling (sub)shell, so a backgrounded call's $! is the server's own PID. -run_env() { - ${EXEC:+exec} env -i "PATH=$PATH" "HOME=$HOME" "DATABASE_URL=$DATABASE_URL" "REDIS_URL=$REDIS_URL" \ +exec_in_stack_env() { + exec env -i "PATH=$PATH" "HOME=$HOME" "DATABASE_URL=$DATABASE_URL" "REDIS_URL=$REDIS_URL" \ "REDIS_DATA_URL=$REDIS_DATA_URL" "AUTH_SECRET=$AUTH_SECRET" "$@" } start_server() { - # shellcheck disable=SC2086 - EXEC=1 run_env "PORT=$PORT" ${SERVER_ENV:-} node dist/api/server.js >>"$DIR/server.log" 2>&1 & + local -a overrides=() + read -r -a overrides <<<"${SERVER_ENV:-}" + (exec_in_stack_env "PORT=$PORT" ${overrides[@]+"${overrides[@]}"} node dist/api/server.js >>"$DIR/server.log" 2>&1) & PID=$! local deadline=$((SECONDS + 60)) until ready; do @@ -74,6 +73,12 @@ STARTED_AT='$(date -u +%Y-%m-%dT%H:%M:%SZ)' EOF } +register_for_teardown() { + DATABASE_URL="" REDIS_URL="" REDIS_DATA_URL="" AUTH_SECRET=$secret PORT="" BASE="" PID="" + write_state + echo "$DIR" >"$pointer" +} + stop_server() { server_alive || return 0 kill -TERM "$PID" @@ -96,10 +101,7 @@ cmd_up() { if docker compose version >/dev/null 2>&1; then COMPOSE="docker compose"; else COMPOSE="docker-compose"; fi mkdir -p "$DIR" set_compose - # Record state before anything can fail so `down` can always clean up. - DATABASE_URL="" REDIS_URL="" REDIS_DATA_URL="" AUTH_SECRET=$secret PORT="" BASE="" PID="" - write_state - echo "$DIR" >"$pointer" + register_for_teardown "${compose[@]}" up -d --wait --wait-timeout 120 >&2 DATABASE_URL="postgres://sqlfs_app:sqlfs_app@$("${compose[@]}" port postgres 5432)/sqlfs" REDIS_URL="redis://$("${compose[@]}" port redis 6379)" @@ -210,8 +212,7 @@ cmd_suite() { if [[ "$name" == integration ]]; then local report report="$DIR/suites/integration-$(date +%s).json" - run_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@" - # Same gate as verify-local.sh: a run that skips or executes nothing is not proof. + (exec_in_stack_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@") node -e ' const r = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); if (!(r.numPassedTests > 0) || r.numFailedTests || r.numPendingTests) { @@ -220,7 +221,7 @@ cmd_suite() { }' "$report" else [[ -f "scripts/verify/$name.mjs" ]] || die "no scripts/verify/$name.mjs" - run_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@" + (exec_in_stack_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@") fi } From 77fc44ddfe162b6307a00d1523573617bda0c75e Mon Sep 17 00:00:00 2001 From: QuangNguyen2609 Date: Sat, 17 Oct 2026 15:38:51 +1030 Subject: [PATCH 3/4] test(verify): share the integration report gate between verify-local and the verify skill Both checked that an integration run executed tests with none failed or skipped, in two copies. assert-integration-report.mjs is now the single check, with node tests for empty, failed and skipped reports. Co-Authored-By: Claude Opus 4.6 --- .claude/skills/verify/scripts/stack.sh | 7 +--- AGENTS.md | 2 +- scripts/verify-local.sh | 9 +---- scripts/verify/assert-integration-report.mjs | 14 +++++++ .../verify/assert-integration-report.test.mjs | 37 +++++++++++++++++++ 5 files changed, 54 insertions(+), 15 deletions(-) create mode 100644 scripts/verify/assert-integration-report.mjs create mode 100644 scripts/verify/assert-integration-report.test.mjs diff --git a/.claude/skills/verify/scripts/stack.sh b/.claude/skills/verify/scripts/stack.sh index f1ead155..3a960620 100755 --- a/.claude/skills/verify/scripts/stack.sh +++ b/.claude/skills/verify/scripts/stack.sh @@ -213,12 +213,7 @@ cmd_suite() { local report report="$DIR/suites/integration-$(date +%s).json" (exec_in_stack_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@") - node -e ' - const r = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); - if (!(r.numPassedTests > 0) || r.numFailedTests || r.numPendingTests) { - console.error(`integration: passed=${r.numPassedTests} failed=${r.numFailedTests} skipped=${r.numPendingTests}`); - process.exit(1); - }' "$report" + node scripts/verify/assert-integration-report.mjs "$report" else [[ -f "scripts/verify/$name.mjs" ]] || die "no scripts/verify/$name.mjs" (exec_in_stack_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@") diff --git a/AGENTS.md b/AGENTS.md index 6dbad365..0dfb1d03 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,7 +49,7 @@ Evidence lands in `scripts/verify/.run/stack-/` (`transcript.jsonl`, `server | `worker-failures`, `sse-finalization`, `input-budgets`, `tenants` | worker loss, stream finalization, input budgets, tenant isolation | `verify:architecture` | | `ingest-admission`, `large-files` | ingest budgets, large-file profiling | manual; large files need cgroup v2 | -`local-api.mjs` is the shared harness: owned servers, tokens, schema validation, `proof.json`. `*-faults.mjs` files are fault injectors loaded by the suites, not entry points. When you add an endpoint or client-visible behavior, add or extend a suite here, add its journey to `features/` and `journeys.sh`, and update `openapi-spec.ts`. +`local-api.mjs` is the shared harness: owned servers, tokens, schema validation, `proof.json`. `assert-integration-report.mjs` is the shared check that an integration run executed tests with none failed or skipped. `*-faults.mjs` files are fault injectors loaded by the suites, not entry points. When you add an endpoint or client-visible behavior, add or extend a suite here, add its journey to `features/` and `journeys.sh`, and update `openapi-spec.ts`. See [READINESS.md](docs/READINESS.md) for verified scope and hosted follow-up. Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup and contribution steps, [DEVELOPER.md](docs/DEVELOPER.md) before changing consistency or session ownership, [README.md](README.md) for API behavior, [CONFIGURATION.md](docs/CONFIGURATION.md) for runtime settings, and [AUTOSCALING.md](docs/AUTOSCALING.md) for probes, metrics and replica scaling. diff --git a/scripts/verify-local.sh b/scripts/verify-local.sh index f28f283e..3db8e900 100644 --- a/scripts/verify-local.sh +++ b/scripts/verify-local.sh @@ -52,14 +52,7 @@ pnpm --dir clients/typescript build "${verify_env[@]}" node scripts/verify/output-cap.mjs "${verify_env[@]}" node scripts/verify/shutdown.mjs "${verify_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$evidence/integration.json" -node --input-type=module - "$evidence/integration.json" <<'JS' -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; -const report = JSON.parse(readFileSync(process.argv[2], "utf8")); -assert.ok(report.numPassedTests > 0, "Integration tests must execute"); -assert.equal(report.numFailedTests, 0, "Integration tests must pass"); -assert.equal(report.numPendingTests, 0, "Integration tests must not silently skip"); -JS +node scripts/verify/assert-integration-report.mjs "$evidence/integration.json" "${verify_env[@]}" node scripts/verify/sdk-clients.mjs if [[ "$profile" == architecture ]]; then diff --git a/scripts/verify/assert-integration-report.mjs b/scripts/verify/assert-integration-report.mjs new file mode 100644 index 00000000..b26d7a7e --- /dev/null +++ b/scripts/verify/assert-integration-report.mjs @@ -0,0 +1,14 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +export function assertIntegrationReport(report) { + assert.ok(report.numPassedTests > 0, "Integration tests must execute"); + assert.equal(report.numFailedTests, 0, "Integration tests must pass"); + assert.equal(report.numPendingTests, 0, "Integration tests must not silently skip"); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + assertIntegrationReport(JSON.parse(readFileSync(process.argv[2], "utf8"))); +} diff --git a/scripts/verify/assert-integration-report.test.mjs b/scripts/verify/assert-integration-report.test.mjs new file mode 100644 index 00000000..e4fee2ea --- /dev/null +++ b/scripts/verify/assert-integration-report.test.mjs @@ -0,0 +1,37 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import { assertIntegrationReport } from "./assert-integration-report.mjs"; + +const passing = { numPassedTests: 3, numFailedTests: 0, numPendingTests: 0 }; + +test("integration report proof rejects empty, failed or skipped runs", () => { + assertIntegrationReport(passing); + for (const report of [ + { ...passing, numPassedTests: 0 }, + { ...passing, numFailedTests: 1 }, + { ...passing, numPendingTests: 1 }, + {}, + ]) { + assert.throws(() => assertIntegrationReport(report)); + } +}); + +test("integration report CLI exits non-zero for a skipped run", () => { + const script = fileURLToPath(new URL("./assert-integration-report.mjs", import.meta.url)); + const directory = mkdtempSync(join(tmpdir(), "integration-report-")); + const write = (name, report) => { + const file = join(directory, name); + writeFileSync(file, JSON.stringify(report)); + return file; + }; + execFileSync(process.execPath, [script, write("pass.json", passing)]); + assert.throws( + () => execFileSync(process.execPath, [script, write("skip.json", { ...passing, numPendingTests: 2 })], { stdio: "pipe" }), + /must not silently skip/, + ); +}); From 2b4408d4700b1bbbab56ad335b4e4b4a560bbd93 Mon Sep 17 00:00:00 2001 From: QuangNguyen2609 Date: Sat, 17 Oct 2026 17:12:17 +1030 Subject: [PATCH 4/4] test(verify): address review findings on the verify skill Redact sensitive keys inside JSON-looking strings and sensitive headers, cap every recorded body at 4 KiB, keep SERVER_ENV values out of logs and doctor output, write state.env shell-safely and owner-only, record the server PID before readiness polling, keep doctor running when the server is down, restore the default server env when a journey fails, accept only suite entry points in `suite`, and correct the auth, exec, sandbox and prerequisite docs. Co-Authored-By: Claude Opus 4.6 --- .claude/skills/verify/SKILL.md | 12 ++--- .claude/skills/verify/features/auth.md | 6 +-- .claude/skills/verify/features/exec.md | 2 +- .claude/skills/verify/features/sandboxes.md | 4 +- .claude/skills/verify/scripts/journeys.sh | 3 ++ .claude/skills/verify/scripts/mcp.mjs | 7 +-- .claude/skills/verify/scripts/record.mjs | 58 +++++++++++++-------- .claude/skills/verify/scripts/stack.sh | 57 +++++++++++--------- AGENTS.md | 9 ++-- 9 files changed, 92 insertions(+), 66 deletions(-) diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md index 400ed398..e0aa2dce 100644 --- a/.claude/skills/verify/SKILL.md +++ b/.claude/skills/verify/SKILL.md @@ -34,7 +34,7 @@ Also, from CLAUDE.md: add `.changeset/*.md` for behavior changes, update `src/ap ## Launch -Preconditions: Docker daemon running, `pnpm setup:dev` done (needs `node_modules` and `clients/python/.venv`). `jq` makes JSON assertions easier but is optional. +Preconditions: Docker daemon running, `pnpm setup:dev` done (needs `node_modules` and `clients/python/.venv`), and `jq` on `PATH`. `journeys.sh` and the feature recipes need `jq`. ```bash $S up # compose stack + pnpm build + API server; ~10s @@ -45,7 +45,7 @@ $S up --no-build # reuse the current dist/ `up` refuses while a stack is recorded for this checkout. Reuse it (`$S doctor`) or `$S down` first. After a failed `up`, run `$S down` too, because the containers may be running. -Server environment overrides: `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false SQL_FS_WORKER_OUTPUT_BYTES=1048576" $S restart`. Overrides persist across later restarts until you pass `SERVER_ENV= $S restart`. Values are split on spaces, so use compact JSON with no spaces. +Server environment overrides: `SERVER_ENV="SQL_FS_WORKERS_ENABLED=false SQL_FS_WORKER_OUTPUT_BYTES=1048576" $S restart`. Overrides persist across later restarts until you pass `SERVER_ENV= $S restart`. Values are split on spaces, so use compact JSON with no spaces. `server.log` and `doctor` show only the override names; the values stay in the owner-only `state.env`. ## Doctor @@ -56,7 +56,7 @@ Run `$S doctor` first, and again whenever anything looks off. It is read-only an - that PID owns the recorded port (via `lsof`) - `/readyz` ok and a freshly minted token gets `200` on `GET /v1/sandboxes` - `node_modules` matches `pnpm-lock.yaml`, compared byte-for-byte with `node_modules/.pnpm/lock.yaml`. After a pull that changed the lockfile, the old install keeps running and unit tests fail for reasons unrelated to your diff. Fix it with `pnpm install --frozen-lockfile && $S restart`. `up` refuses to start on drift. -- `dist/` is not older than any non-test `src/**/*.ts`. **A stale build is the most common false result**: you edited source, but the server and every suite still run the old `dist/`. Fix it with `$S restart`. +- `dist/` is not older than any non-test `src/**/*.ts` or `src/**/*.sql`, so a migration edit also fails until you restart. **A stale build is the most common false result**: you edited source, but the server and every suite still run the old `dist/`. Fix it with `$S restart`. ## Drive @@ -70,7 +70,7 @@ After every source edit: `$S restart` (rebuild + restart, same port, data kept), | `$S mcp list` / `$S mcp TOOL ''` | MCP over Streamable HTTP with the same `AS`/`TENANT`. Exits non-zero on `isError` or a `{"ok":false}` payload. Use `EXPECT=error` to assert one of those. `bash_exec` denials are neither; assert their `exitCode`/`stderr`. | | `$S token [sub] [tenant]` | Print a 24h JWT signed with the stack's `AUTH_SECRET`. | | `$S psql 'SQL'` | Run SQL as the `postgres` superuser on the stack DB. This **bypasses RLS** and is only for inspecting side effects; never use it to prove isolation. Tables: `sandboxes`, `inodes`, `dirents`, `blobs`, `sandbox_epochs`. | -| `$S suite NAME [args]` | Run `scripts/verify/NAME.mjs` with the stack's DB/Redis. Suites start their own servers from `dist/`, so restart (build) first. `NAME=integration [file filters]` runs vitest integration and writes a JSON report. | +| `$S suite NAME [args]` | Run the suite entry point `scripts/verify/NAME.mjs` against the stack's Postgres and Redis. Suites start their own API servers from `dist/`, so a passing suite does not prove the `$S up` server; restart (build) first. An entry point is a non-test file that creates a `LocalApiRun`; helper modules such as `local-api.mjs` or `measurements.mjs` are rejected with the list of entry points. `NAME=integration [file filters]` runs vitest integration and writes a JSON report. | | `$S env` | Print the recorded state (URLs, port, PID, run dir). | Use `curl --json ''` for JSON (sets both headers). Use `--data-binary @file -H 'Content-Type: application/octet-stream'` for raw file bytes. @@ -81,11 +81,11 @@ The feature map in [features/](features/README.md) has the exact journeys. Alway Everything lands in `scripts/verify/.run/stack-/` (gitignored) and survives `down`: -- `transcript.jsonl`: every `api`/`mcp` call with caller, status, and body (≤4 KiB). The stack secret, JWTs (in curl args or bodies), and token/secret/password/authorization keys are redacted. +- `transcript.jsonl`: every `api` call with caller, method, path, curl args, and HTTP status, and every `mcp` call (including `list`) with caller, tool, args, and whether it failed. Each record keeps the response body up to 4 KiB; a longer body is cut, marked `truncated: true`, and `bytes` keeps the full size. The stack secret, JWTs, values of token/secret/password/authorization keys (also inside JSON strings such as `--json` args and MCP text parts), and sensitive header values are redacted. - `server.log`: server stdout/stderr across restarts, with a `--- restart` line at each one. - `services.log`: Postgres/Redis logs, written at `down`. - `suites/-/proof.json` and `requests.jsonl`, plus `suites/integration-*.json`. -- `state.env`: the stack's coordinates. It contains the local-only verification secret. +- `state.env`: the stack's coordinates. It contains the local-only verification secret and any `SERVER_ENV` values, so the run directory is owner-only. `pnpm verify:local` writes its own `scripts/verify/.run/verify-*/` directory with `proof.json` per suite and `integration.json`. diff --git a/.claude/skills/verify/features/auth.md b/.claude/skills/verify/features/auth.md index 395e4688..322de036 100644 --- a/.claude/skills/verify/features/auth.md +++ b/.claude/skills/verify/features/auth.md @@ -1,10 +1,10 @@ # Auth -Every `/v1/*` and `/mcp` call carries a bearer JWT signed with `AUTH_SECRET`. A caller with the secret can mint a token through `POST /v1/auth/bootstrap`. Requests without a valid token are rejected before they touch any sandbox. +Every `/v1/*` and `/mcp` call carries a bearer JWT signed with `AUTH_SECRET`, with two exceptions. `POST /v1/auth/bootstrap` takes no bearer token: a caller proves it holds the secret with `X-Auth-Secret` and gets a token back. When `MCP_API_KEY` is set, `/mcp` also accepts that static key as the bearer value and takes the owner from an identity header. Requests without a valid token are rejected before they touch any sandbox. ## Sub-features -- `auth-required` rejects unauthenticated `/v1` requests with `401 AUTH_REQUIRED`. +- `auth-required` rejects `/v1` requests without a bearer token with `401 AUTH_REQUIRED`. `POST /v1/auth/bootstrap` is exempt. - `auth-bootstrap` issues a token for a `sub` when `X-Auth-Secret` matches. - `auth-bootstrap-denied` rejects a wrong secret with `403 FORBIDDEN` before parsing the body. - `auth-token-cli` mints tokens offline with `pnpm token:create`. @@ -28,6 +28,6 @@ Preconditions: ## Gotchas -- Bootstrap is rate-limited per IP. A burst of bad attempts returns `429` for a while. +- Every `POST /v1/auth/bootstrap` counts toward a per-IP limit, successful mints included: 5 per 60 seconds by default (`BOOTSTRAP_RATE_LIMIT_MAX`, `BOOTSTRAP_RATE_LIMIT_WINDOW_MS`). The `auth` journey makes two calls, so a third run within a minute gets `429`. - `POST /v1/auth/admin` needs `ADMIN_SECRET`, which the stack does not set. It returns `500 ADMIN_NOT_CONFIGURED` unless you pass `SERVER_ENV="ADMIN_SECRET=..." $S restart`. - A token without a `tenant` claim resolves to the `default` tenant. diff --git a/.claude/skills/verify/features/exec.md b/.claude/skills/verify/features/exec.md index 60854605..5b7c1b18 100644 --- a/.claude/skills/verify/features/exec.md +++ b/.claude/skills/verify/features/exec.md @@ -33,4 +33,4 @@ Preconditions: - A non-zero script exit is still HTTP `200`. Assert `exitCode`, not just the status. - Python and JavaScript commands need the runtime enabled (`PATCH {"python":true}`). The first `python3` call can take tens of seconds. - Output above `SQL_FS_WORKER_OUTPUT_BYTES` fails with exit `126`. `$S suite output-cap` covers both worker modes. -- Never auto-retry an execution whose reply was lost. Unknown-outcome errors are typed on purpose; see `worker-failures`. +- The outcome of an `exec` or `execBatch` is unknown after `503` with code `EEXEC_OUTCOME_UNKNOWN`, `EDRIVERFAULT`, or `ECOHERENCE` (writes may have been applied), or after an SDK `TransportError` starting with `network error after` or `script outcome unknown`. Never retry those automatically. Read the sandbox state before running anything else. `worker-failures` covers them. diff --git a/.claude/skills/verify/features/sandboxes.md b/.claude/skills/verify/features/sandboxes.md index 07ca4823..340d24a3 100644 --- a/.claude/skills/verify/features/sandboxes.md +++ b/.claude/skills/verify/features/sandboxes.md @@ -7,7 +7,7 @@ A sandbox is a persistent, owner-scoped filesystem with optional Python and Java - `sandbox-create` creates a sandbox, optionally with seed `files`, and returns `201` with `id`. - `sandbox-get` returns metadata including `owner`. - `sandbox-list` lists only the caller's sandboxes. -- `sandbox-update` toggles `python`, `javascript`, and `network`. +- `sandbox-update` sets `python`, `javascript`, and `network` to `true` or `false`. An omitted flag keeps its current value. - `sandbox-isolation` denies other owners with `403 FORBIDDEN`. - `sandbox-delete` removes the sandbox and its files. @@ -15,7 +15,7 @@ A sandbox is a persistent, owner-scoped filesystem with optional Python and Java - REST: `POST /v1/sandboxes`, `GET /v1/sandboxes`, `GET|PATCH|DELETE /v1/sandboxes/:id`. - MCP: `sandbox_create`, `sandbox_list`, `sandbox_update`, `sandbox_delete`. -- SDKs: `SqlFsClient` in `clients/typescript` and `sqlfs` in `clients/python`. +- SDKs: `Client` from `sql-fs-sdk` (`clients/typescript`) and `Client` from `sqlfs` (`clients/python`). ## Driving it with stack.sh diff --git a/.claude/skills/verify/scripts/journeys.sh b/.claude/skills/verify/scripts/journeys.sh index ccc10f96..ecd2045d 100755 --- a/.claude/skills/verify/scripts/journeys.sh +++ b/.claude/skills/verify/scripts/journeys.sh @@ -3,6 +3,7 @@ set -euo pipefail repo=$(cd "$(dirname "$0")" && git rev-parse --show-toplevel) cd "$repo" S=.claude/skills/verify/scripts/stack.sh +command -v jq >/dev/null || { echo "journeys.sh needs jq on PATH" >&2; exit 2; } doctor=$($S doctor) || { echo "$doctor"; exit 1; } eval "$($S env)" @@ -59,9 +60,11 @@ journey_exec() { eq "$(EXPECT=408 api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"echo p > /home/user/p.txt; sleep 2; echo s > /home/user/s.txt","timeoutMs":100}' | jq -r .code)" EXEC_TIMEOUT "timeout -> 408" eq "$(api GET "/v1/sandboxes/$id/files/home/user/p.txt")" p "timeout kept prefix" eq "$(absent "/v1/sandboxes/$id/files/home/user/s.txt")" absent "timeout dropped suffix" + trap 'SERVER_ENV= $S restart --no-build >/dev/null' EXIT SERVER_ENV="SQL_FS_WORKERS_ENABLED=false" $S restart --no-build >/dev/null eq "$(api POST "/v1/sandboxes/$id/exec-sync" --json '{"script":"cat /home/user/a.txt"}' | jq -c '{exitCode,stdout}')" '{"exitCode":0,"stdout":"hi\n"}' "workers disabled: same result" SERVER_ENV= $S restart --no-build >/dev/null + trap - EXIT drop "$id" } diff --git a/.claude/skills/verify/scripts/mcp.mjs b/.claude/skills/verify/scripts/mcp.mjs index 67b337dd..d8398c81 100755 --- a/.claude/skills/verify/scripts/mcp.mjs +++ b/.claude/skills/verify/scripts/mcp.mjs @@ -12,7 +12,7 @@ function isToolFailure(part) { } } -const { STACK_BASE, STACK_TOKEN, STACK_TRANSCRIPT, STACK_SECRET } = process.env; +const { STACK_BASE, STACK_TOKEN, STACK_TRANSCRIPT, STACK_SECRET, STACK_AS } = process.env; const [tool, rawArgs = "{}"] = process.argv.slice(2); const client = new Client({ name: "verify-skill", version: "1.0.0" }); await client.connect( @@ -23,14 +23,15 @@ await client.connect( try { if (tool === "list") { const { tools } = await client.listTools(); + record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", as: STACK_AS, tool }, tools.map((t) => t.name)); for (const t of tools) console.log(`${t.name}\t${Object.keys(t.inputSchema.properties ?? {}).join(",")}`); } else { const args = JSON.parse(rawArgs); const response = await client.callTool({ name: tool, arguments: args }); - record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", tool, args, response }); const parts = response.content; - for (const part of parts) console.log(part.type === "text" ? part.text : JSON.stringify(part)); const failed = response.isError === true || parts.some(isToolFailure); + record(STACK_TRANSCRIPT, STACK_SECRET, { transport: "mcp", as: STACK_AS, tool, args, failed }, parts); + for (const part of parts) console.log(part.type === "text" ? part.text : JSON.stringify(part)); console.error(`MCP ${tool}: ${failed ? "error" : "ok"}`); const expectError = process.env.EXPECT === "error"; if (failed !== expectError) process.exitCode = 1; diff --git a/.claude/skills/verify/scripts/record.mjs b/.claude/skills/verify/scripts/record.mjs index 4c90ad29..a100bb74 100755 --- a/.claude/skills/verify/scripts/record.mjs +++ b/.claude/skills/verify/scripts/record.mjs @@ -4,12 +4,28 @@ import { fileURLToPath } from "node:url"; const JWT = /eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g; const SENSITIVE_KEY = /token|secret|password|authorization/i; +const HEADER = /^([A-Za-z0-9-]+):/; +const BODY_LIMIT = 4096; -function redact(value, secret) { - if (typeof value === "string") { - const masked = value.replaceAll(secret, "[REDACTED]"); - return masked.replace(JWT, "[REDACTED JWT]"); +function parseJson(text) { + if (!/^\s*[[{]/.test(text)) return undefined; + try { + return JSON.parse(text); + } catch { + return undefined; } +} + +function redactString(value, secret) { + const parsed = parseJson(value); + if (parsed !== undefined) return JSON.stringify(redact(parsed, secret)); + const header = HEADER.exec(value); + if (header && SENSITIVE_KEY.test(header[1])) return `${header[1]}: [REDACTED]`; + return value.replaceAll(secret, "[REDACTED]").replace(JWT, "[REDACTED JWT]"); +} + +function redact(value, secret) { + if (typeof value === "string") return redactString(value, secret); if (Array.isArray(value)) return value.map((item) => redact(item, secret)); if (value && typeof value === "object") { return Object.fromEntries( @@ -19,25 +35,25 @@ function redact(value, secret) { return value; } -export function record(file, secret, event) { - appendFileSync(file, `${JSON.stringify(redact({ at: new Date().toISOString(), ...event }, secret))}\n`); +function serialize(value) { + return typeof value === "string" ? value : JSON.stringify(value); +} + +export function record(file, secret, event, body, bytes = Buffer.byteLength(serialize(body))) { + const redacted = redact(body, secret); + const text = serialize(redacted); + const captured = + Buffer.byteLength(text) <= BODY_LIMIT + ? { body: redacted } + : { truncated: true, body: Buffer.from(text).subarray(0, BODY_LIMIT).toString("utf8") }; + const entry = { at: new Date().toISOString(), ...redact(event, secret), bytes, ...captured }; + appendFileSync(file, `${JSON.stringify(entry)}\n`); } if (process.argv[1] === fileURLToPath(import.meta.url)) { const [file, method, path, status, as, bodyFile, ...args] = process.argv.slice(2); - const bytes = readFileSync(bodyFile); - const text = bytes.toString("utf8"); - let json; - try { - json = JSON.parse(text); - } catch {} - record(file, process.env.STACK_SECRET, { - as, - method, - path, - status: Number(status), - args, - bytes: bytes.length, - ...(json !== undefined ? { json } : { text: text.slice(0, 4096) }), - }); + const raw = readFileSync(bodyFile); + const text = raw.toString("utf8"); + const event = { as, method, path, status: Number(status), args }; + record(file, process.env.STACK_SECRET, event, parseJson(text) ?? text, raw.length); } diff --git a/.claude/skills/verify/scripts/stack.sh b/.claude/skills/verify/scripts/stack.sh index 3a960620..c943dc4e 100755 --- a/.claude/skills/verify/scripts/stack.sh +++ b/.claude/skills/verify/scripts/stack.sh @@ -42,11 +42,19 @@ exec_in_stack_env() { "REDIS_DATA_URL=$REDIS_DATA_URL" "AUTH_SECRET=$AUTH_SECRET" "$@" } +override_names() { + local -a overrides=() + read -r -a overrides <<<"${SERVER_ENV:-}" + local kv + for kv in ${overrides[@]+"${overrides[@]}"}; do printf '%s ' "${kv%%=*}"; done +} + start_server() { local -a overrides=() read -r -a overrides <<<"${SERVER_ENV:-}" (exec_in_stack_env "PORT=$PORT" ${overrides[@]+"${overrides[@]}"} node dist/api/server.js >>"$DIR/server.log" 2>&1) & PID=$! + write_state local deadline=$((SECONDS + 60)) until ready; do if ! kill -0 "$PID" 2>/dev/null; then tail -40 "$DIR/server.log" >&2; die "server exited before ready"; fi @@ -56,21 +64,15 @@ start_server() { } write_state() { - cat >"$DIR/state.env" <"$DIR/state.env" + ) } register_for_teardown() { @@ -99,7 +101,8 @@ cmd_up() { DIR="$runs/$id" PROJECT="sqlfs-$id" if docker compose version >/dev/null 2>&1; then COMPOSE="docker compose"; else COMPOSE="docker-compose"; fi - mkdir -p "$DIR" + mkdir -p "$runs" + mkdir -m 700 "$DIR" set_compose register_for_teardown "${compose[@]}" up -d --wait --wait-timeout 120 >&2 @@ -109,9 +112,7 @@ cmd_up() { maybe_build "${1:-}" PORT=$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})') BASE="http://127.0.0.1:$PORT" - write_state start_server - write_state echo "stack up: $BASE (pid $PID)" echo "evidence: $DIR" } @@ -122,9 +123,8 @@ cmd_restart() { [[ "$requested" == __keep__ ]] || SERVER_ENV=$requested maybe_build "${1:-}" stop_server - echo "--- restart $(date -u +%Y-%m-%dT%H:%M:%SZ) SERVER_ENV=${SERVER_ENV:-}" >>"$DIR/server.log" + echo "--- restart $(date -u +%Y-%m-%dT%H:%M:%SZ) overrides: $(override_names)" >>"$DIR/server.log" start_server - write_state echo "restarted: $BASE (pid $PID)" } @@ -140,12 +140,12 @@ cmd_doctor() { server_alive && ok "server pid $PID runs dist/api/server.js" || fail "server pid ${PID:-?} not running (see $DIR/server.log)" if command -v lsof >/dev/null; then local owner - owner=$(lsof -nP -iTCP:"$PORT" -sTCP:LISTEN -t 2>/dev/null | head -1) + owner=$(lsof -nP -iTCP:"$PORT" -sTCP:LISTEN -t 2>/dev/null | head -1) || true [[ "$owner" == "$PID" ]] && ok "port $PORT owned by pid $PID" || fail "port $PORT listener is '${owner:-none}', expected $PID" fi ready && ok "GET /readyz ok" || fail "GET /readyz not ok" local code - code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(mint doctor)" "$BASE/v1/sandboxes") + code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(mint doctor)" "$BASE/v1/sandboxes") || true [[ "$code" == 200 ]] && ok "minted token accepted (GET /v1/sandboxes 200)" || fail "GET /v1/sandboxes returned $code" deps_current && ok "node_modules matches pnpm-lock.yaml" || fail "node_modules does not match pnpm-lock.yaml -> run: pnpm install --frozen-lockfile && $0 restart" @@ -158,7 +158,7 @@ cmd_doctor() { else ok "dist is current for src (server started $STARTED_AT, HEAD $BUILT_HEAD, now $(git rev-parse --short HEAD))" fi - [[ -n "${SERVER_ENV:-}" ]] && echo "info server env overrides: $SERVER_ENV" + [[ -n "${SERVER_ENV:-}" ]] && echo "info server env overrides (values hidden): $(override_names)" echo "base $BASE" echo "dir $DIR" return "$bad" @@ -192,7 +192,7 @@ cmd_api() { cmd_mcp() { load_state (($# >= 1)) || die "usage: $0 mcp list | $0 mcp TOOL '{json args}' (env: AS=sub, EXPECT=error)" - STACK_BASE=$BASE STACK_TOKEN=$(mint "${AS:-verify-agent}" "${TENANT:-}") STACK_TRANSCRIPT=$DIR/transcript.jsonl \ + STACK_BASE=$BASE STACK_TOKEN=$(mint "${AS:-verify-agent}" "${TENANT:-}") STACK_AS=${AS:-verify-agent} STACK_TRANSCRIPT=$DIR/transcript.jsonl \ STACK_SECRET=$AUTH_SECRET node "$here/mcp.mjs" "$@" } @@ -202,11 +202,17 @@ cmd_psql() { "${compose[@]}" exec -T postgres psql -U postgres -d sqlfs -v ON_ERROR_STOP=1 -c "$1" } +suite_names() { + echo integration + grep -l "new LocalApiRun(" scripts/verify/*.mjs | grep -v '\.test\.mjs$' | xargs -n1 basename | sed 's/\.mjs$//' +} + cmd_suite() { load_state - (($# >= 1)) || die "usage: $0 suite NAME [args] (NAME: integration or a scripts/verify/*.mjs basename)" + (($# >= 1)) || die "usage: $0 suite NAME [args] (NAME: integration or a suite entry point in scripts/verify/)" local name=$1 shift + suite_names | grep -qx "$name" || die "$name is not a suite entry point (have: $(suite_names | tr '\n' ' '))" local -a suite_env=("VERIFY_ENV_FILE=none" "VERIFY_EVIDENCE_DIR=$DIR/suites" "VERIFY_PYTHON=$repo/clients/python/.venv/bin/python") mkdir -p "$DIR/suites" if [[ "$name" == integration ]]; then @@ -215,7 +221,6 @@ cmd_suite() { (exec_in_stack_env "${suite_env[@]}" pnpm test:integration --reporter=default --reporter=json --outputFile="$report" "$@") node scripts/verify/assert-integration-report.mjs "$report" else - [[ -f "scripts/verify/$name.mjs" ]] || die "no scripts/verify/$name.mjs" (exec_in_stack_env "${suite_env[@]}" node "scripts/verify/$name.mjs" "$@") fi } diff --git a/AGENTS.md b/AGENTS.md index 0dfb1d03..e9e06b12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,17 +28,18 @@ S=.claude/skills/verify/scripts/stack.sh $S up # disposable Postgres/Redis + built API server; keeps running $S doctor # read-only health check; FAILs when dist/ is older than src/ $S restart # rebuild and restart after each source edit (data kept) -$S api POST /v1/sandboxes --json '{"name":"x"}' # authed request; EXPECT=403 AS=intruder to assert denial +$S api POST /v1/sandboxes --json '{"name":"x"}' # authed request as verify-agent; EXPECT= asserts the status +EXPECT=403 AS=intruder $S api GET /v1/sandboxes/$ID # another subject is denied a sandbox it does not own $S mcp list # MCP over Streamable HTTP; $S mcp TOOL '' $S psql 'select ...' # inspect side effects (superuser, bypasses RLS: never proof of isolation) -$S suite contracts # any scripts/verify/.mjs, or `integration [filter]`, against the stack -.claude/skills/verify/scripts/journeys.sh # every mapped journey, ~40 assertions in ~10s +$S suite contracts # a suite from the table below, or `integration [filter]`, on the stack's databases +.claude/skills/verify/scripts/journeys.sh # every mapped journey, ~40 assertions in ~10s; needs jq $S down # remove only this run's server, containers, and volumes; evidence stays ``` Evidence lands in `scripts/verify/.run/stack-/` (`transcript.jsonl`, `server.log`, `suites/`). The stack can run alongside `pnpm verify:local`, which starts its own. -`scripts/verify/` holds the live suites. Every suite except `large-files` runs against the stack through `$S suite `; the full gates run them as listed: +`scripts/verify/` holds the live suites. Every suite except `large-files` runs through `$S suite `. It reuses the stack's Postgres and Redis but starts its own API server from `dist/`, so a passing suite does not prove the `$S up` server. The full gates run them as listed: | Suite | Proves | Run by | | --- | --- | --- |