fix: bound ingress queues and reduce replica startup contention

.changeset/balanced-aca-admission.md

@@ -0,0 +1,5 @@
+---
+"sql-fs-api": patch
+---
+
+Update the ACA deployment example with early HTTP concurrency scaling, bounded five-second queues, faster readiness probes, a drain grace period and an immutable image reference. Keep blob caching off in the example so control Redis does not also carry blob traffic.

.changeset/queued-input-cancellation.md

@@ -0,0 +1,7 @@
+---
+"sql-fs-api": patch
+---
+
+Read accepted queued HTTP and MCP request bodies within the existing byte budgets so client disconnects can remove waiting requests before route execution. Preserve streaming, response and retained-work ownership, and unread immediate queue-full refusal. Add independent global and tenant ingress queue limits, including zero waiting, while keeping the current queue sizes and 250 ms wait by default.
+
+Expose ingress capacity, outcome counters and admitted queue wait histograms through metrics.

.changeset/skip-applied-startup-migrations.md

@@ -0,0 +1,5 @@
+---
+"sql-fs-api": patch
+---
+
+Record applied Postgres migrations and their SHA-256 checksums in a transactionally committed ledger per tenant database. Replica startup skips unchanged migrations instead of repeating table DDL while other replicas serve requests. Reject changes to recorded migration files and roll back pending DDL and history together on failure.

.changeset/tidy-ingress-outcomes.md

@@ -0,0 +1,5 @@
+---
+"sql-fs-api": patch
+---
+
+Preserve completed route responses when request input fails after dispatch, and count queued-hook failures as cancelled admissions. Keep overload retry and Redis scaling guidance aligned with operation outcomes and configured cache roles.

.changeset/tidy-runtime-dependencies.md

@@ -0,0 +1,5 @@
+---
+"sql-fs-api": patch
+---
+
+Remove development dependencies from the runtime Docker image after building the API.

.claude/skills/verify/SKILL.md

@@ -22,7 +22,7 @@ Run every row your diff touches. `pnpm check` is always required; the full verif
 | `src/sql-fs/` (fs, transactions, cache, RLS, symlinks) | targeted `pnpm exec vitest run <file>`, `$S suite integration <file>`, [files](features/files.md) / [exec](features/exec.md) drive | `pnpm check` + `pnpm verify:local` |
 | `src/api/routes/`, `auth.ts`, `validation.ts`, `openapi-spec.ts` | the matching feature drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` |
 | `src/api/mcp/` | [mcp](features/mcp.md) drive + `$S suite contracts` | `pnpm check` + `pnpm verify:local` |
-| `src/api/worker-owner/`, `ingress-admission.ts`, `session-manager.ts`, `src/execution/`, `script-scope.ts`, budgets, cancellation, deadlines, body limits | `$S suite worker-failures`, `sse-finalization`, `input-budgets`, `output-cap` | `pnpm check` + `pnpm verify:architecture` |
+| `src/api/worker-owner/`, `ingress-admission.ts`, `session-manager.ts`, `src/execution/`, `script-scope.ts`, budgets, cancellation, deadlines, body limits | `$S suite worker-failures`, `sse-finalization`, `input-budgets`, `ingress-queue`, `output-cap` | `pnpm check` + `pnpm verify:architecture` |
 | `src/api/server.ts` startup/shutdown, `src/api/mcp/server.ts` listener lifecycle, ingress waits | `$S suite shutdown`, `$S suite contracts` | `pnpm check` + `pnpm verify:local` |
 | Tenancy, `TENANT_DATABASES`, owner isolation | `$S suite tenants`, [sandboxes](features/sandboxes.md) isolation steps | `pnpm check` + `pnpm verify:architecture` |
 | `src/redis/`, `redis-blob-cache.ts` | `$S suite integration src/redis` | `pnpm check` + `pnpm verify:local` |

.claude/skills/verify/features/README.md

@@ -46,6 +46,7 @@ These have no hand-driven journey yet. Run the suite through `$S suite <name>`,
 - Worker crash, lease loss, and unknown-outcome errors: `worker-failures`
 - SSE stream finalization: `sse-finalization`
 - Request and input budgets: `input-budgets`
+- Queued upload cancellation and durable absence: `ingress-queue`
 - Output caps: `output-cap`
 - Graceful shutdown with open MCP streams: `shutdown`
 - Full HTTP/MCP contract against `/openapi.json`: `contracts`

.claude/skills/verify/features/files.md

@@ -10,6 +10,7 @@ Clients read and write sandbox files directly, without bash. They can write raw
 - `files-mkdir` creates directories recursively (`204`).
 - `files-tree` lists entries under a prefix to a given depth.
 - `files-delete` removes a file (`204`), after which reads return `404 ENOENT`.
+- `files-queued-cancel` abandons a queued 1 MiB upload and verifies that it never becomes a durable file.
 
 ## How to get to it (user POV)
 
@@ -28,6 +29,7 @@ Preconditions:
 - **Bulk write, seen by bash.** Run `EXPECT=204 $S api POST /v1/sandboxes/$ID/writeFiles --json '{"files":{"/home/user/v/one.txt":"one\n","/home/user/v/two.txt":"two\n"}}'`, then `$S api POST /v1/sandboxes/$ID/exec-sync --json '{"script":"cat /home/user/v/one.txt /home/user/v/two.txt","readOnly":true}' | jq -r .stdout`. It prints `one` and `two`.
 - **Tree.** Run `$S api GET "/v1/sandboxes/$ID/tree?prefix=/home/user/v&depth=3" | jq -c '[.[].path]'`. The output lists `note.txt`, `one.txt`, and `two.txt` under `/home/user/v/`.
 - **Delete.** Run `EXPECT=204 $S api DELETE /v1/sandboxes/$ID/files/home/user/v/note.txt`, then `EXPECT=404 $S api GET /v1/sandboxes/$ID/files/home/user/v/note.txt`. The body is `{"error":"not_found","code":"ENOENT"}`.
+- **Queued cancellation.** Run `$S suite ingress-queue`. It holds ingress with an incomplete body, sends and abandons a queued upload, then checks queue removal before admission and reads the file after capacity returns. All five repetitions must leave no file and no retained bytes. A live 1 MiB upload control must preserve its exact hash.
 
 ## Gotchas
 

.claude/skills/verify/scripts/journeys.sh

@@ -82,6 +82,7 @@ journey_files() {
 	EXPECT=204 api DELETE "/v1/sandboxes/$id/files/home/user/v/note.txt" >/dev/null
 	eq "$(EXPECT=404 api GET "/v1/sandboxes/$id/files/home/user/v/note.txt" | jq -r .code)" ENOENT "deleted file -> ENOENT"
 	drop "$id"
+	$S suite ingress-queue
 }
 
 journey_ingest() {

AGENTS.md

@@ -10,12 +10,12 @@ Use Node 22 from `.node-version`, pnpm 10.30.3 from `package.json`, Python 3.12,
 pnpm setup:dev          # install all three packages from their lockfiles; install git hooks
 pnpm check          # types, formatting, unit tests, builds, and both SDKs; no services or .env needed
 pnpm verify:local   # isolated Postgres/Redis, HTTP/MCP contracts, integration tests, and both live SDKs
-pnpm verify:architecture # also worker failure, stream finalization, input budgets, and multi-tenant proofs
+pnpm verify:architecture # also worker failure, stream finalization, input budgets, ingress queue, and multi-tenant proofs
 ```
 
 `pnpm verify:local` uses random loopback ports and a unique Compose project, applies migrations through server startup, and removes only its own containers and volumes. It ignores `.env` and inherited deployment settings. It writes logs and JSON proof to `scripts/verify/.run/`. A missing Docker daemon or skipped integration test fails the command.
 
-`pnpm verify:architecture` runs the standard verification plus `worker-failures.mjs`, `sse-finalization.mjs`, `input-budgets.mjs`, and `tenants.mjs` in the same disposable stack. Run it for worker ownership, script completion, command protocol, or isolation changes. CI runs this expanded profile on pushes to `main`; PR checks keep the faster static, unit, build, and SDK gates. Resource and large-file profiling remain separate and require a suitable cgroup v2 host.
+`pnpm verify:architecture` runs the standard verification plus `worker-failures.mjs`, `sse-finalization.mjs`, `input-budgets.mjs`, `ingress-queue.mjs`, and `tenants.mjs` in the same disposable stack. Run it for worker ownership, script completion, command protocol, ingress admission, or isolation changes. CI runs this expanded profile on pushes to `main`; PR checks keep the faster static, unit, build, and SDK gates. Resource and large-file profiling remain separate and require a suitable cgroup v2 host.
 
 Use `pnpm check:typescript` or `pnpm check:python` for SDK-only changes. Run a targeted core test with `pnpm exec vitest run src/sql-fs/tests/unit/sql-fs.cache.test.ts`. A nonexistent test selection must fail. `pnpm test:integration` is the lower-level command for an already migrated disposable database with `DATABASE_URL` and `REDIS_URL`; it can skip without those variables. Prefer `pnpm verify:local` for completion evidence.
 
@@ -47,7 +47,7 @@ Evidence lands in `scripts/verify/.run/stack-<id>/` (`transcript.jsonl`, `server
 | `output-cap` | worker output limits in both worker modes | `verify:local` |
 | `shutdown` | graceful shutdown ends MCP listener streams and refuses new connections | `verify:local` |
 | `sdk-clients` (+ `sdk-python.py`) | live TypeScript and Python SDKs | `verify:local` |
-| `worker-failures`, `sse-finalization`, `input-budgets`, `tenants` | worker loss, stream finalization, input budgets, tenant isolation | `verify:architecture` |
+| `worker-failures`, `sse-finalization`, `input-budgets`, `ingress-queue`, `tenants` | worker loss, stream finalization, input budgets, ingress queue admission, tenant isolation | `verify:architecture` |
 | `ingest-admission`, `large-files` | ingest budgets, large-file profiling | manual; large files need cgroup v2 |
 
 `local-api.mjs` is the shared harness: owned servers, tokens, schema validation, `proof.json`. `assert-integration-report.mjs` is the shared check that an integration run executed tests with none failed or skipped. `*-faults.mjs` files are fault injectors loaded by the suites, not entry points. When you add an endpoint or client-visible behavior, add or extend a suite here, add its journey to `features/` and `journeys.sh`, and update `openapi-spec.ts`.

Dockerfile

@@ -1,6 +1,5 @@
 # syntax=docker/dockerfile:1.7
 
-# ------- Builder -------
 FROM node:22-bookworm-slim AS builder
 WORKDIR /app
 RUN corepack enable
@@ -16,14 +15,16 @@ RUN --mount=type=secret,id=proxy_ca \
 COPY . .
 RUN pnpm build
 
-# ------- Runtime -------
+FROM builder AS production-dependencies
+RUN pnpm --config.ignore-scripts=true prune --prod
+
 FROM node:22-slim AS runtime
 ENV NODE_ENV=production
 ENV PORT=8080
 WORKDIR /app
 RUN groupadd -r app && useradd -r -g app app
 COPY --from=builder --chown=app:app /app/dist ./dist
-COPY --from=builder --chown=app:app /app/node_modules ./node_modules
+COPY --from=production-dependencies --chown=app:app /app/node_modules ./node_modules
 COPY --from=builder --chown=app:app /app/package.json ./package.json
 USER app
 EXPOSE 8080

deploy/azure/aca.yaml

@@ -22,9 +22,10 @@ properties:
       - name: ghcr-token
         value: "<GITHUB_PAT>"
   template:
+    terminationGracePeriodSeconds: 90
     containers:
       - name: api
-        image: ghcr.io/<GITHUB_USERNAME>/sql-fs-api:latest
+        image: ghcr.io/<GITHUB_USERNAME>/sql-fs@sha256:<IMAGE_DIGEST>
         resources:
           cpu: 1.0
           memory: 2Gi
@@ -39,28 +40,43 @@ properties:
             secretRef: redis-url
           - name: PORT
             value: "8080"
+          - name: REDIS_BLOB_CACHE_ENABLED
+            value: "false"
+          - name: SQL_FS_MAX_INGRESS_QUEUED_REQUESTS
+            value: "64"
+          - name: SQL_FS_MAX_INGRESS_TENANT_QUEUED_REQUESTS
+            value: "16"
+          - name: SQL_FS_MAX_INGRESS_WAIT_MS
+            value: "5000"
+          - name: SQL_FS_WORKER_QUEUE_WAIT_MS
+            value: "5000"
+          - name: SQL_FS_WORKER_EXECUTION_MS
+            value: "45000"
         probes:
           - type: Liveness
             httpGet:
               path: /healthz
               port: 8080
             periodSeconds: 15
+            timeoutSeconds: 3
           - type: Readiness
             httpGet:
               path: /readyz
               port: 8080
-            periodSeconds: 10
+            periodSeconds: 2
+            timeoutSeconds: 3
           - type: Startup
             httpGet:
               path: /readyz
               port: 8080
             failureThreshold: 30
             periodSeconds: 2
     scale:
-      minReplicas: 1
+      minReplicas: 2
       maxReplicas: 10
+      # HTTP scaling uses ACA's fleet request signal, evaluated every 15 seconds.
       rules:
         - name: http-rule
           http:
             metadata:
-              concurrentRequests: "50"
+              concurrentRequests: "1"