lab: worker job interface (local and SSH transports)

CLAUDE.md

@@ -63,8 +63,12 @@ pytest/ruff config). Each folder with commands has its own `python -m`.
   measures each regime's run-to-run band from repeated reference runs into `knowledge/noise/<regime>.json` (which
   `submit` reads). Exit 0 only if both hold. Needs a GPU for a real target; tested on fakes. `lab/LEDGER.md` is the
   record-format spec: the environment's public interface.
-- `lab/RUNTIME.md` — design, not built (2026-10-08): controller (laptop: ledger, referee, secrets) and worker (GPU
-  VM: jails, clients, profilers) behind a job interface; trust zones and rules; why srt cannot jail GPU code on Linux.
+- `lab/RUNTIME.md` — design (2026-10-08): controller (laptop: ledger, referee, secrets) and worker (GPU VM: jails,
+  clients, profilers) behind a job interface; trust zones and rules; why srt cannot jail GPU code on Linux.
+- `lab/worker.py` — the job interface, built for the measured jobs: `test`, `measure`, `equiv`, `profile` take input
+  trees + JSON args, write an out dir, return JSON; the tools call `self.worker`. `LAB_WORKER=local` (default, in
+  process) or `ssh` (the `lab.vm` VM): inputs by content hash, re-hashed on the worker before every job; GpuBusy,
+  Contaminated, NotReady, ValueError cross intact. Remote is tested through a local shell, not yet over SSH.
 - `lab/TRACE.md` + `lab/trace_contract.py` — the request-trace contract: per-request engine rows (trace_id = the
   client's `X-Trace-Id`, `arrival_ts` epoch s, `*_s` monotonic durations, TPOT = (E2E − TTFT)/(n − 1)) in a dir of
   `*.sqlite` (`requests` + `meta`) or `*.jsonl` + `<stem>.meta.json`, with schema version and drop counts;

lab/README.md

@@ -167,6 +167,10 @@ bridge; image from `python -m lab.safety.container build`, which `vm-setup.sh` r
 sandbox-runtime (`npm install -g @anthropic-ai/sandbox-runtime`); `LAB_JAIL` picks one. `LAB_NO_JAIL=1` waives
 the jail for tests and a box you trust. The agent's own CLI still runs under srt (`lab/RUNTIME.md`).
 
+The GPU work behind `test`, `profile`, `trace`, `kernel`, `hostprof`, `bench`, `equiv` and `submit` runs as worker
+jobs (`lab/worker.py`). `LAB_WORKER=local` (default) runs them in process; `LAB_WORKER=ssh` runs them on the VM
+`lab.vm` names (start it and run `python -m lab.vm setup` first), shipping each tree by content hash.
+
 ## Canaries
 
     python -m lab.canary slow_decode,kv_leak --port 8000

lab/RUNTIME.md

@@ -2,8 +2,41 @@
 
 Design, 2026-10-08. Decided with the user: the lab splits into a controller and a worker; no separate
 measurement VM (cost); NVIDIA only for now; the jail on Linux is a container with the NVIDIA runtime (option A
-below). Built: the container jail (`lab/safety/container.py`). Not built: the split. Open decisions are marked
-**Decide**.
+below). Built: the container jail (`lab/safety/container.py`), GPU exclusivity during measurements, and the
+measured jobs behind the worker interface (`lab/worker.py`). Not built: the agent on the worker (its container, the
+API proxy) and the agent-side jobs. Open decisions are marked **Decide**.
+
+## The line (the user, 2026-10-09)
+
+Don't restrict the agent's search space; restrict only what would let it cheat or cost money. The agent never
+touches (1) secrets and money: API keys, cloud credentials, the HF token; (2) the answer key: held-out workloads,
+correctness reference outputs; (3) the scoreboard: ledger, measurement clients, grader, budget; (4) the GPU
+during its own measurement. Everything else it can have. Decided with it:
+
+- **Internet:** the agent's workbench has it, open: the same capabilities Claude Code has (a shell with network,
+  web search and fetch, any package install), not gated behind tools. Hence nothing secret on the VM (rule 1).
+  Agent-facing tools are the user's to design; the lab builds none.
+- **Installing libraries:** the agent can pip- and apt-install. Its room is a looser *workbench* container: GPU,
+  network, root inside the container (never on the VM host), writable system dirs. The tight container (no
+  network, no capabilities, read-only root) stays for the measurement rooms only (`test`, `profile`, served
+  engines).
+- **Installed libraries reach measurement only through the change.** A measurement room is rebuilt from the
+  agent's code change alone, so a dependency change has to be part of it: the engine's dependency files (e.g.
+  `pyproject.toml`, `uv.lock`) join the target's write surface and are installed from scratch for the
+  measurement. Patching an installed library in the workbench changes no measured result.
+  The measurement room stays offline, a fairness rule rather than a safety one: an engine with network during
+  a measurement could forward requests to a remote GPU or API. So the install is its own build step before the
+  measurement (decided 2026-10-09): package indexes only, no GPU, no secrets, cached by lockfile hash. Anything the
+  engine needs at runtime (draft models, prebuilt kernels, files a library downloads on first use) must be in the
+  change, in the dependency files or saved in the workspace; the tool descriptions state that as a fact once the
+  build step exists (not before: today measurements still use the engine venv on the VM).
+- **Fresh VM per run:** yes (rule 6).
+- **No separate measurement VM** for now: container escape is not a realistic threat at this stage, and it
+  would double GPU cost. Revisit for a public leaderboard.
+- **Refuse rather than kill** (#20) until the workbench can be frozen during a measurement (rule 4).
+
+Not decided here: held-out workloads are sampled from public datasets, so with internet the agent could
+download them; the workload generator's owner decides (rephrased or non-public held-out prompts).
 
 ## Why
 
@@ -27,7 +60,8 @@ cannot signal others), and everything in a jail dies when the jail's command exi
 |---|---|---|---|
 | Controller | laptop (later an always-on box) | session loop, ledger, referee, holdout guard, budget, snapshot store, every secret | no |
 | Worker | GPU VM, host side, outside every jail | job runner, measurement clients, profilers, GPU state | only through the engine's HTTP port |
-| Jails | GPU VM, inside containers | agent's shell and workspace; each served engine | yes, it is the agent's |
+| Workbench | GPU VM, the agent's container | agent's shell and workspace; GPU, internet, root inside | yes, it is the agent's |
+| Measurement rooms | GPU VM, one tight container per job | a tree under `test`/`profile`, a served engine | its code is the agent's; nothing else reaches in |
 
 Rules:
 
@@ -43,10 +77,10 @@ Rules:
    states what was killed (a fact).
 5. **Measurement code is out of reach.** The clients run on the host, outside every container, from a tree the
    jails cannot read; their CPU cores are reserved (cpuset) so the engine cannot starve them.
-6. **One VM per run, destroyed after** (**Decide**; recommended). Nothing from one run can touch the next one's numbers. Setup time is the
+6. **One VM per run, created at run start and deleted at the end** (decided 2026-10-09). Nothing from one run can touch the next one's numbers. Setup time is the
    cost; a prebuilt image with the venv plus a reattached read-only weights disk cuts it.
-7. **Network.** VM egress: the controller tunnel only. Agent jail: the API proxy only. Engine jail: nothing; its
-   port is reachable from the host alone.
+7. **Network.** Workbench: the internet (decided 2026-10-09), so the VM holds nothing secret. Measurement rooms:
+   nothing; a served engine's port is reachable from the host alone.
 
 Left open by not having a separate measurement VM: the clients and the agent's code share a kernel, so a
 kernel or container escape defeats rules 4 and 5. A measurement VM in the same private network closes it later.
@@ -119,6 +153,15 @@ JSON, raw files with hashes, and hardware facts (device, driver, clocks).
 
 Transport now: SSH from the controller, one worker. Transport later: a queue. The job list does not change.
 
+Built (2026-10-09, `lab/worker.py`): `test`, `measure` (bench, submit and the base share it), `equiv` and `profile`
+(all four instruments). The tools keep the policy, the ledger and the budget and call `self.worker`; `LAB_WORKER`
+picks `local` (default: in process, as before) or `ssh` (the VM `lab.vm` names, started and set up beforehand).
+Over SSH each input tree goes by content hash, at most once per worker (`has`, `put`), and the worker re-hashes a
+fresh copy right before every job, refusing a mismatch. The job's out dir comes back as a tar; GpuBusy,
+Contaminated, NotReady and ValueError are raised again on the controller. equiv ships the snapshot's earlier
+answers with the job, so calling it again never re-rolls the gate. Tested through a local shell with the same CLI;
+not yet over SSH to a VM. `agent`, `exec`, `snapshot`, `restore` and `facts` come with the agent container.
+
 ## What moves where
 
 - Controller: `session`, `agent` (SDK side), the tool front ends in `tools`/`evaltools`/`proftools` (policy,
@@ -148,7 +191,7 @@ database, still append-only.
 ## Order
 
 1. ~~Decide the jail~~ (A) and build it. Done, minus the GPU check.
-2. Without a GPU: the job interface with a local transport, the snapshot round trip and rule 3, the API proxy,
-   and the container jail's network, filesystem and freeze behaviour on any Linux box (Docker locally).
+2. Without a GPU: ~~the job interface with a local transport, the snapshot round trip and rule 3~~ (done for the
+   measured jobs); next the agent container with its freeze, and the API proxy.
 3. With a GPU (spend approval): `--gpus` and the profilers inside the container, then a first `lab.session`
    dry run with a tiny budget.