lab: dependency build step (deps installed from scratch, index-only network)

CLAUDE.md

@@ -80,6 +80,13 @@ pytest/ruff config). Each folder with commands has its own `python -m`.
   record-format spec: the environment's public interface.
 - `lab/RUNTIME.md` — design (2026-10-08): controller (laptop: ledger, referee, secrets) and worker (GPU VM: jails,
   clients, profilers) behind a job interface; trust zones and rules; why srt cannot jail GPU code on Linux.
+- `lab/build.py` — the dependency build step: a tree whose change touched the target's `deps` (pyproject: dependency
+  tables only, checked by the audit) gets a venv built by the target's `build` command in a no-GPU, `--network none`
+  container whose only way out is a CONNECT proxy to the package-index hosts; cached by the files' hash; jobs run
+  under `engine.using(build.python_for(tree))`.
+- `lab/gpucheck.py` — `python -m lab.gpucheck` on a GPU VM (as root, like the worker): what the jails need from a real
+  GPU host, PASS/FAIL each: jail GPU, workbench (GPU, root, apt/pip, metadata blocked), quiet, exclusivity, ncu,
+  hostprof, the dependency build.
 - `lab/worker.py` — the job interface, built for the measured jobs: `test`, `measure`, `equiv`, `profile` take input
   trees + JSON args, write an out dir, return JSON; the tools call `self.worker`. `LAB_WORKER=local` (default, in
   process) or `ssh` (the `lab.vm` VM): inputs by content hash, re-hashed on the worker before every job; GpuBusy,
@@ -246,6 +253,20 @@ Linux the jail is a container with the NVIDIA runtime (`lab/safety/container.py`
 check (`vm-setup.sh` prints `jail: gpu ok`) has not run yet. The agent's CLI still runs under srt, which on Ubuntu
 24.04 needs `kernel.apparmor_restrict_unprivileged_userns=0` (in `vm-setup.sh`).
 
+## First agent dry run (Nebius H200, 2026-10-10)
+`LAB_WORKER=ssh`, laptop controller, `claude-fable-5-1`, one session ($2.53 as the proxy priced it, 24 turns, 465 s;
+~1 h 20 min of GPU for the whole run). Validated on the real GPU: the GPU in the network-less jail, metadata blocked,
+base and candidate served jailed through the port bridge, `profile`, `trace` (after mounting nsys's install root),
+`test` (after caching the tests' gpt2 tokenizer), the remote workbench over `ssh -R`, the proxy's pricing. The agent
+fused RMSNorm/RoPE in Triton and added a batch-1 decode graph: seen-split short-tier `single_stream` tpot p99 21.4 ->
+13.1 ms; not submitted, correctness not run (equiv needs `data/equiv/ref`, absent in a worktree). A base dev-tier equiv
+on this engine took > 46 min for 774 of 1,597 items at concurrency 16 (vLLM: ~10 min): the tool descriptions' "10 min
+at tier dev" is wrong for this engine. Workbench processes cannot change clocks or power limits on that platform.
+`python -m lab.gpucheck` then passed every item on Nebius H200s (2026-10-13, three runs, fixes between them):
+jail GPU, quiet, exclusivity, hostprof; workbench (after blocking only 169.254.169.254: the /16 took DNS with it);
+the dependency build (after allowing pypi.nvidia.com); ncu (after linking the real /opt/nvidia install). The jail's
+user can read GPU performance counters on that image.
+
 ## Next
 GPU (needs spend approval): resolve the BF16 puzzle (plain BF16 on the dev tier at concurrency 32 and 128 on one VM,
 ~20 min, ~$1.50); first real run of `python -m regimes run all` against the engine and against vLLM.

lab/RUNTIME.md

@@ -4,8 +4,8 @@ Design, 2026-10-08. Decided with the user: the lab splits into a controller and
 measurement VM (cost); NVIDIA only for now; the jail on Linux is a container with the NVIDIA runtime (option A
 below). Built: the container jail (`lab/safety/container.py`), GPU exclusivity during measurements, and the
 measured jobs behind the worker interface (`lab/worker.py`), the API proxy (`lab/apiproxy.py`), the workbench
-container with its pause during measurements, and the agent on a remote worker. Not built: the dependency build
-step. Open decisions are marked **Decide**.
+container with its pause during measurements, the agent on a remote worker, and the dependency build step
+(`lab/build.py`). Open decisions are marked **Decide**.
 
 ## The line (the user, 2026-10-09)
 
@@ -173,7 +173,8 @@ not yet over SSH to a VM. `agent`, `exec`, `snapshot`, `restore` and `facts` com
   interpreter, weights, CUDA and the CLI binary read-only, so an install in the workbench never reaches a
   measurement. Container root is uid 0 on the host kernel (no user-namespace remap): an escape is host root,
   accepted under decision 5; `userns-remap` is the hardening step. `vm-setup.sh` drops container traffic to
-  169.254.0.0/16 (the cloud metadata service) and checks it.
+  169.254.169.254 (the cloud metadata service; only that address, a cloud's DNS can sit elsewhere in 169.254/16) and
+  checks both that and the workbench's internet.
 - **Pause** (`container.quiet`): around every `measure`, `equiv` and `profile` job, processes in the workbench
   holding the GPU are killed, the container is paused, a second sweep catches any that started before the pause
   landed, and it waits for their memory to be released; thawed after. The killed processes are in the record and
@@ -206,10 +207,31 @@ on the VM:
   on the VM, mounted into the workbench, where a socat serves it on the container's own 127.0.0.1:4000. It listens on
   no interface of the VM.
 - The write guard checks the VM's workspace path; the workbench is removed over SSH when the session ends.
+- The workbench is frozen on the VM (`python -m lab.worker pause`) for the whole of every lab tool call, so the root
+  rsync of a pull or push never races a process in it swapping a file or a directory for a link (2026-10-13 review:
+  a swapped directory could have carried the held-out split out, or let root write anywhere on the VM).
 - Tested without a VM: rsync mirroring (content edits, deletions, links), the VM-side `workbench` command, and the
   argument quoting through bash and ssh (an `ssh` stand-in that prints the remote command). Not yet tested against a
   real VM.
 
+## The dependency build step, as built (2026-10-10)
+
+- The target names its dependency files (`deps`) and the command that installs them (`build`, run with
+  `$UV_PROJECT_ENVIRONMENT` set). For inference-server that is vm-setup.sh's recipe: `uv sync --locked`, then the
+  CUDA torch wheel at the locked version.
+- The agent may change those files; a `pyproject.toml` only in its dependency tables (`[project] dependencies`,
+  `optional-dependencies`, `[dependency-groups]`, `[tool.uv] sources` and `index`): any other change, its test and
+  lint config included, is a violation, and deleting a dependency file is one too.
+- Each job asks `build.python_for(tree)`: a tree whose change left the dependency files as in its base (HEAD~1 of
+  the two-commit pristine tree) runs on the engine's venv; a changed one gets a venv built for it, cached by the
+  files' hash. The build runs in a measurement-room container without the GPU and with `--network none`; its only way
+  out is a host-side CONNECT proxy, over a unix socket, to pypi.org, files.pythonhosted.org and the https hosts its
+  files and command name. A failed build is a failed `test`/`profile` or an `error` record carrying the build's
+  output. The tool descriptions state it as a fact: only the change reaches the engine, installed from scratch, and
+  the engine then runs offline.
+- Tested in Docker-in-Docker: the build room gets 200 from PyPI through the proxy, 403 for another host, and no
+  direct route out. The real inference-server build (torch, CUDA) is untested until the GPU run.
+
 ## What moves where
 
 - Controller: `session`, `agent` (SDK side), the tool front ends in `tools`/`evaltools`/`proftools` (policy,

lab/agent.py

@@ -121,6 +121,7 @@ class ClaudeAgentSDK:
     srt); its model calls go through the lab's API proxy, so no credential is ever inside; lab tools run out here."""
 
     name = "claude"
+    remote = True                       # can work on a Remote worker's host (its CLI over ssh)
 
     @staticmethod
     def containerized() -> bool:
@@ -133,7 +134,10 @@ def _wrapper(self, spec: AgentSpec, cli: Path, base_url: str) -> Path:
             w = spec.remote.workbench(spec.workbench, spec.scratch.name, spec.remote_workspace,
                                       claude_agent_sdk.__version__)
             *ssh, target = spec.remote.prefix
+            # The SDK may launch this more than once, and that sshd need not unlink a socket a launch left behind;
+            # -n: that call must not read the SDK's stdin.
             line = (f'args=$(printf " %q" "$@")\ncmd={shlex.quote(shlex.join(w["argv"]))}\n'
+                    f'{shlex.join(ssh)} -n -T {target} {shlex.quote("rm -f " + shlex.quote(w["sock"]))} || exit 1\n'
                     f'exec {shlex.join(ssh)} -T -o ExitOnForwardFailure=yes -o StreamLocalBindUnlink=yes '
                     f'-R {w["sock"]}:127.0.0.1:{urlsplit(base_url).port} {target} "$cmd$args"')
         elif self.containerized():
@@ -148,7 +152,10 @@ def _wrapper(self, spec: AgentSpec, cli: Path, base_url: str) -> Path:
             cli_tmp = Path(f"{tmp_root}/claude-{os.getuid()}")    # the CLI's own per-project scratch; Bash fails without it
             cli_tmp.mkdir(parents=True, exist_ok=True)
             config = jail.settings([home.resolve(), tmp.resolve(), ws, cli_tmp.resolve()], engine.venv(),
-                                   [base_url.removeprefix("http://")], readonly=_weights(), python=engine.python())
+                                   [base_url.removeprefix("http://")], readonly=_weights(), python=engine.python(),
+                                   local_binding=True)     # srt on macOS reaches 127.0.0.1 only with it (and then
+                                                           # every local port: dev only; measurements that count run
+                                                           # on Linux, where each jail has its own network)
             argv = jail.wrap(config, spec.scratch / "srt.json", [str(cli)])
             keep = " ".join(f'"{k}=${k}"' for k in PASS_ENV)
             sets = " ".join(f"{k}={shlex.quote(v)}" for k, v in cli_env(home, tmp, ws, base_url).items())
@@ -162,10 +169,20 @@ async def _run(self, spec: AgentSpec, base_url: str) -> AgentReply:
         from claude_agent_sdk import (ClaudeAgentOptions, HookMatcher, ResultMessage,
                                       create_sdk_mcp_server, query, tool)
 
+        # The workbench is frozen for every lab tool call, wherever it runs: nothing in it can swap a file or a dir for a
+        # link while the tool snapshots, restores, copies into the workspace, or (Remote) rsyncs it as root on the VM.
+        if spec.remote is not None:
+            frozen = lambda: spec.remote.paused(spec.workbench)                                  # noqa: E731
+        else:
+            frozen = lambda: container.paused(spec.workbench if self.containerized() else None)  # noqa: E731
+
         def adapt(t: ToolSpec):
             async def call(args: dict[str, Any]) -> dict[str, Any]:
                 try:
-                    text = await asyncio.to_thread(t.fn, args)   # a long test run must not block the transport
+                    def run():
+                        with frozen():
+                            return t.fn(args)
+                    text = await asyncio.to_thread(run)          # a long test run must not block the transport
                 except Exception as e:          # the agent sees the refusal, the harness keeps going
                     text = f"{t.name} refused: {e}"
                 return {"content": [{"type": "text", "text": text}]}

lab/apiproxy.py

@@ -54,10 +54,11 @@ def credential() -> dict[str, str]:
 
 
 class _Usage:
-    """Usage read off a response as it streams by: JSON, or SSE message_start + message_delta."""
+    """Usage read off a response as it streams by: JSON, or SSE message_start + message_delta. `complete` once the
+    final output count is known."""
 
     def __init__(self, sse: bool):
-        self.sse, self.buf, self.model, self.usage = sse, b"", None, {}
+        self.sse, self.buf, self.model, self.usage, self.complete = sse, b"", None, {}, False
 
     def feed(self, chunk: bytes) -> None:
         self.buf += chunk
@@ -78,6 +79,7 @@ def _event(self, data: bytes) -> None:
             self.usage.update(e["message"].get("usage") or {})
         elif e.get("type") == "message_delta":
             self.usage.update(e.get("usage") or {})
+            self.complete = True
 
     def done(self) -> None:
         if self.sse:
@@ -88,14 +90,14 @@ def done(self) -> None:
         except ValueError:
             return
         if isinstance(body, dict) and isinstance(body.get("usage"), dict):
-            self.model, self.usage = body.get("model"), body["usage"]
+            self.model, self.usage, self.complete = body.get("model"), body["usage"], True
 
 
 class Proxy:
     """`with Proxy(cap_usd, host) as p: ... p.url`; `p.spent_usd` and `p.usage` are what the session cost."""
 
     def __init__(self, cap_usd: float, host: str = "127.0.0.1", upstream: str = UPSTREAM):
-        self.cap_usd, self.spent_usd, self.requests = cap_usd, 0.0, 0
+        self.cap_usd, self.spent_usd, self.requests, self.reserved_usd = cap_usd, 0.0, 0, 0.0
         self.usage = {"input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0,
                       "cache_creation_input_tokens": 0}
         self.auth = credential()
@@ -142,27 +144,35 @@ def _handle(self, h, body: bytes | None) -> None:
         path = urlsplit(h.path).path
         if path not in ALLOWED and not (path.startswith("/v1/models/") and path.count("/") == 3):
             return self._refuse(h, 403, f"{path} is not forwarded")
-        model = None
+        model, max_tokens = None, 0
         if body:
             try:
                 req = json.loads(body)
             except ValueError:
                 return self._refuse(h, 400, "the request body is not JSON")
-            model = req.get("model")
+            model, max_tokens = req.get("model"), int(req.get("max_tokens") or 0)
             if path == "/v1/messages" and (not isinstance(model, str) or price(model) is None):
                 return self._refuse(h, 403, f"no price for model {model!r}; priced: {sorted(PRICES)}")
             if req.get("speed") == "fast":
                 return self._refuse(h, 403, "fast mode is not priced here")
+        metered = path == "/v1/messages"
+        if metered and (not body or not isinstance(model, str) or price(model) is None):
+            return self._refuse(h, 403, "a /v1/messages request needs a JSON body naming a priced model")
+        # Its worst case is reserved before it goes (input: at most a token per byte of body; output: all of max_tokens),
+        # so concurrent requests cannot overshoot the cap.
+        hold = (len(body) * price(model)[0] + max_tokens * price(model)[1]) / 1e6 if metered else 0.0
         with self.lock:
-            if path == "/v1/messages" and self.spent_usd >= self.cap_usd:
-                return self._refuse(h, 403, f"this session's ${self.cap_usd:.2f} is spent")
+            if metered and self.spent_usd + self.reserved_usd + hold > self.cap_usd:
+                return self._refuse(h, 403, f"this session's ${self.cap_usd:.2f} is spent (or would be by this request)")
+            self.reserved_usd += hold
         headers = {k: v for k, v in h.headers.items() if k.lower() not in HOP | {"x-api-key", "authorization"}}
         headers.update(self.auth, host=self.upstream.netloc, **{"accept-encoding": "identity"})
         if "authorization" in self.auth:
             betas = [b for b in headers.pop("anthropic-beta", "").split(",") if b.strip()]
             headers["anthropic-beta"] = ",".join(dict.fromkeys([*betas, OAUTH_BETA]))
         conn = (http.client.HTTPSConnection if self.upstream.scheme == "https" else http.client.HTTPConnection)(
             self.upstream.netloc, timeout=900)
+        usage = None
         try:
             conn.request(h.command, h.path, body=body, headers=headers)
             resp = conn.getresponse()
@@ -178,13 +188,22 @@ def _handle(self, h, body: bytes | None) -> None:
                 h.wfile.write(b"%x\r\n%s\r\n" % (len(chunk), chunk))
                 h.wfile.flush()
             h.wfile.write(b"0\r\n\r\n")
-            usage.done()
-        finally:
+        finally:                                    # charged however it ended: a dropped client still spent
             conn.close()
-        if path == "/v1/messages" and usage.usage:
-            spent = cost_usd(usage.model if usage.model and price(usage.model) else model, usage.usage)
-            with self.lock:
-                self.spent_usd += spent
+            if usage is not None:
+                usage.done()
+            self._charge(metered and usage is not None, usage, model, max_tokens, hold)
+
+    def _charge(self, metered: bool, usage: "_Usage | None", model: str | None, max_tokens: int, hold: float) -> None:
+        """An incomplete response is charged its input as far as seen and its whole `max_tokens` of output."""
+        got = dict(usage.usage) if usage is not None else {}
+        if metered and got and not usage.complete:
+            got["output_tokens"] = max(got.get("output_tokens") or 0, max_tokens)
+        spent = cost_usd(usage.model if usage.model and price(usage.model) else model, got) if metered and got else 0.0
+        with self.lock:
+            self.reserved_usd -= hold
+            self.spent_usd += spent
+            if metered and got:
                 self.requests += 1
                 for k in self.usage:
-                    self.usage[k] += usage.usage.get(k) or 0
+                    self.usage[k] += got.get(k) or 0