Skip to content

lab: agent workbench container, pause during measurements, API-key proxy - #24

Merged
AdvayMonga merged 10 commits into
mainfrom
lab/workbench
Oct 14, 2026
Merged

lab: agent workbench container, pause during measurements, API-key proxy#24
AdvayMonga merged 10 commits into
mainfrom
lab/workbench

Conversation

@AdvayMonga

@AdvayMonga AdvayMonga commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Stacked on #21 (→ #20 → #19). Retarget as those merge.

Built to the decisions recorded in lab/RUNTIME.md ("The line"): open workbench, measurement offline, nothing secret reachable by the agent.

API proxy (lab/apiproxy.py)

  • The gap it closes: before this PR the model API key sat in the agent CLI's environment, so anything in the agent's shell could read it from /proc.
  • How it works:
    • the CLI now holds the placeholder key lab-proxy and talks to a per-session proxy;
    • the proxy swaps in the lab's key (x-api-key) or OAuth token (Authorization: Bearer + the oauth beta);
    • it forwards only /v1/messages, /v1/messages/count_tokens and /v1/models.
  • Metering: it prices each response's usage, JSON or SSE, including the cache-write breakdown, at first-party rates (from the claude-api reference).
  • Refusals: requests stop once the session's cap is spent. Unpriced models and fast mode are always refused, because they would spend unmetered.
  • What the session is charged: the proxy's figure, not the CLI's report. The session record gains cost.tokens.

Workbench (Linux)

  • The CLI runs in a per-session container via docker run -i, so the Agent SDK keeps its stdio.
  • It gets the GPU, the internet, root inside, writable system dirs and Docker's default capabilities.
  • Mounted read-only: the engine's venv, the interpreter, weights, CUDA and the CLI binary. A package installed in the workbench therefore never reaches a measurement.
  • vm-setup.sh drops container traffic to 169.254.0.0/16 (the cloud metadata service) and checks that.
  • macOS keeps srt; there only the proxy's address is allowed.

Pause during measurements

  • measure, equiv and profile jobs run inside container.quiet(workbench).
  • Before the job: the workbench's GPU holders are killed, the container is paused, a second sweep catches anything started before the pause landed, and the job waits for their GPU memory to be released.
  • After the job: the container is thawed.
  • What was killed appears as workbench_killed in the record and in the tool's result. On Linux, every GPU tool's description states the pause as a fact.
  • lab: no measurement on a shared GPU (refused / contaminated) #20's refusal stays as the backstop for any other process.

Tested

  • Proxy, against a fake API: credential swap, OAuth beta, JSON and SSE pricing, byte-for-byte pass-through, the cap, refusals (unpriced model, fast mode, batches, files), count_tokens free.
  • Wrapper: holds no credential in either mode.
  • Pause: sequencing (kill, pause, sweep, thaw).
  • Docker-in-Docker, live: the workbench is root with writable system dirs and reaches the bridge gateway; pause and thaw work. Lab suite on Linux: 254 passed (the remaining failures are the test box's).
  • Wiring: bench passes the session's workbench and reports workbench_killed.
  • macOS: 251 passed; ruff clean.

Not tested:

  • the real CLI through the proxy (it might call an endpoint outside the three; it would get a 403);
  • the macOS srt path (srt isn't installed on the dev box);
  • --gpus and kill-on-GPU on a real GPU.

Not in this PR

  • The agent on a remote worker: the CLI over SSH, workspace jobs, and the proxy through a tunnel.
  • The dependency build step.
  • userns-remap, so that container root isn't uid 0 on the host.

🤖 Generated with Claude Code

AdvayMonga and others added 7 commits October 9, 2026 00:11
@AdvayMonga @claude
lab: container jail on Linux; runtime design (RUNTIME.md)
535a24f 
srt cannot jail GPU code on Linux: no device passthrough, and each jail gets its own
network namespace, so a served engine is unreachable from the clients. On Linux,
agent code (tests, profilers, served engines) now runs in one container per command:
NVIDIA runtime, --network none, read-only root, non-root user, no capabilities,
cpuset leaving cores for the clients. A served port reaches the host through a socat
bridge over a unix socket. srt stays on macOS; LAB_JAIL overrides.

vm-setup.sh installs docker + NVIDIA Container Toolkit, builds lab-jail, and checks the
GPU inside it. lab/RUNTIME.md is the controller/worker design this starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
lab: no measurement on a shared GPU (refused / contaminated)
a27dedc 
A leftover process on the GPU could slow a bench, equiv or submit or stop the engine
from starting, and the ledger recorded that as the snapshot's result; a contaminated
base measurement was cached and skewed every later submit.

serve.Served now refuses to launch while any process holds the GPU (GpuBusy, nothing
launched) and polls the GPU's processes every second while serving; anything outside
the engine's own process tree (descendants, or the container's processes) makes
exclusive() raise Contaminated. bench/equiv/submit/baseline record verdict refused or
contaminated with gpu_processes as facts; a contaminated run is never evidence, never
cached as the base, and is caught before the holdout guard spends a query.
lab.validate's noise bands get the same check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
lab: worker job interface (local and SSH transports)
c9b43ad 
The GPU work behind test, profile/trace/kernel/hostprof, bench, equiv, submit and the
base measurement now runs as worker jobs (lab/worker.py: test, measure, equiv, profile).
A job takes input trees and JSON args, writes an out dir and returns JSON; the tools keep
the policy, the ledger and the budget and call self.worker.

LAB_WORKER=local (default) runs jobs in process, as before. LAB_WORKER=ssh runs them on
the lab.vm VM: each tree goes by content hash at most once (has/put), the worker
re-hashes a fresh copy before every job and refuses a mismatch, and the out dir comes back
as a tar. GpuBusy, Contaminated, NotReady and ValueError are raised again on the
controller. equiv ships the snapshot's earlier answers with the job, so a second equiv
never re-rolls the gate (it already did not, in process).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
RUNTIME.md: the user's access line and the workbench decisions (2026-…
a79167e 
…10-09)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
RUNTIME.md: open workbench, offline measurement as a fairness rule, t…
a089209 
…he dependency build step (approved 2026-10-09)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
lab: the agent's workbench container, its pause during measurements, …
125cd97 
…and the API-key proxy

API proxy (lab/apiproxy.py): the agent's CLI holds a placeholder key and talks to a
per-session proxy that swaps in the lab's key or OAuth token, forwards only
/v1/messages, count_tokens and /v1/models, prices each response's usage (JSON or SSE)
at first-party rates, and refuses once the session cap is spent (and any unpriced model
or fast mode). The session is charged what the proxy priced. Before, the key sat in the
CLI's environment, readable from /proc by the agent's shell.

Workbench (Linux): the CLI runs in a per-session container with the GPU, the internet,
root inside, writable system dirs; the engine venv, weights and CLI binary read-only so
an install never reaches a measurement. vm-setup.sh blocks the cloud metadata service
for containers.

Pause: measure, equiv and profile jobs kill the workbench's GPU holders, pause it, and
thaw it after; what was killed is recorded as workbench_killed and returned as a fact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga @claude
agent: the CLI's scratch dir is /tmp off macOS (srt branch)
5c05798 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga AdvayMonga mentioned this pull request Oct 10, 2026
Merged
@AdvayMonga
Merge remote-tracking branch 'origin/main' into lab/workbench
cd3ce3f 
# Conflicts:
#	CLAUDE.md
#	lab/LEDGER.md
#	lab/README.md
#	lab/RUNTIME.md
#	lab/agent.py
#	lab/evaltools.py
#	lab/proftools.py
#	lab/safety/container.py
#	lab/tools.py
#	lab/vm-setup.sh
#	lab/worker.py
#	tests/test_lab_container.py
#	tests/test_lab_evaltools.py
@AdvayMonga
AdvayMonga changed the base branch from lab/worker to main October 10, 2026 19:07
@AdvayMonga
Merge remote-tracking branch 'origin/main' into lab/workbench
47599f1 
# Conflicts:
#	CLAUDE.md
#	lab/evaltools.py
#	lab/proftools.py
#	lab/tools.py
#	lab/worker.py
@AdvayMonga
Merge remote-tracking branch 'origin/main' into HEAD
8f4d78a
@AdvayMonga
AdvayMonga merged commit d2ced5d into main Oct 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AdvayMonga