Skip to content

lab: no measurement on a shared GPU (refused / contaminated) - #20

Merged
AdvayMonga merged 5 commits into
mainfrom
lab/gpu-exclusive
Oct 10, 2026
Merged

AdvayMonga merged 5 commits into
mainfrom
lab/gpu-exclusive

Conversation

@AdvayMonga

Copy link
Copy Markdown
Owner

Stacked on #19 (it uses the container jail's process list). Retarget to main once #19 merges.

Bug

The agent's shell can start GPU processes and leave them running in the background. bench, equiv and submit measured the engine without checking what else was on the GPU. A leftover process could slow a measurement or keep the engine from starting, and the ledger recorded that as the snapshot's result. The base measurement is cached per run, so one contaminated base skewed every later submit.

Fix

  • Before launch: serve.Served refuses to start an engine while any process holds the GPU (GpuBusy). No engine is launched.
  • While serving: it checks the GPU's processes every second. Anything outside the engine's own processes makes Served.exclusive() raise Contaminated. The engine's processes are its process tree, or the container's processes from docker top.
  • Records: bench, equiv, submit and baseline record verdict refused or contaminated, with gpu_processes: [{pid, used_mib, name}]. The agent sees the same facts.
  • Consequences of a contaminated run:
    • it is never evidence for submit (bench isn't ok, equiv isn't pass);
    • it is never cached as the base;
    • in submit, it is caught before the holdout guard spends a query.
  • lab.validate: the noise-band runs get the same check.
  • Why refuse rather than kill: the lab can't yet tell which processes are the agent's. Freeze-and-kill (RUNTIME.md rule 4) comes with the agent container.

Not covered: CPU contention, and a GPU burst shorter than the 1 s poll.

Tested

  • Fake GPU process lists:
    • a busy GPU refuses with nothing launched;
    • a foreign process mid-run gives contaminated and leaves no base cache;
    • the engine's own GPU process isn't foreign.
  • compute_apps parsing.
  • descendants, on Linux.
  • The container's pids(), live in Docker-in-Docker.
  • Lab suite: macOS 227 passed. Linux 228 passed; the remaining failures are the test box's (old pytest, no git SHA).
  • Not tested on a real GPU: whether nvidia-smi --query-compute-apps lists only the engine's processes (DCGM or the lab itself would show up as refused on the first run).

🤖 Generated with Claude Code

AdvayMonga and others added 2 commits October 9, 2026 00:11
srt cannot jail GPU code on Linux: no device passthrough, and each jail gets its own
network namespace, so a served engine is unreachable from the clients. On Linux,
agent code (tests, profilers, served engines) now runs in one container per command:
NVIDIA runtime, --network none, read-only root, non-root user, no capabilities,
cpuset leaving cores for the clients. A served port reaches the host through a socat
bridge over a unix socket. srt stays on macOS; LAB_JAIL overrides.

vm-setup.sh installs docker + NVIDIA Container Toolkit, builds lab-jail, and checks the
GPU inside it. lab/RUNTIME.md is the controller/worker design this starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A leftover process on the GPU could slow a bench, equiv or submit or stop the engine
from starting, and the ledger recorded that as the snapshot's result; a contaminated
base measurement was cached and skewed every later submit.

serve.Served now refuses to launch while any process holds the GPU (GpuBusy, nothing
launched) and polls the GPU's processes every second while serving; anything outside
the engine's own process tree (descendants, or the container's processes) makes
exclusive() raise Contaminated. bench/equiv/submit/baseline record verdict refused or
contaminated with gpu_processes as facts; a contaminated run is never evidence, never
cached as the base, and is caught before the holdout guard spends a query.
lab.validate's noise bands get the same check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga
AdvayMonga changed the base branch from design/controller-worker to main October 10, 2026 17:18
@AdvayMonga
AdvayMonga merged commit 78582a3 into main Oct 10, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant