Repository navigation
lab: no measurement on a shared GPU (refused / contaminated) - #20
Merged
Merged
Conversation
srt cannot jail GPU code on Linux: no device passthrough, and each jail gets its own network namespace, so a served engine is unreachable from the clients. On Linux, agent code (tests, profilers, served engines) now runs in one container per command: NVIDIA runtime, --network none, read-only root, non-root user, no capabilities, cpuset leaving cores for the clients. A served port reaches the host through a socat bridge over a unix socket. srt stays on macOS; LAB_JAIL overrides. vm-setup.sh installs docker + NVIDIA Container Toolkit, builds lab-jail, and checks the GPU inside it. lab/RUNTIME.md is the controller/worker design this starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A leftover process on the GPU could slow a bench, equiv or submit or stop the engine from starting, and the ledger recorded that as the snapshot's result; a contaminated base measurement was cached and skewed every later submit. serve.Served now refuses to launch while any process holds the GPU (GpuBusy, nothing launched) and polls the GPU's processes every second while serving; anything outside the engine's own process tree (descendants, or the container's processes) makes exclusive() raise Contaminated. bench/equiv/submit/baseline record verdict refused or contaminated with gpu_processes as facts; a contaminated run is never evidence, never cached as the base, and is caught before the holdout guard spends a query. lab.validate's noise bands get the same check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 9, 2026
# Conflicts: # lab/README.md
# Conflicts: # lab/evaltools.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #19 (it uses the container jail's process list). Retarget to
mainonce #19 merges.Bug
The agent's shell can start GPU processes and leave them running in the background.
bench,equivandsubmitmeasured the engine without checking what else was on the GPU. A leftover process could slow a measurement or keep the engine from starting, and the ledger recorded that as the snapshot's result. The base measurement is cached per run, so one contaminated base skewed every latersubmit.Fix
serve.Servedrefuses to start an engine while any process holds the GPU (GpuBusy). No engine is launched.Served.exclusive()raiseContaminated. The engine's processes are its process tree, or the container's processes fromdocker top.bench,equiv,submitandbaselinerecord verdictrefusedorcontaminated, withgpu_processes: [{pid, used_mib, name}]. The agent sees the same facts.submit(benchisn't ok,equivisn't pass);submit, it is caught before the holdout guard spends a query.lab.validate: the noise-band runs get the same check.Not covered: CPU contention, and a GPU burst shorter than the 1 s poll.
Tested
contaminatedand leaves no base cache;compute_appsparsing.descendants, on Linux.pids(), live in Docker-in-Docker.nvidia-smi --query-compute-appslists only the engine's processes (DCGM or the lab itself would show up asrefusedon the first run).🤖 Generated with Claude Code