Agent tools: no upper caps, instrument flags pass through, bench seed, web tools

lab/LEDGER.md

@@ -13,7 +13,7 @@ on write. Every record: `id`, `at`, `schema` (set by the writer), `kind`, `run`,
 | `test` | `test` | `lint`, `tests` (bools), returncode, output tails, `changed`, `scratch_left_out` |
 | `profile` | `profile` | `bundle` (ledger blob of the profiler output), `workspace_copy`, seconds |
 | `profile` | `trace`, `kernel`, `hostprof` | `bundle` (blob of the instrument's output dir), `workspace_copy`, `instrument` (resolved binary), `argv` (every command run), returncode, seconds, output tail |
-| `bench` | `bench` | `config: {split: seen, tier}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` |
+| `bench` | `bench` | `config: {split: seen, tier, seed}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` |
 | `equiv` | `equiv` | `config: {split: seen, tier}`; `verdict: pass\|fail\|inconclusive`, `passed`, `reasons`, `gates`, `metrics` (accuracy with CI, length, consistency, divergence, per-task flips, unanswered), `thresholds`, `artifacts` |
 | `submit` | `submit` | **held-out shape**: no `result`; top-level `config: {split: heldout, tier: full}` and `metrics: {regime: {base, new, delta_pct, band_pct, verdict}}`, scalars only. The ledger refuses anything else that mentions the held-out split. |
 | `baseline` | the loop, once per run before the first session | `config: {split: seen, tier: short, commit}`; `verdict: ok` with `metrics` shaped like `bench` (the base commit under bench's default regimes: the task's objective and constrained ones, from the cache submit compares against), or `verdict: error`, `reason` |

lab/README.md

@@ -106,21 +106,20 @@ with the referee's rights, snapshot the workspace and write the ledger on every
 |---|---|
 | `test` | lint and the fast suite on a pristine two-commit copy of the workspace, jailed |
 | `profile` | `lab.profile` on the pristine copy, jailed, on `args.corpus_class` (default `steady_interactive`, seen split, `args.seed`) or `args.synthetic`; the bundle goes into the ledger as a blob |
-| `trace` | `nsys profile` around `lab.profile --profiler cuda-range` (window only), then `nsys stats` (kernel, CUDA API, memcpy, NVTX tables as CSV), both jailed; `.nsys-rep` + CSVs as a blob |
-| `kernel` | `ncu --kernel-name regex:<kernel_regex> --launch-skip N --launch-count N --set basic\|full` around the same workload, then `ncu --import --csv --page raw`; `.ncu-rep` + `metrics.csv` as a blob |
-| `hostprof` | `lab.profile --profiler pyspy`: py-spy record (speedscope) for `seconds` under the repeated workload plus one `py-spy dump`; the bundle as a blob |
+| `trace` | `nsys profile` (`args.nsys_args`, default window-only CUDA/NVTX/OS-runtime with CUDA-graph nodes) around `lab.profile --profiler cuda-range` (window only), then `nsys stats` (kernel, CUDA API, memcpy, NVTX tables as CSV), both jailed; `.nsys-rep` + CSVs as a blob |
+| `kernel` | `ncu [--kernel-name regex:<kernel_regex>] --launch-skip N --launch-count N --set <any set> [ncu_args]` around the same workload, then `ncu --import --csv --page raw`; `.ncu-rep` + `metrics.csv` as a blob |
+| `hostprof` | `lab.profile --profiler pyspy`: py-spy record (speedscope, plus `args.pyspy_args`) for `seconds` under the repeated workload plus one `py-spy dump`; the bundle as a blob |
 | `ledger` | read records (this run and earlier ones) |
 | `budget` | dollars left |
 | `restore` | workspace back to a snapshot id (`base` resets) |
 | `note` | a note for the human; recorded, changes nothing |
-| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`) |
+| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`, `args.seed`) |
 | `equiv` | serve it and run the correctness gate (`correctness/`) against the target's reference outputs; pass, fail or inconclusive with every metric; `tier=full` uses `<reference.dir>-full` |
 | `submit` | the only thing that can produce a win. Needs a passing full-tier equiv and a seen bench on this snapshot. Measures the held-out split at the full tier, measures the base commit the same way (once per run, cached outside the jail), and records one aggregate per regime (base, new, delta %, noise band %, verdict improved/regressed/within_band/unknown_band) through the Thresholdout guard (`validity/holdout.py`, state in `lab/ledger/holdout_state.json`, seed `LAB_HOLDOUT_SEED`): the held-out numbers themselves never reach the ledger. Noise bands come from `knowledge/noise/<regime>.json` when measured |
 
-`trace`, `kernel` and `hostprof` take integers in fixed ranges (`requests` 1-256, `prompt_len` 1-32768,
-`max_tokens` 1-4096, `launch_count` 1-64, `launch_skip` 0-100000, `seconds` 1-300, `rate` 1-1000), `set` from
-basic/full, and a `kernel_regex` of at most 200 regex characters (no spaces, quotes, `;`, `&`, backticks, `/` or
-`$(`); nothing else reaches the command line, which is an argv list, never a shell. The instrument is resolved on the
+`trace`, `kernel` and `hostprof` take integers with a lower bound only, any regex and set, and the instrument's
+own flags as a list of strings; the command line is an argv list, never a shell, and runs in the jail, where the
+agent's code runs anyway. Cost is the budget's job, not an argument cap. The instrument is resolved on the
 referee's PATH (refused, and recorded as a refusal, when absent) and its directory is added to the jail's read list.
 The workload is lab.profile's in-process scheduler and backend, so these tools see the engine's kernels and host
 code but not the HTTP front end.

lab/agent.py

@@ -15,7 +15,7 @@
 from lab.safety.hooks import WRITE_TOOLS, write_guard
 
 API_HOST = "api.anthropic.com"
-BUILTIN_TOOLS = ["Read", "Grep", "Glob", "Edit", "Write", "Bash"]
+BUILTIN_TOOLS = ["Read", "Grep", "Glob", "Edit", "Write", "Bash", "WebSearch", "WebFetch"]
 PASS_ENV = ("PATH", "ANTHROPIC_API_KEY", "CLAUDE_CODE_OAUTH_TOKEN", "CLAUDE_CODE_ENTRYPOINT",
             "CLAUDE_AGENT_SDK_VERSION")
 # What a session says when it ends. `stop` means the agent is done with this run.

lab/evaltools.py

@@ -137,21 +137,24 @@ def bench(self, args: dict) -> str:
         t = target.load()
         names = _regimes(args, self.bench_defaults())
         tier = args.get("tier") or "short"
+        seed = args.get("seed", self.seed)
+        if isinstance(seed, bool) or not isinstance(seed, int):
+            raise ValueError("seed must be an integer")
         tree = self._pristine()
         t0 = time.monotonic()
         work, rows = self._work("bench"), []
         try:
             with serve.Served(tree, t.engine, log=self.s.run_dir / "serve-bench.log",
                               jailed=self.serve_jailed, artifacts=work) as srv:
-                results = run_regimes(srv.url, t, names, "seen", tier, self.seed, rows)
+                results = run_regimes(srv.url, t, names, "seen", tier, seed, rows)
         except Exception as e:                      # a tool never crashes the session: the failure is the record
             result = {"verdict": "error", "reason": _why(e), "seconds": time.monotonic() - t0,
                       "artifacts": self._store(work, rows)}
-            self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier})
+            self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier, "seed": seed})
             return f"bench failed: {result['reason']}"
         result = {"verdict": "ok", "metrics": headline(results), "regimes": results, "tier": tier,
                   "seconds": time.monotonic() - t0, "ready_s": srv.ready_s, "artifacts": self._store(work, rows)}
-        self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier})
+        self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier, "seed": seed})
         return json.dumps({"snapshot": snap.id, "split": "seen", "tier": tier, "metrics": result["metrics"]}, indent=1)
 
     # -- equiv --------------------------------------------------------------------------

lab/profile.py

@@ -82,15 +82,15 @@ def _allow_ptrace() -> None:
 
 
 async def _sampled(sched: ContinuousBatchScheduler, prompts: list[list[int]], max_tokens: int, out: Path,
-                   pyspy: tuple[str, float, int]) -> tuple[list, int]:
+                   pyspy: tuple[str, float, int, list[str]]) -> tuple[list, int]:
     """Repeat the workload for `seconds` while py-spy samples this process; one stack dump halfway. (results, rounds)"""
-    exe, seconds, rate = pyspy
+    exe, seconds, rate, extra = pyspy
     _allow_ptrace()
     pid = str(os.getpid())
     with open(out / "pyspy-record.log", "w") as log:
         rec = subprocess.Popen([exe, "record", "--pid", pid, "--duration", str(int(seconds)), "--rate", str(rate),
                                 "--format", "speedscope", "--output", str(out / "pyspy.speedscope.json"),
-                                "--nonblocking"], stdout=log, stderr=log)
+                                "--nonblocking", *extra], stdout=log, stderr=log)
 
     async def dump():
         await asyncio.sleep(seconds / 2)
@@ -134,7 +134,7 @@ def outcomes(results: list, max_tokens: int) -> dict:
 
 async def run(backend: InferenceBackend, settings: Settings, prompts: list[list[int]],
               max_tokens: int, out: Path, warmup: int | None = None, source: dict | None = None,
-              profiler: str = "torch", pyspy: tuple[str, float, int] | None = None) -> Path:
+              profiler: str = "torch", pyspy: tuple[str, float, int, list[str]] | None = None) -> Path:
     """Profile `prompts` through the served scheduler config on `backend`; return the bundle directory.
     `warmup` requests (default: as many as the window) are fired together first, outside the window.
     `profiler`: torch (torch.profiler trace), cuda-range (window marked for nsys/ncu), pyspy (`pyspy` = exe, s, Hz)."""
@@ -219,6 +219,7 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--seconds", type=float, default=20.0, help="pyspy: how long to sample under load")
     ap.add_argument("--rate", type=int, default=100, help="pyspy: samples per second")
     ap.add_argument("--pyspy-bin", default="py-spy")
+    ap.add_argument("--pyspy-arg", action="append", default=[], help="pyspy: extra `py-spy record` flag (repeatable)")
     args = ap.parse_args(argv)
 
     # The served configuration, from the same env the server reads; only the workload is ours.
@@ -245,7 +246,7 @@ def main(argv: list[str] | None = None) -> int:
             prompts = [tok.encode_messages(m, thinking=args.enable_thinking) for m in w["messages"]]
             source = {**w["source"], "thinking": args.enable_thinking}
     out = asyncio.run(run(backend, settings, prompts, args.max_tokens, Path(args.out), warmup=args.warmup,
-                          source=source, profiler=args.profiler, pyspy=(args.pyspy_bin, args.seconds, args.rate)))
+                          source=source, profiler=args.profiler, pyspy=(args.pyspy_bin, args.seconds, args.rate, args.pyspy_arg)))
     print(out)
     return 0