diff --git a/lab/LEDGER.md b/lab/LEDGER.md index aecd476..52471ff 100644 --- a/lab/LEDGER.md +++ b/lab/LEDGER.md @@ -13,7 +13,7 @@ on write. Every record: `id`, `at`, `schema` (set by the writer), `kind`, `run`, | `test` | `test` | `lint`, `tests` (bools), returncode, output tails, `changed`, `scratch_left_out` | | `profile` | `profile` | `bundle` (ledger blob of the profiler output), `workspace_copy`, seconds | | `profile` | `trace`, `kernel`, `hostprof` | `bundle` (blob of the instrument's output dir), `workspace_copy`, `instrument` (resolved binary), `argv` (every command run), returncode, seconds, output tail | -| `bench` | `bench` | `config: {split: seen, tier}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` | +| `bench` | `bench` | `config: {split: seen, tier, seed}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` | | `equiv` | `equiv` | `config: {split: seen, tier}`; `verdict: pass\|fail\|inconclusive`, `passed`, `reasons`, `gates`, `metrics` (accuracy with CI, length, consistency, divergence, per-task flips, unanswered), `thresholds`, `artifacts` | | `submit` | `submit` | **held-out shape**: no `result`; top-level `config: {split: heldout, tier: full}` and `metrics: {regime: {base, new, delta_pct, band_pct, verdict}}`, scalars only. The ledger refuses anything else that mentions the held-out split. | | `baseline` | the loop, once per run before the first session | `config: {split: seen, tier: short, commit}`; `verdict: ok` with `metrics` shaped like `bench` (the base commit under bench's default regimes: the task's objective and constrained ones, from the cache submit compares against), or `verdict: error`, `reason` | diff --git a/lab/README.md b/lab/README.md index b3686fe..273c87b 100644 --- a/lab/README.md +++ b/lab/README.md @@ -106,21 +106,20 @@ with the referee's rights, snapshot the workspace and write the ledger on every |---|---| | `test` | lint and the fast suite on a pristine two-commit copy of the workspace, jailed | | `profile` | `lab.profile` on the pristine copy, jailed, on `args.corpus_class` (default `steady_interactive`, seen split, `args.seed`) or `args.synthetic`; the bundle goes into the ledger as a blob | -| `trace` | `nsys profile` around `lab.profile --profiler cuda-range` (window only), then `nsys stats` (kernel, CUDA API, memcpy, NVTX tables as CSV), both jailed; `.nsys-rep` + CSVs as a blob | -| `kernel` | `ncu --kernel-name regex: --launch-skip N --launch-count N --set basic\|full` around the same workload, then `ncu --import --csv --page raw`; `.ncu-rep` + `metrics.csv` as a blob | -| `hostprof` | `lab.profile --profiler pyspy`: py-spy record (speedscope) for `seconds` under the repeated workload plus one `py-spy dump`; the bundle as a blob | +| `trace` | `nsys profile` (`args.nsys_args`, default window-only CUDA/NVTX/OS-runtime with CUDA-graph nodes) around `lab.profile --profiler cuda-range` (window only), then `nsys stats` (kernel, CUDA API, memcpy, NVTX tables as CSV), both jailed; `.nsys-rep` + CSVs as a blob | +| `kernel` | `ncu [--kernel-name regex:] --launch-skip N --launch-count N --set [ncu_args]` around the same workload, then `ncu --import --csv --page raw`; `.ncu-rep` + `metrics.csv` as a blob | +| `hostprof` | `lab.profile --profiler pyspy`: py-spy record (speedscope, plus `args.pyspy_args`) for `seconds` under the repeated workload plus one `py-spy dump`; the bundle as a blob | | `ledger` | read records (this run and earlier ones) | | `budget` | dollars left | | `restore` | workspace back to a snapshot id (`base` resets) | | `note` | a note for the human; recorded, changes nothing | -| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`) | +| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`, `args.seed`) | | `equiv` | serve it and run the correctness gate (`correctness/`) against the target's reference outputs; pass, fail or inconclusive with every metric; `tier=full` uses `-full` | | `submit` | the only thing that can produce a win. Needs a passing full-tier equiv and a seen bench on this snapshot. Measures the held-out split at the full tier, measures the base commit the same way (once per run, cached outside the jail), and records one aggregate per regime (base, new, delta %, noise band %, verdict improved/regressed/within_band/unknown_band) through the Thresholdout guard (`validity/holdout.py`, state in `lab/ledger/holdout_state.json`, seed `LAB_HOLDOUT_SEED`): the held-out numbers themselves never reach the ledger. Noise bands come from `knowledge/noise/.json` when measured | -`trace`, `kernel` and `hostprof` take integers in fixed ranges (`requests` 1-256, `prompt_len` 1-32768, -`max_tokens` 1-4096, `launch_count` 1-64, `launch_skip` 0-100000, `seconds` 1-300, `rate` 1-1000), `set` from -basic/full, and a `kernel_regex` of at most 200 regex characters (no spaces, quotes, `;`, `&`, backticks, `/` or -`$(`); nothing else reaches the command line, which is an argv list, never a shell. The instrument is resolved on the +`trace`, `kernel` and `hostprof` take integers with a lower bound only, any regex and set, and the instrument's +own flags as a list of strings; the command line is an argv list, never a shell, and runs in the jail, where the +agent's code runs anyway. Cost is the budget's job, not an argument cap. The instrument is resolved on the referee's PATH (refused, and recorded as a refusal, when absent) and its directory is added to the jail's read list. The workload is lab.profile's in-process scheduler and backend, so these tools see the engine's kernels and host code but not the HTTP front end. diff --git a/lab/agent.py b/lab/agent.py index f496d30..6b3ee26 100644 --- a/lab/agent.py +++ b/lab/agent.py @@ -15,7 +15,7 @@ from lab.safety.hooks import WRITE_TOOLS, write_guard API_HOST = "api.anthropic.com" -BUILTIN_TOOLS = ["Read", "Grep", "Glob", "Edit", "Write", "Bash"] +BUILTIN_TOOLS = ["Read", "Grep", "Glob", "Edit", "Write", "Bash", "WebSearch", "WebFetch"] PASS_ENV = ("PATH", "ANTHROPIC_API_KEY", "CLAUDE_CODE_OAUTH_TOKEN", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_AGENT_SDK_VERSION") # What a session says when it ends. `stop` means the agent is done with this run. diff --git a/lab/evaltools.py b/lab/evaltools.py index 3f86457..b37d046 100644 --- a/lab/evaltools.py +++ b/lab/evaltools.py @@ -137,21 +137,24 @@ def bench(self, args: dict) -> str: t = target.load() names = _regimes(args, self.bench_defaults()) tier = args.get("tier") or "short" + seed = args.get("seed", self.seed) + if isinstance(seed, bool) or not isinstance(seed, int): + raise ValueError("seed must be an integer") tree = self._pristine() t0 = time.monotonic() work, rows = self._work("bench"), [] try: with serve.Served(tree, t.engine, log=self.s.run_dir / "serve-bench.log", jailed=self.serve_jailed, artifacts=work) as srv: - results = run_regimes(srv.url, t, names, "seen", tier, self.seed, rows) + results = run_regimes(srv.url, t, names, "seen", tier, seed, rows) except Exception as e: # a tool never crashes the session: the failure is the record result = {"verdict": "error", "reason": _why(e), "seconds": time.monotonic() - t0, "artifacts": self._store(work, rows)} - self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier}) + self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier, "seed": seed}) return f"bench failed: {result['reason']}" result = {"verdict": "ok", "metrics": headline(results), "regimes": results, "tier": tier, "seconds": time.monotonic() - t0, "ready_s": srv.ready_s, "artifacts": self._store(work, rows)} - self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier}) + self._record("bench", "bench", args, result, snap, config={"split": "seen", "tier": tier, "seed": seed}) return json.dumps({"snapshot": snap.id, "split": "seen", "tier": tier, "metrics": result["metrics"]}, indent=1) # -- equiv -------------------------------------------------------------------------- diff --git a/lab/profile.py b/lab/profile.py index d458f5d..f8cf07a 100644 --- a/lab/profile.py +++ b/lab/profile.py @@ -82,15 +82,15 @@ def _allow_ptrace() -> None: async def _sampled(sched: ContinuousBatchScheduler, prompts: list[list[int]], max_tokens: int, out: Path, - pyspy: tuple[str, float, int]) -> tuple[list, int]: + pyspy: tuple[str, float, int, list[str]]) -> tuple[list, int]: """Repeat the workload for `seconds` while py-spy samples this process; one stack dump halfway. (results, rounds)""" - exe, seconds, rate = pyspy + exe, seconds, rate, extra = pyspy _allow_ptrace() pid = str(os.getpid()) with open(out / "pyspy-record.log", "w") as log: rec = subprocess.Popen([exe, "record", "--pid", pid, "--duration", str(int(seconds)), "--rate", str(rate), "--format", "speedscope", "--output", str(out / "pyspy.speedscope.json"), - "--nonblocking"], stdout=log, stderr=log) + "--nonblocking", *extra], stdout=log, stderr=log) async def dump(): await asyncio.sleep(seconds / 2) @@ -134,7 +134,7 @@ def outcomes(results: list, max_tokens: int) -> dict: async def run(backend: InferenceBackend, settings: Settings, prompts: list[list[int]], max_tokens: int, out: Path, warmup: int | None = None, source: dict | None = None, - profiler: str = "torch", pyspy: tuple[str, float, int] | None = None) -> Path: + profiler: str = "torch", pyspy: tuple[str, float, int, list[str]] | None = None) -> Path: """Profile `prompts` through the served scheduler config on `backend`; return the bundle directory. `warmup` requests (default: as many as the window) are fired together first, outside the window. `profiler`: torch (torch.profiler trace), cuda-range (window marked for nsys/ncu), pyspy (`pyspy` = exe, s, Hz).""" @@ -219,6 +219,7 @@ def main(argv: list[str] | None = None) -> int: ap.add_argument("--seconds", type=float, default=20.0, help="pyspy: how long to sample under load") ap.add_argument("--rate", type=int, default=100, help="pyspy: samples per second") ap.add_argument("--pyspy-bin", default="py-spy") + ap.add_argument("--pyspy-arg", action="append", default=[], help="pyspy: extra `py-spy record` flag (repeatable)") args = ap.parse_args(argv) # The served configuration, from the same env the server reads; only the workload is ours. @@ -245,7 +246,7 @@ def main(argv: list[str] | None = None) -> int: prompts = [tok.encode_messages(m, thinking=args.enable_thinking) for m in w["messages"]] source = {**w["source"], "thinking": args.enable_thinking} out = asyncio.run(run(backend, settings, prompts, args.max_tokens, Path(args.out), warmup=args.warmup, - source=source, profiler=args.profiler, pyspy=(args.pyspy_bin, args.seconds, args.rate))) + source=source, profiler=args.profiler, pyspy=(args.pyspy_bin, args.seconds, args.rate, args.pyspy_arg))) print(out) return 0 diff --git a/lab/proftools.py b/lab/proftools.py index 12a739a..da93092 100644 --- a/lab/proftools.py +++ b/lab/proftools.py @@ -1,5 +1,6 @@ """Instrumented runs of the agent's pristine tree, one tool per instrument. Each wraps lab.profile's workload inside -the jail with fixed, validated arguments, keeps the raw output as a ledger blob and returns a short raw summary. +the jail, passes the instrument's own flags through as an argument list, keeps the raw output as a ledger blob and +returns a short raw summary. trace Nsight Systems: timeline of the measured window (.nsys-rep) plus `nsys stats` CSV tables kernel Nsight Compute: kernels whose name matches a regex (.ncu-rep) plus `ncu --import --csv --page raw` @@ -18,27 +19,35 @@ OUT = "_instr" # in the pristine tree: writable in the jail, rebuilt on every call NSYS_REPORTS = "cuda_gpu_kern_sum,cuda_api_sum,cuda_gpu_mem_time_sum,nvtx_sum" -NCU_SETS = ("basic", "full") -KERNEL_RE = re.compile(r"[\w.*+?|()\[\]{}^$\\:,<>~-]{1,200}") # regex syntax only: no spaces, quotes, ; & ` or / +NSYS_FLAGS = ["--trace", "cuda,nvtx,osrt", "--capture-range", "cudaProfilerApi", "--capture-range-end", "stop", + "--cuda-graph-trace", "node"] SUMMARY_CHARS = 2000 -WORKLOAD = {"requests": (8, 1, 256), "prompt_len": (64, 1, 32768), "max_tokens": (32, 1, 4096)} +WORKLOAD = {"requests": (8, 1), "prompt_len": (64, 1), "max_tokens": (32, 1)} # (default, minimum) -def _int(args: dict, name: str, default: int, lo: int, hi: int) -> int: +def _int(args: dict, name: str, default: int, lo: int) -> int: v = args.get(name, default) - if isinstance(v, bool) or not isinstance(v, int) or not lo <= v <= hi: - raise ValueError(f"{name} must be an integer in [{lo}, {hi}]") + if isinstance(v, bool) or not isinstance(v, int) or v < lo: + raise ValueError(f"{name} must be an integer >= {lo}") return v +def _flags(args: dict, name: str, default: list[str] | None = None) -> list[str]: + """An instrument's own flags, passed through as argv items (never through a shell).""" + v = args.get(name, default if default is not None else []) + if not isinstance(v, list) or not all(isinstance(x, str) for x in v): + raise ValueError(f"{name} must be a list of strings") + return list(v) + + def workload(args: dict) -> list[str]: """lab.profile's workload flags from validated integers.""" - return [w for k, (d, lo, hi) in WORKLOAD.items() for w in (f"--{k.replace('_', '-')}", str(_int(args, k, d, lo, hi)))] + return [w for k, (d, lo) in WORKLOAD.items() for w in (f"--{k.replace('_', '-')}", str(_int(args, k, d, lo)))] def kernel_regex(value) -> str: - if not isinstance(value, str) or not KERNEL_RE.fullmatch(value) or "$(" in value: - raise ValueError("kernel_regex must be 1-200 characters: letters, digits and _.*+?|()[]{}^$\\:,<>~-") + if not isinstance(value, str) or not value: + raise ValueError("kernel_regex must be a non-empty string") try: re.compile(value) except re.error as e: @@ -46,8 +55,11 @@ def kernel_regex(value) -> str: return value +FLAGS = {"type": "array", "items": {"type": "string"}} + + def _schema(**extra) -> dict: - props = {k: {"type": "integer", "minimum": lo, "maximum": hi, "default": d} for k, (d, lo, hi) in WORKLOAD.items()} + props = {k: {"type": "integer", "minimum": lo, "default": d} for k, (d, lo) in WORKLOAD.items()} props.update(corpus_class={"type": "string"}, seed={"type": "integer"}, synthetic={"type": "boolean"}) return {"type": "object", "properties": {**props, **extra}} @@ -60,33 +72,35 @@ class ProfTools: """Mixin for lab.tools.Toolbox: needs self._audited, self._pristine, self._record, self._keep.""" def trace(self, args: dict) -> str: - wl = workload(args) + wl, flags = workload(args), _flags(args, "nsys_args", NSYS_FLAGS) wrap = lambda exe, out: [exe, "profile", "--output", str(out / "trace"), "--force-overwrite", "true", - "--trace", "cuda,nvtx,osrt", "--capture-range", "cudaProfilerApi", - "--capture-range-end", "stop", "--cuda-graph-trace", "node", "--stats", "false"] + *flags, "--stats", "false"] post = lambda exe, out: ([exe, "stats", "--report", NSYS_REPORTS, "--format", "csv", "--output", str(out / "stats"), str(out / "trace.nsys-rep")], None) return self._instrumented("trace", args, "nsys", wrap, wl + ["--profiler", "cuda-range"], post, "stats_cuda_gpu_kern_sum.csv") def kernel(self, args: dict) -> str: - rx = kernel_regex(args.get("kernel_regex")) - skip, count = _int(args, "launch_skip", 0, 0, 100000), _int(args, "launch_count", 8, 1, 64) + rx = kernel_regex(args["kernel_regex"]) if "kernel_regex" in args else None + skip, count = _int(args, "launch_skip", 0, 0), _int(args, "launch_count", 8, 1) kset = args.get("set", "basic") - if kset not in NCU_SETS: - raise ValueError(f"set must be one of {NCU_SETS}") - wl = workload(args) - wrap = lambda exe, out: [exe, "--export", str(out / "kernel"), "--force-overwrite", "--kernel-name", - f"regex:{rx}", "--launch-skip", str(skip), "--launch-count", str(count), "--set", kset, - "--profile-from-start", "off", "--target-processes", "all"] + if not isinstance(kset, str) or not kset: + raise ValueError("set must be a non-empty string") + wl, flags = workload(args), _flags(args, "ncu_args") + wrap = lambda exe, out: [exe, "--export", str(out / "kernel"), "--force-overwrite", + *(["--kernel-name", f"regex:{rx}"] if rx else []), "--launch-skip", str(skip), + "--launch-count", str(count), "--set", kset, "--profile-from-start", "off", + "--target-processes", "all", *flags] post = lambda exe, out: ([exe, "--import", str(out / "kernel.ncu-rep"), "--csv", "--page", "raw"], out / "metrics.csv") return self._instrumented("kernel", args, "ncu", wrap, wl + ["--profiler", "cuda-range"], post, "metrics.csv") def hostprof(self, args: dict) -> str: wl = workload(args) - seconds, rate = _int(args, "seconds", 20, 1, 300), _int(args, "rate", 100, 1, 1000) - extra = lambda exe: ["--profiler", "pyspy", "--seconds", str(seconds), "--rate", str(rate), "--pyspy-bin", exe] + seconds, rate = _int(args, "seconds", 20, 1), _int(args, "rate", 100, 1) + more = [w for f in _flags(args, "pyspy_args") for w in ("--pyspy-arg", f)] + extra = lambda exe: ["--profiler", "pyspy", "--seconds", str(seconds), "--rate", str(rate), "--pyspy-bin", exe, + *more] return self._instrumented("hostprof", args, "py-spy", lambda exe, out: [], wl, None, "bundle/*/pyspy-dump.txt", extra) @@ -136,25 +150,27 @@ def _instrumented(self, tool: str, args: dict, binary: str, wrap, flags: list[st def prof_specs(self, gpu) -> list[ToolSpec]: """`gpu(time)` renders the cost fact for a description.""" return [ - ToolSpec("trace", "Run the profile workload of your current workspace under Nsight Systems (nsys), " - "capturing the measured window only. Stores the .nsys-rep and `nsys stats` CSV tables (GPU " - "kernels, CUDA API, memory copies, NVTX) in the ledger; returns the head of the kernel table. " + ToolSpec("trace", "Run the profile workload of your current workspace under Nsight Systems (nsys). " + f"`nsys_args`: the `nsys profile` flags, default {' '.join(NSYS_FLAGS)} (the measured window " + "only); output and overwrite are set by the tool. Stores the .nsys-rep and `nsys stats` CSV tables " + "(GPU kernels, CUDA API, memory copies, NVTX) in the ledger; returns the head of the kernel table. " "Refused where nsys is not installed." + gpu("about 1 min measured at 16 requests x 64 tokens on an H200 (engine start, workload, report export)"), - _schema(), self.trace), + _schema(nsys_args=FLAGS), self.trace), ToolSpec("kernel", "Run the profile workload under Nsight Compute (ncu) on the kernels whose name " - "matches `kernel_regex`: skips `launch_skip` matching launches, then profiles `launch_count` " - "with metric set `set`. Stores the .ncu-rep and its raw metrics CSV in the ledger; returns the " - "head of the CSV. Refused where ncu is not installed." + "matches `kernel_regex` (every kernel if omitted): skips `launch_skip` matching launches, then " + "profiles `launch_count` with section set `set` (any `ncu --list-sets` name); `ncu_args` are " + "appended as further ncu flags. Stores the .ncu-rep and its raw metrics CSV in the ledger; returns " + "the head of the CSV. Refused where ncu is not installed." + gpu("about 1.5 min measured at 4-8 launches on an H200; each profiled launch is replayed once per metric pass, so it grows with launch_count and set=full"), - _schema(kernel_regex={"type": "string", "maxLength": 200}, - launch_skip={"type": "integer", "minimum": 0, "maximum": 100000, "default": 0}, - launch_count={"type": "integer", "minimum": 1, "maximum": 64, "default": 8}, - set={"type": "string", "enum": list(NCU_SETS), "default": "basic"}) - | {"required": ["kernel_regex"]}, self.kernel), + _schema(kernel_regex={"type": "string"}, + launch_skip={"type": "integer", "minimum": 0, "default": 0}, + launch_count={"type": "integer", "minimum": 1, "default": 8}, + set={"type": "string", "default": "basic"}, ncu_args=FLAGS), self.kernel), ToolSpec("hostprof", "Sample the engine process with py-spy for `seconds` while the profile workload " "repeats (scheduler and backend in process, warmup excluded). Stores a speedscope profile and one " - "stack dump taken halfway in the ledger; returns the head of the dump. Refused where py-spy is " + "stack dump taken halfway in the ledger; returns the head of the dump. `pyspy_args` are appended " + "to `py-spy record`. Refused where py-spy is " "not installed." + gpu("about 35 s plus `seconds`, measured on an H200"), - _schema(seconds={"type": "integer", "minimum": 1, "maximum": 300, "default": 20}, - rate={"type": "integer", "minimum": 1, "maximum": 1000, "default": 100}), self.hostprof), + _schema(seconds={"type": "integer", "minimum": 1, "default": 20}, + rate={"type": "integer", "minimum": 1, "default": 100}, pyspy_args=FLAGS), self.hostprof), ] diff --git a/lab/tools.py b/lab/tools.py index 6a51a02..ee06e4b 100644 --- a/lab/tools.py +++ b/lab/tools.py @@ -197,10 +197,12 @@ def gpu(t: str) -> str: {"type": "object", "required": ["text"], "properties": {"text": {"type": "string"}}}, self.note), ToolSpec("bench", "Serve a pristine copy of your current workspace (jailed) and measure it under load " f"regimes on the seen split. `regimes`: default {_names(self.bench_defaults())}; 'all'; or " - f"any of {_names(suite.REGIMES)}. `tier`: short (default) or full. Returns one headline per regime " + f"any of {_names(suite.REGIMES)}. `tier`: short (default) or full. `seed`: the seen-split workload sample " + f"(default {self.seed}). Returns one headline per regime " f"(objective, value, direction, validity), raw. Takes {windows}." + gpu("5-25 min per regime"), {"type": "object", "properties": {"regimes": {"type": "array", "items": {"type": "string"}}, - "tier": {"type": "string", "enum": ["short", "full"]}}}, self.bench), + "tier": {"type": "string", "enum": ["short", "full"]}, + "seed": {"type": "integer"}}}, self.bench), ToolSpec("equiv", "Serve your current workspace and run the correctness gate against the reference " "model's outputs at `tier` dev (default) or full. Returns pass, fail or inconclusive with every " "metric. submit requires a passing full-tier equiv on the snapshot. Takes one engine start plus " diff --git a/tests/test_lab_evaltools.py b/tests/test_lab_evaltools.py index 015e714..259e839 100644 --- a/tests/test_lab_evaltools.py +++ b/tests/test_lab_evaltools.py @@ -118,7 +118,7 @@ def test_bench_serves_and_measures(lab): assert out["split"] == "seen" and out["metrics"]["single_stream"]["valid"] assert out["metrics"]["single_stream"]["value"] is not None rec = next(ledger.records(s.ledger_root, kind="bench")) - assert rec["config"] == {"split": "seen", "tier": "short"} and rec["result"]["verdict"] == "ok" + assert rec["config"] == {"split": "seen", "tier": "short", "seed": tb.seed} and rec["result"]["verdict"] == "ok" assert rec["result"]["ready_s"] is not None and (s.run_dir / "serve-bench.log").exists() @@ -314,3 +314,15 @@ def test_bench_defaults_follow_the_task(): tb.s.task = Task("g", ("long_prompt_short_output",), constraints={"single_stream": {"max_regression_pct": 5}, "long_prompt_short_output": {"max_regression_pct": 1}}) assert tb.bench_defaults() == ("long_prompt_short_output", "single_stream") + + +def test_bench_takes_the_agents_seed(lab, monkeypatch): + from lab import evaltools + tb, _ = lab + seeds = [] + monkeypatch.setattr(evaltools, "run_regimes", lambda url, t, names, split, tier, seed, rows=None: seeds.append(seed) or []) + tb.bench({"regimes": ["single_stream"], "seed": 7}) + tb.bench({"regimes": ["single_stream"]}) + assert seeds == [7, tb.seed] + with pytest.raises(ValueError): + tb.bench({"seed": "7"}) diff --git a/tests/test_lab_profile.py b/tests/test_lab_profile.py index 3977a7e..3109590 100644 --- a/tests/test_lab_profile.py +++ b/tests/test_lab_profile.py @@ -157,9 +157,10 @@ async def test_pyspy_samples_this_process_under_a_repeated_workload(tmp_path, st exe.write_text(FAKE_PYSPY.replace("LOG, ", f"{str(tmp_path / 'calls.log')!r}, ")) exe.chmod(0o755) out = await profile.run(stub_backend_cls(), Settings(max_batch_size=2), profile.synthetic_prompts(2, 4), max_tokens=2, - out=tmp_path / "b", profiler="pyspy", pyspy=(str(exe), 0.5, 50)) + out=tmp_path / "b", profiler="pyspy", pyspy=(str(exe), 0.5, 50, ["--native"])) rec, dump = [json.loads(line) for line in (tmp_path / "calls.log").read_text().splitlines()] assert rec[:3] == ["record", "--pid", str(os.getpid())] and "--nonblocking" in rec and "speedscope" in rec + assert rec[-1] == "--native" assert dump == ["dump", "--pid", str(os.getpid()), "--nonblocking"] assert (out / "pyspy.speedscope.json").exists() and "MainThread" in (out / "pyspy-dump.txt").read_text() meta = json.loads((out / "meta.json").read_text()) diff --git a/tests/test_lab_proftools.py b/tests/test_lab_proftools.py index 998d3da..aa9730f 100644 --- a/tests/test_lab_proftools.py +++ b/tests/test_lab_proftools.py @@ -138,20 +138,18 @@ def test_refused_cleanly_without_the_instrument(tb, monkeypatch, tool, args): assert rec["tool"] == tool and rec["result"]["verdict"] == "refused" and calls(tb) == [] -@pytest.mark.parametrize("bad", ["gemm; rm -rf /", "$(id)", "`id`", "a b", "x\nid", "a'b", 'a"b', "a&b", "../x", - "", "a" * 201, "(", None, 7, ["gemm"]]) -def test_kernel_regex_rejects_injection(tb, bad): +@pytest.mark.parametrize("bad", ["", "(", None, 7, ["gemm"]]) +def test_kernel_regex_must_be_a_regex(tb, bad): with pytest.raises(ValueError): tb.kernel({"kernel_regex": bad}) assert calls(tb) == [] and not list(ledger.records(tb.s.ledger_root)) @pytest.mark.parametrize("tool,args", [ - ("kernel", {"kernel_regex": "gemm", "set": "full --replay-mode application"}), - ("kernel", {"kernel_regex": "gemm", "launch_count": 0}), - ("kernel", {"kernel_regex": "gemm", "launch_count": "4; id"}), - ("trace", {"requests": True}), ("trace", {"requests": "8 --out /"}), ("trace", {"max_tokens": 10**6}), - ("hostprof", {"seconds": 301}), ("hostprof", {"rate": -1}), + ("kernel", {"kernel_regex": "gemm", "set": ""}), ("kernel", {"kernel_regex": "gemm", "launch_count": 0}), + ("kernel", {"kernel_regex": "gemm", "launch_count": "4; id"}), ("kernel", {"ncu_args": "--section X"}), + ("trace", {"requests": True}), ("trace", {"requests": "8 --out /"}), ("trace", {"nsys_args": [1]}), + ("hostprof", {"seconds": 0}), ("hostprof", {"rate": -1}), ("hostprof", {"pyspy_args": "--native"}), ]) def test_arguments_are_validated_before_anything_runs(tb, tool, args): with pytest.raises(ValueError): @@ -159,6 +157,27 @@ def test_arguments_are_validated_before_anything_runs(tb, tool, args): assert calls(tb) == [] +def test_instrument_flags_pass_through_as_single_argv_items_with_no_upper_limits(tb, monkeypatch): + tb.trace({"nsys_args": ["--trace", "cuda,cublas", "--sample", "none"], "requests": 1000, "max_tokens": 8192, + "synthetic": True}) + prof = calls(tb)[0] + assert after(prof, "--trace") == "cuda,cublas" and "--capture-range" not in prof + assert after(prof, "--requests") == "1000" and after(prof, "--max-tokens") == "8192" + tb.kernel({"kernel_regex": "a b; id", "set": "roofline", "launch_count": 500, "ncu_args": ["--section", "Occupancy"]}) + prof = calls(tb)[2] + assert after(prof, "--kernel-name") == "regex:a b; id" and after(prof, "--set") == "roofline" + assert after(prof, "--launch-count") == "500" and prof[prof.index("--section") + 1] == "Occupancy" + tb.kernel({}) + assert "--kernel-name" not in calls(tb)[4] + seen = {} + monkeypatch.setattr(labtools, "default_profile_runner", + lambda tree, argv, read=(): seen.update(argv=argv) or subprocess.CompletedProcess(argv, 1, "", "")) + tb.hostprof({"seconds": 900, "rate": 5000, "pyspy_args": ["--native", "--idle"]}) + a = seen["argv"] + assert after(a, "--seconds") == "900" and after(a, "--rate") == "5000" + assert [a[i + 1] for i, x in enumerate(a) if x == "--pyspy-arg"] == ["--native", "--idle"] + + def test_cost_free_tools_record_wall_time_and_zero(tb): tb.note({"text": "hi"}) tb.restore({"snapshot": "base"})