Observability batch: bundle fixes, passive data, profiler tools, prompt+baseline, trace contract (integrated)

CLAUDE.md

@@ -13,28 +13,39 @@ Hard rule from the user: the loop may receive **facts only, never heuristics or
 the limits, tiers, policy, noise bands, corpus version the referee measures against). Nothing derived from the
 records (counts, best measured, claims vs evidence) goes to the agent; that is `for_researcher.activity`.
 The task (`lab/task.py`, `--task task.toml`: goal, objective regimes, constraints) is stated in full in the brief,
-and a submit returns its `score` under that task: the win condition is known, never inferred.
+and a submit returns its `score` under that task: the win condition is known, never inferred. The system prompt is
+generated per session (`session.system_prompt`: template `lab/prompts/system.md` filled from the target spec and the
+ToolSpec descriptions, which state what each tool does, returns and costs, never when to use it). The brief is the
+task, budget, referee, a BASELINE (the base commit under bench's defaults, measured once per run, `baseline` ledger
+kind; `RunConfig.baseline=False` skips it) and only the run's last `session` record.
 
 ## Layout
 Top-level packages, run from the repo root (no install step; `pyproject.toml` lists optional extras and holds the
 pytest/ruff config). Each folder with commands has its own `python -m`.
 - `targets/<name>.toml` + `lab/target.py` — **the target spec** (since 2026-10-06): model + chat kwargs, engine
   (repo, python, launch command with `{port}`, health path, env, logprobs `api` vllm|sglang|none, write surface,
-  add-only globs, test and lint commands), reference server + its outputs dir, MLPerf latency limits, correctness
+  add-only globs, test and lint commands, optional `[engine.telemetry] env` with `{dir}`), reference server + its outputs dir, MLPerf latency limits, correctness
   tasks and policy, corpus dir. Selected by `LAB_TARGET` or `--target`; `target.load()` raises `NoTarget` otherwise.
   Nothing defaults to the user's engine or to Qwen3; `targets/inference-server.toml` is the committed example and
   what `tests/conftest.py` points at. Surfaces, grader commands, `lab/engine.py`, and the `correctness`,
   `regimes`, `validity` CLIs all read it.
 - `lab/` — moved from inference-server 2026-10-06 (`lab/README.md`). `session.py` (the loop over a dollar budget),
   `agent.py` (the one place a model is called; the agent's jailed shell gets the engine's python on PATH),
-  `tools.py` (metered tools: test, profile, ledger, budget, restore, note) + `evaltools.py` (bench → `regimes` on
+  `tools.py` (metered tools: test, profile, ledger, budget, restore, note; every record's `cost` has wall seconds, and
+  GPU tools charge them at `LAB_GPU_USD_PER_HOUR` / the target's `[cost] gpu_usd_per_hour` to the run's budget) +
+  `proftools.py` (trace = nsys, kernel = ncu, hostprof = py-spy, each wrapping lab.profile's workload in the jail
+  with validated args) + `evaltools.py` (bench → `regimes` on
   the seen split; equiv → `correctness` candidate vs the target's reference; submit → held-out full tier vs the
   base commit, one aggregate per regime through the Thresholdout guard; all three serve the pristine tree via
-  `serve.py`, jailed with local binding allowed), `workspace.py` (exported engine copy + snapshots), `ledger.py`
+  `serve.py`, jailed with local binding allowed; every serve also keeps its passive data, `artifacts.py`: device
+  samples via `gpu.DeviceSampler` (DCGM if `dcgmi`, else nvidia-smi), engine telemetry files from `{dir}`, all
+  client rows, the serve log; bench/equiv as ledger blobs under `result.artifacts`, submit only in
+  `<run>/heldout-private/`, never in the ledger; `artifacts.joined` joins client rows to engine rows by trace id),
+  `workspace.py` (exported engine copy + snapshots), `ledger.py`
   (append-only JSONL, `lab/ledger/` gitignored), `budget.py`, `engine.py` (where the engine repo and its python
   are), `safety/` (write surfaces, srt jail, grader: lint/tests run with the engine's python in the jail),
-  `profile.py` + `bundle.py` + `gpu.py` (profiler harness, staged into the pristine tree under `_harness/` so the
-  jail never opens this repo, which holds `.env` and the ledger), `canary.py` (honest regressions patched into
+  `profile.py` (`--profiler torch|cuda-range|pyspy`) + `bundle.py` + `gpu.py` (profiler harness, staged into the pristine tree under `_harness/` so the
+  jail never opens this repo, which holds `.env` and the ledger; the tool samples its seen-split corpus workload out here, `corpus.sample`, and stages it as `_harness/workload.json`; window-only memory peaks and scheduler counters, warmup sized to the window, op/kernel tables), `canary.py` (honest regressions patched into
   the engine's process), `vm.py` + `providers/` + `vm-setup.sh` (one GPU VM on Verda/Nebius/Crusoe; pushes both
   trees, engine to `LAB_VM_DIR`, this repo to `LAB_VM_ENV_DIR`; `run --env` runs here; `run` stops the VM when done
   unless `--keep`, and every session is capped by `LAB_VM_MAX_MINUTES` via `timeout` + a local watchdog + in-VM
@@ -46,6 +57,12 @@ pytest/ruff config). Each folder with commands has its own `python -m`.
   measures each regime's run-to-run band from repeated reference runs into `knowledge/noise/<regime>.json` (which
   `submit` reads). Exit 0 only if both hold. Needs a GPU for a real target; tested on fakes. `lab/LEDGER.md` is the
   record-format spec: the environment's public interface.
+- `lab/TRACE.md` + `lab/trace_contract.py` — the request-trace contract: per-request engine rows (trace_id = the
+  client's `X-Trace-Id`, `arrival_ts` epoch s, `*_s` monotonic durations, TPOT = (E2E − TTFT)/(n − 1)) in a dir of
+  `*.sqlite` (`requests` + `meta`) or `*.jsonl` + `<stem>.meta.json`, with schema version and drop counts;
+  `python -m lab.trace_contract DIR`. Diagnostics only, never scored. inference-server's `TELEMETRY_DIR` conforms
+  natively (schema 1, inference-server#102); the target names its env vars in `[engine.telemetry] env` (`{dir}`); serve collects
+  the files after every bench/equiv/submit and the contract check's report is stored beside them. vLLM OTel span mapping is in the spec, not implemented.
 - `corpus/` — frozen workload traces (BurstGPT timing, WildChat text), seen/heldout, hashed, `manifest.json`;
   `build_corpus.py` / `fetch_traces.py` rebuild it; any change is a new `corpus_version`.
 - `knowledge/` — measured findings (one JSON each) + `evidence/`; seeded into the ledger as `finding` records.

correctness/client.py

@@ -35,8 +35,9 @@ class NoLogprobs(RuntimeError):
 DEFAULT = Api()
 
 
-def _post(url: str, body: dict, timeout: float = 600) -> dict:
-    req = urllib.request.Request(url, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
+def _post(url: str, body: dict, timeout: float = 600, trace_id: str | None = None) -> dict:
+    headers = {"Content-Type": "application/json", **({"X-Trace-Id": trace_id} if trace_id else {})}
+    req = urllib.request.Request(url, data=json.dumps(body).encode(), headers=headers)
     with urllib.request.urlopen(req, timeout=timeout) as r:
         return json.loads(r.read())
 
@@ -48,9 +49,11 @@ def _chat_body(api: Api, model: str, messages: list[dict], max_tokens: int) -> d
     return body
 
 
-def generate(base_url: str, model: str, messages: list[dict], max_tokens: int = 2048, api: Api = DEFAULT) -> dict:
-    """Greedy. Returns {"text", "completion_tokens", "finish_reason"}."""
-    out = _post(f"{base_url.rstrip('/')}/v1/chat/completions", _chat_body(api, model, messages, max_tokens))
+def generate(base_url: str, model: str, messages: list[dict], max_tokens: int = 2048, api: Api = DEFAULT,
+             trace_id: str | None = None) -> dict:
+    """Greedy. Returns {"text", "completion_tokens", "finish_reason"}. `trace_id` rides as X-Trace-Id."""
+    out = _post(f"{base_url.rstrip('/')}/v1/chat/completions", _chat_body(api, model, messages, max_tokens),
+                trace_id=trace_id)
     ch = out["choices"][0]
     return {"text": ch["message"]["content"] or "", "completion_tokens": (out.get("usage") or {}).get("completion_tokens"),
             "finish_reason": ch.get("finish_reason")}

correctness/run.py

@@ -50,6 +50,7 @@ def _jsonl_read(path: Path) -> list[dict]:
     return [json.loads(l) for l in Path(path).read_text().splitlines() if l.strip()] if Path(path).exists() else []
 
 
+TRACE_PREFIX = "eq"          # X-Trace-Id of a gate request: eq-<item id>, so equiv rows join engine telemetry
 RETRIES = 6                 # a 429/503 (the server shedding under the gate's load) is retried with backoff
 
 
@@ -75,7 +76,8 @@ def collect(url: str, model: str, items: list[dict], out: Path, concurrency: int
 
     def one(it):
         try:
-            g = _retrying(client.generate, url, model, it["messages"], it.get("max_tokens", 2048), api)
+            g = _retrying(client.generate, url, model, it["messages"], it.get("max_tokens", 2048), api,
+                          trace_id=f"{TRACE_PREFIX}-{it['id']}")
         except Exception as e:                      # a failed request is a recorded fact, not a crash
             g = {"text": "", "completion_tokens": None, "finish_reason": None, "error": f"{type(e).__name__}: {e}"[:200]}
         return {"id": it["id"], **g}

feedback/lab_verdict.py

@@ -17,7 +17,7 @@
 from collections import Counter
 from pathlib import Path
 
-KINDS = {"test", "equiv", "bench", "profile", "submit", "finding", "session", "note"}
+KINDS = {"test", "equiv", "bench", "profile", "submit", "finding", "session", "note", "baseline"}
 TOOL_KINDS = ("test", "equiv", "bench", "profile", "submit")
 
 

lab/LEDGER.md

@@ -3,21 +3,47 @@
 Any loop that writes these records gets a verdict and facts back (`python -m feedback <ledger dir>`); the lab's
 tools are one writer. Append-only JSONL, one record per line, `lab/ledger/ledger.jsonl`; `lab/ledger.py` validates
 on write. Every record: `id`, `at`, `schema` (set by the writer), `kind`, `run`, `session`, `snapshot`
-(content hash of the agent's workspace), `snapshot_blob`, `patch`, `cost: {usd}`. Tool records add `tool`, `args`,
+(content hash of the agent's workspace), `snapshot_blob`, `patch`, `cost: {usd}`. Tool records' `cost` also has
+`seconds` (wall time of the call); GPU tools add `gpu_seconds`, `usd_per_hour` and `rate_known`. Tool records add `tool`, `args`,
 `result`. The agent's own words live only under `claim` and never decide anything.
 
 | kind | written by | `result` (or shape) |
 |---|---|---|
 | `session` | the loop | provider, model, turns, seconds, status (`stop`/`continue`), error, `violation`, `claim.note` |
 | `test` | `test` | `lint`, `tests` (bools), returncode, output tails, `changed`, `scratch_left_out` |
 | `profile` | `profile` | `bundle` (ledger blob of the profiler output), `workspace_copy`, seconds |
-| `bench` | `bench` | `config: {split: seen, tier}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s` |
-| `equiv` | `equiv` | `config: {split: seen, tier}`; `verdict: pass\|fail\|inconclusive`, `passed`, `reasons`, `gates`, `metrics` (accuracy with CI, length, consistency, divergence, per-task flips, unanswered), `thresholds` |
+| `profile` | `trace`, `kernel`, `hostprof` | `bundle` (blob of the instrument's output dir), `workspace_copy`, `instrument` (resolved binary), `argv` (every command run), returncode, seconds, output tail |
+| `bench` | `bench` | `config: {split: seen, tier}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` |
+| `equiv` | `equiv` | `config: {split: seen, tier}`; `verdict: pass\|fail\|inconclusive`, `passed`, `reasons`, `gates`, `metrics` (accuracy with CI, length, consistency, divergence, per-task flips, unanswered), `thresholds`, `artifacts` |
 | `submit` | `submit` | **held-out shape**: no `result`; top-level `config: {split: heldout, tier: full}` and `metrics: {regime: {base, new, delta_pct, band_pct, verdict}}`, scalars only. The ledger refuses anything else that mentions the held-out split. |
+| `baseline` | the loop, once per run before the first session | `config: {split: seen, tier: short, commit}`; `verdict: ok` with `metrics` shaped like `bench` (the base commit under bench's default regimes, from the cache submit compares against), or `verdict: error`, `reason` |
 | `finding` | `lab.ledger seed` | a `knowledge/` entry, whole, as `claim`; `source`, `content_sha` |
 | `note` | `note`, `restore`, refusals | free text, or `{verdict: refused, reason}` |
 
 A tool that could not run writes `{verdict: refused | error, reason}` and the agent sees that text; it never
 crashes the session. The integrity verdict (`feedback/lab_verdict.py`) reads only these records: a run is valid
 iff a completed `submit` exists whose snapshot also has a completed seen-split `bench`, a passing `test`, a
 passing `equiv` (tier full) and no failing `equiv`, and no record reports a write-surface violation.
+
+## Passive data (`artifacts`)
+
+Every bench and equiv (including one that errored after serving) keeps the cheap data from the same served run as
+ledger blobs, referenced from `result.artifacts: {device, telemetry, client_rows, serve_log}` (blob paths under the
+ledger root, `telemetry` None when the target has no `[engine.telemetry]` or the engine wrote nothing). The tool's
+reply to the agent never includes them. `lab/artifacts.py` loads them back (`load`) and joins client rows to engine
+rows by trace id (`joined`: client `X-Trace-Id` is `bg-<row id>`).
+
+- `device/`: `meta.json` (`available`; when true `source` dcgm|nvidia-smi, `coarse` (nvidia-smi utilization is
+  coarse, DCGM profiling fields are not), `fields`, `interval_ms`, `exited` if the sampler died) and `samples.csv`,
+  each raw tool line prefixed with epoch seconds. Covers the whole served window, launch to teardown.
+- `telemetry/`: whatever the engine wrote into `{dir}` (inference-server: `<run>.sqlite` with a `requests` table,
+  `events.jsonl`).
+- `client_rows`: JSONL, one per measured request, without generated text. Bench: the regimes' rows with `regime`,
+  `call` (which run_open/run_closed call), `call_started_at` (epoch) and run-clock times; warmups are not kept.
+  Equiv: one per gate item (`id`, `completion_tokens`, `finish_reason`, `error`).
+- `serve_log`: this run's part of the serve log.
+
+**Submit keeps none of this in the ledger.** Its held-out record stays in the strict held-out shape above, with no
+field pointing at passive data, and no blob is written for it. The same data for the held-out serve goes to
+`<run dir>/heldout-private/<snapshot>-<time>/` (same layout, `client_rows.jsonl`), outside the ledger and outside
+every jail: operator-only. The base-commit measurements submit makes keep no passive data.