Ceilings: every measured number with its physical bound and the reference

CLAUDE.md

@@ -33,6 +33,12 @@ pytest/ruff config). Each folder with commands has its own `python -m`.
   `agent.py` (the one place a model is called; the agent's jailed shell gets the engine's python on PATH;
   `LAB_AGENT_PROVIDER` claude = Agent SDK CLI, openai = `agent_openai.py`: any OpenAI-compatible API with its own
   key, https public hosts only, `LAB_MODEL_PRICE` required, loop outside the jail, Bash jailed with no network),
+  `calibrate.py` (every measured number with its context, decided 2026-10-11: `% of ceiling` from `validity/ceiling`
+  on measured peaks, `% of reference` from `knowledge/reference/<target>/<regime>.json` written by `lab.validate`
+  with `perf_reference` (the correctness reference minus its determinism switches, which are slow by design),
+  and for kernels their share of the window; a ceiling or reference is shown only under a matching config
+  fingerprint (model, dtypes, max_model_len, tier, split, corpus version, GPU), else the field says why it is
+  missing; bench, baseline, profile and kernel records carry a `calibrated` block, the brief shows the baseline's),
   `tools.py` (metered tools: test, profile, ledger, budget, restore, note; every record's `cost` has wall seconds, and
   GPU tools charge them at `LAB_GPU_USD_PER_HOUR` / the target's `[cost] gpu_usd_per_hour` to the run's budget) +
   `proftools.py` (trace = nsys, kernel = ncu, hostprof = py-spy, each wrapping lab.profile's workload in the jail
@@ -102,7 +108,14 @@ pytest/ruff config). Each folder with commands has its own `python -m`.
   (`verdict` re-judges saved results under the policy, no server).
 - `canaries/cheat_proxy.py` — cheat proxy in front of any OpenAI-compatible server (`truncate`, `early_eos`,
   `fake_first`, `drop`, `cache` (streamed and not), `inflate_usage`); `python -m canaries --upstream URL --cheat NAME`.
-- `validity/` — `roofline.py` (analytical decode ceiling, no GPU; to be generalized into physical limits), `repeat_audit.py`: repeated-request audit (MLPerf TEST04-style), judges decode time per token, not TTFT, so prefix caching passes and replayed answers fail; `python -m validity repeat --url URL`. `tier_agreement.py`: `tier_agreement(changes)`: Spearman, Kendall tau-b, bootstrap CI, out-of-band disagreements; short tier usable if n ≥ 8, rho ≥ 0.8, CI low ≥ 0.5. `holdout.py`: `HoldoutGuard` (Thresholdout query budget, state persisted atomically under a file lock, deterministic from seed) and `SealedSplit` (hash manifest, one logged unseal per corpus version). Keep sigma small relative to the threshold.
+- `validity/` — `ceiling.py` (physical bounds from the model's HF config and the GPU's peaks: active/total params,
+  bytes streamed per step with MoE experts-touched, decode and prefill floors, the largest batch under a TPOT limit,
+  per-objective regime ceilings: TPOT = decode step floor, TTFT = prefill floor at the p99 prompt, goodput =
+  1 / (prefill compute + output tokens at the decode ceiling, batch capped by TPOT and by KV fitting in HBM); no
+  bound for speedups, cold start or sessions/s; assumptions (one token per step, uniform routing, declared dtypes)
+  are in each `basis`, so a value above 100% means one of them does not hold; `pct_of`, `ceiling_gain` = Amdahl) + `peaks.py` (`python -m validity peaks`: HBM
+  bandwidth and bf16 GEMM measured once per GPU+driver into `knowledge/peaks/<gpu>.json`, run by `vm-setup.sh`; spec
+  sheet as the labelled fallback), `repeat_audit.py`: repeated-request audit (MLPerf TEST04-style), judges decode time per token, not TTFT, so prefix caching passes and replayed answers fail; `python -m validity repeat --url URL`. `tier_agreement.py`: `tier_agreement(changes)`: Spearman, Kendall tau-b, bootstrap CI, out-of-band disagreements; short tier usable if n ≥ 8, rho ≥ 0.8, CI low ≥ 0.5. `holdout.py`: `HoldoutGuard` (Thresholdout query budget, state persisted atomically under a file lock, deterministic from seed) and `SealedSplit` (hash manifest, one logged unseal per corpus version). Keep sigma small relative to the threshold.
 - `kernels/` — kernel equivalence: `check_kernel` (seen + held-out shapes, per-dtype tolerances, edge cases incl. non-contiguous and NaN/Inf, inputs unchanged, no aliasing, memoization, determinism) and `time_kernel` / `check_memoization` (sync, rotated buffers, L2 flush on CUDA). Validated on CPU, MPS and CUDA (H200, 2026-10-04): all tests pass; the timer resolves ~20 µs kernels at ~9% IQR, 0.2 ms at 1.5%. `make_inputs` gets a seeded CPU generator (generate on CPU, then `.to(device)`).
 - `regimes/` — load generation: drives a live OpenAI-compatible server the eight ways it gets used, client-side.
   `client.py` (one streamed chat request: TTFT from *scheduled* send to first real token, TPOT = (E2E − TTFT)/(n − 1),
@@ -232,6 +245,12 @@ a first `python -m lab.session --task ...` dry run with a tiny budget, which wai
 `engine/gpu-validated`) were judged by the old gate: re-judge with `python -m correctness verdict` on their saved
 results before deciding anything. lab `session.py --run` re-resolves
 `base` from HEAD rather than the run's original base (pre-existing; resuming after the engine moved would misaudit);
-`validity/tier_agreement.py` needs real short/full runs; `validity/roofline.py` wants generalizing to the H200 + MoE
+`validity/tier_agreement.py` needs real short/full runs. Ceilings checked on an H200 (Nebius, driver 580.173,
+2026-10-13; peaks 4.14 TB/s, 794 TFLOP/s bf16): every floor below every measurement once goodput charges only the
+prefill the server reports as uncached (vLLM's prefix cache made a prefill-inclusive bound read 147-471%). Plain vLLM:
+single_stream TPOT p99 4.1 ms (37% of ceiling), saturated 5.8 req/s (19%), long prompt 68 req/s (94%); the engine:
+21 ms (7%), 0.097 req/s (0.5%), long prompt invalid (111 requests without a response). Stored in
+`knowledge/reference/inference-server/`. Finding for the corpus: long_prompt_short_output's window repeats each
+prompt ~5x (23,185 requests over 4,794 prompts), which rewards prefix caching.
 target. The agent's tool set (e.g. a knowledge tool) is the user's design: ask first. README stays a few one-line
 feature bullets, not a dev doc.

knowledge/peaks/nvidia-h200-580-126-09.json

@@ -0,0 +1,11 @@
+{
+ "gpu": "NVIDIA H200",
+ "bf16_flops": 793933120974445.2,
+ "bw_bytes_s": 4120900460756.014,
+ "mem_bytes": 150109880320.0,
+ "source": "measured",
+ "driver": "580.126.09",
+ "sm_clock_mhz": 1455,
+ "measured_at": "2026-10-12T05:51:02",
+ "method": "best of 3.0s: 512MiB bf16 copy; 8192^3 bf16 GEMM"
+}

knowledge/peaks/nvidia-h200-580-173-02.json

@@ -0,0 +1,11 @@
+{
+ "gpu": "NVIDIA H200",
+ "bf16_flops": 794182576833900.2,
+ "bw_bytes_s": 4135566038904.868,
+ "mem_bytes": 150109880320.0,
+ "source": "measured",
+ "driver": "580.173.02",
+ "sm_clock_mhz": 1455,
+ "measured_at": "2026-10-13T17:39:50",
+ "method": "best of 3.0s: 512MiB bf16 copy; 8192^3 bf16 GEMM"
+}

knowledge/reference/inference-server/long_prompt_short_output.json

@@ -0,0 +1,56 @@
+{
+ "target": "inference-server",
+ "server": "reference",
+ "launch": "vllm serve Qwen/Qwen3-30B-A3B --dtype bfloat16 --served-model-name Qwen/Qwen3-30B-A3B --max-model-len 32768 --port {port}",
+ "env": {},
+ "regime": "long_prompt_short_output",
+ "objective": "goodput_req_per_s",
+ "value": 68.0,
+ "better": "higher",
+ "valid": true,
+ "summary": {
+  "n": 4089,
+  "ok": 4089,
+  "error": 0,
+  "silent_drop": 0,
+  "truncated": 0,
+  "crash": 0,
+  "attainment": 1.0,
+  "meets_limits": true,
+  "ttft_p50_s": 0.22769945079134501,
+  "ttft_p99_s": 0.40794480072750794,
+  "tpot_p50_s": 0.055327961015874526,
+  "tpot_p99_s": 0.06561631411111396,
+  "output_tokens": 261696,
+  "elapsed_s": 60.0,
+  "req_per_s": 68.15,
+  "good_req_per_s": 68.15,
+  "output_tok_per_s": 4361.6,
+  "client_lag_p99_s": 0.00025313885493538635,
+  "limits": {
+   "ttft_s": 2.0,
+   "tpot_s": 0.1
+  },
+  "valid": true,
+  "invalid_reasons": []
+ },
+ "fingerprint": {
+  "model": "Qwen/Qwen3-30B-A3B",
+  "w_dtype": "bf16",
+  "kv_dtype": "bf16",
+  "max_model_len": 32768,
+  "tier": "short",
+  "split": "seen",
+  "corpus_version": "81739c469a304fac4a486917dee281ae8182881ab64128030b3c6ce1a428c613",
+  "gpu": "NVIDIA H200",
+  "driver": "580.173.02",
+  "sm_clock_mhz": "1980",
+  "seed": 0
+ },
+ "shape": {
+  "prompt_tokens_mean": 5374.133491481562,
+  "prompt_tokens_p99": 6748.0,
+  "output_tokens_mean": 64.0,
+  "concurrency": 2060.467756318616
+ }
+}

knowledge/reference/inference-server/saturated.json

@@ -0,0 +1,56 @@
+{
+ "target": "inference-server",
+ "server": "reference",
+ "launch": "vllm serve Qwen/Qwen3-30B-A3B --dtype bfloat16 --served-model-name Qwen/Qwen3-30B-A3B --max-model-len 32768 --port {port}",
+ "env": {},
+ "regime": "saturated",
+ "objective": "goodput_req_per_s",
+ "value": 5.819394739012707,
+ "better": "higher",
+ "valid": true,
+ "summary": {
+  "n": 528,
+  "ok": 528,
+  "error": 0,
+  "silent_drop": 0,
+  "truncated": 0,
+  "crash": 0,
+  "attainment": 1.0,
+  "meets_limits": true,
+  "ttft_p50_s": 0.08344380399989859,
+  "ttft_p99_s": 1.2710328759999356,
+  "tpot_p50_s": 0.023719139036450395,
+  "tpot_p99_s": 0.030884681000012886,
+  "output_tokens": 402714,
+  "elapsed_s": 90.73108522099983,
+  "req_per_s": 5.819394739012707,
+  "good_req_per_s": 5.819394739012707,
+  "output_tok_per_s": 4438.544948724931,
+  "client_lag_p99_s": 0.00012458699984563282,
+  "limits": {
+   "ttft_s": 2.0,
+   "tpot_s": 0.1
+  },
+  "valid": true,
+  "invalid_reasons": []
+ },
+ "fingerprint": {
+  "model": "Qwen/Qwen3-30B-A3B",
+  "w_dtype": "bf16",
+  "kv_dtype": "bf16",
+  "max_model_len": 32768,
+  "tier": "short",
+  "split": "seen",
+  "corpus_version": "81739c469a304fac4a486917dee281ae8182881ab64128030b3c6ce1a428c613",
+  "gpu": "NVIDIA H200",
+  "driver": "580.173.02",
+  "sm_clock_mhz": "1980",
+  "seed": 0
+ },
+ "shape": {
+  "prompt_tokens_mean": 710.1108091156178,
+  "prompt_tokens_p99": 5613.0,
+  "output_tokens_mean": 761.5467279949822,
+  "concurrency": 1062.8178219140848
+ }
+}

knowledge/reference/inference-server/single_stream.json

@@ -0,0 +1,56 @@
+{
+ "target": "inference-server",
+ "server": "reference",
+ "launch": "vllm serve Qwen/Qwen3-30B-A3B --dtype bfloat16 --served-model-name Qwen/Qwen3-30B-A3B --max-model-len 32768 --port {port}",
+ "env": {},
+ "regime": "single_stream",
+ "objective": "tpot_p99_s",
+ "value": 0.00414084390815831,
+ "better": "lower",
+ "valid": true,
+ "summary": {
+  "n": 20,
+  "ok": 20,
+  "error": 0,
+  "silent_drop": 0,
+  "truncated": 0,
+  "crash": 0,
+  "attainment": 1.0,
+  "meets_limits": true,
+  "ttft_p50_s": 0.026337838000017655,
+  "ttft_p99_s": 0.07710730499991314,
+  "tpot_p50_s": 0.00402400701940755,
+  "tpot_p99_s": 0.00414084390815831,
+  "output_tokens": 15988,
+  "elapsed_s": 65.61843898899997,
+  "req_per_s": 0.3047923770840194,
+  "good_req_per_s": 0.3047923770840194,
+  "output_tok_per_s": 243.6510262409651,
+  "client_lag_p99_s": 0.00013822099992921721,
+  "limits": {
+   "ttft_s": 0.5,
+   "tpot_s": 0.03
+  },
+  "valid": true,
+  "invalid_reasons": []
+ },
+ "fingerprint": {
+  "model": "Qwen/Qwen3-30B-A3B",
+  "w_dtype": "bf16",
+  "kv_dtype": "bf16",
+  "max_model_len": 32768,
+  "tier": "short",
+  "split": "seen",
+  "corpus_version": "81739c469a304fac4a486917dee281ae8182881ab64128030b3c6ce1a428c613",
+  "gpu": "NVIDIA H200",
+  "driver": "580.173.02",
+  "sm_clock_mhz": "1980",
+  "seed": 0
+ },
+ "shape": {
+  "prompt_tokens_mean": 1377.65,
+  "prompt_tokens_p99": 3406.0,
+  "output_tokens_mean": 799.4,
+  "concurrency": 1.0
+ }
+}

lab/LEDGER.md

@@ -13,10 +13,10 @@ on write. Every record: `id`, `at`, `schema` (set by the writer), `kind`, `run`,
 | `test` | `test` | `lint`, `tests` (bools), returncode, output tails, `changed`, `scratch_left_out` |
 | `profile` | `profile` | `bundle` (ledger blob of the profiler output), `workspace_copy`, seconds |
 | `profile` | `trace`, `kernel`, `hostprof` | `bundle` (blob of the instrument's output dir), `workspace_copy`, `instrument` (resolved binary), `argv` (every command run), returncode, seconds, output tail |
-| `bench` | `bench` | `config: {split: seen, tier, seed}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `regimes` (full per-regime results), `ready_s`, `artifacts` |
+| `bench` | `bench` | `config: {split: seen, tier, seed}`; `verdict: ok`, `metrics: {regime: {objective, value, better, valid, invalid_reasons}}`, `calibrated: {regime: {value, ceiling, pct_of_ceiling, bound, ceiling_basis, peaks_source, shape, reference, pct_of_reference, fingerprint}}` (a missing ceiling or reference comes with `ceiling_missing` / `reference_missing` saying why), `regimes` (full per-regime results), `ready_s`, `gpu`, `artifacts` |
 | `equiv` | `equiv` | `config: {split: seen, tier}`; `verdict: pass\|fail\|inconclusive`, `passed`, `reasons`, `gates`, `metrics` (accuracy with CI, length, consistency, divergence, per-task flips, unanswered), `thresholds`, `artifacts` |
 | `submit` | `submit` | **held-out shape**: no `result`; top-level `config: {split: heldout, tier: full}` and `metrics: {regime: {base, new, delta_pct, band_pct, verdict}}`, scalars only. The ledger refuses anything else that mentions the held-out split. |
-| `baseline` | the loop, once per run before the first session | `config: {split: seen, tier: short, commit}`; `verdict: ok` with `metrics` shaped like `bench` (the base commit under bench's default regimes: the task's objective and constrained ones, from the cache submit compares against), or `verdict: error`, `reason` |
+| `baseline` | the loop, once per run before the first session | `config: {split: seen, tier: short, commit}`; `verdict: ok` with `metrics` and `calibrated` shaped like `bench`'s (the base commit under bench's default regimes: the task's objective and constrained ones, from the cache submit compares against), or `verdict: error`, `reason` |
 | `finding` | `lab.ledger seed` | a `knowledge/` entry, whole, as `claim`; `source`, `content_sha` |
 | `note` | `note`, `restore`, refusals | free text, or `{verdict: refused, reason}` |
 

lab/README.md

@@ -105,15 +105,15 @@ with the referee's rights, snapshot the workspace and write the ledger on every
 | tool | does |
 |---|---|
 | `test` | lint and the fast suite on a pristine two-commit copy of the workspace, jailed |
-| `profile` | `lab.profile` on the pristine copy, jailed, on `args.corpus_class` (default `steady_interactive`, seen split, `args.seed`) or `args.synthetic`; the bundle goes into the ledger as a blob |
+| `profile` | `lab.profile` on the pristine copy, jailed, on `args.corpus_class` (default `steady_interactive`, seen split, `args.seed`) or `args.synthetic`; the bundle goes into the ledger as a blob, with each kernel's share of the window (the Amdahl denominator) in the record |
 | `trace` | `nsys profile` (`args.nsys_args`, default window-only CUDA/NVTX/OS-runtime with CUDA-graph nodes) around `lab.profile --profiler cuda-range` (window only), then `nsys stats` (kernel, CUDA API, memcpy, NVTX tables as CSV), both jailed; `.nsys-rep` + CSVs as a blob |
 | `kernel` | `ncu [--kernel-name regex:<kernel_regex>] --launch-skip N --launch-count N --set <any set> [ncu_args]` around the same workload, then `ncu --import --csv --page raw`; `.ncu-rep` + `metrics.csv` as a blob |
 | `hostprof` | `lab.profile --profiler pyspy`: py-spy record (speedscope, plus `args.pyspy_args`) for `seconds` under the repeated workload plus one `py-spy dump`; the bundle as a blob |
 | `ledger` | read records (this run and earlier ones) |
 | `budget` | dollars left |
 | `restore` | workspace back to a snapshot id (`base` resets) |
 | `note` | a note for the human; recorded, changes nothing |
-| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`, `args.seed`) |
+| `bench` | serve the pristine copy and run the load regimes (`regimes/`) on the seen split, short tier; one headline per regime, raw (`args.regimes`, `args.tier`, `args.seed`), each with its context (`lab/calibrate.py`: % of the physical ceiling, % of the reference engine measured the same way; only under a matching config fingerprint) |
 | `equiv` | serve it and run the correctness gate (`correctness/`) against the target's reference outputs; pass, fail or inconclusive with every metric; `tier=full` uses `<reference.dir>-full` |
 | `submit` | the only thing that can produce a win. Needs a passing full-tier equiv and a seen bench on this snapshot. Measures the held-out split at the full tier, measures the base commit the same way (once per run, cached outside the jail), and records one aggregate per regime (base, new, delta %, noise band %, verdict improved/regressed/within_band/unknown_band) through the Thresholdout guard (`validity/holdout.py`, state in `lab/ledger/holdout_state.json`, seed `LAB_HOLDOUT_SEED`): the held-out numbers themselves never reach the ledger. Noise bands come from `knowledge/noise/<regime>.json` when measured |
 

lab/artifacts.py

@@ -19,6 +19,11 @@ def write_rows(work: Path, rows: list[dict]) -> None:
     (Path(work) / ROWS).write_text("".join(json.dumps(r) + "\n" for r in rows))
 
 
+def read_rows(work: Path) -> list[dict]:
+    p = Path(work) / ROWS
+    return [json.loads(x) for x in p.read_text().splitlines() if x.strip()] if p.exists() else []
+
+
 def store(work: Path, root: Path) -> dict:
     """`work`'s passive data as ledger blobs, {device, telemetry, client_rows, serve_log} (None when absent); `work`
     is removed."""