Repository navigation
Observability batch: bundle fixes, passive data, profiler tools, prompt+baseline, trace contract (integrated) - #15
Closed
AdvayMonga wants to merge 11 commits into
Closed
AdvayMonga wants to merge 11 commits into
AdvayMonga wants to merge 11 commits into
Conversation
… tables - warmup sized to the window (default = window requests, fired together, random ids at the window's prompt lengths so no prefix-cache hits); settings recorded in meta - meta: profiler_on + profiler config, GPU snapshot at window start and end, timeline events written/dropped, per-request outcomes (errors, empty, short, full) and token counts; empty responses count as failed - memory.json: CUDA peaks reset at window start; window_peak and lifetime_peak named apart - stats.json: scheduler stats at window start/end plus counter deltas - gpu: throttle reasons (clocks_event_reasons / clocks_throttle_reasons fallback, null if unsupported), application clocks; gpu.csv gets an epoch_s column - ops.json from key_averages, kernels.json for CUDA kernels by name - lab.profile takes --corpus-class/--workload/--prompts/--synthetic (one required); the profile tool samples a seen corpus class outside the jail (lab.corpus.sample) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each served run keeps device samples (DCGM profiling fields if dcgmi, else
nvidia-smi with throttle reasons; epoch-stamped), engine-native telemetry
from the target's optional [engine.telemetry] env ({dir} in the served
tree), every client row without text, and the serve log. Bench and equiv
store them as ledger blobs under result.artifacts; submit keeps them only
in <run>/heldout-private/, never in the ledger. lab/artifacts.py loads a
record's artifacts and joins client rows to engine rows by trace id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ession - lab/prompts/system.md is now a template filled from the target spec and the ToolSpec descriptions (session.system_prompt): target, write surface, each tool with what it does, returns and costs, budget mechanics, the validity rule, how a session ends. No advice. - The run measures its base commit once, before the first session, under bench's default regimes (seen, short), in the same cache submit compares against; recorded as a new `baseline` ledger kind. RunConfig.baseline=False skips it. - The brief: task, budget, referee, baseline (or why it could not be measured), and only the run's last session record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An engine's per-request rows, engine-agnostic: trace_id = the client's X-Trace-Id, arrival_ts epoch seconds, *_s monotonic durations, TPOT = (E2E - TTFT)/(n - 1), terminal states, schema version, and drop counts in a meta table/file. A dir of *.sqlite (requests + meta) or *.jsonl + <stem>.meta.json; python -m lab.trace_contract DIR. inference-server's TELEMETRY_DIR conforms natively from inference-server#102; targets name the env var in [engine.telemetry] dir_env. vLLM OTel mapping noted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… every tool Every tool record's cost now carries its wall seconds; GPU tools (profile, trace, kernel, hostprof, bench, equiv, submit) charge them at LAB_GPU_USD_PER_HOUR or the target's [cost] gpu_usd_per_hour to the run's budget (0 with rate_known false when neither is set), and are refused once the budget is spent. Tool descriptions state each tool's typical cost; lab/README.md has the table. New tools wrap lab.profile's workload in the jail with validated arguments: trace (nsys, window marked with cudaProfilerStart/Stop, plus nsys stats CSVs), kernel (ncu on a validated kernel-name regex with launch skip/count and basic|full set, plus the raw CSV), hostprof (py-spy record + dump on the workload process for N seconds). Each refuses cleanly without its binary and stores its output as a ledger blob. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ervability-integration # Conflicts: # lab/gpu.py
…ent tools trace/kernel/hostprof called lab.profile with no workload source, which bundle-fixes made mandatory; they now stage the same seen-split corpus sample (or --synthetic) as profile, via tools.workload_flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…observability-integration # Conflicts: # lab/evaltools.py # lab/tools.py # tests/test_lab_evaltools.py
…engine repo has no corpus) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One telemetry form, [engine.telemetry] env with {dir}; timeline in its own subdir.
- artifacts.store validates collected rows against lab/TRACE.md (telemetry/contract.json).
- Correctness client sends X-Trace-Id (eq-<item>) so equiv rows join engine rows.
- profile meta.profiler is always {kind, ...}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine's GPU time Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates #10, #11, #14, #12, #13 (in that order) with the cross-branch fixes they needed together. Merge this instead of the five, or merge them in that order and drop this.
Cross-branch fixes:
lab.profile(sharedtools.workload_flags).[engine.telemetry] envwith{dir}; timeline goes to its own subdir so events aren't read as request rows.lab/TRACE.md(telemetry/contract.json).X-Trace-Idso equiv client rows join engine rows.meta.profileris always{kind, ...}.Tests: lab suite 216 passed / 5 skipped; engine-dependent suites (against inference-server #102) 42 passed; ruff clean. Engine-side contract is inference-server #102, merge that first.
GPU validation of the profiler tools (~2 h, ~$9, 3 h cap) is not run; awaits approval.
🤖 Generated with Claude Code