Skip to content

lab: request-trace contract (TRACE.md) and validator - #13

Merged
AdvayMonga merged 1 commit into
mainfrom
feat/trace-contract
Oct 8, 2026
Merged

AdvayMonga merged 1 commit into
mainfrom
feat/trace-contract

Conversation

@AdvayMonga

@AdvayMonga AdvayMonga commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Defines an engine-agnostic request-trace contract: what an engine reports for each request so the researcher can join it with the client's rows. These rows are diagnostics only and are never scored.

  • lab/TRACE.md is the spec. It covers:
    • Layout: a directory of *.sqlite files (tables requests + meta) or *.jsonl files with a <stem>.meta.json.
    • Required fields:
      • trace_id: the client's X-Trace-Id, verbatim
      • arrival_ts: epoch seconds
      • total_s, ttft_s, tpot_s: durations on a monotonic clock, with TPOT = (E2E − TTFT)/(n − 1)
      • schema_version, terminal_state, prompt_tokens, tokens_out
    • Optional typed fields.
    • Meta: schema_version, rows_written, rows_dropped, closed. Drops are never silent.
    • A short section mapping vLLM's OpenTelemetry spans to the contract (no code).
  • lab/trace_contract.py is the validator: check_row, validate(dir), and python -m lab.trace_contract DIR. It rejects:
    • a missing or empty trace_id
    • arrival_ts values that are monotonic readings, epoch ms or epoch ns
    • mixed units: TTFT greater than E2E, queue + prefill greater than TTFT, negative or day-long durations
    • TTFT set when there are no tokens (or missing when there are)
    • a TPOT that doesn't match the definition
    • wrong types, unknown states, and missing or inconsistent meta
  • targets/inference-server.toml gets [engine.telemetry] dir_env = "TELEMETRY_DIR". No tool reads it yet.
  • inference-server conforms natively once telemetry: clock rule, every terminal path, real cache hits, schema + drop counts inference-server#102 is merged, so no adapter is needed. test_inference_server_rows_conform checks this with the engine's own writer and skips on older engines.

Tests: LAB_NO_JAIL=1 pytest -q: 169 passed, 5 skipped. ruff check .: clean. With the engine's python and LAB_ENGINE_REPO set to the #102 branch, the conformance test passes too (27/27).

🤖 Generated with Claude Code

Rebased last in the stack; also carries the cross-branch wiring and review fixes from #15: one [engine.telemetry] env form, timeline in its own subdir, contract check on collected telemetry, X-Trace-Id from the correctness client, meta.profiler always {kind,...}, symlinks dropped from jailed tool output before it is copied out, baseline GPU time charged.

An engine's per-request rows, engine-agnostic: trace_id = the client's X-Trace-Id, arrival_ts epoch
seconds, *_s monotonic durations, TPOT = (E2E - TTFT)/(n - 1), terminal states, schema version, and
drop counts in a meta table/file. A dir of *.sqlite (requests + meta) or *.jsonl + <stem>.meta.json;
python -m lab.trace_contract DIR. inference-server's TELEMETRY_DIR conforms natively from
inference-server#102; targets name the env var in [engine.telemetry] dir_env. vLLM OTel mapping noted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AdvayMonga
AdvayMonga force-pushed the feat/trace-contract branch from a3930a7 to 0e0d2cc Compare October 8, 2026 22:44
@AdvayMonga
AdvayMonga merged commit 167c220 into main Oct 8, 2026
3 checks passed
@AdvayMonga
AdvayMonga deleted the feat/trace-contract branch October 8, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant