Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

## Evidence
<!-- A change under src/inference_server/ is measured, not argued: link the profile bundle,
the benchmark under knowledge/evidence/, or the ledger record, and say what config and
the benchmark in BlameGraph `knowledge/evidence/`, or the ledger record, and say what config and
machine it ran on. Touched no engine file? Say so. -->


## Checks
- [ ] `pytest -q` and `ruff check .` pass locally
- [ ] models/ change: `scripts/gpu_tests/checks.py` run on the lab's GPU VM
- [ ] models/ change: `scripts/gpu_tests/checks.py` run on the environment's GPU VM (BlameGraph `python -m lab.vm`)
76 changes: 6 additions & 70 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ on:
branches: [main]
pull_request:
workflow_dispatch:
schedule:
- cron: "0 7 * * 1" # Mondays 07:00 UTC: corpus provenance, which nothing else exercises

permissions:
contents: read
Expand All @@ -17,43 +15,9 @@ concurrency:
cancel-in-progress: true

jobs:
# Does this change touch the corpus or its builder? Everything else always runs the engine lane.
changes:
runs-on: ubuntu-latest
outputs:
corpus: ${{ steps.classify.outputs.corpus }}
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- id: classify
env:
BASE_REF: ${{ github.base_ref }}
BEFORE: ${{ github.event.before }}
EVENT: ${{ github.event_name }}
run: |
set -eu
if [ "$EVENT" = "workflow_dispatch" ] || [ "$EVENT" = "schedule" ]; then
echo "corpus=true" >> "$GITHUB_OUTPUT"; exit 0
fi
range=""
if [ "$EVENT" = "pull_request" ]; then
range="origin/$BASE_REF...HEAD"
elif [ "$EVENT" = "push" ] && git cat-file -e "$BEFORE^{commit}" 2>/dev/null; then
range="$BEFORE...HEAD"
fi
corpus=false
if [ -z "$range" ] || ! git diff --name-only "$range" > changed.txt; then
corpus=true # no diffable range: run everything
elif grep -qE '^(corpus/|scripts/corpus/|lab/corpus\.py|tests/test_lab_corpus\.py)' changed.txt; then
corpus=true
fi
echo "corpus=$corpus" >> "$GITHUB_OUTPUT"
echo "corpus lane: $corpus"

# Lint and the fast suite: engine, lab, control plane. CPU only; model-heavy tests are
# deselected by tests/conftest.py and no kernel runs (Triton compiles only on CUDA; the lab's
# GPU VM is where kernels get verified, by hand, before a models/ change merges).
# Lint and the fast suite: engine and control plane. CPU only; model-heavy tests are
# deselected by tests/conftest.py and no kernel runs (Triton compiles only on CUDA; the
# environment's GPU VM is where kernels get verified, by hand, before a models/ change merges).
engine:
runs-on: ubuntu-latest
timeout-minutes: 15
Expand Down Expand Up @@ -82,44 +46,16 @@ jobs:
- name: Tests
run: pytest -q

# Provenance of corpus/: rebuilds it from the pinned raw traces and asserts the committed bytes.
corpus:
needs: changes
if: always() && needs.changes.result == 'success' && needs.changes.outputs.corpus == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v7
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
with:
python-version: "3.11"
enable-cache: true
cache-dependency-glob: uv.lock
activate-environment: true
- name: Install deps
run: uv sync --locked --extra dev --extra corpus
- name: Cache raw traces
uses: actions/cache@v6
with:
path: ~/.cache/inference-server/traces
key: traces-${{ hashFiles('scripts/corpus/fetch_traces.py') }}
- name: Fetch and verify raw traces
run: python scripts/corpus/fetch_traces.py
- name: Corpus provenance tests
env:
CORPUS_REQUIRE_RAW: "1"
run: pytest -q tests/test_lab_corpus.py

# The one check branch protection requires. A skipped lane is fine; failed or cancelled is not.
ci-ok:
needs: [changes, engine, corpus]
needs: [engine]
if: always()
runs-on: ubuntu-latest
steps:
- env:
RESULTS: ${{ needs.changes.result }} ${{ needs.engine.result }} ${{ needs.corpus.result }}
RESULTS: ${{ needs.engine.result }}
run: |
echo "changes/engine/corpus: $RESULTS"
echo "engine: $RESULTS"
case "$RESULTS" in
*failure*|*cancelled*) echo "a lane did not pass"; exit 1 ;;
esac
Expand Down
5 changes: 0 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ Thumbs.db
*.pt
*.onnx

# Benchmark outputs
knowledge/evidence/benchmarks/results/

# Jupyter
.ipynb_checkpoints/

Expand All @@ -54,5 +51,3 @@ notes/
# Research loop: raw run panels (regenerable, large)
runs/

# lab ledger: live, not committed for now
/lab/ledger/
38 changes: 12 additions & 26 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Plans are hypotheses; the vision changes as we learn.
- If something written conflicts with what I said recently, follow me and flag the stale source.
Don't quote old plans back at me as a reason not to change course.
- Exception: measured findings (`knowledge/`) stand until re-measured. If a new direction
- Exception: measured findings (BlameGraph `knowledge/`) stand until re-measured. If a new direction
runs into one, raise it once, then do what I decide.

## How we discuss
Expand Down Expand Up @@ -34,30 +34,16 @@
hard rules; a bad idea is what the lab is for.
- When we design the harness, flag anything that drifts from referee into strategy.

## The lab
- `lab/` is the open environment (design: `ENVIRONMENT.md`). The research loop it replaced was
removed on 2026-10-02; tag `archive/research-loop` keeps it. `python -m lab.profile` writes a
raw measurement bundle per run.
- The workload corpus (`corpus/`) is built from real public traces: BurstGPT arrivals and
sessions, WildChat-1M conversations with their real assistant replies, four classes
(`cold_start`, `steady_interactive`, `long_context`, `spike`) split seen/heldout by alternating
trace weeks. `scripts/corpus/fetch_traces.py` + `build_corpus.py` rebuild it; the loader is
`lab/corpus.py`; any change is a new `corpus_version`. See `corpus/README.md`.
- `python -m lab.session` is the runtime: `lab/session.py` loops sessions over a dollar budget
(`lab/budget.py`), `lab/agent.py` is the one place a model is called (Claude via the Agent SDK,
provider seam), `lab/tools.py` the agent's metered tools (each snapshots the workspace and writes
the ledger), `lab/workspace.py` the exported engine copy with content-addressed snapshots, and
`lab/safety/` the referee (surfaces, jail, grader). Lab tools run outside the jail; the agent's
own shell and file tools run inside it. `LAB_NO_JAIL=1` waives the jail (tests, trusted box).
- `python -m lab.canary <names>` serves the engine with deliberate regressions patched in from
outside, for proving the eval harness's coverage. `/v1/completions` accepts token-id prompts and
vLLM's `prompt_logprobs: k` (teacher-forced scoring via `CustomTorchBackend.score_logprobs`),
the output-equivalence surface the harness uses.
- `python -m lab.vm` drives one persistent GPU VM (`LAB_VM_*` env) on `lab/providers/` verda
(default, 1x H100 SXM) or crusoe (1x A100): start, setup, run a command on the pushed tree and
fetch outputs, stop (Verda: hibernate). Credentials stay in env or the provider CLI's config.
- `lab/ledger.py` is the raw knowledge base: append-only JSONL, one record per tool call, written by
tools only; `claim` holds the agent's untrusted words; held-out records carry one aggregate per metric.
`python -m lab.ledger seed` imports `knowledge/` as `finding` records. `lab/ledger/` is gitignored.
## This repo is the engine; the environment is BlameGraph
- This repo holds only the engine and its stack: `src/` (engine, control plane), its tests, `monitoring/`,
and engine tools in `scripts/` (Triton launch tuning, CUDA parity checks, smoke). The agent's workspace is
exported from here; it may write `src/inference_server/` and add `tests/test_*.py`, nothing else.
- Everything that runs, measures or judges the engine lives in github.com/AdvayMonga/BlameGraph (split
2026-10-06; tag `archive/pre-split` here holds the combined tree): the lab (session loop, agent, tools,
ledger, workspace, sandbox/grader, budget), the GPU VMs (`python -m lab.vm`, Verda/Nebius/Crusoe), the
workload corpus, `knowledge/`, load regimes, the correctness gate, canaries and the referee. It finds this
repo via `LAB_ENGINE_REPO` (default `../inference-server`) and runs its code with `.venv/bin/python`.
- `/v1/completions` accepts token-id prompts and vLLM's `prompt_logprobs: k` (teacher-forced scoring via
`CustomTorchBackend.score_logprobs`), the output-equivalence surface the environment uses.
- `TIMELINE_DIR` turns on the engine event timeline (`timeline.py`): one JSONL event per scheduler
decision and a profiler range per phase, both keyed by step id. Off by default, no-op when off.
18 changes: 7 additions & 11 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing

The engine is measured, not argued about. The lab (`ENVIRONMENT.md`) is where that happens.
The engine is measured, not argued about. The environment ([BlameGraph](https://github.com/AdvayMonga/BlameGraph)) is where that happens.

## Setup

Expand All @@ -10,7 +10,7 @@ git config core.hooksPath scripts/hooks # pre-push runs CI's lint + tests in
```

`uv.lock` pins every dependency; Linux takes the CPU torch wheel, macOS the PyPI one (MPS).
The GPU VM (`python -m lab.vm setup`) installs the CUDA wheel at the same locked version.
The GPU VM (BlameGraph's `python -m lab.vm setup`) installs the CUDA wheel at the same locked version.

The hook needs the venv on `PATH` and says so rather than passing silently. `git push
--no-verify` or `SKIP_HOOKS=1 git push` when you mean it.
Expand All @@ -26,21 +26,17 @@ agent get it twice.

## CI

`ci-ok` is the only required check. Two lanes feed it:

| lane | runs | what |
|---|---|---|
| `engine` | always | ruff and the full fast suite: engine, control plane, lab. CPU only; model-heavy tests are deselected and no kernel runs. |
| `corpus` | when `corpus/`, `scripts/corpus/`, `lab/corpus.py` or its test change; weekly | rebuilds the corpus from the pinned raw traces and asserts the committed bytes. |
`ci-ok` is the only required check. One lane feeds it: `engine`, ruff and the full fast suite (engine and
control plane). CPU only; model-heavy tests are deselected and no kernel runs.

Triton compiles only on CUDA, so no lane can execute a kernel. Before merging anything under
`src/inference_server/models/`, run the checks in `scripts/gpu_tests/checks.py` on the lab's
GPU VM: `python -m lab.vm run -- python scripts/gpu_tests/checks.py`.
`src/inference_server/models/`, run the checks in `scripts/gpu_tests/checks.py` on the environment's
GPU VM, from BlameGraph: `python -m lab.vm run -- python scripts/gpu_tests/checks.py`.

## Evidence for an engine change

Today: a PR with the measurement behind it (a profile bundle, a benchmark CSV under
`knowledge/evidence/`, or a ledger record) and a reviewer who reads it. The lab's `submit` tool
BlameGraph's `knowledge/evidence/`, or a ledger record) and a reviewer who reads it. The lab's `submit` tool
will make this mechanical once the eval harness is wired; until then the number in the PR body
is the claim and the reviewer is the gate.

Expand Down
Loading