Feat/metrics

.dockerignore

@@ -19,6 +19,5 @@ data/
 # committed and later removed from it, into an image layer.
 .git/
 sdk/
-tests/
 *.log
 *_out.ipynb

.env.example

@@ -137,3 +137,8 @@ ASAREE_OKF_BUNDLE_ROOT=~
 # not in core: core has no way to know how slow any given product's tools or
 # provider responses are.
 ASAREE_HOOK_TIMEOUT_SECONDS=3600
+
+# Administrative limits for independent post-run metric producers. These are
+# server policy, not per-metric controls exposed in measurement plans.
+ASAREE_METRIC_PRODUCER_MAX_CONCURRENCY=4
+ASAREE_METRIC_PRODUCER_TIMEOUT_SECONDS=30

.gitignore

@@ -15,3 +15,7 @@ backups/
 # Generated by hatch-vcs at build time from the git tag (see pyproject.toml).
 src/asaree/_version.py
 sdk/src/asaree_client/_version.py
+
+# Private local planning and agent guidance
+docs/
+issues/

CLAUDE.md

@@ -43,6 +43,20 @@ or introduce a different aesthetic without being asked.
   leaving the canvas ≥420px) and remembers its width in `localStorage` — so a new tab in it
   should be width-responsive via container queries, not built for one fixed column width.
 
+## Agent skills
+
+### Issue tracker
+
+Implementation issues and specs live in the separate `EpistasisLab/ASAREE_Issues` GitHub repository. See `docs/agents/issue-tracker.md`.
+
+### Triage labels
+
+Use the five canonical Matt Pocock triage labels. See `docs/agents/triage-labels.md`.
+
+### Domain docs
+
+This is a single-context repository with `CONTEXT.md` at the root and private local ADRs under `docs/adr/`. See `docs/agents/domain.md`.
+
 # Git commit conventions
 
 Do not add `Co-Authored-By` or `Generated-with` lines to commits or PRs.

CONTEXT.md

@@ -0,0 +1,106 @@
+# ASAREE
+
+ASAREE runs factorial experiments over agent protocols: a user declares factors,
+metrics, and an agent protocol graph on a canvas; the harness executes every
+factor-combination cell and compares the recorded outcomes.
+
+## Language
+
+### Experiments
+
+**Experiment**:
+A factorial study comparing agent protocol conditions. Owns the design declaration, the protocol canvas, and all runs.
+_Avoid_: Project, study, trial
+
+**Factor**:
+A declared experimental variable with discrete levels, bound to at least one canvas field before cells can run.
+_Avoid_: Parameter, variable
+
+**Cell**:
+One unique factor combination together with all its planned replicates.
+_Avoid_: Condition, treatment (accepted in prose, never in code)
+
+**Replicate**:
+One run of one cell. Counts of pending/run/scored are replicate counts, never cell counts.
+
+**Primary metric**:
+The optional declared metric Results uses to rank cells. An editable experiment may have none; when declared, it must be unique.
+_Avoid_: Main metric, target metric
+
+**Design revision**:
+An immutable snapshot of the generated design. The current design is the one revision with `superseded_at IS NULL`.
+_Avoid_: Design version
+
+### Metrics
+
+**Measurement plan**:
+The experiment's declared set of metrics, their producers, and the bindings that provide each producer's inputs.
+_Avoid_: Metric list, evaluation config
+
+**Metric**:
+A scalar experimental outcome that can be observed per replicate and aggregated across a cell.
+_Avoid_: Measurement, score, result field
+
+**Built-in metric**:
+A metric whose definition and producer ASAREE supplies as one supported capability.
+_Avoid_: Catalog metric, preset metric
+
+**Custom metric**:
+An experiment-owned metric whose definition or producer configuration is supplied by the user.
+_Avoid_: User metric, freeform metric
+
+**Metric producer**:
+The declared mechanism that observes a metric from a completed replicate. A producer may emit several metrics and evaluation artifacts from one evaluation.
+_Avoid_: Calculator, metric tool, extraction rule
+
+**Runtime producer**:
+A metric producer that observes execution facts recorded by the harness, such as cost, tokens, duration, or tool calls.
+_Avoid_: Telemetry metric, process metric
+
+**Deterministic evaluator**:
+A metric producer that applies a reproducible evaluation procedure to declared run outputs and reference inputs.
+_Avoid_: Score agent, metric script
+
+**Model judge**:
+A metric producer that applies a declared rubric to run output using a selected language model.
+_Avoid_: Critic, evaluator agent
+
+**Reported metric**:
+A display/export-only custom metric containing either a configured Agent's completed
+final output or the opaque result of its last matching Script or MCP tool call. ASAREE
+never invokes the source to produce it, validates no result type, and leaves the value
+absent when the Agent did not complete or call the configured tool.
+_Avoid_: Agent metric, parsed metric
+
+**Metric observation**:
+One producer's outcome for one metric on one replicate, including whether it was measured, unavailable, failed, timed out, cancelled, or not applicable.
+_Avoid_: Metric value, score record
+
+**Evaluation artifact**:
+A structured diagnostic such as a confusion matrix, calibration curve, or per-class report that accompanies observations but is not ranked or aggregated as a metric.
+_Avoid_: Array metric, report metric
+
+**Legacy value**:
+A historical raw value outside the canonical aggregatable metric contract whose original producer was not recorded. It stays visible with `legacy.unknown` provenance but is not ranked; ASAREE never infers its producer from a display name, prompt, or tool call.
+_Avoid_: Inferred observation, promoted score
+
+**Context metric**:
+A metric an Agent inspector includes in that agent's system prompt as evaluation guidance; it never exposes a future observation.
+_Avoid_: Prompt metric
+
+### Runs
+
+**Protocol canvas**:
+The draft editable graph. Production runs never read it directly.
+_Avoid_: Workflow, pipeline
+
+**Published revision**:
+The immutable protocol snapshot a run executes; created by publishing the canvas.
+_Avoid_: Protocol version
+
+**Gated pair**:
+An agent node with its critic gate. `approved` and `revisions_used` are recorded on the worker's node-run.
+_Avoid_: Reviewer, supervisor
+
+**Attempt**:
+One ProtocolRun for a replicate; later attempts supersede earlier ones but every attempt's own facts stay immutable.

Dockerfile

@@ -14,7 +14,7 @@
 # python -m ...`, the same subprocess convention used in every dev environment
 # so far. Removing uv here would just move the same "no uv at runtime" problem
 # one level down.
-FROM python:3.13-slim
+FROM python:3.13-slim AS application
 
 WORKDIR /app
 
@@ -111,3 +111,25 @@ ENV PATH="/app/.venv/bin:$PATH"
 EXPOSE 8000
 
 CMD ["uvicorn", "asaree.app:app", "--host", "0.0.0.0", "--port", "8000"]
+
+# Opt-in test image used by the asaree-tests Compose service. Keep pytest and
+# the test tree out of the production image while reusing every application
+# dependency layer above. The git context is required for the same hatch-vcs
+# reason as the application install.
+FROM application AS test
+
+COPY tests/ ./tests/
+# One backend contract test compares the mirrored TypeScript catalog with the
+# Python source of truth. Keep the test image narrow while making that source
+# available at the same repository-relative path used outside containers.
+COPY frontend/src/lib/metricCatalog.ts ./frontend/src/lib/metricCatalog.ts
+RUN --mount=type=bind,from=gitdir,target=/app/.git \
+    --mount=type=cache,target=/root/.cache/uv,sharing=locked \
+    uv sync --frozen --group dev
+
+CMD ["pytest", "tests/", "-q", "--tb=short"]
+
+# Keep the default build target as the production application. Merely adding
+# the test stage must not put test-only dependencies or source in deployed
+# images built without an explicit target.
+FROM application AS runtime

README.md

@@ -1,5 +1,10 @@
 # ASAREE
 
+[![Latest release](https://img.shields.io/github/v/release/EpistasisLab/ASAREE?display_name=tag&sort=semver)](https://github.com/EpistasisLab/ASAREE/releases/latest)
+[![CI](https://github.com/EpistasisLab/ASAREE/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/EpistasisLab/ASAREE/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/Python-%E2%89%A53.12-3776AB?logo=python&logoColor=white)](https://github.com/EpistasisLab/ASAREE/blob/main/pyproject.toml)
+[![License](https://img.shields.io/github/license/EpistasisLab/ASAREE)](https://github.com/EpistasisLab/ASAREE/blob/main/LICENSE)
+
 **A**nalytical **S**andbox for **A**gentic **R**esearch, **E**ngineering, and
 **E**xperimentation — a workbench for running LLM agents as designed
 experiments rather than one-off prompts.
@@ -86,6 +91,31 @@ docker compose up -d    # picks up the changed .env
 — five agents in series building a classifier on a public dataset. It picks up
 exactly where this step leaves off.
 
+## Measurements and Results
+
+The Design panel offers built-in runtime and supervised-ML metrics with their
+producer, required inputs, units, aggregation, direction, and current readiness.
+Custom metrics can read a typed numeric/Boolean structured-output field or use a
+model judge with an explicit model, rubric, bounds, and selected run inputs. A
+judge is an extra model call on every replicate, so its recurring call and cost
+estimate is shown before save. The optional wizard creates the same typed draft
+as the manual editor and cannot bypass the normal validator.
+
+Every completed attempt records one state for every declared metric:
+`measured`, `unavailable` (the required value was absent), `failed` (the producer
+could not evaluate it), or `not_applicable` (the metric does not apply to that
+task). Results keeps those states distinct. Confusion matrices, calibration
+data, and per-class reports are evaluation artifacts: they appear in attempt
+detail but are never ranked or averaged as scalar metrics.
+
+The Results CSV exports scalar values as ordinary analysis columns. Its
+`observation_statuses` JSON column retains each metric's state and error, and
+`evaluation_artifacts` retains structured diagnostics without flattening them
+into misleading scalar columns. The `legacy_values` JSON column keeps old text
+or other non-rankable values with `legacy.unknown` provenance when their
+original producer was never recorded; choose an explicit producer before
+running that legacy declaration again.
+
 ## Everyday commands
 
 ```bash
@@ -95,6 +125,19 @@ docker compose restart asaree-app     # apply an edited .env
 docker compose down                   # stop, keep all data
 ```
 
+Run the backend test suite in Compose when you do not already have the dev
+database exposed to the host. The one-shot runner waits for both migration
+chains and connects to Postgres over the internal Compose network, so it does
+not depend on `localhost`, `POSTGRES_PORT`, or a host PostgreSQL installation:
+
+```bash
+docker compose run --rm --build asaree-tests
+```
+
+The tests themselves still run under pytest; only pytest and its real Postgres
+dependency are placed on the same network. The `test` profile keeps this
+service out of a normal `docker compose up`.
+
 The frontend hot-reloads from your checkout; backend changes need a rebuild.
 
 `docker compose up -d --build` also runs pending database migrations. The

compose.yml

@@ -218,6 +218,32 @@ services:
       - ./data:/app/data
     restart: unless-stopped
 
+  # Integration-test runner on the Compose network. The tests intentionally
+  # use real Postgres (JSONB and core's vector schema rule out SQLite), so run
+  # them beside the database instead of relying on a host-side localhost port.
+  # The profile keeps this one-shot service out of ordinary `docker compose
+  # up`; targeting it explicitly activates it:
+  #
+  #   docker compose run --rm --build asaree-tests
+  asaree-tests:
+    profiles: ["test"]
+    build:
+      context: .
+      target: test
+      secrets:
+        - gh_token
+      additional_contexts:
+        gitdir: ./.git
+    depends_on:
+      asaree-migrate:
+        condition: service_completed_successfully
+    environment:
+      DATABASE_URL: postgresql+asyncpg://agentic:agentic@motoro-postgres:5432/motoro
+      ASAREE_DATABASE_URL: postgresql+asyncpg://agentic:agentic@motoro-postgres:5432/motoro
+      ASAREE_PRODUCT_DATABASE_URL: postgresql+asyncpg://agentic:agentic@motoro-postgres:5432/asaree
+      ASAREE_REDIS_URL: redis://motoro-redis:6379/0
+    restart: "no"
+
   asaree-frontend:
     build:
       context: ./frontend

docs/adr/0001-measurements-have-explicit-producers.md

@@ -0,0 +1,17 @@
+# Built-in metrics are evaluated; custom metrics are reported
+
+ASAREE distinguishes two measurement paths.
+
+Built-in metrics are owned by ASAREE. Their runtime producer reads the immutable attempt snapshot during finalization, computes supported numeric or Boolean observations, and supplies the values used for aggregation, comparison, and ranking.
+
+Custom metrics are owned by the experiment author. A custom declaration points to an Agent, or to a Script/MCP tool directly connected to that Agent. ASAREE does not execute that source as a post-run evaluator. It captures the Agent's completed final output or the result of the last matching tool call already recorded in the Agent's execution trace. The last call is used whether it succeeded or failed.
+
+## Consequences
+
+- Users are responsible for prompts, scripts, MCP tools, and Agent behavior that produce the value they want to record.
+- Custom values are opaque JSON. ASAREE does not coerce, validate, aggregate, rank, or choose a primary custom metric.
+- If the Agent never produces the configured output or call, the observation is `unavailable` and its CSV cell is empty.
+- Test Runs, per-node Play, production Results, and CSV export all consume the same persisted observation shape.
+- Every observation retains producer provenance and a status. A measured JSON `null` remains distinguishable from an unavailable observation through that status and is exported as the literal `null`.
+- Script and MCP configuration is validated for source identity, enabled state, and direct Agent wiring. Evaluator input mappings, result paths, deterministic execution, and side-effect acknowledgements are not part of custom measurement.
+- Historical `deterministic_evaluator` bindings are normalized at the persistence boundary to reported bindings; no runtime adapter exists for them.