diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f5e0c6a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,228 @@ +# Frontend visual style + +The frontend (`frontend/`) uses a deliberate dark, retrofuturist/high-tech visual language. +Preserve it for all new pages and components — don't revert to shadcn's plain default theme +or introduce a different aesthetic without being asked. + +- **Theme**: dark mode only (`` in `index.html`), a graphite background + (not near-black) with a cyan/electric-blue accent (`--primary`). Tokens live in + `frontend/src/index.css`'s `.dark` block — adjust values there, don't hardcode colors in + components. +- **Motifs**: a faint cyan grid backdrop and scanline texture on `body`, glowing buttons + (`components/ui/button.tsx` — every variant glows in its own accent color: `default`/ + `outline`/`secondary`/`ghost` in `--primary`, `destructive` in `--destructive`; only `link` + has no surface to glow), and glowing HUD-style corner brackets on cards + (`components/ui/card.tsx`'s `Card`, plus `components/AuthLayout.tsx`). These live in the + shared primitives, not per-page overrides — a new page gets them for free by using + `Card`/`Button`/`AppHeader`, so build on those rather than hand-rolling styles. +- **Monospace for technical readouts**: IDs, hashes, token prefixes, cell labels, and + key=value data dumps use `font-mono` (see `components/protocol/cells/CellsTable.tsx`, + `ApiTokensSection.tsx`) + to read like a real data/terminal output, not prose. +- **Every button glows uniformly, by explicit choice**: an earlier pass reserved the glow for + one "primary" action per section (`default` variant only), with `outline`/`ghost` left + plain — the user found that inconsistent and asked for uniform glowing buttons instead. + Don't reintroduce the primary-only hierarchy; if a future change to `button.tsx` needs a + visual hierarchy again, ask first rather than assuming the old convention. +- **Tables vs. cards is a zoom-level decision, not a style preference**: a "container" list — + sparse metadata, click one to drill into it (e.g. the Experiments list) — is a card/tile + grid, not a table; a table only earns its place once you're inside that container looking + at dense, precise, multi-attribute records to compare (e.g. an experiment's Cells, with + their metrics/hyperparameters). Don't reach for a table just because a list exists — check + which zoom level you're actually building first. + - Detail on the Experiments tile grid lives in `frontend/src/pages/AGENTS.md`; detail on + the Cells heatmap/table (and the per-agent run tally that replaced the old Agents grid) + lives in `frontend/src/components/protocol/cells/AGENTS.md`. Read the relevant one before + touching `ExperimentsPage.tsx` or anything under `components/protocol/cells/`. +- **There is no separate experiment detail page** — clicking an Experiments tile lands on + `/experiments/{id}/protocol`, the protocol canvas, and everything that used to be on a + static `ExperimentDetailPage` now lives in that canvas's `ExperimentSidePanel` tabs + (Design/Cells/Runs/Results) or its top bar. `/experiments/{id}` is kept only as a redirect + for old bookmarks. Don't reintroduce a static detail page; a new per-experiment view is a + new tab in that panel. The panel is drag-resizable by its right edge (320px–1100px, always + leaving the canvas ≥420px) and remembers its width in `localStorage` — so a new tab in it + should be width-responsive via container queries, not built for one fixed column width. + +## Agent skills + +### Issue tracker + +Implementation issues and specs live in the separate `EpistasisLab/ASAREE_Issues` GitHub repository. See `docs/agents/issue-tracker.md`. + +### Triage labels + +Use the five canonical Matt Pocock triage labels. See `docs/agents/triage-labels.md`. + +### Domain docs + +This is a single-context repository with `CONTEXT.md` at the root and private local ADRs under `docs/adr/`. See `docs/agents/domain.md`. + +# Git commit conventions + +Do not add `Co-Authored-By` or `Generated-with` lines to commits or PRs. + +# Cost-conscious execution + +Favor the cheapest tool/approach that reliably gets the job done. Before doing any of the +following, say in one sentence what you're about to do and why it's needed, and ask first +rather than doing it silently as a routine part of coding: + +- **No Playwright/browser automation or screenshot-based UI verification.** The user checks + UI changes visually themselves once you report them done — see the standing preference to + skip Playwright verification agents. Screenshots are read as images, which cost far more + tokens than text, and driving a browser adds many extra tool round-trips on top. Verify with + `tsc`/`oxlint`/existing tests instead; if something genuinely can't be confirmed without a + browser, say so explicitly rather than skipping verification or reaching for Playwright + unasked. +- **No multi-agent fan-out for routine coding work** — don't invoke the `Workflow` tool or + spawn multiple subagents in parallel just because a task touches several files. Each spawned + agent carries its own multiplied context/token cost. Default to working solo, or at most one + scoped `Explore`/`general-purpose` agent for a targeted search. Reserve `Workflow`/parallel + fan-out for when the user explicitly asks for that scale. +- **No unscoped, repo-wide exploration** ("search everywhere," reading many files in full) when + the request already names specific files or areas — grep/read narrowly first, and only widen + the search if that comes up empty. +- **No speculative full test-suite runs or long builds** — run the narrowest test/lint command + that covers the change under review; ask before running something that takes minutes or spins + up extra infrastructure (dev servers, containers, etc.). + +# Experiment data model + +- **Datasets are a real, stored many-to-many** — the `experiment_datasets` join table + (`models/experiment_dataset.py`, migration `d5a3b90c71e4`), unlike agents below: a dataset + genuinely is a first-class property of an experiment worth a real relationship, matching + what ARES does. It started as a scalar `ResearchExperiment.dataset_id` FK (migration + `a1b2c3d4e5f6`); that column is **dropped** — uncapping the Dataset connector means one + experiment can run against several, so a single winner would have been a lie. Read/write it + through `services/experiments.py`'s `set_experiment_datasets` / + `get_experiment_dataset_ids` / `get_dataset_ids_by_experiment` (the batch one — use it on + list endpoints to avoid an N+1), never by touching the model. + - `position` on the join row preserves **canvas wiring order**, which is the order the + agent's prompt lists the datasets in, so it's user-visible, not cosmetic. + - `dataset_ids` is a **full replacement**, not a merge (`[]` detaches everything). The API + and SDK still accept and return the old scalar `dataset_id`: on write it's the + one-dataset shorthand, on read a view of `dataset_ids[0]`. Don't add a second source of + truth — the join table is it. + - It's set via `PATCH /experiments/{id}`, not at creation: the notebook's Step 1 (create the + experiment) runs *before* Step 2 (register the dataset), so there's nothing to attach yet + at create time — see the `client.experiments.update(...)` call after Step 2 in + `spinal_pipeline.ipynb`. In the GUI it's `ProtocolCanvas.tsx`'s `syncExperimentDatasets` + effect, which PATCHes whenever the canvas's set of Dataset nodes changes. That effect's + ref is deliberately **seeded from the graph as loaded so it never fires on mount** — a + mount-time "reconcile" would see zero Dataset nodes on a notebook-driven experiment and + detach its dataset. Keep that property if you touch it. + - An experiment with no datasets attached is an expected, permanent state (everything + created before the FK existed) — not a bug to backfill. +- **Agents are deliberately NOT a stored relationship** — there's no `experiment_agents` join + table, and none is planned. Agents are reusable per-user templates, not something an + experiment "owns"; asking "which agents ran in this experiment" is answered by scanning the + user's `Run`s for `run_metadata.experiment_id` matches (see `ExperimentAgents` in + `components/protocol/RunsTab.tsx`) and cross-referencing `GET /agents`, not by a new backend + association. `GET /runs` has no server-side `experiment_id` filter (only `agent_id`) — this + fetches every run for the user and filters client-side, which is fine at today's scale but + is the place to add a real filter if a user's run history grows large enough to matter. +- **Cells belong to a design revision, not to the experiment** — `experiment_design_revisions` + (`models/experiment_design_revision.py`, migration `e2f7c4a91b60`). The current design is the + one revision with `superseded_at IS NULL` (a partial unique index enforces "at most one", + rather than convention). This exists because generation used to be purely additive: a design + shrunk from 6 cells to 2 left all 6 behind, so the experiment still read "0/6 scored" and + "run all cells" still launched 6. + - **Never query `FactorialReplicateResult` without joining its owning cell** — that loses + experiment/design-revision scope. Go through + `services/factorial_cells.py`'s `get_replicate`/`list_replicates`/`upsert_replicate`, which scope to the + current revision by default and take an explicit `revision_id` only to read history on + purpose. `experiment_id` lives on the parent cell, and both experiment and revision filters + are applied together because `revision_id` can arrive from a query string. + - **Cell vs. replicate:** a `FactorialCell` is one unique factor combination together + with all its planned `FactorialReplicateResult` children. Group replicate responses by `cell_id`, + and use “replicate” for run/progress/scored counts. “Run all cells” is the experiment-level + action that runs every pending replicate across those cells. + - `generate_design_cells` opens a new revision **only when the new design would drop a cell + the current one has**. Re-clicking generate, changing the seed, widening a factor's levels + or raising `replicates` all keep the current revision and its row ids — history entries are + meant to mark designs that actually discarded something, not every edit. Results for a + label the two revisions share are copied forward; the originals stay in history. + - `ProtocolRun.design_revision_id` **pins** which design a run's result belongs to, so a + regenerate mid-flight can't redirect the write-back (`plan_cell_runs` → + `run_protocol`/`promote_cell_score_metrics`). It's `ondelete="SET NULL"` — run history + outlives the design; cells are `ondelete="CASCADE"` so deleting a revision really does + delete its results. + - Deleting the **current** revision is refused (409) — that's a reset, not a deletion; + regenerating is how you replace it. Revision numbers are `max()+1` over every revision ever, + so a deleted number is never reused. The history UI is `components/protocol/cells/ + DesignHistory.tsx`. +- **A protocol canvas is a draft; production runs use a published revision** — `Protocol.graph` + remains the autosaved draft while `protocol_revisions` stores immutable graph snapshots. A + `ProtocolRun` carries both `design_revision_id` and `protocol_revision_id`; `run_protocol` + loads the latter, so a canvas edit can never hot-patch queued, running, or resumed work. Never + create a production run from `Protocol.graph` directly: publish first and pass the resulting + revision through the planning path. The top bar deliberately says when a visible canvas has + unpublished changes and which published revision production currently uses. +- **A declared factor must bind to at least one canvas field before a cell batch can run** — + deleting/unbinding a node field leaves the factor declared but *unbound*, rather than silently + deleting the experimental treatment. The user must rebind it or remove it, then review the + design impact and regenerate. `services.factor_bindings` is the shared backend guard; the + Design tab shows the same state before generation. + +# Color — meaningful variation, not decoration + +A single cyan accent everywhere read as flat/monotone rather than retrofuturist (that style +leans on multi-hue neon contrast — synthwave, Tron, Blade Runner — not minimalism). Cards can +now be individually re-tinted, but the tint must mean something; don't add color for its own +sake. + +- **Mechanism**: `components/ui/card.tsx`'s `Card` reads a `--card-accent` CSS custom property + (falling back to `--primary`) for its ring/glow/corner-bracket colors — see the comment on + `Card` itself. Set it via `style={cardAccent('var(--chart-3)')}` (`lib/utils.ts`) on any + individual `Card`; anything that also wants the icon/badge/hover-arrow to match reads + `text-[color:var(--card-accent,var(--primary))]` the same way `AgentCard` and the Experiments + tiles do — don't hardcode `text-primary` on elements living inside a re-tintable card. +- **Status-driven tint** (`lib/experiment.ts`'s `cellsStatusAccent`): `--chart-4` (amber) = cells + generated but none scored, `--primary` (cyan) = partially scored, `--chart-3` (emerald) = fully + scored, `--muted-foreground` (dim) = no cells yet. Used on both the Experiments-list tiles and + the detail page's "Cells" stat card — the same status must always resolve to the same color + everywhere, so change it in that one function, not per call site. +- **Hash-driven tint** (`lib/utils.ts`'s `hashToChartHue`): for things with no "done/pending" + status but real category variety — e.g. `AgentCard` tints by `model_config.model`, so agents + sharing an LLM visually match without a hardcoded model→color table that goes stale the moment + a new model ships. Same input always produces the same one of the five `--chart-*` hues. + **Not for protocol-canvas nodes** — see the table below. +- **Table-driven tint** (`lib/nodeAccent.ts`'s `nodeAccent(kind)`): protocol-canvas nodes only. + Thirteen node kinds against five `--chart-*` hues made collisions arithmetic, and they landed + on the confusable pairs (Skill/AI, Dataset/Knowledge, Pattern/Script), so every kind now has an + explicit entry. Five keep the `--chart-*` hue the old hash gave them (agent, dataset, + reason+act, critic gate, and the LLM family) and the other eight moved to a `--node-1`…`--node-8` + slot in `index.css`'s `.dark` block — **fixing a repeat means moving the bucket-mates, not + repainting the canvas**, so don't reassign an anchor's hue without being asked. Both the node + card and its inspector call `nodeAccent` with the same key so they can't drift; a kind with no + entry falls back to `--primary` rather than a hashed hue, so a new node type visibly asks for a + slot. Adjacent hues are assigned to related kinds on purpose (the two MCP kinds, the two OKF + kinds). Note LLM nodes are one hue for the whole family, not one per provider. + - `--node-label` (yellow) is separate from all of them: it's the connector captions on + `AgentNode`/`CriticGateNode`, which are meant to stand out *from* their node, so they + deliberately don't follow `--card-accent` the way everything else inside a card does. +- **Don't hash/rotate a tint just to break up visual monotony** with no underlying meaning (e.g. + cycling colors by array index) — that was considered and rejected in favor of the schemes + above. If a new list of things genuinely has no status or category worth encoding in color, + it's fine for it to stay a single accent. + +# Communication style + +Be concise and execution-oriented. + +For coding tasks: +- Do the requested work without lengthy explanations. +- Make routine implementation decisions yourself. +- Do not present multiple alternatives unless the choice materially affects + architecture, correctness, or requirements. +- Do not enumerate pros and cons for routine decisions. +- Do not explain obvious code changes. +- Do not repeatedly summarize what you are doing. +- Ask questions only when ambiguity would materially change the implementation. +- Prefer implementing over discussing. +- After completing a task, give a short summary of what changed. +- If you identify an important concern, state it briefly and recommend one + course of action rather than presenting many possibilities. + +Keep responses focused, brief, and concise. Give deeper explanations only +when I explicitly ask for them. diff --git a/CLAUDE.md b/CLAUDE.md index 73f237f..43c994c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,228 +1 @@ -# Frontend visual style - -The frontend (`frontend/`) uses a deliberate dark, retrofuturist/high-tech visual language. -Preserve it for all new pages and components — don't revert to shadcn's plain default theme -or introduce a different aesthetic without being asked. - -- **Theme**: dark mode only (`` in `index.html`), a graphite background - (not near-black) with a cyan/electric-blue accent (`--primary`). Tokens live in - `frontend/src/index.css`'s `.dark` block — adjust values there, don't hardcode colors in - components. -- **Motifs**: a faint cyan grid backdrop and scanline texture on `body`, glowing buttons - (`components/ui/button.tsx` — every variant glows in its own accent color: `default`/ - `outline`/`secondary`/`ghost` in `--primary`, `destructive` in `--destructive`; only `link` - has no surface to glow), and glowing HUD-style corner brackets on cards - (`components/ui/card.tsx`'s `Card`, plus `components/AuthLayout.tsx`). These live in the - shared primitives, not per-page overrides — a new page gets them for free by using - `Card`/`Button`/`AppHeader`, so build on those rather than hand-rolling styles. -- **Monospace for technical readouts**: IDs, hashes, token prefixes, cell labels, and - key=value data dumps use `font-mono` (see `components/protocol/cells/CellsTable.tsx`, - `ApiTokensSection.tsx`) - to read like a real data/terminal output, not prose. -- **Every button glows uniformly, by explicit choice**: an earlier pass reserved the glow for - one "primary" action per section (`default` variant only), with `outline`/`ghost` left - plain — the user found that inconsistent and asked for uniform glowing buttons instead. - Don't reintroduce the primary-only hierarchy; if a future change to `button.tsx` needs a - visual hierarchy again, ask first rather than assuming the old convention. -- **Tables vs. cards is a zoom-level decision, not a style preference**: a "container" list — - sparse metadata, click one to drill into it (e.g. the Experiments list) — is a card/tile - grid, not a table; a table only earns its place once you're inside that container looking - at dense, precise, multi-attribute records to compare (e.g. an experiment's Cells, with - their metrics/hyperparameters). Don't reach for a table just because a list exists — check - which zoom level you're actually building first. - - Detail on the Experiments tile grid lives in `frontend/src/pages/CLAUDE.md`; detail on - the Cells heatmap/table (and the per-agent run tally that replaced the old Agents grid) - lives in `frontend/src/components/protocol/cells/CLAUDE.md`. Read the relevant one before - touching `ExperimentsPage.tsx` or anything under `components/protocol/cells/`. -- **There is no separate experiment detail page** — clicking an Experiments tile lands on - `/experiments/{id}/protocol`, the protocol canvas, and everything that used to be on a - static `ExperimentDetailPage` now lives in that canvas's `ExperimentSidePanel` tabs - (Design/Cells/Runs/Results) or its top bar. `/experiments/{id}` is kept only as a redirect - for old bookmarks. Don't reintroduce a static detail page; a new per-experiment view is a - new tab in that panel. The panel is drag-resizable by its right edge (320px–1100px, always - leaving the canvas ≥420px) and remembers its width in `localStorage` — so a new tab in it - should be width-responsive via container queries, not built for one fixed column width. - -## Agent skills - -### Issue tracker - -Implementation issues and specs live in the separate `EpistasisLab/ASAREE_Issues` GitHub repository. See `docs/agents/issue-tracker.md`. - -### Triage labels - -Use the five canonical Matt Pocock triage labels. See `docs/agents/triage-labels.md`. - -### Domain docs - -This is a single-context repository with `CONTEXT.md` at the root and private local ADRs under `docs/adr/`. See `docs/agents/domain.md`. - -# Git commit conventions - -Do not add `Co-Authored-By` or `Generated-with` lines to commits or PRs. - -# Cost-conscious execution - -Favor the cheapest tool/approach that reliably gets the job done. Before doing any of the -following, say in one sentence what you're about to do and why it's needed, and ask first -rather than doing it silently as a routine part of coding: - -- **No Playwright/browser automation or screenshot-based UI verification.** The user checks - UI changes visually themselves once you report them done — see the standing preference to - skip Playwright verification agents. Screenshots are read as images, which cost far more - tokens than text, and driving a browser adds many extra tool round-trips on top. Verify with - `tsc`/`oxlint`/existing tests instead; if something genuinely can't be confirmed without a - browser, say so explicitly rather than skipping verification or reaching for Playwright - unasked. -- **No multi-agent fan-out for routine coding work** — don't invoke the `Workflow` tool or - spawn multiple subagents in parallel just because a task touches several files. Each spawned - agent carries its own multiplied context/token cost. Default to working solo, or at most one - scoped `Explore`/`general-purpose` agent for a targeted search. Reserve `Workflow`/parallel - fan-out for when the user explicitly asks for that scale. -- **No unscoped, repo-wide exploration** ("search everywhere," reading many files in full) when - the request already names specific files or areas — grep/read narrowly first, and only widen - the search if that comes up empty. -- **No speculative full test-suite runs or long builds** — run the narrowest test/lint command - that covers the change under review; ask before running something that takes minutes or spins - up extra infrastructure (dev servers, containers, etc.). - -# Experiment data model - -- **Datasets are a real, stored many-to-many** — the `experiment_datasets` join table - (`models/experiment_dataset.py`, migration `d5a3b90c71e4`), unlike agents below: a dataset - genuinely is a first-class property of an experiment worth a real relationship, matching - what ARES does. It started as a scalar `ResearchExperiment.dataset_id` FK (migration - `a1b2c3d4e5f6`); that column is **dropped** — uncapping the Dataset connector means one - experiment can run against several, so a single winner would have been a lie. Read/write it - through `services/experiments.py`'s `set_experiment_datasets` / - `get_experiment_dataset_ids` / `get_dataset_ids_by_experiment` (the batch one — use it on - list endpoints to avoid an N+1), never by touching the model. - - `position` on the join row preserves **canvas wiring order**, which is the order the - agent's prompt lists the datasets in, so it's user-visible, not cosmetic. - - `dataset_ids` is a **full replacement**, not a merge (`[]` detaches everything). The API - and SDK still accept and return the old scalar `dataset_id`: on write it's the - one-dataset shorthand, on read a view of `dataset_ids[0]`. Don't add a second source of - truth — the join table is it. - - It's set via `PATCH /experiments/{id}`, not at creation: the notebook's Step 1 (create the - experiment) runs *before* Step 2 (register the dataset), so there's nothing to attach yet - at create time — see the `client.experiments.update(...)` call after Step 2 in - `spinal_pipeline.ipynb`. In the GUI it's `ProtocolCanvas.tsx`'s `syncExperimentDatasets` - effect, which PATCHes whenever the canvas's set of Dataset nodes changes. That effect's - ref is deliberately **seeded from the graph as loaded so it never fires on mount** — a - mount-time "reconcile" would see zero Dataset nodes on a notebook-driven experiment and - detach its dataset. Keep that property if you touch it. - - An experiment with no datasets attached is an expected, permanent state (everything - created before the FK existed) — not a bug to backfill. -- **Agents are deliberately NOT a stored relationship** — there's no `experiment_agents` join - table, and none is planned. Agents are reusable per-user templates, not something an - experiment "owns"; asking "which agents ran in this experiment" is answered by scanning the - user's `Run`s for `run_metadata.experiment_id` matches (see `ExperimentAgents` in - `components/protocol/RunsTab.tsx`) and cross-referencing `GET /agents`, not by a new backend - association. `GET /runs` has no server-side `experiment_id` filter (only `agent_id`) — this - fetches every run for the user and filters client-side, which is fine at today's scale but - is the place to add a real filter if a user's run history grows large enough to matter. -- **Cells belong to a design revision, not to the experiment** — `experiment_design_revisions` - (`models/experiment_design_revision.py`, migration `e2f7c4a91b60`). The current design is the - one revision with `superseded_at IS NULL` (a partial unique index enforces "at most one", - rather than convention). This exists because generation used to be purely additive: a design - shrunk from 6 cells to 2 left all 6 behind, so the experiment still read "0/6 scored" and - "run all cells" still launched 6. - - **Never query `FactorialReplicateResult` without joining its owning cell** — that loses - experiment/design-revision scope. Go through - `services/factorial_cells.py`'s `get_replicate`/`list_replicates`/`upsert_replicate`, which scope to the - current revision by default and take an explicit `revision_id` only to read history on - purpose. `experiment_id` lives on the parent cell, and both experiment and revision filters - are applied together because `revision_id` can arrive from a query string. - - **Cell vs. replicate:** a `FactorialCell` is one unique factor combination together - with all its planned `FactorialReplicateResult` children. Group replicate responses by `cell_id`, - and use “replicate” for run/progress/scored counts. “Run all cells” is the experiment-level - action that runs every pending replicate across those cells. - - `generate_design_cells` opens a new revision **only when the new design would drop a cell - the current one has**. Re-clicking generate, changing the seed, widening a factor's levels - or raising `replicates` all keep the current revision and its row ids — history entries are - meant to mark designs that actually discarded something, not every edit. Results for a - label the two revisions share are copied forward; the originals stay in history. - - `ProtocolRun.design_revision_id` **pins** which design a run's result belongs to, so a - regenerate mid-flight can't redirect the write-back (`plan_cell_runs` → - `run_protocol`/`promote_cell_score_metrics`). It's `ondelete="SET NULL"` — run history - outlives the design; cells are `ondelete="CASCADE"` so deleting a revision really does - delete its results. - - Deleting the **current** revision is refused (409) — that's a reset, not a deletion; - regenerating is how you replace it. Revision numbers are `max()+1` over every revision ever, - so a deleted number is never reused. The history UI is `components/protocol/cells/ - DesignHistory.tsx`. -- **A protocol canvas is a draft; production runs use a published revision** — `Protocol.graph` - remains the autosaved draft while `protocol_revisions` stores immutable graph snapshots. A - `ProtocolRun` carries both `design_revision_id` and `protocol_revision_id`; `run_protocol` - loads the latter, so a canvas edit can never hot-patch queued, running, or resumed work. Never - create a production run from `Protocol.graph` directly: publish first and pass the resulting - revision through the planning path. The top bar deliberately says when a visible canvas has - unpublished changes and which published revision production currently uses. -- **A declared factor must bind to at least one canvas field before a cell batch can run** — - deleting/unbinding a node field leaves the factor declared but *unbound*, rather than silently - deleting the experimental treatment. The user must rebind it or remove it, then review the - design impact and regenerate. `services.factor_bindings` is the shared backend guard; the - Design tab shows the same state before generation. - -# Color — meaningful variation, not decoration - -A single cyan accent everywhere read as flat/monotone rather than retrofuturist (that style -leans on multi-hue neon contrast — synthwave, Tron, Blade Runner — not minimalism). Cards can -now be individually re-tinted, but the tint must mean something; don't add color for its own -sake. - -- **Mechanism**: `components/ui/card.tsx`'s `Card` reads a `--card-accent` CSS custom property - (falling back to `--primary`) for its ring/glow/corner-bracket colors — see the comment on - `Card` itself. Set it via `style={cardAccent('var(--chart-3)')}` (`lib/utils.ts`) on any - individual `Card`; anything that also wants the icon/badge/hover-arrow to match reads - `text-[color:var(--card-accent,var(--primary))]` the same way `AgentCard` and the Experiments - tiles do — don't hardcode `text-primary` on elements living inside a re-tintable card. -- **Status-driven tint** (`lib/experiment.ts`'s `cellsStatusAccent`): `--chart-4` (amber) = cells - generated but none scored, `--primary` (cyan) = partially scored, `--chart-3` (emerald) = fully - scored, `--muted-foreground` (dim) = no cells yet. Used on both the Experiments-list tiles and - the detail page's "Cells" stat card — the same status must always resolve to the same color - everywhere, so change it in that one function, not per call site. -- **Hash-driven tint** (`lib/utils.ts`'s `hashToChartHue`): for things with no "done/pending" - status but real category variety — e.g. `AgentCard` tints by `model_config.model`, so agents - sharing an LLM visually match without a hardcoded model→color table that goes stale the moment - a new model ships. Same input always produces the same one of the five `--chart-*` hues. - **Not for protocol-canvas nodes** — see the table below. -- **Table-driven tint** (`lib/nodeAccent.ts`'s `nodeAccent(kind)`): protocol-canvas nodes only. - Thirteen node kinds against five `--chart-*` hues made collisions arithmetic, and they landed - on the confusable pairs (Skill/AI, Dataset/Knowledge, Pattern/Script), so every kind now has an - explicit entry. Five keep the `--chart-*` hue the old hash gave them (agent, dataset, - reason+act, critic gate, and the LLM family) and the other eight moved to a `--node-1`…`--node-8` - slot in `index.css`'s `.dark` block — **fixing a repeat means moving the bucket-mates, not - repainting the canvas**, so don't reassign an anchor's hue without being asked. Both the node - card and its inspector call `nodeAccent` with the same key so they can't drift; a kind with no - entry falls back to `--primary` rather than a hashed hue, so a new node type visibly asks for a - slot. Adjacent hues are assigned to related kinds on purpose (the two MCP kinds, the two OKF - kinds). Note LLM nodes are one hue for the whole family, not one per provider. - - `--node-label` (yellow) is separate from all of them: it's the connector captions on - `AgentNode`/`CriticGateNode`, which are meant to stand out *from* their node, so they - deliberately don't follow `--card-accent` the way everything else inside a card does. -- **Don't hash/rotate a tint just to break up visual monotony** with no underlying meaning (e.g. - cycling colors by array index) — that was considered and rejected in favor of the schemes - above. If a new list of things genuinely has no status or category worth encoding in color, - it's fine for it to stay a single accent. - -# Communication style - -Be concise and execution-oriented. - -For coding tasks: -- Do the requested work without lengthy explanations. -- Make routine implementation decisions yourself. -- Do not present multiple alternatives unless the choice materially affects - architecture, correctness, or requirements. -- Do not enumerate pros and cons for routine decisions. -- Do not explain obvious code changes. -- Do not repeatedly summarize what you are doing. -- Ask questions only when ambiguity would materially change the implementation. -- Prefer implementing over discussing. -- After completing a task, give a short summary of what changed. -- If you identify an important concern, state it briefly and recommend one - course of action rather than presenting many possibilities. - -Keep responses focused, brief, and concise. Give deeper explanations only -when I explicitly ask for them. +@AGENTS.md diff --git a/CONTEXT.md b/CONTEXT.md index 8d1d770..eb9c555 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -88,6 +88,16 @@ _Avoid_: Inferred observation, promoted score A metric an Agent inspector includes in that agent's system prompt as evaluation guidance; it never exposes a future observation. _Avoid_: Prompt metric +### Protocols + +**Model**: +The configured language model an Agent or Critic Gate uses to produce or review output. +_Avoid_: AI, LLM (when naming the protocol role) + +**Model connection**: +The required relationship assigning exactly one Model to an Agent or Critic Gate. +_Avoid_: AI connection, LLM connection + ### Runs **Protocol canvas**: diff --git a/frontend/README.md b/frontend/README.md index 2ea8a28..c7e3b8f 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -3,7 +3,7 @@ React 19 + TypeScript + Vite, styled with Tailwind v4 and a small set of local shadcn-style primitives over [Base UI](https://base-ui.com/) (`src/components/ui`). The visual language is deliberate and documented in the repo root's -[`CLAUDE.md`](../CLAUDE.md) — read that before adding a page or component. +[`AGENTS.md`](../AGENTS.md) — read that before adding a page or component. ## How it runs: dev server only, no build step diff --git a/frontend/src/components/CreateCredentialDialog.tsx b/frontend/src/components/CreateCredentialDialog.tsx index fb8a132..9b52ac9 100644 --- a/frontend/src/components/CreateCredentialDialog.tsx +++ b/frontend/src/components/CreateCredentialDialog.tsx @@ -10,7 +10,7 @@ import { Label } from '@/components/ui/label' import { PasswordInput } from '@/components/ui/password-input' import { cn, HUD_ACCENT_RING_CLASSNAME } from '@/lib/utils' import { LLM_PROVIDER_CATALOG, type LLMProvider } from '@/types/llmSettings' -import { PROVIDER_META } from '@/components/protocol/nodes/LlmNode' +import { PROVIDER_META } from '@/components/protocol/nodes/ModelNode' const PROVIDER_CATALOG = LLM_PROVIDER_CATALOG @@ -21,7 +21,7 @@ export function CreateCredentialDialog({ }: { open: boolean onOpenChange: (open: boolean) => void - // Opening this from a specific LLM node's inspector should land straight + // Opening this from a specific Model node's inspector should land straight // on that provider's fields, not the search screen -- only meaningful with // more than one catalog entry to search through. defaultProvider?: LLMProvider | null diff --git a/frontend/src/components/LlmConnectionCheck.tsx b/frontend/src/components/LlmConnectionCheck.tsx index 3e1db13..fb60b36 100644 --- a/frontend/src/components/LlmConnectionCheck.tsx +++ b/frontend/src/components/LlmConnectionCheck.tsx @@ -6,9 +6,9 @@ import type { LLMConnectionStatus, LLMProvider } from '@/types/llmSettings' // Shared by all three places a credential's health is shown -- the Profile // page's credentials table, the save step in CreateCredentialDialog, and the -// LLM node inspector's Credential field. One definition on purpose: the same +// Model node inspector's Credential field. One definition on purpose: the same // status must always resolve to the same color and the same word everywhere -// (the root CLAUDE.md's rule for status-driven tint), and "Key valid" in one +// (the root AGENTS.md's rule for status-driven tint), and "Key valid" in one // place with "Connected" in another would read as two different claims. // // Colors follow the app's status language: emerald for done/good, amber for diff --git a/frontend/src/components/protocol/AddNodePanel.test.tsx b/frontend/src/components/protocol/AddNodePanel.test.tsx new file mode 100644 index 0000000..7bbe97a --- /dev/null +++ b/frontend/src/components/protocol/AddNodePanel.test.tsx @@ -0,0 +1,38 @@ +import { fireEvent, render, screen } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' +import { AddNodePanel } from './AddNodePanel' + +describe('AddNodePanel', () => { + it('groups node types by protocol role', () => { + render() + + expect(screen.getAllByRole('heading', { level: 3 }).map((heading) => heading.textContent)).toEqual([ + 'Agents', + 'Models', + 'Execution patterns', + 'Knowledge & data', + 'Tools & output', + ]) + }) + + it('only shows categories containing search matches', () => { + render() + + fireEvent.change(screen.getByPlaceholderText('Search node types…'), { target: { value: 'OpenAI' } }) + + expect(screen.getByRole('heading', { name: 'Models' })).toBeInTheDocument() + expect(screen.queryByRole('heading', { name: 'Agents' })).not.toBeInTheDocument() + expect(screen.getByRole('button', { name: /OpenAI/ })).toBeInTheDocument() + }) + + it('shows where Sub-Agents are added and enables them in that connector picker', () => { + const { rerender } = render() + + expect(screen.getByRole('button', { name: /Sub-Agent/ })).toBeDisabled() + expect(screen.getByText("Add from an Agent's Sub-Agents connector")).toBeInTheDocument() + + rerender() + + expect(screen.getByRole('button', { name: /Sub-Agent/ })).toBeEnabled() + }) +}) diff --git a/frontend/src/components/protocol/AddNodePanel.tsx b/frontend/src/components/protocol/AddNodePanel.tsx index fdb6de5..d7b7218 100644 --- a/frontend/src/components/protocol/AddNodePanel.tsx +++ b/frontend/src/components/protocol/AddNodePanel.tsx @@ -7,22 +7,33 @@ import { MCP_SERVER_BROWSE } from './mcpServerCatalog' import { OKF_BUNDLE_BROWSE, OKF_DOCUMENT_BROWSE } from './okfCatalog' import { SKILL_BROWSE } from './skillCatalog' +const NODE_CATEGORIES = [ + { id: 'agents', label: 'Agents' }, + { id: 'models', label: 'Models' }, + { id: 'patterns', label: 'Execution patterns' }, + { id: 'knowledge', label: 'Knowledge & data' }, + { id: 'tools', label: 'Tools & output' }, +] as const + +type NodeCategory = (typeof NODE_CATEGORIES)[number]['id'] + // Every entry here earns its place: GET /api/mcp-servers already backs the // tool picker, GET /datasets already backs the dataset picker, // services.protocol_execution._run_gated_worker already implements the -// critic gate's revision loop, and _resolve_llm_config/_resolve_tool_config/ +// critic gate's revision loop, and _resolve_model_config/_resolve_tool_config/ // _resolve_dataset_configs/_resolve_script_configs already resolve an agent's // respective connectors. "memory" and the two pattern entries are the // exceptions -- each is real in the graph/validation sense (wiring one up is // accepted and does something visually) but has NO runtime effect yet, // documented on the node/inspector itself, not hidden from the catalog. // LLM/Architectural Pattern are each a family of node types (one per -// provider/pattern -- see LlmNodeData/ReasonActPatternNodeData in +// provider/pattern -- see ModelNodeData/ReasonActPatternNodeData in // types/protocols.ts for why), not one generic entry with an internal // picker -- this catalog, filtered to a connector's own family via // allowedTypes, IS that picker. const NODE_CATALOG = [ - { type: 'agent', label: 'Agent', description: 'An LLM agent stage in the pipeline', icon: Bot }, + { type: 'agent', category: 'agents', label: 'Agent', description: 'An LLM agent stage in the pipeline', icon: Bot }, + { type: 'sub_agent', category: 'agents', label: 'Sub-Agent', description: 'A delegated worker callable by one parent Agent', icon: Bot }, // Not a node type -- picking this opens the server browser // (McpServerBrowserPanel), and the node gets created from whichever // server is chosen there. It replaced a plain "MCP Tool" entry that made @@ -33,50 +44,58 @@ const NODE_CATALOG = [ // nothing already on a canvas changes. { type: MCP_SERVER_BROWSE, + category: 'tools', label: 'MCP Servers', description: "Browse available MCP servers and allow-list their tools for an Agent", icon: Server, }, { type: 'critic_gate', + category: 'agents', label: 'Critic Gate', description: "Reviews an upstream Agent's output, requests revisions", icon: ShieldCheck, }, - { type: 'llm_anthropic', label: 'Anthropic', description: "An Agent or Critic Gate's model, temperature, and parameters", icon: Sparkles }, - { type: 'llm_openai', label: 'OpenAI', description: "An Agent or Critic Gate's model, temperature, and parameters", icon: Atom }, + { type: 'model_anthropic', category: 'models', label: 'Anthropic', description: "An Agent or Critic Gate's model, temperature, and parameters", icon: Sparkles }, + { type: 'model_openai', category: 'models', label: 'OpenAI', description: "An Agent or Critic Gate's model, temperature, and parameters", icon: Atom }, { - type: 'llm_azure_foundry', + type: 'model_azure_foundry', + category: 'models', label: 'Azure AI Foundry', description: "An Agent or Critic Gate's model, temperature, and parameters -- routed through your own Azure resource", icon: Cloud, }, { - type: 'llm_openrouter', + type: 'model_openrouter', + category: 'models', label: 'OpenRouter', description: "An Agent or Critic Gate's model, temperature, and parameters -- routed through your own OpenRouter account", icon: Route, }, { - type: 'llm_local', + type: 'model_local', + category: 'models', label: 'Local', description: "An Agent or Critic Gate's model, temperature, and parameters -- routed to a self-hosted OpenAI-compatible server", icon: HardDrive, }, { type: 'pattern_reason_act', + category: 'patterns', label: 'Reason + Act', description: 'Alternates between reasoning and tool calls each iteration until it reaches a final answer', icon: Repeat2, }, { type: 'pattern_single_agent_baseline', + category: 'patterns', label: 'Single-Agent Baseline', description: 'One reasoning pass per iteration, no tool-call loop -- the cheap default when there’s nothing to call', icon: ArrowRight, }, { type: 'memory', + category: 'knowledge', label: 'Memory', description: 'Not yet functional -- declares intent for a future phase', icon: BrainCircuit, @@ -89,6 +108,7 @@ const NODE_CATALOG = [ // there's no picker in the node's inspector -- the dataset IS the node. { type: DATASET_BROWSE, + category: 'knowledge', label: 'Datasets', description: "Browse your registered datasets -- the data an Agent's workspace tools operate on", icon: Database, @@ -100,6 +120,7 @@ const NODE_CATALOG = [ // whole skill library, so it's where registering and deleting live. { type: SKILL_BROWSE, + category: 'knowledge', label: 'Skills', description: 'Browse your Agent Skills -- instructions an Agent opens when their description matches the task', icon: ScrollText, @@ -109,6 +130,7 @@ const NODE_CATALOG = [ // That browser is also the only place bundles are uploaded. { type: OKF_BUNDLE_BROWSE, + category: 'knowledge', label: 'OKF Bundles', description: 'Upload a folder of Markdown concepts an Agent reads and writes as it works', icon: BookMarked, @@ -119,23 +141,32 @@ const NODE_CATALOG = [ // concepts and a document is a single one -- exactly like Skills. { type: OKF_DOCUMENT_BROWSE, + category: 'knowledge', label: 'OKF Documents', description: 'Upload a single Markdown concept an Agent reads and rewrites as it works', icon: FileText, }, { type: 'output_parser', + category: 'tools', label: 'Output Parser', description: "Defines the format an Agent's answer must take, and reads its named, typed fields back out", icon: Braces, }, { type: 'script', + category: 'tools', label: 'Script', description: 'A fixed piece of Python code an Agent passes verbatim into some tool', icon: Code2, }, -] +] satisfies Array<{ + type: string + category: NodeCategory + label: string + description: string + icon: typeof Bot +}> export function AddNodePanel({ onAdd, @@ -152,11 +183,17 @@ export function AddNodePanel({ title?: string }) { const [query, setQuery] = useState('') - const catalog = allowedTypes ? NODE_CATALOG.filter((item) => allowedTypes.includes(item.type)) : NODE_CATALOG + const catalog = allowedTypes + ? NODE_CATALOG.filter((item) => allowedTypes.includes(item.type)) + : NODE_CATALOG const filtered = catalog.filter((item) => item.label.toLowerCase().includes(query.trim().toLowerCase())) + const groups = NODE_CATEGORIES.flatMap((category) => { + const items = filtered.filter((item) => item.category === category.id) + return items.length > 0 ? [{ ...category, items }] : [] + }) return ( -
+

{title}

setQuery(e.target.value)} /> -
+
{filtered.length === 0 &&

No matching node types.

} - {filtered.map((item) => ( - - ))} +
+ {groups.map((group) => ( +
+

+ {group.label} +

+
+ {group.items.map((item) => ( + + ))} +
+
+ ))} +
) diff --git a/frontend/src/components/protocol/AgentNodeInspector.tsx b/frontend/src/components/protocol/AgentNodeInspector.tsx index e3745a8..7778acd 100644 --- a/frontend/src/components/protocol/AgentNodeInspector.tsx +++ b/frontend/src/components/protocol/AgentNodeInspector.tsx @@ -10,7 +10,7 @@ import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs' import { Textarea } from '@/components/ui/textarea' import { defaultSystemPrompt } from './defaultSystemPrompt' import { EditableNodeTitle } from './EditableNodeTitle' -import { FactorBindableField, MakeNodeFactorButton } from './FactorBindableField' +import { FactorBindableField } from './FactorBindableField' import { ReceivesSummary, SendsSummary } from './HandoffSummary' import { NodeInspectorDialog } from './NodeInspectorDialog' import { NodeRunOutputPanel, ReceivedPromptPanel, UnresolvedReferencesNote } from './NodeRunOutputPanel' @@ -23,8 +23,6 @@ import { referenceLabel, seedPromptText } from '@/lib/promptReferences' import type { HandoffPeers, PromptReferenceScope } from '@/lib/promptReferences' import type { AgentNodeConfig, AgentNodeData, NodeRunState, PromptPreview, ProtocolNode } from '@/types/protocols' -const ACCENT = nodeAccent('agent') - // The middle column is where the actual editing happens, so neither side pane // may drag it below a width its labels and textareas still work at. Enforced // at drag start (see useResizablePane) against the frame's measured width. @@ -42,9 +40,10 @@ const GUTTER_WIDTH = 96 // unaffected by which Parameters/Settings tab is active) is shared with the // other node inspectors via `NodeInspectorDialog` -- see that file for why. // -// Three columns: Input, then Parameters/Settings, then Output -- laid out in -// the direction data actually travels, so the agent's configuration sits -// literally between what it is handed and what it produces. +// Agents use three columns: Input, then Parameters/Settings, then Output -- +// laid out in the direction data actually travels. Sub-Agents omit Input +// because they are invoked as tools by their parent rather than participating +// in the previous/next-agent handoff chain. // // Input and Output are always-visible panes rather than tabs because both are // things you check *while* adjusting Parameters, not destinations you tab away @@ -97,7 +96,8 @@ export function AgentNodeInspector({ onDelete: (nodeId: string) => void onClose: () => void }) { - const { requestMakeFactor, requestConnectorAdd, convertLegacyOutputContract } = useProtocolCanvasActions() + const { requestConnectorAdd, convertLegacyOutputContract } = useProtocolCanvasActions() + const isSubAgent = node?.type === 'sub_agent' // Measured at drag start so each pane's ceiling accounts for what the other // one is currently taking; read through a ref because the two hooks below // would otherwise have to reference each other's not-yet-declared width. @@ -124,7 +124,7 @@ export function AgentNodeInspector({ resolveMaxWidth: () => roomFor('input'), recomputeKey: node?.id ?? '', }) - widthsRef.current = { input: inputPane.width, output: outputPane.width } + widthsRef.current = { input: isSubAgent ? 0 : inputPane.width, output: outputPane.width } const experimentQuery = useQuery({ queryKey: ['experiments', experimentId], @@ -134,6 +134,7 @@ export function AgentNodeInspector({ if (!node) return null const data = node.data + const accent = nodeAccent(node.type === 'sub_agent' ? 'sub_agent' : 'agent') const config = data.config const bindings = data.factor_bindings ?? {} // The lead marker is meaningless under any other coordination strategy, so @@ -152,7 +153,8 @@ export function AgentNodeInspector({ // to move the role. Showing it everywhere would invite creating an invalid // canvas, and silently reassigning on click would move a role the user might // only have been inspecting. - const canMarkLead = leadRole !== null && (markedLeadAgentId === null || markedLeadAgentId === node.id) + const canMarkLead = + node.type !== 'sub_agent' && leadRole !== null && (markedLeadAgentId === null || markedLeadAgentId === node.id) // The pre-node way of declaring an output shape, still honoured by the // executor when no parser node is wired (see _resolve_output_contract). // Its presence swaps the section below into the convert-it banner: an agent @@ -184,56 +186,77 @@ export function AgentNodeInspector({ onOpenChange={(open) => { if (!open) onClose() }} - accent={ACCENT} + accent={accent} title={ <> - - onChange(node.id, { ...data, label })} /> - requestMakeFactor(node.id)} /> + + onChange(node.id, { ...data, label })} + /> + bindFactor('active', name)} + onUnbind={() => unbindFactor('active')} + > + {(trigger) => trigger} + } onDelete={() => onDelete(node.id)} onClose={onClose} >
-
-

Input

- {/* First, because it is the context everything below is read - against: every incoming edge delivers, so "what am I even given?" - has to be answerable before the assembled prompt underneath means - anything. */} - - fetchPromptPreview(node.id)} - /> - {/* Below the design-time preview, because it supersedes it: a - placeholder proves nothing about a run that actually happened. */} - {nodeRun?.run_id && ( - // Boxed to match the preview above it -- in this pane the two are a - // matched pair, where elsewhere the panel is one item in a list. -
- + {!isSubAgent && ( + <> +
+

Input

+ {/* First, because it is the context everything below is read + against: every incoming edge delivers, so "what am I even given?" + has to be answerable before the assembled prompt underneath means + anything. */} + + fetchPromptPreview(node.id)} + /> + {/* Below the design-time preview, because it supersedes it: a + placeholder proves nothing about a run that actually happened. */} + {nodeRun?.run_id && ( + // Boxed to match the preview above it -- in this pane the two are a + // matched pair, where elsewhere the panel is one item in a list. +
+ +
+ )} + referenceLabel(ref, referenceScope.names))} + />
- )} - referenceLabel(ref, referenceScope.names))} - /> -
-
+
+ + )} -
+
Parameters @@ -509,10 +532,15 @@ export function AgentNodeInspector({

Output

{/* Mirroring Receives on the far side: who this answer is handed to, stated above the answer itself. */} - + {!isSubAgent && } {/* The received prompt lives in the Input pane instead -- see the layout note at the top of this file. */} - +
diff --git a/frontend/src/components/protocol/DatasetNodeInspector.tsx b/frontend/src/components/protocol/DatasetNodeInspector.tsx index 8061369..b210b05 100644 --- a/frontend/src/components/protocol/DatasetNodeInspector.tsx +++ b/frontend/src/components/protocol/DatasetNodeInspector.tsx @@ -132,8 +132,8 @@ export function DatasetNodeInspector({ onChange(node.id, { ...data, label })} /> {/* In the title row rather than beside a field, because there is no "which dataset" field here to sit beside -- the dataset IS the - node. Same position as the Agent/Pattern inspectors' own - MakeNodeFactorButton, and the same visual identity, but this one + node. Same position and visual identity as the Agent inspector's + own Active factor control, but this one binds one specific field (`config`, the whole node) directly instead of opening a per-node field picker: a Dataset node has exactly one whole-node factor worth making, so a picker listing diff --git a/frontend/src/components/protocol/FactorBindableField.tsx b/frontend/src/components/protocol/FactorBindableField.tsx index 9cf1826..11dfa65 100644 --- a/frontend/src/components/protocol/FactorBindableField.tsx +++ b/frontend/src/components/protocol/FactorBindableField.tsx @@ -31,7 +31,7 @@ import type { DesignFactor } from '@/types/experiments' // NOT its layout -- `children` is a render prop that receives the trigger // element (a "make it a factor" button, a bound "Factor: {name}" badge, or a // disabled button) and decides where to put it. Callers place it inline -// right next to their own field's Label text (see e.g. LlmNodeInspector's +// right next to their own field's Label text (see e.g. ModelNodeInspector's // "Model" Label) rather than trailing after the whole Label+control block -- // a fixed trailing position reads as decoration bolted onto the end of a // row; sitting directly beside the text it labels reads as part of the @@ -51,26 +51,6 @@ import type { DesignFactor } from '@/types/experiments' // (slow, inconsistent chrome -- see CanvasControls.tsx's own reasoning for // the same swap). // -// The Agent/Pattern inspectors' own title-row button (opens the per-node -// field picker, rather than binding one specific field the way every -// FactorBindableField instance below does) -- shares the exact same visual -// identity (icon, text, violet accent, Tooltip) so every "this makes a -// factor" control in the app reads as the same kind of thing regardless of -// which of the two entry points it is. -export function MakeNodeFactorButton({ onClick }: { onClick: () => void }) { - return ( - - - }> - - Make factor - - Bind one of this node's fields to an experimental factor - - - ) -} - // 'text' levelType (a long-form value, e.g. a full system prompt) and every // structured kind (isStructuredLevelType -- the "whole node as a factor" // ones plus tool_names, see factorLevels.ts) escalate straight to @@ -111,7 +91,7 @@ export function FactorBindableField({ // The field's own current value, e.g. config.system_prompt -- omitted for // a boolean field, since its levels are always the fixed [true, false]. currentValue?: unknown - // The field's own already-fetched choices (e.g. LlmNodeInspector's model/ + // The field's own already-fetched choices (e.g. ModelNodeInspector's model/ // effort lists) -- when given, each level row renders as a Select over // these exact values instead of a freeform Input, so a factor's levels // can never drift from what the field itself actually accepts. Passed diff --git a/frontend/src/components/protocol/FactorEditorDialog.tsx b/frontend/src/components/protocol/FactorEditorDialog.tsx index 6029398..be1e767 100644 --- a/frontend/src/components/protocol/FactorEditorDialog.tsx +++ b/frontend/src/components/protocol/FactorEditorDialog.tsx @@ -29,7 +29,7 @@ import { } from './factorLevels' import { ModelField } from './ModelField' import { NODE_INSPECTOR_CONTENT_CLASSNAME } from './NodeInspectorDialog' -import { PROVIDER_META } from './nodes/LlmNode' +import { PROVIDER_META } from './nodes/ModelNode' import { PromptReferenceField } from './PromptReferenceField' import { PythonCodeEditor } from './PythonCodeEditor' import { useDialogAutosave } from './useDialogAutosave' @@ -43,10 +43,10 @@ const PATTERN_OPTIONS = [ type StructuredLevel = Record -// One row of an "llm_config" factor's levels -- mirrors LlmNodeInspector's +// One row of a "model_config" factor's levels -- mirrors ModelNodeInspector's // own Provider/Model/Temperature/Effort/Max tokens fields exactly, since a -// level here IS a whole LLM node's config (protocol_execution.py's -// _resolve_llm_config reads it verbatim, never the node's xyflow type). +// level here IS a whole Model node's config (protocol_execution.py's +// _resolve_model_config reads it verbatim, never the node's xyflow type). function LlmConfigLevelRow({ value, onChange }: { value: StructuredLevel; onChange: (next: StructuredLevel) => void }) { const provider = (value.provider as string) || 'anthropic' const { modelsQuery, models } = useProviderModels(provider) @@ -813,7 +813,7 @@ export function FactorEditorDialog({ value={levelLabels[i] ?? ''} onChange={(e) => setLevelLabels((ls) => ls.map((label, j) => (j === i ? e.target.value : label)))} /> - {levelType === 'llm_config' ? ( + {levelType === 'model_config' ? ( setLevels((ls) => ls.map((l, j) => (j === i ? next : l)))} diff --git a/frontend/src/components/protocol/ModelField.test.tsx b/frontend/src/components/protocol/ModelField.test.tsx new file mode 100644 index 0000000..7414ce4 --- /dev/null +++ b/frontend/src/components/protocol/ModelField.test.tsx @@ -0,0 +1,25 @@ +import { render, screen } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' +import { ModelField } from './ModelField' + +describe('ModelField', () => { + it('prompts for a model when no default is selected', () => { + render( + , + ) + + expect(screen.getByRole('combobox')).toHaveTextContent('Select a model…') + }) +}) diff --git a/frontend/src/components/protocol/ModelField.tsx b/frontend/src/components/protocol/ModelField.tsx index 80084d3..715f81b 100644 --- a/frontend/src/components/protocol/ModelField.tsx +++ b/frontend/src/components/protocol/ModelField.tsx @@ -7,9 +7,9 @@ import type { LLMModelInfo } from '@/types/llmSettings' const NONE_VALUE = '__none__' const CUSTOM_VALUE = '__custom__' -// The Model picker, shared by LlmNodeInspector and an "llm_config" factor +// The Model picker, shared by ModelNodeInspector and a "model_config" factor // level's own row (FactorEditorDialog's LlmConfigLevelRow) -- a factor level -// IS a whole LLM node config (protocol_execution.py's _resolve_llm_config +// IS a whole Model node config (protocol_execution.py's _resolve_model_config // reads it verbatim), so the two have to offer exactly the same choices or a // factor sweep can't express what a single node can. // @@ -93,7 +93,7 @@ export function ModelField({ }} > - {() => known?.label ?? value ?? 'Select a model…'} + {() => known?.label || value || 'Select a model…'} diff --git a/frontend/src/components/protocol/LlmNodeInspector.tsx b/frontend/src/components/protocol/ModelNodeInspector.tsx similarity index 95% rename from frontend/src/components/protocol/LlmNodeInspector.tsx rename to frontend/src/components/protocol/ModelNodeInspector.tsx index fbe9f5b..0600030 100644 --- a/frontend/src/components/protocol/LlmNodeInspector.tsx +++ b/frontend/src/components/protocol/ModelNodeInspector.tsx @@ -12,25 +12,25 @@ import { Skeleton } from '@/components/ui/skeleton' import { FactorBindableField } from './FactorBindableField' import { ModelField } from './ModelField' import { NodeInspectorDialog } from './NodeInspectorDialog' -import { PROVIDER_META } from './nodes/LlmNode' +import { PROVIDER_META } from './nodes/ModelNode' import { useProviderModels } from './useProviderModels' -import type { LlmNodeConfig, LlmNodeData, ProtocolNode } from '@/types/protocols' +import type { ModelNodeConfig, ModelNodeData, ProtocolNode } from '@/types/protocols' import type { LLMProvider } from '@/types/llmSettings' const EFFORT_LEVELS = ['low', 'medium', 'high', 'xhigh', 'max'] as const -// Shared by all three LLM provider node types (llm_anthropic/llm_openai/ -// llm_azure_foundry) -- fields are identical across providers (see -// LlmNodeData's own comment in types/protocols.ts), only the Credential +// Shared by all three LLM provider node types (model_anthropic/model_openai/ +// model_azure_foundry) -- fields are identical across providers (see +// ModelNodeData's own comment in types/protocols.ts), only the Credential // section's content differs, branched on config.provider below. Model/ // Temperature/Effort/Max tokens are exactly the fields AgentNodeInspector/ // CriticGateNodeInspector used to have, relocated here -- this is now the // ONLY place that config lives, resolved at execution time via the agent/ -// critic_gate's required LLM connector (services.protocol_execution's -// _resolve_llm_config). The free-text "Provider" field is gone entirely -- +// critic_gate's required Model connector (services.protocol_execution's +// _resolve_model_config). The free-text "Provider" field is gone entirely -- // provider is fixed by which node type you picked from the "+" panel, not a // field you fill in. -export function LlmNodeInspector({ +export function ModelNodeInspector({ node, experimentId, factorNodeLabel, @@ -38,21 +38,21 @@ export function LlmNodeInspector({ onDelete, onClose, }: { - node: (ProtocolNode & { data: LlmNodeData }) | null + node: (ProtocolNode & { data: ModelNodeData }) | null experimentId: string | null // The agent-traced display label (see bindableFields.ts's // agentTracedLabel) -- distinct from data.label/meta.label, which is this // node's own plain label/provider name shown in the header title. Two - // different agents' LLM nodes can share the exact same plain label (e.g. + // different agents' Model nodes can share the exact same plain label (e.g. // both "Anthropic"), so factor names need this instead to stay // unambiguous. factorNodeLabel: string - onChange: (nodeId: string, data: LlmNodeData) => void + onChange: (nodeId: string, data: ModelNodeData) => void onDelete: (nodeId: string) => void onClose: () => void }) { const [credentialDialogOpen, setCredentialDialogOpen] = useState(false) - // Shown instead of closing outright when a required field (see LlmNode.tsx's + // Shown instead of closing outright when a required field (see ModelNode.tsx's // matching warning-triangle check) is still empty -- lets the user close // anyway rather than trapping them in the inspector, but makes sure they // saw it first. Same convention as ReasonActPatternNodeInspector. @@ -103,7 +103,7 @@ export function LlmNodeInspector({ const bindings = data.factor_bindings ?? {} const meta = PROVIDER_META[provider!] ?? { label: provider, icon: Sparkles } const Icon = meta.icon - const ACCENT = nodeAccent('llm') + const ACCENT = nodeAccent('model') // Unrecognized model (list still loading, discovery failed, or a // hand-typed value not in the catalog) -- default to temperature-only, @@ -133,7 +133,7 @@ export function LlmNodeInspector({ onClose() } - function patchConfig(patch: Partial) { + function patchConfig(patch: Partial) { onChange(node!.id, { ...data, config: { ...config, ...patch } }) } @@ -169,7 +169,7 @@ export function LlmNodeInspector({ fieldPath="config" defaultLabel="Provider & model" nodeLabel={factorNodeLabel} - levelType="llm_config" + levelType="model_config" currentValue={config} boundFactorName={bindings.config} onBind={(name) => bindFactor('config', name)} diff --git a/frontend/src/components/protocol/NodeRunOutputPanel.tsx b/frontend/src/components/protocol/NodeRunOutputPanel.tsx index 2537394..251b61a 100644 --- a/frontend/src/components/protocol/NodeRunOutputPanel.tsx +++ b/frontend/src/components/protocol/NodeRunOutputPanel.tsx @@ -4,7 +4,7 @@ import { ChevronDown, ChevronRight } from 'lucide-react' import { runsApi } from '@/api/client' import { nodeRunBadge } from '@/lib/protocolRun' import { referenceLabel } from '@/lib/promptReferences' -import { hashToChartHue } from '@/lib/utils' +import { cn, hashToChartHue } from '@/lib/utils' import type { NodeRunState } from '@/types/protocols' import type { RunStep } from '@/types/runs' @@ -266,6 +266,7 @@ export function NodeRunOutputPanel({ nodeRun, referenceNames = {}, showReceivedPrompt = true, + resizableOutput = false, }: { nodeRun: NodeRunState | undefined // Node id -> display name, for naming a reference that resolved empty. The @@ -277,6 +278,9 @@ export function NodeRunOutputPanel({ // Agent inspector) -- what an agent was handed is input, and showing it in // both columns would say the split means less than it does. showReceivedPrompt?: boolean + // Agent and Sub-Agent inspectors give the final answer its own draggable + // viewport, matching the resize affordance on their prompt textareas. + resizableOutput?: boolean }) { const badge = nodeRunBadge(nodeRun?.status, Boolean(nodeRun?.truncation)) const unresolved = nodeRun?.unresolved_references ?? [] @@ -345,7 +349,15 @@ export function NodeRunOutputPanel({ through, not text the critic itself wrote -- the Verdict block above is the critic's own contribution. */}

{isGateRun ? 'Passed-through output' : 'Output'}

-

{nodeRun.output_text}

+

+ {nodeRun.output_text} +

) : (

No output yet.

diff --git a/frontend/src/components/protocol/ProtocolCanvas.test.tsx b/frontend/src/components/protocol/ProtocolCanvas.test.tsx index 9cb57a8..cba5297 100644 --- a/frontend/src/components/protocol/ProtocolCanvas.test.tsx +++ b/frontend/src/components/protocol/ProtocolCanvas.test.tsx @@ -6,7 +6,16 @@ import { MemoryRouter } from 'react-router-dom' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { experimentsApi, protocolsApi } from '@/api/client' import { protocolGraphQueryKey } from '@/lib/protocolGraph' -import { defaultAgentNodeData, defaultScriptNodeData, type ProtocolGraph } from '@/types/protocols' +import { + defaultAgentNodeData, + defaultAnthropicModelNodeData, + defaultCriticGateNodeData, + defaultMemoryNodeData, + defaultOutputParserNodeData, + defaultReasonActPatternNodeData, + defaultScriptNodeData, + type ProtocolGraph, +} from '@/types/protocols' import { ProtocolCanvas } from './ProtocolCanvas' vi.mock('./PythonCodeEditor', () => ({ @@ -126,6 +135,77 @@ describe('ProtocolCanvas connector adds', () => { expect(screen.queryByText('Parser')).not.toBeInTheDocument() }) + it('makes occupied single-capacity connector handles non-connectable', async () => { + const agentData = defaultAgentNodeData('Writer') + agentData.config.require_output_parser = true + renderCanvas({ + nodes: [ + { id: 'agent-1', type: 'agent', position: { x: 100, y: 100 }, data: agentData }, + { id: 'gate-1', type: 'critic_gate', position: { x: 400, y: 100 }, data: defaultCriticGateNodeData() }, + { id: 'model-1', type: 'model_anthropic', position: { x: 0, y: 0 }, data: defaultAnthropicModelNodeData() }, + { id: 'memory-1', type: 'memory', position: { x: 0, y: 0 }, data: defaultMemoryNodeData() }, + { id: 'pattern-1', type: 'pattern_reason_act', position: { x: 0, y: 0 }, data: defaultReasonActPatternNodeData() }, + { id: 'parser-1', type: 'output_parser', position: { x: 0, y: 0 }, data: defaultOutputParserNodeData() }, + ], + edges: [ + { id: 'model-agent', source: 'model-1', sourceHandle: 'model', target: 'agent-1', targetHandle: 'model' }, + { id: 'model-gate', source: 'model-1', sourceHandle: 'model', target: 'gate-1', targetHandle: 'model' }, + { id: 'memory-agent', source: 'memory-1', sourceHandle: 'memory', target: 'agent-1', targetHandle: 'memory' }, + { id: 'pattern-agent', source: 'pattern-1', sourceHandle: 'architectural_pattern', target: 'agent-1', targetHandle: 'architectural_pattern' }, + { id: 'parser-agent', source: 'parser-1', sourceHandle: 'output_parser', target: 'agent-1', targetHandle: 'output_parser' }, + ], + }) + + await screen.findByText('Writer') + for (const handleId of ['model', 'memory', 'architectural_pattern', 'output_parser']) { + const handle = document.querySelector(`[data-nodeid="agent-1"][data-handleid="${handleId}"]`) + expect(handle).toBeInTheDocument() + expect(handle).not.toHaveClass('connectable') + } + expect(document.querySelector('[data-nodeid="gate-1"][data-handleid="model"]')).not.toHaveClass('connectable') + expect(document.querySelector('[data-nodeid="model-1"][data-handleid="model"]')).toHaveClass('connectable') + }) + + it('adds a connector-only Sub-Agent from an Agent', async () => { + const user = userEvent.setup() + const { client } = renderCanvas({ + nodes: [{ id: 'agent-1', type: 'agent', position: { x: 100, y: 100 }, data: defaultAgentNodeData('Planner') }], + edges: [], + }) + + fireEvent.click(await screen.findByTitle('Add Sub-Agents')) + await user.click(await screen.findByRole('button', { name: /^Sub-Agent A delegated worker/ })) + + await waitFor(() => { + const graph = client.getQueryData(protocolGraphQueryKey('protocol-1')) + const child = graph?.nodes.find((node) => node.type === 'sub_agent') + const patternEdge = graph?.edges.find( + (edge) => edge.target === child?.id && edge.targetHandle === 'architectural_pattern', + ) + const pattern = graph?.nodes.find((node) => node.id === patternEdge?.source) + expect(child).toBeDefined() + expect(graph?.edges).toContainEqual(expect.objectContaining({ + source: child!.id, + sourceHandle: 'sub_agents', + target: 'agent-1', + targetHandle: 'sub_agents', + })) + expect(child!.position.x).toBe(100) + expect(child!.position.y).toBeGreaterThanOrEqual(400) + expect(pattern).toBeDefined() + expect(pattern!.position.x).toBeGreaterThanOrEqual(child!.position.x - 50) + expect(pattern!.position.x).toBeLessThanOrEqual(child!.position.x + 50) + expect(pattern!.position.y).toBeGreaterThan(100) + expect(pattern!.position.y).toBeLessThan(child!.position.y) + }) + + expect(screen.getByText('Parent')).toBeInTheDocument() + expect(screen.getAllByText('Sub-Agent').length).toBeGreaterThan(0) + expect(screen.queryByText('Input')).not.toBeInTheDocument() + expect(screen.getByText('Output')).toBeInTheDocument() + expect(screen.queryByText('Nothing downstream — this is the final output.')).not.toBeInTheDocument() + }) + it('does not expose custom metric controls in the Script inspector', async () => { const scriptData = defaultScriptNodeData('Score script') scriptData.config.code = 'def evaluate(output): return 1' @@ -205,4 +285,25 @@ describe('ProtocolCanvas connector adds', () => { expect(screen.getByText('Factor name')).toBeInTheDocument() }) + it('binds the Agent inspector header action directly to Active', async () => { + const user = userEvent.setup() + vi.spyOn(experimentsApi, 'get').mockResolvedValue({ + id: 'experiment-1', name: 'Experiment', description: null, hypothesis: null, design_type: 'factorial', task_brief: null, + design_spec: { factors: [], metrics: [] }, measurement_plan: null, dataset_ids: [], dataset_id: null, + locked_at: null, locked_protocol_revision_id: null, locked_design_spec: null, locked_measurement_plan: null, + created_at: '', updated_at: '', archived_at: null, + }) + renderCanvas({ + nodes: [{ id: 'agent-1', type: 'agent', position: { x: 100, y: 100 }, data: defaultAgentNodeData('Writer') }], + edges: [], + }, 'experiment-1') + + fireEvent.doubleClick(await screen.findByText('Writer')) + await user.click(screen.getAllByRole('button', { name: 'Make experimental factor' })[0]) + + expect(await screen.findByText('Writer:Active')).toBeInTheDocument() + expect(screen.getByText('Levels: true, false')).toBeInTheDocument() + expect(screen.queryByText('Bind to a field on the canvas')).not.toBeInTheDocument() + }) + }) diff --git a/frontend/src/components/protocol/ProtocolCanvas.tsx b/frontend/src/components/protocol/ProtocolCanvas.tsx index eb3a016..9b27f01 100644 --- a/frontend/src/components/protocol/ProtocolCanvas.tsx +++ b/frontend/src/components/protocol/ProtocolCanvas.tsx @@ -3,6 +3,7 @@ import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' import { addEdge, Background, + MarkerType, MiniMap, ReactFlow, useEdgesState, @@ -28,15 +29,15 @@ import { nodeDisplayNames } from '@/lib/nodeNames' import { raiseForTruncation, suggestedMaxIterations } from '@/lib/reasonActIterations' import { defaultAgentNodeData, - defaultAnthropicLlmNodeData, - defaultAzureFoundryLlmNodeData, + defaultAnthropicModelNodeData, + defaultAzureFoundryModelNodeData, defaultCriticGateNodeData, defaultDatasetNodeData, - defaultLocalLlmNodeData, + defaultLocalModelNodeData, defaultMcpToolNodeData, defaultMemoryNodeData, - defaultOpenAiLlmNodeData, - defaultOpenRouterLlmNodeData, + defaultOpenAiModelNodeData, + defaultOpenRouterModelNodeData, defaultOutputParserNodeData, defaultReasonActPatternNodeData, defaultScriptNodeData, @@ -46,7 +47,7 @@ import type { AgentNodeData, CriticGateNodeData, DatasetNodeData, - LlmNodeData, + ModelNodeData, McpToolNodeData, MemoryNodeData, NodeRunState, @@ -77,8 +78,14 @@ import { DatasetNodeInspector } from './DatasetNodeInspector' import { DeleteNodeConfirmDialog } from './DeleteNodeConfirmDialog' import { DEFAULT_ZOOM } from './constants' import { FactorEditorDialog } from './FactorEditorDialog' -import { CONNECTOR_CHILD_CLEARANCE, connectorNodeOffsetX, findFreePosition, tidyLayout } from './layout' -import { LlmNodeInspector } from './LlmNodeInspector' +import { + CONNECTOR_CHILD_CLEARANCE, + SUB_AGENT_CHILD_OFFSET_Y, + connectorNodeOffsetX, + findFreePosition, + tidyLayout, +} from './layout' +import { ModelNodeInspector } from './ModelNodeInspector' import { DatasetBrowserPanel } from './DatasetBrowserPanel' import { DATASET_BROWSE, nodeDataForDataset } from './datasetCatalog' import { McpServerBrowserPanel } from './McpServerBrowserPanel' @@ -119,7 +126,7 @@ import { InteractEdge } from './edges/InteractEdge' import { AgentNode } from './nodes/AgentNode' import { CriticGateNode } from './nodes/CriticGateNode' import { DatasetNode } from './nodes/DatasetNode' -import { LlmNode } from './nodes/LlmNode' +import { ModelNode } from './nodes/ModelNode' import { McpClientToolNode } from './nodes/McpClientToolNode' import { McpToolNode } from './nodes/McpToolNode' import { MemoryNode } from './nodes/MemoryNode' @@ -131,17 +138,16 @@ import { OkfBundleNode } from './nodes/OkfBundleNode' import { OkfDocumentNode } from './nodes/OkfDocumentNode' import { SkillNode } from './nodes/SkillNode' import { ProtocolCanvasMenu } from './ProtocolCanvasMenu' +import { + MODEL_NODE_TYPES, + PATTERN_NODE_TYPES, + isProtocolConnectionValid, +} from './connectionValidation' -// One node type per LLM provider / architectural pattern (see LlmNodeData/ +// One node type per LLM provider / architectural pattern (see ModelNodeData/ // ReasonActPatternNodeData's own comments in types/protocols.ts) -- each // connector slot accepts this whole family, not one exact type, mirroring // how the "tool" slot already accepts any mcp_tool node. -const LLM_NODE_TYPES = ['llm_anthropic', 'llm_openai', 'llm_azure_foundry', 'llm_openrouter', 'llm_local'] -const PATTERN_NODE_TYPES = ['pattern_reason_act', 'pattern_single_agent_baseline'] -// The Knowledge slot's family, mirroring _KNOWLEDGE_NODE_TYPES in -// services/protocol_execution.py: a server-side folder or an uploaded single -// concept, both resolved identically into the agent's tool allow-list. -const KNOWLEDGE_NODE_TYPES = ['okf_bundle', 'okf_document'] // The four connector slots that live on an Agent's TOP edge (see // AgentNode.tsx) -- a node feeding one of these is placed ABOVE its agent, // every other slot's source below it. @@ -149,6 +155,7 @@ const TOP_EDGE_SLOTS = new Set(['architectural_pattern', 'skill', const NODE_TYPES = { agent: AgentNode, + sub_agent: AgentNode, // Both MCP-tool types render through the same component (as the five LLM // provider types do) -- they carry identical data and differ only in // whether their server was picked in the browser or in a dropdown. @@ -162,11 +169,11 @@ const NODE_TYPES = { // All five LLM provider types render through the same component -- it // derives icon/accent/placeholder from data.config.provider, not from // which of these five keys it was registered under. - llm_anthropic: LlmNode, - llm_openai: LlmNode, - llm_azure_foundry: LlmNode, - llm_openrouter: LlmNode, - llm_local: LlmNode, + model_anthropic: ModelNode, + model_openai: ModelNode, + model_azure_foundry: ModelNode, + model_openrouter: ModelNode, + model_local: ModelNode, memory: MemoryNode, output_parser: OutputParserNode, dataset: DatasetNode, @@ -209,18 +216,18 @@ function defaultDataFor(nodeType: string): ProtocolNode['data'] { // it. if (nodeType === 'mcp_tool') return defaultMcpToolNodeData() if (nodeType === 'critic_gate') return defaultCriticGateNodeData() - if (nodeType === 'llm_anthropic') return defaultAnthropicLlmNodeData() - if (nodeType === 'llm_openai') return defaultOpenAiLlmNodeData() - if (nodeType === 'llm_azure_foundry') return defaultAzureFoundryLlmNodeData() - if (nodeType === 'llm_openrouter') return defaultOpenRouterLlmNodeData() - if (nodeType === 'llm_local') return defaultLocalLlmNodeData() + if (nodeType === 'model_anthropic') return defaultAnthropicModelNodeData() + if (nodeType === 'model_openai') return defaultOpenAiModelNodeData() + if (nodeType === 'model_azure_foundry') return defaultAzureFoundryModelNodeData() + if (nodeType === 'model_openrouter') return defaultOpenRouterModelNodeData() + if (nodeType === 'model_local') return defaultLocalModelNodeData() if (nodeType === 'memory') return defaultMemoryNodeData() if (nodeType === 'output_parser') return defaultOutputParserNodeData() if (nodeType === 'dataset') return defaultDatasetNodeData() if (nodeType === 'script') return defaultScriptNodeData() if (nodeType === 'pattern_reason_act') return defaultReasonActPatternNodeData() if (nodeType === 'pattern_single_agent_baseline') return defaultSingleAgentBaselinePatternNodeData() - return defaultAgentNodeData() + return defaultAgentNodeData(nodeType === 'sub_agent' ? 'Sub-Agent' : 'Agent') } // The registered datasets this canvas declares, in the order their nodes were @@ -265,7 +272,7 @@ function datasetIdsInGraph(nodes: Node[], factors: DesignFactor[] = EMPTY_FACTOR // Mirrors isValidConnection's own per-slot source-type-family rule -- the // panel that opens for a connector "+" is pre-filtered to that slot's whole -// family of node types (LLM_NODE_TYPES/PATTERN_NODE_TYPES above) rather than +// family of node types (MODEL_NODE_TYPES/PATTERN_NODE_TYPES above) rather than // the full catalog. Tool's own family includes Script alongside mcp_tool // (one connector accepting several kinds of node -- see AgentNode.tsx's own // comment on its Tool handle): a Script is a pure config source with no @@ -273,7 +280,7 @@ function datasetIdsInGraph(nodes: Node[], factors: DesignFactor[] = EMPTY_FACTOR // getting a dedicated one. Dataset used to share it too, but now has its // own slot -- what an agent operates ON, not a capability it operates WITH. const CONNECTOR_PANEL_INFO: Record = { - ai: { allowedTypes: LLM_NODE_TYPES, title: 'Add AI' }, + model: { allowedTypes: MODEL_NODE_TYPES, title: 'Add Model' }, tool: { allowedTypes: [MCP_SERVER_BROWSE, 'script'], title: 'Add Tool' }, memory: { allowedTypes: ['memory'], title: 'Add Memory' }, output_parser: { allowedTypes: ['output_parser'], title: 'Add Output Parser' }, @@ -284,6 +291,7 @@ const CONNECTOR_PANEL_INFO: Record "ai", on every edge, when the connector's caption became "AI". +// * "llm" -> "ai" -> "model", on every edge as the connector vocabulary evolved. +// * "llm_*" -> "model_*", for the five provider-node discriminators. // * "tool" -> "resource" for DATASET-sourced edges only -- Dataset used to // share the Tool slot with mcp_tool/script, which both keep "tool". Hence // the source-type check rather than a blanket swap. @@ -315,14 +324,35 @@ function parserPositionFor(agent: Node, otherNodes: Node[]) { // the right handle and the next autosave persists the fix. This is one of // three layers, none of them load-bearing alone: Alembic data migrations // (3f1a7c9b2e04, b7c2d9e14a35) make stored graphs canonical, the backend keeps -// resolving the old spellings (_LEGACY_AI_HANDLES / _LEGACY_DATASET_HANDLES in +// resolving the old spellings (_LEGACY_MODEL_HANDLES / _LEGACY_DATASET_HANDLES in // services/protocol_execution.py) so a graph that's never opened still runs, // and this covers a tab that loaded before the deploy and is still autosaving // old-spelling edges. +const LEGACY_MODEL_NODE_TYPES: Record = { + llm_anthropic: 'model_anthropic', + llm_openai: 'model_openai', + llm_azure_foundry: 'model_azure_foundry', + llm_openrouter: 'model_openrouter', + llm_local: 'model_local', +} + +function migrateLegacyNodes(graph: ProtocolGraph): Node[] { + return (graph.nodes as Node[]).map((node) => { + const type = LEGACY_MODEL_NODE_TYPES[node.type ?? ''] + return type ? { ...node, type } : node + }) +} + function migrateLegacyHandles(graph: ProtocolGraph): Edge[] { const datasetIds = new Set(graph.nodes.filter((n) => n.type === 'dataset').map((n) => n.id)) return (graph.edges as Edge[]).map((e) => { - if (e.targetHandle === 'llm') return { ...e, sourceHandle: 'ai', targetHandle: 'ai' } + if (e.targetHandle === 'ai' || e.targetHandle === 'llm' || e.sourceHandle === 'ai' || e.sourceHandle === 'llm') { + return { + ...e, + sourceHandle: e.sourceHandle === 'ai' || e.sourceHandle === 'llm' ? 'model' : e.sourceHandle, + targetHandle: e.targetHandle === 'ai' || e.targetHandle === 'llm' ? 'model' : e.targetHandle, + } + } if (e.targetHandle === 'resource' || (e.targetHandle === 'tool' && datasetIds.has(e.source))) { return { ...e, sourceHandle: 'dataset', targetHandle: 'dataset' } } @@ -365,7 +395,7 @@ export const ProtocolCanvas = forwardRef(function ProtocolCanvas({ protocolId, experimentId, initialGraph, hasUnpublishedChanges, publishedRevision, experimentLocked = false }, canvasHandleRef) { - const [nodes, setNodes, onNodesChange] = useNodesState(initialGraph.nodes as Node[]) + const [nodes, setNodes, onNodesChange] = useNodesState(migrateLegacyNodes(initialGraph)) const [edges, setEdges, onEdgesChange] = useEdgesState(migrateLegacyHandles(initialGraph)) const queryClient = useQueryClient() @@ -515,10 +545,6 @@ export const ProtocolCanvas = forwardRef { - if (!experimentLocked) setEdges((eds) => addEdge(connection, eds)) - }, [experimentLocked, setEdges]) - // "Tidy up" -- reposition every node into a generated layout (see // layout.ts's tidyLayout). Goes through this component's own setNodes // rather than useReactFlow().setNodes because the flow is controlled, and @@ -731,11 +757,30 @@ export const ProtocolCanvas = forwardRef new Set(edges.filter((e) => e.targetHandle === 'ai').map((e) => e.target)), [edges]) + const agentIdsWithModel = useMemo(() => new Set(edges.filter((e) => e.targetHandle === 'model').map((e) => e.target)), [edges]) const agentIdsWithParser = useMemo( () => new Set(edges.filter((e) => e.targetHandle === 'output_parser').map((e) => e.target)), [edges], ) + const agentIdsWithSubAgents = useMemo( + () => { + const activeSubAgents = new Set( + nodes.filter((node) => node.type === 'sub_agent' && node.data.active !== false).map((node) => node.id), + ) + return new Set( + edges.filter((edge) => edge.targetHandle === 'sub_agents' && activeSubAgents.has(edge.source)).map((edge) => edge.target), + ) + }, + [edges, nodes], + ) + const connectedActiveSubAgentIds = useMemo(() => { + const activeSubAgents = new Set( + nodes.filter((node) => node.type === 'sub_agent' && node.data.active !== false).map((node) => node.id), + ) + return new Set( + edges.filter((edge) => edge.targetHandle === 'sub_agents' && activeSubAgents.has(edge.source)).map((edge) => edge.source), + ) + }, [edges, nodes]) // The canvas's per-node Play icon is only offered for a node with no // upstream *main* pipeline edge (mirrors services.protocol_execution's @@ -753,7 +798,7 @@ export const ProtocolCanvas = forwardRef { const nodeTypeById = new Map(nodes.map((n) => [n.id, n.type])) + const activeSubAgents = new Set( + nodes.filter((node) => node.type === 'sub_agent' && node.data.active !== false).map((node) => node.id), + ) return new Set( edges .filter((e) => { + if (e.targetHandle === 'sub_agents') return activeSubAgents.has(e.source) if (e.targetHandle === 'knowledge' || e.targetHandle === 'skill') return true if (e.targetHandle !== 'tool') return false const sourceType = nodeTypeById.get(e.source) @@ -926,7 +975,7 @@ export const ProtocolCanvas = forwardRef [n.id, n])) const map = new Map() for (const e of edges) { - if (e.targetHandle !== 'ai') continue - const config = (nodeById.get(e.source)?.data as LlmNodeData | undefined)?.config + if (e.targetHandle !== 'model') continue + const config = (nodeById.get(e.source)?.data as ModelNodeData | undefined)?.config if (config) map.set(e.target, { provider: config.provider, model: config.model }) } return map @@ -948,6 +997,7 @@ export const ProtocolCanvas = forwardRef nodeDisplayNames(nodes), [nodes]) + const nodeTypes = useMemo(() => new Map(nodes.map((node) => [node.id, node.type ?? ''])), [nodes]) // The experiment's declared coordination strategy, which decides what the // main handles MEAN -- whether a lead marker is in force, and whether the @@ -960,6 +1010,41 @@ export const ProtocolCanvas = forwardRef { + if (experimentLocked) return + setEdges((currentEdges) => + isProtocolConnectionValid(connection, nodes, currentEdges, isSequential) + ? addEdge(connection, currentEdges) + : currentEdges, + ) + }, [experimentLocked, isSequential, nodes, setEdges]) + + const renderedEdges = useMemo(() => { + if (!isSequential) return edges + const agentIds = new Set(nodes.filter((node) => node.type === 'agent').map((node) => node.id)) + return edges.map((edge) => { + const isAgentFlow = + !edge.sourceHandle && + !edge.targetHandle && + agentIds.has(edge.source) && + agentIds.has(edge.target) + if (!isAgentFlow) return edge + return { + ...edge, + data: { ...edge.data, sequentialAgentFlow: true }, + markerEnd: { + type: MarkerType.ArrowClosed, + width: 12, + height: 12, + color: 'color-mix(in oklch, var(--muted-foreground), transparent 30%)', + }, + } + }) + }, [edges, isSequential, nodes]) + // Which main-flow sides are already taken. Only consulted under // 'sequential', where the chain rule caps each side at one edge // (validate_sequential_chain), so the "+" stub can hide instead of offering @@ -978,28 +1063,34 @@ export const ProtocolCanvas = forwardRef { return nodes.map((n) => { const patternHostId = patternHostIds.get(n.id) + const isAgentLike = n.type === 'agent' || n.type === 'sub_agent' return { ...n, deletable: !nonDeletablePatternNodeIds.has(n.id), data: { ...n.data, + isSubAgent: n.type === 'sub_agent', runStatus: latestNodeRuns?.[n.id]?.status, runTruncated: Boolean(latestNodeRuns?.[n.id]?.truncation), - missingLlm: n.type === 'agent' && !agentIdsWithLlm.has(n.id), + missingModel: + (n.type === 'agent' || (n.type === 'sub_agent' && connectedActiveSubAgentIds.has(n.id))) && + !agentIdsWithModel.has(n.id), // "Require specific output format" is on, but nothing says what the - // format is. Unlike missingLlm this doesn't stop the run -- the agent + // format is. Unlike missingModel this doesn't stop the run -- the agent // just answers in prose, which is the outcome the switch was flipped // to prevent, so it has to be visible on the card and not only in the // inspector the user has already closed. A legacy stored contract // counts as the answer: the executor falls back to it. missingOutputParser: - n.type === 'agent' && + isAgentLike && (n.data as AgentNodeData).config?.require_output_parser === true && !agentIdsWithParser.has(n.id) && !(n.data as AgentNodeData).config?.output_contract, - canRunAlone: n.type === 'agent' && !agentIdsWithUpstream.has(n.id), - hasPeers: n.type === 'agent' && (peerIdsByAgent.get(n.id)?.length ?? 0) > 0, - llmConfig: n.type === 'agent' ? llmConfigByAgent.get(n.id) ?? null : null, + canRunAlone: isAgentLike && !agentIdsWithUpstream.has(n.id), + hasPeers: + n.type === 'agent' && + ((peerIdsByAgent.get(n.id)?.length ?? 0) > 0 || agentIdsWithSubAgents.has(n.id)), + llmConfig: isAgentLike ? llmConfigByAgent.get(n.id) ?? null : null, // Gated on the strategy, not just the flag: a "Lead" badge left over // from a Peer Collaboration experiment that has since been switched // to Sequential would claim a role nothing acts on. The flag itself @@ -1043,8 +1134,10 @@ export const ProtocolCanvas = forwardRef { if (!experimentId || experimentLocked) return @@ -1397,7 +1485,7 @@ export const ProtocolCanvas = forwardRef n.position), desired, CONNECTOR_CHILD_CLEARANCE) + const position = findFreePosition( + nodes.map((n) => n.position), + desired, + slot === 'sub_agents' ? { width: 320, height: 140 } : CONNECTOR_CHILD_CLEARANCE, + ) const newId = newNodeId() // Execution pattern is capped at one but must never go to zero (see // AgentNode.tsx's own comment) -- its "+" stays visible even once @@ -1420,16 +1514,24 @@ export const ProtocolCanvas = forwardRef e.target === originId && e.targetHandle === 'architectural_pattern') : undefined - setNodes((nds) => - nds - .filter((n) => n.id !== existingPatternEdge?.source) - .concat({ id: newId, type: nodeType, position, data: dataOverride ?? defaultDataFor(nodeType) }), - ) - setEdges((eds) => - eds - .filter((e) => e.id !== existingPatternEdge?.id) - .concat({ id: newNodeId(), source: newId, sourceHandle: slot, target: originId, targetHandle: slot }), - ) + const connectorNode: Node = { id: newId, type: nodeType, position, data: dataOverride ?? defaultDataFor(nodeType) } + const ownerEdge: Edge = { id: newNodeId(), source: newId, sourceHandle: slot, target: originId, targetHandle: slot } + if (nodeType === 'sub_agent') { + const { patternNode, patternEdge } = agentDefaultPattern(newId, position, nodes.map((n) => n.position)) + setNodes((nds) => nds.concat(connectorNode, patternNode)) + setEdges((eds) => eds.concat(ownerEdge, patternEdge)) + } else { + setNodes((nds) => + nds + .filter((n) => n.id !== existingPatternEdge?.source) + .concat(connectorNode), + ) + setEdges((eds) => + eds + .filter((e) => e.id !== existingPatternEdge?.id) + .concat(ownerEdge), + ) + } setPendingConnectorAdd(null) setAddPanelOpen(false) // Picking a node from the connector panel goes straight into that @@ -1528,7 +1630,7 @@ export const ProtocolCanvas = forwardRef n.type === 'agent' && !edges.some((e) => e.target === n.id && e.targetHandle === 'output_parser'), + (n) => (n.type === 'agent' || n.type === 'sub_agent') && !edges.some((e) => e.target === n.id && e.targetHandle === 'output_parser'), ) const asking = parserless.filter((n) => (n.data as AgentNodeData).config?.require_output_parser === true) const host = (asking.length === 1 ? asking : parserless.length === 1 ? parserless : [])[0] @@ -1717,7 +1819,7 @@ export const ProtocolCanvas = forwardRef n.id === selectedNodeId) ?? null // Computed once per selection change, not per FactorBindableField -- an - // LLM/Tool/Memory node's plain label alone doesn't say which agent it + // Model/Tool/Memory node's plain label alone doesn't say which agent it // belongs to (see bindableFields.ts's own comment), so every inspector // that wraps a field in "+ Make experimental factor" gets this instead of // data.label for that purpose specifically; the header title itself still @@ -1757,7 +1859,7 @@ export const ProtocolCanvas = forwardRef { - const sourceNode = nodes.find((n) => n.id === connection.source) - const targetNode = nodes.find((n) => n.id === connection.target) - if (!sourceNode || !targetNode) return false - switch (connection.targetHandle) { - case 'ai': - return ( - LLM_NODE_TYPES.includes(sourceNode.type ?? '') && - (targetNode.type === 'agent' || targetNode.type === 'critic_gate') - ) - case 'tool': - return ( - (MCP_TOOL_NODE_TYPES.includes(sourceNode.type ?? '') || sourceNode.type === 'script') && - targetNode.type === 'agent' - ) - case 'memory': - return sourceNode.type === 'memory' && targetNode.type === 'agent' - case 'output_parser': - // Agent only, deliberately not critic_gate: a gate's answer is a - // pass/fail decision the executor already reads structurally, so - // there is nothing for a contract to extract. - return sourceNode.type === 'output_parser' && targetNode.type === 'agent' - case 'architectural_pattern': - return PATTERN_NODE_TYPES.includes(sourceNode.type ?? '') && targetNode.type === 'agent' - case 'skill': - return sourceNode.type === 'skill' && targetNode.type === 'agent' - case 'dataset': - // Uncapped: a cell's workspace holds one dataset per named SLOT, so - // several datasets on one agent is a supported shape, not a run-time - // error (see AgentNode.tsx's Dataset comment). Wiring order is the - // order the agent's prompt lists the slots in. Note this is still - // distinct from COMPARING datasets across cells, which is a - // 'dataset_config' factor. - return sourceNode.type === 'dataset' && targetNode.type === 'agent' - case 'knowledge': - // The one connector with two source types -- bundles and uploaded - // documents are interchangeable here, since both resolve to the same - // per-directory OKF server. - return KNOWLEDGE_NODE_TYPES.includes(sourceNode.type ?? '') && targetNode.type === 'agent' - default: { - // A plain "main" pipeline edge -- LLM/memory/pattern/mcp_tool/ - // dataset/skill/knowledge/script nodes have no main handle to drag from in - // the first place, so this mostly guards against a stray - // connection, not real interactive use. - const sourceCanFeedMainFlow = - !LLM_NODE_TYPES.includes(sourceNode.type ?? '') && - sourceNode.type !== 'memory' && - sourceNode.type !== 'output_parser' && - !MCP_TOOL_NODE_TYPES.includes(sourceNode.type ?? '') && - sourceNode.type !== 'dataset' && - sourceNode.type !== 'skill' && - !KNOWLEDGE_NODE_TYPES.includes(sourceNode.type ?? '') && - sourceNode.type !== 'script' && - !PATTERN_NODE_TYPES.includes(sourceNode.type ?? '') - if (!sourceCanFeedMainFlow) return false - // Under Sequential the main flow is a chain: one edge out of each - // node, one into each. Enforced here so the canvas refuses the fork - // as you draw it, rather than validate_sequential_chain rejecting the - // whole protocol at publish time. Other strategies leave the main - // flow unrestricted -- a fork is exactly the shape Peer Collaboration - // exists for, and a Critic Gate pipeline routes around agents. - if (!isSequential) return true - return ( - !edges.some((e) => e.source === connection.source && !CONNECTOR_HANDLES.has(e.targetHandle ?? '')) && - !edges.some((e) => e.target === connection.target && !CONNECTOR_HANDLES.has(e.targetHandle ?? '')) - ) - } - } - }, + (connection: Edge | Connection) => isProtocolConnectionValid(connection, nodes, edges, isSequential), [nodes, edges, isSequential], ) @@ -1886,7 +1920,7 @@ export const ProtocolCanvas = forwardRef
{testResultsOpen && testRunQuery.data && ( - setTestResultsOpen(false)} /> + setTestResultsOpen(false)} /> )} {playResultsOpen && playResult && ( - setPlayResultsOpen(false)} /> + setPlayResultsOpen(false)} /> )} {/* One top-left column rather than two independently-positioned overlays: the lock badge and the transcript are both anchored @@ -2132,9 +2166,9 @@ export const ProtocolCanvas = forwardRef setSelectedNodeId(null)} /> - ) : LLM_NODE_TYPES.includes(selectedNode?.type ?? '') ? ( - = { - ai: 'AI', + model: 'Model', tool: 'Tool', memory: 'Memory', architectural_pattern: 'Architectural Pattern', @@ -24,6 +25,7 @@ export const CONNECTOR_SLOT_LABELS: Record = { dataset: 'Dataset', knowledge: 'Knowledge', output_parser: 'Output Parser', + sub_agents: 'Sub-Agents', } export interface ConnectorAddRequest { @@ -70,12 +72,8 @@ interface ProtocolCanvasActions { // ever called for a node with no upstream input (see AgentNode.tsx's own // canRunAlone computation); the backend re-validates this regardless. requestRunNode: (nodeId: string) => void - // The canvas's per-node "Make experimental factor" icon (NodeHoverToolbar, - // last button, every node type) -- opens the same field-picker dialog as - // DesignTab's "Add factor," pre-filtered to this one node's own bindable - // fields. A no-op when the protocol has no linked experiment yet (nothing - // to attach a factor to), same as FactorBindableField's own disabled - // state for that case. + // Critic Gate's canvas-toolbar factor action. Other node fields bind + // directly through their inspector's FactorBindableField controls. requestMakeFactor: (nodeId: string) => void requestEditFactor: (factorName: string) => void // Custom metric declarations live on the experiment rather than in canvas @@ -89,7 +87,7 @@ interface ProtocolCanvasActions { // -- so an interrupted conversion can't leave a graph with the contract in // two places at once (which the backend rejects outright). // - // User-initiated and never automatic, unlike migrateLegacyHandles: that one + // User-initiated and never automatic, unlike migrateLegacyGraph: that one // rewrites an invisible handle string on an edge the user drew themselves, // whereas this one MATERIALISES A NODE the user never placed, and autosave // would then persist it. A graph is a document; nothing edits it on the diff --git a/frontend/src/components/protocol/PythonCodeEditor.tsx b/frontend/src/components/protocol/PythonCodeEditor.tsx index ca15d70..c813927 100644 --- a/frontend/src/components/protocol/PythonCodeEditor.tsx +++ b/frontend/src/components/protocol/PythonCodeEditor.tsx @@ -2,6 +2,7 @@ import CodeMirror, { EditorView } from '@uiw/react-codemirror' import { python } from '@codemirror/lang-python' import { syntaxHighlighting } from '@codemirror/language' import { oneDarkHighlightStyle } from '@codemirror/theme-one-dark' +import { cn } from '@/lib/utils' // A dark theme built from this app's own CSS custom properties (index.css's // .dark block) for the editor's CHROME (background, gutter, active line, @@ -11,7 +12,7 @@ import { oneDarkHighlightStyle } from '@codemirror/theme-one-dark' // itself, not its accompanying background theme) is used verbatim for // those -- a code editor's multi-hue token coloring is its own established // visual language, not something this app's single-accent color convention -// (CLAUDE.md's "Color -- meaningful variation, not decoration") is meant to +// (AGENTS.md's "Color -- meaningful variation, not decoration") is meant to // constrain, the same way a syntax-highlighted code block in a chat UI // doesn't reskin its colors to match the surrounding chrome either. const editorTheme = EditorView.theme( @@ -60,18 +61,29 @@ export function PythonCodeEditor({ value, onChange, rows = 16, + resizable = false, }: { value: string onChange: (value: string) => void rows?: number + resizable?: boolean }) { + const height = `${rows * 1.35}em` + return ( -
+
diff --git a/frontend/src/components/protocol/ReasonActPatternNodeInspector.tsx b/frontend/src/components/protocol/ReasonActPatternNodeInspector.tsx index c237952..d43b966 100644 --- a/frontend/src/components/protocol/ReasonActPatternNodeInspector.tsx +++ b/frontend/src/components/protocol/ReasonActPatternNodeInspector.tsx @@ -8,9 +8,8 @@ import { Input } from '@/components/ui/input' import { Label } from '@/components/ui/label' import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select' import { Switch } from '@/components/ui/switch' -import { FactorBindableField, MakeNodeFactorButton } from './FactorBindableField' +import { FactorBindableField } from './FactorBindableField' import { NodeInspectorDialog } from './NodeInspectorDialog' -import { useProtocolCanvasActions } from './ProtocolCanvasContext' import type { ReasonActPatternConfig, ReasonActPatternNodeData, ProtocolNode } from '@/types/protocols' const OBSERVATION_FORMATS = ['raw', 'summarized'] as const @@ -52,7 +51,6 @@ export function ReasonActPatternNodeInspector({ onChange: (nodeId: string, data: ReasonActPatternNodeData) => void onClose: () => void }) { - const { requestMakeFactor } = useProtocolCanvasActions() // Shown instead of closing outright when a required field (see // ReasonActPatternNode.tsx's matching warning-triangle check) is still // empty -- lets the user close anyway rather than trapping them in the @@ -107,7 +105,6 @@ export function ReasonActPatternNodeInspector({ <>

{data.label || 'Reason + Act'}

- requestMakeFactor(node.id)} /> } onClose={requestClose} diff --git a/frontend/src/components/protocol/ResultsTab.test.tsx b/frontend/src/components/protocol/ResultsTab.test.tsx index d2bf1af..95ea589 100644 --- a/frontend/src/components/protocol/ResultsTab.test.tsx +++ b/frontend/src/components/protocol/ResultsTab.test.tsx @@ -78,6 +78,67 @@ function orderedMetricsFixture() { } describe('results measurement states', () => { + it('shows explicit units for cost and duration in an individual replicate result', async () => { + const replicate = { + replicate_label: 'replicate-1', replicate_number: 1, cell_label: 'model_a', factor_values: { model: 'a' }, + metric_values: { cost_usd: 0.0123, duration_seconds: 65 }, status: 'completed' as const, obsolete: false, + error: null, run_id: 'run-1', protocol_revision_id: 'revision-1', updated_at: '2026-01-01T00:00:00Z', + duration_seconds: 65, node_runs: [], input_tokens: null, output_tokens: null, total_tokens: null, cost_usd: 0.0123, + agent_run_count: 0, reported_usage_count: 0, reported_cost_count: 0, metric_evaluation: null, + metric_observations: [], evaluation_artifacts: [], obsolete_runs: [], superseded_runs: [], + } + vi.mocked(experimentsApi.getRunResults).mockResolvedValue({ + overview, + metric_keys: ['cost_usd', 'duration_seconds'], + metric_types: { cost_usd: 'number', duration_seconds: 'number' }, + metric_aggregations: { cost_usd: 'sum', duration_seconds: 'sum' }, + metric_directions: { cost_usd: 'minimize', duration_seconds: 'minimize' }, + primary_metric: null, + primary_metric_direction: null, + cells: [], + replicates: [replicate], + } satisfies ExperimentRunResults) + + renderWithQuery() + + expect(await screen.findByText('$0.01')).toBeInTheDocument() + expect(screen.getByText('1.1 min')).toBeInTheDocument() + expect(screen.queryByRole('heading', { name: 'Outcome' })).not.toBeInTheDocument() + const usage = screen.getByRole('heading', { name: 'Usage' }).parentElement! + expect(within(usage).getAllByText(/^(Estimated cost|Duration|Total tokens|Agent calls)$/).map((label) => label.textContent)).toEqual([ + 'Estimated cost', + 'Duration', + 'Total tokens', + 'Agent calls', + ]) + }) + + it('shows a measured observation once in Outcome and keeps its provenance there', async () => { + const producer = { binding_id: 'judge', producer_id: 'asaree.judge', kind: 'reported' as const, version: '1' } + const replicate = { + replicate_label: 'replicate-1', replicate_number: 1, cell_label: 'model_a', factor_values: { model: 'a' }, + metric_values: { Quality: 0.9 }, status: 'completed' as const, obsolete: false, error: null, run_id: 'run-1', + protocol_revision_id: 'revision-1', updated_at: '2026-01-01T00:00:00Z', duration_seconds: null, + node_runs: [], input_tokens: null, output_tokens: null, total_tokens: null, cost_usd: null, + agent_run_count: 0, reported_usage_count: 0, reported_cost_count: 0, metric_evaluation: null, + metric_observations: [{ metric_id: 'quality', metric_name: 'Quality', value_type: 'number' as const, status: 'measured' as const, value: 0.9, error: null, attempt_id: 'run-1', producer, input_provenance: {} }], + evaluation_artifacts: [], obsolete_runs: [], superseded_runs: [], + } + vi.mocked(experimentsApi.getRunResults).mockResolvedValue({ + overview, + metric_keys: ['Quality'], metric_types: { Quality: 'number' }, metric_aggregations: { Quality: 'mean' }, + metric_directions: { Quality: 'maximize' }, primary_metric: null, primary_metric_direction: null, + cells: [], replicates: [replicate], + } satisfies ExperimentRunResults) + + renderWithQuery() + + expect(await screen.findByRole('heading', { name: 'Outcome' })).toBeInTheDocument() + expect(screen.getAllByText('0.9')).toHaveLength(1) + expect(screen.getByText('asaree.judge · v1')).toBeInTheDocument() + expect(screen.queryByRole('heading', { name: 'Measurement observations' })).not.toBeInTheDocument() + }) + it('presents metrics in the measurement-plan order declared by the designer', async () => { const { orderedExperiment, results } = orderedMetricsFixture() vi.mocked(experimentsApi.getRunResults).mockResolvedValue(results) @@ -101,6 +162,30 @@ describe('results measurement states', () => { ]) }) + it('shows condition-level cost and duration once under Usage', async () => { + vi.mocked(experimentsApi.getRunResults).mockResolvedValue({ + overview, + metric_keys: ['cost_usd', 'duration_seconds'], + metric_types: { cost_usd: 'number', duration_seconds: 'number' }, + metric_aggregations: { cost_usd: 'sum', duration_seconds: 'sum' }, + metric_directions: { cost_usd: 'minimize', duration_seconds: 'minimize' }, + primary_metric: null, + primary_metric_direction: null, + cells: [{ + cell_label: 'model_a', factor_values: { model: 'a' }, replicate_count: 2, completed_count: 2, + current_completed_count: 2, obsolete_count: 0, metric_means: { cost_usd: 0.0123, duration_seconds: 65 }, + metric_counts: { cost_usd: 2, duration_seconds: 2 }, cost_usd: 0.0123, total_tokens: null, duration_seconds: 65, + }], + replicates: [], + } satisfies ExperimentRunResults) + + renderWithQuery() + + expect(await screen.findByText('$0.01')).toBeInTheDocument() + expect(screen.getByText('1.1 min')).toBeInTheDocument() + expect(screen.queryByRole('heading', { name: 'Outcome metrics' })).not.toBeInTheDocument() + }) + it('ranks measured minimize values before missing values without skipping rank one', async () => { const results = { overview: { ...overview, total_replicates: 2, completed_replicates: 2 }, diff --git a/frontend/src/components/protocol/ResultsTab.tsx b/frontend/src/components/protocol/ResultsTab.tsx index 0795c9d..73d2c39 100644 --- a/frontend/src/components/protocol/ResultsTab.tsx +++ b/frontend/src/components/protocol/ResultsTab.tsx @@ -207,8 +207,17 @@ function Scorecard({ label, help, value, note, icon: Icon }: { label: string; he ) } +function usageSummarizesMetric( + key: string, + result: Pick, +): boolean { + return (key === 'cost_usd' && result.cost_usd !== null) + || (key === 'total_tokens' && result.total_tokens !== null) + || (key === 'duration_seconds' && result.duration_seconds !== null) +} + function CellResultSummary({ cell, metricKeys, metricTypes, metricAggregations }: { cell: ResultCell; metricKeys: string[]; metricTypes: ResultMetricTypes; metricAggregations: ResultMetricAggregations }) { - const metrics = metricKeys.filter((key) => typeof cell.metric_means[key] === 'number') + const metrics = metricKeys.filter((key) => typeof cell.metric_means[key] === 'number' && !usageSummarizesMetric(key, cell)) const hasUsage = cell.cost_usd !== null || cell.total_tokens !== null || cell.duration_seconds !== null return (
@@ -240,8 +249,8 @@ function CellResultSummary({ cell, metricKeys, metricTypes, metricAggregations }

UsageTotals across current replicates in this condition. Provider telemetry can be unavailable for some calls.

{cell.cost_usd !== null &&

Cost

{formatCurrency(cell.cost_usd)}

} - {cell.total_tokens !== null &&

Tokens

{formatNumber(cell.total_tokens)}

} {cell.duration_seconds !== null &&

Duration

{formatDuration(cell.duration_seconds)}

} + {cell.total_tokens !== null &&

Tokens

{formatNumber(cell.total_tokens)}

}
)} @@ -301,7 +310,16 @@ function ReplicateResultDetail({ replicate, metricKeys, metricTypes }: { metricKeys: string[] metricTypes: ResultMetricTypes }) { - const metrics = metricKeys.filter((key) => typeof replicate.metric_values[key] === 'number') + const metrics = metricKeys.filter((key) => { + if (typeof replicate.metric_values[key] !== 'number') return false + return !usageSummarizesMetric(key, replicate) + }) + const summarizedObservations = new Set( + metrics + .map((key) => observationForMetric(replicate, undefined, key)) + .filter((observation): observation is MetricObservation => observation?.status === 'measured'), + ) + const detailedObservations = replicate.metric_observations.filter((observation) => !summarizedObservations.has(observation)) const hasUsage = replicate.cost_usd !== null || replicate.total_tokens !== null || replicate.duration_seconds !== null || replicate.agent_run_count > 0 const timelineOnly = metrics.length === 0 && !hasUsage && !replicate.error const agentNodes = replicate.node_runs.filter((node) => node.agent_run_id) @@ -318,11 +336,18 @@ function ReplicateResultDetail({ replicate, metricKeys, metricTypes }: {

Outcome

- {metrics.map((key) =>

{formatMetricLabel(key)}

{formatResultMetricValue(key, replicate.metric_values[key], metricTypes, true)}

)} + {metrics.map((key) => { + const observation = observationForMetric(replicate, undefined, key) + return
+

{formatMetricLabel(key)}

+

{formatResultMetricValue(key, replicate.metric_values[key], metricTypes, true)}

+ {observation?.status === 'measured' &&

{observation.producer.producer_id} · v{observation.producer.version}

} +
+ })}
)} - {replicate.metric_observations.length > 0 &&

Measurement observations

{replicate.metric_observations.map((observation) => )}
} + {detailedObservations.length > 0 &&

Measurement observations

{detailedObservations.map((observation) => )}
} {(replicate.legacy_values ?? []).length > 0 &&

Legacy values

The original producers were not recorded, so these values are shown without inferred provenance and are not ranked.

{replicate.legacy_values!.map((item) =>

{item.metric_name}

{typeof item.value === 'string' ? item.value : JSON.stringify(item.value, null, 2)}
Legacy · producer unknown
)}
} {replicate.evaluation_artifacts.length > 0 &&

Evaluation artifacts

{replicate.evaluation_artifacts.map((artifact) => )}
} {hasUsage && ( @@ -330,8 +355,8 @@ function ReplicateResultDetail({ replicate, metricKeys, metricTypes }: {

Usage

- +
{replicate.agent_run_count > 0 && (replicate.reported_usage_count < replicate.agent_run_count || replicate.reported_cost_count < replicate.agent_run_count) &&

Usage and cost are shown only where the provider reported them.

} diff --git a/frontend/src/components/protocol/ScriptNodeInspector.tsx b/frontend/src/components/protocol/ScriptNodeInspector.tsx index 16a5687..5257f4b 100644 --- a/frontend/src/components/protocol/ScriptNodeInspector.tsx +++ b/frontend/src/components/protocol/ScriptNodeInspector.tsx @@ -15,7 +15,7 @@ const ACCENT = nodeAccent('script') // is a fixed label, not a picker, so there's nothing to configure there yet. // The whole node is also factor-bindable (bindableFields.ts's 'script_config' // kind) -- comparing two hand-written scoring scripts as an experimental -// factor is a direct use of the same whole-node-config mechanism llm_config/ +// factor is a direct use of the same whole-node-config mechanism model_config/ // tool_config/pattern already have. export function ScriptNodeInspector({ node, @@ -81,6 +81,16 @@ export function ScriptNodeInspector({
+
+ + patchConfig({ description: e.target.value })} + placeholder="What this script does and when the agent should run it" + /> +
+ - patchConfig({ code })} rows={16} /> + patchConfig({ code })} rows={16} resizable />
)} diff --git a/frontend/src/components/protocol/SingleAgentBaselinePatternNodeInspector.tsx b/frontend/src/components/protocol/SingleAgentBaselinePatternNodeInspector.tsx index 1358f40..876d8a9 100644 --- a/frontend/src/components/protocol/SingleAgentBaselinePatternNodeInspector.tsx +++ b/frontend/src/components/protocol/SingleAgentBaselinePatternNodeInspector.tsx @@ -3,9 +3,8 @@ import { nodeAccent } from '@/lib/nodeAccent' import { Input } from '@/components/ui/input' import { Label } from '@/components/ui/label' import { Switch } from '@/components/ui/switch' -import { FactorBindableField, MakeNodeFactorButton } from './FactorBindableField' +import { FactorBindableField } from './FactorBindableField' import { NodeInspectorDialog } from './NodeInspectorDialog' -import { useProtocolCanvasActions } from './ProtocolCanvasContext' import type { SingleAgentBaselinePatternConfig, SingleAgentBaselinePatternNodeData, ProtocolNode } from '@/types/protocols' const ACCENT = nodeAccent('pattern_single_agent_baseline') @@ -35,7 +34,6 @@ export function SingleAgentBaselinePatternNodeInspector({ onChange: (nodeId: string, data: SingleAgentBaselinePatternNodeData) => void onClose: () => void }) { - const { requestMakeFactor } = useProtocolCanvasActions() if (!node) return null const data = node.data @@ -67,7 +65,6 @@ export function SingleAgentBaselinePatternNodeInspector({ <>

{data.label || 'Single-Agent Baseline'}

- requestMakeFactor(node.id)} /> } onClose={onClose} diff --git a/frontend/src/components/protocol/TestRunResults.test.tsx b/frontend/src/components/protocol/TestRunResults.test.tsx index 0e51c95..2f6e974 100644 --- a/frontend/src/components/protocol/TestRunResults.test.tsx +++ b/frontend/src/components/protocol/TestRunResults.test.tsx @@ -1,5 +1,5 @@ import { useState } from 'react' -import { render, screen } from '@testing-library/react' +import { render, screen, within } from '@testing-library/react' import userEvent from '@testing-library/user-event' import { describe, expect, it, vi } from 'vitest' import type { TestRun } from '@/types/protocols' @@ -61,6 +61,43 @@ describe('TestRunResults', () => { expect(screen.getByText('Analyze this.')).toBeInTheDocument() }) + it('orders executable nodes by the canvas and omits configuration-only nodes', () => { + render() + + const progress = screen.getByRole('region', { name: 'Node progress' }) + expect(within(progress).getAllByText(/^(Lead|Researcher)$/).map((node) => node.textContent)).toEqual(['Lead', 'Researcher']) + expect(within(progress).queryByText('OpenAI')).not.toBeInTheDocument() + expect(within(progress).getByText('Available')).toBeInTheDocument() + }) + + it('collapses node progress and calls an unused Sub-Agent not invoked after the run', async () => { + const user = userEvent.setup() + const nodeNames = new Map([['child', 'Researcher']]) + const nodeTypes = new Map([['child', 'sub_agent']]) + const execution_summary = { + node_runs: { child: { status: 'skipped' as const, output_text: null } }, + started_at: '2026-09-16T12:00:00Z', completed_at: null, cancel_requested_at: null, + } + const { rerender } = render() + + const progressToggle = screen.getByRole('button', { name: /Node progress/ }) + await user.click(progressToggle) + expect(progressToggle).toHaveAttribute('aria-expanded', 'false') + + rerender() + await user.click(screen.getByRole('button', { name: /Node progress/ })) + expect(screen.getByText('Not invoked')).toBeInTheDocument() + }) + it('shows a captured metric alongside downstream task progress', () => { render( void; nodeNames?: Map; title?: string }) { +const EXECUTABLE_NODE_TYPES = new Set(['agent', 'sub_agent', 'critic_gate']) + +export function TestRunResults({ + run, + onClose, + nodeNames = new Map(), + nodeTypes = new Map(), + title = 'Test Run Results', +}: { + run: TestRun + onClose: () => void + nodeNames?: Map + nodeTypes?: Map + title?: string +}) { + const [nodeProgressCollapsed, setNodeProgressCollapsed] = useState(false) const running = !TERMINAL_RUN_STATUSES.has(run.status) + const canvasOrder = new Map(Array.from(nodeTypes.keys(), (nodeId, index) => [nodeId, index])) const nodeRuns = Object.entries(run.execution_summary.node_runs) + .filter(([nodeId]) => { + const nodeType = nodeTypes.get(nodeId) + return nodeType === undefined || EXECUTABLE_NODE_TYPES.has(nodeType) + }) + .sort(([leftId], [rightId]) => { + const leftOrder = canvasOrder.get(leftId) ?? Number.MAX_SAFE_INTEGER + const rightOrder = canvasOrder.get(rightId) ?? Number.MAX_SAFE_INTEGER + return leftOrder - rightOrder || (nodeNames.get(leftId) ?? leftId).localeCompare(nodeNames.get(rightId) ?? rightId) + }) + const nodeRunGroups = [ + { label: 'Agents', entries: nodeRuns.filter(([nodeId]) => nodeTypes.get(nodeId) !== 'critic_gate') }, + { label: 'Critic gates', entries: nodeRuns.filter(([nodeId]) => nodeTypes.get(nodeId) === 'critic_gate') }, + ].filter((group) => group.entries.length > 0) const timestamp = new Date(run.created_at) return ( @@ -105,14 +135,36 @@ export function TestRunResults({ run, onClose, nodeNames = new Map(), title = 'T {run.error &&

{run.error}

} {nodeRuns.length > 0 && ( -

Node progress

- {nodeRuns.map(([nodeId, nodeRun]) => { - const badge = nodeRunBadge(nodeRun.status, Boolean(nodeRun.truncation)) - return
- {nodeNames.get(nodeId) ?? nodeId}{badge && {badge.label}} -
{nodeRun.output_text ?
{nodeRun.output_text}
:

No output recorded yet.

}{nodeRun.error &&

{nodeRun.error}

}
-
- })} +
+ +
+ {nodeRunGroups.map((group) => ( +
+

{group.label}

+ {group.entries.map(([nodeId, nodeRun]) => { + const badge = nodeRunBadge(nodeRun.status, Boolean(nodeRun.truncation)) + const label = nodeTypes.get(nodeId) === 'sub_agent' && nodeRun.status === 'skipped' + ? (running ? 'Available' : 'Not invoked') + : badge?.label + return
+ {nodeNames.get(nodeId) ?? nodeId}{badge && {label}} +
{nodeRun.output_text ?
{nodeRun.output_text}
:

No output recorded yet.

}{nodeRun.error &&

{nodeRun.error}

}
+
+ })} +
+ ))} +
)} diff --git a/frontend/src/components/protocol/bindableFields.ts b/frontend/src/components/protocol/bindableFields.ts index 20d1623..95eccef 100644 --- a/frontend/src/components/protocol/bindableFields.ts +++ b/frontend/src/components/protocol/bindableFields.ts @@ -92,7 +92,7 @@ export function pickToolNamesForServer(previousToolNames: string[], availableToo } // Whether a node has at least one field (including a whole-node factor like -// pattern_override/llm_config/tool_config/script_config, which is stored +// pattern_override/model_config/tool_config/script_config, which is stored // under factor_bindings the same way an ordinary field-path binding is -- // see bindableFieldsForNode's own comment) bound to an experimental factor. // Every node component reads this straight off its own `data.factor_bindings` @@ -112,7 +112,13 @@ export function boundFactorCount(data: { factor_bindings?: Record` in a bordered box, like `RunsTab`/`ResultsTab`), not + `components/ui/table`'s page-width rows. The caller (`CellsTab`) owns the one + `overflow-x-auto` wrapper — the column set can outgrow even the maximized overlay. +- **A factorial design gets a heatmap complement above its table, not instead of it** + (`CellsHeatmap.tsx`) — precise numbers still matter, a heatmap just makes the *shape* of a + multi-factor sweep's results scannable at a glance. 1-2 factors render as one grid; 3 facet + into one grid per level of the third. Replicates within each cell are averaged, not just the + first one picked. Color is `color-mix(in oklch, var(--muted), + var(--primary) N%)` (`heatColor` in `lib/experiment.ts`) — low values dim, high values glow + in the theme's own accent, not an unrelated rainbow scale. Its layout is driven by + **container queries** (`@lg`/`@2xl`, with `CellsTab`'s `CellsBody` marking the `@container`), + not by the viewport and not by a compact/roomy prop — it renders both inside the + drag-resizable panel and in the full-viewport overlay, so "how much room is there" is a + question about the box, and dragging the panel wider has to pay off without a width threaded + through three components. Keep it that way if you add more responsive behaviour here. + - **Both axes (factors) and color (metric) are derived from the cells' own data, with + zero setup required** (`deriveFactors`/`availableMetricKeys`/`pickDefaultMetric` in + `lib/experiment.ts`) — NOT from `design_spec.factors`/`task_brief.selection_metric`. + Those two fields are only ever optional hints for a nicer default (`design_spec.factors` + supplies deliberate level *ordering*, and levels that are planned but have no cell yet; + `task_brief.selection_metric` only picks which observed metric key is pre-selected). Do + not make either a hard requirement again: `task_brief` is a notebook-local variable + embedded straight into agent prompts, not something the backend record has — nearly every + real experiment has both fields `null`, and requiring them would mean the heatmap never + renders for anyone who didn't know to separately attach this metadata. Factors are derived + from the union of every cell's `factor_values` keys, excluding a small bookkeeping + blocklist (`replicate`/`seed`/`rep`/`trial`/`iteration`) so a replicate index doesn't + become a spurious extra axis. + - **Which factors exist is decided by the cells, never by `design_spec.factors`** — this + is the bug that made the heatmap "disappear" once already, so don't undo it. + `deriveFactors` used to return a declared spec verbatim whenever one existed. Real data + here has both conventions: the *Myocardial Infarction* experiment's cells are keyed by + the declared display names (`Azure Foundry:Model`, `Critic enabled`), while the *Spinal + Fusion* one declares those same-style names but its cells are keyed `tier`/`effort`/ + `critic` from the design generator. Nothing keeps the two in sync. When they diverged, + every `replicatesMatching()` lookup matched 0 of 80 replicates, every square came out `null`, and + the grid vanished with no error in the console, no failing type, and no visible clue — + it looked exactly like a broken component. A declared spec now only contributes level + order and planned-but-unrun levels for names the cells actually use. The metric selector only shows when there's more than one + numeric key to choose from. For >3 derived factors, <2 cells, or no numeric metric + anywhere on the cells at all it draws no grid — those are exactly the cases where it + can't show anything the table doesn't already say better — but it says so in a one-line + `HeatmapUnavailable` note rather than rendering `null`. That's deliberate and worth + keeping: it used to bail silently, and a heatmap that can disappear for five different + data-shaped reasons without naming one is indistinguishable from a heatmap that's broken. + Route any new bail-out through that component instead of returning `null`. + - **The grid has to survive a 320px column** — it was written for a ~1024px page and moved + into the panel, where `grid-template-columns: auto repeat(N, 1fr)` overflowed: a grid + won't shrink an `auto` track below its content, so one long factor level name (a model + id, a dict-ish value) sized the label track past the panel's whole width and pushed every + square out through the `Card`'s `overflow-hidden` edge — the heatmap read as *missing*, + not as squeezed. It's now `fit-content(7rem) repeat(N, minmax(2.5rem, 1fr))` inside an + `overflow-x-auto` box, with `overflow-hidden` + `truncate` on the label cells (that pair + is what drops their min-content contribution to zero and lets the 7rem cap actually + bind). Don't reintroduce a content-sized track or drop the per-square minimum. +- **Cells/factor_values/metric_values are a `design_type === 'factorial'` concept, not a + universal one** — `CellsTab` and the canvas top bar's own cells/best-metric chips are both + gated behind that check. `design_type` is a plain string specifically so another experiment + type could exist later (ab_experiments, discoveries, etc. are explicitly out of scope on + `ResearchExperiment` itself, not something the frontend invented) — a non-factorial + experiment gets an explicit "Cell-based results aren't available for '{type}' experiments" + line instead of an empty/broken cells view. There is no other experiment type implemented + anywhere in ASAREE today, so this can't be exercised with real data yet — don't take that as + a reason to remove the guard; it's cheap insurance for a boundary ASAREE's own backend + already declared. +- **A dense view that outgrows the panel earns a "maximize" toggle** (`CellsTab`'s `fullscreen` + state, same as `ResultsTab`'s) — a fixed `inset-0 z-50` overlay with an Escape-key handler + and a close button, NOT the browser Fullscreen API (`requestFullscreen()` hides browser + chrome entirely, a bigger commitment than "let me read this table" calls for). Lock + `document.body` scroll while open and restore it on close/unmount. The table is mounted in + exactly one of the two places at a time, never duplicated with its own divergent sort/page + state. Reach for this pattern again for any other panel view that can't fit — don't + reintroduce the Fullscreen API for the same job. +- **The panel-vs-tab split**: Cells is the raw "what did each configuration score" grid; + `ResultsTab` is the statistical analysis computed *on* those numbers (effects, CIs, + non-inferiority). Keep them separate — don't fold cells into Results. +- **Agent lists aren't a table either** — but they're no longer a card grid either. The old + detail page's `AgentsSection` card grid became `ExperimentAgents` in `RunsTab.tsx`: a + compact one-row-per-agent list (icon tinted by model hash, name, model badge, run count, + relative last-used) since it's a footnote to the trial table in a narrow column, not its own + page section. The model-hash tint is the part that carries meaning and must survive any + further restyling; don't go back to ARES's plain name/role/added thin table either. +- **Design history is a collapsed footnote, not a peer of the table** (`DesignHistory.tsx`): + regenerating a design that would drop cells supersedes the whole revision rather than editing + or deleting the old cells (see the root `AGENTS.md`'s "Cells belong to a design revision"), + so an experiment quietly accumulates results the current view doesn't show. The list renders + **only once there are ≥2 revisions** — a permanent "1 revision" row is noise in a 320px + column, and a single current revision is just "the design", which is already the whole rest + of the tab. Selecting a superseded revision swaps the heatmap/table onto its cells with an + explicit read-only banner; that selection lives in `CellsTab` so the tally, the CSV button and + the grid can't disagree about which design they're showing, and a superseded revision gets its + own query key so it never overwrites the `['experiments', id, 'cells']` entry the canvas top + bar and `DesignTab` share. Deleting one is permanent and cascades to its results, so it goes + through a `Dialog` that names the counts (the `DeleteNodeConfirmDialog` convention), not the + lighter inline two-click confirm — and the current revision has no delete affordance at all, + since the server refuses it (409). +- Paginate client-side once a list can realistically exceed ~10-20 rows. diff --git a/frontend/src/components/protocol/cells/CLAUDE.md b/frontend/src/components/protocol/cells/CLAUDE.md index 2fa84a5..43c994c 100644 --- a/frontend/src/components/protocol/cells/CLAUDE.md +++ b/frontend/src/components/protocol/cells/CLAUDE.md @@ -1,121 +1 @@ -# Cells view UI details - -Scoped detail for the Cells tab of the protocol canvas's `ExperimentSidePanel` — this only -loads when you're working in this directory. These rules came from the (now-deleted) static -`ExperimentDetailPage`; they survived the move into the canvas unchanged in substance, only -re-sized for the panel column (384px by default, drag-resizable from 320px up). See the root -`CLAUDE.md`'s "Tables vs. cards is a -zoom-level decision" principle for the general rule this implements: you're *inside* one -experiment here, comparing dense multi-attribute records, which is exactly where a real table -earns its place. - -- **A real table has real columns, not two key=value string dumps** (`CellsTable.tsx`): a - `Factors`/`Metrics` column rendering `"tier=cloud-standard, effort=medium"` as flat text is - not "a table," it's two text blobs wearing a table's styling — you still have to visually - parse every row to compare anything. `CellsTable` gives one real, independently sortable - column per derived factor (reuses `deriveFactors`) plus up to 4 curated metric columns - (`pickMetricColumns` — same preference order as the heatmap's default metric, capped because - a real scored cell can have 6-7 numeric keys and showing all of them makes the table - unreadably wide). One table row is one true cell (a unique factor combination): its metric - columns are replicate means and its status reports scored/total replicates. The API's legacy - `Replicate` rows are individual observations and must be grouped first via - `groupReplicatesIntoCells`. `table-fixed` isn't used since the column set is dynamic; instead: - `truncate` on free-text cells, a shaded uppercase sortable header row, subtle zebra striping, - `font-mono` on technical values. It's styled to the side panel's own compact table idiom (a - plain `text-xs` `` in a bordered box, like `RunsTab`/`ResultsTab`), not - `components/ui/table`'s page-width rows. The caller (`CellsTab`) owns the one - `overflow-x-auto` wrapper — the column set can outgrow even the maximized overlay. -- **A factorial design gets a heatmap complement above its table, not instead of it** - (`CellsHeatmap.tsx`) — precise numbers still matter, a heatmap just makes the *shape* of a - multi-factor sweep's results scannable at a glance. 1-2 factors render as one grid; 3 facet - into one grid per level of the third. Replicates within each cell are averaged, not just the - first one picked. Color is `color-mix(in oklch, var(--muted), - var(--primary) N%)` (`heatColor` in `lib/experiment.ts`) — low values dim, high values glow - in the theme's own accent, not an unrelated rainbow scale. Its layout is driven by - **container queries** (`@lg`/`@2xl`, with `CellsTab`'s `CellsBody` marking the `@container`), - not by the viewport and not by a compact/roomy prop — it renders both inside the - drag-resizable panel and in the full-viewport overlay, so "how much room is there" is a - question about the box, and dragging the panel wider has to pay off without a width threaded - through three components. Keep it that way if you add more responsive behaviour here. - - **Both axes (factors) and color (metric) are derived from the cells' own data, with - zero setup required** (`deriveFactors`/`availableMetricKeys`/`pickDefaultMetric` in - `lib/experiment.ts`) — NOT from `design_spec.factors`/`task_brief.selection_metric`. - Those two fields are only ever optional hints for a nicer default (`design_spec.factors` - supplies deliberate level *ordering*, and levels that are planned but have no cell yet; - `task_brief.selection_metric` only picks which observed metric key is pre-selected). Do - not make either a hard requirement again: `task_brief` is a notebook-local variable - embedded straight into agent prompts, not something the backend record has — nearly every - real experiment has both fields `null`, and requiring them would mean the heatmap never - renders for anyone who didn't know to separately attach this metadata. Factors are derived - from the union of every cell's `factor_values` keys, excluding a small bookkeeping - blocklist (`replicate`/`seed`/`rep`/`trial`/`iteration`) so a replicate index doesn't - become a spurious extra axis. - - **Which factors exist is decided by the cells, never by `design_spec.factors`** — this - is the bug that made the heatmap "disappear" once already, so don't undo it. - `deriveFactors` used to return a declared spec verbatim whenever one existed. Real data - here has both conventions: the *Myocardial Infarction* experiment's cells are keyed by - the declared display names (`Azure Foundry:Model`, `Critic enabled`), while the *Spinal - Fusion* one declares those same-style names but its cells are keyed `tier`/`effort`/ - `critic` from the design generator. Nothing keeps the two in sync. When they diverged, - every `replicatesMatching()` lookup matched 0 of 80 replicates, every square came out `null`, and - the grid vanished with no error in the console, no failing type, and no visible clue — - it looked exactly like a broken component. A declared spec now only contributes level - order and planned-but-unrun levels for names the cells actually use. The metric selector only shows when there's more than one - numeric key to choose from. For >3 derived factors, <2 cells, or no numeric metric - anywhere on the cells at all it draws no grid — those are exactly the cases where it - can't show anything the table doesn't already say better — but it says so in a one-line - `HeatmapUnavailable` note rather than rendering `null`. That's deliberate and worth - keeping: it used to bail silently, and a heatmap that can disappear for five different - data-shaped reasons without naming one is indistinguishable from a heatmap that's broken. - Route any new bail-out through that component instead of returning `null`. - - **The grid has to survive a 320px column** — it was written for a ~1024px page and moved - into the panel, where `grid-template-columns: auto repeat(N, 1fr)` overflowed: a grid - won't shrink an `auto` track below its content, so one long factor level name (a model - id, a dict-ish value) sized the label track past the panel's whole width and pushed every - square out through the `Card`'s `overflow-hidden` edge — the heatmap read as *missing*, - not as squeezed. It's now `fit-content(7rem) repeat(N, minmax(2.5rem, 1fr))` inside an - `overflow-x-auto` box, with `overflow-hidden` + `truncate` on the label cells (that pair - is what drops their min-content contribution to zero and lets the 7rem cap actually - bind). Don't reintroduce a content-sized track or drop the per-square minimum. -- **Cells/factor_values/metric_values are a `design_type === 'factorial'` concept, not a - universal one** — `CellsTab` and the canvas top bar's own cells/best-metric chips are both - gated behind that check. `design_type` is a plain string specifically so another experiment - type could exist later (ab_experiments, discoveries, etc. are explicitly out of scope on - `ResearchExperiment` itself, not something the frontend invented) — a non-factorial - experiment gets an explicit "Cell-based results aren't available for '{type}' experiments" - line instead of an empty/broken cells view. There is no other experiment type implemented - anywhere in ASAREE today, so this can't be exercised with real data yet — don't take that as - a reason to remove the guard; it's cheap insurance for a boundary ASAREE's own backend - already declared. -- **A dense view that outgrows the panel earns a "maximize" toggle** (`CellsTab`'s `fullscreen` - state, same as `ResultsTab`'s) — a fixed `inset-0 z-50` overlay with an Escape-key handler - and a close button, NOT the browser Fullscreen API (`requestFullscreen()` hides browser - chrome entirely, a bigger commitment than "let me read this table" calls for). Lock - `document.body` scroll while open and restore it on close/unmount. The table is mounted in - exactly one of the two places at a time, never duplicated with its own divergent sort/page - state. Reach for this pattern again for any other panel view that can't fit — don't - reintroduce the Fullscreen API for the same job. -- **The panel-vs-tab split**: Cells is the raw "what did each configuration score" grid; - `ResultsTab` is the statistical analysis computed *on* those numbers (effects, CIs, - non-inferiority). Keep them separate — don't fold cells into Results. -- **Agent lists aren't a table either** — but they're no longer a card grid either. The old - detail page's `AgentsSection` card grid became `ExperimentAgents` in `RunsTab.tsx`: a - compact one-row-per-agent list (icon tinted by model hash, name, model badge, run count, - relative last-used) since it's a footnote to the trial table in a narrow column, not its own - page section. The model-hash tint is the part that carries meaning and must survive any - further restyling; don't go back to ARES's plain name/role/added thin table either. -- **Design history is a collapsed footnote, not a peer of the table** (`DesignHistory.tsx`): - regenerating a design that would drop cells supersedes the whole revision rather than editing - or deleting the old cells (see the root `CLAUDE.md`'s "Cells belong to a design revision"), - so an experiment quietly accumulates results the current view doesn't show. The list renders - **only once there are ≥2 revisions** — a permanent "1 revision" row is noise in a 320px - column, and a single current revision is just "the design", which is already the whole rest - of the tab. Selecting a superseded revision swaps the heatmap/table onto its cells with an - explicit read-only banner; that selection lives in `CellsTab` so the tally, the CSV button and - the grid can't disagree about which design they're showing, and a superseded revision gets its - own query key so it never overwrites the `['experiments', id, 'cells']` entry the canvas top - bar and `DesignTab` share. Deleting one is permanent and cascades to its results, so it goes - through a `Dialog` that names the counts (the `DeleteNodeConfirmDialog` convention), not the - lighter inline two-click confirm — and the current revision has no delete affordance at all, - since the server refuses it (409). -- Paginate client-side once a list can realistically exceed ~10-20 rows. +@AGENTS.md diff --git a/frontend/src/components/protocol/cells/CellsTab.tsx b/frontend/src/components/protocol/cells/CellsTab.tsx index a648774..8f8da80 100644 --- a/frontend/src/components/protocol/cells/CellsTab.tsx +++ b/frontend/src/components/protocol/cells/CellsTab.tsx @@ -46,7 +46,7 @@ function CellsBody({ experiment, cells }: { experiment: Experiment; cells: Repli * experiment has cells, so a future non-factorial type gets an explicit * "not available" line instead of an empty, broken-looking grid. * - * Maximize is CLAUDE.md's established convention for a dense view that + * Maximize is AGENTS.md's established convention for a dense view that * outgrows the panel's column (same as ResultsTab): a fixed inset-0 overlay * with an Escape handler and a body-scroll lock, never the browser Fullscreen * API. The table is mounted in exactly one of the two places at a time, so diff --git a/frontend/src/components/protocol/connectionValidation.test.ts b/frontend/src/components/protocol/connectionValidation.test.ts new file mode 100644 index 0000000..9f97d02 --- /dev/null +++ b/frontend/src/components/protocol/connectionValidation.test.ts @@ -0,0 +1,71 @@ +import type { Connection, Edge, Node } from '@xyflow/react' +import { describe, expect, it } from 'vitest' +import { isProtocolConnectionValid } from './connectionValidation' + +const target: Node = { id: 'agent', type: 'agent', position: { x: 0, y: 0 }, data: {} } + +describe('isProtocolConnectionValid', () => { + const cappedSlots = [ + ['model', 'model_openai'], + ['memory', 'memory'], + ['architectural_pattern', 'pattern_reason_act'], + ['output_parser', 'output_parser'], + ] as const + + it.each(cappedSlots)('rejects a second %s connection to the same node', (slot, sourceType) => { + const firstSource: Node = { id: 'first', type: sourceType, position: { x: 0, y: 0 }, data: {} } + const secondSource: Node = { id: 'second', type: sourceType, position: { x: 0, y: 0 }, data: {} } + const existingEdge: Edge = { + id: 'existing', + source: firstSource.id, + sourceHandle: slot, + target: target.id, + targetHandle: slot, + } + const connection: Connection = { + source: secondSource.id, + sourceHandle: slot, + target: target.id, + targetHandle: slot, + } + + expect(isProtocolConnectionValid(connection, [target, firstSource, secondSource], [existingEdge], false)).toBe(false) + }) + + it.each(cappedSlots)('accepts the first %s connection', (slot, sourceType) => { + const source: Node = { id: 'source', type: sourceType, position: { x: 0, y: 0 }, data: {} } + const connection: Connection = { + source: source.id, + sourceHandle: slot, + target: target.id, + targetHandle: slot, + } + + expect(isProtocolConnectionValid(connection, [target, source], [], false)).toBe(true) + }) + + it.each([ + ['tool', 'mcp_tool'], + ['dataset', 'dataset'], + ['skill', 'skill'], + ['knowledge', 'okf_bundle'], + ])('continues to accept multiple %s connections', (slot, sourceType) => { + const firstSource: Node = { id: 'first', type: sourceType, position: { x: 0, y: 0 }, data: {} } + const secondSource: Node = { id: 'second', type: sourceType, position: { x: 0, y: 0 }, data: {} } + const existingEdge: Edge = { + id: 'existing', + source: firstSource.id, + sourceHandle: slot, + target: target.id, + targetHandle: slot, + } + const connection: Connection = { + source: secondSource.id, + sourceHandle: slot, + target: target.id, + targetHandle: slot, + } + + expect(isProtocolConnectionValid(connection, [target, firstSource, secondSource], [existingEdge], false)).toBe(true) + }) +}) diff --git a/frontend/src/components/protocol/connectionValidation.ts b/frontend/src/components/protocol/connectionValidation.ts new file mode 100644 index 0000000..31c98bb --- /dev/null +++ b/frontend/src/components/protocol/connectionValidation.ts @@ -0,0 +1,96 @@ +import { CONNECTOR_HANDLES } from '@/lib/coordinationStrategy' +import { MCP_TOOL_NODE_TYPES } from './mcpServerCatalog' +import type { Connection, Edge, Node } from '@xyflow/react' + +export const MODEL_NODE_TYPES = [ + 'model_anthropic', + 'model_openai', + 'model_azure_foundry', + 'model_openrouter', + 'model_local', +] +export const PATTERN_NODE_TYPES = ['pattern_reason_act', 'pattern_single_agent_baseline'] +export const KNOWLEDGE_NODE_TYPES = ['okf_bundle', 'okf_document'] + +// These slots represent one scalar configuration on their target node. Keep +// this aligned with protocol_execution.py's backend cardinality checks. The +// remaining named slots are collections and deliberately accept many edges. +const SINGLE_CAPACITY_SLOTS = new Set([ + 'model', + 'memory', + 'architectural_pattern', + 'output_parser', +]) + +export function isProtocolConnectionValid( + connection: Edge | Connection, + nodes: Node[], + edges: Edge[], + isSequential: boolean, +): boolean { + const sourceNode = nodes.find((node) => node.id === connection.source) + const targetNode = nodes.find((node) => node.id === connection.target) + if (!sourceNode || !targetNode) return false + + if ( + connection.targetHandle && + SINGLE_CAPACITY_SLOTS.has(connection.targetHandle) && + edges.some( + (edge) => edge.target === connection.target && edge.targetHandle === connection.targetHandle, + ) + ) { + return false + } + + const targetIsAgentLike = targetNode.type === 'agent' || targetNode.type === 'sub_agent' + switch (connection.targetHandle) { + case 'model': + return ( + MODEL_NODE_TYPES.includes(sourceNode.type ?? '') && + (targetIsAgentLike || targetNode.type === 'critic_gate') + ) + case 'tool': + return ( + (MCP_TOOL_NODE_TYPES.includes(sourceNode.type ?? '') || sourceNode.type === 'script') && + targetIsAgentLike + ) + case 'memory': + return sourceNode.type === 'memory' && targetIsAgentLike + case 'output_parser': + return sourceNode.type === 'output_parser' && targetIsAgentLike + case 'architectural_pattern': + return PATTERN_NODE_TYPES.includes(sourceNode.type ?? '') && targetIsAgentLike + case 'skill': + return sourceNode.type === 'skill' && targetIsAgentLike + case 'dataset': + return sourceNode.type === 'dataset' && targetIsAgentLike + case 'knowledge': + return KNOWLEDGE_NODE_TYPES.includes(sourceNode.type ?? '') && targetIsAgentLike + case 'sub_agents': + return ( + sourceNode.type === 'sub_agent' && + targetNode.type === 'agent' && + !edges.some((edge) => edge.source === sourceNode.id && edge.targetHandle === 'sub_agents') + ) + default: { + const sourceCanFeedMainFlow = + !MODEL_NODE_TYPES.includes(sourceNode.type ?? '') && + sourceNode.type !== 'memory' && + sourceNode.type !== 'output_parser' && + !MCP_TOOL_NODE_TYPES.includes(sourceNode.type ?? '') && + sourceNode.type !== 'dataset' && + sourceNode.type !== 'skill' && + !KNOWLEDGE_NODE_TYPES.includes(sourceNode.type ?? '') && + sourceNode.type !== 'script' && + !PATTERN_NODE_TYPES.includes(sourceNode.type ?? '') && + sourceNode.type !== 'sub_agent' && + targetNode.type !== 'sub_agent' + if (!sourceCanFeedMainFlow) return false + if (!isSequential) return true + return ( + !edges.some((edge) => edge.source === connection.source && !CONNECTOR_HANDLES.has(edge.targetHandle ?? '')) && + !edges.some((edge) => edge.target === connection.target && !CONNECTOR_HANDLES.has(edge.targetHandle ?? '')) + ) + } + } +} diff --git a/frontend/src/components/protocol/datasetCatalog.ts b/frontend/src/components/protocol/datasetCatalog.ts index 81ec764..a566cb6 100644 --- a/frontend/src/components/protocol/datasetCatalog.ts +++ b/frontend/src/components/protocol/datasetCatalog.ts @@ -25,6 +25,14 @@ export const DATASET_BROWSE = 'datasets_browse' export function nodeDataForDataset(dataset: Dataset): DatasetNodeData { return { label: dataset.name, - config: { dataset_id: dataset.id, dataset_name: dataset.name, enabled: true }, + config: { + dataset_id: dataset.id, + dataset_name: dataset.name, + description: dataset.description, + target_column: dataset.target_column, + split_state: dataset.train_path && dataset.test_path ? 'split' : 'unsplit', + dictionary_available: Boolean(dataset.dictionary_json), + enabled: true, + }, } } diff --git a/frontend/src/components/protocol/edges/InteractEdge.tsx b/frontend/src/components/protocol/edges/InteractEdge.tsx index d24abfa..acbcdc2 100644 --- a/frontend/src/components/protocol/edges/InteractEdge.tsx +++ b/frontend/src/components/protocol/edges/InteractEdge.tsx @@ -27,21 +27,22 @@ import { useProtocolCanvasActions } from '../ProtocolCanvasContext' // A solid edge is deliberately not one relationship: between two Agent nodes it // is BOTH the left-to-right pipeline edge a normal run walks AND the "these two // may consult each other" edge a Peer Collaboration run reads (undirected). The -// experiment's coordination strategy picks which, so the edge must not commit to -// either -- it looks the same in both, and nothing is annotated onto it. An -// earlier pass captioned peer edges "can consult"; it read as clutter on a -// canvas where most solid edges qualify, and the Design tab already says which -// strategy is in force. +// experiment's coordination strategy picks which. Sequential Agent-to-Agent +// edges get a heavier stroke and an arrow from source to target; under the +// collaboration strategies the same relationship stays undirected. An earlier +// pass captioned peer edges "can consult"; it read as clutter on a canvas where +// most solid edges qualify, and the Design tab already says which strategy is +// in force. // // Note the dashes are NOT the same statement as MemoryNode's dashed ring, // which means "not yet functional"; here they only mean "connector, not // pipeline". Nothing currently renders both, but don't add a third meaning. const EDGE_STROKE = 'color-mix(in oklch, var(--muted-foreground), transparent 30%)' -function edgeStyle(isMainEdge: boolean, hovered: boolean): CSSProperties { +function edgeStyle(isMainEdge: boolean, isSequentialAgentFlow: boolean, hovered: boolean): CSSProperties { return { stroke: hovered ? 'var(--primary)' : EDGE_STROKE, - strokeWidth: hovered ? 2.5 : 2, + strokeWidth: isSequentialAgentFlow ? (hovered ? 3.5 : 3) : hovered ? 2.5 : 2, strokeDasharray: isMainEdge ? undefined : '6 4', filter: hovered ? 'drop-shadow(0 0 5px var(--primary))' : undefined, transition: 'stroke 120ms ease, stroke-width 120ms ease', @@ -63,7 +64,7 @@ function edgeStyle(isMainEdge: boolean, hovered: boolean): CSSProperties { // would. Swapping (which removes both atomically) is still the only way to // change it. "+" (insert a node in the middle) only shows for a plain // "main" edge (no source/targetHandle) -- inserting an arbitrary node into -// a typed connector edge (LLM/Tool/Memory/Pattern) would violate that +// a typed connector edge (Model/Tool/Memory/Pattern) would violate that // connector's own required shape, so it's hidden there too. export function InteractEdge({ id, @@ -77,6 +78,7 @@ export function InteractEdge({ targetPosition, sourceHandleId, targetHandleId, + data, style, markerEnd, }: EdgeProps) { @@ -85,6 +87,7 @@ export function InteractEdge({ const { requestEdgeInsert } = useProtocolCanvasActions() const [edgePath, labelX, labelY] = getBezierPath({ sourceX, sourceY, sourcePosition, targetX, targetY, targetPosition }) const isMainEdge = !sourceHandleId && !targetHandleId + const isSequentialAgentFlow = data?.sequentialAgentFlow === true const isPatternEdge = targetHandleId === 'architectural_pattern' return ( @@ -94,7 +97,7 @@ export function InteractEdge({ diff --git a/frontend/src/components/protocol/factorLevels.ts b/frontend/src/components/protocol/factorLevels.ts index 25d7a5f..71614e5 100644 --- a/frontend/src/components/protocol/factorLevels.ts +++ b/frontend/src/components/protocol/factorLevels.ts @@ -1,8 +1,8 @@ import type { DesignFactor } from '@/types/experiments' -// llm_config/tool_config/pattern/script_config/dataset_config are the "whole +// model_config/tool_config/pattern/script_config/dataset_config are the "whole // node as a factor" kinds (see bindableFields.ts) -- their levels are OBJECTS -// (a whole LLM/Tool/Script/Dataset node config, or a {execution_pattern, +// (a whole Model/Tool/Script/Dataset node config, or a {execution_pattern, // pattern_params} payload), never strings, unlike every other kind here. // // tool_names is the odd one out: its levels are ARRAYS of bare tool names, @@ -15,7 +15,7 @@ export type LevelType = | 'text' | 'number' | 'boolean' - | 'llm_config' + | 'model_config' | 'tool_config' | 'pattern' | 'script_config' @@ -27,7 +27,7 @@ export const LEVEL_TYPE_LABELS: Record = { text: 'Long text', number: 'Number', boolean: 'Boolean', - llm_config: 'Provider & model', + model_config: 'Provider & model', tool_config: 'Server & tools', pattern: 'Execution pattern', script_config: 'Script', @@ -42,7 +42,7 @@ export const LEVEL_TYPE_LABELS: Record = { // its own one-line-Input popover. export function isStructuredLevelType(type: LevelType): boolean { return ( - type === 'llm_config' || + type === 'model_config' || type === 'tool_config' || type === 'pattern' || type === 'script_config' || @@ -67,9 +67,8 @@ export function factorLevelLabels(factor: DesignFactor): string[] { } // Shared visual identity for every "make experimental factor" trigger -- -// FactorBindableField's own per-field triggers plus the Agent/Pattern -// inspectors' title-row buttons (the one spot that opens the per-node -// picker rather than binding a single field directly). A dedicated hue +// FactorBindableField's own per-field triggers, including the Agent +// inspector's title-row Active binding. A dedicated hue // (violet, --chart-2) not already claimed by another meaning elsewhere in // this app (chart-3 = fully scored, chart-4 = generated-but-unscored, // chart-5 ≈ destructive's own hue, primary = every other button) -- fixed, @@ -90,13 +89,13 @@ export function parseLevelValue(raw: string, type: LevelType): unknown { } // A blank starting point for one structured level -- shaped exactly like -// what protocol_execution.py's _resolve_llm_config/_resolve_tool_config/ +// what protocol_execution.py's _resolve_model_config/_resolve_tool_config/ // _resolve_pattern_config already expect, so a freshly-added level is // immediately a valid (if unconfigured) whole-node config rather than an // empty object the executor can't do anything with. export function emptyStructuredLevel(type: LevelType): unknown { switch (type) { - case 'llm_config': + case 'model_config': return { provider: 'anthropic', model: '', temperature: 0.7, max_tokens: 128000 } case 'tool_config': return { server_id: null, server_name: null, tool_names: [], enabled: true } diff --git a/frontend/src/components/protocol/layout.ts b/frontend/src/components/protocol/layout.ts index a451f36..8e43f72 100644 --- a/frontend/src/components/protocol/layout.ts +++ b/frontend/src/components/protocol/layout.ts @@ -21,8 +21,9 @@ const CONNECTOR_X: { agent: Record; critic_gate: Partial< skill: 0.18, dataset: 0.71, knowledge: 0.9, - ai: 0.2, - memory: 0.5, + model: 0.08, + sub_agents: 0.32, + memory: 0.58, tool: 0.8, // Added after the other seven, and placed after Tool rather than among // them: every existing slot keeps the x it already had, because a canvas @@ -31,11 +32,11 @@ const CONNECTOR_X: { agent: Record; critic_gate: Partial< // see AgentNode's own showOutputParser. output_parser: 0.95, }, - critic_gate: { ai: 0.5 }, + critic_gate: { model: 0.5 }, } // The host cards' own widths: AgentNode is w-72, CriticGateNode w-36. -const HOST_WIDTH: Record = { agent: 288, critic_gate: 144 } +const HOST_WIDTH: Record = { agent: 288, sub_agent: 288, critic_gate: 144 } // A connector's node is a CircleNode: a 56px circle under a caption that can // grow to 96px, with the circle -- where its handle is -- centered in @@ -44,6 +45,12 @@ const HOST_WIDTH: Record = { agent: 288, critic_gate: 144 } // and being a whole card-width off is the thing this fixes. const NEW_NODE_HALF_WIDTH = 38 +// A Sub-Agent needs a full satellite row between it and its parent for its +// own required Pattern node. Keeping the two equal-width cards aligned also +// makes the ownership hierarchy read vertically instead of as another step +// in the left-to-right main flow. +export const SUB_AGENT_CHILD_OFFSET_Y = 320 + /** The `left` style for each of a host card's connectors, as a percentage * string — for the ``, its caption and its "+" stub, which must all * sit at the same x. */ @@ -54,11 +61,13 @@ export function connectorLefts(host: 'agent' | 'critic_gate'): Record { - const isHost = (n: TidyNode) => n.type === 'agent' || n.type === 'critic_gate' + const isHost = (n: TidyNode) => n.type === 'agent' || n.type === 'sub_agent' || n.type === 'critic_gate' const hosts = nodes.filter(isHost) // A connector edge carries a targetHandle (the slot it feeds); a main-flow // edge between two hosts doesn't. So the presence of a handle is what @@ -232,9 +241,19 @@ export function tidyLayout(nodes: TidyNode[], edges: TidyEdge[]): Map, ): NodeConfigIssue[] { - const agentIdsWithLlm = new Set(edges.filter((e) => e.targetHandle === 'ai').map((e) => e.target)) + const agentIdsWithModel = new Set(edges.filter((e) => e.targetHandle === 'model').map((e) => e.target)) // suggestedMaxIterations walks the persisted shape (it also runs against a // graph loaded from the server), so convert once rather than per node. const graph = toPersistedGraph(nodes, edges) @@ -68,12 +68,23 @@ export function findNodeConfigIssues( switch (node.type) { case 'agent': - if (!agentIdsWithLlm.has(node.id)) issues.push('No AI connected') + if (!agentIdsWithModel.has(node.id)) issues.push('No Model connected') break - case 'llm_anthropic': - case 'llm_openai': - case 'llm_azure_foundry': { - const config = (node.data as LlmNodeData).config + case 'sub_agent': + if ( + node.data.active !== false && + edges.some((edge) => edge.source === node.id && edge.targetHandle === 'sub_agents') && + !agentIdsWithModel.has(node.id) + ) { + issues.push('No Model connected') + } + break + case 'model_anthropic': + case 'model_openai': + case 'model_azure_foundry': + case 'model_openrouter': + case 'model_local': { + const config = (node.data as ModelNodeData).config let selectedModelInfo: LLMSettingModelsResponse['models'][number] | undefined if (!config?.model) { issues.push('No model set') @@ -87,7 +98,7 @@ export function findNodeConfigIssues( // knowingly incomplete and the inspector's "Custom model..." field // exists to go past it, so an off-catalog id there is a supported // choice, not a misconfigured node worth interrupting a Run for. - // Same gate as LlmNode.tsx's own warning triangle; see the longer + // Same gate as ModelNode.tsx's own warning triangle; see the longer // note there. if (cached?.source === 'api' && models.length > 0 && !selectedModelInfo) { const label = PROVIDER_META[config.provider]?.label ?? config.provider @@ -96,7 +107,7 @@ export function findNodeConfigIssues( } if (config?.max_tokens == null) issues.push('Max tokens is required') // Same "unrecognized model defaults to temperature-only" fallback as - // LlmNodeInspector.tsx's own showTemperature -- required whenever + // ModelNodeInspector.tsx's own showTemperature -- required whenever // it's the field actually offered for this model, so it's never // Motoro's own silent ModelConfig default (0.7) filling the gap. if ((selectedModelInfo?.supports_temperature ?? true) && config?.temperature == null) { diff --git a/frontend/src/components/protocol/nodes/AgentNode.tsx b/frontend/src/components/protocol/nodes/AgentNode.tsx index e7d6efd..e54b9c1 100644 --- a/frontend/src/components/protocol/nodes/AgentNode.tsx +++ b/frontend/src/components/protocol/nodes/AgentNode.tsx @@ -22,8 +22,6 @@ import { NodeSummaryLine } from './NodeSummaryLine' // not its label. The connector captions below are the one thing here that // does NOT follow --card-accent: they're yellow, to be found against the node // rather than to match it (ConnectorHandleLabel). -const ACCENT = nodeAccent('agent') - // Every connector's handle, caption and "+" stub reads its x from here, and so // does the placement of whatever node the connector's own "+" creates (see // layout.ts) -- otherwise a node can land under a connector that isn't the one @@ -40,11 +38,11 @@ export function AgentNode({ // Paired with runStatus, never derivable from it: a truncated run is still // `completed` (see NodeRunState.truncation). runTruncated?: boolean - missingLlm?: boolean + missingModel?: boolean missingOutputParser?: boolean canRunAlone?: boolean // Both injected by ProtocolCanvas: whether a plain Agent-to-Agent edge - // reaches this node, and the model its AI connector resolves to. The + // reaches this node, and the model its Model connector resolves to. The // canvas supplies the wiring; the capability lookup below is this card's. hasPeers?: boolean llmConfig?: { provider?: string; model?: string } | null @@ -59,8 +57,11 @@ export function AgentNode({ // one -- see ProtocolCanvas's `mainEdgeSlots`. mainInFull?: boolean mainOutFull?: boolean + isSubAgent?: boolean } }) { + const isSubAgent = data.isSubAgent === true + const accent = nodeAccent(isSubAgent ? 'sub_agent' : 'agent') const badge = nodeRunBadge(data.runStatus, data.runTruncated) // Peers are offered to the model as function schemas -- that is the only // channel a consultation can be *chosen* through -- so an agent on a model @@ -74,11 +75,11 @@ export function AgentNode({ const peerNeedsToolCalling = !!data.hasPeers && models.find((m) => m.id === data.llmConfig?.model)?.supports_tool_calling === false const warnings = [ - ...(data.missingLlm ? ["No AI connected -- this agent can't run"] : []), + ...(data.missingModel ? ["No Model connected -- this agent can't run"] : []), ...(data.missingOutputParser ? ['A specific output format is required, but no Output Parser says what it is'] : []), - ...(peerNeedsToolCalling ? ["This model can't call tools, so this agent can't consult its connected peers"] : []), + ...(peerNeedsToolCalling ? ["This model can't call tools, so this agent can't consult or delegate to connected agents"] : []), ] const { updateNodeData } = useReactFlow() const { requestRunNode } = useProtocolCanvasActions() @@ -109,13 +110,16 @@ export function AgentNode({ // // The already-wired clause keeps the visible caption in sync with graphs // created outside this UI, where the edge may exist without the flag. + const modelConnections = useNodeConnections({ id, handleType: 'target', handleId: 'model' }) + const memoryConnections = useNodeConnections({ id, handleType: 'target', handleId: 'memory' }) + const patternConnections = useNodeConnections({ id, handleType: 'target', handleId: 'architectural_pattern' }) const parserConnections = useNodeConnections({ id, handleType: 'target', handleId: 'output_parser' }) const showOutputParser = !!data.config?.require_output_parser || parserConnections.length > 0 || !!data.config?.output_contract return (
updateNodeData(id, { active: !isActive })} runAlone={{ canRun: !!data.canRunAlone, onRun: () => requestRunNode(id) }} /> - {/* Steps left to `right-6` when the factor badge is also showing: that - badge straddles this same corner (-right-3, size-7, so 12px out to - 16px in) and this Badge straddles the top border (-top-2.5, h-5), so - at the default right-1.5 the two would sit on top of each other. */} + {/* Sits inside the top-right corner so it stays clear of the Knowledge + connector above the card. Steps left to `right-6` when the factor + badge is also showing because that badge straddles this corner. */} {badge && ( - + {badge.label} )} @@ -143,7 +146,7 @@ export function AgentNode({ Architectural Pattern connector's own label/stub live OUTSIDE the card on the left of this edge, so neither competes for this corner. */} {hasBoundFactor(data) && } - {/* Main flow is left-to-right -- the 4 sub-connectors below stay on the + {/* Main flow is left-to-right -- the bottom sub-connectors stay on the bottom edge regardless, since a config source hangs below a node no matter which way the main flow runs. @@ -154,19 +157,37 @@ export function AgentNode({ decides the cardinality: unrestricted under Peer Collaboration and Critic Gate, but exactly one per side under Sequential, where the chain rule applies and the "+" stub hides once a side is taken. */} - - + {!isSubAgent && ( + <> + + Agent + + + )} + {isSubAgent && ( + <> + + Parent + + )}
{/* Renaming happens in the Inspector's own title now (click it, same as the experiment name) -- not here anymore. */} - {data.label || 'Agent'} + {data.label || (isSubAgent ? 'Sub-Agent' : 'Agent')} {/* Inline on the title row rather than hung off a corner: all three corners are spoken for (run status and the factor badge share the @@ -219,9 +240,10 @@ export function AgentNode({ toolbar is above the stub's z-index and only there on hover, and the visible "+" glyph still clears it). - The 3 bottom sub-connectors: required AI (exactly one), optional - max-1 Memory (visual scaffolding only -- see MemoryNodeData), and - optional repeatable Tool. Script is a repeatable pure config source + The bottom sub-connectors: required Model (exactly one), optional + repeatable Sub-Agents on parent Agents, optional max-1 Memory (visual + scaffolding only -- see MemoryNodeData), optional repeatable Tool, + and optional max-1 Output Parser. Script is a repeatable pure config source too, but deliberately does NOT get its own slot -- it wires into that same Tool connector (one connector accepting a FAMILY of node types, matching Motoro's own @@ -233,13 +255,14 @@ export function AgentNode({ Pattern - {/* Never hides once connected (unlike AI/Memory) -- an execution + {/* Never hides once connected (unlike Model/Memory) -- an execution pattern must never go to zero (Motoro silently falls back to reason_act if left unconnected, undoing the whole point of making the default explicit), so the only way to change it is to @@ -329,24 +352,40 @@ export function AgentNode({ /> Knowledge - {/* Handle id `ai`; graphs saved before the rename carry these edges on - `llm` -- ProtocolCanvas.tsx rewrites those on load + {/* Handle id `model`; graphs saved before the rename carry these edges on + `ai` or `llm` -- ProtocolCanvas.tsx rewrites those on load (migrateLegacyHandles) and the backend keeps accepting both (see - _LEGACY_AI_HANDLES). The node types feeding it are still called - LLM_NODE_TYPES: those name a model family, not this slot. */} + _LEGACY_MODEL_HANDLES). The node types feeding it are called + MODEL_NODE_TYPES: those name a model family, not this slot. */} - AI - + Model + + {!isSubAgent && ( + <> + + Sub-Agents + + + )} Tool @@ -367,8 +406,8 @@ export function AgentNode({ {/* Output Parser -- the field spec the agent's answer is written to and read back out of. Last on the bottom edge, at 95%: it's the only connector here whose work outlives the agent's own turn, so it sits - at the end of the row the run reads left-to-right (AI -> Memory -> - Tool -> Parser). + at the end of the row the run reads left-to-right (Model -> + Sub-Agents -> Memory -> Tool -> Parser). Capped at one (no `alwaysVisible`) -- two contracts would be two answers to "what shape is this agent's output". */} {/* Keep the handle mounted even while its affordances are hidden. @@ -377,9 +416,10 @@ export function AgentNode({ leaves React Flow one measurement behind and the edge can stay visually detached until another canvas update. Opacity hides an unused handle without removing the endpoint React Flow registers. */} - )} - - + {!isSubAgent && ( + <> + + Agent + + + )}
) } diff --git a/frontend/src/components/protocol/nodes/ConnectorHandleLabel.tsx b/frontend/src/components/protocol/nodes/ConnectorHandleLabel.tsx index 4d51686..fc9750f 100644 --- a/frontend/src/components/protocol/nodes/ConnectorHandleLabel.tsx +++ b/frontend/src/components/protocol/nodes/ConnectorHandleLabel.tsx @@ -17,11 +17,11 @@ // those x-positions are forced), so horizontal padding here is the one // dimension that can't grow without them colliding. // -// Offset a step further out than the handle needs (top/bottom-5, right-10) -// so the chip clears the connector's own 8px dot rather than crowding it -- -// the caption names the connector, it isn't part of it. +// Top/bottom captions clear their connector dot. Side captions are centered +// on the same boundary position and sit just above the dot, keeping the Agent +// flow label visually attached to its connector. const LABEL_CLASSNAME = - 'absolute rounded bg-background/70 px-0.5 text-[0.6rem] font-semibold whitespace-nowrap text-[color:var(--node-label)]' + 'pointer-events-none absolute rounded bg-background/70 px-0.5 text-[0.6rem] font-semibold whitespace-nowrap text-[color:var(--node-label)]' export function ConnectorHandleLabel({ left, @@ -34,12 +34,19 @@ export function ConnectorHandleLabel({ // McpToolNode's main output and its Tool connector) each get their own // vertical slot instead of both defaulting to dead center. top?: string - side?: 'bottom' | 'right' | 'top' + side?: 'bottom' | 'left' | 'right' | 'top' children: string }) { if (side === 'right') { return ( - + + {children} + + ) + } + if (side === 'left') { + return ( + {children} ) @@ -47,7 +54,7 @@ export function ConnectorHandleLabel({ if (side === 'top') { // Centered directly ABOVE its handle -- an exact mirror of the bottom // branch below, so the three top-edge captions (Pattern / Skill / - // Resource) read the same way as AI / Memory / Tool do underneath. They + // Resource) read the same way as Model / Memory / Tool do underneath. They // used to hang off to one side of the handle instead, which meant a // caption's own position had to be reasoned about per-connector // (left-hanging near the right corner, right-hanging near the left one); diff --git a/frontend/src/components/protocol/nodes/CriticGateNode.tsx b/frontend/src/components/protocol/nodes/CriticGateNode.tsx index 9fcc654..634d2cb 100644 --- a/frontend/src/components/protocol/nodes/CriticGateNode.tsx +++ b/frontend/src/components/protocol/nodes/CriticGateNode.tsx @@ -1,4 +1,4 @@ -import { Handle, Position, useReactFlow, type NodeProps } from '@xyflow/react' +import { Handle, Position, useNodeConnections, useReactFlow, type NodeProps } from '@xyflow/react' import { ShieldCheck, ShieldOff } from 'lucide-react' import { Badge } from '@/components/ui/badge' import { cardAccent } from '@/lib/utils' @@ -25,6 +25,7 @@ export function CriticGateNode({ const Icon = enabled ? ShieldCheck : ShieldOff const summary = enabled ? `Up to ${data.config?.max_revisions ?? 1} revision(s)` : 'Gate disabled' const badge = nodeRunBadge(data.runStatus, data.runTruncated) + const modelConnections = useNodeConnections({ id, handleType: 'target', handleId: 'model' }) const { updateNodeData } = useReactFlow() const { requestMakeFactor } = useProtocolCanvasActions() @@ -53,7 +54,7 @@ export function CriticGateNode({ )} {hasBoundFactor(data) && } {/* Main pipeline flow is left-to-right -- input on the left, output on - the right, same convention as AgentNode. The AI sub-connector + the right, same convention as AgentNode. The Model sub-connector stays on the bottom edge regardless. */} {summary}

- {/* Same required AI connector an agent node has (handle id `ai`, see - AgentNode.tsx's own note on the pre-rename `llm` spelling) -- no + {/* Same required Model connector an agent node has (handle id `model`, see + AgentNode.tsx's note on the pre-rename spellings) -- no Tool/Memory slots here, gates never use tools and are always single-pass. */} - AI - + Model + 0 ? warnings : undefined} factorCount={boundFactorCount(data)} /> diff --git a/frontend/src/components/protocol/nodes/NodeFactorBadge.tsx b/frontend/src/components/protocol/nodes/NodeFactorBadge.tsx index faae3c7..1e0d1bb 100644 --- a/frontend/src/components/protocol/nodes/NodeFactorBadge.tsx +++ b/frontend/src/components/protocol/nodes/NodeFactorBadge.tsx @@ -11,11 +11,11 @@ import { Split } from 'lucide-react' // nothing about a 16px icon said *what* it marked: a `size-7` disc (exactly // half CircleNode's own `size-14`, so it reads as "attached to this node", // never as a node in its own right) with a "N factor(s)" caption under it, the -// same circle-above-label idiom every Pattern/LLM/Tool node already uses. +// same circle-above-label idiom every Pattern/Model/Tool node already uses. // // Violet (`--chart-2`), not the host node's `--card-accent`: violet is already // this app's factor hue (FACTOR_TRIGGER_CLASSNAME's own button, the factor -// editor dialog), and CLAUDE.md's colour rule wants a tint to mean something +// editor dialog), and AGENTS.md's colour rule wants a tint to mean something // -- here it means "factor", the one thing this badge is for, so it must not // dissolve into whatever hue the node underneath happens to be. // diff --git a/frontend/src/components/protocol/okfCatalog.ts b/frontend/src/components/protocol/okfCatalog.ts index 284173a..866393c 100644 --- a/frontend/src/components/protocol/okfCatalog.ts +++ b/frontend/src/components/protocol/okfCatalog.ts @@ -33,6 +33,7 @@ export function nodeDataForBundle(bundle: OkfBundle): OkfBundleNodeData { server_name: bundle.name, bundle_path: bundle.path, bundle_label: folder, + bundle_description: `Open Knowledge Format bundle ${folder}`, tool_names: bundle.tool_names, enabled: true, }, @@ -57,6 +58,9 @@ export function nodeDataForDocument(doc: OkfDocument): OkfDocumentNodeData { document_id: doc.id, server_name: doc.name, document_title: doc.title, + document_description: doc.description, + document_type: doc.concept_type, + document_tags: doc.tags, document_path: doc.path, tool_names: doc.tool_names, enabled: true, diff --git a/frontend/src/components/protocol/runSummary.ts b/frontend/src/components/protocol/runSummary.ts index a730212..3560858 100644 --- a/frontend/src/components/protocol/runSummary.ts +++ b/frontend/src/components/protocol/runSummary.ts @@ -1,27 +1,34 @@ import type { Edge, Node } from '@xyflow/react' import type { DatasetNodeData, - LlmNodeData, + ModelNodeData, McpToolNodeData, OkfBundleNodeData, OkfDocumentNodeData, SkillNodeData, } from '@/types/protocols' -// Plain duplicate of ProtocolCanvas.tsx's own LLM_NODE_TYPES rather than a +// Plain duplicate of ProtocolCanvas.tsx's own MODEL_NODE_TYPES rather than a // shared import -- same reasoning as nodeConfigIssues.ts's own comment: // keeps this module from being coupled to that component's internals. -const LLM_NODE_TYPES = new Set(['llm_anthropic', 'llm_openai', 'llm_azure_foundry']) +const MODEL_NODE_TYPES = new Set([ + 'model_anthropic', + 'model_openai', + 'model_azure_foundry', + 'model_openrouter', + 'model_local', +]) // Ditto for the MCP-tool family (mcpServerCatalog.ts's MCP_TOOL_NODE_TYPES) // -- every type in it carries the same config, so a run summary reads the // server name off any of them identically. const MCP_TOOL_NODE_TYPES = ['mcp_tool', 'mcp_scikit_learn', 'mcp_client_tool'] -// "llm" and "resource" are the pre-rename spellings of "ai" and "dataset" +// "llm" and "resource" are the pre-rename spellings of "model" and "dataset" // (see migrateLegacyHandles in ProtocolCanvas.tsx) -- kept here, as in that // file's CONNECTOR_HANDLES, so this stays a question of "is this a connector // edge at all" rather than one that silently answers no for a graph that // hasn't been normalised yet. const DEPENDENCY_HANDLES = new Set([ + 'model', 'ai', 'llm', 'tool', @@ -67,7 +74,7 @@ export interface RunSummary { // (a replicate run is the same graph, just with factor_values substituted); for // scope "node" (the per-node Play icon) only that node's own directly-wired // dependencies count, mirroring the one-level connector traversal -// services.protocol_execution's _resolve_llm_config/_resolve_tool_config/ +// services.protocol_execution's _resolve_model_config/_resolve_tool_config/ // _resolve_dataset_configs do server-side -- kept as a client-side duplicate // for the same reason nodeConfigIssues.ts already is. export function summarizeRun(nodes: Node[], edges: Edge[], scope: RunScope): RunSummary { @@ -116,9 +123,9 @@ export function summarizeRun(nodes: Node[], edges: Edge[], scope: RunScope): Run ) const models = uniq( relevantNodes - .filter((n) => LLM_NODE_TYPES.has(n.type ?? '')) + .filter((n) => MODEL_NODE_TYPES.has(n.type ?? '')) .map((n) => { - const config = (n.data as LlmNodeData).config + const config = (n.data as ModelNodeData).config return config?.model ? `${config.provider}/${config.model}` : null }) .filter((m): m is string => !!m), diff --git a/frontend/src/components/protocol/useProviderModels.ts b/frontend/src/components/protocol/useProviderModels.ts index 29bb795..33653e9 100644 --- a/frontend/src/components/protocol/useProviderModels.ts +++ b/frontend/src/components/protocol/useProviderModels.ts @@ -2,7 +2,7 @@ import { useQuery } from '@tanstack/react-query' import { llmSettingsApi } from '@/api/client' // The one place the provider model list is fetched. Every consumer -- the -// canvas node cards, the inspector, an llm_config factor level -- goes +// canvas node cards, the inspector, a model_config factor level -- goes // through this so they share a single cache entry rather than three // near-identical useQuery calls that can drift apart (FactorEditorDialog was // already missing the azure credential gate the other two had). @@ -22,7 +22,7 @@ export const providerModelsKey = (provider: string | undefined) => ['llm-setting // Model lists turn over on the order of weeks, but the default QueryClient // (main.tsx) sets no staleTime, so every inspector open, canvas mount and // window refocus refired this -- data already in cache, request still sent. -// Two LLM nodes for the same provider therefore cost two requests even +// Two Model nodes for the same provider therefore cost two requests even // though they render one identical list. That's also live against a real // 10-per-60s limiter on GET /llm-settings/{provider}/models, so the old // behaviour could 429 a canvas with a few nodes and some tab-switching. diff --git a/frontend/src/index.css b/frontend/src/index.css index 48f9a4c..d76f3d0 100644 --- a/frontend/src/index.css +++ b/frontend/src/index.css @@ -116,7 +116,7 @@ Eight because the canvas used to hash a node kind into the five chart hues, which made repeats arithmetic rather than unlucky: thirteen kinds - across five hues left buckets of three and four, so Skill and AI came + across five hues left buckets of three and four, so Skill and Model came out the same color, as did Dataset/Knowledge and Pattern/Script. The fix keeps one kind per bucket on its familiar chart hue (the canvas shouldn't recolor itself wholesale over a collision) and moves the rest @@ -141,7 +141,7 @@ 227, 253, 280, 307, 333, 359 -- every gap is ~25deg except 25->60, and the wide-looking 60->105 gap is NOT free, because --node-label's yellow sits at 92. So this takes the one real gap (42, between Critic's 25 and - the LLM family's 60) and leans on the lightness/chroma axis the block's + the Model family's 60) and leans on the lightness/chroma axis the block's comment above already establishes for exactly this case: at 0.88/0.1 it's the lightest and least chromatic hue on the canvas, a washed salmon, where both neighbours are saturated (0.18-0.17 chroma) and @@ -149,7 +149,10 @@ that's the point to ask whether thirteen-plus kinds should still be distinguished by color alone. */ --node-9: oklch(0.88 0.1 42); - /* Connector captions (Pattern/Skill/Dataset/Knowledge/AI/Memory/Tool on + /* Sub-Agent uses a bright electric blue for stronger contrast with the + Agent's emerald while remaining in the same cool-hued agent family. */ + --node-10: oklch(0.8 0.17 220); + /* Connector captions (Pattern/Skill/Dataset/Knowledge/Model/Memory/Tool on AgentNode and CriticGateNode). One fixed yellow rather than the host node's own accent: these sit outside the card on the canvas backdrop, where they have to be found rather than merely read, and yellow is the @@ -212,4 +215,4 @@ html { @apply font-sans; } -} \ No newline at end of file +} diff --git a/frontend/src/lib/agentOutputMetrics.ts b/frontend/src/lib/agentOutputMetrics.ts index 3623529..14c799c 100644 --- a/frontend/src/lib/agentOutputMetrics.ts +++ b/frontend/src/lib/agentOutputMetrics.ts @@ -12,7 +12,7 @@ export interface AgentOutputSourceOption { export function agentOutputSourceOptions(graph: ProtocolGraph | undefined): AgentOutputSourceOption[] { if (!graph) return [] return graph.nodes.flatMap((node) => { - if (node.type !== 'agent') return [] + if (node.type !== 'agent' && node.type !== 'sub_agent') return [] const disabledReason = node.data.active === false ? 'Agent is disabled.' : undefined return [{ agentNodeId: node.id, label: node.data.label || node.id, disabledReason }] }) diff --git a/frontend/src/lib/contextualMetricSuggestions.ts b/frontend/src/lib/contextualMetricSuggestions.ts index 53111a2..cc5b642 100644 --- a/frontend/src/lib/contextualMetricSuggestions.ts +++ b/frontend/src/lib/contextualMetricSuggestions.ts @@ -7,7 +7,7 @@ export interface ContextualMetricSuggestion { } function enabledAgent(node: ProtocolNode | undefined): boolean { - return node?.type === 'agent' && node.data.active !== false + return (node?.type === 'agent' || node?.type === 'sub_agent') && node.data.active !== false } function sourceIsCallable(node: ProtocolNode, targetHandle: string | null | undefined): boolean { diff --git a/frontend/src/lib/coordinationStrategy.ts b/frontend/src/lib/coordinationStrategy.ts index 0650491..b20f467 100644 --- a/frontend/src/lib/coordinationStrategy.ts +++ b/frontend/src/lib/coordinationStrategy.ts @@ -3,10 +3,11 @@ import type { AgentNodeData, ProtocolEdge, ProtocolGraph, ProtocolNode } from '@ // Mirrors services.protocol_execution's own _CONNECTOR_HANDLES -- any edge // whose targetHandle ISN'T one of these is a plain "main" pipeline edge. -// Includes the pre-rename "llm" and "resource" spellings for the same reason +// Includes the pre-rename "ai", "llm", and "resource" spellings for the same reason // the backend set does: a graph that hasn't been through migrateLegacyHandles -// yet must not have its AI/Dataset edges misread as main pipeline edges. +// yet must not have its Model/Dataset edges misread as main pipeline edges. export const CONNECTOR_HANDLES = new Set([ + 'model', 'ai', 'llm', 'tool', @@ -17,6 +18,7 @@ export const CONNECTOR_HANDLES = new Set([ 'resource', 'knowledge', 'output_parser', + 'sub_agents', ]) export function isMainEdge(edge: Pick): boolean { diff --git a/frontend/src/lib/experiment.ts b/frontend/src/lib/experiment.ts index 6ee26e0..776185c 100644 --- a/frontend/src/lib/experiment.ts +++ b/frontend/src/lib/experiment.ts @@ -5,10 +5,10 @@ export function factorCount(designSpec: Experiment['design_spec']): number | nul return Array.isArray(factors) ? factors.length : null } -// A "whole node as a factor" level (LLM/Tool config, pattern override) is a +// A "whole node as a factor" level (Model/Tool config, pattern override) is a // plain object, not a scalar -- JS's default `String({...})` collapses every // distinct object to the same "[object Object]", which would silently -// conflate different LLM/Tool/Pattern configs wherever this codebase +// conflate different Model/Tool/Pattern configs wherever this codebase // compares/dedupes/sorts factor values by their string form (this module's own // deriveFactors/replicatesMatching). A canonical (recursively key-sorted) JSON // string is stable across separately-deserialized-but-content-identical @@ -48,7 +48,7 @@ const FACTOR_VALUE_DISPLAY_PRIORITY_KEYS = [ ] /** Human-readable rendering of a factor value for table/badge display -- - * scalars render as-is; a dict-valued "whole node" level (LLM/Tool config, + * scalars render as-is; a dict-valued "whole node" level (Model/Tool config, * pattern override) picks its own most identifying field instead of * JS's default object stringification. */ export function displayFactorValue(value: unknown): string { @@ -241,6 +241,7 @@ const PREFERRED_METRICS = ['average_precision', 'roc_auc', 'accuracy', 'f1'] const METRIC_LABEL_OVERRIDES: Record = { cost_usd: 'cost (USD)', duration_s: 'duration (minutes)', + duration_seconds: 'duration (seconds)', n_features_created: 'n eng. feat.', n_created_selected: 'n eng. feat. selected', frac_created_selected: '% eng. feat. selected', @@ -265,6 +266,7 @@ export function scaledMetricValue(key: string, value: number): number { * scaledMetricValue so a metric can add a unit suffix without needing its * own numeric rescale (or vice versa). */ const METRIC_VALUE_SUFFIXES: Record = { + duration_seconds: ' sec', frac_created_selected: '%', } @@ -327,7 +329,8 @@ export function formatMetricValue(key: string, value: unknown): string { // custom metrics. The Results detail panel surfaces these values in a // compact grid where an ungrouped `12500` is needlessly hard to scan. const formatted = new Intl.NumberFormat('en-US', { maximumFractionDigits: 4 }).format(scaledMetricValue(key, value)) - return `${formatted}${metricValueSuffix(key)}` + const prefix = key === 'cost_usd' ? '$' : '' + return `${prefix}${formatted}${metricValueSuffix(key)}` } /** Every cell whose factor_values match `match` on all of its keys -- i.e. one diff --git a/frontend/src/lib/mcpToolMetrics.ts b/frontend/src/lib/mcpToolMetrics.ts index bcccac6..fecdc97 100644 --- a/frontend/src/lib/mcpToolMetrics.ts +++ b/frontend/src/lib/mcpToolMetrics.ts @@ -24,7 +24,7 @@ export function mcpToolSourceOptions(graph: ProtocolGraph | undefined): McpToolS return graph.edges.flatMap((edge) => { if (edge.targetHandle !== 'tool') return [] const agent = nodes.get(edge.target); const tool = nodes.get(edge.source) - if (!agent || !tool || agent.type !== 'agent' || !isMcpToolNodeType(tool.type)) return [] + if (!agent || !tool || !['agent', 'sub_agent'].includes(agent.type) || !isMcpToolNodeType(tool.type)) return [] const config = 'config' in tool.data ? tool.data.config : undefined const serverId = config && 'server_id' in config ? config.server_id : null const toolNames = config && 'tool_names' in config && Array.isArray(config.tool_names) ? config.tool_names : [] diff --git a/frontend/src/lib/measurementPlan.ts b/frontend/src/lib/measurementPlan.ts index a4d0566..27fc069 100644 --- a/frontend/src/lib/measurementPlan.ts +++ b/frontend/src/lib/measurementPlan.ts @@ -106,7 +106,7 @@ export function localMetricReadinessPreview( const producer = producerLabel(binding.producer_id) if (binding.producer_id === AGENT_OUTPUT_PRODUCER_ID) { const agentNodeId = String(binding.config.agent_node_id ?? '') - const agent = graph?.nodes.find((node) => node.id === agentNodeId && node.type === 'agent') + const agent = graph?.nodes.find((node) => node.id === agentNodeId && (node.type === 'agent' || node.type === 'sub_agent')) if (!agent) return { ready: false, producer, detail: `Agent ${agentNodeId || '(not selected)'} is not available.` } if (agent.data.active === false) return { ready: false, producer, detail: 'The Agent is disabled.' } return { ready: true, producer, detail: 'The Agent final output will be captured after execution.' } @@ -123,7 +123,7 @@ export function localMetricReadinessPreview( const agentNodeId = String(binding.config.agent_node_id ?? '') const script = graphNodes.find((node) => node.id === scriptNodeId && node.type === 'script') const config = script && 'config' in script.data ? script.data.config : undefined - if (!graphNodes.some((node) => node.id === agentNodeId && node.type === 'agent')) { + if (!graphNodes.some((node) => node.id === agentNodeId && (node.type === 'agent' || node.type === 'sub_agent'))) { return { ready: false, producer, detail: `Source Agent ${agentNodeId || '(not selected)'} is not available.` } } if (!graphEdges.some((edge) => edge.source === scriptNodeId && edge.target === agentNodeId && edge.targetHandle === 'tool')) { @@ -141,7 +141,7 @@ export function localMetricReadinessPreview( const toolNodeId = String(binding.config.mcp_node_id ?? '') const tool = graph?.nodes.find((node) => node.id === toolNodeId && ['mcp_tool', 'mcp_scikit_learn', 'mcp_client_tool'].includes(node.type)) const config = tool && 'config' in tool.data ? tool.data.config as unknown as Record : undefined - if (!graph?.nodes.some((node) => node.id === agentNodeId && node.type === 'agent')) return { ready: false, producer, detail: 'The source Agent is unavailable.' } + if (!graph?.nodes.some((node) => node.id === agentNodeId && (node.type === 'agent' || node.type === 'sub_agent'))) return { ready: false, producer, detail: 'The source Agent is unavailable.' } if (!tool || !config) return { ready: false, producer, detail: 'The MCP Tool is unavailable.' } if (!graph?.edges.some((edge) => edge.source === toolNodeId && edge.target === agentNodeId && edge.targetHandle === 'tool')) return { ready: false, producer, detail: 'The source Agent and MCP Tool are not directly connected.' } if (config.enabled === false) return { ready: false, producer, detail: 'The MCP Tool is disabled.' } diff --git a/frontend/src/lib/nodeAccent.ts b/frontend/src/lib/nodeAccent.ts index dff0c70..9aea8f3 100644 --- a/frontend/src/lib/nodeAccent.ts +++ b/frontend/src/lib/nodeAccent.ts @@ -6,7 +6,7 @@ // while the palette was wider than the set of things being colored, but there // are fourteen node kinds and five --chart-* hues, so repeats weren't a risk, // they were arithmetic -- and they landed on exactly the pairs a reader -// confuses: Skill and AI, Dataset and Knowledge, Pattern and Script all came +// confuses: Skill and Model, Dataset and Knowledge, Pattern and Script all came // out the same color. Color on this canvas answers "what kind of node is // this", which is a closed, small, slow-changing set, so it gets a table. // hashToChartHue stays for the open-ended cases it was written for (a model @@ -19,20 +19,21 @@ // kind that anchors each bucket is the one whose color is most load-bearing -- // the agent (the canvas's protagonist), the dataset and the reason+act pattern // (the two collisions were reported against them), the critic gate (red, and -// it's the only node whose job is to stop a run), and the LLM family. +// it's the only node whose job is to stop a run), and the Model family. const NODE_ACCENTS: Record = { agent: 'var(--chart-3)', + sub_agent: 'var(--node-10)', dataset: 'var(--chart-2)', pattern_reason_act: 'var(--chart-1)', critic_gate: 'var(--chart-5)', - // One hue for the whole LLM family (llm_anthropic/llm_openai/...), which is + // One hue for the whole Model family (model_anthropic/model_openai/...), which is // a change: it used to hash the PROVIDER, giving each its own color. That - // read as five unrelated node kinds, and it's what put an AI node on Skill's - // color. Providers stay distinguishable by icon and label (LlmNode's + // read as five unrelated node kinds, and it's what put a Model node on Skill's + // color. Providers stay distinguishable by icon and label (ModelNode's // PROVIDER_META) -- color here means "this is the model", not "which // vendor". --chart-4 because that's what the hash gave the Azure/local - // providers, so the most common AI nodes keep the color they had. - llm: 'var(--chart-4)', + // providers, so the most common Model nodes keep the color they had. + model: 'var(--chart-4)', // The movers. Adjacent slots go to related kinds on purpose (the two MCP // kinds, the two OKF kinds): where two colors are closest, the things they // mark are most alike, so a mix-up costs the least. @@ -50,7 +51,7 @@ const NODE_ACCENTS: Record = { } /** The accent color for a canvas node kind — pass the same key the node card - * and its inspector both use (`'skill'`, `'okf_bundle'`, `'llm'`). An unknown + * and its inspector both use (`'skill'`, `'okf_bundle'`, `'model'`). An unknown * kind falls back to `--primary` rather than to a hashed hue: a new node type * showing up in the theme's own accent is a visible prompt to give it a slot * here, where a plausible-looking hash collision would just look intentional. */ diff --git a/frontend/src/lib/nodeNames.test.ts b/frontend/src/lib/nodeNames.test.ts index cac2a14..29ae241 100644 --- a/frontend/src/lib/nodeNames.test.ts +++ b/frontend/src/lib/nodeNames.test.ts @@ -10,7 +10,7 @@ describe('nodeDisplayNames', () => { it('falls back to the placeholder the unlabelled node shows on its card', () => { const names = nodeDisplayNames([ { id: 'n1', type: 'output_parser', data: { label: '' } }, - { id: 'n2', type: 'llm_anthropic', data: {} }, + { id: 'n2', type: 'model_anthropic', data: {} }, ]) expect(names.get('n1')).toBe('Output Parser') expect(names.get('n2')).toBe('Anthropic') diff --git a/frontend/src/lib/nodeNames.ts b/frontend/src/lib/nodeNames.ts index f0fd38e..6b3e677 100644 --- a/frontend/src/lib/nodeNames.ts +++ b/frontend/src/lib/nodeNames.ts @@ -10,13 +10,14 @@ // placeholders; keep it in step with the node components if one is renamed. const NODE_TYPE_NAMES: Record = { agent: 'Agent', + sub_agent: 'Sub-Agent', critic_gate: 'Critic Gate', dataset: 'Dataset', - llm_anthropic: 'Anthropic', - llm_openai: 'OpenAI', - llm_azure_foundry: 'Azure AI Foundry', - llm_openrouter: 'OpenRouter', - llm_local: 'Local', + model_anthropic: 'Anthropic', + model_openai: 'OpenAI', + model_azure_foundry: 'Azure AI Foundry', + model_openrouter: 'OpenRouter', + model_local: 'Local', mcp_tool: 'MCP Tool', mcp_client_tool: 'MCP Client Tool', memory: 'Memory', diff --git a/frontend/src/lib/promptReferences.ts b/frontend/src/lib/promptReferences.ts index d79b95e..3cab7cc 100644 --- a/frontend/src/lib/promptReferences.ts +++ b/frontend/src/lib/promptReferences.ts @@ -87,7 +87,7 @@ export function isPromptReferenceField(fieldPath: string): boolean { function baseName(node: ProtocolNode): string { const label = (node.data as { label?: string } | undefined)?.label - return label?.trim() || (node.type === 'agent' ? 'Agent' : node.type) + return label?.trim() || (node.type === 'agent' ? 'Agent' : node.type === 'sub_agent' ? 'Sub-Agent' : node.type) } /** A unique, human display name per node id. diff --git a/frontend/src/lib/pythonScriptMetrics.ts b/frontend/src/lib/pythonScriptMetrics.ts index df9907a..f91ec81 100644 --- a/frontend/src/lib/pythonScriptMetrics.ts +++ b/frontend/src/lib/pythonScriptMetrics.ts @@ -16,7 +16,7 @@ export function pythonScriptSourceOptions(graph: ProtocolGraph | undefined): Pyt if (edge.targetHandle !== 'tool') return [] const agent = nodes.get(edge.target) const script = nodes.get(edge.source) - if (!agent || !script || agent.type !== 'agent' || script.type !== 'script') return [] + if (!agent || !script || !['agent', 'sub_agent'].includes(agent.type) || script.type !== 'script') return [] const config = 'config' in script.data ? script.data.config : undefined const code = config && 'code' in config ? config.code : '' const enabled = !(config && 'enabled' in config && config.enabled === false) diff --git a/frontend/src/lib/reasonActIterations.ts b/frontend/src/lib/reasonActIterations.ts index 7bd89be..6bc296c 100644 --- a/frontend/src/lib/reasonActIterations.ts +++ b/frontend/src/lib/reasonActIterations.ts @@ -40,7 +40,13 @@ function connectorCost(node: ProtocolNode): number { if (!isEnabled(node)) return 0 if (node.type === 'script') return SCRIPT_ITERATIONS if (isMcpToolNodeType(node.type)) return CONNECTOR_ITERATIONS - if (node.type === 'skill' || node.type === 'dataset' || node.type === 'okf_bundle' || node.type === 'okf_document') { + if ( + node.type === 'skill' || + node.type === 'dataset' || + node.type === 'okf_bundle' || + node.type === 'okf_document' || + node.type === 'sub_agent' + ) { return CONNECTOR_ITERATIONS } return 0 @@ -66,7 +72,7 @@ export function suggestedMaxIterations(graph: ProtocolGraph, patternNodeId: stri const agentIds = graph.edges .filter((edge) => edge.source === patternNodeId && edge.targetHandle === 'architectural_pattern') .map((edge) => edge.target) - .filter((id) => nodes.get(id)?.type === 'agent') + .filter((id) => ['agent', 'sub_agent'].includes(nodes.get(id)?.type ?? '')) if (agentIds.length === 0) return null // The max across agents, not the sum: each agent runs its own loop, and the diff --git a/frontend/src/pages/AGENTS.md b/frontend/src/pages/AGENTS.md new file mode 100644 index 0000000..5a8c3a7 --- /dev/null +++ b/frontend/src/pages/AGENTS.md @@ -0,0 +1,30 @@ +# Experiments-list page UI details + +Scoped detail for `ExperimentsPage.tsx` — this only loads when you're working in this +directory. See the root `AGENTS.md`'s "Tables vs. cards is a zoom-level decision, not a style +preference" principle for the general rule this detail implements: a "container" list — sparse +metadata, click one to drill into it (e.g. the Experiments list) — is a card/tile grid, not a +table; a table only earns its place once you're inside that container looking at dense, +precise, multi-attribute records to compare (e.g. an experiment's Cells, with their +metrics/hyperparameters). + +- **Tile grids** (`ExperimentsPage.tsx`): `grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 + gap-4` of `Card`s, each with a truncated title + `line-clamp-3` description (`title=` + attribute carries the full text on hover, since the card stays compact rather than + growing to fit long text — the full description belongs one click in) + a small metadata + badge/date — not a dense data dump. Client-side sort via a `Select` + direction + toggle (column-header sort doesn't exist once there are no columns). Don't add a KPI + summary strip above the grid unless there's a real aggregate to show (e.g. cells + scored) — a stat card restating the list's own item count, or one that just repeats a + field already visible on every tile (design type, most-recent item), isn't a summary, + it's noise. Per-tile polish that *is* worth it: a design-type icon next to its badge, a + thin glowing top accent strip, relative time (`"2h ago"`, exact date on hover via + `title`), a short `font-mono` id fragment, a very low-opacity decorative watermark icon, + and a hover lift (`scale-[1.02]` + intensified glow + the trailing arrow shifting to + the card's accent color). +- **A tile click goes to the protocol canvas** (`/experiments/{id}/protocol`), not to a + static detail page — there isn't one, and `/experiments/{id}` exists only as a redirect + (see `App.tsx`). Everything about a single experiment lives in the canvas's side-panel + tabs; the rules for the Cells heatmap/table specifically are in + `components/protocol/cells/AGENTS.md`. +- Paginate client-side once a list can realistically exceed ~10-20 tiles. diff --git a/frontend/src/pages/CLAUDE.md b/frontend/src/pages/CLAUDE.md index 347a024..43c994c 100644 --- a/frontend/src/pages/CLAUDE.md +++ b/frontend/src/pages/CLAUDE.md @@ -1,30 +1 @@ -# Experiments-list page UI details - -Scoped detail for `ExperimentsPage.tsx` — this only loads when you're working in this -directory. See the root `CLAUDE.md`'s "Tables vs. cards is a zoom-level decision, not a style -preference" principle for the general rule this detail implements: a "container" list — sparse -metadata, click one to drill into it (e.g. the Experiments list) — is a card/tile grid, not a -table; a table only earns its place once you're inside that container looking at dense, -precise, multi-attribute records to compare (e.g. an experiment's Cells, with their -metrics/hyperparameters). - -- **Tile grids** (`ExperimentsPage.tsx`): `grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 - gap-4` of `Card`s, each with a truncated title + `line-clamp-3` description (`title=` - attribute carries the full text on hover, since the card stays compact rather than - growing to fit long text — the full description belongs one click in) + a small metadata - badge/date — not a dense data dump. Client-side sort via a `Select` + direction - toggle (column-header sort doesn't exist once there are no columns). Don't add a KPI - summary strip above the grid unless there's a real aggregate to show (e.g. cells - scored) — a stat card restating the list's own item count, or one that just repeats a - field already visible on every tile (design type, most-recent item), isn't a summary, - it's noise. Per-tile polish that *is* worth it: a design-type icon next to its badge, a - thin glowing top accent strip, relative time (`"2h ago"`, exact date on hover via - `title`), a short `font-mono` id fragment, a very low-opacity decorative watermark icon, - and a hover lift (`scale-[1.02]` + intensified glow + the trailing arrow shifting to - the card's accent color). -- **A tile click goes to the protocol canvas** (`/experiments/{id}/protocol`), not to a - static detail page — there isn't one, and `/experiments/{id}` exists only as a redirect - (see `App.tsx`). Everything about a single experiment lives in the canvas's side-panel - tabs; the rules for the Cells heatmap/table specifically are in - `components/protocol/cells/CLAUDE.md`. -- Paginate client-side once a list can realistically exceed ~10-20 tiles. +@AGENTS.md diff --git a/frontend/src/pages/profile/LlmCredentialsSection.tsx b/frontend/src/pages/profile/LlmCredentialsSection.tsx index 509d88a..1263913 100644 --- a/frontend/src/pages/profile/LlmCredentialsSection.tsx +++ b/frontend/src/pages/profile/LlmCredentialsSection.tsx @@ -18,7 +18,7 @@ import { Skeleton } from '@/components/ui/skeleton' import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from '@/components/ui/table' import { CreateCredentialDialog } from '@/components/CreateCredentialDialog' import { ConnectionStatusBadge, useConnectionCheck } from '@/components/LlmConnectionCheck' -import { PROVIDER_META } from '@/components/protocol/nodes/LlmNode' +import { PROVIDER_META } from '@/components/protocol/nodes/ModelNode' import { LLM_PROVIDER_LABELS, type LLMProvider, type LLMSetting } from '@/types/llmSettings' /** On-demand, zero-token credential check -- a button rather than something diff --git a/frontend/src/types/experiments.ts b/frontend/src/types/experiments.ts index 2632113..5e74569 100644 --- a/frontend/src/types/experiments.ts +++ b/frontend/src/types/experiments.ts @@ -15,7 +15,7 @@ export interface DesignFactor { | 'text' | 'number' | 'boolean' - | 'llm_config' + | 'model_config' | 'tool_config' | 'pattern' | 'script_config' diff --git a/frontend/src/types/llmSettings.ts b/frontend/src/types/llmSettings.ts index 871a667..ef1b784 100644 --- a/frontend/src/types/llmSettings.ts +++ b/frontend/src/types/llmSettings.ts @@ -20,7 +20,7 @@ export interface LLMSetting { // Matches ASAREE's backend SUPPORTED_PROVIDERS (credential_resolver.py) -- // every provider a per-user credential can be resolved for. The single // shared source for provider display info -- CreateCredentialDialog's -// provider picker and the LLM node inspector's Credential section both read +// provider picker and the Model node inspector's Credential section both read // this instead of keeping their own copy. export const LLM_PROVIDER_CATALOG: { id: LLMProvider; label: string; description: string }[] = [ { id: 'anthropic', label: 'Anthropic', description: 'Route requests through your own Anthropic account' }, diff --git a/frontend/src/types/protocols.test.ts b/frontend/src/types/protocols.test.ts new file mode 100644 index 0000000..cc27045 --- /dev/null +++ b/frontend/src/types/protocols.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'vitest' +import { + defaultAnthropicModelNodeData, + defaultAzureFoundryModelNodeData, + defaultLocalModelNodeData, + defaultOpenAiModelNodeData, + defaultOpenRouterModelNodeData, +} from './protocols' + +describe('Model node defaults', () => { + it('requires an explicit model selection for every provider', () => { + const defaults = [ + defaultAnthropicModelNodeData(), + defaultOpenAiModelNodeData(), + defaultAzureFoundryModelNodeData(), + defaultOpenRouterModelNodeData(), + defaultLocalModelNodeData(), + ] + + expect(defaults.map(({ config }) => config.model)).toEqual(['', '', '', '', '']) + }) +}) diff --git a/frontend/src/types/protocols.ts b/frontend/src/types/protocols.ts index 5465743..f3228f4 100644 --- a/frontend/src/types/protocols.ts +++ b/frontend/src/types/protocols.ts @@ -26,7 +26,7 @@ export interface ProtocolNode { | AgentNodeData | McpToolNodeData | CriticGateNodeData - | LlmNodeData + | ModelNodeData | MemoryNodeData | OutputParserNodeData | DatasetNodeData @@ -116,13 +116,13 @@ export interface ProtocolRun { id: string protocol_id: string // `limit_reached` is conversation-mode only: the agents were still talking - // when a budget (consultation count, depth, or the conversation wall clock) - // ran out. Distinct from `failed` because the work up to that point is + // when the recursive consultation-depth guard ran out. Distinct from + // `failed` because the work up to that point is // sound -- the transcript is worth reading. status: 'pending' | 'running' | 'finalizing' | 'completed' | 'failed' | 'cancelled' | 'limit_reached' node_runs: Record - // Null for every pipeline run; populated once a conversation-mode run's - // agents start talking. + // Populated once agents communicate, whether through a conversation + // strategy or a pipeline Agent delegating to a Sub-Agent. conversation: Conversation | null error: string | null // Both null for a plain graph run. Set together only for a run created by @@ -216,9 +216,9 @@ export interface AgentModelConfigData { model: string temperature?: number | null effort?: string | null - // Nullable so LlmNodeInspector's Input can be backspaced to empty without + // Nullable so ModelNodeInspector's Input can be backspaced to empty without // snapping to a forced value -- null is a real, persisted "not set yet" - // state flagged by LlmNode's warning triangle and nodeConfigIssues.ts's + // state flagged by ModelNode's warning triangle and nodeConfigIssues.ts's // pre-flight scan, same convention as ReasonActPatternConfig's fields. max_tokens: number | null } @@ -273,10 +273,10 @@ export interface AgentNodeConfig { // a purely presentational flag would orphan it on every canvas already saved. require_output_parser?: boolean // Model, tool assignment, and execution pattern are no longer fields - // here -- resolved from the node's required LLM connector, optional Tool + // here -- resolved from the node's required Model connector, optional Tool // connector(s), and optional Architectural Pattern connector instead (see - // LlmNodeData/McpToolNodeData/ReasonActPatternNodeData and - // services.protocol_execution's _resolve_llm_config/_resolve_tool_config/ + // ModelNodeData/McpToolNodeData/ReasonActPatternNodeData and + // services.protocol_execution's _resolve_model_config/_resolve_tool_config/ // _resolve_pattern_config) -- deliberately kept out of a node's own // settings. // @@ -385,7 +385,7 @@ export interface CriticGateNodeConfig { goal: string description: string system_prompt: string - // Resolved from the gate's required LLM connector instead, same as an + // Resolved from the gate's required Model connector instead, same as an // agent node -- see AgentNodeConfig's own comment. // Critic gates have no separate top-level `active` flag the way // AgentNodeData/McpToolNodeData do -- this field already means exactly @@ -417,25 +417,24 @@ export function defaultCriticGateNodeData(label = 'Critic Gate'): CriticGateNode } } -// The LLM connector's node family -- named to match this app's existing -// LLMProvider/LLMSetting vocabulary. One node type per provider -// (llm_anthropic/llm_openai/llm_azure_foundry/llm_openrouter/llm_local), not +// The Model connector's node family. One node type per provider +// (model_anthropic/model_openai/model_azure_foundry/model_openrouter/model_local), not // one generic node with a Provider field -- a dedicated node per capability // rather than one node with an internal picker. Config shape is identical // across all five (provider is baked into which node type you // picked, not user-editable), so they share this one config/data shape and -// -- see LlmNodeInspector.tsx -- one inspector component, varying only the +// -- see ModelNodeInspector.tsx -- one inspector component, varying only the // hardcoded `provider` each default-data factory below sets. Supplies // model/provider/temperature/effort/max_tokens to whichever agent/ -// critic_gate node(s) it's wired into via their required LLM connector. -// One LLM node's output can fan out to multiple agents (shared config, +// critic_gate node(s) it's wired into via their required Model connector. +// One Model node's output can fan out to multiple agents (shared config, // reused rather than re-entered per agent) -- nothing prevents this, though // there's no dedicated UI for it yet. -export type LlmNodeConfig = AgentModelConfigData +export type ModelNodeConfig = AgentModelConfigData -export interface LlmNodeData { +export interface ModelNodeData { label: string - config: LlmNodeConfig + config: ModelNodeConfig factor_bindings?: Record [key: string]: unknown } @@ -445,34 +444,32 @@ export interface LlmNodeData { // (asaree-spinal-use-case/spinal_pipeline.ipynb) -- used uniformly across // every one of its agents/critics, well under Motoro's own // ModelConfig cap of 200000. -export function defaultAnthropicLlmNodeData(label = 'Anthropic'): LlmNodeData { - return { label, config: { provider: 'anthropic', model: 'claude-sonnet-5', temperature: 0.7, max_tokens: 128000 } } +export function defaultAnthropicModelNodeData(label = 'Anthropic'): ModelNodeData { + return { label, config: { provider: 'anthropic', model: '', temperature: 0.7, max_tokens: 128000 } } } -export function defaultOpenAiLlmNodeData(label = 'OpenAI'): LlmNodeData { - return { label, config: { provider: 'openai', model: 'gpt-5', temperature: 0.7, max_tokens: 128000 } } +export function defaultOpenAiModelNodeData(label = 'OpenAI'): ModelNodeData { + return { label, config: { provider: 'openai', model: '', temperature: 0.7, max_tokens: 128000 } } } -export function defaultAzureFoundryLlmNodeData(label = 'Azure AI Foundry'): LlmNodeData { - return { label, config: { provider: 'azure_foundry', model: 'gpt-5', temperature: 0.7, max_tokens: 128000 } } +export function defaultAzureFoundryModelNodeData(label = 'Azure AI Foundry'): ModelNodeData { + return { label, config: { provider: 'azure_foundry', model: '', temperature: 0.7, max_tokens: 128000 } } } -export function defaultOpenRouterLlmNodeData(label = 'OpenRouter'): LlmNodeData { - return { label, config: { provider: 'openrouter', model: 'anthropic/claude-sonnet-5', temperature: 0.7, max_tokens: 128000 } } +export function defaultOpenRouterModelNodeData(label = 'OpenRouter'): ModelNodeData { + return { label, config: { provider: 'openrouter', model: '', temperature: 0.7, max_tokens: 128000 } } } -// model starts empty -- unlike every other provider here, there's no -// universal default self-hosted model name to assume (see -// CreateCredentialDialog.tsx's requiresApiBase for the matching "no default -// host" reasoning on api_base). The Model field's own required-field -// warning already flags this until the user picks one. -export function defaultLocalLlmNodeData(label = 'Local'): LlmNodeData { +// Every provider starts empty so adding a Model node never silently chooses a +// model the user's credential may not expose. The Model field's required-field +// warning remains until the user makes an explicit catalog/custom selection. +export function defaultLocalModelNodeData(label = 'Local'): ModelNodeData { return { label, config: { provider: 'local', model: '', temperature: 0.7, max_tokens: 128000 } } } // A "Memory" node -- visual/validation scaffolding only for now. Wiring one // into an Agent's Memory connector is accepted by the graph (validated the -// same way LLM/Tool connectors are) but has NO effect on execution yet -- +// same way Model/Tool connectors are) but has NO effect on execution yet -- // porting Motoro's actual episodic-memory service (already built, // Postgres+pgvector-backed, just not yet invoked anywhere in ASAREE's own // execution path) is an explicit, deliberate follow-up, not this phase. @@ -558,6 +555,10 @@ export function defaultOutputParserNodeData(label = 'Output Parser'): OutputPars export interface DatasetNodeConfig { dataset_id: string | null dataset_name: string | null + description?: string | null + target_column?: string | null + split_state?: 'split' | 'unsplit' | null + dictionary_available?: boolean // Absent means enabled, matching every other connector's own convention. enabled?: boolean } @@ -584,6 +585,7 @@ export function defaultDatasetNodeData(label = 'Dataset'): DatasetNodeData { // itself). export interface ScriptNodeConfig { name: string + description?: string language: 'python' code: string } @@ -596,7 +598,7 @@ export interface ScriptNodeData { } export function defaultScriptNodeData(label = 'Script'): ScriptNodeData { - return { label, config: { name: 'script', language: 'python', code: '' } } + return { label, config: { name: 'script', description: '', language: 'python', code: '' } } } // A "Skill" node -- names one registered Agent Skill for the Agent it's wired @@ -665,6 +667,7 @@ export interface OkfBundleNodeConfig { // the folder's own name. bundle_path: string | null bundle_label: string | null + bundle_description?: string | null // The bundle server's tools, BARE (e.g. "read_concept"), cached at // registration. Namespaced "{server_name}.{tool}" at resolve time, matching // McpToolNodeConfig. No per-tool picker in V1: a bundle's tools are a fixed @@ -709,6 +712,9 @@ export interface OkfDocumentNodeConfig { // rewrite the document's frontmatter mid-run, and the canvas card shouldn't // silently rename itself. The inspector shows the live values. document_title: string | null + document_description?: string | null + document_type?: string | null + document_tags?: string[] document_path: string | null // The document server's tools, BARE, cached at registration -- namespaced // "{server_name}.{tool}" at resolve time. No per-tool picker, same reason as @@ -735,11 +741,11 @@ export interface OkfDocumentNodeData { // services.protocol_execution's _resolve_pattern_config reads the wired // node's own config into a real Motoro PatternConfig, passed straight // into create_agent/update_agent. ASAREE-specific, alongside -// LLM/Tool/Memory. +// Model/Tool/Memory. // // One node type per pattern (pattern_reason_act/pattern_single_agent_baseline), // not one generic node with a Pattern-name field -- same reasoning as the -// LLM node family above, and unlike that family these genuinely have +// Model node family above, and unlike that family these genuinely have // different config shapes (Motoro's own `pattern_params` schema per // plugin, see engine/patterns/builtin/*.py), so each gets its own dedicated // inspector rather than sharing one. `PatternConfig` (Motoro) already diff --git a/frontend/src/types/skills.ts b/frontend/src/types/skills.ts index a7e4826..2c98b25 100644 --- a/frontend/src/types/skills.ts +++ b/frontend/src/types/skills.ts @@ -10,6 +10,12 @@ export interface Skill { name: string description: string body: string + frontmatter: { + license?: string + compatibility?: string + metadata?: Record + 'allowed-tools'?: string + } is_system: boolean // The folder the skill was uploaded from, or the .md file for a single-file // upload. Shown back to the user, never used to resolve anything. diff --git a/publications/bioinformatics/existing-tools-comparators.md b/publications/bioinformatics/existing-tools-comparators.md new file mode 100644 index 0000000..df00366 --- /dev/null +++ b/publications/bioinformatics/existing-tools-comparators.md @@ -0,0 +1,245 @@ +# Existing-tool comparators for ASAREE + +Research date: 2026-09-25. This is a primary-source scan prompted by the review +comment that ASAREE is "a potentially useful, openly available analytical +sandbox whose practical value is plausible, but whose advantage over existing +tools has not been demonstrated." Tools released after the manuscript's +comparison cut-off should be labeled as such rather than presented as omissions +from the original submission. + +## What ASAREE is claiming + +The repository describes ASAREE as a workbench for running LLM agents as +designed experiments: users visually assemble a protocol, bind factors to node +configuration, materialize a full factorial design, run replicates, and compare +recorded metrics. It also versions datasets and production protocol snapshots +and exposes tools through MCP ([project README](../../README.md)). The public +use case is an agent sequence for tabular biomedical ML, varied in a 2 × 2 × 2 +model/effort/critic design with ten replicates +([use-case README](README.md)). + +That description touches four established product/research categories. The +reviewer may mean any of them; the first two are the most direct. + +## 1. Evaluation and experiment platforms — closest functional comparators + +### Arize Phoenix (highest-priority comparator) + +Phoenix is open source and explicitly positions itself as a platform for +"experimentation, evaluation, and troubleshooting" of AI/LLM applications. It +supports versioned datasets, side-by-side experiments, deterministic and +LLM-judge evaluators, and repeated runs; its client API exposes a `repetitions` +argument and stores experiment/evaluation results for comparison +([official overview](https://arize.com/docs/phoenix/), +[dataset concepts](https://arize.com/docs/phoenix/learn/datasets-and-experiments/datasets-concepts), +[experiment API](https://arize-phoenix.readthedocs.io/projects/client/api/experiments.html)). + +Why the reviewer may have it in mind: this is the clearest existing open-source +answer to "systematically run a stochastic agent repeatedly against versioned +data and compare scores." ASAREE needs to demonstrate more than the generic +ability to store datasets, traces, repetitions, and scores. Its plausible +distinction is the first-class *factorial design over fields inside a visual +multi-agent protocol*, including factor binding, generated cells/replicates, +immutable executable protocol revisions, and biomedical data-workspace lineage. + +### MLflow GenAI evaluation + +MLflow evaluates tool-using agents from datasets or traces, records the result +as an experiment run, and supplies agent-specific scorers such as tool-call +correctness and efficiency. It also supports deterministic code scorers and LLM +judges; classic MLflow separately evaluates classification and regression models +([agent evaluation](https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/agents/), +[scorers](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/), +[classic model evaluation](https://mlflow.org/docs/latest/ml/evaluation)). + +Why it matters: ASAREE's use case produces ordinary supervised-ML outcomes as +well as agent runtime outcomes. A convincing comparison should show what ASAREE +adds beyond an MLflow-tracked loop over model/agent configurations. + +### LangSmith, Braintrust, W&B Weave, and Promptfoo + +- LangSmith supports offline benchmarks over datasets, comparisons among + application versions, code and model-judge evaluators, pairwise/summary + evaluators, and agent examples that verify expected tool calls + ([official evaluation docs](https://docs.langchain.com/langsmith/evaluation-types)). +- Braintrust defines an evaluation as data + task + scores and stores offline + evaluations as experiments. Its datasets are versioned and can be populated + from production, staging, evaluations, or manual examples + ([experiments](https://www.braintrust.dev/docs/guides/experiments), + [datasets](https://www.braintrust.dev/docs/guides/datasets)). +- W&B Weave logs datasets, model outputs, per-example scores and summary + metrics, then compares multiple evaluations in its UI + ([official EvaluationLogger guide](https://weave-docs.wandb.ai/guides/evaluation/evaluation_logger)). +- Promptfoo is an open-source CLI/library that crosses prompts/models with test + cases and evaluates them through deterministic assertions, custom code, or + model-graded assertions; it explicitly covers agent quality and trajectory + goal success + ([getting started](https://www.promptfoo.dev/docs/getting-started/), + [assertions and metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/)). + +These are direct comparators for the evaluation/result-management layer, but +less direct for ASAREE's visual protocol construction and factorial-design +semantics. They are still likely to be named by an agent-evaluation reviewer. + +### Inspect AI + +The UK AI Security Institute's open-source Inspect framework models evaluations +as datasets, solvers/agents, tools, sandboxes, and scorers; it supports +multi-agent primitives, arbitrary external agents, and several isolated +execution backends ([official documentation](https://inspect.aisi.org.uk/)). + +Inspect is a direct comparator if ASAREE is framed as an *agent evaluation +harness*. It is more benchmark/code oriented than ASAREE's GUI and biomedical +workflow, but its mature sandboxing and task/scorer abstraction set a baseline +for claims about reproducibility and safe execution. + +## 2. Visual agent workflow builders — closest interface comparators + +### AutoGen Studio + +AutoGen Studio offers a drag-and-drop/JSON interface for teams, agents, tools, +models, and termination conditions, plus a playground for running, inspecting, +and debugging multi-agent sessions. Its paper explicitly describes it as a +no-code tool for prototyping, debugging, and evaluating multi-agent workflows +([official docs](https://microsoft.github.io/autogen/0.7.1/user-guide/autogenstudio-user-guide/index.html), +[Microsoft Research publication](https://www.microsoft.com/en-us/research/publication/autogen-studio-a-no-code-developer-tool-for-building-and-debugging-multi-agent-systems/)). + +This is probably the first tool a reviewer will cite against the protocol +canvas. ASAREE should not claim novelty merely for drag-and-drop multi-agent +composition; the defensible comparison is whether a visual workflow can be +turned into a controlled factorial experiment with replicated, statistically +comparable outcomes. + +### Flowise and Langflow + +Flowise is an open-source visual platform for single- and multi-agent workflows, +with models, branching/loops, MCP, traces, datasets, evaluators, and evaluation +runs. Its packaged evaluation feature is documented as Cloud/Enterprise +functionality ([overview](https://docs.flowiseai.com/), +[evaluations](https://docs.flowiseai.com/using-flowise/evaluations)). Langflow's +visual editor connects prompts, models, data, agents, MCP servers, and tools; +flows can be run in a playground, exported as JSON, served through APIs, or +exposed as MCP tools +([visual editor](https://docs.langflow.org/concepts-overview), +[agents](https://docs.langflow.org/components-agents)). + +These are direct comparators for practical visual authoring and MCP integration. +Neither cited documentation foregrounds designed experiments over internal node +fields; that potential gap should be demonstrated with a feature matrix and a +worked head-to-head task rather than asserted. + +## 3. Biomedical/data-science agents — task-level comparators + +### BioMedAgent, Agentomics, Biomni, AIDE, MLAgentBench, and STELLA + +BioMedAgent is a particularly important task-level comparator. It is a +self-evolving multi-agent framework that accepts natural-language biomedical +analysis tasks, learns to use and chain bioinformatics tools, and covers +cross-omics analysis, ML modeling, and pathology image segmentation. Its +authors report a 77% success rate on the 327-task BioMed-AQA benchmark and +external evaluation on BixBench +([Nature Biomedical Engineering article](https://doi.org/10.1038/s41551-026-01634-6), +[official repository](https://github.com/BOBQWERA/BioMedAgent)). It postdates +ASAREE v0.2.0, but is now one of the clearest practical biomedical-agent +comparators and should be acknowledged in a current revision. + +Agentomics is an end-to-end biomedical ML agent system with validation +checkpoints, containerized execution, multiple model providers, and support for +biomedical foundation models. Its Bioinformatics paper benchmarks it across 20 +datasets against human solutions and four agent systems: AIDE, MLAgentBench, +STELLA, and Biomni +([Bioinformatics article](https://academic.oup.com/bioinformatics/article/42/Supplement_1/btag250/8726289)). +The paper characterizes AIDE and MLAgentBench as generalist ML coding agents, +STELLA as a biomedical multi-agent system with manager/developer/critic/tool +agents, and Biomni as a general-purpose biomedical agent with an expert-vetted +tool environment. + +These systems do not have ASAREE's same purpose: they seek a strong ML solution, +whereas ASAREE measures the effects of agent/protocol choices. Nevertheless, +they are important practical baselines for the myocardial/spinal-style use +case. If ASAREE claims usefulness for producing biomedical classifiers, a +reviewer can reasonably ask how its output quality, success rate, wall time, and +cost compare with at least one autonomous ML agent and a non-agent baseline. + +### BRAD and Coala + +BRAD is an open agentic bioinformatics system that connects LLMs to literature, +databases, custom software, and user data, records detailed interaction logs, +and demonstrates an automated biomarker/enrichment workflow +([Bioinformatics article](https://academic.oup.com/bioinformatics/article/41/5/btaf159/8125018)). +Coala converts CWL-described command-line tools into MCP tools and executes them +in containers, targeting reproducible local bioinformatics analysis +([Bioinformatics article](https://academic.oup.com/bioinformatics/article/42/9/btag641/8771239)). + +They are adjacent rather than direct: BRAD overlaps in agentic biomedical +analysis and provenance, while Coala overlaps in MCP tool integration and +reproducible execution. Both weaken broad claims that ASAREE uniquely makes +bioinformatics tools accessible to LLM agents. + +## 4. Conventional analytics, workflow, and AutoML systems — necessary baselines + +- Galaxy is an open web platform explicitly built for accessible, + reproducible, transparent computational biomedical research; it records the + information needed to repeat complete analyses + ([Galaxy project](https://galaxyproject.org/galaxy-project/)). +- KNIME is an open-source visual workflow system for data access, + transformation, analysis, modeling, and visualization + ([official documentation](https://docs.knime.com/ap/latest/)). +- Orange is open-source visual programming for machine learning and data + visualization, with canvas-connected widgets and no-code workflows + ([official site](https://orangedatamining.com/)). +- AutoGluon trains and ranks model families from tabular data and exposes a + model leaderboard; its paper describes highly accurate tabular AutoML from a + single line of Python + ([official tutorial](https://auto.gluon.ai/stable/tutorials/tabular/tabular-quick-start.html), + [paper](https://arxiv.org/abs/2003.06505)). The Agentomics paper also names + auto-sklearn and H2O alongside AutoGluon as established AutoML frameworks + ([Bioinformatics article](https://academic.oup.com/bioinformatics/article/42/Supplement_1/btag250/8726289)). + +These are not agent-experiment platforms. They matter because "analytical +sandbox" and the paper's visual tabular-ML use case overlap with capabilities +biomedical users already recognize. The practical advantage should therefore +be stated narrowly: ASAREE is for causal/comparative study of stochastic agent +protocol configurations, not simply visual analytics, workflow reproducibility, +or automated classifier search. + +## Likely reviewer shortlist + +If space permits only a compact comparison, prioritize: + +1. **Phoenix** — strongest open-source experiment/evaluation overlap, including + versioned datasets and repetitions. +2. **AutoGen Studio** — strongest visual multi-agent-workflow overlap. +3. **MLflow** — strongest bridge between agent evaluation and conventional ML + experiment tracking/evaluation. +4. **BioMedAgent, Agentomics, and Biomni** — strongest task-level overlap for + autonomous biomedical analysis/ML; AIDE and MLAgentBench are useful + generalist agent baselines. +5. **Galaxy or KNIME** — the established biomedical/visual analytical workflow + baseline. + +Flowise/Langflow, LangSmith, Braintrust, Weave, Promptfoo, Inspect, BRAD, Coala, +Orange, and AutoGluon belong in a broader related-work table or supplement. + +## What would actually answer the criticism + +A feature inventory alone will probably not satisfy "advantage ... has not been +demonstrated." A focused revision should: + +1. Add a capability table whose rows distinguish visual multi-agent authoring, + arbitrary internal-field factors, full-factorial cell generation, + repetitions, statistical comparison, immutable executable revisions, + versioned biomedical datasets, deterministic + model-judge metrics, MCP, + self-hosting, and run-level cost/time capture. +2. Recreate one ASAREE experiment in the nearest feasible baseline—preferably + Phoenix or MLflow around an equivalent agent workflow—and report setup/code + burden, completeness of provenance, and whether the factorial comparison can + be expressed natively or only through bespoke orchestration. +3. Add a non-agent practical baseline such as AutoGluon for predictive quality, + time, and cost. This prevents evidence that one agent configuration beats + another from being mistaken for evidence that the sandbox improves the + biomedical analysis. +4. Frame the advantage as **designed experimentation on agent protocols**, not + as visual workflow construction, generic LLM evaluation, AutoML, or a new + bioinformatics agent. Existing tools already substantiate all four broader + categories. diff --git a/publications/bioinformatics/myocardial-anthropic-latest.json b/publications/bioinformatics/myocardial-anthropic-latest.json index a9150cf..c640065 100644 --- a/publications/bioinformatics/myocardial-anthropic-latest.json +++ b/publications/bioinformatics/myocardial-anthropic-latest.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_anthropic", + "type": "model_anthropic", "position": { "x": 1140, "y": 680 @@ -703,57 +703,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -855,8 +855,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/publications/bioinformatics/myocardial-anthropic-v0.2.0.json b/publications/bioinformatics/myocardial-anthropic-v0.2.0.json index 5c0d312..dd8d175 100644 --- a/publications/bioinformatics/myocardial-anthropic-v0.2.0.json +++ b/publications/bioinformatics/myocardial-anthropic-v0.2.0.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_anthropic", + "type": "model_anthropic", "position": { "x": 1140, "y": 680 @@ -766,57 +766,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -939,8 +939,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/publications/bioinformatics/myocardial-azure-foundry-latest.json b/publications/bioinformatics/myocardial-azure-foundry-latest.json index ebc2e63..4775402 100644 --- a/publications/bioinformatics/myocardial-azure-foundry-latest.json +++ b/publications/bioinformatics/myocardial-azure-foundry-latest.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_azure_foundry", + "type": "model_azure_foundry", "position": { "x": 1140, "y": 680 @@ -703,57 +703,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -855,8 +855,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/publications/bioinformatics/myocardial-azure-foundry-v0.2.0.json b/publications/bioinformatics/myocardial-azure-foundry-v0.2.0.json index 9de5d46..419112d 100644 --- a/publications/bioinformatics/myocardial-azure-foundry-v0.2.0.json +++ b/publications/bioinformatics/myocardial-azure-foundry-v0.2.0.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_azure_foundry", + "type": "model_azure_foundry", "position": { "x": 1140, "y": 680 @@ -766,57 +766,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -939,8 +939,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/publications/bioinformatics/myocardial-openai-latest.json b/publications/bioinformatics/myocardial-openai-latest.json index 7e272e9..c29e6c3 100644 --- a/publications/bioinformatics/myocardial-openai-latest.json +++ b/publications/bioinformatics/myocardial-openai-latest.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_openai", + "type": "model_openai", "position": { "x": 1140, "y": 680 @@ -703,57 +703,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -855,8 +855,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/publications/bioinformatics/myocardial-openai-v0.2.0.json b/publications/bioinformatics/myocardial-openai-v0.2.0.json index dd4b115..3b229c0 100644 --- a/publications/bioinformatics/myocardial-openai-v0.2.0.json +++ b/publications/bioinformatics/myocardial-openai-v0.2.0.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_openai", + "type": "model_openai", "position": { "x": 1140, "y": 680 @@ -766,57 +766,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -939,8 +939,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/pyproject.toml b/pyproject.toml index 6ef3686..d4839d0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -94,7 +94,7 @@ dev = [ ] [tool.uv.sources] -motoro = { git = "https://github.com/EpistasisLab/motoro.git", tag = "v0.6.1" } +motoro = { git = "https://github.com/EpistasisLab/motoro.git", tag = "v0.8.0" } asaree-workspace-core = { path = "workspace-core", editable = true } # Every mcp-servers/ package is declared here, including asaree-sklearn-core # (which the six servers each depend on): uv resolves `tool.uv.sources` from diff --git a/spinal-use-case.json b/spinal-use-case.json index c886f6a..9991c43 100644 --- a/spinal-use-case.json +++ b/spinal-use-case.json @@ -5,7 +5,7 @@ "nodes": [ { "id": "llm-shared", - "type": "llm_azure_foundry", + "type": "model_azure_foundry", "position": { "x": 1140, "y": 680 @@ -766,57 +766,57 @@ "id": "e-llm-dc", "source": "llm-shared", "target": "agent-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fte", "source": "llm-shared", "target": "agent-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-fs", "source": "llm-shared", "target": "agent-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-mlm", "source": "llm-shared", "target": "agent-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatedc", "source": "llm-shared", "target": "gate-dc", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefte", "source": "llm-shared", "target": "gate-fte", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatefs", "source": "llm-shared", "target": "gate-fs", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-llm-gatemlm", "source": "llm-shared", "target": "gate-mlm", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-dc", @@ -939,8 +939,8 @@ "id": "e-llm-score", "source": "llm-shared", "target": "agent-score", - "sourceHandle": "ai", - "targetHandle": "ai" + "sourceHandle": "model", + "targetHandle": "model" }, { "id": "e-pattern-score", diff --git a/src/asaree/api/datasets.py b/src/asaree/api/datasets.py index 454834c..52026c5 100644 --- a/src/asaree/api/datasets.py +++ b/src/asaree/api/datasets.py @@ -20,6 +20,7 @@ from asaree.models.dataset_workspace_event import WorkspaceEventType from asaree.services.dataset_workspace_events import list_events, record_event from asaree.services.datasets import ( + DatasetNameConflictError, DatasetValidationError, create_dataset, delete_dataset, @@ -41,8 +42,8 @@ async def _get_owned_dataset(db: DbSession, dataset_id: uuid.UUID, user: Current async def _get_owned_dataset_by_name(db: DbSession, name: str, user: CurrentUser) -> Any: - dataset = await get_dataset_by_name(db, name) - if dataset is None or dataset.owner_id != user.id: + dataset = await get_dataset_by_name(db, name, owner_id=user.id) + if dataset is None: raise HTTPException(status_code=404, detail="No such dataset") return dataset @@ -129,7 +130,7 @@ async def create_dataset_endpoint( ) -> DatasetResponse: """Stores the raw file, verbatim -- never splits it. See a split's own two endpoints below (`.../split/quick`, `.../split/manual`).""" - if await get_dataset_by_name(db, name) is not None: + if await get_dataset_by_name(db, name, owner_id=user.id) is not None: raise HTTPException(status_code=409, detail="A dataset with this name already exists") try: dataset = await create_dataset( @@ -141,6 +142,8 @@ async def create_dataset_endpoint( description=description, dictionary_json=dictionary_json, ) + except DatasetNameConflictError as exc: + raise HTTPException(status_code=409, detail="A dataset with this name already exists") from exc except DatasetValidationError as exc: raise HTTPException(status_code=422, detail=str(exc)) from exc return _dataset_response(dataset) diff --git a/src/asaree/api/experiments.py b/src/asaree/api/experiments.py index bad4794..dd9ce16 100644 --- a/src/asaree/api/experiments.py +++ b/src/asaree/api/experiments.py @@ -66,6 +66,7 @@ normalize_design_spec, validate_metric_values, ) +from asaree.services.protocol_graph_schema import normalize_protocol_graph from asaree.services.protocol_revisions import get_published_revision, is_draft_published, publish_protocol from asaree.services.protocol_runs import list_experiment_trials from asaree.services.protocols import ( @@ -461,6 +462,9 @@ async def import_experiment_definition_endpoint( ): raise HTTPException(status_code=422, detail="The imported published canvas must contain nodes and edges") + graph = normalize_protocol_graph(body.graph) + published_graph = normalize_protocol_graph(body.published_graph) if body.published_graph is not None else None + try: design_spec = normalize_design_spec(body.design_spec, validate_metrics=True) except ValueError as exc: @@ -495,22 +499,22 @@ async def import_experiment_definition_endpoint( owner_id=user.id, description=body.protocol_description, experiment_id=experiment.id, - graph=body.published_graph or body.graph, + graph=published_graph or graph, ) if body.measurement_plan is not None: await _require_valid_measurement_plan( db, document=experiment.measurement_plan, metrics=(experiment.design_spec or {}).get("metrics"), - graph=body.published_graph or body.graph, + graph=published_graph or graph, experiment_id=experiment.id, owner_id=experiment.owner_id, allow_preserved_bindings=False, ) - if body.published_graph is not None: + if published_graph is not None: await publish_protocol(db, protocol) - if body.graph != body.published_graph: - protocol.graph = body.graph + if graph != published_graph: + protocol.graph = graph await db.flush() except IntegrityError as exc: if "uq_research_experiments_owner_name" in str(exc.orig): @@ -849,7 +853,7 @@ async def list_design_revisions_endpoint( id=s.revision.id, revision=s.revision.revision, superseded_at=s.revision.superseded_at, - design_spec=s.revision.design_spec, + design_spec=normalize_design_spec(s.revision.design_spec), cell_count=s.cell_count, replicate_count=s.replicate_count, scored_replicate_count=s.scored_replicate_count, diff --git a/src/asaree/api/mcp_servers.py b/src/asaree/api/mcp_servers.py index a5784e6..e3b0a5c 100644 --- a/src/asaree/api/mcp_servers.py +++ b/src/asaree/api/mcp_servers.py @@ -24,6 +24,7 @@ from fastapi import APIRouter, HTTPException from motoro.mcp.registry import get_registry from motoro.services import mcp_service +from motoro.services.mcp_service import MCPServerNameConflictError from pydantic import BaseModel from asaree.deps import CurrentUser @@ -92,8 +93,11 @@ async def _capabilities_with_tool_annotations(config: Any, *, refresh: bool = Fa tools = capabilities.get("tools") if capabilities else None if not isinstance(tools, list): return capabilities - entry = get_registry().servers.get(config.name) + assert capabilities is not None + entry = get_registry().servers.get(config.id) client = entry.client if entry is not None else None + if client is None: + return capabilities session = getattr(client, "_session", None) if session is None: return capabilities @@ -108,7 +112,7 @@ async def _capabilities_with_tool_annotations(config: Any, *, refresh: bool = Fa if tool.annotations is not None } cached = (frozenset(tool.name for tool in discovered.tools), annotation_map) - client._asaree_tool_annotations = cached + client._asaree_tool_annotations = cached # type: ignore[attr-defined] except Exception: return capabilities annotation_map = cached[1] @@ -136,7 +140,7 @@ async def _to_response(config: Any, *, refresh_annotations: bool = False) -> Ser @router.post("", response_model=ServerResponse, status_code=201) async def register_server_endpoint(body: RegisterServerRequest, user: CurrentUser) -> ServerResponse: - if await mcp_service.get_server_by_name(body.name) is not None: + if await mcp_service.get_server_by_name(body.name, owner_id=user.id) is not None: raise HTTPException(status_code=409, detail="A server with this name already exists") try: config = await mcp_service.register_server( @@ -147,6 +151,8 @@ async def register_server_endpoint(body: RegisterServerRequest, user: CurrentUse headers=body.headers, owner_id=user.id, ) + except MCPServerNameConflictError as exc: + raise HTTPException(status_code=409, detail="A server with this name already exists") from exc except ValueError as exc: raise HTTPException(status_code=422, detail=str(exc)) from exc return await _to_response(config, refresh_annotations=True) @@ -180,6 +186,8 @@ async def update_server_endpoint(server_id: uuid.UUID, body: UpdateServerRequest url=body.url, headers=body.headers, ) + except MCPServerNameConflictError as exc: + raise HTTPException(status_code=409, detail="A server with this name already exists") from exc except ValueError as exc: raise HTTPException(status_code=422, detail=str(exc)) from exc assert config is not None # existence already checked above diff --git a/src/asaree/api/protocols.py b/src/asaree/api/protocols.py index 96068f3..d2eb723 100644 --- a/src/asaree/api/protocols.py +++ b/src/asaree/api/protocols.py @@ -35,6 +35,7 @@ validate_single_node_runnable, validate_stage_plan, ) +from asaree.services.protocol_graph_schema import normalize_protocol_graph from asaree.services.protocol_revisions import ( get_published_revision, get_revision, @@ -284,7 +285,7 @@ async def _protocol_response(db: DbSession, protocol: Any) -> ProtocolResponse: name=protocol.name, description=protocol.description, experiment_id=protocol.experiment_id, - graph=protocol.graph, + graph=normalize_protocol_graph(protocol.graph), published_revision_id=published.id if published else None, published_revision=published.revision if published else None, has_unpublished_changes=not is_draft_published(protocol, published), @@ -366,6 +367,9 @@ async def update_protocol_endpoint( async def publish_protocol_endpoint(protocol_id: uuid.UUID, user: CurrentUser, db: DbSession) -> ProtocolResponse: """Make the current autosaved canvas the immutable version future runs use.""" protocol = await _get_owned_protocol(db, protocol_id, user) + published = await get_published_revision(db, protocol) + if published is not None and is_draft_published(protocol, published): + return await _protocol_response(db, protocol) if protocol.experiment_id: experiment = await get_experiment(db, protocol.experiment_id) if experiment is not None and experiment.locked_at is not None: @@ -409,7 +413,13 @@ async def get_protocol_revision_endpoint( revision = await get_revision(db, revision_id) if revision is None or revision.protocol_id != protocol_id: raise HTTPException(status_code=404, detail="No such protocol revision") - return ProtocolRevisionResponse.model_validate(revision) + return ProtocolRevisionResponse( + id=revision.id, + protocol_id=revision.protocol_id, + revision=revision.revision, + graph=normalize_protocol_graph(revision.graph), + published_at=revision.published_at, + ) @router.delete("/{protocol_id}", status_code=204) @@ -532,7 +542,7 @@ async def preview_node_prompt_endpoint( protocol = await _get_owned_protocol(db, protocol_id, user) try: text = await preview_node_prompt( - body.graph if body.graph is not None else protocol.graph, + normalize_protocol_graph(body.graph) if body.graph is not None else protocol.graph, node_id, owner_id=user.id, experiment_id=protocol.experiment_id, diff --git a/src/asaree/api/skills.py b/src/asaree/api/skills.py index 20e6572..138f209 100644 --- a/src/asaree/api/skills.py +++ b/src/asaree/api/skills.py @@ -117,8 +117,8 @@ async def upload_skill_endpoint( return SkillResponse.model_validate(skill) -async def _folder_payload(files: list[UploadFile]) -> list[tuple[str, str]]: - """``(path relative to the skill folder, text)`` for one directory upload. +async def _folder_payload(files: list[UploadFile]) -> list[tuple[str, bytes]]: + """``(path relative to the skill folder, bytes)`` for one directory upload. The leading ``webkitRelativePath`` segment is the folder the user picked — ``code-simplification/SKILL.md`` — and core's ``parse_skill_bundle`` wants @@ -129,7 +129,7 @@ async def _folder_payload(files: list[UploadFile]) -> list[tuple[str, str]]: the user dragged loose files, and a skill assembled from an unknown directory layout is not the directory they have on disk. """ - payload: list[tuple[str, str]] = [] + payload: list[tuple[str, bytes]] = [] for upload in files: name = upload.filename or "" parts = [p for p in name.replace("\\", "/").split("/") if p] @@ -138,7 +138,7 @@ async def _folder_payload(files: list[UploadFile]) -> list[tuple[str, str]]: status_code=422, detail=f"{name or 'A file'} didn't come from a folder — pick the skill's own folder.", ) - payload.append(("/".join(parts[1:]), _decode(await upload.read()))) + payload.append(("/".join(parts[1:]), await upload.read())) return payload diff --git a/src/asaree/mcp_servers/script_server.py b/src/asaree/mcp_servers/script_server.py index 40b5ce1..adddb8e 100644 --- a/src/asaree/mcp_servers/script_server.py +++ b/src/asaree/mcp_servers/script_server.py @@ -22,6 +22,14 @@ arrangement as the Dataset connector and the workspace tools, so wiring a script is the only gesture needed to let the agent run it. +When a Dataset is also wired, the subprocess receives a short-lived runtime +manifest through :mod:`asaree.script_context`. That stable API resolves either +an unsplit raw file or a workspace's ``v0_raw`` training partition without +exposing the held-out test partition or making user code parse ``state.json``. +Legacy scripts that explicitly read ``state.json`` receive the same training +reference through an isolated, execution-only compatibility view; it is never +written at the real workspace root. + **This is isolation, not a sandbox.** The script runs as a subprocess of this server, with a deny-by-default environment (see ``_ENV_PASSTHROUGH``) and a timeout, but it runs as the same user with the same filesystem. That is the trust @@ -39,12 +47,15 @@ import os import subprocess import sys +import tempfile from pathlib import Path from typing import Any from mcp.server import FastMCP from mcp.server.fastmcp import Context +from asaree.services.dataset_workspaces import raw_training_data_locators + INSTRUCTIONS = """\ Run a Python script wired into this step and report what it printed. @@ -52,7 +63,8 @@ call run_wired_script(script=...) using the name or id listed in the run prompt. \ Scripts arrive as ambient run context, so there is nothing to paste or retype. \ They are plain Python -- no required entry point, no dataset needed. Read stdout \ -for the result.""" +for the result. A wired script can resolve an attached authorized training input \ +with `from asaree.script_context import training_input`.""" mcp = FastMCP("asaree-script", instructions=INSTRUCTIONS) @@ -65,6 +77,12 @@ _META_KEY_SCRIPT_PATH = "motoro.ambient.script_path" _META_KEY_SCRIPT_PATHS = "motoro.ambient.script_paths" _META_KEY_WORKSPACE_ID = "motoro.workspace_id" +_META_KEY_DATA_PATH = "motoro.ambient.data_path" +_META_KEY_TARGET_COLUMN = "motoro.ambient.target_column" +_META_KEY_DATASET_NAMES = "motoro.ambient.dataset_names" +_META_KEY_DATASET_MODE = "motoro.ambient.dataset_mode" + +_RUN_CONTEXT_ENV = "ASAREE_RUN_CONTEXT" # Truncation budgets, matching the sklearn servers': a tool result is read by a # model, so a script that prints in a loop must not cost more context than the @@ -174,6 +192,74 @@ def _working_dir(workspace_id: str, script: Path) -> Path: return script.parent +def _runtime_manifest(ctx: Context[Any, Any, Any] | None, workspace_id: str) -> dict[str, Any]: + """Build the model-inaccessible dataset contract for a wired subprocess. + + Workspace inputs always name ``v0_raw.train`` rather than HEAD: assessment + scripts must see the registered training partition even after later stages + have transformed HEAD. With no workspace, the ambient path is the raw + unsplit registration resolved by protocol execution. Neither route ever + includes the held-out test path. + """ + raw_names = _ambient_value(ctx, _META_KEY_DATASET_NAMES) + names = [str(name) for name in raw_names if isinstance(name, str)] if isinstance(raw_names, list) else [] + dataset_mode = _ambient(ctx, _META_KEY_DATASET_MODE) + # An explicitly wired unsplit dataset must not inherit a durable workspace + # left by an older protocol revision for the same experiment/cell. + locators = raw_training_data_locators(workspace_id) if workspace_id and dataset_mode != "raw_unsplit" else {} + training_inputs: list[dict[str, Any]] = [] + for slot, locator in locators.items(): + recorded_name = str(locator.get("name") or "") + if names and recorded_name not in names and not (len(locators) == 1 and len(names) == 1): + continue + name = names[0] if len(locators) == 1 and len(names) == 1 else recorded_name + training_inputs.append( + { + "name": name, + "path": str(locator.get("data_path") or ""), + "target_column": str(locator.get("target_column") or ""), + "mode": "workspace", + "slot": slot, + "workspace_version": "v0_raw", + } + ) + + if not training_inputs and not locators: + data_path = _ambient(ctx, _META_KEY_DATA_PATH) + if data_path: + training_inputs.append( + { + "name": names[0] if len(names) == 1 else "", + "path": data_path, + "target_column": _ambient(ctx, _META_KEY_TARGET_COLUMN), + "mode": "raw_unsplit", + "slot": None, + "workspace_version": None, + } + ) + return {"schema_version": 1, "training_inputs": training_inputs} + + +def _legacy_unsplit_state(manifest: dict[str, Any]) -> dict[str, Any] | None: + """A read-only workspace-shaped view for scripts written before the API. + + This is never placed at a real workspace root. It exists only in an + isolated execution directory while a single unsplit input's script runs, + so workspace tools cannot mistake it for a seeded train/test lineage. + """ + inputs = manifest.get("training_inputs") + if not isinstance(inputs, list) or len(inputs) != 1: + return None + item = inputs[0] + if not isinstance(item, dict) or item.get("mode") != "raw_unsplit" or not item.get("path"): + return None + return { + "target_column": str(item.get("target_column") or ""), + "head": "v0_raw", + "versions": [{"id": "v0_raw", "train": str(item["path"])}], + } + + @mcp.tool() def run_wired_script( code: str = "", @@ -274,29 +360,61 @@ def run_wired_script( # printed -- the whole value of a partial result is that it survives. env["PYTHONUNBUFFERED"] = "1" result: dict[str, Any] = {"code_sha256": code_sha256, "script": script_file.name} + cwd = _working_dir(workspace_id, script_file) + manifest = _runtime_manifest(ctx, workspace_id) + # Preserve cwd for ordinary/helper-based scripts. The isolated legacy + # view is only needed when the authored source explicitly expects the old + # state-file contract. + legacy_state = _legacy_unsplit_state(manifest) if "state.json" in source else None + compatibility_dir: Path | None = None + if legacy_state is not None: + try: + # Never put this view at the workspace root: it has no test + # partition and must not make workspace_status report a real + # workspace. Keep artifacts created by the script in this unique + # run directory after the compatibility files are removed. + compatibility_dir = Path(tempfile.mkdtemp(prefix=f".{script_file.stem}-unsplit-", dir=script_file.parent)) + cwd = compatibility_dir + (cwd / "state.json").write_text(json.dumps(legacy_state), encoding="utf-8") + except OSError as e: + return json.dumps({**result, "error": f"could not prepare the unsplit dataset view: {e}"}) try: - completed = subprocess.run( # noqa: S603 -- user-authored script, by design; see module docstring - [sys.executable, str(script_file)], - cwd=str(_working_dir(workspace_id, script_file)), - env=env, - capture_output=True, - text=True, - errors="replace", - timeout=timeout, - ) - except subprocess.TimeoutExpired as e: - logger.warning("wired_script_timeout", extra={"script": str(script_file), "timeout": timeout}) - return json.dumps( - { - **result, - "timed_out": True, - "error": f"the script was killed after {timeout}s.", - "stdout": _clip(_as_text(e.stdout), _STDOUT_CHARS), - "stderr": _clip(_as_text(e.stderr), _STDERR_CHARS, tail=True), - } - ) + with tempfile.TemporaryDirectory(prefix=".asaree-script-context-", dir=cwd) as context_dir: + context_path = Path(context_dir) / "context.json" + context_path.write_text(json.dumps(manifest), encoding="utf-8") + env[_RUN_CONTEXT_ENV] = str(context_path) + try: + completed = subprocess.run( # noqa: S603 -- user-authored script, by design; see module docstring + [sys.executable, str(script_file)], + cwd=str(cwd), + env=env, + capture_output=True, + text=True, + errors="replace", + timeout=timeout, + ) + except subprocess.TimeoutExpired as e: + logger.warning("wired_script_timeout", extra={"script": str(script_file), "timeout": timeout}) + return json.dumps( + { + **result, + "timed_out": True, + "error": f"the script was killed after {timeout}s.", + "stdout": _clip(_as_text(e.stdout), _STDOUT_CHARS), + "stderr": _clip(_as_text(e.stderr), _STDERR_CHARS, tail=True), + } + ) + except OSError as e: + return json.dumps({**result, "error": f"could not start the script: {e}"}) except OSError as e: - return json.dumps({**result, "error": f"could not start the script: {e}"}) + return json.dumps({**result, "error": f"could not prepare the script runtime context: {e}"}) + finally: + if compatibility_dir is not None: + try: + (compatibility_dir / "state.json").unlink(missing_ok=True) + compatibility_dir.rmdir() # succeeds only when the script left no artifacts + except OSError: + pass result["exit_code"] = completed.returncode result["stdout"] = _clip(completed.stdout, _STDOUT_CHARS) diff --git a/src/asaree/mcp_servers/workspace_server.py b/src/asaree/mcp_servers/workspace_server.py index f86a12b..0c5a91b 100644 --- a/src/asaree/mcp_servers/workspace_server.py +++ b/src/asaree/mcp_servers/workspace_server.py @@ -369,7 +369,9 @@ async def open_workspace( name: Registered dataset name (must be a pre-split train/test registration, and owned by the user who started this run). Optional — resolved from _meta when the run has exactly one dataset wired; with several, this - picks between them and the error lists the candidates. + picks between them and the error lists the candidates. Inside an + ASAREE run an explicit name must be one of those wired candidates; + outside a run any owned registration may be named. target_column: Override target column; defaults to the registry's. stage: For a stage that hands off through a scratch directory (the default for every stage — see the stage-plan flags at the top of this @@ -415,6 +417,14 @@ async def open_workspace( ) } ) + run_scoped = bool(resolve_workspace_id_from_ctx("", ctx, required=False)) + if name.strip() and run_scoped and resolved_name not in candidates: + return json.dumps( + { + "error": f"Dataset {resolved_name!r} is not wired into this run.", + "wired_datasets": candidates, + } + ) try: owner_id = uuid.UUID(resolve_owner_id_from_ctx(ctx, required=True)) diff --git a/src/asaree/migrations/versions/a6c4e2f8190b_rename_model_connector_schema.py b/src/asaree/migrations/versions/a6c4e2f8190b_rename_model_connector_schema.py new file mode 100644 index 0000000..7d7a74d --- /dev/null +++ b/src/asaree/migrations/versions/a6c4e2f8190b_rename_model_connector_schema.py @@ -0,0 +1,156 @@ +"""rename Model connector schema identifiers + +Revision ID: a6c4e2f8190b +Revises: 9e4a7b2c1d30 +Create Date: 2026-09-23 00:00:00.000000 +""" + +from __future__ import annotations + +from collections.abc import Sequence + +from alembic import op + +revision: str = "a6c4e2f8190b" +down_revision: str | None = "9e4a7b2c1d30" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + + +def _rewrite_node_types(table: str, column: str, mapping: dict[str, str]) -> None: + cases = " ".join(f"WHEN n->>'type' = '{old}' THEN '{new}'" for old, new in mapping.items()) + old_values = ", ".join(f"'{value}'" for value in mapping) + op.execute( + f""" + UPDATE {table} AS target_row + SET {column} = jsonb_set( + target_row.{column}, + '{{nodes}}', + ( + SELECT jsonb_agg( + CASE + WHEN n->>'type' IN ({old_values}) + THEN jsonb_set(n, '{{type}}', to_jsonb(CASE {cases} ELSE n->>'type' END)) + ELSE n + END + ORDER BY ord + ) + FROM jsonb_array_elements(target_row.{column}->'nodes') WITH ORDINALITY AS items(n, ord) + ) + ) + WHERE jsonb_typeof(target_row.{column}->'nodes') = 'array' + AND jsonb_array_length(target_row.{column}->'nodes') > 0 + AND EXISTS ( + SELECT 1 + FROM jsonb_array_elements(target_row.{column}->'nodes') AS n + WHERE n->>'type' IN ({old_values}) + ) + """ + ) + + +def _rewrite_handles(table: str, column: str, old_values: tuple[str, ...], new_value: str) -> None: + old_sql = ", ".join(f"'{value}'" for value in old_values) + for field in ("sourceHandle", "targetHandle"): + op.execute( + f""" + UPDATE {table} AS target_row + SET {column} = jsonb_set( + target_row.{column}, + '{{edges}}', + ( + SELECT jsonb_agg( + CASE + WHEN edge->>'{field}' IN ({old_sql}) + THEN jsonb_set(edge, '{{{field}}}', '"{new_value}"') + ELSE edge + END + ORDER BY ord + ) + FROM jsonb_array_elements(target_row.{column}->'edges') WITH ORDINALITY AS items(edge, ord) + ) + ) + WHERE jsonb_typeof(target_row.{column}->'edges') = 'array' + AND jsonb_array_length(target_row.{column}->'edges') > 0 + AND EXISTS ( + SELECT 1 + FROM jsonb_array_elements(target_row.{column}->'edges') AS edge + WHERE edge->>'{field}' IN ({old_sql}) + ) + """ + ) + + +def _rewrite_factor_level_type(table: str, column: str, old_value: str, new_value: str) -> None: + op.execute( + f""" + UPDATE {table} AS target_row + SET {column} = jsonb_set( + target_row.{column}, + '{{factors}}', + ( + SELECT jsonb_agg( + CASE + WHEN factor->>'level_type' = '{old_value}' + THEN jsonb_set(factor, '{{level_type}}', '"{new_value}"') + ELSE factor + END + ORDER BY ord + ) + FROM jsonb_array_elements(target_row.{column}->'factors') WITH ORDINALITY AS items(factor, ord) + ) + ) + WHERE jsonb_typeof(target_row.{column}->'factors') = 'array' + AND jsonb_array_length(target_row.{column}->'factors') > 0 + AND EXISTS ( + SELECT 1 + FROM jsonb_array_elements(target_row.{column}->'factors') AS factor + WHERE factor->>'level_type' = '{old_value}' + ) + """ + ) + + +def _rewrite_graphs(node_types: dict[str, str], handles: tuple[str, ...], new_handle: str) -> None: + for table in ("protocols", "protocol_revisions"): + _rewrite_node_types(table, "graph", node_types) + _rewrite_handles(table, "graph", handles, new_handle) + + +def _rewrite_design_specs(old_value: str, new_value: str) -> None: + for table, column in ( + ("research_experiments", "design_spec"), + ("research_experiments", "locked_design_spec"), + ("experiment_design_revisions", "design_spec"), + ): + _rewrite_factor_level_type(table, column, old_value, new_value) + + +def upgrade() -> None: + _rewrite_graphs( + { + "llm_anthropic": "model_anthropic", + "llm_openai": "model_openai", + "llm_azure_foundry": "model_azure_foundry", + "llm_openrouter": "model_openrouter", + "llm_local": "model_local", + }, + ("ai", "llm"), + "model", + ) + _rewrite_design_specs("llm_config", "model_config") + + +def downgrade() -> None: + _rewrite_graphs( + { + "model_anthropic": "llm_anthropic", + "model_openai": "llm_openai", + "model_azure_foundry": "llm_azure_foundry", + "model_openrouter": "llm_openrouter", + "model_local": "llm_local", + }, + ("model",), + "ai", + ) + _rewrite_design_specs("model_config", "llm_config") diff --git a/src/asaree/migrations/versions/b2c7e4d91a60_scope_dataset_names_per_owner.py b/src/asaree/migrations/versions/b2c7e4d91a60_scope_dataset_names_per_owner.py new file mode 100644 index 0000000..65c2bb6 --- /dev/null +++ b/src/asaree/migrations/versions/b2c7e4d91a60_scope_dataset_names_per_owner.py @@ -0,0 +1,59 @@ +"""scope registered dataset names per owner + +Revision ID: b2c7e4d91a60 +Revises: a6c4e2f8190b +Create Date: 2026-09-23 00:00:00.000000 +""" + +from __future__ import annotations + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op + +revision: str = "b2c7e4d91a60" +down_revision: str | None = "a6c4e2f8190b" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + + +def upgrade() -> None: + op.drop_index("ix_registered_datasets_name", table_name="registered_datasets", if_exists=True) + op.create_index( + "uq_registered_datasets_owner_name", + "registered_datasets", + ["owner_id", "name"], + unique=True, + if_not_exists=True, + ) + + +def downgrade() -> None: + op.execute( + sa.text( + """ + DO $$ + BEGIN + IF EXISTS ( + SELECT 1 + FROM registered_datasets + GROUP BY name + HAVING count(*) > 1 + ) THEN + RAISE EXCEPTION + 'cannot downgrade dataset name scoping: duplicate names exist across owners'; + END IF; + END + $$ + """ + ) + ) + op.drop_index("uq_registered_datasets_owner_name", table_name="registered_datasets", if_exists=True) + op.create_index( + "ix_registered_datasets_name", + "registered_datasets", + ["name"], + unique=True, + if_not_exists=True, + ) diff --git a/src/asaree/models/dataset.py b/src/asaree/models/dataset.py index ab61082..492d751 100644 --- a/src/asaree/models/dataset.py +++ b/src/asaree/models/dataset.py @@ -11,7 +11,7 @@ ``register_manual_split``), so ``train_path``/``test_path`` are nullable: a freshly-registered dataset has a raw file and no split yet, same as an experiment created before the ``dataset_id`` FK existed permanently has -``dataset_id: null`` (see CLAUDE.md's own Experiment data model section) — +``dataset_id: null`` (see AGENTS.md's own Experiment data model section) — not a bug to backfill, just a real, valid state. This split-off-registration design is deliberate, not an oversight: scientific splitting needs vary per experiment (stratified holdout, group-aware holdout, k-fold, time-based, @@ -31,7 +31,7 @@ import uuid -from sqlalchemy import Float, ForeignKey, Integer, String, Text +from sqlalchemy import Float, ForeignKey, Index, Integer, String, Text from sqlalchemy.dialects.postgresql import UUID from sqlalchemy.orm import Mapped, mapped_column @@ -40,14 +40,15 @@ class RegisteredDataset(Base, TimestampMixin): __tablename__ = "registered_datasets" + __table_args__ = (Index("uq_registered_datasets_owner_name", "owner_id", "name", unique=True),) id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=generate_uuid) - name: Mapped[str] = mapped_column(String(255), nullable=False, unique=True, index=True) + name: Mapped[str] = mapped_column(String(255), nullable=False) # The original uploaded file, verbatim -- never modified, never # re-derived. The one thing registration itself is responsible for. # Nullable purely for a dataset registered before this column existed # (this concept didn't exist yet, so there's nothing to backfill it - # from -- same "permanent, valid null" reasoning CLAUDE.md's own + # from -- same "permanent, valid null" reasoning AGENTS.md's own # Experiment data model section gives for a pre-migration # dataset_id) -- every dataset registered from here on always has one. raw_path: Mapped[str | None] = mapped_column(Text, nullable=True) diff --git a/src/asaree/models/experiment_dataset.py b/src/asaree/models/experiment_dataset.py index 2e9e851..b04ef8c 100644 --- a/src/asaree/models/experiment_dataset.py +++ b/src/asaree/models/experiment_dataset.py @@ -9,7 +9,7 @@ itself changed shape and the column was dropped (migration ``d5a3b90c71e4``, which backfills one row per existing non-null ``dataset_id``). -Deliberately NOT the same call as CLAUDE.md's "agents are not a stored +Deliberately NOT the same call as AGENTS.md's "agents are not a stored relationship". An experiment's datasets are answerable only from the canvas graph otherwise, and unlike agents (reusable per-user templates that an experiment merely borrows) a dataset is part of what the experiment IS -- diff --git a/src/asaree/models/protocol_run.py b/src/asaree/models/protocol_run.py index c3fbb1f..6771f7b 100644 --- a/src/asaree/models/protocol_run.py +++ b/src/asaree/models/protocol_run.py @@ -57,7 +57,7 @@ class ProtocolRun(Base, TimestampMixin): # "evaluation_completed_at": iso}. attempt_result: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True) # The agent-to-agent transcript, present only once a run's agents actually - # consult each other -- null for every single-agent and pipeline run. + # communicate -- including a pipeline Agent delegating to a Sub-Agent. # Shape: {"state": , "messages": [{"message_id", "sequence", # "from_agent_id", "to_agent_id", "parts": [...], "created_at"}, ...]} # Kept alongside node_runs and for the same reason: it is an append-only, diff --git a/src/asaree/script_context.py b/src/asaree/script_context.py new file mode 100644 index 0000000..8c05072 --- /dev/null +++ b/src/asaree/script_context.py @@ -0,0 +1,89 @@ +"""Stable access to datasets authorized for an ASAREE wired script. + +The script runner publishes a short-lived JSON manifest and points this module +at it with ``ASAREE_RUN_CONTEXT``. User scripts therefore do not need to know +the workspace ``state.json`` format or receive filesystem paths through an +agent/model argument. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +_CONTEXT_ENV = "ASAREE_RUN_CONTEXT" + + +class ScriptContextError(ValueError): + """The wired-script runtime context is absent, invalid, or ambiguous.""" + + +@dataclass(frozen=True) +class TrainingInput: + """One authorized training input attached to the running script.""" + + name: str + path: Path + target_column: str + mode: str + slot: str | None = None + workspace_version: str | None = None + + +def _manifest() -> dict[str, Any]: + context_path = os.environ.get(_CONTEXT_ENV, "") + if not context_path: + raise ScriptContextError("This process has no ASAREE wired-script runtime context.") + try: + payload = json.loads(Path(context_path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise ScriptContextError(f"Could not read the ASAREE runtime context: {exc}") from exc + if not isinstance(payload, dict) or payload.get("schema_version") != 1: + raise ScriptContextError("Unsupported ASAREE wired-script runtime context.") + return payload + + +def training_inputs() -> tuple[TrainingInput, ...]: + """Return every authorized training input attached to this script.""" + raw_inputs = _manifest().get("training_inputs", []) + if not isinstance(raw_inputs, list): + raise ScriptContextError("ASAREE runtime training_inputs must be a list.") + resolved: list[TrainingInput] = [] + for item in raw_inputs: + if not isinstance(item, dict): + raise ScriptContextError("ASAREE runtime training input must be an object.") + raw_path = item.get("path") + if not isinstance(raw_path, str) or not raw_path: + raise ScriptContextError("ASAREE runtime training input has no path.") + path = Path(raw_path).resolve() + if not path.is_file(): + raise ScriptContextError(f"Authorized training input does not exist: {path}") + resolved.append( + TrainingInput( + name=str(item.get("name") or ""), + path=path, + target_column=str(item.get("target_column") or ""), + mode=str(item.get("mode") or ""), + slot=str(item["slot"]) if item.get("slot") is not None else None, + workspace_version=( + str(item["workspace_version"]) if item.get("workspace_version") is not None else None + ), + ) + ) + return tuple(resolved) + + +def training_input(*, name: str | None = None, slot: str | None = None) -> TrainingInput: + """Resolve one attached training input, requiring a selector if ambiguous.""" + inputs = training_inputs() + matches = [item for item in inputs if (name is None or item.name == name) and (slot is None or item.slot == slot)] + if len(matches) == 1: + return matches[0] + if not matches: + requested = f" name={name!r}" if name is not None else f" slot={slot!r}" if slot is not None else "" + raise ScriptContextError(f"No authorized training input matches{requested}.") + choices = ", ".join(item.slot or item.name or "" for item in matches) + raise ScriptContextError(f"Several training inputs are attached ({choices}); select by name or slot.") diff --git a/src/asaree/services/agent_messenger.py b/src/asaree/services/agent_messenger.py index c2c43d2..315d767 100644 --- a/src/asaree/services/agent_messenger.py +++ b/src/asaree/services/agent_messenger.py @@ -4,11 +4,11 @@ method wide: the engine projects the peers a caller declared into callable function schemas and hands any resulting call straight back here. Everything that decides *whether* the call happens lives in this module -- authorization, -message identity and ordering, the transcript, the budget, and cancellation. +message identity and ordering, the transcript, recursion safety, and cancellation. The engine never learns what an ASAREE canvas is. **The reply is a call result, not an exception.** A peer that may not be -reached, a spent budget and a cancelled conversation all come back as an +reached, a recursion-depth refusal and a cancelled conversation all come back as an :class:`~motoro.engine.ports.AgentReply` with a state the calling model can read, so it absorbs the outcome and still writes a real answer. Only genuine infrastructure failure raises. @@ -35,6 +35,7 @@ from __future__ import annotations import asyncio +import json import logging import time import uuid @@ -63,16 +64,6 @@ logger = logging.getLogger(__name__) -#: Total peer turns allowed per protocol run, across the whole conversation -#: tree. Every one of these is a full agent run with its own Reason/Plan/Act -#: cycle and its own tokens, so this is a cost cap as much as a loop guard. -_MAX_PEER_EXECUTIONS = 8 - -#: The real backstop. Deliberately **not** extended by peer time the way an -#: individual agent's own deadline is (see :mod:`asaree.services.deadline`): -#: no agent is charged for delegating, but the total stays bounded. -_MAX_CONVERSATION_DURATION = timedelta(minutes=5) - #: How deep consultations may nest. Two is enough for "Planner asks Critic, #: Critic asks a clarifying question back" -- the shape this feature exists for #: -- without letting a chain of agents each delegate one level further. @@ -84,7 +75,7 @@ USER_PARTICIPANT = "user" #: How much of one earlier message a briefing reproduces. A peer's own analysis -#: can run to thousands of tokens, and eight of them would crowd out the +#: can run to thousands of tokens, and several of them would crowd out the #: question actually being asked. Truncation is marked so the reading model can #: tell a cut-off answer from a short one. _MAX_BRIEFING_CHARS_PER_MESSAGE = 1500 @@ -112,13 +103,13 @@ class AgentMessenger: The consultation path (:meth:`send`) is single-threaded by design: invariant 5 is that one agent executes at a time and a consulting agent - blocks on its peer's reply, so ``sequence``, the budget counters and the + blocks on its peer's reply, so ``sequence`` and the turn stack are only ever touched from a single logical call stack. :func:`execute_supervisor_architecture` is the deliberate exception -- it dispatches workers concurrently rather than having a model ask for them -- and it uses the *transcript* only, through :meth:`record`, which is locked. - It never calls :meth:`send`, so nothing about the budget or the turn stack + It never calls :meth:`send`, so nothing about the recursion guard or the turn stack is ever touched concurrently. """ @@ -145,8 +136,6 @@ def __init__( #: else. ``None`` means "whatever this cell already stages through". self._stage_plan = stage_plan self._entry_agent_id = entry_agent_id - self._started_at = time.monotonic() - self._executions = 0 self._sequence = 0 #: Canvas node ids of the agents whose turns are currently on the stack, #: innermost last. This -- not anything the engine hands back -- is who @@ -167,7 +156,7 @@ def __init__( self._display_names = { str(n.get("id")): str((n.get("data") or {}).get("label") or "").strip() or str(n.get("id")) for n in graph.get("nodes") or [] - if n.get("type") == "agent" + if n.get("type") in ("agent", "sub_agent") } self._state = "working" #: Set when a cap is what stopped the conversation, so the run can land @@ -298,9 +287,26 @@ def _briefing(self, *, from_agent_id: str, to_agent_id: str, exclude_message_id: consultation of a conversation reads exactly as it did before. """ entries: list[str] = [] + target = next((n for n in self._graph.get("nodes") or [] if str(n.get("id")) == to_agent_id), None) + pair_only = target is not None and target.get("type") == "sub_agent" + original_user_message_id = next( + (m["message_id"] for m in self._messages if m["from_agent_id"] == USER_PARTICIPANT), + None, + ) for message in self._messages: if message["message_id"] == exclude_message_id: continue + if pair_only: + sender = message["from_agent_id"] + recipient = message["to_agent_id"] + if not ( + {sender, recipient}.issubset({from_agent_id, to_agent_id}) + or ( + sender == USER_PARTICIPANT + and (recipient == from_agent_id or message["message_id"] == original_user_message_id) + ) + ): + continue body = _text_of(message["parts"]) if not body: continue @@ -374,7 +380,6 @@ async def send( # sequential, which prevents a race but not this. head_before = head_data_locator(self._workspace_id)[0] if self._workspace_id else "" - self._executions += 1 # The caller's own clock stops for exactly this span, failures # included: it waited either way, and charging it for a peer's # failure is the same unfairness as charging it for a peer's @@ -468,19 +473,6 @@ async def _refusal(self, from_agent_id: str, to_agent_id: str) -> str | None: f"Consultations are already nested {_MAX_CONSULT_DEPTH} deep, which is the limit. " "Answer with what you have rather than delegating further." ) - if self._executions >= _MAX_PEER_EXECUTIONS: - self.limit_reached = True - return ( - f"This conversation has used all {_MAX_PEER_EXECUTIONS} of its peer consultations. " - "Answer with what you already have." - ) - if time.monotonic() - self._started_at >= _MAX_CONVERSATION_DURATION.total_seconds(): - self.limit_reached = True - return ( - f"This conversation has run for its full {int(_MAX_CONVERSATION_DURATION.total_seconds())} seconds. " - "Answer with what you already have." - ) - async with get_session() as db: run = await get_protocol_run(db, self._protocol_run_id) if run is not None and run.cancel_requested_at is not None: @@ -527,13 +519,17 @@ async def _run_peer( # workspace seeded and its `data_path` bound before it can run a script, # exactly like any other node. ambient_meta, dataset = await _node_run_context( - self._graph, to_agent_id, self._workspace_id, self._owner_id, stage_plan=self._stage_plan + self._graph, + to_agent_id, + self._workspace_id, + self._owner_id, + protocol_run_id=self._protocol_run_id, + stage_plan=self._stage_plan, ) - # A consultation reply is prose the asking agent reads, never a typed - # value anything binds to, so whatever the peer's parser extracted (if - # it even ran) is discarded here rather than stored under the peer's - # node run: this turn isn't that node's own pipeline run. - output_text, error, run_id, _extraction = await _run_agent_node( + # A configured Output Parser still defines a consulted worker's reply + # contract. Return its compact payload to the parent and retain the + # extraction for metrics; fall back to prose when no payload exists. + output_text, error, run_id, extraction = await _run_agent_node( node, protocol_id=self._protocol_id, protocol_run_id=self._protocol_run_id, @@ -549,19 +545,31 @@ async def _run_peer( # The canvas shows a consulted peer as a node that ran, because it did. # A peer consulted twice keeps only its latest turn here; the full # sequence is the transcript's job, not node_runs'. + effective_output = output_text + if extraction and extraction.get("payload") is not None: + effective_output = json.dumps(extraction["payload"], separators=(",", ":"), default=str) + patch: dict[str, Any] = { + "status": "cancelled" if error == _AGENT_CANCELLED else ("failed" if error else "completed"), + "output_text": effective_output, + "error": None if error == _AGENT_CANCELLED else error, + "run_id": str(run_id) if run_id else None, + **(extraction or {}), + } + if error is None: + patch.update( + { + "last_successful_output_text": effective_output, + "last_successful_run_id": str(run_id) if run_id else None, + } + ) async with get_session() as db: await update_node_run( db, self._protocol_run_id, to_agent_id, - { - "status": "cancelled" if error == _AGENT_CANCELLED else ("failed" if error else "completed"), - "output_text": output_text, - "error": None if error == _AGENT_CANCELLED else error, - "run_id": str(run_id) if run_id else None, - }, + patch, ) - return output_text, error, str(run_id) if run_id else None + return effective_output, error, str(run_id) if run_id else None async def execute_conversation( @@ -620,7 +628,12 @@ async def execute_conversation( if ambient_meta is None: ambient_meta, entry_dataset = await _node_run_context( - graph, entry_agent_id, workspace_id, owner_id, stage_plan=stage_plan + graph, + entry_agent_id, + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan, ) unsplit_dataset = unsplit_dataset or entry_dataset.unsplit_name @@ -857,7 +870,13 @@ async def _turn( async with get_session() as db: await update_node_run(db, protocol_run_id, node_id, {"status": "running"}) ambient_meta, dataset = await _node_run_context( - graph, node_id, workspace_id, owner_id, slot_prefix=slot_prefix, stage_plan=stage_plan + graph, + node_id, + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + slot_prefix=slot_prefix, + stage_plan=stage_plan, ) prompt = _build_user_input( nodes[node_id], @@ -1051,6 +1070,7 @@ async def record_sequential_transcript( node_runs: dict[str, Any], entry_prompt: str, state: str, + messenger: AgentMessenger | None = None, ) -> None: """Render a finished sequential run as an A2A conversation document. @@ -1080,7 +1100,7 @@ async def record_sequential_transcript( through. *entry_prompt* is the head agent's own built ``user_input``, which stands in as what the user asked. """ - messenger = AgentMessenger( + messenger = messenger or AgentMessenger( protocol_id=protocol_id, protocol_run_id=protocol_run_id, owner_id=owner_id, @@ -1088,11 +1108,18 @@ async def record_sequential_transcript( entry_agent_id=chain[0], workspace_id=None, ) - messenger.append( - from_agent_id=USER_PARTICIPANT, - to_agent_id=chain[0], - parts=[{"kind": "text", "text": entry_prompt}], - ) + # A pipeline messenger may already contain this entry because the head + # agent delegated. Keep one user-to-head opening, then add the ordinary + # sequential handoffs around the nested delegation messages. + if not any( + message["from_agent_id"] == USER_PARTICIPANT and message["to_agent_id"] == chain[0] + for message in messenger._messages + ): + messenger.append( + from_agent_id=USER_PARTICIPANT, + to_agent_id=chain[0], + parts=[{"kind": "text", "text": entry_prompt}], + ) def _outcome(node_id: str) -> tuple[str, str]: run = node_runs.get(node_id) or {} diff --git a/src/asaree/services/csv_export.py b/src/asaree/services/csv_export.py index 0fbc846..07c1dd3 100644 --- a/src/asaree/services/csv_export.py +++ b/src/asaree/services/csv_export.py @@ -3,7 +3,7 @@ One row per replicate; one column per factor_values/metric_values key seen across all replicates -- unlike the Cells table UI's own pickMetricColumns, which caps displayed metric columns at 4 for on-screen readability (frontend/src/pages/ -CLAUDE.md), a CSV has no such density constraint, so every key is included. +AGENTS.md), a CSV has no such density constraint, so every key is included. """ from __future__ import annotations diff --git a/src/asaree/services/dataset_workspaces.py b/src/asaree/services/dataset_workspaces.py index d45dbb5..d716f6c 100644 --- a/src/asaree/services/dataset_workspaces.py +++ b/src/asaree/services/dataset_workspaces.py @@ -66,10 +66,11 @@ async def fetch_owned_registration(name: str, owner_id: uuid.UUID) -> dict[str, (``asaree/api/datasets.py``). """ async with get_session() as db: - dataset = await get_dataset_by_name(db, name) - if dataset is None or dataset.owner_id != owner_id: + dataset = await get_dataset_by_name(db, name, owner_id=owner_id) + if dataset is None: return None return { + "description": dataset.description, "target_column": dataset.target_column, "raw_path": dataset.raw_path, "train_path": dataset.train_path, @@ -292,3 +293,39 @@ def slot_data_locators(workspace_id: str) -> dict[str, dict[str, str]]: "target_column": str(state.get("target_column") or ""), } return locators + + +def raw_training_data_locators(workspace_id: str) -> dict[str, dict[str, str]]: + """Every slot's authorized ``v0_raw`` training input, keyed by slot. + + This is the script-facing counterpart to :func:`slot_data_locators`, which + deliberately names HEAD for modeling tools. A read-only assessment script + needs the registered training partition instead: later accepted stages may + have dropped, encoded, or selected columns. The held-out test path is + intentionally never returned. + + Total like the other locators: an absent or unreadable workspace is ``{}``. + """ + try: + ws = Workspace(workspace_id) + if not ws.exists(): + return {} + slots = ws.slots() + except WorkspaceError: + return {} + except (OSError, ValueError) as e: + logger.warning("workspace_raw_locator_failed", extra={"workspace_id": workspace_id, "error": str(e)}) + return {} + + locators: dict[str, dict[str, str]] = {} + for key, state in slots.items(): + raw = next((v for v in state.get("versions", []) if v.get("id") == "v0_raw"), None) + train = str((raw or {}).get("train") or "") + if not train: + continue + locators[key] = { + "name": str(state.get("name") or key.split(":", 1)[-1]), + "data_path": train, + "target_column": str(state.get("target_column") or ""), + } + return locators diff --git a/src/asaree/services/datasets.py b/src/asaree/services/datasets.py index 6693304..aa30231 100644 --- a/src/asaree/services/datasets.py +++ b/src/asaree/services/datasets.py @@ -23,6 +23,7 @@ import pandas as pd from sklearn.model_selection import GroupShuffleSplit, train_test_split from sqlalchemy import select +from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession from asaree.config import get_settings @@ -34,6 +35,10 @@ class DatasetValidationError(ValueError): """A dataset request that fails validation before anything is written.""" +class DatasetNameConflictError(ValueError): + """A dataset name is already in use in this owner's namespace.""" + + def _split( df: pd.DataFrame, *, test_size: float, seed: int, target_column: str | None, group_column: str | None ) -> tuple[pd.DataFrame, pd.DataFrame, str | None]: @@ -102,7 +107,11 @@ async def create_dataset( owner_id=owner_id, ) db.add(dataset) - await db.flush() + try: + await db.flush() + except IntegrityError as exc: + shutil.rmtree(dest, ignore_errors=True) + raise DatasetNameConflictError(f"Dataset name '{name}' is already in use") from exc await db.refresh(dataset) return dataset @@ -222,8 +231,17 @@ async def get_dataset(db: AsyncSession, dataset_id: uuid.UUID) -> RegisteredData return (await db.execute(select(RegisteredDataset).where(RegisteredDataset.id == dataset_id))).scalar_one_or_none() -async def get_dataset_by_name(db: AsyncSession, name: str) -> RegisteredDataset | None: - return (await db.execute(select(RegisteredDataset).where(RegisteredDataset.name == name))).scalar_one_or_none() +async def get_dataset_by_name( + db: AsyncSession, name: str, *, owner_id: uuid.UUID +) -> RegisteredDataset | None: + return ( + await db.execute( + select(RegisteredDataset).where( + RegisteredDataset.owner_id == owner_id, + RegisteredDataset.name == name, + ) + ) + ).scalar_one_or_none() async def list_datasets(db: AsyncSession, *, owner_id: uuid.UUID) -> Sequence[RegisteredDataset]: diff --git a/src/asaree/services/design_generation.py b/src/asaree/services/design_generation.py index 3a8e0dc..70f6f7d 100644 --- a/src/asaree/services/design_generation.py +++ b/src/asaree/services/design_generation.py @@ -76,7 +76,7 @@ def generate_design(factors: list[dict[str, Any]]) -> list[dict[str, Any]]: return [dict(zip(names, values, strict=True)) for values in itertools.product(*levels_lists)] -# Checked in order for a dict-valued level (e.g. a whole LLM/Tool/Dataset node +# Checked in order for a dict-valued level (e.g. a whole Model/Tool/Dataset node # config or a pattern-override payload bound as a single factor) -- whichever # of these identifying keys is present first names the slug, since one of them # is always the thing a human actually wants to see in a cell label diff --git a/src/asaree/services/llm_model_discovery.py b/src/asaree/services/llm_model_discovery.py index 6f67e37..cbee83f 100644 --- a/src/asaree/services/llm_model_discovery.py +++ b/src/asaree/services/llm_model_discovery.py @@ -200,7 +200,7 @@ async def _discover_anthropic(setting: UserLLMSetting) -> tuple[list[ModelInfo], Failure falls back to the curated catalog rather than surfacing an error: the catalog is stale, not wrong, and an empty dropdown is a worse answer than an incomplete one. The returned source is "static" in that case, so - callers gating on "api" (LlmNode's unrecognized-model warning) correctly + callers gating on "api" (ModelNode's unrecognized-model warning) correctly stay quiet about ids this list can't vouch for. """ api_key = decrypt_api_key(setting) @@ -302,7 +302,7 @@ async def _discover_local(setting: UserLLMSetting) -> tuple[list[ModelInfo], str implement -- but that route isn't a hard requirement of being "an OpenAI-compatible chat endpoint", so a server that doesn't expose it is a normal, expected outcome (surfaced via ``source="error"`` purely so the - inspector's note actually renders -- see LlmNodeInspector.tsx, which only + inspector's note actually renders -- see ModelNodeInspector.tsx, which only shows ``note`` for that source -- not because it's a real error). """ base = setting.api_base.rstrip("/") if setting.api_base else "" diff --git a/src/asaree/services/metrics.py b/src/asaree/services/metrics.py index 7505b54..b3e4f4c 100644 --- a/src/asaree/services/metrics.py +++ b/src/asaree/services/metrics.py @@ -328,6 +328,8 @@ def normalize_design_spec( for factor in factors: if not isinstance(factor, dict) or not isinstance(factor.get("name"), str): continue + if factor.get("level_type") == "llm_config": + factor["level_type"] = "model_config" levels = factor.get("levels") if not isinstance(levels, list): continue diff --git a/src/asaree/services/okf_bundles.py b/src/asaree/services/okf_bundles.py index 80b5fe9..c547799 100644 --- a/src/asaree/services/okf_bundles.py +++ b/src/asaree/services/okf_bundles.py @@ -426,7 +426,7 @@ async def register_bundle(*, owner_id: uuid.UUID, relative_path: str | None) -> """ path = validate_bundle_path(relative_path) name = server_name_for(owner_id, path) - existing = await mcp_service.get_server_by_name(name) + existing = await mcp_service.get_server_by_name(name, owner_id=owner_id) if existing is not None: return existing return await mcp_service.register_server( diff --git a/src/asaree/services/protocol_execution.py b/src/asaree/services/protocol_execution.py index 36a7c5d..f9126d7 100644 --- a/src/asaree/services/protocol_execution.py +++ b/src/asaree/services/protocol_execution.py @@ -91,6 +91,8 @@ from asaree.services.run_tools import gather_tools from asaree.services.runtime_metrics import finalize_attempt_measurement from asaree.services.system_mcp_servers import ( + DATASET_DICTIONARY_AGENT_TOOLS, + EDA_SERVER_NAME, SCIKIT_LEARN_SERVER_NAME, SCRIPT_AGENT_TOOLS, SCRIPT_SERVER_NAME, @@ -175,7 +177,7 @@ class ProtocolValidationError(Exception): # # 1. CAPABILITY -- what the agent can DO: the model, the execution pattern, the # tool allow-list, knowledge servers, skills. Route: resolve it into the -# agent's stored config (``_resolve_llm_config``, ``_resolve_tool_config``, +# agent's stored config (``_resolve_model_config``, ``_resolve_tool_config``, # ``_resolve_pattern_config``, ``_resolve_skill_config``, ...) and let # Motoro carry it on ``RunContext``. Never prompt text: a capability is # something the runtime arranges, not something the model is told about. @@ -210,7 +212,7 @@ class ProtocolValidationError(Exception): # ``run_wired_script``, granted by ``_resolve_script_tool_config``. Wiring a # script is what declares that the agent should run one. # -# The connector-typed slots on an agent/critic_gate node. ai/tool/memory are +# The connector-typed slots on an agent/critic_gate node. model/tool/memory are # a deliberately closed set; architectural_pattern and dataset are # ASAREE-specific -- architectural_pattern for ARES's pluggable # architectural patterns, dataset for the data an agent operates ON as @@ -222,13 +224,14 @@ class ProtocolValidationError(Exception): # data-flow) is any edge whose targetHandle is one of these -- everything # else. The type marker always lives on the target side of an edge. # -# "llm" and "resource" are in here purely as pre-rename spellings of "ai" and -# "dataset" (see _LEGACY_AI_HANDLES/_LEGACY_DATASET_HANDLES): an un-migrated +# "ai", "llm", and "resource" are pre-rename spellings of "model" and +# "dataset" (see _LEGACY_MODEL_HANDLES/_LEGACY_DATASET_HANDLES): an un-migrated # edge must still be recognised as a connector, or it would be misread as a # main pipeline edge and turn a perfectly good graph into a cycle/ordering # error. _CONNECTOR_HANDLES = frozenset( { + "model", "ai", "llm", "tool", @@ -239,6 +242,7 @@ class ProtocolValidationError(Exception): "skill", "knowledge", "output_parser", + "sub_agents", } ) @@ -249,9 +253,11 @@ class ProtocolValidationError(Exception): # a single generic node with a Provider/kind field -- config shape is identical # across LLM providers (provider is baked into the node type instead of a # user-editable field), but genuinely differs per architectural pattern (see -# each pattern's own NodeConfig on the frontend), so the LLM family shares +# each pattern's own NodeConfig on the frontend), so the Model family shares # one inspector while each pattern gets its own. -_LLM_NODE_TYPES = frozenset({"llm_anthropic", "llm_openai", "llm_azure_foundry"}) +_MODEL_NODE_TYPES = frozenset( + {"model_anthropic", "model_openai", "model_azure_foundry", "model_openrouter", "model_local"} +) # Only two builtin execution patterns exist in Motoro today # (engine/patterns/builtin/) -- PatternConfig already has unused slots for # safety_patterns/coordination_pattern/knowledge_patterns/quality_patterns/ @@ -350,13 +356,14 @@ class ProtocolValidationError(Exception): # Capped at one, like Memory and unlike Tool/Skill/Knowledge: two field specs # for one output is an ambiguity, not a richer declaration. _OUTPUT_PARSER_NODE_TYPES = frozenset({"output_parser"}) +_SUB_AGENT_NODE_TYPES = frozenset({"sub_agent"}) -# Every node type that's a pure config source -- never gets its own execution -# turn, never a pipeline "final output" (see sink_node_ids/run_protocol's -# main loop), and may only ever emit its own connector-typed edge (see the -# "outgoing wrong handle" check in topological_order below). +# Every node type skipped by the main pipeline walk and excluded as a final +# output. Most are pure config sources; Sub-Agent is the deliberate exception: +# it executes only as a nested delegated turn. All may emit only their own +# connector-typed edge (see topological_order's outgoing-handle check). _PURE_CONFIG_SOURCE_TYPES = ( - _LLM_NODE_TYPES + _MODEL_NODE_TYPES | _EXECUTION_PATTERN_NODE_TYPES | _MEMORY_NODE_TYPES | _MCP_TOOL_NODE_TYPES @@ -365,20 +372,21 @@ class ProtocolValidationError(Exception): | _SKILL_NODE_TYPES | _KNOWLEDGE_NODE_TYPES | _OUTPUT_PARSER_NODE_TYPES + | _SUB_AGENT_NODE_TYPES ) -# Which connector handle each pure-config-source node type may exclusively +# Which connector handle each pipeline-skipped node type may exclusively # emit into, and the human-facing label for that handle -- both keyed off # the same family grouping so a new provider/pattern node type only needs -# adding to _LLM_NODE_TYPES/_EXECUTION_PATTERN_NODE_TYPES above, not a +# adding to _MODEL_NODE_TYPES/_EXECUTION_PATTERN_NODE_TYPES above, not a # second lookup. _NODE_TYPE_TO_HANDLE: dict[str, str] = { - **{t: "ai" for t in _LLM_NODE_TYPES}, + **{t: "model" for t in _MODEL_NODE_TYPES}, **{t: "architectural_pattern" for t in _EXECUTION_PATTERN_NODE_TYPES}, **{t: "memory" for t in _MEMORY_NODE_TYPES}, # Script still shares the Tool connector rather than getting its own slot # -- one connector accepting a FAMILY of node types (see this dict's own - # docstring above _LLM_NODE_TYPES). Both are pure config sources an + # docstring above _MODEL_NODE_TYPES). Both are pure config sources an # agent's Tool "+" panel can add (AddNodePanel filters its catalog by # CONNECTOR_PANEL_INFO.tool's allowedTypes on the frontend); which one a # given wired node actually IS is recovered by checking the source node's @@ -394,13 +402,15 @@ class ProtocolValidationError(Exception): **{t: "skill" for t in _SKILL_NODE_TYPES}, **{t: "knowledge" for t in _KNOWLEDGE_NODE_TYPES}, **{t: "output_parser" for t in _OUTPUT_PARSER_NODE_TYPES}, + **{t: "sub_agents" for t in _SUB_AGENT_NODE_TYPES}, } # The user-facing name of each connector slot -- mirrors # CONNECTOR_SLOT_LABELS on the frontend, so a validation error always names # the connector by the caption printed next to it on the canvas. _HANDLE_LABELS: dict[str, str] = { - "ai": "AI", - "llm": "AI", # pre-rename spelling, same slot -- see _LEGACY_AI_HANDLES + "model": "Model", + "ai": "Model", # pre-rename spellings, same slot -- see _LEGACY_MODEL_HANDLES + "llm": "Model", "memory": "Memory", "architectural_pattern": "Architectural Pattern", "tool": "Tool", @@ -409,14 +419,15 @@ class ProtocolValidationError(Exception): "skill": "Skill", "knowledge": "Knowledge", "output_parser": "Output Parser", + "sub_agents": "Sub-Agents", } # Connector slots have been renamed twice since graphs started being saved, # and a stored graph is an opaque JSONB blob, so every spelling has to keep # resolving: # -# "llm" -> "ai" the AI connector (its caption was renamed first, the -# handle id after -- migration 3f1a7c9b2e04) +# "llm" -> "ai" -> "model" the Model connector (migrations +# 3f1a7c9b2e04 and the Model-schema migration) # "tool" -> "resource" for a Dataset source, when Dataset stopped sharing # the Tool slot (same migration) # "resource" -> "dataset" when that slot, whose only member is the Dataset @@ -430,7 +441,7 @@ class ProtocolValidationError(Exception): # old-spelling edges at whatever moment the new backend goes live, and an # SDK/notebook caller pinned to an older graph shape keeps working. Nothing # creates an old-spelling edge going forward -- isValidConnection won't. -_LEGACY_AI_HANDLES = frozenset({"ai", "llm"}) +_LEGACY_MODEL_HANDLES = frozenset({"model", "ai", "llm"}) _LEGACY_DATASET_HANDLES = frozenset({"dataset", "resource", "tool"}) # Keyed by the CURRENT slot id -- every spelling an edge into that slot may # legitimately still carry *on the handle alone*, i.e. every rename that was @@ -441,7 +452,7 @@ class ProtocolValidationError(Exception): # out by ALSO checking its source node's type, which is why the wider # _LEGACY_DATASET_HANDLES is applied at its own call sites instead. _LEGACY_HANDLES_BY_SLOT: dict[str, frozenset[str]] = { - "ai": _LEGACY_AI_HANDLES, + "model": _LEGACY_MODEL_HANDLES, "dataset": frozenset({"dataset", "resource"}), } @@ -573,7 +584,7 @@ def derive_stage_plan(graph: dict[str, Any]) -> Any: stage_ids: list[str] = [] for nid in ordered: node = nodes[nid] - if node.get("type") != "agent": + if node.get("type") not in ("agent", "sub_agent"): continue for edge in _edges_with_handle(graph, nid, "tool", direction="incoming"): source = nodes.get(str(edge.get("source"))) @@ -910,6 +921,7 @@ def resolve_conversation_entry_id(graph: dict[str, Any]) -> str: _NODE_TYPE_DISPLAY_NAMES: dict[str, str] = { "agent": "Agent", + "sub_agent": "Sub-Agent", "critic_gate": "Critic Gate", "mcp_tool": "MCP Tool", "mcp_scikit_learn": "Scikit-learn MCP", @@ -923,9 +935,11 @@ def resolve_conversation_entry_id(graph: dict[str, Any]) -> str: "output_parser": "Output Parser", "pattern_reason_act": "Reason + Act", "pattern_single_agent_baseline": "Single-Agent Baseline", - "llm_anthropic": "Anthropic", - "llm_openai": "OpenAI", - "llm_azure_foundry": "Azure AI Foundry", + "model_anthropic": "Anthropic", + "model_openai": "OpenAI", + "model_azure_foundry": "Azure AI Foundry", + "model_openrouter": "OpenRouter", + "model_local": "Local", } @@ -934,7 +948,7 @@ def _node_display_name(node: dict[str, Any]) -> str: user has set one (matching what they'd actually see in the inspector header/on the card), else the same placeholder text the frontend shows for an unnamed node of that type (EditableNodeTitle's own `placeholder` - prop, or the provider label for the three LLM node types). Never the + prop, or the provider label for the three Model node types). Never the bare internal node id -- that's graph bookkeeping (see newNodeId on the frontend), meaningless to a user reading a failed-validation message.""" data = node.get("data") @@ -981,7 +995,7 @@ def _kahn_order(graph: dict[str, Any]) -> tuple[dict[str, dict[str, Any]], list[ Split out of :func:`topological_order` for callers that only want to know what order the canvas draws -- :func:`derive_stage_plan` reads a half-built - graph while the user is still wiring it, and a missing AI connection there + graph while the user is still wiring it, and a missing Model connection there is a thing to report on the canvas, not a reason for stage derivation to raise. Returns the node map, the walk, and whether every node was reached (``False`` is the cycle signature); unreached nodes are appended in @@ -1033,7 +1047,7 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> for nid, node in nodes.items(): if node.get("type") != "critic_gate": continue - # Main-pipeline incoming edges only -- a gate's own LLM connector + # Main-pipeline incoming edges only -- a gate's own Model connector # edge is a separate concept (validated below) and must not count # towards "how many things feed this gate on the main pipeline." ups = _upstream_ids(graph, nid) @@ -1063,15 +1077,20 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> node_type = node.get("type") name = _node_display_name(node) - if node_type in ("agent", "critic_gate"): - llm_edges = _edges_with_handle(graph, nid, "ai", direction="incoming") - if len(llm_edges) != 1: + sub_agent_is_callable = ( + node_type == "sub_agent" + and _is_node_active(node) + and bool(_edges_with_handle(graph, nid, "sub_agents", direction="outgoing")) + ) + if node_type in ("agent", "critic_gate") or sub_agent_is_callable: + model_edges = _edges_with_handle(graph, nid, "model", direction="incoming") + if len(model_edges) != 1: raise ProtocolValidationError( - f"Node {name!r} must have exactly one AI connection (found {len(llm_edges)})." + f"Node {name!r} must have exactly one Model connection (found {len(model_edges)})." ) - llm_source = nodes.get(llm_edges[0]["source"]) - if llm_source is None or llm_source.get("type") not in _LLM_NODE_TYPES: - raise ProtocolValidationError(f"Node {name!r}'s AI connection must come from an AI node.") + model_source = nodes.get(model_edges[0]["source"]) + if model_source is None or model_source.get("type") not in _MODEL_NODE_TYPES: + raise ProtocolValidationError(f"Node {name!r}'s Model connection must come from a Model node.") tool_edges = _edges_with_handle(graph, nid, "tool", direction="incoming") memory_edges = _edges_with_handle(graph, nid, "memory", direction="incoming") @@ -1080,7 +1099,8 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> skill_edges = _edges_with_handle(graph, nid, "skill", direction="incoming") knowledge_edges = _edges_with_handle(graph, nid, "knowledge", direction="incoming") parser_edges = _edges_with_handle(graph, nid, "output_parser", direction="incoming") - if node_type == "agent": + sub_agent_edges = _edges_with_handle(graph, nid, "sub_agents", direction="incoming") + if node_type in ("agent", "sub_agent"): # The Tool connector accepts a family of source types -- an # mcp_tool node contributes a callable capability, while a # Script node contributes declarative config/context (see @@ -1183,6 +1203,15 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> f"Node {name!r} has both an Output Parser connection and its own stored output contract. " "Convert the stored one to a node, or remove it, so there is one output shape." ) + if node_type == "agent": + for edge in sub_agent_edges: + child = nodes.get(edge["source"]) + if child is None or child.get("type") != "sub_agent": + raise ProtocolValidationError( + f"Node {name!r}'s Sub-Agents connection must come from a Sub-Agent node." + ) + elif sub_agent_edges: + raise ProtocolValidationError(f"Sub-Agent node {name!r} cannot own other Sub-Agents.") elif ( tool_edges or memory_edges @@ -1191,12 +1220,26 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> or skill_edges or knowledge_edges or parser_edges + or sub_agent_edges ): raise ProtocolValidationError( f"Only Agent nodes can have a Tool, Memory, Architectural Pattern, Skill, Dataset, " f"Knowledge, or Output Parser connection (node {name!r})." ) + if node_type == "sub_agent": + parent_edges = _edges_with_handle(graph, nid, "sub_agents", direction="outgoing") + if len(parent_edges) > 1: + raise ProtocolValidationError( + f"Sub-Agent node {name!r} can have exactly one parent (found {len(parent_edges)})." + ) + for edge in parent_edges: + parent = nodes.get(str(edge.get("target"))) + if parent is None or parent.get("type") != "agent": + raise ProtocolValidationError( + f"Sub-Agent node {name!r}'s Parent connection must lead to an Agent node." + ) + if node_type in _NODE_TYPE_TO_HANDLE: expected_handle = _NODE_TYPE_TO_HANDLE[node_type] allowed_handles = ( @@ -1221,7 +1264,8 @@ def topological_order(graph: dict[str, Any], *, require_acyclic: bool = True) -> # "Dataset" either way. leading_label = ( _NODE_TYPE_DISPLAY_NAMES[node_type] - if node_type in _DATASET_NODE_TYPES | _SCRIPT_NODE_TYPES | _KNOWLEDGE_NODE_TYPES + if node_type + in _DATASET_NODE_TYPES | _SCRIPT_NODE_TYPES | _KNOWLEDGE_NODE_TYPES | _SUB_AGENT_NODE_TYPES else handle_label ) raise ProtocolValidationError( @@ -1251,11 +1295,9 @@ def sink_node_ids(graph: dict[str, Any]) -> list[str]: """Every node with no outgoing edges -- used both to validate a graph is runnable per-cell (exactly one sink required, see ``plan_cell_runs``) and by ``run_protocol`` itself to find the node whose output becomes a cell's - result. Excludes every pure-config-source node type (every LLM provider/ - architectural pattern node, plus ``memory`` and ``mcp_tool``) -- these are - never a pipeline's "final output," whether or not they're connected to - anything (an unwired one would otherwise falsely count as an extra - sink).""" + result. Excludes every pipeline-skipped node type: connector config sources + and Sub-Agents, whose output belongs to a nested delegated turn rather than + the main pipeline. An unwired one must not falsely count as an extra sink.""" nodes, downstream, _upstream = _adjacency(graph) return [ nid for nid, node in nodes.items() if not downstream[nid] and node.get("type") not in _PURE_CONFIG_SOURCE_TYPES @@ -1419,9 +1461,7 @@ def _compute_workspace_id( def _materialize_script(workspace_id: str | None, node_id: str, code: str) -> str: """Write a wired Script node's code next to the run's workspace; return its path. - ``""`` when there's nowhere to put it -- no workspace id (an unlinked - protocol run) or the write failed. The caller falls back to inlining the - code in the prompt, which is what this replaces. + ``""`` when no materialization id was supplied or the write failed. The file lives under the run's own workspace directory because that directory is already the shared surface between this process and the MCP @@ -1450,8 +1490,33 @@ def _materialize_script(workspace_id: str | None, node_id: str, code: str) -> st return str(path) +def _script_workspace_id(workspace_id: str | None, protocol_run_id: uuid.UUID | None, agent_node_id: str) -> str | None: + """Choose real workspace storage or an isolated standalone-run directory.""" + safe_agent = _UNSAFE_WORKSPACE_LABEL_CHAR.sub("_", agent_node_id) + return workspace_id or (f"_protocol_runs/{protocol_run_id}/{safe_agent}" if protocol_run_id else None) + + +def _cleanup_adhoc_scripts(ambient_meta: dict[str, Any] | None) -> None: + """Remove standalone-run script files after their consuming agent stops.""" + adhoc_root = (Path(WORKSPACE_ROOT).resolve() / "_protocol_runs").resolve() + for item in (ambient_meta or {}).get("script_paths") or []: + path = Path(str(item.get("path") or "")).resolve() + if adhoc_root not in path.parents: + continue + with contextlib.suppress(OSError): + path.unlink() + for directory in (path.parent, path.parent.parent, path.parent.parent.parent): + with contextlib.suppress(OSError): + directory.rmdir() + + def _ambient_meta_for( - graph: dict[str, Any], node_id: str, workspace_id: str | None = None, *, slots: tuple[str, ...] = () + graph: dict[str, Any], + node_id: str, + workspace_id: str | None = None, + *, + script_workspace_id: str | None = None, + slots: tuple[str, ...] = (), ) -> dict[str, Any]: """The node's Reference-route values, for Motoro's caller-ambient ``_meta``. @@ -1539,7 +1604,7 @@ def _ambient_meta_for( if not code: continue script_node_id = str(config.get("node_id") or f"script-{index}") - script_path = _materialize_script(workspace_id, script_node_id, str(code)) + script_path = _materialize_script(script_workspace_id or workspace_id, script_node_id, str(code)) if script_path: script_paths.append( { @@ -1654,6 +1719,18 @@ async def _resolve_node_dataset( "workspace_preseed_failed", extra={"node_id": node_id, "dataset": name, "error": "not found"} ) continue + # Enrich this run's in-memory graph with current catalog metadata. The + # published graph still owns identity/order; the registry owns mutable + # descriptive facts, so old nodes gain the same discovery context as + # newly-created ones without rewriting a revision. + for config in _resolve_dataset_configs(graph, node_id): + if str(config.get("dataset_name") or "") == name: + config.update( + description=reg.get("description"), + target_column=reg.get("target_column"), + split_state="split" if reg.get("train_path") and reg.get("test_path") else "unsplit", + dictionary_available=bool(reg.get("dictionary_json")), + ) if not (reg.get("train_path") and reg.get("test_path")): # Unsplit: no workspace to seed (``seed_cell_workspace`` says why), # so the raw file itself becomes the run's dataset and the agent @@ -1708,6 +1785,7 @@ async def _node_run_context( workspace_id: str | None, owner_id: uuid.UUID, *, + protocol_run_id: uuid.UUID | None = None, slot_prefix: str | None = None, stage_plan: Any = None, ) -> tuple[dict[str, Any], NodeDataset]: @@ -1721,16 +1799,19 @@ async def _node_run_context( Dataset connector the seeding is a no-op and the path comes from whatever an earlier node in the run already seeded. - An unsplit dataset supplies that path itself, and only as a fallback: a - workspace HEAD always wins, because a cell that has one has already moved - past the raw file (and a later Score step must fit on the engineered - matrix, not on the upload). + An explicitly wired unsplit dataset supplies the path itself and wins over + any existing workspace HEAD. A workspace is durable across reruns, so its + HEAD may belong to a dataset that was wired to an older protocol revision; + allowing it to override the current connector would silently give this + node data it is no longer attached to. Nodes with no Dataset connector + still inherit workspace HEAD, which is how later Score steps consume the + engineered matrix. *slot_prefix* gives this node a private workspace lineage (see - :func:`_resolve_node_dataset`). When it is set the ambient view is narrowed - to the slots that were just seeded for it, so a worker sharing a cell - workspace with several sibling workers still sees exactly one HEAD -- its - own -- rather than everybody's.""" + :func:`_resolve_node_dataset`). Any node that wires Dataset connectors has + its ambient view narrowed to the slots just resolved for those connectors; + this also ensures a worker sharing a cell workspace with sibling workers + sees its own HEAD rather than everybody's.""" dataset = await _resolve_node_dataset( graph, node_id, workspace_id, owner_id, slot_prefix=slot_prefix, stage_plan=stage_plan ) @@ -1738,22 +1819,29 @@ async def _node_run_context( graph, node_id, workspace_id, - slots=tuple(slot for _name, slot in dataset.seeded) if slot_prefix else (), + script_workspace_id=_script_workspace_id(workspace_id, protocol_run_id, node_id), + # A node with Dataset connectors sees only the slots those connectors + # just resolved. Nodes with none keep the whole-cell view so downstream + # stages can consume the workspace produced upstream. + slots=tuple(slot for _name, slot in dataset.seeded), ) - if dataset.data_path and "data_path" not in ambient_meta: + if dataset.data_path: + ambient_meta.pop("data_slots", None) ambient_meta["data_path"] = dataset.data_path + ambient_meta.pop("target_column", None) if dataset.target_column: ambient_meta["target_column"] = dataset.target_column + ambient_meta["dataset_mode"] = "raw_unsplit" return ambient_meta, dataset -def _resolve_llm_config(graph: dict[str, Any], node_id: str) -> dict[str, Any]: +def _resolve_model_config(graph: dict[str, Any], node_id: str) -> dict[str, Any]: """The node's connected ``llm`` node's own config -- agent/critic_gate nodes no longer carry ``model_config_data`` themselves, it's resolved - from the required LLM connector instead (``topological_order`` already + from the required Model connector instead (``topological_order`` already validated it exists exactly once).""" nodes, _downstream, _upstream = _adjacency(graph) - edges = _edges_with_handle(graph, node_id, "ai", direction="incoming") + edges = _edges_with_handle(graph, node_id, "model", direction="incoming") if not edges: return {} source = nodes.get(edges[0]["source"]) @@ -2091,6 +2179,27 @@ def _connected_agent_ids(graph: dict[str, Any], node_id: str) -> list[str]: return peers +def _sub_agent_ids(graph: dict[str, Any], parent_id: str) -> list[str]: + """Active Sub-Agents owned by *parent_id*, in canvas wiring order.""" + nodes = {str(n.get("id")): n for n in graph.get("nodes") or [] if n.get("id")} + if (nodes.get(parent_id) or {}).get("type") != "agent": + return [] + children: list[str] = [] + for edge in graph.get("edges") or []: + if edge.get("target") != parent_id or edge.get("targetHandle") != "sub_agents": + continue + child_id = str(edge.get("source")) + child = nodes.get(child_id) + if ( + child is not None + and child.get("type") == "sub_agent" + and _is_node_active(child) + and child_id not in children + ): + children.append(child_id) + return children + + def _can_deliver_communication(graph: dict[str, Any], from_agent_id: str, to_agent_id: str) -> bool: """Live authorization check, re-run for every consultation. @@ -2102,7 +2211,9 @@ def _can_deliver_communication(graph: dict[str, Any], from_agent_id: str, to_age """ if from_agent_id == to_agent_id: return False - return to_agent_id in _connected_agent_ids(graph, from_agent_id) + return to_agent_id in _connected_agent_ids(graph, from_agent_id) or to_agent_id in _sub_agent_ids( + graph, from_agent_id + ) async def resolve_agent_card( @@ -2122,7 +2233,7 @@ async def resolve_agent_card( """ nodes = {str(n.get("id")): n for n in graph.get("nodes") or [] if n.get("id")} node = nodes.get(node_id) - if node is None or node.get("type") != "agent": + if node is None or node.get("type") not in ("agent", "sub_agent") or not _is_node_active(node): return None config = (node.get("data") or {}).get("config") or {} # The registered skill documents, not the ids: a peer reads names and @@ -2135,7 +2246,7 @@ async def resolve_agent_card( description=config.get("description") or "", goal=config.get("goal") or "", skills=[dict(s) for s in skills], - model=_resolve_llm_config(graph, node_id).get("model"), + model=_resolve_model_config(graph, node_id).get("model"), metadata=metadata, ) @@ -2154,13 +2265,25 @@ async def resolve_available_agents( are bound no new function schemas (invariant 11). """ cards: list[dict[str, Any]] = [] - for peer_id in _connected_agent_ids(graph, node_id): + for peer_id in [*_connected_agent_ids(graph, node_id), *_sub_agent_ids(graph, node_id)]: card = await resolve_agent_card(graph, peer_id, owner_id=owner_id, metadata=metadata) if card is not None: cards.append(card.to_dict()) return cards +async def resolve_available_sub_agents( + graph: dict[str, Any], node_id: str, *, owner_id: uuid.UUID +) -> list[dict[str, Any]]: + """Serialized cards for active Sub-Agents owned by one parent Agent.""" + cards: list[dict[str, Any]] = [] + for child_id in _sub_agent_ids(graph, node_id): + card = await resolve_agent_card(graph, child_id, owner_id=owner_id) + if card is not None: + cards.append(card.to_dict()) + return cards + + def _resolve_pattern_config(graph: dict[str, Any], node_id: str) -> dict[str, Any]: """``{"execution_pattern": slug, "pattern_params": {slug: {...}}}`` from the node's connected execution-pattern node, or ``{}`` if none is @@ -2276,6 +2399,7 @@ def _resolve_knowledge_config(graph: dict[str, Any], node_id: str) -> dict[str, nodes, _downstream, _upstream = _adjacency(graph) server_names: list[str] = [] tool_names: list[str] = [] + tool_descriptions: dict[str, str] = {} for edge in _edges_with_handle(graph, node_id, "knowledge", direction="incoming"): source = nodes.get(edge["source"]) if source is None or source.get("type") not in _KNOWLEDGE_NODE_TYPES: @@ -2287,8 +2411,21 @@ def _resolve_knowledge_config(graph: dict[str, Any], node_id: str) -> dict[str, if not server_name or server_name in server_names: continue server_names.append(server_name) - tool_names.extend(f"{server_name}.{name}" for name in bundle_config.get("tool_names") or []) - return {"server_names": server_names, "tool_names": tool_names} + label = str( + bundle_config.get("document_title") + or bundle_config.get("bundle_label") + or (source.get("data") or {}).get("label") + or server_name + ) + summary = str(bundle_config.get("document_description") or bundle_config.get("bundle_description") or "") + for name in bundle_config.get("tool_names") or []: + full_name = f"{server_name}.{name}" + tool_names.append(full_name) + tool_descriptions[full_name] = f"Knowledge source: {label}. {summary}".strip() + resolved: dict[str, Any] = {"server_names": server_names, "tool_names": tool_names} + if tool_descriptions: + resolved["tool_descriptions"] = tool_descriptions + return resolved def _declares_a_field(contract: Any) -> bool: @@ -2320,7 +2457,7 @@ def _resolve_output_contract(graph: dict[str, Any], node_id: str) -> dict[str, A from the SDK or a notebook, at any time. There is no cutover date after which nothing produces one. - So this is unlike ``_LEGACY_AI_HANDLES``, which covers a rename whose + So this is unlike ``_LEGACY_MODEL_HANDLES``, which covers a rename whose stored data really was migrated: nothing here ever becomes dead code. The two sources are never merged and never race -- ``topological_order`` @@ -2357,13 +2494,13 @@ def _resolve_output_contract(graph: dict[str, Any], node_id: str) -> dict[str, A if not parser_config.get("enabled", True): continue contract = parser_config.get("output_contract") - if _declares_a_field(contract): + if isinstance(contract, dict) and _declares_a_field(contract): return dict(contract) if wired: return None node = nodes.get(node_id) or {} legacy = ((node.get("data") or {}).get("config") or {}).get("output_contract") - return dict(legacy) if _declares_a_field(legacy) else None + return dict(legacy) if isinstance(legacy, dict) and _declares_a_field(legacy) else None def _output_shape_block(contract: dict[str, Any] | None) -> str: @@ -2458,6 +2595,9 @@ def _resolve_dataset_tool_config(graph: dict[str, Any], node_id: str, *, unsplit if unsplit_dataset: server_names.append(SCIKIT_LEARN_SERVER_NAME) tool_names.extend(f"{SCIKIT_LEARN_SERVER_NAME}.{name}" for name in UNSPLIT_DATASET_AGENT_TOOLS) + if any(config.get("dictionary_available") for config in _resolve_dataset_configs(graph, node_id)): + server_names.append(EDA_SERVER_NAME) + tool_names.extend(f"{EDA_SERVER_NAME}.{name}" for name in DATASET_DICTIONARY_AGENT_TOOLS) return {"server_names": server_names, "tool_names": tool_names} @@ -2497,6 +2637,7 @@ def _merge_tool_configs(*configs: dict[str, Any]) -> dict[str, Any]: """ server_names: list[str] = [] tool_names: list[str] = [] + tool_descriptions: dict[str, str] = {} for config in configs: for name in config.get("server_names") or []: if name not in server_names: @@ -2504,7 +2645,11 @@ def _merge_tool_configs(*configs: dict[str, Any]) -> dict[str, Any]: for name in config.get("tool_names") or []: if name not in tool_names: tool_names.append(name) - return {"server_names": server_names, "tool_names": tool_names} + tool_descriptions.update(config.get("tool_descriptions") or {}) + resolved: dict[str, Any] = {"server_names": server_names, "tool_names": tool_names} + if tool_descriptions: + resolved["tool_descriptions"] = tool_descriptions + return resolved def _is_node_active(node: dict[str, Any]) -> bool: @@ -2901,6 +3046,72 @@ def validate_prompt_references(*, graph: dict[str, Any]) -> None: ) +def _resource_catalog(graph: dict[str, Any], node_id: str) -> str: + """Compact semantic metadata for references wired into one agent. + + Paths, ids, and source bodies stay out of the prompt. This block gives the + model only enough meaning to choose among already-authorized resources; + native tool schemas remain the interface for reading or executing them. + """ + sections: list[str] = [] + + datasets = _resolve_dataset_configs(graph, node_id) + if datasets: + lines = [] + for config in datasets: + name = str(config.get("dataset_name") or "dataset") + facts = [str(config.get("description") or "").strip()] + if config.get("target_column"): + facts.append(f"target={config['target_column']}") + if config.get("split_state"): + facts.append(f"state={config['split_state']}") + if config.get("dictionary_available"): + facts.append("data dictionary available through an authorized EDA tool") + detail = "; ".join(fact for fact in facts if fact) + lines.append(f"- {name}: {detail}" if detail else f"- {name}") + sections.append("Available datasets:\n" + "\n".join(lines)) + + nodes, _downstream, _upstream = _adjacency(graph) + knowledge_lines: list[str] = [] + for edge in _edges_with_handle(graph, node_id, "knowledge", direction="incoming"): + source = nodes.get(edge["source"]) + if source is None or source.get("type") not in _KNOWLEDGE_NODE_TYPES: + continue + config = (source.get("data") or {}).get("config") or {} + if not config.get("enabled", True): + continue + label = str( + config.get("document_title") + or config.get("bundle_label") + or (source.get("data") or {}).get("label") + or "knowledge source" + ) + facts = [str(config.get("document_description") or config.get("bundle_description") or "").strip()] + if config.get("document_type"): + facts.append(f"type={config['document_type']}") + tags = config.get("document_tags") or [] + if tags: + facts.append("tags=" + ", ".join(str(tag) for tag in tags)) + detail = "; ".join(fact for fact in facts if fact) + knowledge_lines.append(f"- {label}: {detail}" if detail else f"- {label}") + if knowledge_lines: + sections.append( + "Available knowledge sources (use their list/search/get tools to disclose content on demand):\n" + + "\n".join(knowledge_lines) + ) + + scripts = [config for config in _resolve_script_configs(graph, node_id) if config.get("code")] + if scripts: + lines = [] + for index, config in enumerate(scripts, start=1): + name = str(config.get("name") or f"script-{index}") + description = str(config.get("description") or "").strip() + lines.append(f"- {name}: {description}" if description else f"- {name}") + sections.append("Available scripts (source remains out of context until execution):\n" + "\n".join(lines)) + + return "\n\n".join(sections) + + def _build_user_input( node: dict[str, Any], graph: dict[str, Any], @@ -2937,9 +3148,10 @@ def _build_user_input( more than one is wired. *script_bound* says the wired scripts reached ``_meta`` as paths - (``_ambient_meta_for``). When it didn't -- an unlinked protocol run has no - workspace directory to write them to -- the code is inlined here as before, - because a prompt the model can copy from beats no scripts at all. + (``_ambient_meta_for``). Production callers provide either the experiment + workspace or an isolated standalone-run directory, so source stays out of + the prompt in both cases. The false branch is a defensive diagnostic for a + materialization failure. *seeded_datasets* are the ``(dataset name, workspace slot)`` pairs ASAREE already opened on the agent's behalf (``_resolve_node_dataset``). When @@ -2993,6 +3205,10 @@ def _build_user_input( if upstream_context: parts.append(upstream_context) + resource_catalog = _resource_catalog(graph, node["id"]) + if resource_catalog: + parts.append(resource_catalog) + dataset_configs = _resolve_dataset_configs(graph, node["id"]) if dataset_configs and experiment_id is not None and effective_cell_label is not None: dataset_names = [str(c["dataset_name"]) for c in dataset_configs] @@ -3084,7 +3300,7 @@ def _build_user_input( ) else: listed = "\n".join( - f'- {str(config.get("name") or f"script-{index}")!r} (id: {config["node_id"]!r})' + f"- {str(config.get('name') or f'script-{index}')!r} (id: {config['node_id']!r})" for index, config in enumerate(script_configs, start=1) ) parts.append( @@ -3095,25 +3311,12 @@ def _build_user_input( "select by id." ) elif script_configs: - # No workspace directory to write it to (see _materialize_script), so - # fall back to what this did before: paste it and ask for a verbatim - # copy. Costs prompt tokens on every turn and is only as faithful as - # the model's transcription -- which is the whole reason the path - # above exists. - if len(script_configs) == 1: - parts.append( - "Script to pass verbatim as the relevant tool's own code argument (run_wired_script's or " - f"run_model_script's `code`):\n```python\n{script_configs[0]['code']}\n```" - ) - else: - blocks = [] - for index, config in enumerate(script_configs, start=1): - name = str(config.get("name") or f"script-{index}") - blocks.append(f"Script {name!r} (id: {config['node_id']!r}):\n```python\n{config['code']}\n```") - parts.append( - "Scripts to pass verbatim as the relevant tool's own code argument " - "(run_wired_script's or run_model_script's `code`):\n" + "\n\n".join(blocks) - ) + parts.append( + "Script context:\n" + "A script is wired into this unlinked run, but no isolated run workspace exists in which to materialize " + "it. Its source has deliberately not been inserted into the prompt. Link the protocol to an experiment " + "to execute wired scripts." + ) # The shape block is the only *prose* this function composes. Everything # else appended here is either the user's own text or a labelled, fenced @@ -3209,6 +3412,14 @@ async def _preview_node_dataset(graph: dict[str, Any], node_id: str, owner_id: u reg = await fetch_owned_registration(name, owner_id) if reg is None: continue + for config in _resolve_dataset_configs(graph, node_id): + if str(config.get("dataset_name") or "") == name: + config.update( + description=reg.get("description"), + target_column=reg.get("target_column"), + split_state="split" if reg.get("train_path") and reg.get("test_path") else "unsplit", + dictionary_available=bool(reg.get("dictionary_json")), + ) if not (reg.get("train_path") and reg.get("test_path")): if solo: return NodeDataset( @@ -3276,7 +3487,7 @@ async def preview_node_prompt( node = nodes.get(node_id) if node is None: raise ProtocolValidationError(f"No node {node_id!r} on this canvas.") - if node.get("type") != "agent": + if node.get("type") not in ("agent", "sub_agent"): raise ProtocolValidationError(f"{_node_display_name(node)} is not an agent, so it is never given a prompt.") node_runs = { @@ -3575,14 +3786,14 @@ async def _run_agent_node( # into the description instead, purely as a human label. agent_name = f"protocol-{protocol_id}-{node['id']}" # Model/tool/execution-pattern are no longer fields on the agent's own - # config -- resolved from its required LLM connector, its (optional, + # config -- resolved from its required Model connector, its (optional, # repeatable) Tool connectors, and its optional Architectural Pattern # connector instead (topological_order already validated their shape). # output_contract joined them, with one extra argument the others didn't # need: extraction is a second LLM call per run, so its cost belongs on the # canvas. Unlike the three above, the node's own field is still read as a # fallback and always will be -- see _resolve_output_contract. - model_config_data = {k: v for k, v in _resolve_llm_config(graph, node["id"]).items() if v is not None} + model_config_data = {k: v for k, v in _resolve_model_config(graph, node["id"]).items() if v is not None} model_config = ModelConfig(**model_config_data) # Four connectors feed one allow-list. The Knowledge connector's OKF # bundles and documents are MCP servers like any other, so they land here @@ -3644,6 +3855,16 @@ async def _run_agent_node( ) assert agent is not None + resolved_ambient = ( + ambient_meta + if ambient_meta is not None + else _ambient_meta_for( + graph, + node["id"], + workspace_id, + script_workspace_id=_script_workspace_id(workspace_id, protocol_run_id, str(node["id"])), + ) + ) run = await create_run( agent_id=agent.id, user_input=user_input, @@ -3656,15 +3877,7 @@ async def _run_agent_node( # Precomputed by the caller when it also needed to know whether the # script got bound (_build_user_input's script_bound); recomputed # here only for a caller that didn't care. - **( - {"ambient_meta": resolved_ambient} - if ( - resolved_ambient := ( - ambient_meta if ambient_meta is not None else _ambient_meta_for(graph, node["id"], workspace_id) - ) - ) - else {} - ), + **({"ambient_meta": resolved_ambient} if resolved_ambient else {}), }, ) timeout = agent.max_run_duration_seconds or get_settings().worker_job_timeout_seconds @@ -3681,6 +3894,8 @@ async def _run_agent_node( return None, f"run exceeded its {timeout}s execution budget", run.id, None except Exception as e: # noqa: BLE001 -- same boundary reasoning as execute_run_task return None, f"{type(e).__name__}: {e}", run.id, None + finally: + _cleanup_adhoc_scripts(resolved_ambient) finished = await get_run(run.id) if finished is None: @@ -3720,10 +3935,10 @@ async def _run_critic( ``CRITIC_TOOLS = []`` / ``SINGLE_PASS_PATTERN``), and its ``output_contract`` is always :data:`CRITIC_OUTPUT_CONTRACT` -- not whatever (if anything) is in the node's own config. Model is resolved - from its required LLM connector, same as an agent node.""" + from its required Model connector, same as an agent node.""" config = gate["data"]["config"] agent_name = f"protocol-{protocol_id}-{gate['id']}" - model_config_data = {k: v for k, v in _resolve_llm_config(graph, gate["id"]).items() if v is not None} + model_config_data = {k: v for k, v in _resolve_model_config(graph, gate["id"]).items() if v is not None} model_config = ModelConfig(**model_config_data) pattern_config = PatternConfig(execution_pattern="single_agent_baseline").model_dump() goal = config.get("goal") or "Review the given output and return an approval verdict with feedback." @@ -3834,7 +4049,12 @@ async def _run_gated_worker( # worker against the same references, so re-materializing the script per # attempt would only rewrite an identical file. worker_ambient, worker_dataset = await _node_run_context( - graph, worker["id"], workspace_id, owner_id, stage_plan=stage_plan + graph, + worker["id"], + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan, ) base_instruction = _build_user_input( worker, @@ -4202,21 +4422,23 @@ def validate_single_node_runnable(graph: dict[str, Any], node_id: str) -> dict[s node = nodes.get(node_id) if node is None: raise ProtocolValidationError(f"No such node: {node_id!r}") - if node.get("type") != "agent": - raise ProtocolValidationError("Only Agent nodes can be run on their own.") + if node.get("type") not in ("agent", "sub_agent"): + raise ProtocolValidationError("Only Agent and Sub-Agent nodes can be run on their own.") if _upstream_ids(graph, node_id): raise ProtocolValidationError( "This agent has upstream input from another node -- running it alone isn't supported yet. " "Use the canvas's main Run button to run the whole pipeline." ) - llm_edges = _edges_with_handle(graph, node_id, "ai", direction="incoming") - if len(llm_edges) != 1: + model_edges = _edges_with_handle(graph, node_id, "model", direction="incoming") + if len(model_edges) != 1: + raise ProtocolValidationError( + f"Node {_node_display_name(node)!r} must have exactly one Model connection (found {len(model_edges)})." + ) + model_source = nodes.get(model_edges[0]["source"]) + if model_source is None or model_source.get("type") not in _MODEL_NODE_TYPES: raise ProtocolValidationError( - f"Node {_node_display_name(node)!r} must have exactly one AI connection (found {len(llm_edges)})." + f"Node {_node_display_name(node)!r}'s Model connection must come from a Model node." ) - llm_source = nodes.get(llm_edges[0]["source"]) - if llm_source is None or llm_source.get("type") not in _LLM_NODE_TYPES: - raise ProtocolValidationError(f"Node {_node_display_name(node)!r}'s AI connection must come from an AI node.") return node @@ -4248,16 +4470,16 @@ def validate_conversation_entry(graph: dict[str, Any], node_id: str) -> dict[str # or the consultation fails partway through a run the user already paid for. for participant_id in [node_id, *peers]: participant = nodes[participant_id] - llm_edges = _edges_with_handle(graph, participant_id, "ai", direction="incoming") - if len(llm_edges) != 1: + model_edges = _edges_with_handle(graph, participant_id, "model", direction="incoming") + if len(model_edges) != 1: raise ProtocolValidationError( - f"Node {_node_display_name(participant)!r} must have exactly one AI connection " - f"(found {len(llm_edges)})." + f"Node {_node_display_name(participant)!r} must have exactly one Model connection " + f"(found {len(model_edges)})." ) - llm_source = nodes.get(str(llm_edges[0]["source"])) - if llm_source is None or llm_source.get("type") not in _LLM_NODE_TYPES: + model_source = nodes.get(str(model_edges[0]["source"])) + if model_source is None or model_source.get("type") not in _MODEL_NODE_TYPES: raise ProtocolValidationError( - f"Node {_node_display_name(participant)!r}'s AI connection must come from an AI node." + f"Node {_node_display_name(participant)!r}'s Model connection must come from a Model node." ) return node @@ -4295,7 +4517,12 @@ async def _run_single_node( experiment = await get_experiment(db, experiment_id) if experiment_id else None single_design_spec = experiment.design_spec if experiment is not None else None ambient_meta, node_dataset = await _node_run_context( - graph, node["id"], workspace_id, owner_id, stage_plan=stage_plan_spec(single_design_spec, graph=graph) + graph, + node["id"], + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan_spec(single_design_spec, graph=graph), ) user_input = _build_user_input( node, @@ -4467,7 +4694,12 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: entry_agent_id = resolve_conversation_entry_id(graph) # already validated above entry_node = next(n for n in graph["nodes"] if str(n.get("id")) == entry_agent_id) ambient_meta, entry_dataset = await _node_run_context( - graph, entry_agent_id, workspace_id, owner_id, stage_plan=stage_plan + graph, + entry_agent_id, + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan, ) node_run, conversation_status = await execute_conversation( protocol_run_id, @@ -4515,7 +4747,12 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: # own Dataset/Script cues are rebuilt inside each of its two turns, # which is where the slot keys it will actually be given are known. ambient_meta, supervisor_dataset = await _node_run_context( - graph, roles.supervisor, workspace_id, owner_id, stage_plan=stage_plan + graph, + roles.supervisor, + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan, ) node_run, supervisor_status = await execute_supervisor_architecture( protocol_run_id, @@ -4553,6 +4790,26 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: sinks = sink_node_ids(graph) result_node_id = sinks[0] if len(sinks) == 1 else None + pipeline_messenger = None + if order: + pipeline_parents = [ + str(node.get("id")) + for node in order + if node.get("type") == "agent" and _sub_agent_ids(graph, str(node.get("id"))) + ] + if pipeline_parents: + from asaree.services.agent_messenger import AgentMessenger + + pipeline_messenger = AgentMessenger( + protocol_id=protocol_id, + protocol_run_id=protocol_run_id, + owner_id=owner_id, + graph=graph, + entry_agent_id=pipeline_parents[0], + workspace_id=workspace_id, + stage_plan=stage_plan, + ) + for node in order: node_id = node["id"] if node_id in node_runs: @@ -4578,12 +4835,16 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: continue if node.get("type") in _PURE_CONFIG_SOURCE_TYPES: - # Pure config sources -- never get their own execution turn (see - # _resolve_llm_config/_resolve_tool_config). Memory and - # architectural-pattern nodes are visual scaffolding only this - # phase: connecting one declares intent for a future phase, but - # has no runtime effect yet. - node_runs[node_id] = {"status": "completed", "output_text": None, "error": None} + # A Sub-Agent has no automatic pipeline turn. It starts skipped + # and AgentMessenger overwrites that status only if its parent + # actually invokes it; this also keeps its declared metrics blank + # when it was available but unused. The other members are pure + # config sources and count as resolved by the graph walk. + node_runs[node_id] = ( + {"status": "skipped"} + if node.get("type") == "sub_agent" + else {"status": "completed", "output_text": None, "error": None} + ) async with get_session() as db: await update_node_run(db, protocol_run_id, node_id, node_runs[node_id]) continue @@ -4647,7 +4908,12 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: ) else: ambient_meta, node_dataset = await _node_run_context( - graph, node_id, workspace_id, owner_id, stage_plan=stage_plan + graph, + node_id, + workspace_id, + owner_id, + protocol_run_id=protocol_run_id, + stage_plan=stage_plan, ) user_input = _build_user_input( node, @@ -4669,18 +4935,44 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: node_runs, unresolved_out=unresolved, ) - output_text, error, run_id, extraction = await _run_agent_node( - node, - protocol_id=protocol_id, - protocol_run_id=protocol_run_id, - owner_id=owner_id, - user_input=user_input, - graph=graph, - system_prompt=node_system_prompt, - workspace_id=workspace_id, - ambient_meta=ambient_meta, - unsplit_dataset=node_dataset.unsplit_name, - ) + available_sub_agents = await resolve_available_sub_agents(graph, node_id, owner_id=owner_id) + if pipeline_messenger is not None and available_sub_agents: + from asaree.services.agent_messenger import USER_PARTICIPANT + + pipeline_messenger.append( + from_agent_id=USER_PARTICIPANT, + to_agent_id=node_id, + parts=[{"kind": "text", "text": user_input}], + ) + await pipeline_messenger.checkpoint() + with pipeline_messenger.turn(node_id): + output_text, error, run_id, extraction = await _run_agent_node( + node, + protocol_id=protocol_id, + protocol_run_id=protocol_run_id, + owner_id=owner_id, + user_input=user_input, + graph=graph, + system_prompt=node_system_prompt, + workspace_id=workspace_id, + ambient_meta=ambient_meta, + available_agents=available_sub_agents, + agent_messenger=pipeline_messenger, + unsplit_dataset=node_dataset.unsplit_name, + ) + else: + output_text, error, run_id, extraction = await _run_agent_node( + node, + protocol_id=protocol_id, + protocol_run_id=protocol_run_id, + owner_id=owner_id, + user_input=user_input, + graph=graph, + system_prompt=node_system_prompt, + workspace_id=workspace_id, + ambient_meta=ambient_meta, + unsplit_dataset=node_dataset.unsplit_name, + ) if error == _AGENT_CANCELLED: node_runs[node_id] = { @@ -4709,6 +5001,10 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: failed = True async with get_session() as db: await update_node_run(db, protocol_run_id, node_id, node_runs[node_id]) + if pipeline_messenger is not None: + pipeline_messenger.set_state("canceled" if cancelled else ("failed" if failed else "completed")) + await pipeline_messenger.checkpoint() + if coordination_strategy_slug(design_spec) == "sequential": # A chain's handoffs are agent-to-agent messages, so they get the same # transcript a conversation does. Best-effort: a transcript is a view of @@ -4732,6 +5028,7 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: # transcript is meant to read as what was asked. entry_prompt=_node_seed_prompt(head) if head else "", state="canceled" if cancelled else ("failed" if failed else "completed"), + messenger=pipeline_messenger, ) except Exception: logger.exception("sequential_transcript_failed", extra={"protocol_run_id": str(protocol_run_id)}) @@ -4740,9 +5037,8 @@ async def run_protocol(protocol_run_id: uuid.UUID) -> None: if cancelled: await set_status(db, protocol_run_id, status="cancelled") elif failed: - # A conversation that ran out of budget gets its own terminal status - # rather than being flattened into "failed" -- it's the one failure - # mode the user fixes by raising a cap, not by fixing the protocol. + # Recursive conversation-depth exhaustion retains its distinct + # terminal status instead of being flattened into "failed". await set_status(db, protocol_run_id, status=failure_status, error=failure_error) else: await set_status(db, protocol_run_id, status="finalizing") diff --git a/src/asaree/services/protocol_graph_schema.py b/src/asaree/services/protocol_graph_schema.py new file mode 100644 index 0000000..f036ed2 --- /dev/null +++ b/src/asaree/services/protocol_graph_schema.py @@ -0,0 +1,99 @@ +"""Canonical names for structural fields in persisted protocol graphs.""" + +from __future__ import annotations + +import hashlib +import json +from copy import deepcopy +from typing import Any + +_LEGACY_MODEL_NODE_TYPES = { + "llm_anthropic": "model_anthropic", + "llm_openai": "model_openai", + "llm_azure_foundry": "model_azure_foundry", + "llm_openrouter": "model_openrouter", + "llm_local": "model_local", +} +_LEGACY_MODEL_HANDLES = frozenset({"ai", "llm"}) + + +def normalize_protocol_graph(graph: dict[str, Any]) -> dict[str, Any]: + """Return *graph* with legacy Model discriminators made canonical. + + Only schema-owned node ``type`` and edge handle fields are rewritten. + Node ids, labels, prompts, and arbitrary config values are opaque user + data and must never be changed by a vocabulary migration. + """ + normalized = deepcopy(graph) + nodes = normalized.get("nodes") + if isinstance(nodes, list): + for node in nodes: + if not isinstance(node, dict): + continue + node_type = node.get("type") + if isinstance(node_type, str) and node_type in _LEGACY_MODEL_NODE_TYPES: + node["type"] = _LEGACY_MODEL_NODE_TYPES[node_type] + + edges = normalized.get("edges") + if isinstance(edges, list): + for edge in edges: + if not isinstance(edge, dict): + continue + for field in ("sourceHandle", "targetHandle"): + if edge.get(field) in _LEGACY_MODEL_HANDLES: + edge[field] = "model" + return normalized + + +def functional_protocol_graph(graph: dict[str, Any]) -> dict[str, Any]: + """Return the canonical execution-relevant definition of *graph*. + + Canvas layout and React Flow bookkeeping remain useful in the autosaved + draft, but they do not change what production executes. Node ids are + retained because prompt references and edges address them; edge ids are + omitted because execution identifies an edge by its endpoints and handles. + Sorting makes equivalent JSON arrays produce the same fingerprint. + """ + normalized = normalize_protocol_graph(graph) + nodes = normalized.get("nodes") + edges = normalized.get("edges") + + functional_nodes = [ + { + "id": node.get("id"), + "type": node.get("type"), + "data": deepcopy(node.get("data")), + } + for node in nodes or [] + if isinstance(node, dict) + ] + functional_edges = [ + { + "source": edge.get("source"), + "target": edge.get("target"), + "sourceHandle": edge.get("sourceHandle"), + "targetHandle": edge.get("targetHandle"), + } + for edge in edges or [] + if isinstance(edge, dict) + ] + + def canonical_json(value: dict[str, Any]) -> str: + return json.dumps(value, sort_keys=True, separators=(",", ":")) + + functional_nodes.sort(key=canonical_json) + functional_edges.sort(key=canonical_json) + return {"nodes": functional_nodes, "edges": functional_edges} + + +def functional_protocol_graph_hash(graph: dict[str, Any]) -> str: + """Return a stable SHA-256 fingerprint of the functional canvas.""" + encoded = json.dumps( + functional_protocol_graph(graph), + sort_keys=True, + separators=(",", ":"), + ).encode() + return hashlib.sha256(encoded).hexdigest() + + +__all__ = ["functional_protocol_graph", "functional_protocol_graph_hash", "normalize_protocol_graph"] diff --git a/src/asaree/services/protocol_revisions.py b/src/asaree/services/protocol_revisions.py index b8a577e..636382a 100644 --- a/src/asaree/services/protocol_revisions.py +++ b/src/asaree/services/protocol_revisions.py @@ -13,6 +13,7 @@ from asaree.models.protocol import Protocol from asaree.models.protocol_revision import ProtocolRevision from asaree.models.protocol_run import ProtocolRun +from asaree.services.protocol_graph_schema import functional_protocol_graph_hash async def get_revision(db: AsyncSession, revision_id: uuid.UUID) -> ProtocolRevision | None: @@ -27,6 +28,10 @@ async def get_published_revision(db: AsyncSession, protocol: Protocol) -> Protoc async def publish_protocol(db: AsyncSession, protocol: Protocol) -> ProtocolRevision: """Freeze the protocol's current draft as its next production revision.""" + published = await get_published_revision(db, protocol) + if published is not None and is_draft_published(protocol, published): + return published + highest = ( await db.execute( select(func.max(ProtocolRevision.revision)).where(ProtocolRevision.protocol_id == protocol.id) @@ -79,7 +84,9 @@ async def publish_protocol(db: AsyncSession, protocol: Protocol) -> ProtocolRevi def is_draft_published(protocol: Protocol, published: ProtocolRevision | None) -> bool: - return published is not None and protocol.graph == published.graph + return published is not None and functional_protocol_graph_hash(protocol.graph) == functional_protocol_graph_hash( + published.graph + ) __all__ = ["get_published_revision", "get_revision", "is_draft_published", "publish_protocol"] diff --git a/src/asaree/services/protocols.py b/src/asaree/services/protocols.py index 20a473e..5f81ceb 100644 --- a/src/asaree/services/protocols.py +++ b/src/asaree/services/protocols.py @@ -11,6 +11,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from asaree.models.protocol import Protocol +from asaree.services.protocol_graph_schema import normalize_protocol_graph _DEFAULT_GRAPH: dict[str, Any] = {"nodes": [], "edges": []} _SETTABLE_FIELDS = frozenset({"name", "description", "experiment_id", "graph"}) @@ -86,7 +87,7 @@ async def create_protocol( name=name, description=description, experiment_id=experiment_id, - graph=graph if graph is not None else dict(_DEFAULT_GRAPH), + graph=normalize_protocol_graph(graph) if graph is not None else dict(_DEFAULT_GRAPH), owner_id=owner_id, ) db.add(protocol) @@ -140,6 +141,8 @@ async def update_protocol( protocol = await get_protocol(db, protocol_id) if protocol is None: return None + if isinstance(fields.get("graph"), dict): + fields = {**fields, "graph": normalize_protocol_graph(fields["graph"])} for key, value in fields.items(): setattr(protocol, key, value) await db.flush() diff --git a/src/asaree/services/reported_metrics.py b/src/asaree/services/reported_metrics.py index c47cea1..84b252f 100644 --- a/src/asaree/services/reported_metrics.py +++ b/src/asaree/services/reported_metrics.py @@ -85,7 +85,9 @@ async def _last_matching_call( ) -> Mapping[str, Any] | None: agent_node_id = str(binding.config.get("agent_node_id") or "") node_run = (run.node_runs or {}).get(agent_node_id) - agent_run_id = node_run.get("run_id") if isinstance(node_run, Mapping) else None + agent_run_id = ( + node_run.get("last_successful_run_id") or node_run.get("run_id") if isinstance(node_run, Mapping) else None + ) try: steps = await get_run_steps(uuid.UUID(str(agent_run_id))) except (TypeError, ValueError): @@ -135,11 +137,16 @@ async def collect_reported_metrics( if binding.producer_id not in REPORTED_PRODUCER_IDS: continue node_run = (run.node_runs or {}).get(str(binding.config.get("agent_node_id") or "")) + successful_output = ( + node_run.get("last_successful_output_text", node_run.get("output_text")) + if isinstance(node_run, Mapping) + else None + ) agent_output_available = ( binding.producer_id == AGENT_OUTPUT_PRODUCER_ID and isinstance(node_run, Mapping) - and node_run.get("status") == "completed" - and node_run.get("output_text") is not None + and (node_run.get("status") == "completed" or "last_successful_output_text" in node_run) + and successful_output is not None ) call = ( None if binding.producer_id == AGENT_OUTPUT_PRODUCER_ID else await _last_matching_call(run, binding, graph) @@ -175,7 +182,7 @@ async def collect_reported_metrics( metric_name=metric.name, value_type=metric.value_type, value=( - node_run.get("output_text") + successful_output if agent_output_available else call.get("result") if call is not None @@ -262,7 +269,7 @@ def _agent_issue( preserved_binding_ids, has_preserved_value=isinstance(agent_id, str) and bool(agent_id), ) - if agent is None or agent.get("type") != "agent": + if agent is None or agent.get("type") not in ("agent", "sub_agent"): return ValidationIssue(f"{prefix}_agent_missing", "The source Agent is unavailable.", path, severity) if _node_data(agent).get("active") is False: return ValidationIssue(f"{prefix}_agent_disabled", "The source Agent is disabled.", path, severity) diff --git a/src/asaree/services/run_tools.py b/src/asaree/services/run_tools.py index be053c5..d5289aa 100644 --- a/src/asaree/services/run_tools.py +++ b/src/asaree/services/run_tools.py @@ -49,13 +49,15 @@ def gather_tools(agent: Any) -> list[dict[str, Any]]: names -- the namespaced form has to keep its ``.`` for ``lookup_tool`` to resolve it directly, which costs the provider-facing name a sanitising hash suffix, so it isn't worth applying to tools that don't need it. - Collisions are computed over the whole registry, not just this run's - allow-list, because ``lookup_tool``'s bare-name index is registry-wide. + Collisions are computed over the owner's visible registry, not just this + run's allow-list, because a bare name still has to be unambiguous among + every server that owner can use. """ tool_names = set((agent.tool_config_data or {}).get("tool_names") or []) + description_prefixes = (agent.tool_config_data or {}).get("tool_descriptions") or {} if not tool_names: return [] - catalog = get_registry().get_all_tools() + catalog = get_registry().get_all_tools(owner_id=agent.owner_id) servers_by_bare_name: dict[str, set[str]] = {} for tool in catalog: bare = str(tool.get("tool_name") or "") @@ -66,5 +68,9 @@ def gather_tools(agent: Any) -> list[dict[str, Any]]: continue if len(servers_by_bare_name.get(str(tool.get("tool_name") or ""), ())) > 1: tool = {**tool, "tool_name": tool["name"]} + prefix = str(description_prefixes.get(tool["name"]) or "").strip() + if prefix: + description = str(tool.get("description") or "").strip() + tool = {**tool, "description": f"{prefix}\n\n{description}" if description else prefix} admitted.append(tool) return admitted diff --git a/src/asaree/services/skill_sources.py b/src/asaree/services/skill_sources.py index 9cf98df..be137b4 100644 --- a/src/asaree/services/skill_sources.py +++ b/src/asaree/services/skill_sources.py @@ -16,12 +16,10 @@ capped small, and a server-side cache keyed by URL is state with a TTL to get wrong for no gain at this size. -**Nothing is ever written to disk and nothing is executed.** The archive is -walked in memory and only regular files whose paths pass core's -``validate_bundle_path`` survive, so a skill from a stranger's repository -arrives as exactly the same rows as one the user picked out of a folder -- -scripts refused, text only, the same caps. That is the whole security posture -for the *contents*; the fetch itself is guarded below. +**Nothing is ever written to disk or executed during acquisition.** The archive +is walked in memory and only regular, contained paths survive. Scripts and +binary assets are preserved as inert skill resources; execution is a separate +model-controlled runtime action under the product's trust policy. """ from __future__ import annotations @@ -204,18 +202,15 @@ async def _download_archive(source: GithubSource) -> tuple[bytes, str]: ) -def _read_archive(raw: bytes) -> dict[str, str]: - """``{repo-relative path: text}`` for every readable file in the tarball. +def _read_archive(raw: bytes) -> dict[str, bytes]: + """``{repo-relative path: bytes}`` for every regular file in the tarball. - Three things are dropped rather than raised over, because a repository is - full of them and none is the user's mistake: non-regular members - (symlinks, hardlinks, devices -- nothing here should be able to point - *out* of the archive), files that are not UTF-8, and anything - ``validate_bundle_path`` refuses. That last one is what keeps a fetched - skill identical to an uploaded one: scripts, images and hidden paths never - become rows, and the rule lives in core rather than being restated here. + Non-regular members (symlinks, hardlinks, devices) and invalid/hidden paths + are dropped rather than raised over because repositories commonly contain + them. Regular resources remain bytes so scripts and binary assets survive + with the same semantics as a directly uploaded skill folder. """ - files: dict[str, str] = {} + files: dict[str, bytes] = {} extracted = 0 with tarfile.open(fileobj=io.BytesIO(raw), mode="r:gz") as archive: for index, member in enumerate(archive): @@ -237,14 +232,11 @@ def _read_archive(raw: bytes) -> dict[str, str]: handle = archive.extractfile(member) if handle is None: # pragma: no cover -- isfile() already excludes these continue - try: - files[relative] = handle.read().decode("utf-8") - except UnicodeDecodeError: - continue + files[relative] = handle.read() return files -def _skill_dirs(files: dict[str, str], subdirectory: str) -> list[str]: +def _skill_dirs(files: dict[str, bytes], subdirectory: str) -> list[str]: """Directories holding a ``SKILL.md``, at or under *subdirectory*.""" prefix = f"{subdirectory}/" if subdirectory else "" found: list[str] = [] @@ -260,7 +252,7 @@ def _skill_dirs(files: dict[str, str], subdirectory: str) -> list[str]: return sorted(found) -def _bundle_at(files: dict[str, str], subdirectory: str, siblings: Sequence[str] = ()) -> list[tuple[str, str]]: +def _bundle_at(files: dict[str, bytes], subdirectory: str, siblings: Sequence[str] = ()) -> list[tuple[str, bytes]]: """One skill directory's subtree, re-rooted so ``SKILL.md`` is at the top. Re-rooting is the point: core's ``parse_skill_bundle`` wants paths relative @@ -306,10 +298,10 @@ async def discover_skills(url: str) -> tuple[GithubSource, list[DiscoveredSkill] discovered: list[DiscoveredSkill] = [] for directory in directories: bundle = _bundle_at(files, directory, directories) - entry = next((text for path, text in bundle if path.lower() == SKILL_MD.lower()), "") + entry_bytes = next((content for path, content in bundle if path.lower() == SKILL_MD.lower()), b"") try: - parsed = parse_skill_markdown(entry) - except SkillFormatError: + parsed = parse_skill_markdown(entry_bytes.decode("utf-8")) + except (SkillFormatError, UnicodeDecodeError): # Listed, not refused: a repo may hold one malformed skill among # ten good ones, and dropping it silently would look like it # simply is not there. Registering it is what surfaces the reason. @@ -333,7 +325,7 @@ async def discover_skills(url: str) -> tuple[GithubSource, list[DiscoveredSkill] return resolved, discovered -async def fetch_skill_bundle(url: str, subdirectory: str) -> tuple[GithubSource, list[tuple[str, str]]]: +async def fetch_skill_bundle(url: str, subdirectory: str) -> tuple[GithubSource, list[tuple[str, bytes]]]: """One skill's files, ready for ``create_skill_from_bundle``. *subdirectory* is repo-relative and comes from a prior diff --git a/src/asaree/services/system_mcp_servers.py b/src/asaree/services/system_mcp_servers.py index 64ec6e3..3bde193 100644 --- a/src/asaree/services/system_mcp_servers.py +++ b/src/asaree/services/system_mcp_servers.py @@ -36,6 +36,7 @@ SCRIPT_SERVER_NAME = "asaree-script" OKF_SERVER_NAME = "motoro-okf" SCIKIT_LEARN_SERVER_NAME = "scikit-learn-mcp" +EDA_SERVER_NAME = "asaree-sklearn-eda" # The workspace tools every agent with a Dataset connector wired gets, without # the user having to also drag an asaree-workspace Tool node onto the canvas @@ -94,6 +95,8 @@ "train_test_split", ) +DATASET_DICTIONARY_AGENT_TOOLS: Final[tuple[str, ...]] = ("get_data_dictionary",) + # (server name, module to run). Every module here is importable from this # repo's own venv -- asaree.* is ASAREE, motoro.* comes from the pinned Motoro # dependency, and the asaree_sklearn_* packages are the mcp-servers/ path @@ -272,12 +275,12 @@ async def refresh_system_server_capabilities() -> None: registry = get_registry().servers for name, _ in SYSTEM_MCP_SERVERS: try: - entry = registry.get(name) - if entry is None or not entry.client.connected: - continue config = await get_server_by_name(name) if config is None: continue + entry = registry.get(config.id) + if entry is None or not entry.client.connected: + continue capabilities = config.capabilities or {} live = sorted(t.name for t in entry.client.tools) stored = sorted(t.get("name", "") for t in capabilities.get("tools") or []) diff --git a/tests/fixtures/prompts/spinal_fte.txt b/tests/fixtures/prompts/spinal_fte.txt index 42aed46..51a7d39 100644 --- a/tests/fixtures/prompts/spinal_fte.txt +++ b/tests/fixtures/prompts/spinal_fte.txt @@ -9,6 +9,9 @@ task_brief: {"domain": "elective spinal fusion surgeries, Cedars-Sinai Medical C DC accepted v1_dc. <<>> +Available datasets: +- spinal-fusion + Dataset context: A dataset is registered for this run. Call open_workspace() before doing any data work -- it takes no arguments here; which dataset and which workspace both arrive as ambient run context. Its response names what it opened. @@ -22,4 +25,4 @@ Write your answer as you normally would. Then, as the very last thing in your re ```json {"engineering_recipe": null, "encoding_map": null, "n_features_out": null, "notes_for_fs": null} ``` -Replace each null with the value your answer establishes, leaving it null where your answer establishes none. Write nothing after the block. \ No newline at end of file +Replace each null with the value your answer establishes, leaving it null where your answer establishes none. Write nothing after the block. diff --git a/tests/fixtures/spinal_graph.json b/tests/fixtures/spinal_graph.json index 24bc666..3cb2518 100644 --- a/tests/fixtures/spinal_graph.json +++ b/tests/fixtures/spinal_graph.json @@ -38,58 +38,58 @@ { "id": "node-msza682j-6y15rt43", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-w2vslmwn", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-njszbkth", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-hiy7igv5", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-1x5e60c9", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-7qbpg0yc", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-2ty4tcke", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-i5qx7sfa", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-hkq1v06s", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-qvpuwtef", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-j2ekhe90", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-c6k6eyan", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-4glesw3r", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-ffxyvn78", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-tpg8d3td", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-gvf1u5gg", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-1tww3o22", @@ -211,9 +211,9 @@ { "id": "node-msza682j-ptkvuihb", "source": "node-msza682j-etejc4b2", - "sourceHandle": "ai", + "sourceHandle": "model", "target": "node-msza682j-tqhfa0mx", - "targetHandle": "ai" + "targetHandle": "model" }, { "id": "node-msza682j-iy6j9ws7", @@ -258,7 +258,7 @@ "x": 487.0905726370586, "y": 909.7169944010625 }, - "type": "llm_azure_foundry" + "type": "model_azure_foundry" }, { "data": { diff --git a/tests/test_agent_cards.py b/tests/test_agent_cards.py index 1095ff6..689fdcd 100644 --- a/tests/test_agent_cards.py +++ b/tests/test_agent_cards.py @@ -72,7 +72,7 @@ def test_non_agent_nodes_are_never_peers() -> None: "nodes": [ _agent("planner"), {"id": "gate", "type": "critic_gate", "data": {"config": {}}}, - {"id": "llm", "type": "llm_anthropic", "data": {"config": {}}}, + {"id": "llm", "type": "model_anthropic", "data": {"config": {}}}, ], "edges": [_edge("planner", "gate"), _edge("llm", "planner")], } @@ -181,8 +181,8 @@ async def test_the_card_carries_the_node_id_not_the_motoro_agent_id() -> None: async def test_the_model_family_comes_from_the_wired_ai_connector() -> None: graph = _pair_graph() - graph["nodes"].append({"id": "llm", "type": "llm_anthropic", "data": {"config": {"model": "claude-sonnet-5"}}}) - graph["edges"].append(_edge("llm", "critic", "ai")) + graph["nodes"].append({"id": "llm", "type": "model_anthropic", "data": {"config": {"model": "claude-sonnet-5"}}}) + graph["edges"].append(_edge("llm", "critic", "model")) card = await pe.resolve_agent_card(graph, "critic", owner_id=OWNER) assert card is not None @@ -190,7 +190,7 @@ async def test_the_model_family_comes_from_the_wired_ai_connector() -> None: async def test_a_non_agent_node_has_no_card() -> None: - graph = {"nodes": [{"id": "llm", "type": "llm_anthropic", "data": {"config": {}}}], "edges": []} + graph = {"nodes": [{"id": "llm", "type": "model_anthropic", "data": {"config": {}}}], "edges": []} assert await pe.resolve_agent_card(graph, "llm", owner_id=OWNER) is None assert await pe.resolve_agent_card(graph, "missing", owner_id=OWNER) is None diff --git a/tests/test_agent_messenger.py b/tests/test_agent_messenger.py index c8ab634..ae2f714 100644 --- a/tests/test_agent_messenger.py +++ b/tests/test_agent_messenger.py @@ -1,4 +1,4 @@ -"""Consultation delivery: authorization, budgets, transcript, and the clock. +"""Consultation delivery: authorization, recursion, transcript, and the clock. The property under test throughout is that **a refusal is an answer, not an error**. Every cap and every authorization failure below asserts on the returned @@ -8,7 +8,7 @@ DB-free by construction: ``get_session`` and the five service calls the messenger makes through it are stubbed, so what runs is exactly the ordering, -budget and authorization logic and nothing else. +recursion and authorization logic and nothing else. """ from __future__ import annotations @@ -51,7 +51,7 @@ def __init__(self, **kwargs: Any) -> None: @pytest.fixture def stubs(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]: - """Cut every DB call, keeping the real authorization and budget logic.""" + """Cut every DB call, keeping the real authorization and recursion logic.""" state: dict[str, Any] = { "checkpoints": [], "node_runs": [], @@ -145,6 +145,20 @@ async def test_a_consultation_runs_the_peer_and_returns_its_words(stubs: dict[st assert stubs["peer_runs"][0][1] == "What is weak here?" +async def test_a_parsed_reply_returns_compact_json_and_records_the_payload( + stubs: dict[str, Any], monkeypatch: pytest.MonkeyPatch +) -> None: + async def _run_agent_node(*_args: Any, **_kwargs: Any) -> tuple[str, None, uuid.UUID, dict[str, Any]]: + return "prose", None, uuid.uuid4(), {"payload": {"score": 0.91}} + + monkeypatch.setattr(am, "_run_agent_node", _run_agent_node) + reply = await _ask(_messenger()) + + assert reply.text == '{"score":0.91}' + assert stubs["node_runs"][-1][1]["payload"] == {"score": 0.91} + assert stubs["node_runs"][-1][1]["last_successful_output_text"] == '{"score":0.91}' + + async def test_the_peer_is_handed_the_messenger_so_it_can_consult_back(stubs: dict[str, Any]) -> None: """Recursion is the mechanism, bounded by depth -- not something bolted on top of a flat exchange.""" @@ -385,22 +399,16 @@ async def test_a_stop_between_consultations_prevents_further_ones(stubs: dict[st # ---------------------------------------------------------------------- -# Budgets +# Recursion guard # ---------------------------------------------------------------------- -async def test_the_execution_budget_is_spent_not_bypassed( - stubs: dict[str, Any], monkeypatch: pytest.MonkeyPatch -) -> None: - monkeypatch.setattr(am, "_MAX_PEER_EXECUTIONS", 2) +async def test_sequential_consultations_are_not_capped(stubs: dict[str, Any]) -> None: messenger = _messenger() - assert (await _ask(messenger)).state == "completed" - assert (await _ask(messenger)).state == "completed" - exhausted = await _ask(messenger) - assert exhausted.state == "rejected" - assert "consultations" in exhausted.text - assert len(stubs["peer_runs"]) == 2 - assert messenger.limit_reached is True + for _ in range(10): + assert (await _ask(messenger)).state == "completed" + assert len(stubs["peer_runs"]) == 10 + assert messenger.limit_reached is False async def test_depth_is_capped_and_the_cap_is_a_reply(stubs: dict[str, Any], monkeypatch: pytest.MonkeyPatch) -> None: @@ -442,20 +450,6 @@ async def test_depth_is_released_when_a_consultation_returns( assert (await _ask(messenger)).state == "completed" -async def test_the_conversation_wall_clock_is_the_backstop( - stubs: dict[str, Any], monkeypatch: pytest.MonkeyPatch -) -> None: - """Peer time is given back to individual agents but never to this cap -- - otherwise nothing would bound the total.""" - from datetime import timedelta - - monkeypatch.setattr(am, "_MAX_CONVERSATION_DURATION", timedelta(seconds=0)) - reply = await _ask(_messenger()) - assert reply.state == "rejected" - assert "seconds" in reply.text - assert stubs["peer_runs"] == [] - - # ---------------------------------------------------------------------- # Timeout accounting # ---------------------------------------------------------------------- @@ -666,6 +660,58 @@ async def test_a_completed_chain_reads_as_a_conversation(stubs: dict[str, Any]) assert am._text_of(conversation["messages"][3]["parts"]) == "AUC 0.81." +async def test_sequential_handoffs_preserve_nested_delegation_messages(stubs: dict[str, Any]) -> None: + messenger = am.AgentMessenger( + protocol_id=PROTOCOL_ID, + protocol_run_id=RUN_ID, + owner_id=OWNER, + graph=_chain_graph(), + entry_agent_id="a", + workspace_id=None, + ) + messenger.append( + from_agent_id=am.USER_PARTICIPANT, + to_agent_id="a", + parts=[{"kind": "text", "text": "Fit a model."}], + ) + messenger.append( + from_agent_id="a", + to_agent_id="worker", + parts=[{"kind": "text", "text": "Check the assumptions."}], + ) + messenger.append( + from_agent_id="worker", + to_agent_id="a", + parts=[{"kind": "text", "text": "Assumptions pass."}], + ) + + await am.record_sequential_transcript( + RUN_ID, + protocol_id=PROTOCOL_ID, + owner_id=OWNER, + graph=_chain_graph(), + chain=["a", "b", "c"], + node_runs={ + "a": {"status": "completed", "output_text": "Cleaned."}, + "b": {"status": "completed", "output_text": "Fitted."}, + "c": {"status": "completed", "output_text": "AUC 0.81."}, + }, + entry_prompt="Fit a model.", + state="completed", + messenger=messenger, + ) + + conversation = stubs["checkpoints"][-1] + assert [(m["from_agent_id"], m["to_agent_id"]) for m in conversation["messages"]] == [ + (am.USER_PARTICIPANT, "a"), + ("a", "worker"), + ("worker", "a"), + ("a", "b"), + ("b", "c"), + ("c", am.USER_PARTICIPANT), + ] + + async def test_the_transcript_stops_where_the_chain_stopped(stubs: dict[str, Any]) -> None: """A step that never ran has nothing to hand on, and a placeholder for it would read as an agent that answered.""" diff --git a/tests/test_dataset_workspaces.py b/tests/test_dataset_workspaces.py index e1f0900..f4d6ee6 100644 --- a/tests/test_dataset_workspaces.py +++ b/tests/test_dataset_workspaces.py @@ -12,7 +12,7 @@ import pytest from asaree_workspace_core import workspace as ws_module -from asaree.services.dataset_workspaces import head_data_locator +from asaree.services.dataset_workspaces import head_data_locator, raw_training_data_locators def _write_state(root: Path, workspace_id: str, state: dict) -> None: @@ -41,6 +41,13 @@ def test_head_data_locator_names_the_head_version(tmp_path: Path, monkeypatch: p # from the file they're handed, so naming the frozen test parquet would # invite fitting on it. assert head_data_locator("exp1/cellA") == ("/ws/v1_dc/train.parquet", "outcome") + assert raw_training_data_locators("exp1/cellA") == { + "dataset:default": { + "name": "default", + "data_path": "/uploads/train.parquet", + "target_column": "outcome", + } + } def test_head_data_locator_is_total(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: @@ -52,3 +59,4 @@ def test_head_data_locator_is_total(tmp_path: Path, monkeypatch: pytest.MonkeyPa _write_state(tmp_path, "exp1/cellB", {"target_column": "outcome", "head": "v9", "versions": []}) assert head_data_locator("exp1/cellB") == ("", "") # HEAD missing from state + assert raw_training_data_locators("exp1/never-seeded") == {} diff --git a/tests/test_datasets.py b/tests/test_datasets.py index d88a18d..333062c 100644 --- a/tests/test_datasets.py +++ b/tests/test_datasets.py @@ -17,10 +17,12 @@ from asaree.models.database import dispose_engine, get_session from asaree.models.user import User from asaree.services.datasets import ( + DatasetNameConflictError, DatasetValidationError, create_dataset, delete_dataset, get_dataset, + get_dataset_by_name, quick_split_dataset, register_manual_split, ) @@ -67,6 +69,55 @@ async def test_create_dataset_stores_raw_file_without_splitting(owner_id: uuid.U await delete_dataset(db, dataset.id) +async def test_dataset_names_are_scoped_per_owner(owner_id: uuid.UUID) -> None: + name = f"shared-dataset-{uuid.uuid4().hex}" + async with get_session() as db: + other = User( + email=f"dataset-other-{uuid.uuid4().hex}@example.com", + hashed_password="not-a-real-hash", + display_name="Other Dataset User", + ) + db.add(other) + await db.flush() + other_owner_id = other.id + + dataset_ids: list[uuid.UUID] = [] + try: + for current_owner_id in (owner_id, other_owner_id): + async with get_session() as db: + dataset = await create_dataset(db, name=name, csv_bytes=_CSV, owner_id=current_owner_id) + dataset_ids.append(dataset.id) + + async with get_session() as db: + first = await get_dataset_by_name(db, name, owner_id=owner_id) + second = await get_dataset_by_name(db, name, owner_id=other_owner_id) + assert first is not None and first.id == dataset_ids[0] + assert second is not None and second.id == dataset_ids[1] + finally: + for dataset_id in dataset_ids: + async with get_session() as db: + await delete_dataset(db, dataset_id) + async with get_session() as db: + other = await db.get(User, other_owner_id) + if other is not None: + await db.delete(other) + + +async def test_dataset_names_remain_unique_within_one_owner(owner_id: uuid.UUID) -> None: + name = f"duplicate-dataset-{uuid.uuid4().hex}" + async with get_session() as db: + dataset = await create_dataset(db, name=name, csv_bytes=_CSV, owner_id=owner_id) + dataset_id = dataset.id + + try: + with pytest.raises(DatasetNameConflictError, match="already in use"): + async with get_session() as db: + await create_dataset(db, name=name, csv_bytes=_CSV, owner_id=owner_id) + finally: + async with get_session() as db: + await delete_dataset(db, dataset_id) + + async def test_create_dataset_rejects_unparseable_csv(owner_id: uuid.UUID) -> None: async with get_session() as db: with pytest.raises(DatasetValidationError, match="could not parse CSV"): diff --git a/tests/test_design_generation.py b/tests/test_design_generation.py index 3d65400..f8f5dc9 100644 --- a/tests/test_design_generation.py +++ b/tests/test_design_generation.py @@ -82,7 +82,7 @@ def test_replicate_label_maps_to_its_cell_and_number() -> None: def test_cell_label_for_dict_valued_level_prefers_identifying_key() -> None: - """A whole-node factor (LLM/Tool config, pattern override) is a dict, not + """A whole-node factor (Model/Tool config, pattern override) is a dict, not a scalar -- the label should read as the thing a human recognizes ("claude-sonnet-5"), not Python's own dict repr.""" combo = {"llm": {"provider": "anthropic", "model": "claude-sonnet-5", "temperature": 0.7}} diff --git a/tests/test_llm_model_discovery.py b/tests/test_llm_model_discovery.py index 6cbf75e..8e58292 100644 --- a/tests/test_llm_model_discovery.py +++ b/tests/test_llm_model_discovery.py @@ -478,7 +478,7 @@ async def test_local_discovery_no_listing_route_surfaces_as_error_source_so_the_ monkeypatch: pytest.MonkeyPatch, ) -> None: """A server with no GET /models route is a normal, expected outcome -- not - a real error -- but LlmNodeInspector.tsx only renders `note` when + a real error -- but ModelNodeInspector.tsx only renders `note` when `source == "error"`, so that's the source used to actually surface it.""" response = _FakeResponse({}, status_code=404) monkeypatch.setattr(discovery.httpx, "AsyncClient", lambda **kwargs: _SimpleClient(response, captured={})) diff --git a/tests/test_mcp_server_annotations.py b/tests/test_mcp_server_annotations.py index 70a76ce..dff2c00 100644 --- a/tests/test_mcp_server_annotations.py +++ b/tests/test_mcp_server_annotations.py @@ -1,3 +1,4 @@ +import uuid from types import SimpleNamespace import pytest @@ -19,9 +20,11 @@ async def list_tools(self) -> SimpleNamespace: return SimpleNamespace(tools=[SimpleNamespace(name="score", annotations=_Annotations())]) client = SimpleNamespace(_session=_Session()) - registry = SimpleNamespace(servers={"quality": SimpleNamespace(client=client)}) + server_id = uuid.uuid4() + registry = SimpleNamespace(servers={server_id: SimpleNamespace(client=client)}) monkeypatch.setattr(mcp_servers, "get_registry", lambda: registry) config = SimpleNamespace( + id=server_id, name="quality", capabilities={"tools": [{"name": "score", "description": "Scores an answer", "input_schema": {}}]}, ) diff --git a/tests/test_metrics.py b/tests/test_metrics.py index 55874a6..94dc507 100644 --- a/tests/test_metrics.py +++ b/tests/test_metrics.py @@ -222,6 +222,14 @@ def test_design_spec_adds_short_default_labels_for_legacy_factor_levels() -> Non } +def test_design_spec_normalizes_legacy_model_factor_kind() -> None: + spec = normalize_design_spec( + {"factors": [{"name": "Agent:Model", "level_type": "llm_config", "levels": [{"model": "gpt-5"}]}]} + ) + + assert spec["factors"][0]["level_type"] == "model_config" + + def test_level_labels_change_the_material_design_because_they_name_cells() -> None: without_labels = {"factors": [{"name": "Agent:System prompt", "levels": ["a", "b"]}], "replicates": 2} with_labels = { diff --git a/tests/test_prompt_preview.py b/tests/test_prompt_preview.py index acceb17..361c5af 100644 --- a/tests/test_prompt_preview.py +++ b/tests/test_prompt_preview.py @@ -183,7 +183,7 @@ async def test_an_unwired_predecessor_previews_as_the_block_it_will_deliver() -> async def test_a_node_that_is_not_an_agent_has_no_prompt_to_preview() -> None: graph = _chain("Draft it.") - graph["nodes"].append({"id": "llm", "type": "llm_anthropic", "data": {"label": "Claude", "config": {}}}) + graph["nodes"].append({"id": "llm", "type": "model_anthropic", "data": {"label": "Claude", "config": {}}}) with pytest.raises(ProtocolValidationError, match="not an agent"): await preview_node_prompt(graph, "llm", owner_id=OWNER_ID) diff --git a/tests/test_prompt_references.py b/tests/test_prompt_references.py index 0ed0a57..3f38691 100644 --- a/tests/test_prompt_references.py +++ b/tests/test_prompt_references.py @@ -220,10 +220,10 @@ def test_connector_nodes_are_out_of_scope() -> None: graph = { "nodes": [ _agent("a"), - {"id": "llm", "type": "llm_anthropic", "data": {"label": "", "config": {}}}, + {"id": "llm", "type": "model_anthropic", "data": {"label": "", "config": {}}}, _agent("b"), ], - "edges": [_edge("a", "b"), _edge("llm", "b", "ai")], + "edges": [_edge("a", "b"), _edge("llm", "b", "model")], } assert referenceable_node_ids(graph, "b") == ["a"] diff --git a/tests/test_protocol_execution.py b/tests/test_protocol_execution.py index dc4a428..a15afa6 100644 --- a/tests/test_protocol_execution.py +++ b/tests/test_protocol_execution.py @@ -31,12 +31,12 @@ ) from asaree.services.protocol_revisions import publish_protocol from asaree.services.protocol_runs import create_protocol_run, request_protocol_run_cancellation -from asaree.services.protocols import create_protocol, delete_protocol +from asaree.services.protocols import create_protocol, delete_protocol, update_protocol def _graph(node_ids: list[str], edges: list[tuple[str, str]]) -> dict: # "step" is a deliberately-unregistered node type -- not "agent" (needs - # an LLM connector) and not "mcp_tool" (now handle-restricted to its own + # a Model connector) and not "mcp_tool" (now handle-restricted to its own # Tool connector, see _MCP_TOOL_NODE_TYPES) -- so these pure DAG-shape # tests (topological order, cycle detection, sink detection) can wire # plain edges freely with zero setup. topological_order only applies @@ -57,13 +57,13 @@ def _edges(*pairs: tuple[str, str]) -> list[dict]: def _llm_node(node_id: str = "llm", config: dict | None = None) -> dict: - # llm_anthropic -- one arbitrary member of the LLM node-type family - # (pe._LLM_NODE_TYPES); which one doesn't matter for these DAG-shape/ + # model_anthropic -- one arbitrary member of the Model node-type family + # (pe._MODEL_NODE_TYPES); which one doesn't matter for these DAG-shape/ # validation tests, only that it's a family member. - return {"id": node_id, "type": "llm_anthropic", "data": {"label": "", "config": config or {}}} + return {"id": node_id, "type": "model_anthropic", "data": {"label": "", "config": config or {}}} -def _llm_edge(source: str, target: str, handle: str = "ai") -> dict: +def _llm_edge(source: str, target: str, handle: str = "model") -> dict: # `handle` is only ever overridden to exercise the pre-rename "llm" # spelling that migration 3f1a7c9b2e04 rewrites -- see # test_legacy_llm_handle_still_resolves. @@ -213,13 +213,78 @@ def _knowledge_edge(source: str, target: str) -> dict: def _agent_with_llm(node_id: str, llm_id: str = "llm") -> tuple[dict, dict]: - """A minimal valid agent + its required LLM connector edge -- the + """A minimal valid agent + its required Model connector edge -- the boilerplate every connector-validation test below needs just to get - past the "every agent needs exactly one AI connection" rule so it can + past the "every agent needs exactly one Model connection" rule so it can test the thing it actually cares about.""" return _node(node_id, "agent"), _llm_edge(llm_id, node_id) +def _sub_agent_edge(child: str, parent: str) -> dict: + return { + "id": f"{child}-{parent}-sub-agent", + "source": child, + "sourceHandle": "sub_agents", + "target": parent, + "targetHandle": "sub_agents", + } + + +def test_sub_agent_is_a_callable_connector_not_a_pipeline_sink() -> None: + parent, parent_model = _agent_with_llm("parent", "parent-model") + child = _node("child", "sub_agent") + graph = { + "nodes": [parent, child, _llm_node("parent-model"), _llm_node("child-model")], + "edges": [parent_model, _llm_edge("child-model", "child"), _sub_agent_edge("child", "parent")], + } + + topological_order(graph) + + assert pe.sink_node_ids(graph) == ["parent"] + assert pe._sub_agent_ids(graph, "parent") == ["child"] + assert pe._can_deliver_communication(graph, "parent", "child") is True + assert pe._can_deliver_communication(graph, "child", "parent") is False + + +def test_sub_agent_cannot_have_two_parents() -> None: + parent_a, parent_a_model = _agent_with_llm("parent-a", "model-a") + parent_b, parent_b_model = _agent_with_llm("parent-b", "model-b") + child = _node("child", "sub_agent") + graph = { + "nodes": [parent_a, parent_b, child, _llm_node("model-a"), _llm_node("model-b"), _llm_node("model-c")], + "edges": [ + parent_a_model, + parent_b_model, + _llm_edge("model-c", "child"), + _sub_agent_edge("child", "parent-a"), + _sub_agent_edge("child", "parent-b"), + ], + } + + with pytest.raises(ProtocolValidationError, match="exactly one parent"): + topological_order(graph) + + +def test_only_active_connected_sub_agents_require_a_model() -> None: + parent, parent_model = _agent_with_llm("parent", "parent-model") + model = _llm_node("parent-model") + orphan = _node("orphan", "sub_agent") + inactive = _node("inactive", "sub_agent") + inactive["data"]["active"] = False + graph = { + "nodes": [parent, model, orphan, inactive], + "edges": [parent_model, _sub_agent_edge("inactive", "parent")], + } + + topological_order(graph) + + active = _node("active", "sub_agent") + graph["nodes"].append(active) + graph["edges"].append(_sub_agent_edge("active", "parent")) + with pytest.raises(ProtocolValidationError, match="must have exactly one Model"): + topological_order(graph) + + def test_linear_order() -> None: order = [n["id"] for n in topological_order(_graph(["a", "b", "c"], [("a", "b"), ("b", "c")]))] assert order == ["a", "b", "c"] @@ -372,10 +437,9 @@ def test_a_reader_downstream_of_a_deactivated_node_sees_that_nodes_name() -> Non def test_build_user_input_cues_dataset_without_dictating_ids() -> None: - # The ids the prompt used to spell out -- experiment_id, cell_label, the - # dataset name -- all reach open_workspace as ambient _meta now. Anything - # the model has to retype is something it can retype wrong, so the prompt - # keeps only the part _meta can't carry: that there IS a dataset waiting. + # Operational ids reach open_workspace as ambient _meta. The dataset's + # descriptive metadata is deliberately visible in the resource catalog so + # the model can decide whether and how to use the resource. agent, agent_llm_edge = _agent_with_llm("a") dataset = _dataset_node(dataset_name="spinal-fusion-v1") graph = {"nodes": [agent, dataset], "edges": [agent_llm_edge, _dataset_edge("dataset1", "a")]} @@ -384,11 +448,43 @@ def test_build_user_input_cues_dataset_without_dictating_ids() -> None: ) assert "Dataset context:" in result assert "open_workspace()" in result - assert "spinal-fusion-v1" not in result + assert "Available datasets:" in result + assert "spinal-fusion-v1" in result assert str(uuid.UUID(int=1)) not in result assert "tier_a__rep_0" not in result +def test_resource_catalog_exposes_meaning_but_not_bodies_or_paths() -> None: + agent, agent_llm_edge = _agent_with_llm("a") + dataset = _dataset_node() + dataset["data"]["config"].update( + description="Postoperative outcomes cohort", target_column="fusion", dictionary_available=True + ) + knowledge = _okf_bundle_node() + knowledge["data"]["config"].update(bundle_label="Spine ontology", bundle_description="Clinical concepts") + script = _script_node(code="SECRET_SCRIPT_BODY") + script["data"]["config"]["description"] = "Compute the validated score" + graph = { + "nodes": [agent, dataset, knowledge, script], + "edges": [ + agent_llm_edge, + _dataset_edge(dataset["id"], "a"), + _knowledge_edge(knowledge["id"], "a"), + _script_edge(script["id"], "a"), + ], + } + + catalog = pe._resource_catalog(graph, "a") + + assert "Postoperative outcomes cohort" in catalog + assert "target=fusion" in catalog + assert "data dictionary available" in catalog + assert "Spine ontology: Clinical concepts" in catalog + assert "scoring-script: Compute the validated score" in catalog + assert "SECRET_SCRIPT_BODY" not in catalog + assert "/home/r/okf/spine" not in catalog + + def test_ambient_meta_carries_every_wired_dataset_name() -> None: agent, agent_llm_edge = _agent_with_llm("a") graph = { @@ -458,6 +554,7 @@ def test_build_user_input_states_the_dataset_is_already_open_when_preseeded() -> def _registration(**overrides: object) -> dict[str, object]: """A split registration as ``fetch_owned_registration`` returns one.""" return { + "description": "A registered test dataset", "target_column": "outcome", "raw_path": "/data/raw.csv", "train_path": "/data/train.parquet", @@ -641,6 +738,7 @@ async def _never(**_kwargs: object) -> None: assert dataset.seeded == () assert ambient["data_path"] == "/data/spine/raw.csv" assert ambient["target_column"] == "outcome" + assert ambient["dataset_mode"] == "raw_unsplit" # And the prompt says so, because a model left to infer it reaches for # open_workspace -- which has nothing to open. @@ -657,22 +755,71 @@ async def _never(**_kwargs: object) -> None: assert "train_test_split" in result -async def test_a_workspace_head_wins_over_an_unsplit_raw_file(monkeypatch: pytest.MonkeyPatch) -> None: - # The raw file is a fallback, never an override: a cell with a workspace has - # already moved past the upload, and a Score step must fit the engineered - # matrix at HEAD rather than the raw CSV. +async def test_an_attached_unsplit_dataset_wins_over_a_stale_workspace_head( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # A workspace survives reruns and may belong to a Dataset connector from an + # older published revision. The current node's explicit unsplit attachment + # must win; only nodes with no Dataset connector inherit upstream HEAD. async def _reg(name: str, owner_id: uuid.UUID) -> dict[str, object]: - return _registration(train_path=None, test_path=None) + return _registration(train_path=None, test_path=None, raw_path="/data/current.csv") monkeypatch.setattr(pe, "fetch_owned_registration", _reg) monkeypatch.setattr(pe, "head_data_locator", lambda wid: ("/ws/v2_fte/train.parquet", "outcome")) + monkeypatch.setattr( + pe, + "slot_data_locators", + lambda wid: { + "dataset:old": {"data_path": "/ws/old.parquet", "target_column": "old_target"}, + "dataset:other": {"data_path": "/ws/other.parquet", "target_column": "other_target"}, + }, + ) agent, agent_llm_edge = _agent_with_llm("a") graph = { "nodes": [agent, _dataset_node(dataset_name="spine-raw")], "edges": [agent_llm_edge, _dataset_edge("dataset1", "a")], } ambient, _dataset = await pe._node_run_context(graph, "a", "exp1/cellA", uuid.UUID(int=7)) - assert ambient["data_path"] == "/ws/v2_fte/train.parquet" + assert ambient["data_path"] == "/data/current.csv" + assert ambient["target_column"] == "outcome" + assert ambient["dataset_mode"] == "raw_unsplit" + assert "data_slots" not in ambient + + +async def test_an_attached_split_dataset_sees_only_its_workspace_slot( + monkeypatch: pytest.MonkeyPatch, +) -> None: + async def _resolved(*_args: object, **_kwargs: object) -> pe.NodeDataset: + return pe.NodeDataset(seeded=(("current", "dataset:current"),)) + + monkeypatch.setattr(pe, "_resolve_node_dataset", _resolved) + monkeypatch.setattr( + pe, + "slot_data_locators", + lambda wid: { + "dataset:current": { + "name": "current", + "data_path": "/ws/current/train.parquet", + "target_column": "outcome", + }, + "dataset:unwired": { + "name": "unwired", + "data_path": "/ws/unwired/train.parquet", + "target_column": "other_target", + }, + }, + ) + agent, agent_llm_edge = _agent_with_llm("a") + graph = { + "nodes": [agent, _dataset_node(dataset_name="current")], + "edges": [agent_llm_edge, _dataset_edge("dataset1", "a")], + } + + ambient, _dataset = await pe._node_run_context(graph, "a", "exp1/cellA", uuid.UUID(int=7)) + + assert ambient["data_path"] == "/ws/current/train.parquet" + assert ambient["target_column"] == "outcome" + assert "data_slots" not in ambient def test_dataset_connector_grants_the_workspace_tools() -> None: @@ -697,6 +844,22 @@ def test_dataset_connector_grants_the_workspace_tools() -> None: assert pe._resolve_dataset_tool_config(bare, "a") == {"server_names": [], "tool_names": []} +def test_dataset_with_dictionary_grants_only_the_dictionary_reader() -> None: + agent, agent_llm_edge = _agent_with_llm("a") + dataset = _dataset_node(dataset_name="spinal-fusion-v1") + dataset["data"]["config"]["dictionary_available"] = True + graph = { + "nodes": [agent, dataset], + "edges": [agent_llm_edge, _dataset_edge("dataset1", "a")], + } + + resolved = pe._resolve_dataset_tool_config(graph, "a") + + assert "asaree-sklearn-eda" in resolved["server_names"] + eda_tools = {name for name in resolved["tool_names"] if name.startswith("asaree-sklearn-eda.")} + assert eda_tools == {"asaree-sklearn-eda.get_data_dictionary"} + + def test_an_unsplit_dataset_grants_the_tools_its_prompt_names() -> None: """The gap the first sequential demo run fell into: an unsplit registration has no workspace, so the Dataset block tells the agent NOT to call @@ -868,15 +1031,15 @@ def test_build_user_input_lists_multiple_bound_scripts() -> None: assert "print('second')" not in result -def test_build_user_input_inlines_script_when_it_could_not_be_bound() -> None: - # No workspace to write it to (an unlinked protocol run): a prompt the - # model can copy from beats no script at all. +def test_build_user_input_does_not_inline_script_when_it_could_not_be_bound() -> None: + # Source code is never prompt content. An unlinked run reports the missing + # materialization context instead of asking the model to retranscribe code. agent, agent_llm_edge = _agent_with_llm("a") script = _script_node(code="print('hello')") graph = {"nodes": [agent, script], "edges": [agent_llm_edge, _script_edge("script1", "a")]} result = pe._build_user_input(agent, graph, {}, script_bound=False) - assert "Script to pass verbatim" in result - assert "print('hello')" in result + assert "no isolated run workspace" in result + assert "print('hello')" not in result def test_build_user_input_omits_script_block_when_unwired() -> None: @@ -985,14 +1148,33 @@ def _locator(workspace_id: str) -> tuple[str, str]: def test_ambient_meta_omits_script_path_without_a_workspace() -> None: - # Nowhere to write it, so no path -- and _build_user_input falls back to - # inlining rather than cueing a file that doesn't exist. + # The low-level helper still requires an explicit materialization surface. agent, agent_llm_edge = _agent_with_llm("a") script = _script_node(code="print('hello')") graph = {"nodes": [agent, script], "edges": [agent_llm_edge, _script_edge("script1", "a")]} assert pe._ambient_meta_for(graph, "a", None) == {} +def test_standalone_run_materializes_script_out_of_band(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(pe, "WORKSPACE_ROOT", str(tmp_path)) + agent, agent_llm_edge = _agent_with_llm("a") + script = _script_node(code="print('hello')") + graph = {"nodes": [agent, script], "edges": [agent_llm_edge, _script_edge("script1", "a")]} + + meta = pe._ambient_meta_for(graph, "a", script_workspace_id="_protocol_runs/run-1") + + path = Path(meta["script_path"]) + assert path.read_text() == "print('hello')" + assert "_protocol_runs/run-1" in str(path) + pe._cleanup_adhoc_scripts(meta) + assert not path.exists() + + +def test_standalone_agents_get_distinct_script_directories() -> None: + run_id = uuid.uuid4() + assert pe._script_workspace_id(None, run_id, "agent-a") != pe._script_workspace_id(None, run_id, "agent-b") + + # --- _run_gated_worker (mocked -- no real LLM calls) ------------------------- @@ -1528,6 +1710,10 @@ async def test_plan_cell_runs_runs_an_obsolete_completed_replicate(owner_id: uui protocol_revision_id=old_revision.id, ) completed_run.status = "completed" + new_graph = _graph(["a", "b"], [("a", "b")]) + new_graph["nodes"][0]["data"]["config"] = {"revision": "new"} + protocol = await update_protocol(db, protocol_id, fields={"graph": new_graph}) + assert protocol is not None new_revision = await publish_protocol(db, protocol) await db.flush() await upsert_replicate( @@ -1544,7 +1730,7 @@ async def test_plan_cell_runs_runs_an_obsolete_completed_replicate(owner_id: uui protocol_id=protocol_id, experiment_id=experiment_id, owner_id=owner_id, - graph=graph, + graph=new_graph, protocol_revision_id=new_revision.id, ) assert [run.replicate_label for run in runs] == ["cell-obsolete"] @@ -1693,7 +1879,7 @@ async def test_run_protocol_substitutes_factor_and_writes_back_to_cell( ) -> None: """End-to-end (minus the actual LLM call): a run created with cell_label/factor_values set gets the substituted value resolvable via - the worker's LLM connector, and the sink node's output lands on the + the worker's Model connector, and the sink node's output lands on the right replicate via the real upsert_replicate -- proves apply_factor_bindings is actually wired into run_protocol, not just correct in isolation. Model config lives on the connected `llm` node now, not the agent's own @@ -1702,7 +1888,7 @@ async def test_run_protocol_substitutes_factor_and_writes_back_to_cell( received_workspace_ids = [] async def fake_run_agent_node(node, *, graph, workspace_id=None, **kwargs): - received_configs.append(pe._resolve_llm_config(graph, node["id"])) + received_configs.append(pe._resolve_model_config(graph, node["id"])) received_workspace_ids.append(workspace_id) return f"output for {node['id']}", None, None, None @@ -1720,7 +1906,7 @@ async def fake_run_agent_node(node, *, graph, workspace_id=None, **kwargs): "nodes": [ { "id": "llm1", - "type": "llm_anthropic", + "type": "model_anthropic", "data": { "config": {"temperature": 0.9}, "factor_bindings": {"config.temperature": "Temperature"}, @@ -1728,7 +1914,7 @@ async def fake_run_agent_node(node, *, graph, workspace_id=None, **kwargs): }, {"id": "worker", "type": "agent", "data": {"config": {}}}, ], - "edges": [{"id": "llm1-worker", "source": "llm1", "target": "worker", "targetHandle": "ai"}], + "edges": [{"id": "llm1-worker", "source": "llm1", "target": "worker", "targetHandle": "model"}], }, ) protocol_id = protocol.id @@ -2171,12 +2357,12 @@ async def test_monitor_protocol_run_sets_event_once_cancellation_requested(owner await delete_protocol(db, protocol_id) # cascades the created ProtocolRun -# --- LLM / Tool / Memory connector validation (pure) ------------------------- +# --- Model / Tool / Memory connector validation (pure) ------------------------- def test_agent_missing_llm_connection_raises() -> None: graph = {"nodes": [_node("a", "agent")], "edges": []} - with pytest.raises(ProtocolValidationError, match="exactly one AI connection"): + with pytest.raises(ProtocolValidationError, match="exactly one Model connection"): topological_order(graph) @@ -2186,7 +2372,7 @@ def test_agent_duplicate_llm_connection_raises() -> None: "nodes": [llm1, llm2, _node("a", "agent")], "edges": [_llm_edge("llm1", "a"), _llm_edge("llm2", "a")], } - with pytest.raises(ProtocolValidationError, match="exactly one AI connection"): + with pytest.raises(ProtocolValidationError, match="exactly one Model connection"): topological_order(graph) @@ -2197,33 +2383,34 @@ def test_critic_gate_missing_llm_connection_raises() -> None: "nodes": [llm, worker, _node("g1", "critic_gate")], "edges": [worker_llm_edge, {"id": "w1-g1", "source": "w1", "target": "g1"}], } - with pytest.raises(ProtocolValidationError, match="exactly one AI connection"): + with pytest.raises(ProtocolValidationError, match="exactly one Model connection"): topological_order(graph) -def test_legacy_llm_handle_still_resolves() -> None: - # The AI connector's handle id was "llm" before it was renamed to "ai" - # (migration 3f1a7c9b2e04 rewrites stored graphs). An un-migrated edge -- +@pytest.mark.parametrize("legacy_handle", ["ai", "llm"]) +def test_legacy_model_handle_still_resolves(legacy_handle: str) -> None: + # The Model connector's handles were "llm" and then "ai" before "model". + # Data migrations rewrite stored graphs, but an un-migrated edge -- # or one autosaved by a browser tab still running the pre-rename JS -- # must resolve identically: same wiring, same model config, no "exactly - # one AI connection" error from the edge being read as a main pipeline + # one Model connection" error from the edge being read as a main pipeline # edge instead. llm = _llm_node(config={"provider": "anthropic", "model": "claude-sonnet-4-5"}) agent = _node("a", "agent") graph = { "nodes": [llm, agent], - "edges": [_llm_edge("llm", "a", handle="llm")], + "edges": [_llm_edge("llm", "a", handle=legacy_handle)], } assert [n["id"] for n in topological_order(graph)] == ["llm", "a"] - assert pe._resolve_llm_config(graph, "a")["model"] == "claude-sonnet-4-5" + assert pe._resolve_model_config(graph, "a")["model"] == "claude-sonnet-4-5" def test_llm_connection_from_non_llm_source_raises() -> None: graph = { "nodes": [_node("t1", "step"), _node("a", "agent")], - "edges": [{"id": "t1-a-ai", "source": "t1", "target": "a", "targetHandle": "ai"}], + "edges": [{"id": "t1-a-ai", "source": "t1", "target": "a", "targetHandle": "model"}], } - with pytest.raises(ProtocolValidationError, match="must come from an AI node"): + with pytest.raises(ProtocolValidationError, match="must come from a Model node"): topological_order(graph) @@ -2305,7 +2492,7 @@ def test_llm_node_with_plain_outgoing_edge_raises() -> None: "nodes": [llm, agent, _node("b", "agent")], "edges": [agent_llm_edge, {"id": "llm-b", "source": "llm", "target": "b"}], } - with pytest.raises(ProtocolValidationError, match="AI node .* can only connect to a node's AI slot"): + with pytest.raises(ProtocolValidationError, match="Model node .* can only connect to a node's Model slot"): topological_order(graph) @@ -2648,6 +2835,10 @@ def test_resolve_knowledge_config_namespaces_tool_names() -> None: "okf-bundle-spine-abc12345.list_concepts", "okf-bundle-spine-abc12345.read_concept", ], + "tool_descriptions": { + "okf-bundle-spine-abc12345.list_concepts": "Knowledge source: spine.", + "okf-bundle-spine-abc12345.read_concept": "Knowledge source: spine.", + }, } @@ -2718,6 +2909,11 @@ def test_resolve_knowledge_config_mixes_bundles_and_documents() -> None: "okf-bundle-spine-abc12345.read_concept", "okf-doc-spinal-cord-def45678.read_concept", ], + "tool_descriptions": { + "okf-bundle-spine-abc12345.list_concepts": "Knowledge source: spine.", + "okf-bundle-spine-abc12345.read_concept": "Knowledge source: spine.", + "okf-doc-spinal-cord-def45678.read_concept": "Knowledge source: Spinal cord.", + }, } @@ -2825,19 +3021,21 @@ def test_valid_llm_tool_memory_wiring_passes() -> None: assert set(order) == {"llm", "a", "tool1", "memory", "pattern"} -def test_llm_connection_accepts_any_provider_node_type() -> None: - # Membership, not equality -- llm_openai/llm_azure_foundry are just as - # valid an LLM connector source as _llm_node()'s default llm_anthropic. - agent1, agent1_llm_edge = _agent_with_llm("a1", llm_id="openai") - agent2, agent2_llm_edge = _agent_with_llm("a2", llm_id="foundry") - openai_llm = {"id": "openai", "type": "llm_openai", "data": {"label": "", "config": {}}} - foundry_llm = {"id": "foundry", "type": "llm_azure_foundry", "data": {"label": "", "config": {}}} +def test_model_connection_accepts_every_provider_node_type() -> None: + providers = ["anthropic", "openai", "azure_foundry", "openrouter", "local"] + agents_and_edges = [_agent_with_llm(f"a{index}", llm_id=provider) for index, provider in enumerate(providers)] + agents = [agent for agent, _edge in agents_and_edges] + edges = [edge for _agent, edge in agents_and_edges] + models = [ + {"id": provider, "type": f"model_{provider}", "data": {"label": "", "config": {}}} + for provider in providers + ] graph = { - "nodes": [agent1, agent2, openai_llm, foundry_llm], - "edges": [agent1_llm_edge, agent2_llm_edge], + "nodes": [*agents, *models], + "edges": edges, } order = [n["id"] for n in topological_order(graph)] - assert set(order) == {"a1", "a2", "openai", "foundry"} + assert set(order) == {*(agent["id"] for agent in agents), *providers} def test_architectural_pattern_connection_accepts_any_pattern_node_type() -> None: @@ -2852,23 +3050,23 @@ def test_architectural_pattern_connection_accepts_any_pattern_node_type() -> Non assert set(order) == {"llm", "a", "baseline"} -# --- LLM / Tool connector resolution (pure) ----------------------------------- +# --- Model / Tool connector resolution (pure) ----------------------------------- -def test_resolve_llm_config_returns_connected_node_config() -> None: +def test_resolve_model_config_returns_connected_node_config() -> None: llm = _llm_node(config={"provider": "anthropic", "model": "claude-sonnet-5", "temperature": 0.5}) agent, agent_llm_edge = _agent_with_llm("a") graph = {"nodes": [llm, agent], "edges": [agent_llm_edge]} - assert pe._resolve_llm_config(graph, "a") == { + assert pe._resolve_model_config(graph, "a") == { "provider": "anthropic", "model": "claude-sonnet-5", "temperature": 0.5, } -def test_resolve_llm_config_empty_when_unconnected() -> None: +def test_resolve_model_config_empty_when_unconnected() -> None: graph = {"nodes": [_node("a", "agent")], "edges": []} - assert pe._resolve_llm_config(graph, "a") == {} + assert pe._resolve_model_config(graph, "a") == {} def test_resolve_dataset_configs_returns_connected_node_config() -> None: @@ -3395,11 +3593,11 @@ def test_sequential_rejects_a_loop() -> None: def test_sequential_ignores_connector_fan_in() -> None: - """One LLM node feeding every agent in the chain is the normal shape. It is + """One Model node feeding every agent in the chain is the normal shape. It is a fan-in on the graph and must not read as one on the chain.""" graph = _chain_graph("a", "b", "c") - graph["nodes"] = [n for n in graph["nodes"] if n["type"] != "llm_anthropic"] + [_llm_node("shared")] - graph["edges"] = [e for e in graph["edges"] if e.get("targetHandle") != "ai"] + graph["nodes"] = [n for n in graph["nodes"] if n["type"] != "model_anthropic"] + [_llm_node("shared")] + graph["edges"] = [e for e in graph["edges"] if e.get("targetHandle") != "model"] graph["edges"] += [_llm_edge("shared", a) for a in ("a", "b", "c")] validate_coordination_strategy(_SEQUENTIAL, graph=graph) @@ -3907,7 +4105,7 @@ def test_validate_single_node_runnable_missing_node_raises() -> None: def test_validate_single_node_runnable_rejects_non_agent_type() -> None: node = _node("g1", "critic_gate") graph = {"nodes": [node], "edges": []} - with pytest.raises(ProtocolValidationError, match="Only Agent nodes"): + with pytest.raises(ProtocolValidationError, match="Only Agent and Sub-Agent nodes"): pe.validate_single_node_runnable(graph, "g1") @@ -3921,7 +4119,7 @@ def test_validate_single_node_runnable_rejects_a_node_with_upstream_input() -> N def test_validate_single_node_runnable_rejects_zero_llm_connections() -> None: graph = {"nodes": [_node("a", "agent")], "edges": []} - with pytest.raises(ProtocolValidationError, match="must have exactly one AI connection"): + with pytest.raises(ProtocolValidationError, match="must have exactly one Model connection"): pe.validate_single_node_runnable(graph, "a") @@ -3929,7 +4127,7 @@ def test_validate_single_node_runnable_rejects_llm_edge_from_wrong_node_type() - agent = _node("a", "agent") not_an_llm = _node("x", "agent") graph = {"nodes": [agent, not_an_llm], "edges": [_llm_edge("x", "a")]} - with pytest.raises(ProtocolValidationError, match="must come from an AI node"): + with pytest.raises(ProtocolValidationError, match="must come from a Model node"): pe.validate_single_node_runnable(graph, "a") @@ -3976,8 +4174,8 @@ def test_validate_conversation_entry_rejects_a_peer_with_no_model() -> None: """A peer's own wiring is checked too: it will really run, and finding out mid-conversation costs the user a run they already paid for.""" graph = _conversation_graph("a", "b") - graph["edges"] = [e for e in graph["edges"] if e.get("target") != "b" or e.get("targetHandle") != "ai"] - with pytest.raises(ProtocolValidationError, match="exactly one AI connection"): + graph["edges"] = [e for e in graph["edges"] if e.get("target") != "b" or e.get("targetHandle") != "model"] + with pytest.raises(ProtocolValidationError, match="exactly one Model connection"): pe.validate_conversation_entry(graph, "a") @@ -4001,7 +4199,7 @@ async def test_run_single_node_ignores_an_unrelated_broken_sibling_node( ) -> None: """The whole point of a narrower, per-node check: a single-node Play run must not fail because some OTHER node elsewhere in the same graph is - unrelated and broken (e.g. missing its own LLM connector) -- only + unrelated and broken (e.g. missing its own Model connector) -- only topological_order's full-graph walk cares about that.""" async def fake_run_agent_node(node, *, user_input, **_kwargs): @@ -4011,7 +4209,7 @@ async def fake_run_agent_node(node, *, user_input, **_kwargs): target, target_llm_edge = _agent_with_llm("target") target["data"]["config"] = {"prompt": "do the one thing", "goal": ""} - broken_sibling = _node("broken", "agent") # no LLM connector at all + broken_sibling = _node("broken", "agent") # no Model connector at all graph = {"nodes": [target, broken_sibling, _llm_node()], "edges": [target_llm_edge]} diff --git a/tests/test_protocol_graph_schema.py b/tests/test_protocol_graph_schema.py new file mode 100644 index 0000000..0c08c3b --- /dev/null +++ b/tests/test_protocol_graph_schema.py @@ -0,0 +1,119 @@ +from __future__ import annotations + +from copy import deepcopy + +from asaree.services.protocol_graph_schema import functional_protocol_graph_hash, normalize_protocol_graph + + +def test_normalize_protocol_graph_rewrites_only_model_schema_fields() -> None: + graph = { + "nodes": [ + { + "id": "llm_openai-user-id", + "type": "llm_openai", + "data": {"label": "AI node", "config": {"prompt": "keep llm_openai and ai verbatim"}}, + } + ], + "edges": [ + { + "id": "ai-edge-id", + "source": "llm_openai-user-id", + "target": "agent", + "sourceHandle": "llm", + "targetHandle": "ai", + } + ], + } + + normalized = normalize_protocol_graph(graph) + + assert normalized["nodes"][0] == { + "id": "llm_openai-user-id", + "type": "model_openai", + "data": {"label": "AI node", "config": {"prompt": "keep llm_openai and ai verbatim"}}, + } + assert normalized["edges"][0] == { + "id": "ai-edge-id", + "source": "llm_openai-user-id", + "target": "agent", + "sourceHandle": "model", + "targetHandle": "model", + } + assert graph["nodes"][0]["type"] == "llm_openai" + assert graph["edges"][0]["targetHandle"] == "ai" + + +def test_normalize_protocol_graph_preserves_canonical_graph() -> None: + graph = { + "nodes": [{"id": "model", "type": "model_local", "data": {}}], + "edges": [{"source": "model", "target": "agent", "targetHandle": "model"}], + } + + assert normalize_protocol_graph(graph) == graph + + +def test_functional_hash_ignores_canvas_only_state_and_collection_order() -> None: + first = { + "nodes": [ + { + "id": "agent", + "type": "agent", + "position": {"x": 10, "y": 20}, + "selected": True, + "measured": {"width": 288, "height": 180}, + "data": {"label": "Agent", "config": {"system_prompt": "Answer carefully"}}, + }, + {"id": "model", "type": "model_openai", "position": {"x": 30, "y": 40}, "data": {}}, + ], + "edges": [ + { + "id": "edge-one", + "source": "model", + "target": "agent", + "sourceHandle": "model", + "targetHandle": "model", + "selected": True, + } + ], + "viewport": {"x": 100, "y": 200, "zoom": 1.5}, + } + second = { + "nodes": [ + {"id": "model", "type": "model_openai", "position": {"x": 900, "y": 800}, "data": {}}, + { + "id": "agent", + "type": "agent", + "position": {"x": -10, "y": -20}, + "dragging": True, + "data": {"config": {"system_prompt": "Answer carefully"}, "label": "Agent"}, + }, + ], + "edges": [ + { + "id": "replacement-ui-id", + "source": "model", + "target": "agent", + "sourceHandle": "model", + "targetHandle": "model", + } + ], + } + + assert functional_protocol_graph_hash(first) == functional_protocol_graph_hash(second) + + +def test_functional_hash_changes_for_node_configuration_and_edge_reconnection() -> None: + graph = { + "nodes": [ + {"id": "agent-a", "type": "agent", "position": {"x": 0, "y": 0}, "data": {"config": {}}}, + {"id": "agent-b", "type": "agent", "position": {"x": 0, "y": 0}, "data": {"config": {}}}, + ], + "edges": [{"id": "edge", "source": "agent-a", "target": "agent-b"}], + } + configured = deepcopy(graph) + configured["nodes"][0]["data"]["config"]["system_prompt"] = "Changed" + reconnected = deepcopy(graph) + reconnected["edges"][0]["source"] = "agent-b" + + assert functional_protocol_graph_hash(configured) != functional_protocol_graph_hash(graph) + assert functional_protocol_graph_hash(reconnected) != functional_protocol_graph_hash(graph) diff --git a/tests/test_protocol_runs.py b/tests/test_protocol_runs.py index 1b02769..a2e35ec 100644 --- a/tests/test_protocol_runs.py +++ b/tests/test_protocol_runs.py @@ -616,6 +616,17 @@ async def test_list_experiment_trials_marks_runs_obsolete_after_a_new_canvas_pub fields={"run_id": legacy_run.id}, ) + protocol = await update_protocol( + db, + protocol_id, + fields={ + "graph": { + "nodes": [{"id": "new-step", "type": "step", "data": {}}], + "edges": [], + } + }, + ) + assert protocol is not None second_revision = await publish_protocol(db, protocol) assert second_revision.id != first_revision.id diff --git a/tests/test_protocols.py b/tests/test_protocols.py index 0ed46fe..743a6ce 100644 --- a/tests/test_protocols.py +++ b/tests/test_protocols.py @@ -12,13 +12,16 @@ import uuid from collections.abc import AsyncIterator +from copy import deepcopy import pytest import pytest_asyncio +from sqlalchemy import func, select import asaree.models.dataset # noqa: F401 -- registers registered_datasets for research_experiments' FK import asaree.models.experiment # noqa: F401 -- registers research_experiments for the FK from asaree.models.database import dispose_engine, get_session +from asaree.models.protocol_revision import ProtocolRevision from asaree.models.user import User from asaree.services.experiments import create_experiment from asaree.services.protocol_revisions import get_published_revision, is_draft_published, publish_protocol @@ -121,6 +124,44 @@ async def test_publish_freezes_the_draft_and_advances_the_revision(owner_id: uui await delete_protocol(db, protocol.id) +async def test_publish_ignores_layout_changes_and_does_not_duplicate_revision(owner_id: uuid.UUID) -> None: + graph = { + "nodes": [ + { + "id": "agent", + "type": "agent", + "position": {"x": 0, "y": 0}, + "data": {"config": {"system_prompt": "Original"}}, + } + ], + "edges": [], + } + async with get_session() as db: + protocol = await create_protocol(db, name="meaningful-publish", owner_id=owner_id, graph=graph) + first = await publish_protocol(db, protocol) + + moved = deepcopy(graph) + moved["nodes"][0]["position"] = {"x": 500, "y": 600} + await update_protocol(db, protocol.id, fields={"graph": moved}) + assert is_draft_published(protocol, first) is True + assert (await publish_protocol(db, protocol)).id == first.id + count = ( + await db.execute( + select(func.count()).select_from(ProtocolRevision).where(ProtocolRevision.protocol_id == protocol.id) + ) + ).scalar_one() + assert count == 1 + + changed = deepcopy(moved) + changed["nodes"][0]["data"]["config"]["system_prompt"] = "Changed" + await update_protocol(db, protocol.id, fields={"graph": changed}) + assert is_draft_published(protocol, first) is False + + await update_protocol(db, protocol.id, fields={"graph": moved}) + assert is_draft_published(protocol, first) is True + await delete_protocol(db, protocol.id) + + async def test_rename_sync_follows_the_experiment_name(owner_id: uuid.UUID) -> None: """An auto-named protocol tracks its experiment's name; a hand-named one (and a protocol on another experiment) is left alone.""" diff --git a/tests/test_reported_metrics.py b/tests/test_reported_metrics.py index de6fd9a..0ca8267 100644 --- a/tests/test_reported_metrics.py +++ b/tests/test_reported_metrics.py @@ -216,6 +216,31 @@ async def test_agent_output_report_is_unavailable_when_agent_did_not_complete() assert result.observations[0].value is None +@pytest.mark.asyncio +async def test_sub_agent_metric_keeps_latest_success_after_a_failed_retry() -> None: + run = SimpleNamespace( + id=uuid4(), + replicate_result_id=None, + node_runs={ + "worker": { + "status": "failed", + "output_text": None, + "last_successful_output_text": '{"score":0.91}', + "last_successful_run_id": str(uuid4()), + } + }, + ) + + result = await collect_reported_metrics( + run, + _plan("asaree.agent_output", {"agent_node_id": "worker"}), + {"nodes": [{"id": "worker", "type": "sub_agent", "data": {}}], "edges": []}, + ) + + assert result.observations[0].status == "measured" + assert result.observations[0].value == '{"score":0.91}' + + @pytest.mark.asyncio async def test_agent_output_plan_requires_an_active_agent() -> None: plan = _plan("asaree.agent_output", {"agent_node_id": "agent"}) @@ -232,6 +257,12 @@ async def test_agent_output_plan_requires_an_active_agent() -> None: assert valid.valid assert [issue.code for issue in disabled.issues] == ["agent_output_agent_disabled"] + sub_agent = await validate_reported_measurement_plan( + plan, + {"nodes": [{"id": "agent", "type": "sub_agent", "data": {}}]}, + ) + assert sub_agent.valid + async def _async_value(value): return value diff --git a/tests/test_run_tools.py b/tests/test_run_tools.py index 8dca16d..0c4cc53 100644 --- a/tests/test_run_tools.py +++ b/tests/test_run_tools.py @@ -3,6 +3,7 @@ from __future__ import annotations +import uuid from typing import Any import pytest @@ -15,11 +16,13 @@ class _FakeRegistry: def __init__(self, servers: dict[str, list[str]]) -> None: self.servers = servers - def get_all_tools(self) -> list[dict[str, Any]]: + def get_all_tools(self, *, owner_id: uuid.UUID | None = None) -> list[dict[str, Any]]: + assert owner_id == _OWNER_ID return [ { "name": f"{server}.{tool}", "server": server, + "server_id": str(uuid.uuid5(uuid.NAMESPACE_DNS, server)), "tool_name": tool, "description": f"{tool} on {server}", "input_schema": {"type": "object", "properties": {}}, @@ -30,8 +33,14 @@ def get_all_tools(self) -> list[dict[str, Any]]: class _FakeAgent: - def __init__(self, tool_names: list[str] | None) -> None: - self.tool_config_data = None if tool_names is None else {"tool_names": tool_names} + def __init__(self, tool_names: list[str] | None, tool_descriptions: dict[str, str] | None = None) -> None: + self.tool_config_data = ( + None if tool_names is None else {"tool_names": tool_names, "tool_descriptions": tool_descriptions or {}} + ) + self.owner_id = _OWNER_ID + + +_OWNER_ID = uuid.uuid4() @pytest.fixture @@ -59,6 +68,14 @@ def test_admits_only_named_tools(registry: _FakeRegistry) -> None: assert tools[0]["tool_name"] == "describe_dataset" +def test_resource_description_is_prefixed_to_the_live_tool_description(registry: _FakeRegistry) -> None: + name = "okf-doc-hair-concentrations.search_concepts" + [tool] = run_tools.gather_tools(_FakeAgent([name], {name: "Knowledge source: Hair concentrations."})) + + assert tool["description"].startswith("Knowledge source: Hair concentrations.\n\n") + assert tool["description"].endswith("search_concepts on okf-doc-hair-concentrations") + + def test_colliding_bare_name_is_namespaced(registry: _FakeRegistry) -> None: """Two servers exposing ``ping`` must not both bind a tool called ``ping`` -- the provider rejects duplicate names and the run dies before its first @@ -88,9 +105,9 @@ def test_colliding_bare_name_is_namespaced(registry: _FakeRegistry) -> None: assert {t["name"] for t in tools} == set(agent.tool_config_data["tool_names"]) -def test_collision_is_registry_wide_not_allowlist_wide(registry: _FakeRegistry) -> None: +def test_collision_is_owner_catalog_wide_not_allowlist_wide(registry: _FakeRegistry) -> None: """Only one server's ``ping`` is granted, but the other is still connected - -- ``lookup_tool``'s bare-name index spans the whole registry, so a bare - ``ping`` could resolve to the server this run was never granted.""" + -- bare ``ping`` would still be ambiguous among servers visible to this + owner even though only one is granted to this run.""" tools = run_tools.gather_tools(_FakeAgent(["scikit-learn-mcp.ping"])) assert [t["tool_name"] for t in tools] == ["scikit-learn-mcp.ping"] diff --git a/tests/test_script_server.py b/tests/test_script_server.py index 8f64c31..29f8f69 100644 --- a/tests/test_script_server.py +++ b/tests/test_script_server.py @@ -122,7 +122,163 @@ def test_credentials_do_not_reach_the_script(monkeypatch: pytest.MonkeyPatch, tm monkeypatch.setenv("ASAREE_INTERNAL_MCP_API_KEY", "super-secret") monkeypatch.setenv("ASAREE_PRODUCT_DATABASE_URL", "postgresql://user:pw@host/db") out = _run(ctx=_wire(tmp_path, "import os\nprint(sorted(k for k in os.environ if 'ASAREE' in k))")) - assert out["stdout"].strip() == "[]" + assert out["stdout"].strip() == "['ASAREE_RUN_CONTEXT']" + + +def test_unsplit_dataset_reaches_script_through_runtime_context(tmp_path: Path) -> None: + raw = tmp_path / "cohort.csv" + raw.write_text("record_id,outcome\n1,1\n") + script = tmp_path / "wired.py" + script.write_text( + "import json\n" + "from asaree.script_context import training_input\n" + "item = training_input()\n" + "print(json.dumps({'name': item.name, 'path': str(item.path), " + "'target': item.target_column, 'mode': item.mode}))\n" + ) + ctx = _FakeCtx( + { + "motoro.ambient.script_path": str(script), + "motoro.ambient.dataset_names": ["runtime-cohort"], + "motoro.ambient.data_path": str(raw), + "motoro.ambient.target_column": "outcome", + } + ) + + out = _run(ctx=ctx) + + assert out["exit_code"] == 0 + assert json.loads(out["stdout"]) == { + "name": "runtime-cohort", + "path": str(raw), + "target": "outcome", + "mode": "raw_unsplit", + } + + +def test_legacy_workspace_script_profiles_unsplit_input_without_creating_a_workspace( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + raw = tmp_path / "cohort.csv" + raw.write_text("record_id,outcome\n1,1\n") + workspace_root = tmp_path / "workspaces" + workspace_dir = workspace_root / "exp1" / "cellA" + workspace_dir.mkdir(parents=True) + monkeypatch.setenv("ASAREE_DATASET_WORKSPACE_DIR", str(workspace_root)) + script = tmp_path / "profile-data.py" + script.write_text( + "import json\n" + "from pathlib import Path\n" + "state = json.loads(Path('state.json').read_text())\n" + "raw = next(v for v in state['versions'] if v['id'] == 'v0_raw')\n" + "assert 'test' not in raw\n" + "output = Path('profile-result.json').resolve()\n" + "output.write_text(json.dumps({'source': raw['train'], 'target': state['target_column']}))\n" + "print(json.dumps({'output': str(output), 'source': raw['train']}))\n" + ) + ctx = _FakeCtx( + { + "motoro.ambient.script_path": str(script), + "motoro.workspace_id": "exp1/cellA", + "motoro.ambient.dataset_names": ["runtime-cohort"], + "motoro.ambient.data_path": str(raw), + "motoro.ambient.target_column": "outcome", + } + ) + + out = _run(ctx=ctx) + + assert out["exit_code"] == 0 + payload = json.loads(out["stdout"]) + artifact = Path(payload["output"]) + assert payload["source"] == str(raw) + assert artifact.is_file() + assert json.loads(artifact.read_text()) == {"source": str(raw), "target": "outcome"} + assert not (artifact.parent / "state.json").exists() + assert not (workspace_dir / "state.json").exists() + + +def test_workspace_context_exposes_raw_train_not_head_or_test( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + raw_train = tmp_path / "train.parquet" + raw_train.touch() + monkeypatch.setattr( + ss, + "raw_training_data_locators", + lambda _workspace_id: { + "dataset:cohort": { + "name": "cohort", + "data_path": str(raw_train), + "target_column": "outcome", + } + }, + ) + manifest = ss._runtime_manifest( + _FakeCtx( + { + "motoro.ambient.dataset_names": ["cohort"], + "motoro.ambient.data_path": "/workspace/v2_fte/train.parquet", + "motoro.ambient.target_column": "outcome", + } + ), + "exp/cell", + ) + + assert manifest == { + "schema_version": 1, + "training_inputs": [ + { + "name": "cohort", + "path": str(raw_train), + "target_column": "outcome", + "mode": "workspace", + "slot": "dataset:cohort", + "workspace_version": "v0_raw", + } + ], + } + + +def test_explicit_unsplit_context_ignores_a_stale_workspace( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + current = tmp_path / "current.csv" + current.touch() + monkeypatch.setattr( + ss, + "raw_training_data_locators", + lambda _workspace_id: { + "dataset:old": { + "name": "old", + "data_path": "/workspace/old/v0_raw/train.parquet", + "target_column": "old_target", + } + }, + ) + + manifest = ss._runtime_manifest( + _FakeCtx( + { + "motoro.ambient.dataset_names": ["current"], + "motoro.ambient.dataset_mode": "raw_unsplit", + "motoro.ambient.data_path": str(current), + "motoro.ambient.target_column": "outcome", + } + ), + "exp/cell", + ) + + assert manifest["training_inputs"] == [ + { + "name": "current", + "path": str(current), + "target_column": "outcome", + "mode": "raw_unsplit", + "slot": None, + "workspace_version": None, + } + ] def test_runs_in_the_cells_workspace_when_there_is_one(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: diff --git a/tests/test_skill_sources.py b/tests/test_skill_sources.py index 77a4312..7a0ad4a 100644 --- a/tests/test_skill_sources.py +++ b/tests/test_skill_sources.py @@ -118,24 +118,25 @@ def test_reads_files_relative_to_the_repo_root() -> None: assert sorted(files) == ["SKILL.md", "docs/REF.md"] -def test_drops_files_core_would_refuse() -> None: - # Scripts, binaries and dotfiles are dropped rather than raised over: a - # repository is full of them and none is the user's mistake. The rule is - # core's validate_bundle_path, not a second copy of it here. +def test_keeps_scripts_but_drops_hidden_files() -> None: + # Portable skill resources include scripts. Repository housekeeping stays + # excluded and never becomes part of the installed bundle. files = ss._read_archive( make_archive({"SKILL.md": SKILL, "scripts/run.py": "print(1)", ".github/workflows/ci.yml": "on: push"}) ) - assert sorted(files) == ["SKILL.md"] + assert sorted(files) == ["SKILL.md", "scripts/run.py"] -def test_drops_files_that_are_not_utf8() -> None: +def test_keeps_files_that_are_not_utf8() -> None: buffer = io.BytesIO() with tarfile.open(fileobj=buffer, mode="w:gz") as archive: for name, raw in (("repo-main/SKILL.md", SKILL.encode()), ("repo-main/logo.md", b"\xff\xfe\x00")): info = tarfile.TarInfo(name=name) info.size = len(raw) archive.addfile(info, io.BytesIO(raw)) - assert sorted(ss._read_archive(buffer.getvalue())) == ["SKILL.md"] + files = ss._read_archive(buffer.getvalue()) + assert sorted(files) == ["SKILL.md", "logo.md"] + assert files["logo.md"] == b"\xff\xfe\x00" def test_ignores_symlinks_and_other_non_files() -> None: @@ -198,7 +199,7 @@ def test_a_bundle_is_re_rooted_at_the_skill_directory() -> None: files = ss._read_archive( make_archive({"skills/foo/SKILL.md": SKILL, "skills/foo/references/x.md": "ref", "README.md": "hi"}) ) - assert ss._bundle_at(files, "skills/foo") == [("SKILL.md", SKILL), ("references/x.md", "ref")] + assert ss._bundle_at(files, "skills/foo") == [("SKILL.md", SKILL.encode()), ("references/x.md", b"ref")] def test_a_skill_does_not_swallow_a_nested_skill() -> None: @@ -209,4 +210,4 @@ def test_a_skill_does_not_swallow_a_nested_skill() -> None: files = ss._read_archive(make_archive({"SKILL.md": SKILL, "NOTES.md": "notes", "skills/foo/SKILL.md": SKILL})) directories = ss._skill_dirs(files, "") assert directories == ["", "skills/foo"] - assert ss._bundle_at(files, "", directories) == [("NOTES.md", "notes"), ("SKILL.md", SKILL)] + assert ss._bundle_at(files, "", directories) == [("NOTES.md", b"notes"), ("SKILL.md", SKILL.encode())] diff --git a/tests/test_spinal_compat.py b/tests/test_spinal_compat.py index ed12a18..18f8708 100644 --- a/tests/test_spinal_compat.py +++ b/tests/test_spinal_compat.py @@ -120,7 +120,7 @@ def test_fixture_is_the_published_protocol(graph: dict[str, Any]) -> None: assert by_type["critic_gate"] == 4 assert by_type["dataset"] == 1 assert by_type["script"] == 1 - assert by_type["llm_azure_foundry"] == 1 + assert by_type["model_azure_foundry"] == 1 # -- validation ---------------------------------------------------------- @@ -298,7 +298,9 @@ def _assert_golden(graph: dict[str, Any]) -> None: # derived from the topology; both are withdrawn, so there is nothing left # here for the walk to add. actual = _prompt(graph, _GOLDEN_NODE_ID, node_runs=_GOLDEN_NODE_RUNS) - assert actual == golden.read_text() + # The fixture is an ordinary text file and may end with its conventional + # storage newline; that newline is not part of the assembled prompt. + assert actual == golden.read_text().removesuffix("\n") def test_the_assembled_prompt_matches_its_snapshot(graph: dict[str, Any]) -> None: @@ -335,15 +337,18 @@ def test_a_deactivated_node_passes_its_input_through_verbatim(graph: dict[str, A assert pe._upstream_output_text(graph, fte_id, node_runs) == "DC accepted v1_dc." -def test_the_score_agents_script_is_inlined_when_no_workspace_exists(graph: dict[str, Any]) -> None: - """SF-Score has the Script node wired. With a workspace the code reaches it - as an ambient ``script_path``; without one it is inlined in the prompt, and - that fallback is what an unlinked run still relies on.""" +def test_the_score_agents_script_source_stays_out_of_an_unlinked_prompt(graph: dict[str, Any]) -> None: + """SF-Score has the Script node wired. Production runs materialize its + source into either the experiment workspace or an isolated standalone-run + directory. If neither context exists, progressive disclosure still keeps + the source out of the prompt and reports that execution is unavailable.""" text = _prompt(graph, _AGENTS[4][0], script_bound=False) script = next(n for n in graph["nodes"] if n.get("type") == "script") code = (script["data"].get("config") or {}).get("code") or "" assert code, "fixture's Script node lost its code" - assert code in text + assert code not in text + assert "Available scripts (source remains out of context until execution):" in text + assert "Link the protocol to an experiment to execute wired scripts." in text def test_a_script_bound_prompt_does_not_inline_the_code(graph: dict[str, Any]) -> None: @@ -363,13 +368,13 @@ def test_the_model_and_effort_factors_still_bind(graph: dict[str, Any]) -> None: node. ``apply_factor_bindings`` runs before any validation, so a broken binding would make every cell run the same arm.""" patched = pe.apply_factor_bindings(graph, {"Azure Foundry:Model": "claude-opus-5", "Azure Foundry:Effort": "xhigh"}) - llm = next(n for n in patched["nodes"] if n.get("type") == "llm_azure_foundry") + llm = next(n for n in patched["nodes"] if n.get("type") == "model_azure_foundry") config = llm["data"]["config"] assert config["model"] == "claude-opus-5" assert config["effort"] == "xhigh" # The original is untouched -- apply_factor_bindings deep-copies, which is # what lets replicates of different arms share one stored graph. - original = next(n for n in graph["nodes"] if n.get("type") == "llm_azure_foundry") + original = next(n for n in graph["nodes"] if n.get("type") == "model_azure_foundry") assert original["data"]["config"]["model"] != "claude-opus-5" diff --git a/tests/test_workspace_server.py b/tests/test_workspace_server.py new file mode 100644 index 0000000..52c8ec3 --- /dev/null +++ b/tests/test_workspace_server.py @@ -0,0 +1,63 @@ +"""Dataset-scope boundaries on the ASAREE workspace MCP server.""" + +from __future__ import annotations + +import json +from typing import Any + +from asaree.mcp_servers import workspace_server as ws + + +class _FakeCtx: + def __init__(self, extra: dict[str, Any]) -> None: + self.request_context = type("_R", (), {"meta": type("_M", (), {"model_extra": extra})()})() + + +async def test_open_workspace_rejects_an_explicit_dataset_not_wired_to_the_run() -> None: + ctx = _FakeCtx( + { + "motoro.workspace_id": "experiment/cell", + "motoro.ambient.dataset_names": ["attached"], + } + ) + + result = json.loads(await ws.open_workspace(name="other-registered-dataset", ctx=ctx)) + + assert result == { + "error": "Dataset 'other-registered-dataset' is not wired into this run.", + "wired_datasets": ["attached"], + } + + +async def test_open_workspace_still_allows_the_wired_name_and_outside_run_calls() -> None: + wired_ctx = _FakeCtx( + { + "motoro.workspace_id": "experiment/cell", + "motoro.ambient.dataset_names": ["attached"], + } + ) + + wired = json.loads(await ws.open_workspace(name="attached", ctx=wired_ctx)) + outside = json.loads( + await ws.open_workspace( + experiment_id="experiment", + cell_label="cell", + name="other-registered-dataset", + ) + ) + + # Both calls pass dataset-scope validation and stop later only because this + # unit-test context deliberately has no authenticated owner id. + assert "owner resolution" in wired["error"] + assert "owner resolution" in outside["error"] + + +async def test_open_workspace_rejects_explicit_data_when_the_run_wires_none() -> None: + ctx = _FakeCtx({"motoro.workspace_id": "experiment/cell"}) + + result = json.loads(await ws.open_workspace(name="registered-but-unwired", ctx=ctx)) + + assert result == { + "error": "Dataset 'registered-but-unwired' is not wired into this run.", + "wired_datasets": [], + } diff --git a/uv.lock b/uv.lock index a5b55a6..3ec1cd2 100644 --- a/uv.lock +++ b/uv.lock @@ -252,7 +252,7 @@ requires-dist = [ { name = "bcrypt", specifier = ">=4.2.0" }, { name = "email-validator", specifier = ">=2.2.0" }, { name = "fastapi", specifier = ">=0.121.0" }, - { name = "motoro", git = "https://github.com/EpistasisLab/motoro.git?tag=v0.6.1" }, + { name = "motoro", git = "https://github.com/EpistasisLab/motoro.git?tag=v0.8.0" }, { name = "pandas", specifier = ">=2.2.0" }, { name = "pyarrow", specifier = ">=18.0.0" }, { name = "pydantic", specifier = ">=2.10.0" }, @@ -1876,8 +1876,8 @@ wheels = [ [[package]] name = "motoro" -version = "0.6.1" -source = { git = "https://github.com/EpistasisLab/motoro.git?tag=v0.6.1#9ddf869964d8e1f4dc3287122474950a1c275a01" } +version = "0.8.0" +source = { git = "https://github.com/EpistasisLab/motoro.git?tag=v0.8.0#123cdbbe273d88e39bd5cfaeaf47b63f80c60370" } dependencies = [ { name = "alembic" }, { name = "asyncpg" },