From 30726446487a5a5d734b6225cce5907ab073f93e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 12:07:50 +0000 Subject: [PATCH 01/12] Plan named resource instances and a shared Resources section Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- specs/resource-instances.md | 460 ++++++++++++++++++++++++++++++++++++ 1 file changed, 460 insertions(+) create mode 100644 specs/resource-instances.md diff --git a/specs/resource-instances.md b/specs/resource-instances.md new file mode 100644 index 0000000..ce9a9f6 --- /dev/null +++ b/specs/resource-instances.md @@ -0,0 +1,460 @@ +# Proposal: Named resource instances and a Resources section + +> **Status: PLAN, for discussion.** Nothing here is implemented. It builds on +> `specs/execution-inputs.md`, `specs/resource-arguments.md` (whose §Q.2 anticipated "named +> instances") and `specs/evals.md` §B.2. + +## Context + +Resources work, and arguments made them parameterisable. One case still can't be expressed: +**side effects that don't feed any slot, assembled from modular pieces that need each other's +outputs.** + +The motivating example is a task hierarchy for odin. `list_tasks` reads it back through a tool, so +no prompt slot ever takes the child tasks. The prompt takes `taskId` (the root) and nothing else: + +``` +root = seededTask({ title: "Todo app" }) → taskId slot +child1 = seededTask({ title: "Add drag reorder", parentId: root.taskId }) (no slot) +child2 = seededTask({ title: "Persist to localStorage", parentId: root.taskId }) (no slot) +``` + +Three things stop this today. The first two are why the user reported it. The third came up while +reading the registry. + +1. **Nothing runs unless a slot reads it.** Resources are acquired lazily by + `resolveExecutionInputs` as slots resolve. An unselected resource is never created, so + `child1` cannot exist. +2. **A resource has at most one instance per run, as far as the panel is concerned.** The wire + allows two bindings of one URI with different `args`, but the panel holds arguments per URI + (`ResourceArgs`, `resource-arguments.md` §I). Even on the wire, an instance's identity *is* its + recipe (`canonicalArgumentKey`), so two siblings with identical arguments collapse into one, + and editing a text field changes which instance a reference means. +3. **The registry rejects the hierarchy outright.** Cycle detection runs on the resource object, + ignoring the argument key (`resource-registry.ts`, `instantiate`): + `chain.some(c => c.resource === target)`. A child whose `parentId` argument is + `seededTask.taskId` puts `seededTask` in its own chain, so it fails with + `Resource dependency cycle`. `resource-arguments.md` §D chose this deliberately, on the grounds + that "a resource that recursively seeds itself with different arguments … does not exist". This + is that case. + +**The ask:** a **Resources** section at the top of the playground's Execute panel, the eval's +Inputs panel, and a dataset row's details, sharing one implementation. It lists: + +- every resource selected for a slot, +- their code-wired dependencies (`db`), where any arguments they take can be configured, +- resources added with no slot binding at all, which run for their side effects, +- several instances of one resource, each with a name, so a slot (or another instance's + argument) can pick a specific one. + +--- + +## A. The model: a run declares named instances; slots reference them + +Today a run is a set of **slot bindings**, and resources appear *inside* them, inline, identified +by recipe. The proposal adds a second, sibling part to every run: a map of **named instances**. +Slots and arguments then refer to instances by name. + +```ts +/** One resource instance a run creates, whether or not anything reads it. */ +export interface ResourceInstanceInput { + /** The resource's root `ResourceInfo.uri`. Never an output URI. */ + uri: string; + /** Values for its declared arguments. Same as `ExecutionInput`'s `resource.args`. */ + args?: Record; + /** + * Code-wired dependency key → instance name, only where the run has more than one + * instance of that dependency's resource (§D). Usually absent. + */ + deps?: Record; + /** Recorded on every run, sent back only by a replay. Same contract as today's (§F). */ + receipt?: unknown; +} + +export type ExecutionInput = + | … // value, object, resource, dataset, input: unchanged + /** An instance declared in the run's `resources`, or one of its outputs. */ + | { kind: "instance"; name: string; output?: string }; +``` + +The hierarchy, as a request: + +```jsonc +{ + "resources": { + "root": { "uri": "tasks.ts#seededTask", "args": { "title": { "kind": "value", … } } }, + "child1": { "uri": "tasks.ts#seededTask", "args": { + "title": { "kind": "value", … }, + "parentId": { "kind": "instance", "name": "root", "output": "taskId" } } }, + "child2": { "uri": "tasks.ts#seededTask", "args": { … same parentId … } } + }, + "functionInputs": [ { "kind": "instance", "name": "root", "output": "taskId" } ] +} +``` + +What the change buys: + +- **Identity is the name, not the recipe.** `child1` and `child2` stay two tasks even with + identical arguments. Editing an argument changes what an instance *does*, not which instance a + reference means. That is the same split `resource-arguments.md` §C made between URI and `args`, + carried one level further. +- **Unbound instances are ordinary entries.** Every instance in `resources` is created when the + run starts, whether or not anything references it. "Side effect only" isn't a separate concept: + it's an instance nobody reads. +- **Referencing another instance's output is one small node**, not a copy of its recipe. Today + `parentId` would have to repeat `root`'s whole `{ uri, args }` inline and rely on the memo key + matching. That's the drift problem `evals.md` §B.2 invented `input` to avoid for slots. +- **Arguments are stored once per instance**, not stamped onto every reference. The panel's + `ResourceArgs`/claims machinery, which works out which chip "owns" the form, is no longer needed + (§H). +- **`instance` is the same kind of node as `input`.** Both say "the value something else in this + run produced", and both resolve through the lease, so the same instance comes back. + +### A.1 The inline `resource` variant stays, but nothing new writes it + +`{ kind: "resource", uri, args }` keeps resolving exactly as it does now, as an anonymous instance +keyed by recipe. Traces, dataset rows, saved evals and `localStorage` all hold it, and the +resolver can carry both forms indefinitely at very little cost. The UI stops emitting it. When a +surface loads inline references for editing, it **hoists** them: each distinct +`(uri, canonicalArgumentKey)` becomes an instance named after its export, and the references are +rewritten to `instance` nodes. Hoisting groups by the same key the lease memoizes on, so a hoisted +run creates exactly the instances the inline one did. Stored data is never bulk-migrated. It's +rewritten only when someone edits and saves it. + +### A.2 Names + +- **Default name is the export name** (`seededTask`), suffixed `2`, `3`… on collision. Users + rename to something meaningful (`root`, `child1`). +- **Grammar:** `[A-Za-z_][A-Za-z0-9_-]*`. No dots, because the UI shows outputs as `root.taskId`. +- **Renaming rewrites every reference in the same scope** (slots, other instances' args, check + args, a row's cells). This is a pure function over the editor state (§H). +- **Why names rather than opaque ids with a display label:** names are what makes a recorded + trace, a dataset row and an eval binding readable (`taskId ← root.taskId`). They also let an + eval refer to an instance each *row* declares (§E). The cost is that a rename has to rewrite + references. Within one scope that's mechanical. Across scopes (an eval naming a row's instance) + it can't be done, and §E says what happens instead. + +## B. Server: instances are acquired eagerly, keyed by name + +`resolveExecutionInputs` gains the run's instances in its `ResolutionContext`: + +```ts +interface ResolutionContext { + row?: Pick; + bindings?: InputBindings; + /** The run's named instances: eval-level merged with row-level (§E), or the panel's. */ + instances?: Record; +} +``` + +And the lease gains one entry point beside `acquire`: + +```ts +/** Creates (or reuses) the named instance, resolving its args under the same context. */ +acquireInstance(name: string, spec: ResourceInstanceInput, binding: ResourceBinding): Promise; +``` + +Rules: + +1. **Eager.** Before resolving slots, the resolver acquires *every* instance in + `context.instances`, in parallel. References between instances order them naturally: `child1` + awaits `root` because its `parentId` argument resolves through it. Slot resolution then hits + the memo. +2. **Memo key is `(resource object, "@" + name)`.** It lives in the same `InstancesByKey` map + as inline instances. The `@` prefix can't collide with a canonical argument key, which is + either `""` or a JSON object. Reset locks, disposal and server scope keep working without + changes. +3. **Cycle detection runs on instance identity, not on the resource object.** The chain records + `(resource, key)`. `child1 → root` is two different keys of one resource, so it is allowed. + `root.parentId ← child1.taskId` together with `child1.parentId ← root.taskId` is a cycle and + fails, naming the instances. For anonymous inline instances the old resource-object rule stays + exactly as it is: nothing new can produce those, and relaxing it there would buy nothing. +4. **Validated up front, as a 400**, together with the existing `input` checks: an `instance` node + naming a missing instance, an invalid name, a cycle among instances, or two instances of a + `server`-scoped resource. A server-scoped resource has exactly one instance per process, so two + names would alias one object, which is never what someone meant. The panel doesn't offer it. + Output keys are still checked at acquire time by the existing + `no output at '…'` error. +5. **Receipts are recorded per instance**, and `stampReceipts` writes them into the recorded + `resources` map (§F). `lease.receipts()` grows an `instances` half keyed by name. Its existing + `uri` / `uri@key` keys keep serving the inline form. + +Nothing about `create`, `ResourceInstance`, arguments, validation, `reset` or scope changes for +resource authors. **This is invisible in playground modules.** + +## C. Where `resources` lives on every surface + +| Surface | Today | Adds | +| --- | --- | --- | +| `ExecuteRequest` | `functionInputs`, `executeInputs` | `resources?: Record` | +| Playground `StoredInputs` (`localStorage`) | same two | `resources?` | +| `EvalInputs` (JSON column `evals.inputs`) | same two | `resources?`. No migration, it's JSON | +| Check bindings (`EvalCheck.args`) | `ExecutionInput`s | may contain `instance` nodes | +| `DatasetRow` | `cells` | `resources?`. New nullable `dataset_rows.resources` blob, plus the local-directory provider's file shape | +| `DatasetRowUpdate` | `cells` | `resources?: Record` | +| `eval_row_results.row_cells` snapshot | cells | the row's `resources` too, so a result shows what it ran with | +| `PromptID` / recorded trace inputs | `functionInputs`, `executeInputs`, defs | `resources?`, with receipts stamped | +| `NamedInputs` flows (`named-inputs.ts`) | slot inputs | carry `resources` alongside (panel ↔ row ↔ trace) | + +The `dataset_rows` column is the only real schema migration. Every other row in the table is a +new optional field on an existing JSON value. + +## D. Dependencies + +A code-wired dependency (`inputs: { db }`) is resolved by the registry today with no binding and +key `""`. The section shows dependencies, and the user wants to be able to configure their +arguments. So the rule becomes: + +> When resolving a code-wired dependency of a named instance, the registry uses the instance of +> that resource **declared in the run**, if there is one. If there are several, it uses the one +> `deps` names. If there are none, it falls back to today's anonymous key `""`. + +- **The panel shows dependencies derived from what's selected**, transitively, using a new + `ResourceInfo.dependencies?: Record` (input key → URI). The provider already + partitions `inputs` into dependencies and parameters, so this costs nothing to compute. A derived + dependency isn't stored. It's a read-only row ("◆ db · shared · used by root, child1, + child2"), with a preview of its value where `ResourceInfo.value` already provides one. +- **A dependency with arguments gets a form**, and editing it materialises a stored instance named + after its export. Its arguments have to be stored somewhere, and an instance is that place. + Under the rule above, every dependent then uses it. (Server-scoped resources still can't take + arguments, so in practice this means run-scoped dependencies like a per-run workspace.) +- **Several instances of one dependency are a later step (§K.5).** Then each dependent shows a + small picker for that input (`db: [db ▾]`), stored in `deps`. This is the "rebind a + resource-valued input" that `resource-arguments.md` §M left out, restricted to instances *of + the same resource*. That keeps it type-safe by construction, which was §M's reason for + deferring it. + +## E. Evals and dataset rows: two layers, one namespace + +An eval run assembles one run per row. Its instances are the eval's `resources` merged with the +row's `resources`, in a single namespace, so either layer can reference the other's instances by +name: + +- **Eval-level** instances apply to every row: "seed this fixture workspace for every case". +- **Row-level** instances belong to that case: this row's hierarchy. A row cell, or the eval's + `taskId` binding, can say `instance root.taskId`, and each row supplies its own `root`. +- **On a name collision the row wins.** That lets an eval define a default `root` that particular + rows override. It also means a typo can shadow something silently, so the run's recorded + `resources` (§F) always shows the merged result, and eval validation reports a reference that + names an instance **no** row and not the eval declares. (The alternative, treating any collision + as an error, is open question §L.1.) +- **Renames don't cross layers.** Renaming `root` in one row's details rewrites that row's cells + only. An eval binding that named `root` then fails for that row with + `Row 12: no resource instance 'root'`. It's an `error`, not a `fail`, as `evals.md` §B.3 already + treats bad bindings. + +Copying between surfaces carries instances along. "Add to dataset" from the panel writes the +panel's `resources` into the new row, so the instance references in its cells still resolve. +"Fill from row" loads the row's instances into the panel's section. A playground trace → +row/panel round trip carries `resources` with its receipts stripped, as cells already are. + +## F. Traces and replay + +The recorded inputs gain `resources`, with each instance's receipt stamped on, next to +`functionInputs`/`executeInputs`. Replay is still "POST what was recorded". The hierarchy now +reproduces properly: `root`'s receipt brings back its id, and `child1.parentId` is a *reference* +to `root.taskId`, not a copied string, so it follows. If the hierarchy had been written as copied +ids, the replay's children would point at the original run's parent. + +## G. UI: the Resources section + +``` +Resources [ + Add resource ] +┌────────────────────────────────────────────────────────────────────────┐ +│ ◆ db Local D1 (.wrangler state) · shared dependency │ +│ used by root, child1, child2 │ +├────────────────────────────────────────────────────────────────────────┤ +│ ◆ root ✎ Seeded task · per run [⋯] │ +│ → taskId, toolsContext.*.rootTaskId │ +│ title [ Todo app ] │ +│ status [ triaged ] │ +│ parentId [ ] ⋯ │ +├────────────────────────────────────────────────────────────────────────┤ +│ ◆ child1 ✎ Seeded task · per run [⋯] │ +│ not bound to a slot: runs for its side effects │ +│ title [ Add drag reorder ] │ +│ parentId [ ◆ root.taskId ] ⋯ │ +└────────────────────────────────────────────────────────────────────────┘ +taskId * [ ◆ root.taskId ⋯ ] +``` + +- **One card per instance**, then derived dependencies. A card shows the editable name, the + resource label, scope, what references it ("→ taskId") or that nothing does, and the argument + form. Removing an instance asks first if anything references it, then clears those references. + The `[⋯]` menu holds Rename, Duplicate (the quickest way to make `child2`) and Remove. +- **Argument forms are `ExecutionInputEditor`s, as today**, so an argument can be a typed value, an + output of another instance, another slot (`input`), or a column (in an eval). Instances are + excluded from their own forms and from any form that would close a cycle, the same way + `resource-arguments.md` §H already excludes self-dependency. +- **The slot picker gets a "Resources in this run" group first**, listing instances with their + outputs as a submenu (`root ▸ Task ID`). Below it, **"New ▸"** holds the existing catalog tree. + Picking from the catalog adds an instance with the default name *and* binds the slot. That + keeps today's one-click flow: picking `seededTask.taskId` for `taskId` and then `root ▸ Title` + for `taskInfo` still gives one task. +- **A chip in a slot is only a reference** (`◆ root.taskId`), and clicking it scrolls to the card. + Forms never appear under chips any more, so "same instance as `taskId` above" goes away. +- **Unbinding a slot doesn't remove the instance**, because instances outliving bindings is the + point. The card switches to "not bound to a slot", which is visible at the top of the panel, so a + forgotten instance doesn't quietly keep seeding rows. (§L.2 has the alternative of auto-pruning + instances the picker created.) +- **"+ Add resource"** opens the same catalog `SourcePicker` uses, filtered to root resources. + Server-scoped and static resources that already have an instance are disabled, with the reason + as a tooltip. +- **Placement:** above the slots in the playground panel and the eval Inputs panel. In row details + it goes above the fields, with the same commit-on-blur behaviour `RowFields` already uses for + typed edits, and immediate commits for add, remove and rename. +- **Empty state:** when no instances or dependencies exist, the section collapses to the header and + its Add button, so prompts without resources look exactly as they do now, apart from one row. + +## H. Sharing the code + +All three surfaces currently hold the same state in three slightly different ways +(`PlaygroundExecution` lines 215–370, `EvalView` 120–150 and 420–440, `DatasetRowDetails` +130–260). Each one re-derives `ResourceArgs`, runs `computeClaims`, and builds its own +`ResourceArgsContext`. The plan replaces that with: + +- **`run-resources-state.ts`** (pure, unit-tested; the client counterpart of §A): + - `InstanceSelections = Record }>`, + which replaces `ResourceArgs`; + - `toWireResources` / `fromWireResources`, plus `hoistInlineResources(inputs[])` for §A.1; + - `addInstance`, `renameInstance` (rewrites every reference in the given selections), + `removeInstance`, `duplicateInstance`, `referencesTo`, `derivedDependencies`, + `excludedFromArgs` (cycle-safe sources for a card's form). +- **`ResourcesSection.tsx`**: a controlled component, + `{ instances, onChange, resources, resourceSlots, describePseudo, referencedBy }`. Persistence + stays with each surface: `localStorage`, eval save, row commit. +- **`pseudo-sources.ts` gains `instanceUri(name, output?)`** and maps it to and from + `{ kind: "instance" }` in `pseudoInput`/`pseudoUriOf`, the same way columns and `input` slots + already go through the picker. The picker group comes from the instances plus `functionSlots` + matches (an instance of `seededTask` fits wherever `seededTask.taskId`'s URI fits), so **no new + server-side matching** is needed. + +Removed: `computeClaims`, `ClaimOwner`, `ResourceArgsContext.claimed`/`resourceArgs`/`depth`, +`ResourceArgumentsForm`, `MAX_RESOURCE_ARG_DEPTH`, `resourceArgsFor`'s recursion, and the +"same instance as" note. `ResourceArgsContext` shrinks to what an editor needs for chips +(`instances`, `describePseudo`). That's a net deletion of several hundred lines across +`resource-args-context.ts`, `execution-input-state.ts`, `SourceRow.tsx` and the three surfaces. + +## I. Alternatives considered + +**Keep instances inline, and add `instance?: string` to the `resource` node as a memo-key +discriminator.** It's the smallest server change. But unbound side effects still have nowhere to +live, arguments are still duplicated on every reference, `parentId` still repeats `root`'s whole +recipe, and the panel keeps the claims machinery. Rejected. + +**Make the section a UI over today's wire**: instances are the distinct `(uri, args)` pairs, +plus a `sideEffects: ExecutionInput[]` list. Identical siblings still collapse, a rename has +nothing to attach to, and a child still trips the cycle rule. Rejected. + +**One resource per hierarchy** (`seededTaskTree` taking `tasks: { key, title, parentKey? }[]`). +This **works today with no platform change**, and for a hierarchy that is pure data it's a good +pattern to document. It doesn't cover the general case: it can't mix different resources (a task, +a user, a message thread); outputs are declared statically, so an inner node's id can't be offered +to a slot; and every new shape needs a new tree resource. Recommended as a pattern, not as the +answer. + +**A `setup()` playground export** run before every prompt. It duplicates the resource lifecycle +(scope, reset, dispose, receipts) for no gain, and it isn't per-run configurable. Rejected. + +**Opaque instance ids with display labels.** Renames become free, but every recorded trace, +dataset row and eval binding then reads `r3.taskId`, and an eval can't refer to a per-row instance +by meaning. Rejected (§A.2). Open question §L.3 keeps it under review. + +## J. Non-goals + +- Ordering side effects that have no data dependency between them. Instances run in parallel, and + a reference is the only ordering. Nothing needs an `after:` field yet. +- Showing row resources in the dataset *grid*. A count badge at most. The details pane is where + they're edited. +- Rebinding a dependency to a *different* resource. §D only rebinds between instances of the same + resource. +- Server-scoped resources taking arguments. That's still refused (`resource-arguments.md` §Q.1). + +## K. Phasing + +1. **Server.** `ResourceInstanceInput`, the `instance` variant, `ExecuteRequest.resources`, eager + acquisition, name-keyed memo, instance-identity cycle rule, 400 validation, the dependency rule + (§D, without `deps`), per-instance receipts and recording. `ResourceInfo.dependencies`. Testable + end to end with a hand-written request seeding odin's hierarchy. +2. **Shared client state and `ResourcesSection`**, adopted by the playground panel: picker group, + hoisting `localStorage`, deleting the claims machinery. +3. **Evals.** `EvalInputs.resources`, the section in the Inputs panel, instance references in check + args, and the eval runner passing instances into `ResolutionContext`. +4. **Datasets.** `DatasetRow.resources` (Turso migration plus the local-directory provider), + `updateRows`, the section in row details, `named-inputs.ts` flows, eval/row merge (§E), and + the `eval_row_results` snapshot. +5. *(when wanted)* `deps`: several instances of a dependency, with the per-input picker. + +Steps 1–2 are enough for the odin hierarchy in the playground. Step 4 is what makes it a dataset +case. + +## L. Open questions + +1. **Row/eval name collision: row wins, or an error?** Proposed: row wins (§E), so an eval can + carry defaults. The cost is silent shadowing. +2. **Should instances the picker created be pruned when their last reference goes?** Proposed: no. + Instances stay, labelled "not bound to a slot". Auto-pruning matches today's feel (choosing + another source for a slot makes the old resource disappear), but it needs a stored `auto` flag + and makes "why did my instance vanish" possible. +3. **Names or ids** (§A.2, §I). Proposed: names. +4. **Should the inline `resource` variant be deprecated from the wire eventually?** Proposed: keep + resolving it forever, since old traces need it, but stop emitting it from the UI (§A.1). +5. **Does `input` (slot → slot) still earn its place?** Proposed: yes. `toolsContext.*.rootTaskId ← + input taskId` says "whatever `taskId` is", which stays correct when `taskId` is bound to a + column instead of an instance. The two overlap but do different jobs. + +## M. Files + +- `src/shared/types.ts`: `ResourceInstanceInput`, the `instance` variant, + `ExecuteRequest.resources`, `ResourceInfo.dependencies`. +- `src/prompt/execution-inputs.ts`: `ResolutionContext.instances`, eager acquisition, the + `instance` case, validation, `stampReceipts` for instances. +- `src/shared/input-references.ts`: instance references in cycle checks and `inputReferenceProblems`. +- `src/prompt/playground/resource-registry.ts`: `acquireInstance`, the `@name` key, the + instance-identity cycle rule, the dependency-resolution rule, per-instance receipts. +- `src/prompt/file/file-prompt-provider.ts`: `ResourceInfo.dependencies`, and passing + `resources` through `resolveInputs`. +- `src/server/api-routes.ts`: accepting and recording `resources`. +- `src/trace/trace-types.ts`, `src/trace/prompt-tracer.ts`, `src/sdk/vercel-ai-sdk/telemetry.ts`, + `src/trace/otel-trace-ingestor.ts`: `resources` on recorded inputs. +- `src/eval/eval-types.ts`, `src/eval/eval-runner.ts`, `src/shared/eval-problems.ts`: + `EvalInputs.resources`, the eval/row merge, validation. +- `src/dataset/dataset-types.ts`, `turso-dataset-provider.ts` plus a migration under + `src/dataset/db/migrations/`, `local-directory-dataset-provider.ts`, + `dataset-provider-contract.ts`: `DatasetRow.resources`, `updateRows`. +- `src/client/components/run-resources-state.ts` *(new)* and `ResourcesSection.tsx` *(new)*. +- `src/client/components/pseudo-sources.ts`, `SourcePicker.tsx`, `source-tree.ts`: instance URIs + and the "Resources in this run" group. +- `src/client/components/PlaygroundExecution.tsx`, `EvalView.tsx`, `DatasetRowDetails.tsx`, + `dataset-row-editing.ts`, `named-inputs.ts`: adopting the section. +- `src/client/components/SourceRow.tsx`, `ExecutionInputEditor.tsx`, `resource-args-context.ts`, + `execution-input-state.ts`: removing per-chip argument forms and claims. +- `docs/`: the resource authoring guide gains "several instances" and "side-effect-only + resources", and the tree-resource pattern from §I. + +## N. Verification + +- **Registry (`vitest`, `MemoryFileProvider`):** two named instances of one resource with + identical args run `create` twice. An instance whose argument references another instance of + the *same* resource resolves, which is the hierarchy regression for problem 3 in the Context. A + two-instance reference cycle fails with both names. An instance nobody references is still + created and disposed. Slots referencing an instance hit the memo, so `create` runs once. A + dependency resolves to the run's declared instance of that resource and gets its args. A + server-scoped resource declared twice is a 400. Inline `resource` references resolve exactly as + before, keeping the existing tests green unchanged. +- **Receipts and replay:** recorded `resources` carry per-instance receipts. POSTing them back + reproduces `root`'s id, and `child1.parentId` matches it. +- **Client state:** `hoistInlineResources` groups by `(uri, argument key)` so a hoisted run creates + the same instances. `renameInstance` rewrites slot, argument and check references, and nothing + else. `removeInstance` clears references. A stored selection from before this change restores + with the same behaviour. +- **Evals and datasets:** row instances shadow eval instances. An eval binding to a name a row + lacks gives that row an `error` naming it. "Add to dataset" carries instances, and "Fill from + row" restores them. +- **Playwright (`*.pw.tsx`):** adding a resource with no slot shows the card as unbound. Picking + from "New ▸" adds an instance and binds it. A second slot picking `root ▸ Title` reuses it. + Duplicate, then rename, updates every chip. The same `ResourcesSection` renders in the panel, + the eval Inputs panel and row details (one harness per host). +- **End to end:** in odin's playground, add `root`, `child1` and `child2` with `parentId` set to + `root.taskId`, bind only `taskId`, run, and see `list_tasks` return the three-node tree in the + trace. From e747bf09e1569ff430ddb97fea5976911ec163b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 13:02:46 +0000 Subject: [PATCH 02/12] Drop the inline resource variant from the instances plan Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- specs/resource-instances.md | 91 +++++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 40 deletions(-) diff --git a/specs/resource-instances.md b/specs/resource-instances.md index ce9a9f6..d3e58f6 100644 --- a/specs/resource-instances.md +++ b/specs/resource-instances.md @@ -1,7 +1,8 @@ # Proposal: Named resource instances and a Resources section -> **Status: PLAN, for discussion.** Nothing here is implemented. It builds on -> `specs/execution-inputs.md`, `specs/resource-arguments.md` (whose §Q.2 anticipated "named +> **Status: PLAN, agreed 2026-10-08.** Nothing here is implemented yet. §L's proposed answers were +> accepted, and the inline `resource` variant is removed outright rather than kept (§A.1). +> It builds on `specs/execution-inputs.md`, `specs/resource-arguments.md` (whose §Q.2 anticipated "named > instances") and `specs/evals.md` §B.2. ## Context @@ -60,7 +61,7 @@ Slots and arguments then refer to instances by name. export interface ResourceInstanceInput { /** The resource's root `ResourceInfo.uri`. Never an output URI. */ uri: string; - /** Values for its declared arguments. Same as `ExecutionInput`'s `resource.args`. */ + /** Values for its declared arguments, by parameter name. */ args?: Record; /** * Code-wired dependency key → instance name, only where the run has more than one @@ -72,7 +73,8 @@ export interface ResourceInstanceInput { } export type ExecutionInput = - | … // value, object, resource, dataset, input: unchanged + | … // value, object, dataset, input: unchanged + // `{ kind: "resource", uri, args, receipt }` is removed (§A.1) /** An instance declared in the run's `resources`, or one of its outputs. */ | { kind: "instance"; name: string; output?: string }; ``` @@ -110,16 +112,26 @@ What the change buys: - **`instance` is the same kind of node as `input`.** Both say "the value something else in this run produced", and both resolve through the lease, so the same instance comes back. -### A.1 The inline `resource` variant stays, but nothing new writes it +### A.1 The inline `resource` variant is removed -`{ kind: "resource", uri, args }` keeps resolving exactly as it does now, as an anonymous instance -keyed by recipe. Traces, dataset rows, saved evals and `localStorage` all hold it, and the -resolver can carry both forms indefinitely at very little cost. The UI stops emitting it. When a -surface loads inline references for editing, it **hoists** them: each distinct -`(uri, canonicalArgumentKey)` becomes an instance named after its export, and the references are -rewritten to `instance` nodes. Hoisting groups by the same key the lease memoizes on, so a hoisted -run creates exactly the instances the inline one did. Stored data is never bulk-migrated. It's -rewritten only when someone edits and saves it. +`{ kind: "resource", uri, args, receipt }` goes away entirely. Nobody depends on stored traces, +rows or evals yet, so there's nothing to migrate and no compatibility path to carry. Anything in +`localStorage`, a local dataset or a saved eval that still holds the old node is treated like any +other unparseable stored input: a slot holding it restores empty, and the server rejects it with a +400 naming the slot. + +Dropping it, rather than keeping it as an anonymous instance keyed by recipe, removes a lot: + +- **One resolution path.** No anonymous instances, so the lease memoizes only by instance name. + `canonicalArgumentKey`, `ResourceBinding.key` and the `uri@key` receipt keys all go. +- **One cycle rule.** Cycles are checked on instance names alone. The resource-object rule, and the + special case for keeping it on inline references, both go. +- **No hoisting.** The client never has to turn inline references into instances on load, so + `hoistInlineResources` and its tests never get written. +- **Receipts live in one place**, on the instance. `stampReceipts` no longer walks slot trees + looking for resource nodes. +- **`args` exist in exactly one place.** The client's `resourceArgsFor` stamping, and the + last-writer-wins merge in `fromExecutionInput`, go. ### A.2 Names @@ -160,24 +172,24 @@ Rules: `context.instances`, in parallel. References between instances order them naturally: `child1` awaits `root` because its `parentId` argument resolves through it. Slot resolution then hits the memo. -2. **Memo key is `(resource object, "@" + name)`.** It lives in the same `InstancesByKey` map - as inline instances. The `@` prefix can't collide with a canonical argument key, which is - either `""` or a JSON object. Reset locks, disposal and server scope keep working without - changes. -3. **Cycle detection runs on instance identity, not on the resource object.** The chain records - `(resource, key)`. `child1 → root` is two different keys of one resource, so it is allowed. - `root.parentId ← child1.taskId` together with `child1.parentId ← root.taskId` is a cycle and - fails, naming the instances. For anonymous inline instances the old resource-object rule stays - exactly as it is: nothing new can produce those, and relaxing it there would buy nothing. +2. **Memo key is the instance name** for run-scoped resources. A run-scoped instance map becomes + `Map`. Server-scoped resources keep their single per-process instance, and reset + locks and disposal work as they do now. +3. **Cycle detection runs on instance names, not on the resource object.** `child1 → root` is two + instances of one resource, so it is allowed. `root.parentId ← child1.taskId` together with + `child1.parentId ← root.taskId` is a cycle and fails, naming the instances. Code-wired + dependencies (`inputs: { db }`) keep today's resource-object check, since an undeclared + dependency has no name. 4. **Validated up front, as a 400**, together with the existing `input` checks: an `instance` node naming a missing instance, an invalid name, a cycle among instances, or two instances of a `server`-scoped resource. A server-scoped resource has exactly one instance per process, so two names would alias one object, which is never what someone meant. The panel doesn't offer it. Output keys are still checked at acquire time by the existing `no output at '…'` error. -5. **Receipts are recorded per instance**, and `stampReceipts` writes them into the recorded - `resources` map (§F). `lease.receipts()` grows an `instances` half keyed by name. Its existing - `uri` / `uri@key` keys keep serving the inline form. +5. **Receipts are recorded per instance**, keyed by name: `lease.receipts()` returns + `Record`, and the route writes each one onto its entry in the recorded + `resources` map (§F). Code-wired dependency instances that were never declared (the anonymous + `db` fallback in §D) record under their URI, as now. Nothing about `create`, `ResourceInstance`, arguments, validation, `reset` or scope changes for resource authors. **This is invisible in playground modules.** @@ -316,7 +328,7 @@ All three surfaces currently hold the same state in three slightly different way - **`run-resources-state.ts`** (pure, unit-tested; the client counterpart of §A): - `InstanceSelections = Record }>`, which replaces `ResourceArgs`; - - `toWireResources` / `fromWireResources`, plus `hoistInlineResources(inputs[])` for §A.1; + - `toWireResources` / `fromWireResources`; - `addInstance`, `renameInstance` (rewrites every reference in the given selections), `removeInstance`, `duplicateInstance`, `referencesTo`, `derivedDependencies`, `excludedFromArgs` (cycle-safe sources for a card's form). @@ -377,7 +389,7 @@ by meaning. Rejected (§A.2). Open question §L.3 keeps it under review. (§D, without `deps`), per-instance receipts and recording. `ResourceInfo.dependencies`. Testable end to end with a hand-written request seeding odin's hierarchy. 2. **Shared client state and `ResourcesSection`**, adopted by the playground panel: picker group, - hoisting `localStorage`, deleting the claims machinery. + deleting the claims machinery. 3. **Evals.** `EvalInputs.resources`, the section in the Inputs panel, instance references in check args, and the eval runner passing instances into `ResolutionContext`. 4. **Datasets.** `DatasetRow.resources` (Turso migration plus the local-directory provider), @@ -388,18 +400,18 @@ by meaning. Rejected (§A.2). Open question §L.3 keeps it under review. Steps 1–2 are enough for the odin hierarchy in the playground. Step 4 is what makes it a dataset case. -## L. Open questions +## L. Questions (decided 2026-10-08) -1. **Row/eval name collision: row wins, or an error?** Proposed: row wins (§E), so an eval can +1. **Row/eval name collision: row wins, or an error?** Decided: row wins (§E), so an eval can carry defaults. The cost is silent shadowing. -2. **Should instances the picker created be pruned when their last reference goes?** Proposed: no. +2. **Should instances the picker created be pruned when their last reference goes?** Decided: no. Instances stay, labelled "not bound to a slot". Auto-pruning matches today's feel (choosing another source for a slot makes the old resource disappear), but it needs a stored `auto` flag and makes "why did my instance vanish" possible. -3. **Names or ids** (§A.2, §I). Proposed: names. -4. **Should the inline `resource` variant be deprecated from the wire eventually?** Proposed: keep - resolving it forever, since old traces need it, but stop emitting it from the UI (§A.1). -5. **Does `input` (slot → slot) still earn its place?** Proposed: yes. `toolsContext.*.rootTaskId ← +3. **Names or ids** (§A.2, §I). Decided: names. +4. ~~**Should the inline `resource` variant be deprecated from the wire eventually?**~~ **Decided: + removed now** (§A.1). Nothing stored needs it. +5. **Does `input` (slot → slot) still earn its place?** Decided: yes. `toolsContext.*.rootTaskId ← input taskId` says "whatever `taskId` is", which stays correct when `taskId` is bound to a column instead of an instance. The two overlap but do different jobs. @@ -440,14 +452,13 @@ case. two-instance reference cycle fails with both names. An instance nobody references is still created and disposed. Slots referencing an instance hit the memo, so `create` runs once. A dependency resolves to the run's declared instance of that resource and gets its args. A - server-scoped resource declared twice is a 400. Inline `resource` references resolve exactly as - before, keeping the existing tests green unchanged. + server-scoped resource declared twice is a 400, and so is an old inline `resource` node. Existing + registry tests that use inline references are rewritten to declare instances. - **Receipts and replay:** recorded `resources` carry per-instance receipts. POSTing them back reproduces `root`'s id, and `child1.parentId` matches it. -- **Client state:** `hoistInlineResources` groups by `(uri, argument key)` so a hoisted run creates - the same instances. `renameInstance` rewrites slot, argument and check references, and nothing - else. `removeInstance` clears references. A stored selection from before this change restores - with the same behaviour. +- **Client state:** `renameInstance` rewrites slot, argument and check references, and nothing + else. `removeInstance` clears references. A stored selection holding an old inline node restores + as an empty slot rather than throwing. - **Evals and datasets:** row instances shadow eval instances. An eval binding to a name a row lacks gives that row an `error` naming it. "Add to dataset" carries instances, and "Fill from row" restores them. From e3e8780e51da5d907ca442293a147113b7afe000 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 13:26:50 +0000 Subject: [PATCH 03/12] Resolve resources as named run instances on the server Runs, evals and dataset rows declare resources by name (uri + args), and slots reference them with { kind: "instance", name, output? }. Every declared instance is created, so side-effect resources need no slot. Removes the inline resource variant. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- src/dataset/dataset-provider-contract.ts | 85 ++- src/dataset/dataset-provider.ts | 2 +- src/dataset/dataset-types.ts | 39 +- .../migration.sql | 4 + .../snapshot.json | 200 +++++++ src/dataset/db/migrations/bundled.ts | 9 + src/dataset/db/schema.ts | 2 + src/dataset/turso-dataset-provider.ts | 110 ++-- src/eval/eval-runner.ts | 5 + src/eval/eval-types.ts | 11 +- src/index.ts | 5 +- src/mcp/eval-tools.ts | 102 +++- src/mcp/server.test.ts | 13 +- src/mcp/server.ts | 17 +- src/mcp/tool-helpers.ts | 20 +- src/prompt/execution-inputs.test.ts | 475 ++++++++-------- src/prompt/execution-inputs.ts | 241 ++++---- src/prompt/file/file-prompt-provider.test.ts | 13 +- src/prompt/file/file-prompt-provider.ts | 15 +- src/prompt/playground/check.test.ts | 3 +- .../playground/resource-registry.test.ts | 517 ++++++++++-------- src/prompt/playground/resource-registry.ts | 375 +++++++------ src/prompt/prompt-provider.ts | 10 +- src/sdk/typesafe-sdk/telemetry.ts | 1 + src/sdk/vercel-ai-sdk/telemetry.ts | 2 + src/server/api-routes.test.ts | 16 +- src/server/handlers/datasets.test.ts | 26 +- src/server/handlers/datasets.ts | 32 +- src/server/handlers/evals.ts | 16 +- src/server/handlers/prompts.ts | 1 + src/server/run-prompt.ts | 17 +- src/shared/dataset-cells.test.ts | 100 ++-- src/shared/dataset-cells.ts | 124 +++-- src/shared/eval-prefill.ts | 1 + src/shared/eval-problems.ts | 19 +- src/shared/input-references.ts | 159 ++++-- src/shared/types.ts | 69 ++- src/trace/otel-attributes.ts | 6 + src/trace/prompt-tracer.ts | 6 + src/trace/trace-types.ts | 5 + 40 files changed, 1826 insertions(+), 1047 deletions(-) create mode 100644 src/dataset/db/migrations/20261008131520_row_resources/migration.sql create mode 100644 src/dataset/db/migrations/20261008131520_row_resources/snapshot.json diff --git a/src/dataset/dataset-provider-contract.ts b/src/dataset/dataset-provider-contract.ts index 5ec52ce..dc5c720 100644 --- a/src/dataset/dataset-provider-contract.ts +++ b/src/dataset/dataset-provider-contract.ts @@ -117,14 +117,21 @@ export function runDatasetProviderContractTests( fields: [{ def: def("ticket") }, { def: def("db", "Db") }], }); const resource: ExecutionInput = { - kind: "resource", - uri: ".evalution/playground/tasks.ts#seededTask", - args: { title: text("Buy milk") }, + kind: "instance", + name: "task", + output: "taskId", + }; + const resources = { + task: { + uri: ".evalution/playground/tasks.ts#seededTask", + args: { title: text("Buy milk") }, + }, }; const added = await provider.addRows(dataset.id, [ { cells: { "0": text("first"), "1": resource }, source: { kind: "playground", promptId: "p#x", providerId: "files" }, + resources, }, { cells: { "0": text("second") } }, ]); @@ -134,6 +141,8 @@ export function runDatasetProviderContractTests( const rows = await provider.listRows(dataset.id); expect(rows).toEqual(added); expect(rows[0].cells["1"]).toEqual(resource); + expect(rows[0].resources).toEqual(resources); + expect(rows[1].resources).toBeUndefined(); expect(rows[0].source).toEqual({ kind: "playground", promptId: "p#x", @@ -232,34 +241,38 @@ export function runDatasetProviderContractTests( cells: { "0": text("a"), "1": { - kind: "resource", - uri: "pg.ts#seededTask", - args: { title: text("Milk"), owner: text("ann") }, + kind: "value", + value: { + kind: "object", + properties: { + title: { kind: "primitive", value: "Milk" }, + owner: { kind: "primitive", value: "ann" }, + }, + }, }, }, }, { cells: { "1": { - kind: "resource", - uri: "pg.ts#otherTask", + kind: "object", // `title` again, merged by name; `due` is new. - args: { due: text("today"), title: text("Eggs") }, + properties: { due: text("today"), title: text("Eggs") }, }, "2": { kind: "object", properties: { - db: { kind: "resource", uri: "pg.ts#db" }, + db: { kind: "instance", name: "db" }, userId: text("u1"), }, }, }, }, - // A resource without arguments adds no keys, but still marks the - // field as one a resource fills. + // A resource instance adds no keys, but marks the field as one a + // resource fills. { cells: { - "1": { kind: "resource", uri: "pg.ts#blank" }, + "1": { kind: "instance", name: "task", output: "taskId" }, // A typed-in object's properties are keys too. "3": { kind: "value", @@ -313,9 +326,9 @@ export function runDatasetProviderContractTests( fields: [{ def: def("a") }, { def: def("b", "Task") }], }); const resource: ExecutionInput = { - kind: "resource", - uri: "pg.ts#seededTask", - args: { title: text("Milk") }, + kind: "instance", + name: "task", + output: "taskId", }; const [first, second] = await provider.addRows(dataset.id, [ { cells: { "0": text("one"), "1": resource } }, @@ -356,6 +369,38 @@ export function runDatasetProviderContractTests( expect(row.cells["1"]).toEqual(replacement); }); + it("sets and removes resource instances by name, leaving the rest", async () => { + const provider = await makeProvider(); + const { dataset, first, second } = await seeded(provider); + const root = { uri: "pg.ts#seededTask", args: { title: text("Root") } }; + const child = { + uri: "pg.ts#seededTask", + args: { + title: text("Child"), + parentId: { + kind: "instance", + name: "root", + output: "taskId", + } satisfies ExecutionInput, + }, + }; + await provider.updateRows(dataset.id, [ + { rowId: first.id, cells: {}, resources: { root, child } }, + ]); + expect((await provider.listRows(dataset.id))[0].resources).toEqual({ + root, + child, + }); + + await provider.updateRows(dataset.id, [ + { rowId: first.id, cells: {}, resources: { child: null } }, + ]); + const rows = await provider.listRows(dataset.id); + expect(rows[0].resources).toEqual({ root }); + expect(rows[0].cells).toEqual(first.cells); + expect(rows[1]).toEqual(second); + }); + it("fills an empty cell, and clears one by removing its key", async () => { const provider = await makeProvider(); const { dataset, first, second } = await seeded(provider); @@ -375,7 +420,7 @@ export function runDatasetProviderContractTests( it.each([ [ "a resource cell", - { kind: "resource", uri: "pg.ts#db" } satisfies ExecutionInput, + { kind: "instance", name: "db" } satisfies ExecutionInput, ], [ "an object cell", @@ -705,7 +750,7 @@ export function runDatasetProviderContractTests( }; await provider.addRows(dataset.id, [ { cells: { "0": objectCell } }, - { cells: { "0": { kind: "resource", uri: "res://db" } } }, + { cells: { "0": { kind: "instance", name: "db" } } }, ]); const result = await provider.queryRows( dataset.id, @@ -722,8 +767,8 @@ export function runDatasetProviderContractTests( "SELECT data FROM rows", ); expect(JSON.parse(all.rows[1].data as string)).toEqual({ - kind: "resource", - uri: "res://db", + kind: "instance", + name: "db", }); }); diff --git a/src/dataset/dataset-provider.ts b/src/dataset/dataset-provider.ts index f1d8a81..8d40979 100644 --- a/src/dataset/dataset-provider.ts +++ b/src/dataset/dataset-provider.ts @@ -40,7 +40,7 @@ export interface ListRowsOptions { } /** A row as {@link DatasetProvider.addRows} takes it: id and timestamp are minted. */ -export type NewDatasetRow = Pick; +export type NewDatasetRow = Pick; /** Thrown by a {@link DatasetProvider} when the dataset named doesn't exist. */ export class DatasetNotFoundError extends Error { diff --git a/src/dataset/dataset-types.ts b/src/dataset/dataset-types.ts index d264b35..d8fde81 100644 --- a/src/dataset/dataset-types.ts +++ b/src/dataset/dataset-types.ts @@ -6,7 +6,12 @@ * to exercise prompts. See `specs/datasets.md`. */ -import type { ExecutionInput, PropDefinition } from "../shared/types.ts"; +import type { + ExecutionInput, + PropDefinition, + ResourceInstanceInput, + RunResources, +} from "../shared/types.ts"; import type { PromptID } from "../trace/trace-types.ts"; /** @@ -76,8 +81,14 @@ export type DatasetRowSource = */ export interface DatasetRow { id: string; - /** Field id → the input for that field. Never carries a receipt. */ + /** Field id → the input for that field. */ cells: Record; + /** + * Resource instances this row's runs declare, by name — its cells (and an + * eval's bindings) may name them with `instance` references. Never carries + * a receipt. See `specs/resource-instances.md` §E. + */ + resources?: RunResources; /** Where the row came from. */ source?: DatasetRowSource; /** Creation timestamp (ms). */ @@ -91,29 +102,25 @@ export interface DatasetRow { export interface DatasetRowUpdate { rowId: string; cells: Record; + /** + * The row's resource instances to set, by name, or `null` to remove one. + * Instances not named are left as they are. + */ + resources?: Record; } /** * What one field's cells hold, beyond a single value — found in the rows, - * not the schema: a field's `def` is the slot's type (`Db`), which says - * nothing about which resource a row picked to fill it or what arguments - * that resource took. + * not the schema: a field's `def` is the slot's type, which says nothing + * about the shape of the object a row typed in for it. */ export interface DatasetFieldShape { /** - * The keys one level inside the field's cells: a `resource` cell's - * argument names, an `object` cell's property names, and the property - * names of a typed-in object value. Merged by name across rows, roughly in - * the order they first appear. + * The keys one level inside the field's cells: an `object` cell's property + * names, and the property names of a typed-in object value. Merged by name + * across rows, roughly in the order they first appear. */ keys: string[]; - /** - * Whether any of the field's cells is a resource. A table that splits the - * field into its keys still needs a column naming the resource, since - * different rows may name different ones; a field of plain objects needs - * no such column. - */ - resource?: boolean; } /** diff --git a/src/dataset/db/migrations/20261008131520_row_resources/migration.sql b/src/dataset/db/migrations/20261008131520_row_resources/migration.sql new file mode 100644 index 0000000..4f2b9f2 --- /dev/null +++ b/src/dataset/db/migrations/20261008131520_row_resources/migration.sql @@ -0,0 +1,4 @@ +-- SPDX-License-Identifier: AGPL-3.0-only +-- Copyright (c) 2026 Alexander Corrado + +ALTER TABLE `dataset_rows` ADD `resources` blob; diff --git a/src/dataset/db/migrations/20261008131520_row_resources/snapshot.json b/src/dataset/db/migrations/20261008131520_row_resources/snapshot.json new file mode 100644 index 0000000..d95e6fc --- /dev/null +++ b/src/dataset/db/migrations/20261008131520_row_resources/snapshot.json @@ -0,0 +1,200 @@ +{ + "version": "7", + "dialect": "sqlite", + "id": "30558ebf-c3d8-4571-a95c-df42a830fa30", + "prevIds": [ + "e3b838f0-f65b-47e5-ba4d-a1ac9629fee6" + ], + "ddl": [ + { + "name": "dataset_rows", + "entityType": "tables" + }, + { + "name": "datasets", + "entityType": "tables" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "dataset_id", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "blob", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "cells", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "blob", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "source", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "blob", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "resources", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "created_at", + "entityType": "columns", + "table": "dataset_rows" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "name", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "fields", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "next_field_id", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "prompt", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "created_at", + "entityType": "columns", + "table": "datasets" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_at", + "entityType": "columns", + "table": "datasets" + }, + { + "columns": [ + "dataset_id" + ], + "tableTo": "datasets", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_dataset_rows_dataset_id_datasets_id_fk", + "entityType": "fks", + "table": "dataset_rows" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "dataset_rows_pk", + "table": "dataset_rows", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "datasets_pk", + "table": "datasets", + "entityType": "pks" + }, + { + "columns": [ + { + "value": "dataset_id", + "isExpression": false + }, + { + "value": "created_at", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "idx_dataset_rows_dataset", + "entityType": "indexes", + "table": "dataset_rows" + } + ], + "renames": [] +} \ No newline at end of file diff --git a/src/dataset/db/migrations/bundled.ts b/src/dataset/db/migrations/bundled.ts index d7139eb..c84b44d 100644 --- a/src/dataset/db/migrations/bundled.ts +++ b/src/dataset/db/migrations/bundled.ts @@ -19,5 +19,14 @@ export const bundledMigrations: MigrationMeta[] = [ "\nCREATE TABLE `datasets` (\n\t`id` text PRIMARY KEY,\n\t`name` text NOT NULL,\n\t`fields` text NOT NULL,\n\t`next_field_id` integer NOT NULL,\n\t`prompt` text,\n\t`created_at` real NOT NULL,\n\t`updated_at` real NOT NULL\n);\n", "\nCREATE INDEX `idx_dataset_rows_dataset` ON `dataset_rows` (`dataset_id`,`created_at`);" ] + }, + { + "name": "20261008131520_row_resources", + "hash": "3d47cdd5b4a04ceef5601b30f690fc37da4ff071e6ce6c0af10be84c8ffd23e7", + "folderMillis": 1791465320000, + "bps": true, + "sql": [ + "-- SPDX\u002DLicense-Identifier: AGPL-3.0-only\n-- Copyright (c) 2026 Alexander Corrado\n\nALTER TABLE `dataset_rows` ADD `resources` blob;\n" + ] } ]; diff --git a/src/dataset/db/schema.ts b/src/dataset/db/schema.ts index adb1cce..04a6d55 100644 --- a/src/dataset/db/schema.ts +++ b/src/dataset/db/schema.ts @@ -70,6 +70,8 @@ export const datasetRows = sqliteTable( cells: jsonb("cells").notNull(), /** JSONB `DatasetRowSource`. */ source: jsonb("source"), + /** JSONB `RunResources` — the row's resource instances. Absent when it declares none. */ + resources: jsonb("resources"), /** Creation timestamp (ms). */ createdAt: real("created_at").notNull(), }, diff --git a/src/dataset/turso-dataset-provider.ts b/src/dataset/turso-dataset-provider.ts index f4c8341..d25b16e 100644 --- a/src/dataset/turso-dataset-provider.ts +++ b/src/dataset/turso-dataset-provider.ts @@ -12,7 +12,12 @@ import type { Database } from "@tursodatabase/sync"; import { and, asc, desc, eq, inArray, type SQL, sql } from "drizzle-orm"; import { drizzle } from "drizzle-orm/tursodatabase-sync"; import { matchKey } from "../shared/dataset-fields.ts"; -import type { ExecutionInput, PropDefinition } from "../shared/types.ts"; +import type { + ExecutionInput, + PropDefinition, + ResourceInstanceInput, + RunResources, +} from "../shared/types.ts"; import { type ReadOnlyQueryRunner, runReadOnlyQuery, @@ -77,12 +82,34 @@ function rowToDataset(row: typeof datasets.$inferSelect): Dataset { function mergedCells( cells: Record, ): SQL | undefined { - const entries = Object.entries(cells); + return mergedKeys(sql`${datasetRows.cells}`, cells); +} + +/** + * {@link mergedCells} for the `resources` column, which may be NULL: a row + * that declared none. Instance names are `[A-Za-z0-9_-]` (checked by the + * caller), so they too need no escaping inside a quoted path. + */ +function mergedResources( + resources: Record | undefined, +): SQL | undefined { + return mergedKeys( + sql`coalesce(${datasetRows.resources}, jsonb('{}'))`, + resources ?? {}, + ); +} + +/** Sets and removes whole top-level keys of the JSONB object `column`. */ +function mergedKeys( + column: SQL, + values: Record, +): SQL | undefined { + const entries = Object.entries(values); if (entries.length === 0) return undefined; const set = entries.filter(([, cell]) => cell !== null); const cleared = entries.filter(([, cell]) => cell === null); const path = (fieldId: string) => `$."${fieldId}"`; - let expr: SQL = sql`${datasetRows.cells}`; + let expr: SQL = column; if (set.length > 0) { expr = sql`jsonb_set(${expr}, ${sql.join( set.map( @@ -236,6 +263,7 @@ export class TursoDatasetProvider implements DatasetProvider { id: datasetRows.id, cells: jsonColumn(datasetRows.cells), source: jsonColumn(datasetRows.source), + resources: jsonColumn(datasetRows.resources), createdAt: datasetRows.createdAt, }) .from(datasetRows) @@ -248,12 +276,19 @@ export class TursoDatasetProvider implements DatasetProvider { .limit(limit ?? Number.MAX_SAFE_INTEGER) .offset(offset); - return rows.map(row => ({ - id: row.id, - cells: parseJson>(row.cells) ?? {}, - ...(row.source && { source: parseJson(row.source) }), - createdAt: row.createdAt, - })); + return rows.map(row => { + const resources = row.resources + ? parseJson(row.resources) + : undefined; + return { + id: row.id, + cells: parseJson>(row.cells) ?? {}, + ...(row.source && { source: parseJson(row.source) }), + ...(resources && + Object.keys(resources).length > 0 && { resources }), + createdAt: row.createdAt, + }; + }); } async describeRows(datasetId: string): Promise { @@ -262,8 +297,8 @@ export class TursoDatasetProvider implements DatasetProvider { .from(datasetRows) .where(eq(datasetRows.datasetId, datasetId)); - // One level into each cell, wherever that level lives: a resource's - // `args`, an object's `properties`, or those of a typed-in object value. + // One level into each cell, wherever that level lives: an object's + // `properties`, or those of a typed-in object value. // They share a namespace per field, since a table shows any of them as // columns under the field. Ordered by the first row a key appears in, // then its position there — near enough to declaration order. @@ -274,7 +309,6 @@ export class TursoDatasetProvider implements DatasetProvider { json_each( field.value, CASE json_extract(field.value, '$.kind') - WHEN 'resource' THEN '$.args' WHEN 'object' THEN '$.properties' ELSE '$.value.properties' END @@ -284,29 +318,9 @@ export class TursoDatasetProvider implements DatasetProvider { ORDER BY field.key, min(${datasetRows}.rowid), min(inner_key.id) `); - // Whether each field is ever filled by a resource — over every cell, - // including those of a resource that takes no arguments. - const resources = await this.db.all<{ - field: string; - resource: number; - }>(sql` - SELECT - field.key AS field, - max(json_extract(field.value, '$.kind') = 'resource') AS resource - FROM ${datasetRows}, json_each(${datasetRows.cells}) AS field - WHERE ${datasetRows.datasetId} = ${datasetId} - GROUP BY field.key - `); - const isResource = new Set( - resources.filter(r => Number(r.resource) === 1).map(r => r.field), - ); - const fields: Record = {}; for (const { field, key } of found) { - fields[field] ??= { - keys: [], - ...(isResource.has(field) && { resource: true }), - }; + fields[field] ??= { keys: [] }; fields[field].keys.push(key); } return { rowCount: Number(count), fields }; @@ -404,7 +418,7 @@ export class TursoDatasetProvider implements DatasetProvider { ); const now = Date.now(); - const minted: DatasetRow[] = rows.map(({ cells, source }) => { + const minted: DatasetRow[] = rows.map(({ cells, source, resources }) => { // Absent cells aren't stored: a missing field is a missing key. const present = Object.entries(cells).filter( ([, cell]) => cell !== undefined && cell !== null, @@ -420,6 +434,8 @@ export class TursoDatasetProvider implements DatasetProvider { id: mintRowId(), cells: Object.fromEntries(present), ...(source && { source }), + ...(resources && + Object.keys(resources).length > 0 && { resources }), createdAt: now, }; }); @@ -431,6 +447,7 @@ export class TursoDatasetProvider implements DatasetProvider { datasetId, cells: r.cells, source: r.source ?? null, + resources: r.resources ?? null, createdAt: r.createdAt, })), ); @@ -452,7 +469,11 @@ export class TursoDatasetProvider implements DatasetProvider { ): Promise { // A batch that names no cells changes nothing, so it's not recorded: // no `updatedAt` bump, no change event. - const changes = updates.some(u => Object.keys(u.cells).length > 0); + const changes = updates.some( + u => + Object.keys(u.cells).length > 0 || + Object.keys(u.resources ?? {}).length > 0, + ); await this.serializeWrite(() => this.db.transaction(async tx => { const [row] = await tx @@ -466,7 +487,7 @@ export class TursoDatasetProvider implements DatasetProvider { // Everything is checked before anything is written, so a bad update // anywhere in the batch leaves every row as it was. - for (const { cells } of updates) { + for (const { cells, resources } of updates) { for (const fieldId of Object.keys(cells)) { if (!fieldIds.has(fieldId)) { throw new DatasetValidationError( @@ -474,6 +495,13 @@ export class TursoDatasetProvider implements DatasetProvider { ); } } + for (const name of Object.keys(resources ?? {})) { + if (!/^[A-Za-z_][A-Za-z0-9_-]*$/.test(name)) { + throw new DatasetValidationError( + `"${name}" is not a valid resource name`, + ); + } + } } const rowIds = [...new Set(updates.map(u => u.rowId))]; if (rowIds.length > 0) { @@ -495,12 +523,16 @@ export class TursoDatasetProvider implements DatasetProvider { } } - for (const { rowId, cells } of updates) { + for (const { rowId, cells, resources } of updates) { const merged = mergedCells(cells); - if (!merged) continue; + const mergedRes = mergedResources(resources); + if (!merged && !mergedRes) continue; await tx .update(datasetRows) - .set({ cells: merged }) + .set({ + ...(merged && { cells: merged }), + ...(mergedRes && { resources: mergedRes }), + }) .where( and( eq(datasetRows.id, rowId), diff --git a/src/eval/eval-runner.ts b/src/eval/eval-runner.ts index 1946c98..f84c09e 100644 --- a/src/eval/eval-runner.ts +++ b/src/eval/eval-runner.ts @@ -579,6 +579,11 @@ export class EvalRunner { }, ), executeInputs: inputs.executeInputs, + // The eval's instances and the row's, in one namespace — the row's + // win on a name both declare (`specs/resource-instances.md` §E). + ...((inputs.resources || job.row.resources) && { + resources: { ...inputs.resources, ...job.row.resources }, + }), }, { tracer: this.options.tracer, diff --git a/src/eval/eval-types.ts b/src/eval/eval-types.ts index 095fd92..528b944 100644 --- a/src/eval/eval-types.ts +++ b/src/eval/eval-types.ts @@ -10,6 +10,7 @@ import type { ExecutionInput, PromptRef, + RunResources, VariationConflict, VariationId, VersionId, @@ -36,14 +37,20 @@ export interface EvalCheck { /** * One binding per prompt slot, keyed by slot name, in the shape the execute * panel persists. A binding may name a column (`dataset`), another slot - * (`input`), a resource with arguments, a typed-in value, or an object - * mixing those. + * (`input`), a resource instance (`instance`), a typed-in value, or an + * object mixing those. */ export interface EvalInputs { /** Function parameter name → binding. */ functionInputs: Record; /** Execute parameter name → binding. */ executeInputs: Record; + /** + * Resource instances every row's run declares, merged with the row's own + * (`DatasetRow.resources`) — the row's win on a name both declare. See + * `specs/resource-instances.md` §E. + */ + resources?: RunResources; } /** A saved eval: data, stored by an `EvalProvider`. See `specs/evals.md` §A. */ diff --git a/src/index.ts b/src/index.ts index 30581d3..d4a13d4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -103,7 +103,6 @@ export { LocalFileProvider } from "./file-provider-local.ts"; export { MemoryFileProvider } from "./file-provider-memory.ts"; export { OverlayFileProvider } from "./file-provider-overlay.ts"; export { - canonicalArgumentKey, collectInputSlots, findInputCycle, type InputBindings, @@ -114,7 +113,7 @@ export { matchSourcesToSlots, namedBindings, type ResolutionContext, - type ResourceResolver, + type InstanceResolver, resolveExecutionInput, resolveExecutionInputs, stampReceipts, @@ -165,7 +164,9 @@ export { type StaticResourceDefinition, } from "./prompt/playground/resource.ts"; export { + type DeclaredInstance, DEFAULT_PLAYGROUND_INCLUDE_PATTERNS, + isValidInstanceName, type PlaygroundModuleError, type RegisteredCheck, type RegisteredResource, diff --git a/src/mcp/eval-tools.ts b/src/mcp/eval-tools.ts index 5979de2..25c0593 100644 --- a/src/mcp/eval-tools.ts +++ b/src/mcp/eval-tools.ts @@ -49,6 +49,7 @@ import type { ExecutionInput, NormalizedPrompt, PromptID, + RunResources, } from "../shared/types.ts"; import { columnNames, @@ -123,22 +124,9 @@ function bindingFrom( ), }; } - case "resource": { - const args = binding.args as Record | undefined; - return { - ...(binding as ExecutionInput & { kind: "resource" }), - ...(args && { - args: Object.fromEntries( - Object.entries(args).map(([k, v]) => [ - k, - bindingFrom(v, fields, `${path}.args.${k}`), - ]), - ), - }), - }; - } default: - // A typed-in value or a slot reference: checked when it's saved. + // A typed-in value, a slot reference or a resource instance: checked + // when it's saved. return binding as ExecutionInput; } } @@ -162,15 +150,24 @@ function describeBinding( kind: "object", properties: describeBindings(input.properties, fields), }; - case "resource": - return input.args - ? { ...input, args: describeBindings(input.args, fields) } - : input; default: return input; } } +/** A stored eval's resource instances as a tool shows them, their arguments described as bindings are. */ +function describeResources( + resources: RunResources, + fields: readonly DatasetField[] | undefined, +): Record { + return Object.fromEntries( + Object.entries(resources).map(([name, spec]) => [ + name, + spec.args ? { ...spec, args: describeBindings(spec.args, fields) } : spec, + ]), + ); +} + function describeBindings( bindings: Record, fields: readonly DatasetField[] | undefined, @@ -195,13 +192,26 @@ function checkIdFor(uri: string, taken: Set): string { const binding = z .record(z.string(), z.unknown()) .describe( - 'How to fill a slot: {column: ""} for the row\'s value (a field sharing its name with an earlier one goes by its query column, e.g. "city#3"), {json: } for a fixed value, {kind: "input", half: "function" | "execute", path} for what the run bound to one of the prompt\'s own slots, {kind: "object", properties: {name: binding}} to fill an object property by property, or {kind: "resource", uri, args?: {name: binding}}.', + 'How to fill a slot: {column: ""} for the row\'s value (a field sharing its name with an earlier one goes by its query column, e.g. "city#3"), {json: } for a fixed value, {kind: "input", half: "function" | "execute", path} for what the run bound to one of the prompt\'s own slots, {kind: "object", properties: {name: binding}} to fill an object property by property, or {kind: "instance", name, output?} for one of the eval\'s (or each row\'s) named resource instances, or an output of it.', ); const bindings = z .record(z.string(), binding) .describe("Parameter name → binding."); +const resourceSpec = z + .object({ + uri: z + .string() + .describe("The resource's uri (`#`), never an output's."), + args: bindings + .optional() + .describe("The resource's argument name → binding."), + }) + .describe( + "A resource instance every row's run creates, whether or not a slot names it. A row's own instance of the same name replaces it.", + ); + const checkSpec = z.object({ id: z .string() @@ -317,6 +327,9 @@ export function registerEvalTools( inputs: { functionInputs: describeBindings(def.inputs.functionInputs, fields), executeInputs: describeBindings(def.inputs.executeInputs, fields), + ...(def.inputs.resources && { + resources: describeResources(def.inputs.resources, fields), + }), }, checks: def.checks.map(check => ({ id: check.id, @@ -346,6 +359,28 @@ export function registerEvalTools( return out; }; + const resourcesFrom = ( + given: Record | null> | undefined, + current: RunResources | undefined, + fields: readonly DatasetField[] | undefined, + ): RunResources | undefined => { + if (!given) return current; + const out: RunResources = { ...current }; + for (const [name, spec] of Object.entries(given)) { + if (spec === null) { + delete out[name]; + continue; + } + out[name] = { + uri: spec.uri, + ...(spec.args && { + args: bindingsFrom(spec.args, {}, fields, `resources.${name}.args`), + }), + }; + } + return Object.keys(out).length > 0 ? out : undefined; + }; + const checksFrom = ( given: z.infer[], fields: readonly DatasetField[] | undefined, @@ -487,6 +522,12 @@ export function registerEvalTools( executeInputs: bindings .optional() .describe("Execute parameter name → binding."), + resources: z + .record(z.string(), resourceSpec) + .optional() + .describe( + "Named resource instances (letters, digits, '_' and '-') every row's run creates. Bindings name them with {kind: \"instance\", name, output?}.", + ), checks: z.array(checkSpec).optional(), autoBind: autoBindParam, providerId: providerIdParam("eval"), @@ -515,6 +556,12 @@ export function registerEvalTools( "executeInputs", ), }; + const resources = resourcesFrom( + args.resources, + undefined, + setting.fields, + ); + if (resources) inputs.resources = resources; let checks = checksFrom(args.checks ?? [], setting.fields); if (args.autoBind !== false) { ({ inputs, checks } = autoBound(setting, inputs, checks)); @@ -553,6 +600,12 @@ export function registerEvalTools( .record(z.string(), binding.nullable()) .optional() .describe("Execute parameter name → binding, or null to unbind."), + resources: z + .record(z.string(), resourceSpec.nullable()) + .optional() + .describe( + "Resource instances to add or replace by name, or null to remove one; the rest are left.", + ), checks: z.array(checkSpec).optional(), autoBind: z .boolean() @@ -593,6 +646,12 @@ export function registerEvalTools( "executeInputs", ), }; + const resources = resourcesFrom( + args.resources, + current.inputs.resources, + setting.fields, + ); + if (resources) inputs.resources = resources; let checks = args.checks ? checksFrom(args.checks, setting.fields) : current.checks; @@ -619,6 +678,9 @@ export function registerEvalTools( proposed.inputs.executeInputs, args.executeInputs, ), + ...(proposed.inputs.resources && { + resources: proposed.inputs.resources, + }), }; } if (args.checks) checks = proposed.checks; diff --git a/src/mcp/server.test.ts b/src/mcp/server.test.ts index ccb4fba..e0bfadc 100644 --- a/src/mcp/server.test.ts +++ b/src/mcp/server.test.ts @@ -481,7 +481,7 @@ describe("MCP server", () => { expect(added.id).toBe("1"); }); - it("sets resource cells with update_rows, as add_rows does", async () => { + it("sets resource cells and row resources with update_rows, as add_rows does", async () => { const { call, datasets } = await setup(); const dataset = await call("create_dataset", { name: "Resources", @@ -494,21 +494,26 @@ describe("MCP server", () => { datasetId: dataset.id, rows: [{ values: { title: "Milk" } }], }); - const resource = { - kind: "resource", + const notes = { uri: "src/fixtures.ts#notes", args: { title: { kind: "value", value: { kind: "primitive", value: "Milk" } }, }, }; + const resource = { kind: "instance", name: "notes" }; await call("update_rows", { datasetId: dataset.id, updates: [ - { rowId: rowIds[0], cells: { title: resource, notes: resource } }, + { + rowId: rowIds[0], + cells: { title: resource, notes: resource }, + resources: { notes }, + }, ], }); const [row] = await datasets.listRows(dataset.id); expect(row.cells).toEqual({ "0": resource, "1": resource }); + expect(row.resources).toEqual({ notes }); }); it("updates and deletes rows, and renames and deletes fields", async () => { diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 749750a..ce16cd3 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -61,6 +61,7 @@ import { jsonToPropValue } from "../shared/json-prop-value.ts"; import type { AnnotationSource, ExecuteResponse, + RunResources, ExecutionInput, NormalizedPrompt, Span, @@ -75,6 +76,8 @@ import { describeField, describeRow, executionInput, + resourceInstance, + runResources, findField, findPrompt, findTrace, @@ -393,6 +396,7 @@ export function createMcpServer( .describe( "Execute parameters as unresolved inputs, in place of `executeArgs`.", ), + resources: runResources.optional(), wait: z .boolean() .optional() @@ -430,6 +434,9 @@ export function createMcpServer( valueInputs([value])[0], ]), ), + ...(args.resources && { + resources: args.resources as RunResources, + }), }, ); const response = unwrap(result); @@ -942,7 +949,13 @@ export function createMcpServer( .record(z.string(), executionInput) .optional() .describe( - "Field name → an unresolved input, for a value plain JSON can't express (e.g. a resource).", + "Field name → an unresolved input, for a value plain JSON can't express (e.g. a resource instance).", + ), + resources: z + .record(z.string(), resourceInstance.nullable()) + .optional() + .describe( + "The row's own resource instances, which its cells (and an eval's bindings) can name. On update, an instance set to null is removed and those left out are untouched.", ), }; @@ -966,6 +979,7 @@ export function createMcpServer( await handleAddRows(provider, datasetId, { rows: rows.map(row => ({ cells: cellsFrom(fields, row.values, row.cells), + ...(row.resources && { resources: row.resources }), })), }), ); @@ -995,6 +1009,7 @@ export function createMcpServer( updates: updates.map(update => ({ rowId: update.rowId, cells: cellsFrom(fields, update.values, update.cells), + ...(update.resources && { resources: update.resources }), })), }), { updated: updates.length }, diff --git a/src/mcp/tool-helpers.ts b/src/mcp/tool-helpers.ts index 47bc6b2..9470c85 100644 --- a/src/mcp/tool-helpers.ts +++ b/src/mcp/tool-helpers.ts @@ -259,5 +259,23 @@ export const providerIdParam = (kind: string) => export const executionInput = z .record(z.string(), z.unknown()) .describe( - 'An unresolved input: {kind: "value", value: PropValue}, {kind: "object", properties: {...}}, or {kind: "resource", uri, args?}.', + 'An unresolved input: {kind: "value", value: PropValue}, {kind: "object", properties: {...}}, or {kind: "instance", name, output?} naming one of the run\'s resource instances (or an output of it).', + ); + +/** One named resource instance, as a tool parameter. */ +export const resourceInstance = z.object({ + uri: z + .string() + .describe("The resource's uri (`#`), never an output's."), + args: z + .record(z.string(), executionInput) + .optional() + .describe("The resource's argument name → unresolved input."), +}); + +/** A run's named resource instances, as a tool parameter. */ +export const runResources = z + .record(z.string(), resourceInstance) + .describe( + "Named resource instances (letters, digits, '_' and '-'), each created once per run whether or not anything names it — so one can seed data purely as a side effect. Inputs (and other instances' args) name them with {kind: \"instance\", name, output?}.", ); diff --git a/src/prompt/execution-inputs.test.ts b/src/prompt/execution-inputs.test.ts index 605a08a..7c9eb00 100644 --- a/src/prompt/execution-inputs.test.ts +++ b/src/prompt/execution-inputs.test.ts @@ -2,13 +2,17 @@ // Copyright (c) 2026 Alexander Corrado import { describe, expect, it, vi } from "vitest"; -import type { ExecutionInput, PropDefinition } from "../shared/types.ts"; +import type { + ExecutionInput, + PropDefinition, + RunResources, +} from "../shared/types.ts"; import { - canonicalArgumentKey, collectInputSlots, findInputCycle, type InputBindings, type InputSource, + type InstanceResolver, inputReferenceProblems, matchSourcesToSlots, namedBindings, @@ -16,6 +20,45 @@ import { resolveExecutionInputs, stampReceipts, } from "./execution-inputs.ts"; +import type { DeclaredInstance } from "./playground/resource-registry.ts"; + +/** + * An in-memory {@link InstanceResolver}: memoizes by name the way the lease + * does, creating each instance with `create` — by default, a record of what + * it was declared with and the arguments it resolved to. + */ +function fakeResolver( + create: ( + name: string, + decl: DeclaredInstance, + args: Record | undefined, + ) => unknown = (name, decl, args) => ({ name, uri: decl.uri, args }), +) { + const declared = new Map(); + const memo = new Map>(); + const created: string[] = []; + const resolver: InstanceResolver = { + declare: async instances => { + for (const [name, decl] of Object.entries(instances)) { + declared.set(name, decl); + } + }, + acquire: async (name, output) => { + let pending = memo.get(name); + if (!pending) { + const decl = declared.get(name); + if (!decl) throw new Error(`No resource instance named '${name}'`); + created.push(name); + pending = (async () => + create(name, decl, await decl.binding?.resolve?.()))(); + memo.set(name, pending); + } + const value: any = await pending; + return output === undefined ? value : value?.[output]; + }, + }; + return { resolver, created, declared }; +} const str = (name: string, optional = false): PropDefinition => ({ name, @@ -166,6 +209,8 @@ describe("resolveExecutionInput", () => { }); it("grafts a resource into one field of an otherwise hand-edited object", async () => { + const { resolver } = fakeResolver(name => ({ handle: name })); + await resolver.declare({ db: { uri: "pg.ts#db" } }); const resolved = await resolveExecutionInput( { kind: "object", @@ -174,14 +219,14 @@ describe("resolveExecutionInput", () => { kind: "value", value: { kind: "primitive", value: "ws_1" }, }, - db: { kind: "resource", uri: "pg.ts#db" }, + db: { kind: "instance", name: "db" }, }, }, - async uri => ({ handle: uri }), + resolver, ); expect(resolved).toEqual({ workspaceId: "ws_1", - db: { handle: "pg.ts#db" }, + db: { handle: "db" }, }); }); @@ -213,255 +258,139 @@ describe("resolveExecutionInput", () => { it("explains itself when a resource is referenced with no resolver", async () => { await expect( - resolveExecutionInput({ kind: "resource", uri: "pg.ts#db" }), + resolveExecutionInput({ kind: "instance", name: "db" }), ).rejects.toThrow(/does not offer resources/); }); }); describe("resolveExecutionInputs", () => { - it("resolves both halves through one resolver, so a shared resource is created once", async () => { - const created = vi.fn(async (uri: string) => ({ uri })); - // A memoizing resolver is the registry's lease in production; here the - // point is only that both halves go through the *same* one. - const memo = new Map>(); - const resolveResource = (uri: string) => { - const existing = memo.get(uri); - if (existing) return existing; - const pending = created(uri); - memo.set(uri, pending); - return pending; - }; - + it("resolves both halves through one resolver, so a shared instance is created once", async () => { + const { resolver, created } = fakeResolver(); const { functionParams, executeValues } = await resolveExecutionInputs( { - functionInputs: [{ kind: "resource", uri: "pg.ts#db" }], + functionInputs: [{ kind: "instance", name: "db" }], executeInputs: { toolsContext: { kind: "object", - properties: { db: { kind: "resource", uri: "pg.ts#db" } }, + properties: { db: { kind: "instance", name: "db" } }, }, }, + resources: { db: { uri: "pg.ts#db" } }, }, - resolveResource, + resolver, ); - expect(created).toHaveBeenCalledTimes(1); + expect(created).toEqual(["db"]); expect(functionParams[0]).toBe(executeValues.toolsContext.db); }); - it("resolves a resource used both as an argument and as a prompt input to one instance per run", async () => { - const created = vi.fn(async (uri: string) => ({ uri })); - const memo = new Map>(); - const resolveResource = (uri: string) => { - const existing = memo.get(uri); - if (existing) return existing; - const pending = created(uri); - memo.set(uri, pending); - return pending; - }; - - const dbRef: ExecutionInput = { kind: "resource", uri: "pg.ts#db" }; - const { functionParams, executeValues } = await resolveExecutionInputs( + it("creates every declared instance, even one no slot names", async () => { + const { resolver, created } = fakeResolver(); + await resolveExecutionInputs( { - // `db` filling a prompt slot directly... - functionInputs: [dbRef], - // ...and filling an argument of a different resource. - executeInputs: { - seeded: { - kind: "resource", - uri: "pg.ts#seeded", - args: { db: dbRef }, - }, + functionInputs: [], + resources: { + db: { uri: "pg.ts#db" }, + seeded: { uri: "pg.ts#seeded" }, }, }, - resolveResource, + resolver, ); - - expect(created).toHaveBeenCalledTimes(2); // db, seeded — not db twice - expect(functionParams[0]).toBe(await memo.get("pg.ts#db")); - expect(executeValues.seeded).toEqual({ uri: "pg.ts#seeded" }); - }); -}); - -describe("canonicalArgumentKey", () => { - it("encodes absent args and an empty object identically", () => { - expect(canonicalArgumentKey(undefined)).toBe(""); - expect(canonicalArgumentKey({})).toBe(""); + expect(created.sort()).toEqual(["db", "seeded"]); }); - it("is insensitive to key order", () => { - const a = canonicalArgumentKey({ - title: { kind: "value", value: { kind: "primitive", value: "x" } }, - status: { kind: "value", value: { kind: "primitive", value: "y" } }, - }); - const b = canonicalArgumentKey({ - status: { kind: "value", value: { kind: "primitive", value: "y" } }, - title: { kind: "value", value: { kind: "primitive", value: "x" } }, - }); - expect(a).toBe(b); - }); - - it("differs for genuinely different arguments", () => { - const a = canonicalArgumentKey({ - title: { kind: "value", value: { kind: "primitive", value: "x" } }, - }); - const b = canonicalArgumentKey({ - title: { kind: "value", value: { kind: "primitive", value: "y" } }, - }); - expect(a).not.toBe(b); - }); -}); - -describe("resource arguments and receipts on resolveExecutionInput (specs/resource-arguments.md §C, §D)", () => { - it("builds a binding whose key matches canonicalArgumentKey and resolves nested args recursively", async () => { - const resolveResource = vi.fn(async (uri: string, binding?: any) => ({ - uri, - args: binding ? await binding.resolve() : undefined, - })); - - const args = { - title: { - kind: "value" as const, - value: { kind: "primitive" as const, value: "Todo app" }, - }, - owner: { kind: "resource" as const, uri: "pg.ts#owner" }, - }; - const result: any = await resolveExecutionInput( - { kind: "resource", uri: "pg.ts#seeded", args }, - resolveResource, + it("resolves an instance's arguments, which may name another instance's output", async () => { + const { resolver } = fakeResolver((name, _decl, args) => + name === "root" ? { id: "tsk_root" } : { id: "tsk_child", ...args }, ); - - expect(resolveResource).toHaveBeenCalledWith( - "pg.ts#seeded", - expect.objectContaining({ key: canonicalArgumentKey(args) }), + const { functionParams } = await resolveExecutionInputs( + { + functionInputs: [{ kind: "instance", name: "child" }], + resources: { + root: { uri: "pg.ts#task" }, + child: { + uri: "pg.ts#task", + args: { + title: { + kind: "value", + value: { kind: "primitive", value: "Child" }, + }, + parentId: { kind: "instance", name: "root", output: "id" }, + }, + }, + }, + }, + resolver, ); - expect(result.args).toEqual({ - title: "Todo app", - owner: { uri: "pg.ts#owner", args: undefined }, + expect(functionParams[0]).toEqual({ + id: "tsk_child", + title: "Child", + parentId: "tsk_root", }); }); - it("calls the resolver with no binding at all for a reference with neither args nor a receipt", async () => { - const resolveResource = vi.fn(async (uri: string) => uri); - await resolveExecutionInput( - { kind: "resource", uri: "pg.ts#db" }, - resolveResource, + it("declares an instance with no arguments or receipt without a binding", async () => { + const { resolver, declared } = fakeResolver(); + await resolveExecutionInputs( + { resources: { db: { uri: "pg.ts#db" } } }, + resolver, ); - expect(resolveResource).toHaveBeenCalledWith("pg.ts#db"); - expect(resolveResource).toHaveBeenCalledTimes(1); - expect(resolveResource.mock.calls[0]).toHaveLength(1); + expect(declared.get("db")).toStrictEqual({ uri: "pg.ts#db" }); }); it("passes a recorded receipt through to the resolver on a replay", async () => { - const resolveResource = vi.fn( - async (_uri: string, binding?: any) => binding?.receipt, - ); - const result = await resolveExecutionInput( - { kind: "resource", uri: "pg.ts#seeded", receipt: { taskId: "tsk_abc" } }, - resolveResource, - ); - expect(result).toEqual({ taskId: "tsk_abc" }); - }); -}); - -describe("stampReceipts", () => { - it("sets a receipt on a resource reference by its uri when it has no arguments", () => { - const stamped = stampReceipts( - { functionInputs: [{ kind: "resource", uri: "pg.ts#db" }] }, - { "pg.ts#db": "db-receipt" }, - ); - expect(stamped.functionInputs?.[0]).toMatchObject({ - kind: "resource", - uri: "pg.ts#db", - receipt: "db-receipt", - }); - }); - - it("sets a receipt keyed by uri@key when the reference has arguments", () => { - const args = { - title: { - kind: "value" as const, - value: { kind: "primitive" as const, value: "Todo app" }, - }, - }; - const key = canonicalArgumentKey(args); - const stamped = stampReceipts( + const { resolver, declared } = fakeResolver(); + await resolveExecutionInputs( { - executeInputs: { - seeded: { kind: "resource", uri: "pg.ts#seeded", args }, + resources: { + seeded: { uri: "pg.ts#seeded", receipt: { taskId: "tsk_abc" } }, }, }, - { [`pg.ts#seeded@${key}`]: { taskId: "tsk_abc" } }, + resolver, ); - expect(stamped.executeInputs?.seeded).toMatchObject({ - receipt: { taskId: "tsk_abc" }, + expect(declared.get("seeded")?.binding?.receipt).toEqual({ + taskId: "tsk_abc", }); + expect(declared.get("seeded")?.binding?.resolve).toBeUndefined(); }); - it("stamps a resource nested inside an object input", () => { - const stamped = stampReceipts( - { - executeInputs: { - toolsContext: { - kind: "object", - properties: { db: { kind: "resource", uri: "pg.ts#db" } }, - }, - }, - }, - { "pg.ts#db": "db-receipt" }, - ); - expect(stamped.executeInputs?.toolsContext).toEqual({ - kind: "object", - properties: { - db: { - kind: "resource", - uri: "pg.ts#db", - args: undefined, - receipt: "db-receipt", - }, - }, - }); + it("explains itself when a run declares instances but the provider offers no resources", async () => { + await expect( + resolveExecutionInputs({ resources: { db: { uri: "pg.ts#db" } } }), + ).rejects.toThrow(/Cannot create resource 'db'.*does not offer resources/); }); +}); - it("stamps a resource nested inside another resource's own args", () => { - const args = { owner: { kind: "resource" as const, uri: "pg.ts#owner" } }; - const stamped = stampReceipts( - { - functionInputs: [{ kind: "resource", uri: "pg.ts#seeded", args }], +describe("stampReceipts", () => { + it("sets each instance's receipt by name", () => { + const resources: RunResources = { + db: { uri: "pg.ts#db" }, + seeded: { + uri: "pg.ts#seeded", + args: { + title: { kind: "value", value: { kind: "primitive", value: "T" } }, + }, }, - { "pg.ts#owner": "owner-receipt" }, - ); - const seeded = stamped.functionInputs?.[0] as Extract< - ExecutionInput, - { kind: "resource" } - >; - expect(seeded.args?.owner).toMatchObject({ receipt: "owner-receipt" }); + }; + expect( + stampReceipts(resources, { + db: "db-receipt", + seeded: { taskId: "tsk_abc" }, + }), + ).toEqual({ + db: { uri: "pg.ts#db", receipt: "db-receipt" }, + seeded: { ...resources.seeded, receipt: { taskId: "tsk_abc" } }, + }); }); it("drops a replayed receipt that this run didn't reproduce", () => { - const stamped = stampReceipts( - { - functionInputs: [ - { kind: "resource", uri: "pg.ts#db", receipt: "old-receipt" }, - ], - }, - {}, - ); - expect(stamped.functionInputs?.[0]).toStrictEqual({ - kind: "resource", - uri: "pg.ts#db", - }); + expect( + stampReceipts({ db: { uri: "pg.ts#db", receipt: "old-receipt" } }, {}), + ).toStrictEqual({ db: { uri: "pg.ts#db" } }); }); - it("leaves inputs untouched when no receipts were produced", () => { - const inputs = { - functionInputs: [{ kind: "resource" as const, uri: "pg.ts#db" }], - }; - expect(stampReceipts(inputs, undefined)).toBe(inputs); - expect(stampReceipts(inputs, {})).not.toBe(inputs); - expect(stampReceipts(inputs, {}).functionInputs?.[0]).toMatchObject({ - uri: "pg.ts#db", - }); + it("returns nothing for a run that declared no instances", () => { + expect(stampReceipts(undefined, { db: "r" })).toBeUndefined(); }); }); @@ -475,10 +404,11 @@ describe("dataset and input references", () => { const row = { cells: { "0": text("Set up CI"), - "1": { kind: "resource", uri: "pg.ts#db" } as ExecutionInput, + "1": { kind: "instance", name: "db" } as ExecutionInput, }, }; - const resolve = vi.fn(async (uri: string) => `created:${uri}`); + const { resolver: resolve } = fakeResolver(name => `created:${name}`); + await resolve.declare({ db: { uri: "pg.ts#db" } }); expect( await resolveExecutionInput({ kind: "dataset", field: "0" }, resolve, { row, @@ -488,7 +418,7 @@ describe("dataset and input references", () => { await resolveExecutionInput({ kind: "dataset", field: "1" }, resolve, { row, }), - ).toBe("created:pg.ts#db"); + ).toBe("created:db"); }); it("resolves a column the row leaves empty to nothing", async () => { @@ -501,16 +431,20 @@ describe("dataset and input references", () => { it("binds a resource argument to a column", async () => { const seen: unknown[] = []; - await resolveExecutionInput( + const { resolver } = fakeResolver((_name, _decl, args) => { + seen.push(args); + return "task"; + }); + await resolveExecutionInputs( { - kind: "resource", - uri: "pg.ts#seededTask", - args: { title: { kind: "dataset", field: "0" } }, - }, - async (_uri, binding) => { - seen.push(await binding?.resolve()); - return "task"; + resources: { + task: { + uri: "pg.ts#seededTask", + args: { title: { kind: "dataset", field: "0" } }, + }, + }, }, + resolver, { row: { cells: { "0": text("Plan it") } } }, ); expect(seen).toEqual([{ title: "Plan it" }]); @@ -519,7 +453,7 @@ describe("dataset and input references", () => { it("resolves an input reference to the other slot's value, through the same resolver", async () => { const bindings: InputBindings = { functionInputs: { - taskId: { kind: "resource", uri: "pg.ts#seeded.taskId" }, + taskId: { kind: "instance", name: "seeded", output: "taskId" }, }, executeInputs: { toolsContext: { @@ -535,23 +469,19 @@ describe("dataset and input references", () => { }, }, }; - // The lease memoizes; the stand-in counts how often it's asked. - const calls: string[] = []; - const resolve = async (uri: string) => { - calls.push(uri); - return "tsk_1"; - }; + const { resolver, created } = fakeResolver(() => ({ taskId: "tsk_1" })); const { functionParams, executeValues } = await resolveExecutionInputs( { functionInputs: [bindings.functionInputs.taskId], executeInputs: bindings.executeInputs, + resources: { seeded: { uri: "pg.ts#seeded" } }, }, - resolve, + resolver, { bindings }, ); expect(functionParams).toEqual(["tsk_1"]); expect(executeValues.toolsContext.list_tasks.rootTaskId).toBe("tsk_1"); - expect(calls.every(uri => uri === "pg.ts#seeded.taskId")).toBe(true); + expect(created).toEqual(["seeded"]); }); it("reads a nested path through a typed-in object, and off a resolved value", async () => { @@ -564,11 +494,12 @@ describe("dataset and input references", () => { properties: { title: { kind: "primitive", value: "T" } }, }, }, - seeded: { kind: "resource", uri: "pg.ts#seeded" }, + seeded: { kind: "instance", name: "seeded" }, }, executeInputs: {}, }; - const resolve = async () => ({ taskId: "tsk_9" }); + const { resolver: resolve } = fakeResolver(() => ({ taskId: "tsk_9" })); + await resolve.declare({ seeded: { uri: "pg.ts#seeded" } }); expect( await resolveExecutionInput( { kind: "input", half: "function", path: "info.title" }, @@ -649,18 +580,20 @@ describe("findInputCycle", () => { ).toEqual(["a.x", "a.x"]); }); - it("follows references in a resource's arguments", () => { + it("follows references in a resource instance's arguments", () => { expect( findInputCycle({ - functionInputs: { - taskId: { - kind: "resource", + resources: { + seeded: { uri: "pg.ts#seeded", args: { root: { kind: "input", half: "execute", path: "ctx.root" }, }, }, }, + functionInputs: { + taskId: { kind: "instance", name: "seeded" }, + }, executeInputs: { ctx: { kind: "object", @@ -673,6 +606,41 @@ describe("findInputCycle", () => { ).toBeDefined(); }); + it("finds two instances that each take the other's output", () => { + expect( + findInputCycle({ + resources: { + a: { + uri: "pg.ts#task", + args: { parentId: { kind: "instance", name: "b", output: "id" } }, + }, + b: { + uri: "pg.ts#task", + args: { parentId: { kind: "instance", name: "a", output: "id" } }, + }, + }, + functionInputs: {}, + executeInputs: {}, + }), + ).toEqual(["resource 'a'", "resource 'b'", "resource 'a'"]); + }); + + it("accepts one instance taking another of the same resource's output", () => { + expect( + findInputCycle({ + resources: { + root: { uri: "pg.ts#task" }, + child: { + uri: "pg.ts#task", + args: { parentId: { kind: "instance", name: "root", output: "id" } }, + }, + }, + functionInputs: { taskId: { kind: "instance", name: "child" } }, + executeInputs: {}, + }), + ).toBeUndefined(); + }); + it("accepts references that don't loop", () => { expect( findInputCycle({ @@ -705,4 +673,47 @@ describe("inputReferenceProblems", () => { expect(problems[0]).toMatch(/'missing'/); expect(problems[1]).toMatch(/cycle: b → c → b/); }); + + it("names an instance the run doesn't declare, unless rows may declare it", () => { + const bindings: InputBindings = { + functionInputs: { a: { kind: "instance", name: "db" } }, + executeInputs: {}, + resources: { + seeded: { + uri: "pg.ts#seeded", + args: { db: { kind: "instance", name: "other" } }, + }, + }, + }; + const signature = { functionParameters: [str("a")] }; + expect(inputReferenceProblems(bindings, signature)).toEqual([ + "'a' names resource 'db', which this run doesn't declare", + "resource 'seeded' names resource 'other', which this run doesn't declare", + ]); + expect( + inputReferenceProblems(bindings, signature, { + undeclaredInstances: true, + }), + ).toEqual([]); + }); + + it("checks an `input` reference among an instance's arguments", () => { + expect( + inputReferenceProblems( + { + functionInputs: {}, + executeInputs: {}, + resources: { + seeded: { + uri: "pg.ts#seeded", + args: { title: { kind: "input", half: "function", path: "nope" } }, + }, + }, + }, + { functionParameters: [str("a")] }, + ), + ).toEqual([ + "resource 'seeded' names input 'nope', which the prompt doesn't have", + ]); + }); }); diff --git a/src/prompt/execution-inputs.ts b/src/prompt/execution-inputs.ts index 0fa6ea5..027c9c8 100644 --- a/src/prompt/execution-inputs.ts +++ b/src/prompt/execution-inputs.ts @@ -8,11 +8,13 @@ import { type InputSlot, inputKey, } from "../shared/input-references.ts"; -import type { ExecutionInput, PropType, PropValue } from "../shared/types.ts"; -import { - type ResourceBinding, - receiptKeyOf, -} from "./playground/resource-registry.ts"; +import type { + ExecutionInput, + PropType, + PropValue, + RunResources, +} from "../shared/types.ts"; +import type { DeclaredInstance } from "./playground/resource-registry.ts"; export { collectInputSlots, @@ -112,51 +114,15 @@ export function matchSourcesToSlots( } /** - * How an {@link ExecutionInput} of kind `resource` is turned into a value. - * - * `binding` is passed when the resource has arguments and/or a replay - * receipt to hand back — see {@link ResourceBinding} and - * `specs/resource-arguments.md` §D. A resolver that ignores it (as any - * resolver predating arguments does) still gets the resource's plain value, - * unparameterized, the same as before. - */ -export type ResourceResolver = ( - uri: string, - binding?: ResourceBinding, -) => Promise; - -/** - * The stable JSON encoding of a resource reference's `args` — object keys - * sorted recursively, computed over the *unresolved* recipe so a resolved - * argument that turns out to be a live handle never has to be compared or - * hashed. Absent `args` and `{}` both encode to `""`, which is what makes an - * existing argument-free reference key identically to before this existed. - * - * This is the lease's memoization key for one (resource, arguments) pair — - * see `specs/resource-arguments.md` §D. + * How a run's named resource instances are created — what + * `ResourceRegistry.lease()` hands back satisfies it. See + * `specs/resource-instances.md` §B. */ -export function canonicalArgumentKey( - args: Record | undefined, -): string { - if (!args || Object.keys(args).length === 0) return ""; - return stableStringify(args); -} - -/** `JSON.stringify`, but with every object's keys sorted first. */ -function stableStringify(value: unknown): string { - if (value === null || typeof value !== "object") { - return JSON.stringify(value); - } - if (Array.isArray(value)) { - return `[${value.map(stableStringify).join(",")}]`; - } - const keys = Object.keys(value).sort(); - return `{${keys - .map( - k => - `${JSON.stringify(k)}:${stableStringify((value as Record)[k])}`, - ) - .join(",")}}`; +export interface InstanceResolver { + /** Declares the run's instances. Called once, before anything is acquired. */ + declare(instances: Record): Promise; + /** The declared instance `name`'s value, or its `output`. Created once per run. */ + acquire(name: string, output?: string): Promise; } /** @@ -219,16 +185,17 @@ export function bindingAt( * created in the process that will run the prompt. * * @param input - The unresolved input. - * @param resolveResource - Creates the value behind a `resource` reference. + * @param resolver - Creates the run's resource instances, for `instance` + * references. Without one, an `instance` reference fails. * @param context - The row and bindings `dataset` and `input` references * resolve against. See {@link ResolutionContext}. */ export function resolveExecutionInput( input: ExecutionInput, - resolveResource?: ResourceResolver, + resolver?: InstanceResolver, context?: ResolutionContext, ): Promise { - return resolveWithin(input, resolveResource, context ?? {}, []); + return resolveWithin(input, resolver, context ?? {}, []); } /** @@ -238,12 +205,12 @@ export function resolveExecutionInput( */ async function resolveWithin( input: ExecutionInput, - resolveResource: ResourceResolver | undefined, + resolver: InstanceResolver | undefined, context: ResolutionContext, chain: readonly string[], ): Promise { const recurse = (child: ExecutionInput, nextChain = chain) => - resolveWithin(child, resolveResource, context, nextChain); + resolveWithin(child, resolver, context, nextChain); switch (input.kind) { case "value": @@ -258,36 +225,13 @@ async function resolveWithin( return Object.fromEntries(entries); } - case "resource": { - if (!resolveResource) { + case "instance": { + if (!resolver) { throw new Error( - `Cannot resolve resource '${input.uri}': this provider does not offer resources.`, + `Cannot resolve resource '${input.name}': this provider does not offer resources.`, ); } - // No binding at all — not even one carrying an empty `args` — for a - // reference with neither arguments nor a receipt, so a lease sees - // exactly the call it would have seen before either existed. - if (!input.args && input.receipt === undefined) { - return resolveResource(input.uri); - } - const binding: ResourceBinding = { - key: canonicalArgumentKey(input.args), - // Recursion is free: an argument is itself an `ExecutionInput`, so - // resolving the whole `args` map is exactly resolving an `object` - // input's `properties` — see the `object` case above. Evaluated - // lazily, only once the lease has decided this binding doesn't hit - // its memo (see `ResourceBinding.resolve`). - resolve: async () => { - const entries = await Promise.all( - Object.entries(input.args ?? {}).map( - async ([k, v]) => [k, await recurse(v)] as const, - ), - ); - return Object.fromEntries(entries); - }, - receipt: input.receipt, - }; - return resolveResource(input.uri, binding); + return resolver.acquire(input.name, input.output); } case "dataset": { @@ -324,10 +268,13 @@ async function resolveWithin( /** * Resolve a whole execute request's inputs. * - * Both halves are resolved together, and `resolveResource` is expected to - * memoize: a run-scoped resource referenced by both a function input and an - * execute input must be created **once** per run, which is only decidable when - * every input is in view. + * The run's named resource instances (`inputs.resources`) are declared to + * `resolver` and **all** created first — whether or not any slot names + * them, which is what lets an instance exist purely for its side effects + * (`specs/resource-instances.md` §B). Each instance's arguments resolve + * under the same `context`, so they may name another instance's output, + * another slot, or a dataset cell. Slots then resolve, and an `instance` + * reference among them gets the instance already created. * * @param context - What `dataset` and `input` references resolve against. */ @@ -335,23 +282,39 @@ export async function resolveExecutionInputs( inputs: { functionInputs?: readonly ExecutionInput[]; executeInputs?: Record; + resources?: RunResources; }, - resolveResource?: ResourceResolver, + resolver?: InstanceResolver, context?: ResolutionContext, ): Promise<{ functionParams: any[]; executeValues: Record }> { + const resources = Object.entries(inputs.resources ?? {}); + if (resources.length > 0) { + if (!resolver) { + throw new Error( + `Cannot create resource '${resources[0][0]}': this provider does not offer resources.`, + ); + } + await resolver.declare( + Object.fromEntries( + resources.map(([name, spec]) => [ + name, + declaredInstance(spec, resolver, context), + ]), + ), + ); + await Promise.all(resources.map(([name]) => resolver.acquire(name))); + } + const functionParams = await Promise.all( (inputs.functionInputs ?? []).map(i => - resolveExecutionInput(i, resolveResource, context), + resolveExecutionInput(i, resolver, context), ), ); const executeEntries = await Promise.all( Object.entries(inputs.executeInputs ?? {}).map( async ([name, input]) => - [ - name, - await resolveExecutionInput(input, resolveResource, context), - ] as const, + [name, await resolveExecutionInput(input, resolver, context)] as const, ), ); @@ -362,59 +325,55 @@ export async function resolveExecutionInputs( } /** - * Returns `inputs` with every resource reference's `receipt` set from - * `receipts` — the shape `ResourceLease.receipts()` / - * `ResolvedPromptInputs.receipts` produce — so the inputs recorded on a - * trace carry what this run's resources actually produced. - * - * A receipt is looked up by the same key `ResourceRegistry` records it - * under: the reference's own `uri`, or `` `${uri}@${key}` `` where `key` is - * {@link canonicalArgumentKey} of its (already-unresolved) `args` — so a - * fresh run's receipt lands on the exact reference that produced it, however - * many differently-bound references to the same resource a request has. - * Nested references (an argument that is itself a resource, a resource - * inside an `object` input) are stamped too. See - * `specs/resource-arguments.md` §K. + * What the lease is told about one instance: its arguments as a thunk — + * resolved only once the instance is actually created, under the run's own + * context — and its replay receipt. + */ +function declaredInstance( + spec: RunResources[string], + resolver: InstanceResolver, + context: ResolutionContext | undefined, +): DeclaredInstance { + const args = Object.entries(spec.args ?? {}); + if (args.length === 0 && spec.receipt === undefined) return { uri: spec.uri }; + return { + uri: spec.uri, + binding: { + ...(args.length > 0 && { + resolve: async () => + Object.fromEntries( + await Promise.all( + args.map( + async ([k, v]) => + [k, await resolveExecutionInput(v, resolver, context)] as const, + ), + ), + ), + }), + ...(spec.receipt !== undefined && { receipt: spec.receipt }), + }, + }; +} + +/** + * Returns `resources` with each instance's `receipt` set from `receipts` — + * the by-name shape `ResourceLease.receipts()` produces — so the inputs + * recorded on a trace carry what this run's resources actually produced. An + * incoming receipt (a replay's) never outlives a run whose `create` didn't + * produce one. See `specs/resource-arguments.md` §K. * - * @param inputs - The unresolved inputs a request was sent with. + * @param resources - The instances a request was sent with. * @param receipts - What the run's resolution produced, if anything did. */ export function stampReceipts( - inputs: { - functionInputs?: readonly ExecutionInput[]; - executeInputs?: Record; - }, + resources: RunResources | undefined, receipts: Record | undefined, -): { - functionInputs?: readonly ExecutionInput[]; - executeInputs?: Record; -} { - if (!receipts) return inputs; - - const stampAll = (map: Record) => - Object.fromEntries(Object.entries(map).map(([k, v]) => [k, stamp(v)])); - - const stamp = (input: ExecutionInput): ExecutionInput => { - switch (input.kind) { - case "resource": { - // Built fresh rather than spread from `input`, so a replay's incoming - // receipt never outlives a run whose `create` didn't produce one. - const stamped: ExecutionInput = { kind: "resource", uri: input.uri }; - if (input.args) stamped.args = stampAll(input.args); - const receipt = - receipts[receiptKeyOf(input.uri, canonicalArgumentKey(input.args))]; - if (receipt !== undefined) stamped.receipt = receipt; - return stamped; - } - case "object": - return { ...input, properties: stampAll(input.properties) }; - default: - return input; - } - }; - - return { - functionInputs: inputs.functionInputs?.map(stamp), - executeInputs: inputs.executeInputs && stampAll(inputs.executeInputs), - }; +): RunResources | undefined { + if (!resources) return undefined; + return Object.fromEntries( + Object.entries(resources).map(([name, { receipt: _, ...spec }]) => { + const receipt = receipts?.[name]; + return [name, receipt === undefined ? spec : { ...spec, receipt }]; + }), + ); } diff --git a/src/prompt/file/file-prompt-provider.test.ts b/src/prompt/file/file-prompt-provider.test.ts index b6a0907..7956e24 100644 --- a/src/prompt/file/file-prompt-provider.test.ts +++ b/src/prompt/file/file-prompt-provider.test.ts @@ -736,7 +736,8 @@ describe("FilePromptProvider resource inputs", () => { it("resolves a resource reference into the value execute receives", async () => { const provider = withResource(); const resolved = await provider.resolveInputs("x.prompt.ts#greet", { - functionInputs: [{ kind: "resource", uri: "x.playground.ts#taskId" }], + functionInputs: [{ kind: "instance", name: "task" }], + resources: { task: { uri: "x.playground.ts#taskId" } }, }); // `execute` still takes plain `any[]` — the provider contract did not @@ -744,9 +745,7 @@ describe("FilePromptProvider resource inputs", () => { expect(resolved.functionParams).toEqual(["tsk_seeded"]); // A receipt lets a trace display what the run used, even though replaying // it would mint a new one. - expect(resolved.receipts).toEqual({ - "x.playground.ts#taskId": "tsk_seeded", - }); + expect(resolved.receipts).toEqual({ task: "tsk_seeded" }); (globalThis as any).__disposed = 0; await resolved.release?.(); @@ -775,7 +774,8 @@ describe("FilePromptProvider resource inputs", () => { sdk: new VercelAISDK(), }); const resolved = await provider.resolveInputs("x.prompt.ts#greet", { - functionInputs: [{ kind: "resource", uri: ref }], + functionInputs: [{ kind: "instance", name: "task" }], + resources: { task: { uri: ref } }, }); expect(resolved.functionParams).toEqual(["tsk_1"]); } @@ -785,7 +785,8 @@ describe("FilePromptProvider resource inputs", () => { const provider = withResource(); await expect( provider.resolveInputs("x.prompt.ts#greet", { - functionInputs: [{ kind: "resource", uri: "x.playground.ts#missing" }], + functionInputs: [{ kind: "instance", name: "task" }], + resources: { task: { uri: "x.playground.ts#missing" } }, }), ).rejects.toThrow(/not found/); }); diff --git a/src/prompt/file/file-prompt-provider.ts b/src/prompt/file/file-prompt-provider.ts index 8f37ee1..a2238ff 100644 --- a/src/prompt/file/file-prompt-provider.ts +++ b/src/prompt/file/file-prompt-provider.ts @@ -16,6 +16,7 @@ import type { PromptInputSources, PromptStyle, PropDefinition, + RunResources, } from "../../shared/types.ts"; import type { PromptSpanInfo } from "../../trace/prompt-tracer.ts"; import { @@ -1239,16 +1240,13 @@ export class FilePromptProvider inputs: { functionInputs?: readonly ExecutionInput[]; executeInputs?: Record; + resources?: RunResources; }, context?: ResolutionContext, ): Promise { - // One lease across both halves: a run-scoped resource named by a function - // input *and* an execute input must be created once, not twice. + // One lease for the whole run: each named instance is created once, + // however many slots and arguments name it. const lease = this.resources.lease(); - const resolveResource = ( - uri: string, - binding?: Parameters[1], - ) => lease.acquire(uri, binding); // More inputs resolved later in the same lease — a check's — see the // same row and bindings the run's own did. const resolveMore = async (more: Record) => @@ -1258,7 +1256,7 @@ export class FilePromptProvider async ([name, input]) => [ name, - await resolveExecutionInput(input, resolveResource, context), + await resolveExecutionInput(input, lease, context), ] as const, ), ), @@ -1266,7 +1264,7 @@ export class FilePromptProvider try { const { functionParams, executeValues } = await resolveExecutionInputs( inputs, - resolveResource, + lease, context, ); return { @@ -1417,6 +1415,7 @@ export class FilePromptProvider ? [...inputs.functionInputs] : undefined, executeInputs: inputs?.executeInputs, + resources: inputs?.resources, parameterDefinitions: snapshot?.functionParameters, executeParameterDefinitions: snapshot?.executeParameters, version, diff --git a/src/prompt/playground/check.test.ts b/src/prompt/playground/check.test.ts index 3eb4da5..5ae2807 100644 --- a/src/prompt/playground/check.test.ts +++ b/src/prompt/playground/check.test.ts @@ -71,7 +71,8 @@ describe("lease.resolveDeclared", () => { }); const [{ check: c }] = await reg.checks(); const lease = reg.lease(); - const runValue = await lease.acquire(".evalution/playground/checks.ts#db"); + await lease.declare({ db: { uri: ".evalution/playground/checks.ts#db" } }); + const runValue = await lease.acquire("db"); const inputs = await lease.resolveDeclared(c.inputs, "c", async () => ({ title: "Set up CI", })); diff --git a/src/prompt/playground/resource-registry.test.ts b/src/prompt/playground/resource-registry.test.ts index 8bf7ffc..61af53e 100644 --- a/src/prompt/playground/resource-registry.test.ts +++ b/src/prompt/playground/resource-registry.test.ts @@ -5,12 +5,16 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { pathToFileURL } from "node:url"; -import { describe, expect, it, vi } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; import { LocalFileProvider } from "../../file-provider-local.ts"; import { MemoryFileProvider } from "../../file-provider-memory.ts"; import { resolveExecutionInputs } from "../execution-inputs.ts"; import { resource } from "./resource.ts"; -import { ResourceRegistry } from "./resource-registry.ts"; +import { + type ResourceBinding, + type ResourceLease, + ResourceRegistry, +} from "./resource-registry.ts"; const ROOT = "/proj"; const p = (...segments: string[]) => path.join(ROOT, ...segments); @@ -29,6 +33,29 @@ const HELPER = pathToFileURL( const importHelper = `import { resource } from ${JSON.stringify(HELPER)};`; +/** + * Declares `uri` (a resource, or one of its outputs) as an instance on + * `lease` and acquires it — the shape every run goes through. The instance + * is named after the resource's export unless `name` says otherwise. + */ +async function acquireUri( + lease: ResourceLease, + uri: string, + binding?: ResourceBinding, + name?: string, +): Promise { + const [file, fragment] = uri.split("#"); + const [exported, ...output] = fragment.split("."); + const instanceName = name ?? exported; + await lease.declare({ + [instanceName]: { uri: `${file}#${exported}`, ...(binding && { binding }) }, + }); + return lease.acquire( + instanceName, + output.length > 0 ? output.join(".") : undefined, + ); +} + function registry(files: Record) { const fileProvider = new MemoryFileProvider(files); return { @@ -140,7 +167,7 @@ describe("resource lifecycle", () => { // resolves them by identity — which is what makes the dependency typed at // the call site and immune to key collisions across playground files. const lease = reg.lease(); - expect(await lease.acquire("a/seed.playground.ts#seeded")).toBe( + expect(await acquireUri(lease, "a/seed.playground.ts#seeded")).toBe( "seeded-with-the-db", ); await lease.release(); @@ -179,7 +206,7 @@ describe("resource lifecycle", () => { const lease = reg.lease(); expect( - await lease.acquire("agents/odin.playground.ts#seededRootTask"), + await acquireUri(lease, "agents/odin.playground.ts#seededRootTask"), ).toBe("tsk_from_real-db"); await lease.release(); } finally { @@ -201,8 +228,8 @@ describe("resource lifecycle", () => { }); const lease = reg.lease(); - const first = await lease.acquire("x.playground.ts#thing"); - const second = await lease.acquire("x.playground.ts#thing"); + const first = await acquireUri(lease, "x.playground.ts#thing"); + const second = await acquireUri(lease, "x.playground.ts#thing"); // One value per run, however many slots reference it — which is why // resolution takes every input together. expect(second).toBe(first); @@ -224,11 +251,11 @@ describe("resource lifecycle", () => { }); const first = reg.lease(); - const a = await first.acquire("x.playground.ts#thing"); + const a = await acquireUri(first, "x.playground.ts#thing"); await first.release(); const second = reg.lease(); - const b = await second.acquire("x.playground.ts#thing"); + const b = await acquireUri(second, "x.playground.ts#thing"); await second.release(); // Releasing a lease must not touch a value that outlives the run. @@ -252,7 +279,7 @@ describe("resource lifecycle", () => { }); const lease = reg.lease(); - await expect(lease.acquire("cycle.playground.ts#A")).rejects.toThrow( + await expect(acquireUri(lease, "cycle.playground.ts#A")).rejects.toThrow( /cycle/i, ); }); @@ -270,7 +297,7 @@ describe("resource lifecycle", () => { // A value that outlives the run must not close over one that doesn't. const lease = reg.lease(); - await expect(lease.acquire("x.playground.ts#longLived")).rejects.toThrow( + await expect(acquireUri(lease, "x.playground.ts#longLived")).rejects.toThrow( /run-scoped/, ); }); @@ -284,9 +311,9 @@ describe("resource lifecycle", () => { }); const lease = reg.lease(); - await lease.acquire("x.playground.ts#seeded"); + await acquireUri(lease, "x.playground.ts#seeded"); expect(lease.receipts()).toEqual({ - "x.playground.ts#seeded": "tsk_abc123", + seeded: "tsk_abc123", }); await lease.release(); }); @@ -300,7 +327,7 @@ describe("static value resources", () => { }); const lease = reg.lease(); - expect(await lease.acquire("x.playground.ts#apiKey")).toBe("secret"); + expect(await acquireUri(lease, "x.playground.ts#apiKey")).toBe("secret"); await lease.release(); }); @@ -333,7 +360,7 @@ describe("static value resources", () => { }); const lease = reg.lease(); - expect(await lease.acquire("x.playground.ts#client")).toBe("client-secret"); + expect(await acquireUri(lease, "x.playground.ts#client")).toBe("client-secret"); await lease.release(); }); @@ -344,7 +371,7 @@ describe("static value resources", () => { }); const lease = reg.lease(); - await lease.acquire("x.playground.ts#apiKey"); + await acquireUri(lease, "x.playground.ts#apiKey"); // A literal has no lifecycle to tear down; releasing must not throw // trying to call a `dispose` it never had. await expect(lease.release()).resolves.toBeUndefined(); @@ -380,7 +407,7 @@ describe("value previews in the panel description", () => { expect(beforeRun.value).toBeUndefined(); const lease = reg.lease(); - await lease.acquire("x.playground.ts#config"); + await acquireUri(lease, "x.playground.ts#config"); await lease.release(); const [afterRun] = reg.describe(await reg.sources()); @@ -398,7 +425,7 @@ describe("value previews in the panel description", () => { }); const lease = reg.lease(); - await lease.acquire("x.playground.ts#seeded"); + await acquireUri(lease, "x.playground.ts#seeded"); await lease.release(); const [described] = reg.describe(await reg.sources()); @@ -448,39 +475,37 @@ describe("combined execute inputs (specs/combined-execute-inputs.md §C.2)", () }); const lease = reg.lease(); - const uri = "db.playground.ts#db"; - // The shape combined mode expands `db` into: one `resource` node per tool - // member, all naming the same uri — exactly what `ResourceRegistry.lease` - // is supposed to memoize by identity rather than by how many times it's - // referenced. - const resourceNode = { kind: "resource" as const, uri }; + // The shape combined mode expands `db` into: one `instance` node per tool + // member, all naming the same instance — created once, however many + // times it's referenced. + const instanceNode = { kind: "instance" as const, name: "db" }; const executeInputs = { toolsContext: { kind: "object" as const, properties: { list_tasks: { kind: "object" as const, - properties: { db: resourceNode }, + properties: { db: instanceNode }, }, create_task: { kind: "object" as const, - properties: { db: resourceNode }, + properties: { db: instanceNode }, }, update_task: { kind: "object" as const, - properties: { db: resourceNode }, + properties: { db: instanceNode }, }, post_message: { kind: "object" as const, - properties: { db: resourceNode }, + properties: { db: instanceNode }, }, }, }, }; const { executeValues } = await resolveExecutionInputs( - { executeInputs }, - resourceUri => lease.acquire(resourceUri), + { executeInputs, resources: { db: { uri: "db.playground.ts#db" } } }, + lease, ); const ctx = executeValues.toolsContext as Record; @@ -582,8 +607,8 @@ describe("resource outputs (specs/resource-hierarchy.md §A, §C)", () => { }); const lease = reg.lease(); - const id = await lease.acquire("tasks.playground.ts#taskA.id"); - const title = await lease.acquire("tasks.playground.ts#taskA.title"); + const id = await acquireUri(lease, "tasks.playground.ts#taskA.id"); + const title = await acquireUri(lease, "tasks.playground.ts#taskA.title"); expect(id).toBe("tsk_a"); expect(title).toBe("Fix it"); expect((globalThis as any).__creates).toBe(1); @@ -592,7 +617,7 @@ describe("resource outputs (specs/resource-hierarchy.md §A, §C)", () => { expect((globalThis as any).__disposes).toBe(1); }); - it("fails naming the resource and the key when create() doesn't return a declared value", async () => { + it("fails naming the instance and the key when create() doesn't return a declared value", async () => { const { registry: reg } = registry({ [p("tasks.playground.ts")]: `${importHelper} export const taskA = resource({ @@ -603,9 +628,9 @@ describe("resource outputs (specs/resource-hierarchy.md §A, §C)", () => { const lease = reg.lease(); await expect( - lease.acquire("tasks.playground.ts#taskA.title"), + acquireUri(lease, "tasks.playground.ts#taskA.title"), ).rejects.toThrow( - "Resource 'tasks.playground.ts#taskA': no output at 'title'", + "Resource instance 'taskA': no output 'title'", ); }); @@ -630,12 +655,9 @@ describe("resource outputs (specs/resource-hierarchy.md §A, §C)", () => { }); const lease = reg.lease(); - await lease.acquire("tasks.playground.ts#taskA.id"); - await lease.acquire("tasks.playground.ts#taskA.handle"); - expect(lease.receipts()).toEqual({ - "tasks.playground.ts#taskA.id": "taskA-receipt", - "tasks.playground.ts#taskA.handle": "taskA-receipt", - }); + await acquireUri(lease, "tasks.playground.ts#taskA.id"); + await acquireUri(lease, "tasks.playground.ts#taskA.handle"); + expect(lease.receipts()).toEqual({ taskA: "taskA-receipt" }); }); it("previews each declared value of a static `value` resource through describe", async () => { @@ -727,7 +749,7 @@ describe("resource groups (specs/resource-hierarchy.md §B, §D)", () => { }); const lease = reg.lease(); - expect(await lease.acquire("db.playground.ts#db")).toBe("db_1"); + expect(await acquireUri(lease, "db.playground.ts#db")).toBe("db_1"); await lease.release(); const sources = await reg.sources(); @@ -779,7 +801,7 @@ describe("duplicate module instances", () => { // The dependency object came from the plain instance while the registry // registered the freshly-evaluated one. Resolving by identity alone would // fail here with "not exported from any playground module". - expect(await lease.acquire("odin.playground.ts#seeded")).toBe( + expect(await acquireUri(lease, "odin.playground.ts#seeded")).toBe( // …and the value has to come from the re-evaluated module: the instance // the dependency points at is a key, never something to create from, or // an edit to the resource would never take effect. @@ -806,8 +828,8 @@ describe("duplicate module instances", () => { // Picked directly for one slot and pulled in as a dependency for another: // one `create()`, or the seeded row lands in a different database than the // one the tools query. - const direct: any = await lease.acquire("db.playground.ts#db"); - const viaNeeds = await lease.acquire("odin.playground.ts#seeded"); + const direct: any = await acquireUri(lease, "db.playground.ts#db"); + const viaNeeds = await acquireUri(lease, "odin.playground.ts#seeded"); expect(direct).toEqual({ tag: "fresh", created: 1 }); expect(viaNeeds).toBe("tsk_fresh_1"); await lease.release(); @@ -838,69 +860,6 @@ const schemaHelper = ` `; describe("resource arguments (specs/resource-arguments.md)", () => { - it("runs create() once for two acquisitions with equal arguments, and twice for different ones", async () => { - const { registry: reg } = registry({ - [p("x.playground.ts")]: `${importHelper} - ${schemaHelper} - globalThis.__creates = 0; - export const seeded = resource({ - inputs: { title: str() }, - create: ({ title }) => { - globalThis.__creates++; - return { value: title }; - }, - });`, - }); - - const lease = reg.lease(); - const bindingFor = (title: string) => ({ - key: JSON.stringify({ title }), - resolve: async () => ({ title }), - }); - - const a1 = await lease.acquire( - "x.playground.ts#seeded", - bindingFor("Todo app"), - ); - const a2 = await lease.acquire( - "x.playground.ts#seeded", - bindingFor("Todo app"), - ); - const b = await lease.acquire( - "x.playground.ts#seeded", - bindingFor("Other app"), - ); - - expect(a1).toBe("Todo app"); - expect(a2).toBe("Todo app"); - expect(b).toBe("Other app"); - expect((globalThis as any).__creates).toBe(2); - await lease.release(); - }); - - it("treats absent args and {} as the same key, and an unparameterized acquire as before this existed", async () => { - const { registry: reg } = registry({ - [p("x.playground.ts")]: `${importHelper} - globalThis.__creates = 0; - export const thing = resource({ - create: () => { - globalThis.__creates++; - return { value: globalThis.__creates }; - }, - });`, - }); - - const lease = reg.lease(); - const noBinding = await lease.acquire("x.playground.ts#thing"); - const emptyKey = await lease.acquire("x.playground.ts#thing", { - key: "", - resolve: async () => ({}), - }); - expect(noBinding).toBe(emptyKey); - expect((globalThis as any).__creates).toBe(1); - await lease.release(); - }); - it("passes resolved dependencies and validated arguments to create in one object", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: `${importHelper} @@ -914,15 +873,14 @@ describe("resource arguments (specs/resource-arguments.md)", () => { }); const lease = reg.lease(); - const result = await lease.acquire("x.playground.ts#seeded", { - key: JSON.stringify({ title: "hi" }), + const result = await acquireUri(lease, "x.playground.ts#seeded", { resolve: async () => ({ title: "hi" }), }); expect(result).toBe("the-db:hi"); await lease.release(); }); - it("fails naming the resource and the parameter when a value fails its schema, before create runs", async () => { + it("fails naming the instance and the parameter when a value fails its schema, before create runs", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: `${importHelper} ${schemaHelper} @@ -935,12 +893,11 @@ describe("resource arguments (specs/resource-arguments.md)", () => { const lease = reg.lease(); await expect( - lease.acquire("x.playground.ts#seeded", { - key: JSON.stringify({ title: 42 }), + acquireUri(lease, "x.playground.ts#seeded", { resolve: async () => ({ title: 42 }), }), ).rejects.toThrow( - "Resource 'x.playground.ts#seeded': invalid value for 'title'", + "Resource 'seeded (x.playground.ts#seeded)': invalid value for 'title'", ); expect((globalThis as any).__creates).toBe(0); }); @@ -956,15 +913,11 @@ describe("resource arguments (specs/resource-arguments.md)", () => { }); const lease = reg.lease(); - const value = await lease.acquire("x.playground.ts#seeded", { - key: "", - resolve: async () => ({}), - }); - expect(value).toBe("triaged"); + expect(await acquireUri(lease, "x.playground.ts#seeded")).toBe("triaged"); await lease.release(); }); - it("does not resolve arguments at all for a binding that hits the memo", async () => { + it("resolves an instance's arguments once, however often it's acquired", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: `${importHelper} ${schemaHelper} @@ -976,35 +929,12 @@ describe("resource arguments (specs/resource-arguments.md)", () => { const lease = reg.lease(); const resolve = vi.fn(async () => ({ title: "Todo app" })); - await lease.acquire("x.playground.ts#seeded", { key: "k1", resolve }); - await lease.acquire("x.playground.ts#seeded", { key: "k1", resolve }); - expect(resolve).toHaveBeenCalledTimes(1); - await lease.release(); - }); - - it("keys receipts by uri with no arguments and by uri@key with them", async () => { - const { registry: reg } = registry({ - [p("x.playground.ts")]: `${importHelper} - ${schemaHelper} - export const plain = resource({ - create: () => ({ value: "p", receipt: "p-receipt" }), - }); - export const seeded = resource({ - inputs: { title: str() }, - create: ({ title }) => ({ value: title, receipt: "tsk_" + title }), - });`, - }); - - const lease = reg.lease(); - await lease.acquire("x.playground.ts#plain"); - await lease.acquire("x.playground.ts#seeded", { - key: JSON.stringify({ title: "hi" }), - resolve: async () => ({ title: "hi" }), - }); - expect(lease.receipts()).toEqual({ - "x.playground.ts#plain": "p-receipt", - [`x.playground.ts#seeded@${JSON.stringify({ title: "hi" })}`]: "tsk_hi", + await lease.declare({ + seeded: { uri: "x.playground.ts#seeded", binding: { resolve } }, }); + await lease.acquire("seeded"); + await lease.acquire("seeded"); + expect(resolve).toHaveBeenCalledTimes(1); await lease.release(); }); @@ -1024,8 +954,7 @@ describe("resource arguments (specs/resource-arguments.md)", () => { const lease = reg.lease(); await expect( - lease.acquire("x.playground.ts#bad", { - key: JSON.stringify({ title: "x" }), + acquireUri(lease, "x.playground.ts#bad", { resolve: async () => ({ title: "x" }), }), ).rejects.toThrow(/server-scoped.*arguments/i); @@ -1047,7 +976,7 @@ describe("resource arguments (specs/resource-arguments.md)", () => { ); const lease = reg.lease(); - await expect(lease.acquire("x.playground.ts#bad")).rejects.toThrow( + await expect(acquireUri(lease, "x.playground.ts#bad")).rejects.toThrow( /'bad': input 'title' must be a resource or a Standard Schema.*not a string/i, ); }); @@ -1060,8 +989,7 @@ describe("resource arguments (specs/resource-arguments.md)", () => { const lease = reg.lease(); await expect( - lease.acquire("x.playground.ts#apiKey", { - key: JSON.stringify({ x: 1 }), + acquireUri(lease, "x.playground.ts#apiKey", { resolve: async () => ({ x: 1 }), }), ).rejects.toThrow( @@ -1069,7 +997,7 @@ describe("resource arguments (specs/resource-arguments.md)", () => { ); }); - it("fails with a cycle error naming both when an argument names the resource itself", async () => { + it("fails with a cycle error when an argument names the instance itself", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: `${importHelper} ${schemaHelper} @@ -1080,15 +1008,15 @@ describe("resource arguments (specs/resource-arguments.md)", () => { }); const lease = reg.lease(); - const selfBinding = { - key: "k", - resolve: async () => ({ - title: await lease.acquire("x.playground.ts#self", selfBinding), - }), - }; - await expect( - lease.acquire("x.playground.ts#self", selfBinding), - ).rejects.toThrow(/cycle/i); + await lease.declare({ + self: { + uri: "x.playground.ts#self", + binding: { + resolve: async () => ({ title: await lease.acquire("self") }), + }, + }, + }); + await expect(lease.acquire("self")).rejects.toThrow(/cycle/i); }); it("fails with the lifetime error when a server-scoped resource is given a run-scoped resource as an argument", async () => { @@ -1107,12 +1035,189 @@ describe("resource arguments (specs/resource-arguments.md)", () => { }); const lease = reg.lease(); + await expect(acquireUri(lease, "x.playground.ts#longLived")).rejects.toThrow( + /run-scoped/, + ); + }); +}); + +describe("named instances (specs/resource-instances.md)", () => { + const tasksModule = `${importHelper} + ${schemaHelper} + export const db = resource({ + inputs: { name: str({ default: "main" }) }, + create: ({ name }) => ({ value: { name } }), + }); + export const task = resource({ + inputs: { db, title: str(), parentId: str({ default: "" }) }, + create: ({ db, title, parentId }) => { + const id = "tsk_" + (globalThis.__created.length + 1); + globalThis.__created.push({ id, title, parentId, db: db.name }); + return { + value: { id, title }, + receipt: { id }, + dispose: () => { globalThis.__disposed.push(id); }, + }; + }, + });`; + + beforeEach(() => { + (globalThis as any).__created = []; + (globalThis as any).__disposed = []; + }); + + const titled = + (title: string, extra: () => Promise> = async () => ({})) => + ({ resolve: async () => ({ title, ...(await extra()) }) }); + + it("creates two instances of one resource even with identical arguments", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + a: { uri: "x.playground.ts#task", binding: titled("Same") }, + b: { uri: "x.playground.ts#task", binding: titled("Same") }, + }); + const a: any = await lease.acquire("a"); + const b: any = await lease.acquire("b"); + expect(a.id).not.toBe(b.id); + expect(await lease.acquire("a")).toBe(a); + expect((globalThis as any).__created).toHaveLength(2); + await lease.release(); + expect((globalThis as any).__disposed).toEqual( + expect.arrayContaining([a.id, b.id]), + ); + }); + + it("lets one instance take another instance of the same resource as an argument", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + root: { uri: "x.playground.ts#task", binding: titled("Root") }, + child: { + uri: "x.playground.ts#task", + binding: titled("Child", async () => ({ + parentId: await lease.acquire("root", "id"), + })), + }, + }); + const child: any = await lease.acquire("child"); + const root: any = await lease.acquire("root"); + expect((globalThis as any).__created).toEqual([ + { id: root.id, title: "Root", parentId: "", db: "main" }, + { id: child.id, title: "Child", parentId: root.id, db: "main" }, + ]); + await lease.release(); + }); + + it("fails with a cycle error naming both when two instances take each other's output", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + a: { + uri: "x.playground.ts#task", + binding: titled("A", async () => ({ + parentId: await lease.acquire("b", "id"), + })), + }, + b: { + uri: "x.playground.ts#task", + binding: titled("B", async () => ({ + parentId: await lease.acquire("a", "id"), + })), + }, + }); + await expect(lease.acquire("a")).rejects.toThrow( + /cycle: a \(x\.playground\.ts#task\) → b \(x\.playground\.ts#task\) → a/i, + ); + }); + + it("wires a code dependency to the run's one declared instance of it, arguments and all", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + staging: { + uri: "x.playground.ts#db", + binding: { resolve: async () => ({ name: "staging" }) }, + }, + t: { uri: "x.playground.ts#task", binding: titled("T") }, + }); + await lease.acquire("t"); + expect((globalThis as any).__created[0].db).toBe("staging"); + expect(await lease.acquire("staging")).toEqual({ name: "staging" }); + await lease.release(); + }); + + it("refuses a code dependency the run declares more than one instance of", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + db1: { uri: "x.playground.ts#db" }, + db2: { uri: "x.playground.ts#db" }, + t: { uri: "x.playground.ts#task", binding: titled("T") }, + }); + await expect(lease.acquire("t")).rejects.toThrow( + /more than one instance of it \(db1, db2\)/, + ); + }); + + it("refuses a server-scoped resource declared under two names", async () => { + const { registry: reg } = registry({ + [p("x.playground.ts")]: `${importHelper} + export const shared = resource({ scope: "server", create: () => ({ value: 1 }) });`, + }); await expect( - lease.acquire("x.playground.ts#longLived", { - key: "", - resolve: async () => ({}), + reg.lease().declare({ + one: { uri: "x.playground.ts#shared" }, + two: { uri: "x.playground.ts#shared" }, }), - ).rejects.toThrow(/run-scoped/); + ).rejects.toThrow(/server-scoped.*'one' and 'two'/); + }); + + it("refuses an invalid name, an output uri, and a resource that doesn't exist", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await expect( + lease.declare({ "1st": { uri: "x.playground.ts#db" } }), + ).rejects.toThrow(/Invalid resource instance name '1st'/); + await expect( + lease.declare({ t: { uri: "x.playground.ts#task.id" } }), + ).rejects.toThrow(/names an output, not a resource/); + await expect( + lease.declare({ t: { uri: "x.playground.ts#nope" } }), + ).rejects.toThrow(/'x\.playground\.ts#nope' not found/); + await expect(lease.acquire("never")).rejects.toThrow( + "No resource instance named 'never'", + ); + }); + + it("fails naming the instance when an output it doesn't have is read", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + t: { uri: "x.playground.ts#task", binding: titled("T") }, + }); + await expect(lease.acquire("t", "nope")).rejects.toThrow( + "Resource instance 't': no output 'nope'", + ); + await lease.release(); + }); + + it("keys receipts by instance name", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + first: { uri: "x.playground.ts#task", binding: titled("1") }, + second: { uri: "x.playground.ts#task", binding: titled("2") }, + main: { uri: "x.playground.ts#db" }, + }); + await lease.acquire("first"); + await lease.acquire("second"); + await lease.acquire("main"); + expect(lease.receipts()).toEqual({ + first: { id: "tsk_1" }, + second: { id: "tsk_2" }, + }); + await lease.release(); }); }); @@ -1130,14 +1235,12 @@ describe("receipt round trip (specs/resource-arguments.md §E)", () => { }); const fresh = reg.lease(); - const freshValue = await fresh.acquire("x.playground.ts#seeded"); + const freshValue = await acquireUri(fresh, "x.playground.ts#seeded"); await fresh.release(); expect(freshValue).toBe("tsk_new"); const replay = reg.lease(); - const replayValue = await replay.acquire("x.playground.ts#seeded", { - key: "", - resolve: async () => ({}), + const replayValue = await acquireUri(replay, "x.playground.ts#seeded", { receipt: { taskId: "tsk_abc123" }, }); await replay.release(); @@ -1148,26 +1251,6 @@ describe("receipt round trip (specs/resource-arguments.md §E)", () => { ]); }); - it("does not widen the memo key — two references with equal receipts still resolve to one instance", async () => { - const { registry: reg } = registry({ - [p("x.playground.ts")]: `${importHelper} - globalThis.__creates = 0; - export const seeded = resource({ - create: (_inputs, receipt) => { - globalThis.__creates++; - return { value: "v", receipt: receipt ?? "r" }; - }, - });`, - }); - - const lease = reg.lease(); - const binding = { key: "", resolve: async () => ({}), receipt: "r" }; - await lease.acquire("x.playground.ts#seeded", binding); - await lease.acquire("x.playground.ts#seeded", binding); - expect((globalThis as any).__creates).toBe(1); - await lease.release(); - }); - it("still runs, mints, and records a new receipt when create ignores the one it was handed", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: `${importHelper} @@ -1178,13 +1261,11 @@ describe("receipt round trip (specs/resource-arguments.md §E)", () => { }); const lease = reg.lease(); - const value = await lease.acquire("x.playground.ts#seeded", { - key: "", - resolve: async () => ({}), + const value = await acquireUri(lease, "x.playground.ts#seeded", { receipt: "ignored", }); expect(value).toBe(1); - expect(lease.receipts()).toEqual({ "x.playground.ts#seeded": "receipt-1" }); + expect(lease.receipts()).toEqual({ seeded: "receipt-1" }); await lease.release(); }); }); @@ -1198,12 +1279,10 @@ describe("failures in author-written resource code", () => { });`, }); - const error = (await reg - .lease() - .acquire("x.playground.ts#broken") + const error = (await acquireUri(reg.lease(), "x.playground.ts#broken") .catch(e => e)) as Error; expect(error.message).toBe( - "Resource 'x.playground.ts#broken': create() failed — boom", + "Resource 'broken (x.playground.ts#broken)': create() failed — boom", ); expect(error.cause).toBeInstanceOf(TypeError); expect((error.cause as Error).message).toBe("boom"); @@ -1221,12 +1300,10 @@ describe("failures in author-written resource code", () => { });`, }); - const error = (await reg - .lease() - .acquire("x.playground.ts#db") + const error = (await acquireUri(reg.lease(), "x.playground.ts#db") .catch(e => e)) as Error; expect(error.message).toMatch( - /^Resource 'x\.playground\.ts#db': reset\(\) failed — /, + /^Resource 'db \(x\.playground\.ts#db\)': reset\(\) failed — /, ); expect(error.cause).toBeInstanceOf(TypeError); }); @@ -1238,7 +1315,7 @@ describe("failures in author-written resource code", () => { export const top = resource({ inputs: { dep }, create: () => ({ value: 1 }) });`, }); - await expect(reg.lease().acquire("x.playground.ts#top")).rejects.toThrow( + await expect(acquireUri(reg.lease(), "x.playground.ts#top")).rejects.toThrow( "Resource 'x.playground.ts#dep': create() failed — nope", ); }); @@ -1260,14 +1337,14 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { }); const first = reg.lease(); - await first.acquire("x.playground.ts#db"); - await first.acquire("x.playground.ts#db"); // same lease, same instance: no extra reset + await acquireUri(first, "x.playground.ts#db"); + await acquireUri(first, "x.playground.ts#db"); // same lease, same instance: no extra reset await first.release(); expect((globalThis as any).__creates).toBe(1); expect((globalThis as any).__resets).toBe(1); const second = reg.lease(); - await second.acquire("x.playground.ts#db"); + await acquireUri(second, "x.playground.ts#db"); await second.release(); expect((globalThis as any).__creates).toBe(1); // memoized: still one create() expect((globalThis as any).__resets).toBe(2); // reset again for the new lease @@ -1283,7 +1360,7 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { }); const lease = reg.lease(); - await lease.acquire("x.playground.ts#thing"); + await acquireUri(lease, "x.playground.ts#thing"); await lease.release(); expect((globalThis as any).__resets).toBe(0); }); @@ -1302,12 +1379,12 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { }); const first = reg.lease(); - await first.acquire("x.playground.ts#db"); + await acquireUri(first, "x.playground.ts#db"); (globalThis as any).__order.push("first-acquired"); let secondAcquired = false; const second = reg.lease(); - const secondAcquire = second.acquire("x.playground.ts#db").then(() => { + const secondAcquire = acquireUri(second, "x.playground.ts#db").then(() => { secondAcquired = true; (globalThis as any).__order.push("second-acquired"); }); @@ -1344,13 +1421,13 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { const leaseTwo = reg.lease(); const oneDone = (async () => { - await leaseOne.acquire("a.playground.ts#res"); - await leaseOne.acquire("b.playground.ts#res"); + await acquireUri(leaseOne, "a.playground.ts#res"); + await acquireUri(leaseOne, "b.playground.ts#res"); await leaseOne.release(); })(); const twoDone = (async () => { - await leaseTwo.acquire("b.playground.ts#res"); - await leaseTwo.acquire("a.playground.ts#res"); + await acquireUri(leaseTwo, "b.playground.ts#res"); + await acquireUri(leaseTwo, "a.playground.ts#res"); await leaseTwo.release(); })(); @@ -1393,7 +1470,7 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { "x.playground.ts#task", "x.playground.ts#db", ].map(uri => - lease.acquire(uri).then(value => { + acquireUri(lease, uri).then(value => { (globalThis as any).__order.push(uri); return value; }), @@ -1426,16 +1503,16 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { const leaseTwo = reg.lease(); const oneDone = (async () => { await Promise.all([ - leaseOne.acquire("b.playground.ts#res"), - leaseOne.acquire("a.playground.ts#res"), - leaseOne.acquire("b.playground.ts#res"), + acquireUri(leaseOne, "b.playground.ts#res"), + acquireUri(leaseOne, "a.playground.ts#res"), + acquireUri(leaseOne, "b.playground.ts#res"), ]); await leaseOne.release(); })(); const twoDone = (async () => { await Promise.all([ - leaseTwo.acquire("a.playground.ts#res"), - leaseTwo.acquire("b.playground.ts#res"), + acquireUri(leaseTwo, "a.playground.ts#res"), + acquireUri(leaseTwo, "b.playground.ts#res"), ]); await leaseTwo.release(); })(); @@ -1456,7 +1533,7 @@ describe("resource reset (specs/resource-arguments.md §F)", () => { const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); try { const lease = reg.lease(); - await lease.acquire("x.playground.ts#thing"); + await acquireUri(lease, "x.playground.ts#thing"); await lease.release(); expect(warn).toHaveBeenCalledWith(expect.stringMatching(/run-scoped/i)); } finally { diff --git a/src/prompt/playground/resource-registry.ts b/src/prompt/playground/resource-registry.ts index b25ebb0..2b38c04 100644 --- a/src/prompt/playground/resource-registry.ts +++ b/src/prompt/playground/resource-registry.ts @@ -296,42 +296,38 @@ function isPlainSerializable( } /** - * A resource reference's arguments, as the lease sees them. + * A declared instance's arguments and replay receipt, as the lease sees them. * - * Built by `resolveExecutionInput` from an `ExecutionInput`'s `args` — the - * layer that knows how to turn one into a value — so the registry never has - * to know `ExecutionInput` exists. See `specs/resource-arguments.md` §D. + * Built by `resolveExecutionInputs` from a `ResourceInstanceInput` — the + * layer that knows how to turn its `args` into values — so the registry never + * has to know `ExecutionInput` exists. See `specs/resource-instances.md` §B. */ export interface ResourceBinding { - /** - * Canonical key for these arguments — identity for memoization. The stable - * JSON encoding of the *unresolved* arguments (object keys sorted, absent - * and `{}` both encoding to `""`), so a resolved argument that happens to - * be a live handle never has to be compared or hashed. - */ - key: string; /** * Resolves the arguments, keyed by parameter name. Called at most once per - * (resource, key), lazily — a binding that turns out to hit the memo must - * not evaluate its arguments at all, and a failed `create` must not have - * side effects from arguments no one asked for. + * instance, lazily — only when the instance is actually created, so a + * failed `create` has no side effects from arguments no one asked for. + * Absent when the instance was declared with no arguments. */ - resolve(): Promise>; - /** A past run's receipt, on a replay. Passed to `create`; never part of {@link key}. */ + resolve?(): Promise>; + /** A past run's receipt, on a replay. Passed to `create`. */ receipt?: unknown; } /** - * The `ResourceLease.receipts()` key for a source acquired with argument key - * `key`: `@` when `key` is non-empty (the resource took - * arguments), or bare `` when it's `""` (it didn't) — so an existing - * unparameterized receipt's key is byte-identical to before arguments - * existed. Exported so `stampReceipts` (`execution-inputs.ts`) can compute - * the same key from the unresolved recipe when recording a fresh run's - * receipts onto its inputs. See `specs/resource-arguments.md` §D, §K. + * One named instance a run declares — what {@link ResourceLease.declare} + * takes. See `specs/resource-instances.md` §A. */ -export function receiptKeyOf(uri: string, key: string): string { - return key ? `${uri}@${key}` : uri; +export interface DeclaredInstance { + /** The resource's root `uri` (`tasks.ts#seededTask`), never an output's. */ + uri: string; + /** Its arguments and replay receipt, if it has either. */ + binding?: ResourceBinding; +} + +/** Whether `name` is a valid instance name: `[A-Za-z_][A-Za-z0-9_-]*`. */ +export function isValidInstanceName(name: string): boolean { + return /^[A-Za-z_][A-Za-z0-9_-]*$/.test(name); } /** @@ -520,9 +516,16 @@ export async function validateArguments( /** Resources already warned about a `reset` that will never run, so the message appears once each. */ const warnedRunScopedReset = new WeakSet>(); -/** One link in a resource's dependency chain, for cycle detection and error messages. */ +/** + * One link in a resource's dependency chain, for cycle detection and error + * messages. `key` is the instance's memo key — its name for a declared + * run-scoped instance, `""` for a server-scoped or undeclared one — so two + * instances of one resource are two links, and a child task seeded from its + * parent's id is not mistaken for a cycle (`specs/resource-instances.md` §B). + */ interface ChainLink { resource: Resource; + key: string; label: string; } @@ -533,29 +536,35 @@ interface ChainLink { * otherwise has no way to know it's being called from inside another * resource's own creation. * - * This is what lets `A`'s argument resolving back to `A` itself (or to `B`, - * which takes `A` as one of *its* arguments) be caught as the same kind of - * cycle a static `inputs` cycle is, rather than deadlocking on a promise - * awaiting its own settlement. See `specs/resource-arguments.md` §D. + * This is what lets an instance's argument resolving back to the instance + * itself (or to one that takes it as one of *its* arguments) be caught as the + * same kind of cycle a static `inputs` cycle is, rather than deadlocking on a + * promise awaiting its own settlement. */ const chainContext = new AsyncLocalStorage(); /** A handle on the resources created for one execution. */ export interface ResourceLease { /** - * Creates (or reuses) the value for `uri`, resolving its `inputs` first. - * Calling twice for the same resource (with the same `binding`'s `key`, if - * any) within one lease returns the same value — which is why resolution - * takes every input together. - * - * @param binding - The resource's arguments and replay receipt, if any. See - * {@link ResourceBinding} and `specs/resource-arguments.md` §D. + * Declares the run's named instances. Called once, before anything is + * acquired. Rejects a name that is invalid, a `uri` that names no resource + * or names an output rather than a resource, and a `server`-scoped or + * static resource declared under more than one name — it has one instance + * per process, so two names would alias it. See + * `specs/resource-instances.md` §B. + */ + declare(instances: Record): Promise; + /** + * Creates (or reuses) the declared instance `name`, resolving its `inputs` + * first, and returns its value — or, given `output`, that top-level + * property of it. Every call for the same name within one lease returns the + * same instance. */ - acquire(uri: string, binding?: ResourceBinding): Promise; + acquire(name: string, output?: string): Promise; /** - * A serializable summary of each resource acquired, keyed by `uri` — or by - * `` `${uri}@${key}` `` when it was acquired with arguments (`key` being - * the {@link ResourceBinding.key} that produced the instance). + * A serializable summary of what each declared instance produced, by + * instance name — its `create()`'s `receipt`. Instances that produced none + * are absent. */ receipts(): Record; /** @@ -577,9 +586,22 @@ export interface ResourceLease { release(): Promise; } -/** A resource's memoized instances, one per argument key (`""` for none). See `specs/resource-arguments.md` §D. */ +/** + * A resource's memoized instances, by memo key: the instance name for a + * declared run-scoped instance, `""` for a server-scoped or undeclared one. + */ type InstancesByKey = Map | Instance>; +/** One lease's own state, threaded through instantiation. */ +interface LeaseState { + runInstances: Map, InstancesByKey>; + resetLocks: LeaseResetLocks; + /** The run's declared instances, by name, with their registrations. */ + declared: Map; + /** Receipts recorded so far, by instance name. */ + receipts: Map; +} + /** * Discovers **playground modules** and resolves the resources they export. * @@ -709,7 +731,7 @@ export class ResourceRegistry { */ describe(sources: readonly RegisteredSource[]): ResourceInfo[] { return sources.map(s => { - const { params } = partitionInputs(s.resource.resource); + const { deps, params } = partitionInputs(s.resource.resource); const scope = s.resource.resource.scope ?? defaultScope(s.resource.resource); const invalidArgs = scope === "server" && params.length > 0; @@ -734,6 +756,17 @@ export class ResourceRegistry { value, }; if (invalidArgs) info.error = serverScopedArgumentsError(s.resource.uri); + if (s.outputPath.length === 0) { + const dependencies = Object.fromEntries( + deps.flatMap(([name, dep]) => { + const uri = this.identities.get(dep)?.uri; + return uri ? [[name, uri] as const] : []; + }), + ); + if (Object.keys(dependencies).length > 0) { + info.dependencies = dependencies; + } + } if (s.group.length > 0) info.group = [...s.group]; if (s.outputPath.length > 0) { info.parent = s.resource.uri; @@ -780,51 +813,63 @@ export class ResourceRegistry { * together by {@link ResourceLease.release}. */ lease(): ResourceLease { - const runInstances = new Map, InstancesByKey>(); - const acquired = new Map(); - const resetLocks = this.resetLocks.forLease(); + const state: LeaseState = { + runInstances: new Map(), + resetLocks: this.resetLocks.forLease(), + declared: new Map(), + receipts: new Map(), + }; let released = false; - const acquire = async ( - uri: string, - binding?: ResourceBinding, - ): Promise => { - const { rootUri, outputPath } = parseSourceUri(uri); - const registered = (await this.byUri()).get(rootUri); - if (!registered) throw new Error(`Resource '${uri}' not found`); - const key = binding?.key ?? ""; - // A fresh top-level call has no ambient chain — `[]`, same as before - // arguments existed. Reached instead from inside another resource's own - // `create()` (its argument resolved back through here), the chain that - // creation is running under is picked up via {@link chainContext} rather - // than resetting to empty, which is what lets an argument cycle be - // caught the same way a static `inputs` cycle is. - const chain = chainContext.getStore() ?? []; - const instance = await this.instantiate( - registered.resource, - rootUri, - runInstances, - chain, - key, - binding, - resetLocks, + const declare = async ( + instances: Record, + ): Promise => { + const byUri = await this.byUri(); + const sharedBy = new Map, string>(); + for (const [name, decl] of Object.entries(instances)) { + if (!isValidInstanceName(name)) { + throw new Error( + `Invalid resource instance name '${name}': use letters, digits, '_' and '-', not starting with a digit or '-'.`, + ); + } + const { rootUri, outputPath } = parseSourceUri(decl.uri); + if (outputPath.length > 0) { + throw new Error( + `Resource instance '${name}': '${decl.uri}' names an output, not a resource — declare '${rootUri}' and read the output where it's used.`, + ); + } + const registered = byUri.get(rootUri); + if (!registered) { + throw new Error( + `Resource instance '${name}': resource '${decl.uri}' not found`, + ); + } + const target = registered.resource; + if ((target.scope ?? defaultScope(target)) === "server") { + const other = sharedBy.get(target); + if (other) { + throw new Error( + `Resource '${rootUri}' is shared by every run (server-scoped), so it can only be declared once — '${other}' and '${name}' would be the same instance.`, + ); + } + sharedBy.set(target, name); + } + state.declared.set(name, { ...decl, registered }); + } + }; + + const acquire = async (name: string, output?: string): Promise => { + await this.byUri(); + const instance = await this.instantiateDeclared( + name, + state, + chainContext.getStore() ?? [], ); - const read = readOutputPath(instance.value, outputPath); + if (output === undefined) return instance.value; + const read = readOutputPath(instance.value, [output]); if (!read.found) { - throw new Error( - `Resource '${rootUri}': no output at '${outputPath.join(".")}'`, - ); + throw new Error(`Resource instance '${name}': no output '${output}'`); } - // A root reference and an output reference name the *same instance*, so - // both record that instance's receipt, full stop — the output's own - // value (even when it's plain JSON) is no longer substituted here. - // That was right when a receipt was only ever displayed, but the - // moment a receipt is fed back to `create` (§E), replaying an output - // reference must not hand it the output's bare value where `create` - // expects its own receipt shape. An author who wants the id shown in a - // trace puts it in the receipt, which is what makes this no loss. - const receipt = instance.receipt; - acquired.set(receiptKeyOf(uri, key), { receipt }); return read.value; }; @@ -838,9 +883,8 @@ export class ResourceRegistry { deps, label, "run", - runInstances, + state, chainContext.getStore() ?? [], - resetLocks, ); if (params.length > 0) { Object.assign( @@ -852,22 +896,18 @@ export class ResourceRegistry { }; return { + declare, acquire, resolveDeclared, - receipts: () => - Object.fromEntries( - [...acquired].flatMap(([uri, e]) => - e.receipt === undefined ? [] : [[uri, e.receipt] as const], - ), - ), + receipts: () => Object.fromEntries(state.receipts), release: async () => { if (released) return; released = true; - const pending = [...runInstances.values()].flatMap(byKey => [ + const pending = [...state.runInstances.values()].flatMap(byKey => [ ...byKey.values(), ]); - runInstances.clear(); - resetLocks.releaseAll(); + state.runInstances.clear(); + state.resetLocks.releaseAll(); await Promise.all( pending.map(async p => { try { @@ -884,33 +924,65 @@ export class ResourceRegistry { // #region Instantiation /** - * Creates (or reuses) the instance for `target`, resolving its `inputs` - * first — its dependencies, and, if `binding` is given, its arguments. + * Creates (or reuses) the run's declared instance `name`, recording its + * receipt. Keyed by name for a run-scoped resource, so two instances of one + * resource are two `create()`s even with identical arguments; a + * server-scoped one has a single instance per process however it's named. + */ + private async instantiateDeclared( + name: string, + state: LeaseState, + chain: readonly ChainLink[], + ): Promise { + const decl = state.declared.get(name); + if (!decl) throw new Error(`No resource instance named '${name}'`); + const target = decl.registered.resource; + const label = `${name} (${decl.registered.uri})`; + // Checked here rather than in `create`: a static resource's one instance + // exists from discovery on, so `create` never runs for it. + if ("value" in target && decl.binding?.resolve) { + throw new Error(`Resource '${label}': static resources take no arguments`); + } + const scope = target.scope ?? defaultScope(target); + const instance = await this.instantiate( + target, + label, + state, + chain, + scope === "server" ? "" : name, + decl.binding, + ); + if (instance.receipt !== undefined) { + state.receipts.set(name, instance.receipt); + } + return instance; + } + + /** + * Creates (or reuses) the instance for `target` under memo key `key`, + * resolving its `inputs` first — its dependencies, and, if `binding` + * resolves any, its arguments. * * Keyed on the resource **object**, not on its `uri`: a dependency is named * by reference, and a resource reached only that way (from a module the * discovery patterns don't cover) still has to resolve. `label` is what the - * resource is called in errors — its `uri` where it has one. Further keyed - * on `key` — {@link ResourceBinding.key}, or `""` for a resource with no - * arguments — so two references to one resource with different arguments - * resolve to two instances (`specs/resource-arguments.md` §D). + * resource is called in errors. * * `chain` is the dependency path taken to get here; it turns a cycle into a - * legible error instead of a stack overflow. Cycles are detected on the - * resource object alone, ignoring `key`: a resource that recursively seeds - * itself with different arguments is not a case that exists, and erring - * toward rejecting it is the safe side. + * legible error instead of a stack overflow. A cycle is the same + * (resource, key) reached twice — so `child1.parentId ← root.taskId` is + * two instances of one resource and fine, while two instances each taking + * the other's id is not. */ private async instantiate( target: Resource, label: string, - runInstances: Map, InstancesByKey>, - chain: readonly { resource: Resource; label: string }[], + state: LeaseState, + chain: readonly ChainLink[], key: string, binding: ResourceBinding | undefined, - resetLocks: LeaseResetLocks, ): Promise { - if (chain.some(c => c.resource === target)) { + if (chain.some(c => c.resource === target && c.key === key)) { throw new Error( `Resource dependency cycle: ${[...chain, { label }] .map(c => c.label) @@ -919,7 +991,8 @@ export class ResourceRegistry { } const scope = target.scope ?? defaultScope(target); - const cache = scope === "server" ? this.serverInstances : runInstances; + const cache = + scope === "server" ? this.serverInstances : state.runInstances; let byKey = cache.get(target); if (!byKey) { @@ -929,21 +1002,13 @@ export class ResourceRegistry { let pending = byKey.get(key); if (!pending) { - const nextChain = [...chain, { resource: target, label }]; + const nextChain = [...chain, { resource: target, key, label }]; // Ambient for the duration of `create()` — including its `await // binding.resolve()`, which is how an argument that resolves back // through `ResourceLease.acquire` (a public entry point with no chain // of its own) still sees this chain. See {@link chainContext}. pending = chainContext.run(nextChain, () => - this.create( - target, - label, - scope, - runInstances, - nextChain, - binding, - resetLocks, - ), + this.create(target, label, scope, state, nextChain, binding), ); byKey.set(key, pending); const settledByKey = byKey; @@ -973,8 +1038,10 @@ export class ResourceRegistry { const instance = await pending; if (scope === "server" && instance.reset) { - const sortKey = key ? `${label}@${key}` : label; - await resetLocks.enter(target, key, sortKey, async () => { + // Sorted by the resource's own uri, which every lease agrees on — not + // `label`, which carries whatever instance name this run gave it. + const sortKey = (await this.byIdentity()).get(target)?.uri ?? label; + await state.resetLocks.enter(target, key, sortKey, async () => { try { await instance.reset!(); } catch (err) { @@ -1000,17 +1067,11 @@ export class ResourceRegistry { target: Resource, label: string, scope: ResourceScope, - runInstances: Map, InstancesByKey>, - chain: readonly { resource: Resource; label: string }[], + state: LeaseState, + chain: readonly ChainLink[], binding: ResourceBinding | undefined, - resetLocks: LeaseResetLocks, ): Promise { if ("value" in target) { - if (binding) { - throw new Error( - `Resource '${label}': static resources take no arguments`, - ); - } return target; } @@ -1020,21 +1081,13 @@ export class ResourceRegistry { throw new Error(serverScopedArgumentsError(label)); } - const resolved = await this.resolveDeps( - deps, - label, - scope, - runInstances, - chain, - resetLocks, - ); + const resolved = await this.resolveDeps(deps, label, scope, state, chain); if (params.length > 0) { - // Lazy, and only evaluated here — never for a binding that hits the - // memo — because arguments must not be resolved (and a resource used - // as one of them must not be created) for a slot no one is actually - // filling from this call. - const argValues = (await binding?.resolve()) ?? {}; + // Lazy, and only evaluated here — never for an instance that's already + // memoized — because arguments must not be resolved (and an instance + // used as one of them must not be created) for nothing. + const argValues = (await binding?.resolve?.()) ?? {}; Object.assign( resolved, await validateArguments(params, argValues, `Resource '${label}'`), @@ -1058,14 +1111,19 @@ export class ResourceRegistry { /** * Creates (or reuses) each of `deps` — code-wired dependencies, named by * resource object — for something `label` names, which lives for `scope`. + * + * A dependency is the run's declared instance of that resource when it + * declares exactly one, so arguments configured in the Resources section + * reach the instance that depends on it. With none declared it is the + * undeclared instance (memo key `""`), created with no arguments; with + * several it's ambiguous and fails. See `specs/resource-instances.md` §D. */ private async resolveDeps( deps: readonly [string, Resource][], label: string, scope: ResourceScope, - runInstances: Map, InstancesByKey>, + state: LeaseState, chain: readonly ChainLink[], - resetLocks: LeaseResetLocks, ): Promise> { const resolved: Record = {}; for (const [name, dep] of deps) { @@ -1084,16 +1142,27 @@ export class ResourceRegistry { `run-scoped. A value that outlives the run must not close over one that doesn't.`, ); } + + const declaredAs = [...state.declared] + .filter(([, d]) => d.registered.resource === depTarget) + .map(([instanceName]) => instanceName); + if (declaredAs.length > 1) { + throw new Error( + `Resource '${label}' needs '${depLabel}' as '${name}', but this run declares ` + + `more than one instance of it (${declaredAs.join(", ")}) — it can't tell which.`, + ); + } resolved[name] = ( - await this.instantiate( - depTarget, - depLabel, - runInstances, - chain, - "", - undefined, - resetLocks, - ) + declaredAs.length === 1 + ? await this.instantiateDeclared(declaredAs[0], state, chain) + : await this.instantiate( + depTarget, + depLabel, + state, + chain, + "", + undefined, + ) ).value; } return resolved; diff --git a/src/prompt/prompt-provider.ts b/src/prompt/prompt-provider.ts index 4a25772..93a41b1 100644 --- a/src/prompt/prompt-provider.ts +++ b/src/prompt/prompt-provider.ts @@ -13,6 +13,7 @@ import type { PromptStyle, PropDefinition, RebaseResult, + RunResources, VariationConflict, VariationId, VariationInfo, @@ -50,6 +51,8 @@ export interface ExecuteOptions { inputs?: { functionInputs?: readonly ExecutionInput[]; executeInputs?: Record; + /** The run's named resource instances, receipts stamped. */ + resources?: RunResources; }; /** * Called once the run is over, successfully or not. @@ -223,9 +226,9 @@ export interface ResolvedPromptInputs { /** Named values for {@link ExecuteOptions.executeValues}. */ executeValues: Record; /** - * Serializable summaries of what any resources produced, keyed by `uri`, so - * a trace can show the value a run used even though replaying it would mint - * a new one. + * Serializable summaries of what the run's resource instances produced, + * keyed by instance name, so a trace can show the value a run used and a + * replay can hand it back to `create`. */ receipts?: Record; /** @@ -374,6 +377,7 @@ export interface PromptProvider< inputs: { functionInputs?: readonly ExecutionInput[]; executeInputs?: Record; + resources?: RunResources; }, context?: ResolutionContext, ): Promise; diff --git a/src/sdk/typesafe-sdk/telemetry.ts b/src/sdk/typesafe-sdk/telemetry.ts index f79a789..ae1565b 100644 --- a/src/sdk/typesafe-sdk/telemetry.ts +++ b/src/sdk/typesafe-sdk/telemetry.ts @@ -156,6 +156,7 @@ export class TypeSafeTelemetry extends BaseTraceIngestor { functionParameters: identity.functionParameters, functionInputs: identity.functionInputs, executeInputs: identity.executeInputs, + resources: identity.resources, parameterDefinitions: identity.parameterDefinitions, executeParameterDefinitions: identity.executeParameterDefinitions, version: identity.version, diff --git a/src/sdk/vercel-ai-sdk/telemetry.ts b/src/sdk/vercel-ai-sdk/telemetry.ts index 878083e..1f772a3 100644 --- a/src/sdk/vercel-ai-sdk/telemetry.ts +++ b/src/sdk/vercel-ai-sdk/telemetry.ts @@ -336,6 +336,7 @@ export class VercelAISDKTelemetry functionParameters: identity.functionParameters, functionInputs: identity.functionInputs, executeInputs: identity.executeInputs, + resources: identity.resources, parameterDefinitions: identity.parameterDefinitions, executeParameterDefinitions: identity.executeParameterDefinitions, version: identity.version, @@ -387,6 +388,7 @@ export class VercelAISDKTelemetry functionParameters: identity.functionParameters, functionInputs: identity.functionInputs, executeInputs: identity.executeInputs, + resources: identity.resources, parameterDefinitions: identity.parameterDefinitions, version: identity.version, variation: identity.variation, diff --git a/src/server/api-routes.test.ts b/src/server/api-routes.test.ts index df4c7bd..75e17cc 100644 --- a/src/server/api-routes.test.ts +++ b/src/server/api-routes.test.ts @@ -883,9 +883,9 @@ describe("dataset routes", () => { json("POST", { rows: [ { - cells: { - "0": { - kind: "resource", + cells: { "0": { kind: "instance", name: "task" } }, + resources: { + task: { uri: "pg.ts#seededTask", args: { title: text("Milk") }, // Stripped by the server: a row is a recipe, not a replay. @@ -912,7 +912,8 @@ describe("dataset routes", () => { expect(got.dataset.name).toBe("Tickets"); // An overview, not the rows: they're paged in separately. expect(got.rowCount).toBe(1); - expect(got.fields).toEqual({ "0": { keys: ["title"], resource: true } }); + // A resource instance is named, not spelled out, so it has no keys. + expect(got.fields).toEqual({}); expect(got.rows).toBeUndefined(); const rows = (await ( @@ -920,10 +921,9 @@ describe("dataset routes", () => { `/api/datasets/${DATASET_PROVIDER_ID}/${dataset.id}/rows?offset=0&limit=10`, ) ).json()) as any[]; - expect(rows[0].cells["0"]).toEqual({ - kind: "resource", - uri: "pg.ts#seededTask", - args: { title: text("Milk") }, + expect(rows[0].cells["0"]).toEqual({ kind: "instance", name: "task" }); + expect(rows[0].resources).toEqual({ + task: { uri: "pg.ts#seededTask", args: { title: text("Milk") } }, }); expect(rows[0].source).toEqual({ kind: "trace", diff --git a/src/server/handlers/datasets.test.ts b/src/server/handlers/datasets.test.ts index b213015..c736ce2 100644 --- a/src/server/handlers/datasets.test.ts +++ b/src/server/handlers/datasets.test.ts @@ -395,11 +395,35 @@ describe("handleUpdateRows", () => { it("sets a resource in any field, as adding a row does", async () => { const { update, cells } = await seeded(); - const resource: ExecutionInput = { kind: "resource", uri: "pg.ts#task" }; + const resource: ExecutionInput = { kind: "instance", name: "task" }; expect((await update({ "0": resource, "3": resource })).status).toBe(204); expect(await cells()).toMatchObject({ "0": resource, "3": resource }); }); + it("sets and removes a row's resource instances by name, stripping receipts", async () => { + const { provider, dataset, row } = await seeded(); + const updateResources = (resources: Record) => + handleUpdateRows(provider, dataset.id, { + updates: [{ rowId: row.id, cells: {}, resources }], + }); + expect( + ( + await updateResources({ + db: { uri: "pg.ts#db" }, + task: { uri: "pg.ts#task", receipt: "tsk_1" }, + }) + ).status, + ).toBe(204); + expect((await updateResources({ db: null })).status).toBe(204); + const [stored] = await provider.listRows(dataset.id); + expect(stored.resources).toEqual({ task: { uri: "pg.ts#task" } }); + + expect((await updateResources({ "1bad": { uri: "pg.ts#db" } })).status).toBe( + 400, + ); + expect((await updateResources({ db: { uri: "" } })).status).toBe(400); + }); + it("rejects an unknown field or row, and a malformed body", async () => { const { provider, dataset, update } = await seeded(); expect((await update({ z: value("x") })).status).toBe(400); diff --git a/src/server/handlers/datasets.ts b/src/server/handlers/datasets.ts index 940c67b..303ba98 100644 --- a/src/server/handlers/datasets.ts +++ b/src/server/handlers/datasets.ts @@ -26,6 +26,7 @@ import { fitsPrimitiveBase, InvalidCellError, parseCell, + parseResources, primitiveBase, } from "../../shared/dataset-cells.ts"; import { @@ -202,7 +203,16 @@ function parseRows(value: unknown): NewDatasetRow[] { cells[fieldId] = parseCell(cell, `rows[${i}].cells.${fieldId}`); } const source = parseSource(row.source, `rows[${i}].source`); - return { cells, ...(source && { source }) }; + const resources = + row.resources === undefined || row.resources === null + ? undefined + : parseResources(row.resources, { path: `rows[${i}].resources` }); + return { + cells, + ...(source && { source }), + ...(resources && + Object.keys(resources).length > 0 && { resources }), + }; }); } @@ -251,7 +261,20 @@ function parseUpdates( } cells[fieldId] = parsed; } - return { rowId: update.rowId, cells }; + if (update.resources === undefined) return { rowId: update.rowId, cells }; + if (!isRecord(update.resources)) { + throw new BadRequest(`updates[${i}].resources must be an object`); + } + // `null` removes an instance; everything else is checked as a row's + // resources are when it's added. + const set = Object.fromEntries( + Object.entries(update.resources).filter(([, v]) => v !== null), + ); + const parsed = parseResources(set, { path: `updates[${i}].resources` }); + const resources = Object.fromEntries( + Object.keys(update.resources).map(name => [name, parsed[name] ?? null]), + ); + return { rowId: update.rowId, cells, resources }; }); } @@ -410,7 +433,7 @@ export async function handleDeleteDataset( } } -/** `POST /api/datasets/:providerId/:id/rows` — body `{ rows: [{ cells, source? }] }`. */ +/** `POST /api/datasets/:providerId/:id/rows` — body `{ rows: [{ cells, source?, resources? }] }`. */ export async function handleAddRows( provider: DatasetProvider, datasetId: string, @@ -427,7 +450,8 @@ export async function handleAddRows( /** * `PATCH /api/datasets/:providerId/:id/rows` — body - * `{ updates: [{ rowId, cells }] }`, where a `null` cell clears. All or + * `{ updates: [{ rowId, cells, resources? }] }`, where a `null` cell or + * resource clears it. All or * nothing: one bad update rejects the batch. */ export async function handleUpdateRows( diff --git a/src/server/handlers/evals.ts b/src/server/handlers/evals.ts index a20af91..efa008c 100644 --- a/src/server/handlers/evals.ts +++ b/src/server/handlers/evals.ts @@ -26,7 +26,11 @@ import type { NewEvalDefinition, TraceCheckResult, } from "../../eval/eval-types.ts"; -import { InvalidCellError, parseBinding } from "../../shared/dataset-cells.ts"; +import { + InvalidCellError, + parseBinding, + parseResources, +} from "../../shared/dataset-cells.ts"; import type { ExecutionInput, PromptID } from "../../shared/types.ts"; import { errorResult, type HandlerResult } from "./result.ts"; @@ -113,7 +117,9 @@ function parseBindings( function parseInputs(value: unknown): EvalInputs { if (value === undefined) return { functionInputs: {}, executeInputs: {} }; if (!isRecord(value)) { - throw new BadRequest("inputs must be { functionInputs, executeInputs }"); + throw new BadRequest( + "inputs must be { functionInputs, executeInputs, resources? }", + ); } return { functionInputs: parseBindings( @@ -121,6 +127,12 @@ function parseInputs(value: unknown): EvalInputs { "inputs.functionInputs", ), executeInputs: parseBindings(value.executeInputs, "inputs.executeInputs"), + ...(value.resources !== undefined && { + resources: parseResources(value.resources, { + columns: true, + path: "inputs.resources", + }), + }), }; } diff --git a/src/server/handlers/prompts.ts b/src/server/handlers/prompts.ts index 4b79f9d..1744cb9 100644 --- a/src/server/handlers/prompts.ts +++ b/src/server/handlers/prompts.ts @@ -115,6 +115,7 @@ export async function handleExecutePrompt( { functionInputs: request?.functionInputs ?? [], executeInputs: request?.executeInputs ?? {}, + ...(request?.resources && { resources: request.resources }), }, { tracer, traceProviderId: defaultTraceProviderId }, ); diff --git a/src/server/run-prompt.ts b/src/server/run-prompt.ts index 62acdd1..61bd48a 100644 --- a/src/server/run-prompt.ts +++ b/src/server/run-prompt.ts @@ -29,6 +29,7 @@ import type { ExecutionInput, NormalizedPrompt, PromptRef, + RunResources, } from "../shared/types.ts"; /** @@ -89,14 +90,17 @@ export async function runPrompt( const inputs: { functionInputs: ExecutionInput[]; executeInputs: Record; + resources?: RunResources; } = { functionInputs: [...(request.functionInputs ?? [])], executeInputs: { ...request.executeInputs }, + ...(request.resources && { resources: { ...request.resources } }), }; // The request is itself a complete set of bindings, so `input` references - // resolve against it (`specs/evals.md` §B.2.1). Unknown targets and cycles - // are refused before anything is created. + // resolve against it (`specs/evals.md` §B.2.1). Unknown targets — a slot + // the prompt lacks, an instance the run doesn't declare — and cycles are + // refused before anything is created. const bindings = namedBindings(prompt.functionParameters, inputs); const problems = inputReferenceProblems(bindings, prompt); if (problems.length > 0) { @@ -133,9 +137,14 @@ export async function runPrompt( } const { functionParams, executeValues } = resolved; // What actually gets recorded on the trace: the request as sent, with each - // resource reference's receipt filled in from what this run's resolution + // resource instance's receipt filled in from what this run's resolution // produced — see `specs/resource-arguments.md` §K. - const recordedInputs = stampReceipts(inputs, resolved.receipts); + const recordedResources = stampReceipts(inputs.resources, resolved.receipts); + const recordedInputs = { + functionInputs: inputs.functionInputs, + executeInputs: inputs.executeInputs, + ...(recordedResources && { resources: recordedResources }), + }; let settle!: () => void; const settled = new Promise(resolve => { diff --git a/src/shared/dataset-cells.test.ts b/src/shared/dataset-cells.test.ts index 5d0047c..c621bbd 100644 --- a/src/shared/dataset-cells.test.ts +++ b/src/shared/dataset-cells.test.ts @@ -7,62 +7,45 @@ import { fitsPrimitiveBase, InvalidCellError, parseCell, + parseResources, primitiveBase, stripReceipts, } from "./dataset-cells.ts"; describe("stripReceipts", () => { - it("removes receipts inside objects and resource arguments", () => { + it("removes each instance's receipt, keeping its arguments", () => { + const args = { + parentId: { kind: "instance" as const, name: "root", output: "id" }, + }; expect( stripReceipts({ - kind: "object", - properties: { - task: { - kind: "resource", - uri: "a#t", - receipt: 1, - args: { db: { kind: "resource", uri: "a#db", receipt: 2 } }, - }, - }, + root: { uri: "a#t", receipt: 1 }, + child: { uri: "a#t", args, receipt: 2 }, }), - ).toEqual({ - kind: "object", - properties: { - task: { - kind: "resource", - uri: "a#t", - args: { db: { kind: "resource", uri: "a#db" } }, - }, - }, - }); + ).toEqual({ root: { uri: "a#t" }, child: { uri: "a#t", args } }); }); }); describe("parseCell", () => { - it("accepts the three storable variants and strips receipts", () => { + it("accepts the storable variants", () => { const value = { kind: "value", value: { kind: "primitive", value: 1 } }; expect(parseCell(value)).toEqual(value); - expect( - parseCell({ - kind: "object", - properties: { r: { kind: "resource", uri: "a#r", receipt: "x" } }, - }), - ).toEqual({ + const object = { kind: "object", - properties: { r: { kind: "resource", uri: "a#r" } }, - }); + properties: { r: { kind: "instance", name: "db", output: "url" } }, + }; + expect(parseCell(object)).toEqual(object); }); it("names where a nested problem is", () => { expect(() => parseCell({ - kind: "resource", - uri: "a#r", - args: { n: { kind: "dataset", field: "1" } }, + kind: "object", + properties: { n: { kind: "dataset", field: "1" } }, }), ).toThrow( new InvalidCellError( - "cell.args.n is a column reference; a dataset cell must hold a value, object, resource, or slot reference", + "cell.properties.n is a column reference; a dataset cell must hold a value, object, resource, or slot reference", ), ); }); @@ -78,8 +61,9 @@ describe("parseCell", () => { "text", { kind: "value" }, { kind: "object", properties: [] }, - { kind: "resource", uri: "" }, - { kind: "resource", uri: "a", args: "x" }, + { kind: "instance", name: "" }, + { kind: "instance", name: "a", output: 1 }, + { kind: "resource", uri: "a#r" }, { kind: "input", half: "other", path: "x" }, { kind: "input", half: "function", path: "" }, ])("rejects %j", cell => { @@ -87,6 +71,52 @@ describe("parseCell", () => { }); }); +describe("parseResources", () => { + it("accepts named instances, stripping receipts", () => { + expect( + parseResources({ + db: { uri: "a#db", receipt: "r" }, + task: { + uri: "a#task", + args: { + db: { kind: "instance", name: "db" }, + title: { kind: "value", value: { kind: "primitive", value: "T" } }, + }, + }, + }), + ).toEqual({ + db: { uri: "a#db" }, + task: { + uri: "a#task", + args: { + db: { kind: "instance", name: "db" }, + title: { kind: "value", value: { kind: "primitive", value: "T" } }, + }, + }, + }); + }); + + it("lets arguments name a column only when asked to", () => { + const resources = { + task: { uri: "a#task", args: { title: { kind: "dataset", field: "0" } } }, + }; + expect(() => parseResources(resources)).toThrow( + /resources\.task\.args\.title is a column reference/, + ); + expect(parseResources(resources, { columns: true })).toEqual(resources); + }); + + it.each([ + [null, /resources must be an object/], + [{ "1st": { uri: "a#t" } }, /resources\.1st: a resource name/], + [{ t: "a#t" }, /resources\.t must be an object/], + [{ t: { uri: "" } }, /resources\.t\.uri must be a non-empty string/], + [{ t: { uri: "a#t", args: [] } }, /resources\.t\.args must be an object/], + ])("rejects %j", (value, message) => { + expect(() => parseResources(value)).toThrow(message); + }); +}); + describe("primitiveBase", () => { it("reads a recorded base, or a bare primitive's syntax", () => { expect( diff --git a/src/shared/dataset-cells.ts b/src/shared/dataset-cells.ts index 0b5573f..20d8da3 100644 --- a/src/shared/dataset-cells.ts +++ b/src/shared/dataset-cells.ts @@ -11,41 +11,27 @@ import { isPrimitiveFieldType, type PrimitiveFieldType, } from "./dataset-fields.ts"; -import type { ExecutionInput, PropType, PropValue } from "./types.ts"; +import type { + ExecutionInput, + PropType, + PropValue, + RunResources, +} from "./types.ts"; /** - * `input` without any resource receipt, at any depth. + * `resources` without any receipt. * * A receipt makes `create` reconstruct a *past* run's identity; a dataset row * is a recipe for new runs, so two rows keeping the same receipt would * collide on the id it names the moment the dataset is run. */ -export function stripReceipts(input: ExecutionInput): ExecutionInput { - switch (input.kind) { - case "resource": { - const { receipt: _receipt, args, ...rest } = input; - return args - ? { - ...rest, - args: Object.fromEntries( - Object.entries(args).map(([k, v]) => [k, stripReceipts(v)]), - ), - } - : rest; - } - case "object": - return { - kind: "object", - properties: Object.fromEntries( - Object.entries(input.properties).map(([k, v]) => [ - k, - stripReceipts(v), - ]), - ), - }; - default: - return input; - } +export function stripReceipts(resources: RunResources): RunResources { + return Object.fromEntries( + Object.entries(resources).map(([name, { receipt: _receipt, ...spec }]) => [ + name, + spec, + ]), + ); } /** Thrown by {@link parseCell} for anything that isn't a storable cell. */ @@ -59,10 +45,10 @@ function isRecord(value: unknown): value is Record { /** * Checks that `value` is a well-formed {@link ExecutionInput} a dataset cell - * may hold, and returns it with receipts removed. + * may hold, and returns it. * - * The check is structural: `value` must be one of the `value`, `object`, or - * `resource` variants, recursively. The `dataset` variant is refused — a row + * The check is structural: `value` must be one of the `value`, `object`, + * `instance` or `input` variants, recursively. The `dataset` variant is refused — a row * holds copies, not references to other rows. A `value`'s `PropValue` is only * checked for being an object with a `kind`; its full shape is the editor's * business, and it round-trips as JSON either way. @@ -115,25 +101,17 @@ function parseInput( ), }; } - case "resource": { - if (typeof value.uri !== "string" || value.uri === "") { - throw new InvalidCellError(`${path}.uri must be a non-empty string`); + case "instance": { + if (typeof value.name !== "string" || value.name === "") { + throw new InvalidCellError(`${path}.name must be a non-empty string`); } - if (value.args !== undefined && !isRecord(value.args)) { - throw new InvalidCellError(`${path}.args must be an object`); + if (value.output !== undefined && typeof value.output !== "string") { + throw new InvalidCellError(`${path}.output must be a string`); } - const args = value.args as Record | undefined; return { - kind: "resource", - uri: value.uri, - ...(args && { - args: Object.fromEntries( - Object.entries(args).map(([k, v]) => [ - k, - recurse(v, `${path}.args.${k}`), - ]), - ), - }), + kind: "instance", + name: value.name, + ...(value.output !== undefined && { output: value.output }), }; } case "input": { @@ -164,9 +142,59 @@ function parseInput( ); default: throw new InvalidCellError( - `${path}.kind must be "value", "object", "resource", or "input"`, + `${path}.kind must be "value", "object", "instance", or "input"`, + ); + } +} + +/** + * Checks that `value` is a well-formed {@link RunResources} map — a dataset + * row's, or an eval's — and returns it with receipts removed. Each + * instance's `args` are checked as {@link parseCell} checks a cell, or, with + * `columns`, as {@link parseBinding} does. Names are checked for shape only; + * whether the `uri` names a real resource is the provider's business at run + * time. See `specs/resource-instances.md` §C. + * + * @param columns - Whether arguments may name a column of the row being run. + * @param path - Where `value` sits, for the error message. + * @throws {InvalidCellError} naming the first problem found. + */ +export function parseResources( + value: unknown, + { columns = false, path = "resources" }: { columns?: boolean; path?: string } = {}, +): RunResources { + if (!isRecord(value)) { + throw new InvalidCellError(`${path} must be an object`); + } + const out: RunResources = {}; + for (const [name, spec] of Object.entries(value)) { + const at = `${path}.${name}`; + if (!/^[A-Za-z_][A-Za-z0-9_-]*$/.test(name)) { + throw new InvalidCellError( + `${at}: a resource name uses letters, digits, '_' and '-', not starting with a digit or '-'`, ); + } + if (!isRecord(spec)) throw new InvalidCellError(`${at} must be an object`); + if (typeof spec.uri !== "string" || spec.uri === "") { + throw new InvalidCellError(`${at}.uri must be a non-empty string`); + } + if (spec.args !== undefined && !isRecord(spec.args)) { + throw new InvalidCellError(`${at}.args must be an object`); + } + const args = spec.args as Record | undefined; + out[name] = { + uri: spec.uri, + ...(args && { + args: Object.fromEntries( + Object.entries(args).map(([k, v]) => [ + k, + parseInput(v, `${at}.args.${k}`, columns), + ]), + ), + }), + }; } + return out; } /** diff --git a/src/shared/eval-prefill.ts b/src/shared/eval-prefill.ts index efccde9..69901b4 100644 --- a/src/shared/eval-prefill.ts +++ b/src/shared/eval-prefill.ts @@ -178,6 +178,7 @@ export function prefillBindings(input: PrefillInput): Prefilled { stored?.executeInputs, "exec", ), + ...(input.inputs.resources && { resources: input.inputs.resources }), }; const checks = input.checks.map(check => { diff --git a/src/shared/eval-problems.ts b/src/shared/eval-problems.ts index 1cc1a61..c81bb43 100644 --- a/src/shared/eval-problems.ts +++ b/src/shared/eval-problems.ts @@ -36,8 +36,6 @@ function columnRefs(input: ExecutionInput): string[] { return [input.field]; case "object": return Object.values(input.properties).flatMap(columnRefs); - case "resource": - return Object.values(input.args ?? {}).flatMap(columnRefs); default: return []; } @@ -52,8 +50,6 @@ function slotRefs( return [input]; case "object": return Object.values(input.properties).flatMap(slotRefs); - case "resource": - return Object.values(input.args ?? {}).flatMap(slotRefs); default: return []; } @@ -93,6 +89,10 @@ function unboundRequired( * - a binding to a slot, column, or check that no longer exists; * - an `input` reference to a slot the prompt doesn't have, or a cycle. * + * An `instance` reference to a resource the eval doesn't declare isn't one: + * each row may declare it (`specs/resource-instances.md` §E), and a row that + * doesn't fails on its own when it runs. + * * Empty when the eval can run. A half-bound eval is still a legitimate * draft: this gates Run, never Save. */ @@ -135,7 +135,16 @@ export function evalProblems( } } } - if (prompt) problems.push(...inputReferenceProblems(inputs, prompt)); + for (const [name, spec] of Object.entries(inputs.resources ?? {})) { + for (const [arg, binding] of Object.entries(spec.args ?? {})) { + missingColumns(binding, `Resource '${name}': '${arg}'`); + } + } + if (prompt) { + problems.push( + ...inputReferenceProblems(inputs, prompt, { undeclaredInstances: true }), + ); + } const slots = prompt && slotPaths(prompt); for (const check of checks) { diff --git a/src/shared/input-references.ts b/src/shared/input-references.ts index 6fb7566..e4172d6 100644 --- a/src/shared/input-references.ts +++ b/src/shared/input-references.ts @@ -8,7 +8,12 @@ * §B.2. */ -import type { ExecutionInput, PropDefinition, PropType } from "./types.ts"; +import type { + ExecutionInput, + PropDefinition, + PropType, + RunResources, +} from "./types.ts"; /** * One place an input can be plugged in: a prompt's parameter, or any slot @@ -71,6 +76,11 @@ export interface InputBindings { functionInputs: Record; /** Execute parameter name → its binding. */ executeInputs: Record; + /** + * The run's named resource instances, whose arguments may themselves hold + * `input` and `instance` references. See `specs/resource-instances.md`. + */ + resources?: RunResources; } /** An `input` reference's target, as one string: `execute:` prefixed for an execute slot. */ @@ -91,6 +101,7 @@ export function namedBindings( inputs: { functionInputs?: readonly ExecutionInput[]; executeInputs?: Record; + resources?: RunResources; }, ): InputBindings { const functionInputs: Record = {}; @@ -98,85 +109,116 @@ export function namedBindings( const input = inputs.functionInputs?.[i]; if (input) functionInputs[param.name] = input; }); - return { functionInputs, executeInputs: { ...inputs.executeInputs } }; + return { + functionInputs, + executeInputs: { ...inputs.executeInputs }, + ...(inputs.resources && { resources: inputs.resources }), + }; +} + +/** + * Somewhere a reference can sit, or point: a slot (or a path within one), or + * one of the run's resource instances (whose arguments count as the + * instance itself). + */ +type RefLocation = + | { half: "function" | "execute"; path: string } + | { instance: string }; + +/** One `input` or `instance` reference found in a set of bindings, and where it sits. */ +interface FoundRef { + /** Where the reference sits. */ + at: RefLocation; + /** What it names. */ + target: RefLocation; } -/** One `input` reference found in a set of bindings, and where it sits. */ -interface FoundInputRef { - /** The slot the reference sits in (a resource's arguments count as its own slot). */ - at: { half: "function" | "execute"; path: string }; - /** The slot it names. */ - target: { half: "function" | "execute"; path: string }; +/** How a {@link RefLocation} reads in a message: a slot as {@link inputKey}, an instance as `resource 'name'`. */ +function locationKey(loc: RefLocation): string { + return "instance" in loc + ? `resource '${loc.instance}'` + : inputKey(loc.half, loc.path); } -/** Every `input` reference inside `bindings`, with where it sits. */ -function findInputRefs(bindings: InputBindings): FoundInputRef[] { - const found: FoundInputRef[] = []; - const walk = ( - node: ExecutionInput, - half: "function" | "execute", - path: string, - ) => { +/** Every `input` and `instance` reference inside `bindings`, with where it sits. */ +function findRefs(bindings: InputBindings): FoundRef[] { + const found: FoundRef[] = []; + const walk = (node: ExecutionInput, at: RefLocation) => { switch (node.kind) { case "object": for (const [k, child] of Object.entries(node.properties)) { - walk(child, half, `${path}.${k}`); + walk( + child, + "instance" in at ? at : { half: at.half, path: `${at.path}.${k}` }, + ); } break; - case "resource": - // Arguments are resolved as part of this slot's own value, so a - // reference among them sits here, however deep. - for (const arg of Object.values(node.args ?? {})) walk(arg, half, path); - break; case "input": - found.push({ - at: { half, path }, - target: { half: node.half, path: node.path }, - }); + found.push({ at, target: { half: node.half, path: node.path } }); + break; + case "instance": + found.push({ at, target: { instance: node.name } }); break; default: break; } }; for (const [name, node] of Object.entries(bindings.functionInputs)) { - walk(node, "function", name); + walk(node, { half: "function", path: name }); } for (const [name, node] of Object.entries(bindings.executeInputs)) { - walk(node, "execute", name); + walk(node, { half: "execute", path: name }); + } + for (const [name, spec] of Object.entries(bindings.resources ?? {})) { + // Arguments are resolved as part of creating the instance, so a + // reference among them sits at the instance itself, however deep. + for (const arg of Object.values(spec.args ?? {})) { + walk(arg, { instance: name }); + } } return found; } +/** Whether resolving whatever sits at `at` means resolving `target` first. */ +function reaches(at: RefLocation, target: RefLocation): boolean { + if ("instance" in at || "instance" in target) { + return ( + "instance" in at && + "instance" in target && + at.instance === target.instance + ); + } + return at.half === target.half && overlaps(at.path, target.path); +} + /** Whether one dotted path is the other, or contains it. */ export function overlaps(a: string, b: string): boolean { return a === b || a.startsWith(`${b}.`) || b.startsWith(`${a}.`); } /** - * A cycle among `bindings`' `input` references — `a` ← input `b` ← input - * `a` — as the slot paths that form it, first repeated last; or `undefined` - * when there is none. Resolving a slot means resolving every reference that - * sits inside it or around it, which is what an edge here is. See - * `specs/evals.md` §B.2. + * A cycle among `bindings`' `input` and `instance` references — `a` ← input + * `b` ← input `a`, or two resource instances each taking the other's output + * as an argument — as the places that form it, first repeated last; or + * `undefined` when there is none. Resolving a slot means resolving every + * reference that sits inside it or around it, and creating an instance means + * resolving every reference among its arguments, which is what an edge here + * is. See `specs/evals.md` §B.2 and `specs/resource-instances.md` §B. */ export function findInputCycle(bindings: InputBindings): string[] | undefined { - const refs = findInputRefs(bindings); + const refs = findRefs(bindings); const edges = refs.map(ref => - refs.filter( - other => - other.at.half === ref.target.half && - overlaps(other.at.path, ref.target.path), - ), + refs.filter(other => reaches(other.at, ref.target)), ); - const state = new Map(); - const stack: FoundInputRef[] = []; + const state = new Map(); + const stack: FoundRef[] = []; - const visit = (ref: FoundInputRef): string[] | undefined => { + const visit = (ref: FoundRef): string[] | undefined => { const seen = state.get(ref); if (seen === "done") return undefined; if (seen === "visiting") { const loop = stack.slice(stack.indexOf(ref)); - return [...loop, ref].map(r => inputKey(r.at.half, r.at.path)); + return [...loop, ref].map(r => locationKey(r.at)); } state.set(ref, "visiting"); stack.push(ref); @@ -217,21 +259,38 @@ export function slotPaths( } /** - * What's wrong with `bindings`' `input` references against a prompt's - * signature: a target slot that doesn't exist, and a cycle. Empty when - * nothing is. The execute route answers a non-empty list with a 400, and an - * eval run refuses to start. See `specs/evals.md` §B.2.1. + * What's wrong with `bindings`' `input` and `instance` references against a + * prompt's signature: a target slot that doesn't exist, a resource instance + * the run doesn't declare, and a cycle. Empty when nothing is. The execute + * route answers a non-empty list with a 400, and an eval run refuses to + * start. See `specs/evals.md` §B.2.1. + * + * @param options.undeclaredInstances - Allow `instance` references to names + * `bindings.resources` doesn't declare — an eval's, which each dataset row + * may declare instead (`specs/resource-instances.md` §E). */ export function inputReferenceProblems( bindings: InputBindings, signature: InputSignature, + options: { undeclaredInstances?: boolean } = {}, ): string[] { const slots = slotPaths(signature); + const instances = bindings.resources ?? {}; const problems: string[] = []; - for (const ref of findInputRefs(bindings)) { - if (!slots[ref.target.half].has(ref.target.path)) { + for (const ref of findRefs(bindings)) { + const at = "instance" in ref.at ? locationKey(ref.at) : `'${locationKey(ref.at)}'`; + if ("instance" in ref.target) { + if ( + !options.undeclaredInstances && + !Object.hasOwn(instances, ref.target.instance) + ) { + problems.push( + `${at} names resource '${ref.target.instance}', which this run doesn't declare`, + ); + } + } else if (!slots[ref.target.half].has(ref.target.path)) { problems.push( - `'${inputKey(ref.at.half, ref.at.path)}' names input '${inputKey(ref.target.half, ref.target.path)}', which the prompt doesn't have`, + `${at} names input '${inputKey(ref.target.half, ref.target.path)}', which the prompt doesn't have`, ); } } diff --git a/src/shared/types.ts b/src/shared/types.ts index 5bd5562..830dbfc 100644 --- a/src/shared/types.ts +++ b/src/shared/types.ts @@ -471,6 +471,14 @@ export interface ResourceInfo { * `specs/resource-arguments.md` §G. */ parameters?: PropDefinition[]; + /** + * The resource's code-wired dependencies — its resource-valued `inputs` + * entries — as input name → the dependency's {@link uri}. Only the + * resource itself carries this, and only for dependencies discovery + * registered. The panel lists them in its Resources section. See + * `specs/resource-instances.md` §D. + */ + dependencies?: Record; } /** @@ -524,27 +532,17 @@ export type ExecutionInput = * any slot has yet. */ | { kind: "object"; properties: Record } - /** A code-defined resource, created by the provider at run time. */ + /** + * One of the run's named resource instances ({@link RunResources}), or one + * of its outputs — created by the provider at run time, once per run + * however many places name it. See `specs/resource-instances.md` §A. + */ | { - kind: "resource"; - /** The resource's {@link ResourceInfo.uri}. */ - uri: string; - /** - * Values for the resource's declared arguments, by parameter name. - * Absent when the resource takes none. Recursive — an argument may - * itself be a typed-in value, an object, another resource, or (later) - * a dataset cell — because those are exactly the variants this union - * already has. Never folded into `uri`: identity and binding change on - * different schedules. See `specs/resource-arguments.md` §C. - */ - args?: Record; - /** - * What this resource's `create()` produced. Recorded on every run - * (never sent by the panel); sent back only by a replay, so `create` - * can reconstruct rather than re-mint. See - * `specs/resource-arguments.md` §E. - */ - receipt?: unknown; + kind: "instance"; + /** The instance's name in the run's {@link RunResources}. */ + name: string; + /** A top-level output of its value (`taskId`). Absent for the value itself. */ + output?: string; } /** * A cell of the dataset row being run, by `DatasetField.id`. Only @@ -568,6 +566,35 @@ export type ExecutionInput = path: string; }; +/** + * One code-defined resource instance a run creates, whether or not any slot + * reads it — the run's resources are the instances it declares, by name. See + * `specs/resource-instances.md` §A. + */ +export interface ResourceInstanceInput { + /** The resource's root {@link ResourceInfo.uri} — never an output's. */ + uri: string; + /** + * Values for the resource's declared arguments, by parameter name. Absent + * when it takes none. Recursive — an argument may be a typed-in value, an + * object, another instance's output, a dataset cell, or another slot. + */ + args?: Record; + /** + * What this instance's `create()` produced. Recorded on every run (never + * sent by the panel); sent back only by a replay, so `create` can + * reconstruct rather than re-mint. See `specs/resource-arguments.md` §E. + */ + receipt?: unknown; +} + +/** + * A run's named resource instances, by name — `[A-Za-z_][A-Za-z0-9_-]*`. + * Every one is created when the run starts, and {@link ExecutionInput}s of + * kind `instance` refer to them by name. + */ +export type RunResources = Record; + /** * What the client knows about a check — a playground export made with * `check()`, or a built-in from `evalution/checks`. See `specs/evals.md` @@ -692,6 +719,8 @@ export interface ExecuteRequest { functionInputs?: ExecutionInput[]; /** By name, keyed on {@link NormalizedPrompt.executeParameters} entries. */ executeInputs?: Record; + /** The run's named resource instances. See {@link RunResources}. */ + resources?: RunResources; } /** diff --git a/src/trace/otel-attributes.ts b/src/trace/otel-attributes.ts index 3de1667..21927ca 100644 --- a/src/trace/otel-attributes.ts +++ b/src/trace/otel-attributes.ts @@ -369,6 +369,7 @@ export function readInputs( PromptID, | "functionInputs" | "executeInputs" + | "resources" | "parameterDefinitions" | "executeParameterDefinitions" > { @@ -383,6 +384,11 @@ export function readInputs( ...(parsed.executeInputs && typeof parsed.executeInputs === "object" ? { executeInputs: parsed.executeInputs } : {}), + ...(parsed.resources && + typeof parsed.resources === "object" && + !Array.isArray(parsed.resources) + ? { resources: parsed.resources } + : {}), ...(Array.isArray(parsed.parameterDefinitions) && { parameterDefinitions: parsed.parameterDefinitions, }), diff --git a/src/trace/prompt-tracer.ts b/src/trace/prompt-tracer.ts index ecde936..07ed995 100644 --- a/src/trace/prompt-tracer.ts +++ b/src/trace/prompt-tracer.ts @@ -107,6 +107,11 @@ export interface PromptSpanInfo { functionInputs?: unknown[]; /** The unresolved execute-parameter inputs, keyed by parameter name. */ executeInputs?: Record; + /** + * The run's named resource instances, by name, each with the receipt its + * `create()` produced. See `specs/resource-instances.md` §F. + */ + resources?: Record; /** * The prompt's parameter definitions as they stood when the run was * launched, recorded beside the inputs. @@ -170,6 +175,7 @@ export function getPromptSpanAttributes( ? JSON.stringify({ functionInputs: prompt.functionInputs, executeInputs: prompt.executeInputs, + resources: prompt.resources, parameterDefinitions: prompt.parameterDefinitions, executeParameterDefinitions: prompt.executeParameterDefinitions, }) diff --git a/src/trace/trace-types.ts b/src/trace/trace-types.ts index 45c1937..330a644 100644 --- a/src/trace/trace-types.ts +++ b/src/trace/trace-types.ts @@ -33,6 +33,11 @@ export interface PromptID { functionInputs?: unknown[]; /** The unresolved execute-parameter inputs, keyed by parameter name. */ executeInputs?: Record; + /** + * The run's named resource instances, by name, with the receipt each + * `create()` produced. See `specs/resource-instances.md` §F. + */ + resources?: Record; /** * The prompt's parameter definitions as they stood when the run was * launched, so a replay can diff them against today's signature rather than From a58eb2c7a27bc6fe3d172d6ecc1bde9e8462c9db Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 14:15:57 +0000 Subject: [PATCH 04/12] Add the Resources section: named instances in every host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The playground panel, the eval Inputs panel and dataset row details now share one Resources section above their slots. Each named instance gets a card with its name, resource, what references it (or that it runs only for its side effects), and its argument form. Slots hold `◆ name.output` references; picking from the catalog under "New" adds an instance and binds it in one step, and "Resources in this run" reuses an existing one. - run-resources-state.ts: pure instance state (add, adopt a catalog pick, rename, remove, duplicate, references, derived dependencies, cycle-safe argument sources), unit-tested. - ResourcesSection.tsx and instanceSourceContext replace the per-slot argument forms, claims and "same instance as" notes. - Hosts persist instances with their other inputs: localStorage, the eval definition, and the dataset row (adopt and bind save as one update). - Instances of static resources preview their value in the slot. - Dataset grid shows instance cells as chips; describeRows drops the resource flag. - Browser tests rewritten for the section, with new ones for unbound instances, New vs. reuse, rename/duplicate, remove, hierarchy and cycles, and the eval and row-details hosts. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- src/client/App.tsx | 20 +- src/client/api.ts | 2 +- src/client/components/AddToDatasetMenu.tsx | 10 +- src/client/components/CombinedInputEditor.tsx | 19 +- src/client/components/DatasetRowDetails.tsx | 433 +++++++++------ src/client/components/DatasetView.tsx | 45 +- src/client/components/EvalView.tsx | 206 ++++--- .../components/ExecutionInputEditor.tsx | 44 +- src/client/components/PlaygroundExecution.tsx | 334 +++++------- src/client/components/ResourcesSection.tsx | 384 +++++++++++++ src/client/components/SourcePicker.tsx | 29 +- src/client/components/SourceRow.tsx | 265 +++------ src/client/components/TraceView.tsx | 2 + .../__fixtures__/datasetView.pw.tsx | 178 +++--- .../components/__fixtures__/evalView.pw.tsx | 54 ++ .../__fixtures__/executionFixtures.ts | 39 +- .../__fixtures__/playgroundExecution.pw.tsx | 508 ++++++++++-------- src/client/components/combined-inputs.test.ts | 2 +- src/client/components/combined-inputs.ts | Bin 10355 -> 9710 bytes src/client/components/dataset-grid.test.ts | 46 +- src/client/components/dataset-grid.ts | 95 ++-- src/client/components/dataset-preview.test.ts | 16 +- src/client/components/dataset-preview.ts | 22 +- .../components/dataset-row-editing.test.ts | 90 ++-- src/client/components/dataset-row-editing.ts | 109 ++-- src/client/components/eval-bindings.test.ts | 8 +- .../components/execution-input-state.test.ts | 238 ++------ .../components/execution-input-state.ts | 190 ++----- src/client/components/named-inputs.test.ts | 55 +- src/client/components/named-inputs.ts | 78 ++- src/client/components/pseudo-sources.test.ts | 96 +++- src/client/components/pseudo-sources.ts | 177 +++++- .../components/resource-args-context.ts | 193 ------- .../components/run-resources-state.test.ts | 218 ++++++++ src/client/components/run-resources-state.ts | 388 +++++++++++++ src/client/components/source-context.ts | 43 ++ src/client/components/source-tree.ts | 15 +- src/client/components/use-run-instances.ts | 67 +++ src/client/styles.css | 218 ++++++-- src/dataset/dataset-provider-contract.ts | 5 +- src/dataset/turso-dataset-provider.ts | 45 +- src/index.ts | 8 +- src/mcp/server.ts | 6 +- src/prompt/execution-inputs.test.ts | 10 +- src/prompt/execution-inputs.ts | 5 +- .../playground/resource-registry.test.ts | 44 +- src/prompt/playground/resource-registry.ts | 10 +- src/server/handlers/datasets.test.ts | 6 +- src/server/handlers/datasets.ts | 3 +- src/shared/dataset-cells.ts | 8 +- src/shared/eval-prefill.ts | 49 +- src/shared/input-references.ts | 3 +- src/shared/instance-names.ts | 13 + 53 files changed, 3187 insertions(+), 1964 deletions(-) create mode 100644 src/client/components/ResourcesSection.tsx delete mode 100644 src/client/components/resource-args-context.ts create mode 100644 src/client/components/run-resources-state.test.ts create mode 100644 src/client/components/run-resources-state.ts create mode 100644 src/client/components/source-context.ts create mode 100644 src/client/components/use-run-instances.ts create mode 100644 src/shared/instance-names.ts diff --git a/src/client/App.tsx b/src/client/App.tsx index 97658d3..fa8e592 100644 --- a/src/client/App.tsx +++ b/src/client/App.tsx @@ -18,6 +18,7 @@ import type { NormalizedPrompt, PromptID, PromptRef, + RunResources, SSEData, } from "../shared/types"; import { getAgentCommand, getAgents, renamePrompt } from "./api"; @@ -182,7 +183,7 @@ function flashPane(paneId: string) { block: "nearest", inline: "nearest", }); - // Restart-safe, as in `SourceRow.tsx`'s `scrollToRow`: removing the class + // Restart-safe, as in `ResourcesSection.tsx`'s `scrollToInstance`: removing the class // and forcing a reflow before adding it back makes a repeat click flash // again instead of no-op-ing into an animation already in flight. el.classList.remove("pane-flash"); @@ -783,12 +784,17 @@ function App() { const current = findPrompt(recorded); const fill = current && hasRecordedInputs(recorded) - ? panelFill(fromTrace(recorded, current), current, { - type: "trace", - description: `trace ${traceId.slice(0, 8)}…`, - providerId: traceProviderId, - traceId, - }) + ? panelFill( + fromTrace(recorded, current), + current, + { + type: "trace", + description: `trace ${traceId.slice(0, 8)}…`, + providerId: traceProviderId, + traceId, + }, + recorded.resources as RunResources | undefined, + ) : undefined; // At the variation the run applied, else the version it ran against — // the server opens head when that version is what's on disk now. diff --git a/src/client/api.ts b/src/client/api.ts index adf385f..6d7de3f 100644 --- a/src/client/api.ts +++ b/src/client/api.ts @@ -665,7 +665,7 @@ export async function deleteDataset( export async function addDatasetRows( providerId: string, datasetId: string, - rows: Pick[], + rows: Pick[], ): Promise { const res = await fetch(datasetUrl(providerId, datasetId, "/rows"), { method: "POST", diff --git a/src/client/components/AddToDatasetMenu.tsx b/src/client/components/AddToDatasetMenu.tsx index 9642901..7cfbc93 100644 --- a/src/client/components/AddToDatasetMenu.tsx +++ b/src/client/components/AddToDatasetMenu.tsx @@ -15,6 +15,7 @@ import type { DatasetRowSource, DatasetSummary, PromptID, + RunResources, } from "../../shared/types"; import { addDatasetRows, @@ -25,6 +26,7 @@ import { import { countMatches, type NamedInputs, + rowResources, type SkippedInput, toCells, } from "./named-inputs"; @@ -34,6 +36,8 @@ import { useAnchoredPopover } from "./use-anchored-popover"; interface Props { /** What would be added — one row's worth. */ inputs: NamedInputs; + /** The resource instances `inputs` may reference, kept on the row. */ + resources?: RunResources; /** The schema "New dataset…" creates. */ newDatasetFields: Omit[]; /** @@ -86,6 +90,7 @@ function describeSkipped(skipped: SkippedInput[]): string { */ export function AddToDatasetMenu({ inputs, + resources, newDatasetFields, prompt, source, @@ -150,7 +155,10 @@ export function AddToDatasetMenu({ fields: readonly DatasetField[], ) => { const { cells, skipped } = toCells(inputs, fields); - await addDatasetRows(target.providerId, target.id, [{ cells, source }]); + const kept = rowResources(resources); + await addDatasetRows(target.providerId, target.id, [ + { cells, source, ...(kept && { resources: kept }) }, + ]); setOutcome({ ok: true, message: `Added to ${target.name}${describeSkipped(skipped)}`, diff --git a/src/client/components/CombinedInputEditor.tsx b/src/client/components/CombinedInputEditor.tsx index 3b37f78..db27a84 100644 --- a/src/client/components/CombinedInputEditor.tsx +++ b/src/client/components/CombinedInputEditor.tsx @@ -7,7 +7,7 @@ import type { CombinedSelection, FieldGroup } from "./combined-inputs"; import { memberCount } from "./combined-inputs"; import { ExecutionInputEditor } from "./ExecutionInputEditor"; import type { SlotSelection } from "./execution-input-state"; -import { nestedField, type ResourceArgsContext } from "./resource-args-context"; +import type { SourceContext } from "./source-context"; interface Props { /** The fan-out slot itself (e.g. `toolsContext`). */ @@ -28,8 +28,8 @@ interface Props { * edit to that row. */ overwritten?: Record; - /** Forwarded to each row's {@link ExecutionInputEditor} unchanged — a chip with an argument form under it is still a chip. */ - argsContext?: ResourceArgsContext; + /** Forwarded to each row's {@link ExecutionInputEditor} unchanged. */ + context?: SourceContext; } /** @@ -46,7 +46,7 @@ export function CombinedInputEditor({ resources, slots, overwritten, - argsContext, + context, }: Props) { const totalMembers = memberCount(groups); @@ -98,16 +98,7 @@ export function CombinedInputEditor({ } resources={resources} slots={rowSlots(slots, propDef.name, group)} - // The row's identity for claiming an argument form (§I) has to - // agree with `computeClaims`, which only ever sees the - // *expanded* selection — so this uses the same representative - // member `rowSlots`/`droppedCandidates` already privilege, - // rather than the group's own synthetic name, which no - // expanded path actually has. - argsContext={ - argsContext && - nestedField(argsContext, `${group.members[0]}.${group.name}`) - } + context={context} /> ); diff --git a/src/client/components/DatasetRowDetails.tsx b/src/client/components/DatasetRowDetails.tsx index 10c2891..9ed0cd0 100644 --- a/src/client/components/DatasetRowDetails.tsx +++ b/src/client/components/DatasetRowDetails.tsx @@ -14,21 +14,35 @@ import type { Dataset, DatasetField, DatasetRow, + DatasetRowUpdate, ExecutionInput, + PromptInputSources, PropValue, ResourceInfo, } from "../../shared/types"; import { DetailRow, type Fact, FactGroup, FactsGrid } from "./DetailsPane"; import { previewCell, propValueToJson, resourceName } from "./dataset-preview"; import { + changesAnything, chosenResources, type DatasetInputSources, type RowEditorState, - rowCellChanges, + rowChanges, rowEditorState, } from "./dataset-row-editing"; import { ExecutionInputEditor } from "./ExecutionInputEditor"; -import { computeClaims } from "./resource-args-context"; +import { withInstanceResources, withInstanceSlots } from "./pseudo-sources"; +import { + type InstanceEdit, + instanceSourceContext, + ResourcesSection, +} from "./ResourcesSection"; +import { + adoptCatalogPick, + type InstanceSelections, + referencesTo, + retargetSelections, +} from "./run-resources-state"; import { formatTimestamp } from "./trace/format.ts"; import { CalendarIcon, SpansIcon } from "./trace/icons.tsx"; import { JsonView } from "./trace/JsonView.tsx"; @@ -59,7 +73,7 @@ function ValueView({ value }: { value: PropValue }) { ); } -/** Named inputs — an object's properties or a resource's arguments. */ +/** Named inputs — an object's properties or a resource instance's arguments. */ function KeyedInputs({ inputs }: { inputs: Record }) { return (
@@ -75,7 +89,7 @@ function KeyedInputs({ inputs }: { inputs: Record }) { /** * A dataset cell in full: the one-line previews the table has room for, - * expanded — long text to read, and a resource's arguments beneath it. + * expanded — long text to read, and objects key by key. */ export function InputView({ input }: { input: ExecutionInput }) { switch (input.kind) { @@ -83,19 +97,14 @@ export function InputView({ input }: { input: ExecutionInput }) { return ; case "object": return ; - case "resource": + case "instance": return ( -
- - - ◆ - - {resourceName(input.uri)} + + + ◆ - {input.args && Object.keys(input.args).length > 0 && ( - - )} -
+ {previewCell(input)} + ); case "dataset": case "input": @@ -108,212 +117,326 @@ export function InputView({ input }: { input: ExecutionInput }) { /** A stable stand-in for "no resources", so memos keyed on it hold. */ const NO_RESOURCES: ResourceInfo[] = []; +/** A row's resource instances, read-only: each with its resource and arguments. */ +function RowResources({ row }: { row: DatasetRow }) { + const entries = Object.entries(row.resources ?? {}); + if (entries.length === 0) return null; + return ( + Resources}> +
+ {entries.map(([name, spec]) => ( +
+ ◆ {name} +
+ + {resourceName(spec.uri)} + + {spec.args && Object.keys(spec.args).length > 0 && ( + + )} +
+
+ ))} +
+ + ); +} + /** - * A row's fields as the execute panel edits its slots: each in - * `ExecutionInputEditor` against the field's `def`, offering whatever - * resources `sources` maps onto it, with their arguments. + * A row's resource instances and fields, as the execute panel edits its own: + * the Resources section first, then each field in `ExecutionInputEditor` + * against the field's `def`, offering the row's instances and whatever + * resources `sources` maps onto it. * * Typed edits stay a local draft and are committed on blur, on Enter * (Shift+Enter still types a newline), or when the editor goes away, rather * than on every keystroke — each commit is a save. Choosing or dropping a - * resource commits at once, as there's nothing more to type. A commit sends - * every field whose cell changed, since a resource's arguments are shared by - * every field that picked it. + * source, and adding, removing or renaming an instance, commits at once, as + * there's nothing more to type. A commit sends every field and instance that + * changed. */ function RowFields({ fields, row, sources, - onChangeCells, + onChangeRow, }: { fields: readonly DatasetField[]; row: DatasetRow; sources: DatasetInputSources | undefined; - onChangeCells?: (cells: Record) => void; + onChangeRow?: (update: Omit) => void; }) { - const resources = sources?.resources ?? NO_RESOURCES; - const resourcesByUri = useMemo( - () => new Map(resources.map(r => [r.uri, r])), - [resources], - ); - const [state, setState] = useState(() => - rowEditorState(row.cells, resourcesByUri), + const catalog = sources?.resources ?? NO_RESOURCES; + const catalogByUri = useMemo( + () => new Map(catalog.map(r => [r.uri, r])), + [catalog], ); + const [state, setState] = useState(() => rowEditorState(row)); + // Updates apply to the latest state synchronously, not to the one this + // render saw: a catalog pick adds an instance and binds it in one gesture. + const stateRef = useRef(state); const [dirty, setDirty] = useState(false); - // New stored cells — the refetch after a save — replace a clean draft. - const [stored, setStored] = useState(row.cells); - if (stored !== row.cells) { - setStored(row.cells); - if (!dirty) setState(rowEditorState(row.cells, resourcesByUri)); + // A new stored row — the refetch after a save — replaces a clean draft. + const [stored, setStored] = useState(row); + if (stored !== row) { + setStored(row); + if (!dirty) { + const fresh = rowEditorState(row); + stateRef.current = fresh; + setState(fresh); + } } - // A resource or `object` cell is only editable with resources to offer: + // An instance or `object` cell is only editable with resources to offer: // without them it would read as a resource that's gone. const isEditable = (field: DatasetField) => { const input = row.cells[field.id]; - return !!onChangeCells && (!!sources || !input || input.kind === "value"); + return !!onChangeRow && (!!sources || !input || input.kind === "value"); }; const editableIds = fields.filter(isEditable).map(f => f.id); const commit = (next: RowEditorState) => { setDirty(false); - const changes = rowCellChanges( + const changes = rowChanges( editableIds, - next, - row.cells, - resourcesByUri, + sources ? next : { ...next, instances: rowEditorState(row).instances }, + row, ); - if (Object.keys(changes).length > 0) onChangeCells?.(changes); + if (changesAnything(changes)) onChangeRow?.(changes); }; - const update = (next: RowEditorState) => { - setState(next); - if (chosenResources(next) !== chosenResources(state)) commit(next); + const update = ( + next: RowEditorState | ((latest: RowEditorState) => RowEditorState), + ) => { + const prev = stateRef.current; + const value = typeof next === "function" ? next(prev) : next; + stateRef.current = value; + setState(value); + if (chosenResources(value) !== chosenResources(prev)) commit(value); else setDirty(true); }; // Selecting another row in the grid swaps this editor out (it's keyed by // row) without a blur: the focused textarea is simply removed. So a draft // still pending then is committed on the way out, to the row it was made - // on — `onChangeCells` is bound to that row. + // on — `onChangeRow` is bound to that row. const commitPending = () => { - if (dirty) commit(state); + if (dirty) commit(stateRef.current); + }; + const commitOnEnter = (e: KeyboardEvent) => { + // An editor that used Enter itself — to pick a suggestion, say — has + // already claimed it. + if ( + e.key !== "Enter" || + e.shiftKey || + e.defaultPrevented || + e.nativeEvent.isComposing + ) { + return; + } + e.preventDefault(); + commitPending(); }; const commitRef = useRef(commitPending); commitRef.current = commitPending; useEffect(() => () => commitRef.current(), []); - const claimed = useMemo( - () => - computeClaims( - fields.map(f => ({ - path: fieldPath(f), - label: f.def.name, - selection: state.selections[f.id], - })), - state.resourceArgs, - resourcesByUri, + const { instances } = state; + const sectionSources = useMemo( + () => ({ + resources: withInstanceResources(catalog, instances), + functionSlots: {}, + executeSlots: {}, + resourceSlots: Object.fromEntries( + Object.entries(sources?.resourceSlots ?? {}).map(([uri, slots]) => [ + uri, + withInstanceSlots(slots, catalog, instances), + ]), ), - [fields, state, resourcesByUri], + }), + [catalog, instances, sources], ); - return fields.map(field => { - const input = row.cells[field.id]; - return ( - - {field.def.name} - - {shortSyntax(field.def.type.syntax, 32)} - - - } - > -
- {isEditable(field) ? ( -
) => { - if (!e.currentTarget.contains(e.relatedTarget as Node | null)) { - commitPending(); - } - }} - onKeyDown={(e: KeyboardEvent) => { - // An editor that used Enter itself — to pick a suggestion, - // say — has already claimed it. - if ( - e.key !== "Enter" || - e.shiftKey || - e.defaultPrevented || - e.nativeEvent.isComposing - ) { - return; - } - e.preventDefault(); - commitPending(); - }} - > - - update({ - ...state, - selections: { ...state.selections, [field.id]: selection }, - }) - } - resources={resources} - slots={sources?.fieldSlots[field.id] ?? {}} - argsContext={ - sources && { - resourceArgs: state.resourceArgs, - onResourceArgsChange: (uri, next) => - update({ - ...state, - resourceArgs: { ...state.resourceArgs, [uri]: next }, - }), - resourceSlots: sources.resourceSlots, - claimed, - path: fieldPath(field), - depth: 0, - } - } - /> -
- ) : input ? ( - - ) : ( - — - )} - {onChangeCells && input && ( - - )} -
- - ); + const changeInstances = ( + next: + | InstanceSelections + | ((latest: InstanceSelections) => InstanceSelections), + edit?: InstanceEdit, + ) => + update(latest => { + const value = typeof next === "function" ? next(latest.instances) : next; + if (!edit) return { ...latest, instances: value }; + const [from, to] = + edit.kind === "rename" ? [edit.from, edit.to] : [edit.name, null]; + return { + instances: value, + selections: retargetSelections(latest.selections, from, to), + }; + }); + + const context = instanceSourceContext({ + instances, + catalogByUri, + adopt: uri => { + const picked = adoptCatalogPick( + stateRef.current.instances, + uri, + catalogByUri, + ); + // Left a draft: the pick that follows binds the new instance, and + // saves both at once. + const latest = stateRef.current; + const next = { ...latest, instances: picked.instances }; + stateRef.current = next; + setState(next); + setDirty(true); + return picked.uri; + }, }); -} -/** A field's row identity, for "same instance as" links between fields. */ -function fieldPath(field: DatasetField): string { - return `field.${field.id}`; + // Every field is one `DetailRow`; the section sits above them as one more. + const editableRows = !!onChangeRow && !!sources; + return ( + <> + {editableRows ? ( + Resources} + > +
) => { + if (!e.currentTarget.contains(e.relatedTarget as Node | null)) { + commitPending(); + } + }} + onKeyDown={commitOnEnter} + > + + referencesTo(name, instances, [ + { + selections: Object.fromEntries( + fields.flatMap(f => + state.selections[f.id] + ? [[f.def.name, state.selections[f.id]!] as const] + : [], + ), + ), + }, + ]) + } + context={context} + /> +
+ + ) : ( + + )} + {fields.map(field => { + const input = row.cells[field.id]; + return ( + + {field.def.name} + + {shortSyntax(field.def.type.syntax, 32)} + + + } + > +
+ {isEditable(field) ? ( +
) => { + if ( + !e.currentTarget.contains(e.relatedTarget as Node | null) + ) { + commitPending(); + } + }} + onKeyDown={commitOnEnter} + > + + update(latest => ({ + ...latest, + selections: { + ...latest.selections, + [field.id]: selection, + }, + })) + } + resources={sectionSources.resources} + slots={withInstanceSlots( + sources?.fieldSlots[field.id], + catalog, + instances, + )} + context={sources && context} + /> +
+ ) : input ? ( + + ) : ( + — + )} + {onChangeRow && input && ( + + )} +
+ + ); + })} + + ); } /** * One dataset row in the details pane: when it was added and where from, * then every field — the same facts-over-blocks layout as a span's details. * - * With `onChangeCells`, every field is edited in place as the execute panel - * edits a parameter, and any cell can be cleared. Resources are offered from - * `sources` — the linked prompt's, since a dataset has none of its own. - * Without them, a resource or `object` cell stays read-only. See + * With `onChangeRow`, every field is edited in place as the execute panel + * edits a parameter, and any cell can be cleared. The row's resource + * instances are edited above the fields, from the catalog in `sources` — the + * linked prompt's, since a dataset has none of its own. Without it, the + * instances and any instance or `object` cell stay read-only. See * `specs/datasets.md` §P.2. */ export function DatasetRowDetails({ dataset, row, onOpenTrace, - onChangeCells, + onChangeRow, sources, }: { dataset: Dataset; row: DatasetRow; onOpenTrace: (providerId: string, traceId: string) => void; /** - * Sets fields' cells, by field id, or clears them with `null`. Read-only - * without. + * Sets fields' cells, by field id, or clears them with `null`, and sets or + * removes the row's resource instances by name. Read-only without. */ - onChangeCells?: (cells: Record) => void; + onChangeRow?: (update: Omit) => void; /** The resources the fields can be filled from, if any. */ sources?: DatasetInputSources; }) { @@ -357,7 +480,7 @@ export function DatasetRowDetails({ fields={dataset.fields} row={row} sources={sources} - onChangeCells={onChangeCells} + onChangeRow={onChangeRow} />
diff --git a/src/client/components/DatasetView.tsx b/src/client/components/DatasetView.tsx index b4dddd2..c5a2b70 100644 --- a/src/client/components/DatasetView.tsx +++ b/src/client/components/DatasetView.tsx @@ -43,7 +43,7 @@ import type { DatasetField, DatasetRow, DatasetRowsOverview, - ExecutionInput, + DatasetRowUpdate, NormalizedPrompt, PromptID, } from "../../shared/types"; @@ -596,13 +596,18 @@ function DatasetView({ if (!dataset || !linkedPrompt || !row) return; onOpenInPlayground( linkedPrompt, - panelFill(fromRow(dataset, row), linkedPrompt, { - type: "dataset", - description: `${dataset.name}, row ${index + 1}`, - providerId, - datasetId, - name: dataset.name, - }), + panelFill( + fromRow(dataset, row), + linkedPrompt, + { + type: "dataset", + description: `${dataset.name}, row ${index + 1}`, + providerId, + datasetId, + name: dataset.name, + }, + row.resources, + ), ); }; @@ -675,9 +680,8 @@ function DatasetView({ * reloads the visible rows from the server, dropping whatever optimistic * edits are on them — any still saving refetch again when they land. */ - const saveEdits = useCallback( - (edits: CellEdit[]) => { - const updates = groupEdits(edits); + const saveUpdates = useCallback( + (updates: DatasetRowUpdate[]) => { if (updates.length === 0) return; pager.patch(updates); setLoaded(n => n + 1); @@ -690,18 +694,9 @@ function DatasetView({ }, [pager, providerId, datasetId, overview.rowCount], ); - - /** Sets or clears cells of one row, by field id — the details pane's edits. */ - const saveCells = useCallback( - (rowId: string, cells: Record) => - saveEdits( - Object.entries(cells).map(([fieldId, cell]) => ({ - rowId, - fieldId, - cell, - })), - ), - [saveEdits], + const saveEdits = useCallback( + (edits: CellEdit[]) => saveUpdates(groupEdits(edits)), + [saveUpdates], ); /** A single edit, a paste over a range, or a fill — one batch either way. */ @@ -968,7 +963,9 @@ function DatasetView({ dataset={dataset} row={selectedRow} onOpenTrace={onOpenTrace} - onChangeCells={cells => saveCells(selectedRow.id, cells)} + onChangeRow={update => + saveUpdates([{ rowId: selectedRow.id, ...update }]) + } sources={rowSources} /> diff --git a/src/client/components/EvalView.tsx b/src/client/components/EvalView.tsx index c34808b..9a6a7f4 100644 --- a/src/client/components/EvalView.tsx +++ b/src/client/components/EvalView.tsx @@ -56,9 +56,7 @@ import { evalProblems, prefillBindings } from "./eval-bindings"; import { deleteRunQuestion, formatRate, passRate } from "./eval-summary"; import { fromExecutionInput, - type ResourceArgs, readStoredInputs, - resourceArgsFor, type Selections, type SlotSelection, toExecutionInput, @@ -69,12 +67,22 @@ import { columnUri, describePseudoSource, NEW_COLUMN_URI, + withInstanceSources, withPseudoSources, } from "./pseudo-sources"; import { - computeClaims, - type ResourceArgsContext, -} from "./resource-args-context"; + type InstanceEdit, + instanceSourceContext, + ResourcesSection, +} from "./ResourcesSection"; +import { + adoptCatalogPick, + fromWireResources, + type InstanceSelections, + referencesTo, + retargetSelections, + toWireResources, +} from "./run-resources-state"; import { formatTimestampCompact } from "./trace/format.ts"; import { PromptLinkIcon, @@ -124,32 +132,44 @@ interface EditorState { exec: Selections; /** Check id → its parameters' selections. */ checks: Record; - /** Shared by every slot and check parameter, as in the panel. */ - args: ResourceArgs; + /** + * The eval's own resource instances, which every row gets — a row's own + * instance of the same name wins (`specs/resource-instances.md` §E). + */ + instances: InstanceSelections; } /** Editor state recovered from saved bindings. */ function editorStateOf( def: Pick, - resourcesByUri: ReadonlyMap, ): EditorState { - const args: ResourceArgs = {}; const restore = (inputs: Record) => Object.fromEntries( - Object.entries(inputs).map(([name, input]) => { - const recovered = fromExecutionInput(input, resourcesByUri); - Object.assign(args, recovered.resourceArgs); - return [name, recovered.selection]; - }), + Object.entries(inputs).map(([name, input]) => [ + name, + fromExecutionInput(input), + ]), ); return { fn: restore(def.inputs.functionInputs), exec: restore(def.inputs.executeInputs), checks: Object.fromEntries(def.checks.map(c => [c.id, restore(c.args)])), - args, + instances: fromWireResources(def.inputs.resources), }; } +/** `selections` as inputs by name, empty ones left out. */ +function foldSelections( + selections: Selections, +): Record { + return Object.fromEntries( + Object.entries(selections).flatMap(([k, s]) => { + const input = toExecutionInput(s); + return input ? [[k, input] as const] : []; + }), + ); +} + /** A new check's id: short, and unique within the eval. */ function mintCheckId(existing: readonly EvalCheck[]): string { let n = existing.length + 1; @@ -194,7 +214,7 @@ function EvalView({ fn: {}, exec: {}, checks: {}, - args: {}, + instances: {}, }); /** Bindings made for the user and not yet touched — see `Prefilled.matched`. */ const [matched, setMatched] = useState>(new Set()); @@ -232,7 +252,9 @@ function EvalView({ if (loadedId.current !== evalId) { loadedId.current = evalId; setName(loaded.name); - setState(editorStateOf(loaded, resourcesByUri)); + const loadedState = editorStateOf(loaded); + stateRef.current = loadedState; + setState(loadedState); } }) .catch(err => !cancelled && setLoadError(err.message)); @@ -306,35 +328,19 @@ function EvalView({ /** The bindings `next` folds to. */ const foldInputs = (next: EditorState): EvalInputs => { - const resolve = (uri: string) => - resourceArgsFor(uri, next.args, resourcesByUri); - const fold = (sel: Selections) => - Object.fromEntries( - Object.entries(sel).flatMap(([k, s]) => { - const input = toExecutionInput(s, resolve); - return input ? [[k, input] as const] : []; - }), - ); - return { functionInputs: fold(next.fn), executeInputs: fold(next.exec) }; - }; - const foldChecks = (next: EditorState, checks: EvalCheck[]): EvalCheck[] => { - const resolve = (uri: string) => - resourceArgsFor(uri, next.args, resourcesByUri); - return checks.map(c => ({ - ...c, - args: Object.fromEntries( - Object.entries(next.checks[c.id] ?? {}).flatMap(([k, s]) => { - const input = toExecutionInput(s, resolve); - return input ? [[k, input] as const] : []; - }), - ), - })); + const resources = toWireResources(next.instances); + return { + functionInputs: foldSelections(next.fn), + executeInputs: foldSelections(next.exec), + ...(resources && { resources }), + }; }; + const foldChecks = (next: EditorState, checks: EvalCheck[]): EvalCheck[] => + checks.map(c => ({ ...c, args: foldSelections(next.checks[c.id] ?? {}) })); // The latest state and definition, for a change that lands after an // `await` (a new column), when the render's own `state` may be stale. const stateRef = useRef(state); - stateRef.current = state; const defRef = useRef(def); defRef.current = def; @@ -363,11 +369,12 @@ function EvalView({ stored: readStoredInputs(prompt), }); if (proposed.matched.length === 0) return; - const next = editorStateOf( - { inputs: proposed.inputs, checks: proposed.checks }, - resourcesByUri, - ); + const next = editorStateOf({ + inputs: proposed.inputs, + checks: proposed.checks, + }); setMatched(prev => new Set([...prev, ...proposed.matched])); + stateRef.current = next; setState(next); save({ inputs: proposed.inputs, checks: proposed.checks }); }, [def?.id, prompt?.id, fields, checkInfos, checkIdsKey]); @@ -396,14 +403,17 @@ function EvalView({ // Stable while only values change, so typing doesn't remount editors. const sources = useStructurallyStable( - withPseudoSources(prompt?.inputSources, { - functionParameters, - executeParameters, - bindings: inputs, - fields: fields ?? [], - offerMismatches: true, - newColumn: true, - }), + withPseudoSources( + withInstanceSources(prompt?.inputSources, state.instances), + { + functionParameters, + executeParameters, + bindings: inputs, + fields: fields ?? [], + offerMismatches: true, + newColumn: true, + }, + ), ); const pseudoOptions = { functionParameters, executeParameters, fields }; const checkSources = useStructurallyStable( @@ -418,31 +428,42 @@ function EvalView({ ), ); - const claimed = computeClaims( - [ - ...functionParameters.map(p => ({ - path: `fn.${p.name}`, - label: p.name, - selection: state.fn[p.name], - })), - ...executeParameters.map(p => ({ - path: `exec.${p.name}`, - label: p.name, - selection: state.exec[p.name], - })), - ], - state.args, - resourcesByUri, - ); - const argsContextFor = (path: string): ResourceArgsContext => ({ - resourceArgs: state.args, - onResourceArgsChange: (uri, next) => - commit({ ...state, args: { ...state.args, [uri]: next } }), - resourceSlots: sources.resourceSlots ?? {}, - claimed, - path, - depth: 0, - describePseudo: (uri, type) => + const changeInstances = ( + next: + | InstanceSelections + | ((latest: InstanceSelections) => InstanceSelections), + edit?: InstanceEdit, + ) => { + const prev = stateRef.current; + const instances = typeof next === "function" ? next(prev.instances) : next; + if (!edit) return commit({ ...prev, instances }); + const [from, to] = + edit.kind === "rename" ? [edit.from, edit.to] : [edit.name, null]; + commit({ + fn: retargetSelections(prev.fn, from, to), + exec: retargetSelections(prev.exec, from, to), + checks: Object.fromEntries( + Object.entries(prev.checks).map(([id, sel]) => [ + id, + retargetSelections(sel, from, to), + ]), + ), + instances, + }); + }; + const context = instanceSourceContext({ + instances: state.instances, + catalogByUri: resourcesByUri, + adopt: uri => { + const picked = adoptCatalogPick( + stateRef.current.instances, + uri, + resourcesByUri, + ); + changeInstances(picked.instances); + return picked.uri; + }, + describeOther: (uri, type) => describePseudoSource(uri, type, pseudoOptions), }); @@ -498,7 +519,8 @@ function EvalView({ }); }; if (relPath === undefined) { - commit({ ...state, [which]: { ...state[which], [name]: selection } }); + const prev = stateRef.current; + commit({ ...prev, [which]: { ...prev[which], [name]: selection } }); return; } void addColumn( @@ -562,6 +584,7 @@ function EvalView({ if (!def) return; const { [id]: _, ...rest } = state.checks; const nextState = { ...state, checks: rest }; + stateRef.current = nextState; setState(nextState); save( { @@ -622,7 +645,6 @@ function EvalView({ onChange, sources: from, slots, - path, }: { key: string; label: string; @@ -631,7 +653,6 @@ function EvalView({ onChange: (selection: SlotSelection) => void; sources: PromptInputSources; slots: PromptInputSources["functionSlots"]; - path: string; }, ) => (
@@ -658,7 +679,7 @@ function EvalView({ onChange={onChange} resources={from.resources} slots={slots} - argsContext={argsContextFor(path)} + context={context} />
); @@ -673,7 +694,6 @@ function EvalView({ onChange: selection => changeSlot(which)(param.name, selection), sources, slots: which === "fn" ? sources.functionSlots : sources.executeSlots, - path: `${which}.${param.name}`, }), ); @@ -872,7 +892,6 @@ function EvalView({ changeCheckParam(check)(param.name, selection), sources: checkSources[check.id] ?? EMPTY_SOURCES, slots: checkSources[check.id]?.functionSlots ?? {}, - path: `check.${check.id}.${param.name}`, }), )} @@ -935,6 +954,25 @@ function EvalView({ > {prompt ? ( <> + {((prompt.inputSources?.resources.length ?? 0) > 0 || + Object.keys(state.instances).length > 0) && ( + + referencesTo(instance, state.instances, [ + { selections: state.fn }, + { selections: state.exec }, + ...Object.values(state.checks).map(selections => ({ + selections, + })), + ]) + } + context={context} + /> + )} {renderSlots(functionParameters, "fn")} {executeParameters.length > 0 && ( <> diff --git a/src/client/components/ExecutionInputEditor.tsx b/src/client/components/ExecutionInputEditor.tsx index c85b5a8..363f3ca 100644 --- a/src/client/components/ExecutionInputEditor.tsx +++ b/src/client/components/ExecutionInputEditor.tsx @@ -12,8 +12,9 @@ import type { ResourceInfo, } from "../../shared/types"; import { SELF, type SlotSelection } from "./execution-input-state"; -import { nestedField, type ResourceArgsContext } from "./resource-args-context"; +import { isPseudoUri } from "./pseudo-sources"; import { SourceRow } from "./SourceRow"; +import type { SourceContext } from "./source-context"; interface Props { /** The slot being edited. */ @@ -29,14 +30,8 @@ interface Props { * name, as computed provider-side. */ slots: Record; - /** - * Argument-editor state and how to change it, threaded down so a chosen - * resource with declared `parameters` can render its own argument form - * beneath the chip — see `SourceRow` and `specs/resource-arguments.md` §I. - * Absent (e.g. in a host that doesn't wire it up) simply means no resource - * here ever shows an argument form, whether or not it has parameters. - */ - argsContext?: ResourceArgsContext; + /** How chips read, and how a catalog pick becomes an instance — see {@link SourceContext}. */ + context?: SourceContext; } /** @@ -60,7 +55,7 @@ export function ExecutionInputEditor({ onChange, resources, slots, - argsContext, + context, }: Props) { const byUri = useMemo( () => new Map(resources.map(r => [r.uri, r])), @@ -73,20 +68,23 @@ export function ExecutionInputEditor({ // dropping focus after every character. `selection` changes on every // keystroke (it carries the typed value), so neither `chooseResource` nor // the plugin below may depend on it directly; a ref keeps both reading the - // latest `selection` anyway, just not as a dependency. `argsContext` is the - // same story and then some — its `resourceArgs` changes on every keystroke - // of *any* argument form anywhere in the panel, not just this slot's. + // latest `selection` anyway, just not as a dependency. `context` is the + // same story: hosts build it inline, so it's new on every render. const selectionRef = useRef(selection); selectionRef.current = selection; const onChangeRef = useRef(onChange); onChangeRef.current = onChange; - const argsContextRef = useRef(argsContext); - argsContextRef.current = argsContext; + const contextRef = useRef(context); + contextRef.current = context; const chooseResource = useCallback((path: string, uri: string | null) => { const current = selectionRef.current; const next = { ...(current.resources ?? {}) }; - if (uri) next[path] = uri; + // A pick from the catalog adds an instance to the run; the slot holds a + // reference to it, like any other pick from "Resources in this run". + const adopt = contextRef.current?.adopt; + const chosen = uri && !isPseudoUri(uri) && adopt ? adopt(uri) : uri; + if (chosen) next[path] = chosen; else delete next[path]; onChangeRef.current({ ...current, resources: next }); }, []); @@ -112,16 +110,6 @@ export function ExecutionInputEditor({ match: matches, component: (props: ItemEditorProps) => { const path = relativePath(props.path); - const current = argsContextRef.current; - // `path` is this field's own position relative to the top-level - // slot (e.g. "list_tasks.db") — folded into the row's identity so - // it can't collide with an unrelated row that happens to share a - // name (see `ResourceArgsContext.path`). Empty only if the plugin - // ever matched the slot's own root item rather than a nested - // field, which isn't a case ts-proppy produces today; guarded - // rather than assumed. - const nestedContext = - current && path ? nestedField(current, path) : current; // A matched node that is itself an object (not opaque — an opaque // leaf never reaches this recursive call at all, since `SourceRow` // only renders `children` for an editable, i.e. non-opaque, type) @@ -143,7 +131,7 @@ export function ExecutionInputEditor({ matching={matchingFor(slots, propDef.name, props.path, byUri)} chosen={selectionRef.current.resources?.[path]} onChoose={uri => chooseResource(path, uri)} - argsContext={nestedContext} + context={contextRef.current} > chooseResource(SELF, uri)} - argsContext={argsContext} + context={context} > >; }; -/** - * A top-level slot's own unique position — the root every nested - * {@link ResourceArgsContext.path} within it is built from. Namespaced by - * `which` so a function parameter and an execute parameter of the same name - * (a real case — `PromptInputSources.functionSlots` / - * `executeSlots` are already a parallel split for exactly this reason) don't - * collide. - */ -function slotPath(which: "fn" | "exec", name: string): string { - return `${which}.${name}`; -} - /** A stable stand-in for a prompt with no execute parameters, so memos keyed on them hold. */ const NO_PARAMETERS: PropDefinition[] = []; +/** The same, for a prompt with no resources in scope. */ +const NO_RESOURCES: ResourceInfo[] = []; + +/** `selections` as inputs by slot name, empty slots left out. */ +function namedInputs( + defs: readonly PropDefinition[], + selections: Selections, +): Record { + return Object.fromEntries( + defs.flatMap(d => { + const input = toExecutionInput(selections[d.name]); + return input ? [[d.name, input] as const] : []; + }), + ); +} + +/** Editor state for a set of stored or recorded inputs, by slot name. */ +function restoreSelections( + inputs: Record = {}, +): Selections { + return Object.fromEntries( + Object.entries(inputs).map(([name, input]) => [ + name, + fromExecutionInput(input), + ]), + ); +} /** * Restore the panel from what was saved last time. @@ -130,50 +156,34 @@ const NO_PARAMETERS: PropDefinition[] = []; * An {@link ExecutionInput} is what gets stored, rather than a bare value: a * resource choice has no value to save, and storing the recipe is what lets a * template come back as a template rather than as the string it flattened to. - * - * Arguments ride inside those same stored inputs (a resource node's `args`) - * rather than in a second key — `resourceArgs` here is *derived*, by pulling - * every `args` a stored reference carries back out into - * `specs/resource-arguments.md` §J's shared-by-uri shape, from wherever in - * the tree it turns up. + * The run's resource instances are stored beside the slots, as the request + * sends them. */ function loadStored(prompt: NormalizedPrompt): { fn: Selections; exec: Selections; layout: LayoutChoices; - resourceArgs: ResourceArgs; + instances: InstanceSelections; } { const empty = { fn: {}, exec: {}, layout: { fn: {}, exec: {} }, - resourceArgs: {}, + instances: {}, }; try { const raw = localStorage.getItem(paramStorageKey(prompt)); if (!raw) return empty; const parsed = JSON.parse(raw) as StoredInputs; if (!parsed || typeof parsed !== "object") return empty; - const resourcesByUri = new Map( - (prompt.inputSources?.resources ?? []).map(r => [r.uri, r]), - ); - const resourceArgs: ResourceArgs = {}; - const restore = (inputs: Record = {}) => - Object.fromEntries( - Object.entries(inputs).map(([name, input]) => { - const recovered = fromExecutionInput(input, resourcesByUri); - Object.assign(resourceArgs, recovered.resourceArgs); - return [name, recovered.selection]; - }), - ); return { - fn: restore(parsed.functionInputs), - exec: restore(parsed.executeInputs), + fn: restoreSelections(parsed.functionInputs), + exec: restoreSelections(parsed.executeInputs), layout: { fn: parsed.layout?.functionSlots ?? {}, exec: parsed.layout?.executeSlots ?? {}, }, - resourceArgs, + instances: fromWireResources(parsed.resources), }; } catch { /* ignore (private browsing, quota, corrupt entry, etc.) */ @@ -212,9 +222,6 @@ function PlaygroundExecution({ const [layoutChoices, setLayoutChoices] = useState( stored.layout, ); - const [resourceArgs, setResourceArgs] = useState( - stored.resourceArgs, - ); const [overwrittenNotes, setOverwrittenNotes] = useState({ fn: {}, exec: {}, @@ -226,108 +233,64 @@ function PlaygroundExecution({ const executeParameters = prompt.executeParameters ?? NO_PARAMETERS; const broken = brokenResources(prompt.inputSources); + const catalog = prompt.inputSources?.resources ?? NO_RESOURCES; const resourcesByUri = useMemo( - () => - new Map( - (prompt.inputSources?.resources ?? []).map(r => [r.uri, r]), - ), - [prompt.inputSources], + () => new Map(catalog.map(r => [r.uri, r])), + [catalog], + ); + const { instances, setInstances, adopt } = useRunInstances( + stored.instances, + resourcesByUri, ); // The other slots, as sources for `input` references (`specs/evals.md` // §B.2.1): offered where their type fits and no cycle would close. const bindings = useMemo(() => { - const resolve = (uri: string) => - resourceArgsFor(uri, resourceArgs, resourcesByUri); - const named = (defs: readonly PropDefinition[], sel: Selections) => - Object.fromEntries( - defs.flatMap(d => { - const input = toExecutionInput(sel[d.name], resolve); - return input ? [[d.name, input] as const] : []; - }), - ); + const resources = toWireResources(instances); return { - functionInputs: named(prompt.functionParameters, functionSelections), - executeInputs: named(executeParameters, executeSelections), + functionInputs: namedInputs( + prompt.functionParameters, + functionSelections, + ), + executeInputs: namedInputs(executeParameters, executeSelections), + ...(resources && { resources }), }; }, [ prompt.functionParameters, executeParameters, functionSelections, executeSelections, - resourceArgs, - resourcesByUri, + instances, ]); // Stable while only values change, so typing doesn't remount editors. const sources = useStructurallyStable( - withPseudoSources(prompt.inputSources, { + withPseudoSources(withInstanceSources(prompt.inputSources, instances), { functionParameters: prompt.functionParameters, executeParameters, bindings, }), ); - // Panel order — the first slot to reach a given resource `uri` owns its - // argument form; every later selection of it just points back (§I). - // Recomputed purely from the current selections on every render (rather - // than mutated as rows render) so it behaves identically under React - // StrictMode's double-invoked renders. - const claimed = useMemo( - () => - computeClaims( - [ - ...prompt.functionParameters.map(p => ({ - path: slotPath("fn", p.name), - label: p.name, - selection: functionSelections[p.name], - })), - ...executeParameters.map(p => ({ - path: slotPath("exec", p.name), - label: p.name, - selection: executeSelections[p.name], - })), - ], - resourceArgs, - resourcesByUri, - ), - [ - prompt.functionParameters, - executeParameters, - functionSelections, - executeSelections, - resourceArgs, - resourcesByUri, - ], - ); - - const persist = ( - fn: Selections, - exec: Selections, - layout: LayoutChoices, - args: ResourceArgs, - ) => { + // Every edit lands in storage, whichever piece of state it changed. + // biome-ignore lint/correctness/useExhaustiveDependencies: `bindings` is derived from exactly the state that matters. + useEffect(() => { try { - const resolveArgs = (uri: string) => - resourceArgsFor(uri, args, resourcesByUri); - const collect = (defs: readonly PropDefinition[], sel: Selections) => - Object.fromEntries( - defs.flatMap(d => { - const input = toExecutionInput(sel[d.name], resolveArgs); - return input ? [[d.name, input] as const] : []; - }), - ); localStorage.setItem( paramStorageKey(prompt), JSON.stringify({ - functionInputs: collect(prompt.functionParameters, fn), - executeInputs: collect(executeParameters, exec), - layout: { functionSlots: layout.fn, executeSlots: layout.exec }, + functionInputs: bindings.functionInputs, + executeInputs: bindings.executeInputs, + ...(bindings.resources && { resources: bindings.resources }), + layout: { + functionSlots: layoutChoices.fn, + executeSlots: layoutChoices.exec, + }, } satisfies StoredInputs), ); } catch { /* ignore (private browsing, quota, etc.) */ } - }; + }, [bindings, layoutChoices]); const change = (which: "fn" | "exec") => (name: string, selection: SlotSelection) => { @@ -341,34 +304,30 @@ function PlaygroundExecution({ : executeSelections; setFunctionSelections(fn); setExecuteSelections(exec); - persist(fn, exec, layoutChoices, resourceArgs); }; - const onResourceArgsChange = (uri: string, next: Selections) => { - const args = { ...resourceArgs, [uri]: next }; - setResourceArgs(args); - persist(functionSelections, executeSelections, layoutChoices, args); + const changeInstances = ( + next: Parameters[0], + edit?: InstanceEdit, + ) => { + setInstances(next); + if (!edit) return; + const [from, to] = + edit.kind === "rename" ? [edit.from, edit.to] : [edit.name, null]; + setFunctionSelections(s => retargetSelections(s, from, to)); + setExecuteSelections(s => retargetSelections(s, from, to)); }; - // Not itself a valid row's context — `path` is meaningless at this level — - // but every field *other* than `path` is shared, so each slot spreads this - // and sets its own `path` (see `renderSlots`) rather than repeating the rest. - const argsContextBase: Omit = { - resourceArgs, - onResourceArgsChange, - resourceSlots: sources.resourceSlots ?? {}, - claimed, - depth: 0, - describePseudo: (uri, type) => + const context = instanceSourceContext({ + instances, + catalogByUri: resourcesByUri, + adopt, + describeOther: (uri, type) => describePseudoSource(uri, type, { functionParameters: prompt.functionParameters, executeParameters, }), - }; - - /** Builds the `args` a chosen resource should carry, from the current {@link resourceArgs} state. */ - const resolveArgs = (uri: string) => - resourceArgsFor(uri, resourceArgs, resourcesByUri); + }); /** * A slot's effective layout (§B precedence): the user's own stored choice, @@ -388,7 +347,7 @@ function PlaygroundExecution({ )?.[def.name]; return resolveLayout( groups, - toExecutionInput(selections[def.name], resolveArgs), + toExecutionInput(selections[def.name]), layoutChoices[which][def.name], hint, ); @@ -410,13 +369,9 @@ function PlaygroundExecution({ if (next === "combined") { const selections = which === "fn" ? fn : exec; - const storedInput = toExecutionInput(selections[def.name], resolveArgs); + const storedInput = toExecutionInput(selections[def.name]); const { selection, overwritten } = collapseLossy(groups, storedInput); - // Any `args` the round trip carries came from `resourceArgs` itself (via - // `resolveArgs`), so only the plain selection is needed back out. - const merged = fromExecutionInput( - expand(groups, selection, resolveArgs), - ).selection; + const merged = fromExecutionInput(expand(groups, selection)); if (which === "fn") fn = { ...fn, [def.name]: merged }; else exec = { ...exec, [def.name]: merged }; setFunctionSelections(fn); @@ -428,9 +383,7 @@ function PlaygroundExecution({ setOverwrittenNotes(prev => withEntry(prev, which, def.name, undefined)); } - const layout = withEntry(layoutChoices, which, def.name, next); - setLayoutChoices(layout); - persist(fn, exec, layout, resourceArgs); + setLayoutChoices(withEntry(layoutChoices, which, def.name, next)); }; /** @@ -457,10 +410,7 @@ function PlaygroundExecution({ }): PartialExecuteRequest | null => { const functionInputs: Record = {}; for (const param of prompt.functionParameters) { - const input = toExecutionInput( - functionSelections[param.name], - resolveArgs, - ); + const input = toExecutionInput(functionSelections[param.name]); if (input) functionInputs[param.name] = input; else if ( requireAll && @@ -474,10 +424,7 @@ function PlaygroundExecution({ const executeInputs: Record = {}; for (const param of executeParameters) { - const input = toExecutionInput( - executeSelections[param.name], - resolveArgs, - ); + const input = toExecutionInput(executeSelections[param.name]); if (input) executeInputs[param.name] = input; else if (requireAll && !param.optional) { setError(missingInputMessage(param, !!prompt.inputSources)); @@ -485,7 +432,11 @@ function PlaygroundExecution({ } } - return { functionInputs, executeInputs }; + return { + functionInputs, + executeInputs, + ...(bindings.resources && { resources: bindings.resources }), + }; }; /** The request to run: every slot, positionally, defaults filled in. */ @@ -499,7 +450,11 @@ function PlaygroundExecution({ ? { kind: "value", value: param.defaultValue } : { kind: "value", value: { kind: "primitive", value: undefined } }), ); - return { functionInputs, executeInputs: collected.executeInputs }; + return { + functionInputs, + executeInputs: collected.executeInputs, + ...(collected.resources && { resources: collected.resources }), + }; }; // Applies a fill once: matched slots overwritten, every other slot left as @@ -509,39 +464,37 @@ function PlaygroundExecution({ if (!fill || appliedFills.has(fill.nonce)) return; appliedFills.add(fill.nonce); - const recoveredArgs: ResourceArgs = {}; - const restore = (inputs: Record) => - Object.fromEntries( - Object.entries(inputs).map(([name, input]) => { - const recovered = fromExecutionInput(input, resourcesByUri); - Object.assign(recoveredArgs, recovered.resourceArgs); - return [name, recovered.selection]; - }), - ); const kept = [ ...prompt.functionParameters .filter( p => !(p.name in fill.functionInputs) && - toExecutionInput(functionSelections[p.name], resolveArgs), + toExecutionInput(functionSelections[p.name]), ) .map(p => p.name), ...executeParameters .filter( p => !(p.name in fill.executeInputs) && - toExecutionInput(executeSelections[p.name], resolveArgs), + toExecutionInput(executeSelections[p.name]), ) .map(p => p.name), ]; - const fn = { ...functionSelections, ...restore(fill.functionInputs) }; - const exec = { ...executeSelections, ...restore(fill.executeInputs) }; - const args = { ...resourceArgs, ...recoveredArgs }; - setFunctionSelections(fn); - setExecuteSelections(exec); - setResourceArgs(args); - persist(fn, exec, layoutChoices, args); + setFunctionSelections({ + ...functionSelections, + ...restoreSelections(fill.functionInputs), + }); + setExecuteSelections({ + ...executeSelections, + ...restoreSelections(fill.executeInputs), + }); + // The fill's instances replace any of the same name; the rest stay, as + // unfilled slots do. + if (fill.resources) { + const filled = fromWireResources(fill.resources); + setInstances(latest => ({ ...latest, ...filled })); + } setFillNotice({ from: fill.from, kept, skipped: fill.skipped }); }, [fill?.nonce]); @@ -579,10 +532,6 @@ function PlaygroundExecution({ const groups = fanOutGroups(def); const combinable = isCombinable(groups); const layout = combinable ? layoutFor(which, def, groups) : "expanded"; - const slotArgsContext: ResourceArgsContext = { - ...argsContextBase, - path: slotPath(which, def.name), - }; return (
@@ -630,24 +579,19 @@ function PlaygroundExecution({ propDef={def} groups={groups} selection={ - collapseLossy( - groups, - toExecutionInput(selections[def.name], resolveArgs), - ).selection + collapseLossy(groups, toExecutionInput(selections[def.name])) + .selection } onChange={combined => - // As in `toggleLayout`: `args` in the round trip are already - // in `resourceArgs`, edited directly by each row's own form. change(which)( def.name, - fromExecutionInput(expand(groups, combined, resolveArgs)) - .selection, + fromExecutionInput(expand(groups, combined)), ) } resources={sources?.resources ?? []} slots={slots} overwritten={overwrittenNotes[which][def.name]} - argsContext={slotArgsContext} + context={context} /> ) : ( change(which)(def.name, selection)} resources={sources?.resources ?? []} slots={slots} - argsContext={slotArgsContext} + context={context} /> )}
@@ -670,6 +614,7 @@ function PlaygroundExecution({ prompt.providerId && ( )} + {(catalog.length > 0 || Object.keys(instances).length > 0) && ( + + referencesTo(name, instances, [ + { selections: functionSelections }, + { selections: executeSelections }, + ]) + } + context={context} + /> + )} {renderSlots( prompt.functionParameters, functionSelections, diff --git a/src/client/components/ResourcesSection.tsx b/src/client/components/ResourcesSection.tsx new file mode 100644 index 0000000..259d54d --- /dev/null +++ b/src/client/components/ResourcesSection.tsx @@ -0,0 +1,384 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { useMemo, useState } from "react"; +import type { PromptInputSources, ResourceInfo } from "../../shared/types"; +import { ExecutionInputEditor } from "./ExecutionInputEditor"; +import { describeInstanceSource, parseInstanceUri } from "./pseudo-sources"; +import { + addInstance, + derivedDependencies, + duplicateInstance, + excludedFromArgs, + type InstanceSelections, + instanceNameProblem, + removeInstance, + renameInstance, +} from "./run-resources-state"; +import SourcePicker from "./SourcePicker"; +import type { SourceContext } from "./source-context"; + +/** + * A change to the run's instances that the host has to carry into its own + * slots too: references to a renamed instance follow it, and references to a + * removed one are cleared (see `retargetSelections`). + */ +export type InstanceEdit = + | { kind: "rename"; from: string; to: string } + | { kind: "remove"; name: string }; + +/** How long a card stays flagged `.pg-row-highlight` after {@link scrollToInstance} lands on it. */ +const HIGHLIGHT_MS = 1200; + +/** + * Scrolls instance `name`'s card into view and briefly flashes it — what a + * chip naming the instance does when clicked. + */ +export function scrollToInstance(name: string) { + const el = document.querySelector( + `[data-instance-card="${CSS.escape(name)}"]`, + ); + if (!el) return; + el.scrollIntoView({ behavior: "smooth", block: "center" }); + // Removing the class and forcing a reflow restarts the animation, so a + // repeat click flashes again. + el.classList.remove("pg-row-highlight"); + void el.offsetWidth; + el.classList.add("pg-row-highlight"); + window.setTimeout( + () => el.classList.remove("pg-row-highlight"), + HIGHLIGHT_MS, + ); +} + +interface Props { + /** The run's instances, by name, in display order. */ + instances: InstanceSelections; + /** + * Called with the next instances (or how to get them from the latest — + * an argument pick from the catalog adds an instance through + * `context.adopt` just before), and with the edit when slots must follow + * it. + */ + onChange: ( + next: + | InstanceSelections + | ((latest: InstanceSelections) => InstanceSelections), + edit?: InstanceEdit, + ) => void; + /** + * Every source the host's slots are offered — instances, columns and + * prompt slots included — whose `resourceSlots` say what fits each + * argument. + */ + sources: PromptInputSources; + /** The resources an instance can be created from: the prompt's own `inputSources.resources`. */ + catalog: readonly ResourceInfo[]; + /** What references instance `name` (`taskId`, `child1.parentId`) — see `referencesTo`. */ + referencedBy: (name: string) => string[]; + /** Forwarded to each argument's editor. */ + context?: SourceContext; +} + +/** + * The run's named resource instances, one card each, above the slots: its + * name, its resource, what references it, and its argument form. Then the + * dependencies the run doesn't declare, which are created with no arguments + * and shared. Controlled: the host persists `instances` however it persists + * the rest of its inputs. See `specs/resource-instances.md` §G. + */ +export function ResourcesSection({ + instances, + onChange, + sources, + catalog, + referencedBy, + context, +}: Props) { + const catalogByUri = useMemo( + () => new Map(catalog.map(r => [r.uri, r])), + [catalog], + ); + const named = Object.entries(instances); + const dependencies = derivedDependencies(instances, catalogByUri); + + // Only resources themselves, not their outputs; a server-scoped one that + // already has an instance can't have a second. + const addable = catalog.filter( + r => + !r.parent && + !r.error && + !( + (r.scope === "server" || r.value !== undefined) && + named.some(([, i]) => i.uri === r.uri) + ), + ); + + const add = (uri: string | null) => { + if (!uri) return; + onChange(addInstance(instances, uri).instances); + }; + + return ( +
+
+ Resources + {addable.length > 0 && ( + + )} +
+ {named.map(([name, instance]) => ( + + ))} + {dependencies.map(dep => { + const info = catalogByUri.get(dep.uri); + return ( +
+
+ + ◆ {info?.label ?? dep.uri} + + + {scopeLabel(info)} · dependency + + {!dep.ambiguous && ( + + )} +
+
+ {dep.ambiguous + ? `⚠ ${dep.usedBy.join(", ")} can't tell which to use: ${dep.ambiguous.join(", ")}` + : `used by ${dep.usedBy.join(", ")}`} +
+
+ ); + })} +
+ ); +} + +/** "shared" for a resource created once per server, "per run" otherwise. */ +function scopeLabel(info: ResourceInfo | undefined): string { + return info?.scope === "server" ? "shared" : "per run"; +} + +function InstanceCard({ + name, + instances, + resource, + onChange, + sources, + referencedBy, + context, +}: { + name: string; + instances: InstanceSelections; + resource: ResourceInfo | undefined; + onChange: Props["onChange"]; + sources: PromptInputSources; + referencedBy: string[]; + context?: SourceContext; +}) { + const instance = instances[name]; + const [draftName, setDraftName] = useState(null); + const problem = + draftName === null + ? undefined + : instanceNameProblem(draftName, instances, name); + + const commitRename = () => { + if (draftName === null) return; + if (!problem && draftName !== name) { + onChange(renameInstance(instances, name, draftName), { + kind: "rename", + from: name, + to: draftName, + }); + } + setDraftName(null); + }; + + const remove = () => { + if ( + referencedBy.length > 0 && + !window.confirm( + `Remove '${name}'? ${referencedBy.join(", ")} will be cleared.`, + ) + ) { + return; + } + onChange(removeInstance(instances, name), { kind: "remove", name }); + }; + + // An argument may name any other instance whose creation doesn't need + // this one first. + const excluded = excludedFromArgs(instances, name); + const argSlots = Object.fromEntries( + Object.entries(sources.resourceSlots?.[instance.uri] ?? {}).map( + ([path, uris]) => [ + path, + uris.filter(uri => { + const ref = parseInstanceUri(uri); + return !ref || !excluded.has(ref.name); + }), + ], + ), + ); + const params = resource?.parameters ?? []; + + return ( +
+
+ {draftName === null ? ( + + ) : ( + setDraftName(e.target.value)} + onBlur={commitRename} + onKeyDown={e => { + if (e.key === "Enter") commitRename(); + if (e.key === "Escape") setDraftName(null); + }} + /> + )} + + {resource ? resource.label : "Resource no longer available"} + {resource && ` · ${scopeLabel(resource)}`} + + + {resource?.scope !== "server" && ( + + )} + + +
+ {problem &&
{problem}
} +
+ {referencedBy.length > 0 + ? `→ ${referencedBy.join(", ")}` + : "not bound to a slot: runs for its side effects"} +
+ {params.length > 0 && ( +
+ {params.map(param => ( +
+ {param.name} + + onChange(latest => { + const current = latest[name]; + if (!current) return latest; + return { + ...latest, + [name]: { + ...current, + args: { ...current.args, [param.name]: next }, + }, + }; + }) + } + resources={sources.resources} + slots={argSlots} + context={context} + /> +
+ ))} +
+ )} +
+ ); +} + +/** + * The {@link SourceContext} a host hands its slots and the section's own + * argument forms: instance chips name the instance and open its card, a pick + * from the catalog adds an instance, and every other pseudo-source is + * described by `describeOther` (columns, prompt slots). + */ +export function instanceSourceContext({ + instances, + catalogByUri, + adopt, + describeOther, +}: { + instances: InstanceSelections; + catalogByUri: ReadonlyMap; + adopt: (uri: string) => string; + describeOther?: SourceContext["describePseudo"]; +}): SourceContext { + return { + adopt, + describePseudo: (uri, slotType) => { + const instance = describeInstanceSource(uri, instances, catalogByUri); + if (!instance) return describeOther?.(uri, slotType); + const ref = parseInstanceUri(uri)!; + return { + ...instance, + ...(!instance.missing && { onOpen: () => scrollToInstance(ref.name) }), + }; + }, + }; +} diff --git a/src/client/components/SourcePicker.tsx b/src/client/components/SourcePicker.tsx index 0d192b6..2104281 100644 --- a/src/client/components/SourcePicker.tsx +++ b/src/client/components/SourcePicker.tsx @@ -37,6 +37,7 @@ export default function SourcePicker({ editable, label, onChoose, + triggerLabel, }: { /** Every source offered to the prompt, unfiltered — for the tree's group/value context. */ resources: readonly ResourceInfo[]; @@ -52,6 +53,12 @@ export default function SourcePicker({ label: string; /** Called with the chosen uri, or `null` for "Custom" / no resource. */ onChoose: (uri: string | null) => void; + /** + * Text for a labelled trigger button ("＋ Add resource") in place of the + * slot's `⋯` icon — a picker that adds something rather than filling a + * slot, so it has no "Custom" or placeholder row either. + */ + triggerLabel?: string; }) { const [open, setOpen] = useState(false); const [filter, setFilter] = useState(""); @@ -94,23 +101,31 @@ export default function SourcePicker({ ) : undefined; - const headerKind = editable ? "custom" : !chosen ? "placeholder" : undefined; + const headerKind = triggerLabel + ? undefined + : editable + ? "custom" + : !chosen + ? "placeholder" + : undefined; return (
{open && diff --git a/src/client/components/SourceRow.tsx b/src/client/components/SourceRow.tsx index b48f92b..de4a0cf 100644 --- a/src/client/components/SourceRow.tsx +++ b/src/client/components/SourceRow.tsx @@ -5,15 +5,9 @@ import { useMemo } from "react"; import { ItemEditor } from "ts-proppy/react"; import { jsonToPropValue } from "../../shared/json-prop-value"; import type { PropDefinition, ResourceInfo } from "../../shared/types"; -import { ExecutionInputEditor } from "./ExecutionInputEditor"; -import { rootResourceUri, type Selections } from "./execution-input-state"; import { isPseudoUri } from "./pseudo-sources"; -import { - MAX_RESOURCE_ARG_DEPTH, - nested, - type ResourceArgsContext, -} from "./resource-args-context"; import SourcePicker from "./SourcePicker"; +import type { SourceContext } from "./source-context"; import { combinedLabel } from "./source-tree"; /** @@ -36,46 +30,15 @@ function chipLabel( /** `ItemEditor` needs an `onChange` even for the read-only preview — `disabled` already keeps it from ever firing. */ function noop() {} -/** How long a row stays flagged `.pg-row-highlight` after `scrollToRow` lands on it. */ -const HIGHLIGHT_MS = 1200; - -/** - * Scrolls the row identified by `path` (the same string `SourceRow` tags its - * own `.pg-slot` with, via `data-row-path`) into view and briefly flashes it - * — what the chip's "same instance as ▲ X" link does, so picking the same - * resource twice reads as "here's the other one", not just a name. - */ -function scrollToRow(path: string) { - const el = document.querySelector( - `[data-row-path="${CSS.escape(path)}"]`, - ); - if (!el) return; - el.scrollIntoView({ behavior: "smooth", block: "center" }); - // Restarts the animation even if this row was just highlighted and hasn't - // finished fading — removing the class and forcing a reflow before adding - // it back is what makes a repeat click flash again instead of no-op-ing. - el.classList.remove("pg-row-highlight"); - void el.offsetWidth; - el.classList.add("pg-row-highlight"); - window.setTimeout( - () => el.classList.remove("pg-row-highlight"), - HIGHLIGHT_MS, - ); -} - /** * A slot's source dropdown and the control beneath it. * - * With a resource chosen the editor is replaced — by a read-only preview of - * the resource's value where the server already has one to show (see - * `ResourceInfo.value`), or otherwise by a chip naming the resource. Most - * resources' values don't exist until the run creates them, which is when a - * chip is all there is to show. (Seeding an editor from a stored value is - * what a dataset row is for.) - * - * When the chosen resource declares `parameters` (`specs/resource-arguments.md` - * §B), its argument form renders as a block beneath the chip — see - * {@link ResourceArgumentsForm}. + * With a source chosen the editor is replaced — by a read-only preview of a + * catalog resource's value where the server already has one to show (see + * `ResourceInfo.value`), or otherwise by a chip naming it: one of the run's + * resource instances (`◆ root.taskId`), a column, or another slot. A chip is + * only a reference: an instance's arguments live on its card in the + * Resources section, which the chip opens. * * Shared by {@link ExecutionInputEditor} (a slot's own row and every nested * one) and {@link CombinedInputEditor} (one row per deduped group) so both @@ -87,7 +50,7 @@ export function SourceRow({ matching, chosen, onChoose, - argsContext, + context, children, }: { propDef: PropDefinition; @@ -96,57 +59,74 @@ export function SourceRow({ matching: ResourceInfo[]; chosen: string | undefined; onChoose: (uri: string | null) => void; - /** See {@link ResourceArgsContext}. Absent means no argument form is ever rendered. */ - argsContext?: ResourceArgsContext; + /** See {@link SourceContext}. */ + context?: SourceContext; children: React.ReactNode; }) { const editable = propDef.type.kind !== "opaque"; const selected = chosen ? matching.find(r => r.uri === chosen) : undefined; - // A stored choice whose resource is gone — renamed, deleted, or in a module + const pseudo = + chosen && isPseudoUri(chosen) + ? (context?.describePseudo?.(chosen, propDef.type) ?? { note: "" }) + : undefined; + // A stored choice whose source is gone — renamed, deleted, or in a module // that now fails to load. It stays listed, and selected, because it is still // what a run would send: dropping it silently would show an editor while - // submitting the resource behind it. - const stale = chosen && !selected ? chosen : undefined; + // submitting the source behind it. + const stale = chosen && !selected && !pseudo?.label ? chosen : undefined; - // A resource already selected by an earlier row is the *same instance* - // here too (§I), whether or not it takes arguments — the chip says so - // instead of repeating "created once per server"/"created for each run", - // which would otherwise wrongly imply a second one gets made. const resourcesByUri = useMemo( () => new Map(resources.map(r => [r.uri, r])), [resources], ); - const pseudo = - selected && isPseudoUri(selected.uri) - ? (argsContext?.describePseudo?.(selected.uri, propDef.type) ?? { - note: "", - }) - : undefined; - const sharedWith = - selected && !pseudo && argsContext - ? argsContext.claimed.get(rootResourceUri(selected.uri, resourcesByUri)) - : undefined; - const sameInstanceAs = - sharedWith && sharedWith.path !== argsContext?.path - ? sharedWith - : undefined; - if (matching.length === 0 && !stale && editable) return <>{children}; + if (matching.length === 0 && !stale && !pseudo && editable) { + return <>{children}; + } + + const pseudoLabel = pseudo + ? (pseudo.label ?? + (selected ? chipLabel(selected, resourcesByUri) : undefined)) + : undefined; - const row = ( -
+ return ( +
- {selected && pseudo ? ( + {editable && selected?.value !== undefined ? ( + // The real value beats naming it: only a live handle or a run-only + // resource (no `ResourceInfo.value` — see `ResourceRegistry.describe`) + // falls back to a chip. An instance of a static resource is that + // value, so it's previewed too. +
+ +
+ ) : pseudo && pseudoLabel !== undefined ? ( - {selected.label} + {pseudo.onOpen ? ( + + ) : ( + pseudoLabel + )} {(pseudo.warning || pseudo.note) && ( {pseudo.warning ? `⚠ ${pseudo.warning}` : pseudo.note} @@ -154,43 +134,14 @@ export function SourceRow({ )} ) : selected ? ( - // The real value beats naming it: only a live handle or a run-only - // resource (no `ResourceInfo.value` — see `ResourceRegistry.describe`) - // falls back to the chip. - editable && selected.value !== undefined ? ( -
- -
- ) : ( - - {chipLabel(selected, resourcesByUri)} - - {sameInstanceAs ? ( - <> - same instance as{" "} - - - ) : selected.scope === "server" ? ( - "created once per server" - ) : ( - "created for each run" - )} - - - ) + + {chipLabel(selected, resourcesByUri)} + + {selected.scope === "server" + ? "created once per server" + : "created for each run"} + + ) : stale ? (
Resource no longer available @@ -218,7 +169,7 @@ export function SourceRow({ {/* An opaque slot has no editor to fall back to, so its picker offers no "Custom" — picking a source is the only way to fill it. */} - {(matching.length > 0 || stale) && ( + {(matching.length > 0 || stale || pseudo) && ( ); - - const argsForm = - selected && !pseudo && argsContext ? ( - - ) : null; - - if (!argsForm) return row; - return ( -
- {row} - {argsForm} -
- ); -} - -/** - * The block beneath a chosen resource's chip: its own argument editors, only - * for the row that owns its first selection (`specs/resource-arguments.md` - * §I's "one binding per resource per prompt" rule) — every other selection - * of the same resource already says so on its chip (see `sameInstanceAs` in - * {@link SourceRow}), so it takes no space here at all. Ownership itself is - * decided elsewhere, purely, by `computeClaims` — this only reads the answer - * (by comparing full paths, not bare names — two rows can easily share a - * name, like a prompt's own `taskId` and some other resource's own `taskId` - * argument), which is what keeps it safe to call twice (React StrictMode) - * without the two calls disagreeing. - */ -function ResourceArgumentsForm({ - selected, - resources, - resourcesByUri, - argsContext, -}: { - selected: ResourceInfo; - resources: readonly ResourceInfo[]; - /** `resources`, by `uri`. */ - resourcesByUri: ReadonlyMap; - argsContext: ResourceArgsContext; -}) { - // `selected` may be one of the resource's own output values - // (`seededTask.taskId`) rather than the resource itself — only the root - // carries `parameters`, and the args form (and its state) belongs to the - // root regardless of which value was actually picked for this slot, so - // that `seededTask.taskId` here and `seededTask.title` on another slot - // share one set of arguments (§I). - const root = - resourcesByUri.get(rootResourceUri(selected.uri, resourcesByUri)) ?? - selected; - - const params = root.parameters; - if (!params || params.length === 0) return null; - if (argsContext.depth >= MAX_RESOURCE_ARG_DEPTH) return null; - - const claimedBy = argsContext.claimed.get(root.uri); - // No owner found is defensive only (computeClaims should always have - // visited whatever's actually selected) — default to rendering rather than - // hiding, so a gap in that computation never silently loses editability. - if (claimedBy && claimedBy.path !== argsContext.path) { - return null; - } - - const slots = argsContext.resourceSlots[root.uri] ?? {}; - const selections = argsContext.resourceArgs[root.uri] ?? {}; - const setSelections = (next: Selections) => - argsContext.onResourceArgsChange(root.uri, next); - - return ( -
- {params.map(param => ( -
- {param.name} - - setSelections({ ...selections, [param.name]: next }) - } - resources={resources} - slots={slots} - argsContext={nested(argsContext, `args.${param.name}`)} - /> -
- ))} -
- ); } diff --git a/src/client/components/TraceView.tsx b/src/client/components/TraceView.tsx index 28bbf98..e592d2a 100644 --- a/src/client/components/TraceView.tsx +++ b/src/client/components/TraceView.tsx @@ -7,6 +7,7 @@ import type { AgentInfo } from "../../shared/agent"; import type { NormalizedPrompt, PromptID, + RunResources, Span, Trace, TraceLiveEvent, @@ -270,6 +271,7 @@ function TraceView({ traceInputs.length > 0 && ({ inputs: traceInputs, + resources: rootPrompt.resources as RunResources | undefined, newDatasetFields: fieldsForTrace(rootPrompt, currentPrompt, traceInputs), prompt: currentPrompt?.providerId ? { diff --git a/src/client/components/__fixtures__/datasetView.pw.tsx b/src/client/components/__fixtures__/datasetView.pw.tsx index 45767af..8f87419 100644 --- a/src/client/components/__fixtures__/datasetView.pw.tsx +++ b/src/client/components/__fixtures__/datasetView.pw.tsx @@ -88,7 +88,7 @@ async function mockDataset( page: Page, dataset: Dataset, rows: DatasetRow[], - fields: Record = {}, + fields: Record = {}, ) { const pages: [number, number][] = []; const base = `**/api/datasets/local/${dataset.id}`; @@ -130,8 +130,10 @@ const SUPPORT_ROWS: DatasetRow[] = [ id: "r1", cells: { "0": text("My order never arrived"), - "1": { - kind: "resource", + "1": { kind: "instance", name: "seededTask" }, + }, + resources: { + seededTask: { uri: ".evalution/playground/tasks.ts#seededTask", args: { title: text("Buy milk") }, }, @@ -143,15 +145,16 @@ const SUPPORT_ROWS: DatasetRow[] = [ id: "r2", cells: { "0": text("Refund please"), - // Takes no `title`: that column is n/a for this row. - "1": { kind: "resource", uri: ".evalution/playground/tasks.ts#blank" }, + "1": { kind: "instance", name: "blank" }, }, + resources: { blank: { uri: ".evalution/playground/tasks.ts#blank" } }, createdAt: 2, }, { id: "r3", cells: {}, createdAt: 3 }, ]; -const SUPPORT_SHAPE = { "1": { keys: ["title"], resource: true } }; +/** No field has keys: a cell naming an instance doesn't expand. */ +const SUPPORT_SHAPE = {}; /** * Serves `POST …/:id/rows` — what the trailing row sends — appending to @@ -203,9 +206,7 @@ test("DatasetView draws value and resource cells, sparse cells as dashes", async await expect( inGrid(component, "gridcell", "My order never arrived"), ).toBeAttached(); - await expect( - inGrid(component, "gridcell", '◆ seededTask(title: "Buy milk")'), - ).toBeAttached(); + await expect(inGrid(component, "gridcell", "◆ seededTask")).toBeAttached(); await expect(inGrid(component, "gridcell", "trace ↗")).toBeAttached(); // Row 3 has nothing in `task`, nor a source; row 2 has no source. (Its // empty `ticket` is an editor's, which reads as empty.) @@ -223,13 +224,12 @@ test("the table ends after its last row, with nothing ruled below it", async ({ await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); // Glide rules its whole canvas, so what's below the last row is covered: - // group header + header + 3 rows and the trailing "New row", one past the - // closing rule. + // header + 3 rows and the trailing "New row", one past the closing rule. const cover = component.locator(".dataset-grid-fill"); const box = await cover.boundingBox(); const grid = await component.locator(".dataset-grid").boundingBox(); if (!box || !grid) throw new Error("grid not laid out"); - expect(box.y - grid.y).toBe(GRID.group + GRID.header + 4 * GRID.row + 1); + expect(box.y - grid.y).toBe(GRID.header + 4 * GRID.row + 1); // It reaches the bottom, so no ruled lines survive below it. expect(Math.round(box.y + box.height)).toBe(Math.round(grid.y + grid.height)); @@ -261,35 +261,11 @@ test("the table ends after its last row, with nothing ruled below it", async ({ expect(stacking.scroller.zIndex).toBeGreaterThan(stacking.cover.zIndex); }); -test("clicking a field's group header splits it into a column per argument", async ({ - mount, - page, -}) => { - await mockDataset(page, SUPPORT, SUPPORT_ROWS, SUPPORT_SHAPE); - const component = await mount( - , - ); - await expect(inGrid(component, "columnheader", "task")).toBeAttached(); - await expect(inGrid(component, "columnheader", "title")).toHaveCount(0); - - // `task` is the second column: 240px past `ticket`. - await clickGrid(page, GRID.marker + 240 + 60, GRID.group / 2); - - await expect(inGrid(component, "columnheader", "title")).toBeAttached(); - // The head names the resource; the argument has its own column. - await expect(inGrid(component, "gridcell", "◆ seededTask")).toBeAttached(); - await expect(inGrid(component, "gridcell", '"Buy milk"')).toBeAttached(); - await expect(inGrid(component, "gridcell", "n/a")).toHaveCount(1); - - // And back. - await clickGrid(page, GRID.marker + 240 + 60, GRID.group / 2); - await expect(inGrid(component, "columnheader", "title")).toHaveCount(0); -}); - -test("a field of typed-in objects expands into its properties, with no column of its own", async ({ - mount, - page, -}) => { +/** + * Serves `tickets` with a `taskInfo` field of typed-in objects, whose cells + * expand into their `title` and `description`. + */ +function mockTaskInfo(page: Page) { const info: ExecutionInput = { kind: "value", value: { @@ -300,7 +276,7 @@ test("a field of typed-in objects expands into its properties, with no column of }, }, }; - await mockDataset( + return mockDataset( page, { ...SUPPORT, @@ -319,6 +295,13 @@ test("a field of typed-in objects expands into its properties, with no column of [{ id: "r1", cells: { "0": text("Hi"), "1": info }, createdAt: 1 }], { "1": { keys: ["title", "description"] } }, ); +} + +test("a field of typed-in objects expands into its properties, with no column of its own", async ({ + mount, + page, +}) => { + await mockTaskInfo(page); const component = await mount( , ); @@ -346,10 +329,10 @@ test("expanding a field and resizing a column are remembered per dataset", async page, }) => { await page.evaluate(() => localStorage.clear()); - await mockDataset(page, SUPPORT, SUPPORT_ROWS, SUPPORT_SHAPE); + await mockTaskInfo(page); const view = ; const component = await mount(view); - // Columns start at their defaults: marker + ticket + task + source. + // Columns start at their defaults: marker + ticket + taskInfo + source. await expect(gridWidth(component)).toHaveAttribute("style", /width: 608px/); await clickGrid(page, GRID.marker + 240 + 60, GRID.group / 2); @@ -367,9 +350,9 @@ test("expanding a field and resizing a column are remembered per dataset", async await component.unmount(); const reopened = await mount(view); - // Expanded, and `ticket` still 360 wide: 32 + 360 + 140 + 160 + 96. + // Expanded, and `ticket` still 360 wide: 32 + 360 + 160 + 160 + 96. await expect(inGrid(reopened, "columnheader", "title")).toBeAttached(); - await expect(gridWidth(reopened)).toHaveAttribute("style", /width: 788px/); + await expect(gridWidth(reopened)).toHaveAttribute("style", /width: 808px/); // Deleting the dataset takes its layout with it. await page.route("**/api/datasets/local/tickets", route => @@ -404,14 +387,19 @@ test("selecting a row shows it in full in the details pane", async ({ ); await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); - await clickCell(page, 40, 0, true); + await clickCell(page, 40, 0, false); const pane = component.getByRole("region", { name: "Row details" }); await expect(pane).toContainText("Row 1"); await expect(pane).toContainText("r1"); await expect(pane.getByText("My order never arrived")).toBeVisible(); - await expect(pane.getByText("seededTask")).toBeVisible(); - await expect(pane.getByText("title")).toBeVisible(); + // The row's resources come first, each with its arguments; the field + // names the instance it's bound to. + await expect(pane.getByText("◆ seededTask", { exact: true })).toBeVisible(); + await expect(pane.getByText("title", { exact: true })).toBeVisible(); await expect(pane.getByText("Buy milk")).toBeVisible(); + await expect( + pane.locator(".dataset-detail-cell").getByText("seededTask"), + ).toBeVisible(); // Beside the grid when there's room. await expect(pane).toHaveClass(/trace-details-pane/); @@ -440,7 +428,7 @@ test("double-clicking a cell to edit it in place doesn't open the details pane", // pause between its clicks is long enough for a pane to open in. const at = { x: box.x + GRID.marker + 40, - y: box.y + GRID.group + GRID.header + GRID.row / 2, + y: box.y + GRID.header + GRID.row / 2, }; await page.mouse.click(at.x, at.y); await page.waitForTimeout(100); @@ -452,10 +440,10 @@ test("double-clicking a cell to edit it in place doesn't open the details pane", // A click that doesn't go on to edit still opens it, once it has paused... await page.keyboard.press("Escape"); - await clickCell(page, 240 + 40, 0, true); + await clickCell(page, 240 + 40, 0, false); await expect(pane).toContainText("Row 1"); // ...and, open, it follows the selection at once. - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await expect(pane).toContainText("Row 2"); }); @@ -487,7 +475,7 @@ test("the details pane opens a row in the playground and deletes it", async ({ ); await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); const pane = component.getByRole("region", { name: "Row details" }); await expect(pane).toContainText("Row 2"); await pane.getByRole("button", { name: "Open row in playground" }).click(); @@ -599,7 +587,7 @@ test("typing into a string cell saves it, showing it at once", async ({ await untilRowsShown(component); // Row 3's `ticket` is empty. - await clickCell(page, 40, 2, true); + await clickCell(page, 40, 2, false); await typeIntoCell(page, "Late delivery"); await expect @@ -610,11 +598,11 @@ test("typing into a string cell saves it, showing it at once", async ({ await expect(inGrid(component, "gridcell", "Late delivery")).toBeAttached(); // A resource cell is read-only in the grid: no editor opens there. - await clickCell(page, 240 + 40, 0, true); + await clickCell(page, 240 + 40, 0, false); await pressInGrid(page, "Enter"); await expect(page.locator(".dataset-grid-portal textarea")).toHaveCount(0); // Delete clears an editable cell to `null`, and skips a read-only one. - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await pressInGrid(page, "Delete"); await expect.poll(() => sent).toHaveLength(2); expect(sent[1]).toEqual({ @@ -638,10 +626,10 @@ test("a failed save reloads the row from the server and says why under the heade ); await untilRowsShown(component); - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await typeIntoCell(page, "Changed"); await expect(inGrid(component, "gridcell", "Changed")).toBeAttached(); - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await typeIntoCell(page, "Changed again"); await expect(inGrid(component, "gridcell", "Changed again")).toBeAttached(); @@ -732,7 +720,7 @@ test("the details pane edits a value cell, committing on Enter, and clears any c ); await untilRowsShown(component); - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); const pane = component.getByRole("region", { name: "Row details" }); const editor = pane.locator(".dataset-detail-editor textarea"); await expect(editor).toHaveValue("Refund please"); @@ -749,7 +737,7 @@ test("the details pane edits a value cell, committing on Enter, and clears any c await expect(inGrid(component, "gridcell", "Refund now")).toBeAttached(); // The resource is read-only here too, but can be cleared. - await expect(pane.getByText("blank")).toBeVisible(); + await expect(pane.getByText("◆ blank", { exact: true })).toBeVisible(); await pane.getByRole("button", { name: "Clear task" }).click(); await expect.poll(() => sent).toHaveLength(2); expect(sent[1]).toEqual({ @@ -773,9 +761,9 @@ test("a details-pane edit is saved when the grid is clicked, on the same row or const editor = pane.locator(".dataset-detail-editor textarea"); // Another cell of the same row: the pane stays, and the edit is saved. - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await editor.fill("Refund now"); - await clickCell(page, 300, 1, true); + await clickCell(page, 300, 1, false); await expect .poll(() => sent) .toEqual([ @@ -784,9 +772,9 @@ test("a details-pane edit is saved when the grid is clicked, on the same row or // Another row: the pane moves on, and the edit is saved to the row it // was made on. - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); await editor.fill("Refund today"); - await clickCell(page, 40, 0, true); + await clickCell(page, 40, 0, false); await expect(editor).toHaveValue("My order never arrived"); await expect.poll(() => sent).toHaveLength(2); expect(sent[1]).toEqual({ @@ -838,47 +826,51 @@ test("the details pane offers the linked prompt's resources, with their argument ); await untilRowsShown(component); - await clickCell(page, 40, 1, true); + await clickCell(page, 40, 1, false); const pane = component.getByRole("region", { name: "Row details" }); - await expect(pane.getByText("Blank task")).toBeVisible(); + const resources = pane.getByRole("region", { name: "Resources" }); + await expect(resources.locator("[data-instance-card='blank']")).toBeVisible(); - // Choosing a resource is saved at once. + // Choosing a resource from New adds an instance of it to the row and + // binds the field to it, saved at once. await pane.getByRole("button", { name: "Source for task" }).click(); + await page.getByRole("menuitem", { name: "New" }).click(); await page .getByRole("menuitem", { name: "Seeded task", exact: true }) .click(); + await expect( + resources.locator("[data-instance-card='seededTask']"), + ).toBeVisible(); await expect - .poll(() => sent) - .toEqual([ - { - updates: [ - { - rowId: "r2", - cells: { "1": { kind: "resource", uri: seeded.uri } }, - }, - ], - }, - ]); + .poll(() => sent.at(-1)) + .toEqual({ + updates: [ + { + rowId: "r2", + cells: { "1": { kind: "instance", name: "seededTask" } }, + // Only what changed: `blank` stays as it is. + resources: { seededTask: { uri: seeded.uri } }, + }, + ], + }); + // One save, not one for the instance and another for the binding. + expect(sent).toHaveLength(1); - // Its argument is typed in like any value: a draft until Enter. - const title = pane - .locator(".dataset-detail-editor") - .nth(1) + // Its argument is typed into its card like any value: a draft until Enter. + const title = resources + .locator("[data-instance-card='seededTask']") .locator("textarea"); await title.fill("Buy eggs"); expect(sent).toHaveLength(1); await title.press("Enter"); await expect.poll(() => sent).toHaveLength(2); - expect(sent[1]).toEqual({ + expect(sent.at(-1)).toEqual({ updates: [ { rowId: "r2", - cells: { - "1": { - kind: "resource", - uri: seeded.uri, - args: { title: text("Buy eggs") }, - }, + cells: {}, + resources: { + seededTask: { uri: seeded.uri, args: { title: text("Buy eggs") } }, }, }, ], @@ -1017,7 +1009,7 @@ test("clicking the trailing row appends an empty row and selects it", async ({ ); await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); - await clickCell(page, 40, 3, true); + await clickCell(page, 40, 3, false); await expect(component.getByText("4 rows")).toBeVisible(); await expect.poll(() => bodies).toEqual([{ rows: [{ cells: {} }] }]); @@ -1104,14 +1096,14 @@ test("two rows appended at once both stay while the first one's refetch lands", ); await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); - await clickCell(page, 40, 3, true); + await clickCell(page, 40, 3, false); await expect(component.getByText("4 rows")).toBeVisible(); const refetched = page.waitForResponse( r => r.request().method() === "GET" && r.url().endsWith("/api/datasets/local/tickets"), ); - await clickCell(page, 40, 4, true); + await clickCell(page, 40, 4, false); await expect(component.getByText("5 rows")).toBeVisible(); await expect.poll(() => bodies).toHaveLength(2); @@ -1134,7 +1126,7 @@ test("a row that fails to append is taken back, with the error shown", async ({ ); await expect(component.getByTestId("data-grid-canvas")).toBeVisible(); - await clickCell(page, 40, 3, true); + await clickCell(page, 40, 3, false); await expect(component.getByText("4 rows")).toBeVisible(); await expect(component.getByText("disk full")).toBeVisible(); await expect(component.getByText("3 rows")).toBeVisible(); diff --git a/src/client/components/__fixtures__/evalView.pw.tsx b/src/client/components/__fixtures__/evalView.pw.tsx index 75224ce..617f05a 100644 --- a/src/client/components/__fixtures__/evalView.pw.tsx +++ b/src/client/components/__fixtures__/evalView.pw.tsx @@ -142,6 +142,60 @@ test("binds a same-named column to the prompt's input, marked as matched, and sa await expect(component.getByRole("button", { name: /Run/ })).toBeEnabled(); }); +test("the inputs panel has the run's Resources above the slots, saved with the eval", async ({ + mount, + page, +}) => { + const seeded = { + uri: ".evalution/playground/tasks.ts#seededTask", + label: "Seeded task", + scope: "run" as const, + parameters: [string("title")], + }; + const withResources = { + ...TRIAGE, + inputSources: { + resources: [seeded], + functionSlots: {}, + executeSlots: {}, + }, + } as NormalizedPrompt; + const patches = await mockEval(page, EVAL); + const component = await mount( + , + ); + + const resources = component.getByRole("region", { name: "Resources" }); + await resources.getByRole("button", { name: "＋ Add resource" }).click(); + await page + .getByRole("menuitem", { name: "Seeded task", exact: true }) + .click(); + const card = resources.locator("[data-instance-card='seededTask']"); + await expect(card).toContainText("not bound to a slot"); + + // Its argument can take a column, as any slot in an eval can. + await card.locator(".pg-slot-source-wrap").click(); + await page + .getByRole("menuitem", { name: /ticket/ }) + .first() + .click(); + await expect + .poll(() => patches.at(-1)?.inputs) + .toMatchObject({ + resources: { + seededTask: { + uri: seeded.uri, + args: { title: { kind: "dataset", field: "0" } }, + }, + }, + }); +}); + test("lists an unbound check parameter as a problem, and won't run until it's bound", async ({ mount, page, diff --git a/src/client/components/__fixtures__/executionFixtures.ts b/src/client/components/__fixtures__/executionFixtures.ts index 900fb50..163ec50 100644 --- a/src/client/components/__fixtures__/executionFixtures.ts +++ b/src/client/components/__fixtures__/executionFixtures.ts @@ -174,26 +174,6 @@ export const TITLE_GENERATOR: ResourceInfo = { scope: "run", }; -/** - * A second parameterized resource whose own argument happens to be named - * `taskId` — the exact same name as a top-level prompt parameter a test - * might also be filling from {@link PARAMETERIZED_SEEDED_TASK}. Regression - * fixture for the name-collision bug a bare-label "who owns this form" check - * fell into: two rows named `taskId` at different depths in the tree are not - * the same row. - */ -export const SEEDED_RUN_TASK_ID_PARAM: PropDefinition = { - name: "taskId", - type: { kind: "primitive", syntax: "TaskId", base: "string" }, - optional: false, -}; -export const SEEDED_RUN: ResourceInfo = { - uri: "seeded-task.playground.ts#seededRun", - label: "Seeded run", - scope: "run", - parameters: [SEEDED_RUN_TASK_ID_PARAM], -}; - /** * A static `value` resource the server already knows, so the panel can * preview it — unlike {@link SEEDED_TASK}, which only exists once a run @@ -281,3 +261,22 @@ export function sourcesFor( ...(resourceSlots ? { resourceSlots } : {}), }; } + +/** + * A task that can be seeded under another — `specs/resource-instances.md`'s + * hierarchy case: its `parentId` argument takes the id of another instance + * of the very same resource. + */ +export const TREE_TASK: ResourceInfo = { + uri: "tree.playground.ts#task", + label: "Seeded task", + scope: "run", + parameters: [ + SEEDED_TASK_PARAM, + { + name: "parentId", + type: { kind: "primitive", syntax: "TaskId", base: "string" }, + optional: true, + }, + ], +}; diff --git a/src/client/components/__fixtures__/playgroundExecution.pw.tsx b/src/client/components/__fixtures__/playgroundExecution.pw.tsx index 5ecdd1f..b1b9db2 100644 --- a/src/client/components/__fixtures__/playgroundExecution.pw.tsx +++ b/src/client/components/__fixtures__/playgroundExecution.pw.tsx @@ -9,7 +9,6 @@ import { OPAQUE_DB, odinDbSlots, PARAMETERIZED_SEEDED_TASK, - SEEDED_RUN, SEEDED_TASK, sourcesFor, TASK_A, @@ -21,6 +20,7 @@ import { TASKS_LIBRARY, TITLE_GENERATOR, TOOLS_CONTEXT_WITH_NESTED_DB, + TREE_TASK, WORKSPACE_RESOURCE, } from "./executionFixtures"; import { PlaygroundExecutionHarness } from "./PlaygroundExecutionHarness"; @@ -38,6 +38,17 @@ async function chooseSource(page: Page, trigger: Locator, label: string) { await page.getByRole("menuitem", { name: label, exact: true }).click(); } +/** + * Like {@link chooseSource}, for a catalog entry filed under **New**: where + * the menu offers more than the catalog (the run's instances, another + * slot), the catalog sits one submenu down. + */ +async function chooseNew(page: Page, trigger: Locator, label: string) { + await trigger.click(); + await page.getByRole("menuitem", { name: "New" }).click(); + await page.getByRole("menuitem", { name: label, exact: true }).click(); +} + async function mockExecute(page: Page) { await page.route("**/api/**", route => route.fulfill({ @@ -328,11 +339,15 @@ test("picking a resource replaces the editor with a labelled chip, and it surviv ); // A resource's value doesn't exist until the run creates it, so there is - // nothing to seed an editor with — the chip says what will happen instead. + // nothing to seed an editor with. The pick added an instance of it to the + // run, and the chip names that instance and what it is. const chip = component.locator(".pg-slot-chip"); + await expect(chip).toContainText("◆ seededRootTask"); await expect(chip).toContainText(SEEDED_TASK.label); - await expect(chip).toContainText("created for each run"); await expect(component.locator("textarea, input")).toHaveCount(0); + await expect( + component.locator("[data-instance-card='seededRootTask']"), + ).toContainText("per run"); await component.unmount(); const remounted = await mount(); @@ -341,120 +356,6 @@ test("picking a resource replaces the editor with a labelled chip, and it surviv ); }); -test("choosing the same resource for a second slot says 'same instance as' on its chip, even with no arguments to share", async ({ - mount, - page, -}) => { - // Regression: the "one binding per resource" note used to only exist for - // a resource with declared arguments — a plain resource (no `parameters`) - // picked twice showed two identical, unrelated-looking chips with no hint - // that a run creates it once and reuses it, not twice. - const component = await mount( - , - ); - - const rows = component.locator(".pg-exec-param"); - await chooseSource( - page, - rows.nth(0).locator(".pg-slot-source-wrap"), - SEEDED_TASK.label, - ); - await chooseSource( - page, - rows.nth(1).locator(".pg-slot-source-wrap"), - SEEDED_TASK.label, - ); - - // The first (owning) selection keeps its normal scope note. - await expect(rows.nth(0).locator(".pg-slot-chip-note")).toContainText( - "created for each run", - ); - // The second names the row it's the same instance as, instead — as a link - // (a chevron pointing up at it, not the word "above") rather than plain text. - const note = rows.nth(1).locator(".pg-slot-chip-note"); - await expect(note).toContainText("same instance as"); - await expect(note).not.toContainText("above"); - const link = note.getByRole("button", { name: /taskId/ }); - await expect(link).toContainText("▲"); - await expect(link).toContainText("taskId"); - // Neither row wastes space on an args form — this resource has none. - await expect(component.locator(".pg-args-form")).toHaveCount(0); -}); - -test("clicking a 'same instance as' link scrolls to and briefly highlights the row it points at", async ({ - mount, - page, -}) => { - const component = await mount( - , - ); - - const rows = component.locator(".pg-exec-param"); - await chooseSource( - page, - rows.nth(0).locator(".pg-slot-source-wrap"), - SEEDED_TASK.label, - ); - await chooseSource( - page, - rows.nth(1).locator(".pg-slot-source-wrap"), - SEEDED_TASK.label, - ); - - const ownerSlot = rows.nth(0).locator(".pg-slot"); - await expect(ownerSlot).not.toHaveClass(/pg-row-highlight/); - - await rows - .nth(1) - .locator(".pg-slot-chip-note") - .getByRole("button", { name: /taskId/ }) - .click(); - - // The owning row (not the one clicked) is the one that flashes. - await expect(ownerSlot).toHaveClass(/pg-row-highlight/); - await expect(rows.nth(1).locator(".pg-slot")).not.toHaveClass( - /pg-row-highlight/, - ); - // And the flash is temporary, not a lasting state change. - await expect(ownerSlot).not.toHaveClass(/pg-row-highlight/, { - timeout: 2000, - }); -}); - test("a resource the server already has a value for previews it, read-only, through the slot's own editor", async ({ mount, page, @@ -541,8 +442,9 @@ test("an opaque slot with a matching resource makes the picker its only control" await expect(page.getByRole("menuitem")).toHaveText([DB_RESOURCE.label]); await page.getByRole("menuitem", { name: DB_RESOURCE.label }).click(); - await expect(component.locator(".pg-slot-chip")).toContainText( - "created once per server", + await expect(component.locator(".pg-slot-chip")).toContainText("◆ db"); + await expect(component.locator("[data-instance-card='db']")).toContainText( + "shared", ); }); @@ -954,20 +856,16 @@ test.describe("SourcePicker (specs/resource-hierarchy.md §E)", () => { .getByRole("menuitem", { name: TASK_A_ID.label, exact: true }) .click(); - // Task A has three matching outputs, so the dropdown row that chose this - // one was itself labelled `"${TASK_A.label} → ${TASK_A_ID.label}"` — the - // chip repeats that same composite, arrow and all, now that the tree - // (and its breadcrumb) is no longer on screen to say which task this is. + // The pick added an instance of Task A, and the chip names its output + // as `instance.output`, with the resource it's an instance of. const chip = component.locator(".pg-slot-chip"); - await expect(chip).toHaveText( - `${TASK_A.label} → ${TASK_A_ID.label}created for each run`, - ); + await expect(chip).toHaveText(`◆ taskA.id${TASK_A.label}`); await expect(component.locator("textarea, input")).toHaveCount(0); await component.unmount(); const remounted = await mount(taskRefHarness("task-picker-prompt")); - await expect(remounted.locator(".pg-slot-chip")).toContainText( - `${TASK_A.label} → ${TASK_A_ID.label}`, + await expect(remounted.locator(".pg-slot-chip")).toHaveText( + `◆ taskA.id${TASK_A.label}`, ); }); @@ -982,7 +880,7 @@ test.describe("SourcePicker (specs/resource-hierarchy.md §E)", () => { // The row itself is labelled "Resource → Value" (its one declared value // collapsed the submenu away, same as Task A's narrowed-down one did) — // and it's still that value being chosen underneath, so the chip - // afterward names the same pair. + // afterward names that output of the new instance. const taskBRow = page.getByRole("menuitem", { name: `${TASK_B.label} → ${TASK_B_ID.label}`, exact: true, @@ -991,7 +889,7 @@ test.describe("SourcePicker (specs/resource-hierarchy.md §E)", () => { await taskBRow.click(); await expect(component.locator(".pg-slot-chip")).toHaveText( - `${TASK_B.label} → ${TASK_B_ID.label}created for each run`, + `◆ taskB.id${TASK_B.label}`, ); }); @@ -1169,7 +1067,7 @@ test.describe("resource arguments (specs/resource-arguments.md §I, §J)", () => await expect(remountedForm.locator("select")).toHaveValue("open"); }); - test("choosing the same resource for a second slot marks its chip 'same instance as' instead of repeating the form", async ({ + test("the nested title argument offers Title generator as a source, and picking it shows a chip", async ({ mount, page, }) => { @@ -1181,157 +1079,303 @@ test.describe("resource arguments (specs/resource-arguments.md §I, §J)", () => type: { kind: "primitive", syntax: "TaskId", base: "string" }, optional: false, }, - { - name: "taskTitle", - type: { kind: "primitive", syntax: "string", base: "string" }, - optional: false, - }, ]} inputSources={sourcesFor( - { - taskId: [PARAMETERIZED_SEEDED_TASK.uri], - taskTitle: [PARAMETERIZED_SEEDED_TASK.uri], - }, + { taskId: [PARAMETERIZED_SEEDED_TASK.uri] }, [PARAMETERIZED_SEEDED_TASK, TITLE_GENERATOR], + "functionSlots", + { [PARAMETERIZED_SEEDED_TASK.uri]: { title: [TITLE_GENERATOR.uri] } }, )} />, ); - - const rows = component.locator(".pg-exec-param"); await chooseSource( page, - rows.nth(0).locator(".pg-slot-source-wrap"), + component.locator(".pg-slot-source-wrap"), PARAMETERIZED_SEEDED_TASK.label, ); - const firstForm = rows.nth(0).locator(".pg-args-form").first(); - await firstForm.locator("textarea, input").first().fill("Shared title"); - await chooseSource( - page, - rows.nth(1).locator(".pg-slot-source-wrap"), - PARAMETERIZED_SEEDED_TASK.label, + const form = component.locator(".pg-args-form").first(); + const titleTrigger = form.locator(".pg-slot-source-wrap").first(); + await expect(titleTrigger).toBeVisible(); + // Beside the prompt's own `taskId` slot, which fits a string too. + await chooseNew(page, titleTrigger, TITLE_GENERATOR.label); + await expect(form.locator(".pg-slot-chip").first()).toContainText( + TITLE_GENERATOR.label, ); + }); +}); + +test.describe("resource instances (specs/resource-instances.md §G)", () => { + const TASK_ID = { + name: "taskId", + type: { + kind: "primitive" as const, + syntax: "TaskId", + base: "string" as const, + }, + optional: false, + }; + const PARENT_TASK_ID = { ...TASK_ID, name: "parentTaskId" }; + + /** The section's card for instance `name`. */ + const card = (scope: Locator, name: string) => + scope.locator(`[data-instance-card="${name}"]`); - // The second selection doesn't get its own editable form — its chip - // says it's the same instance as the row that owns it, and that's it: - // no form, no extra vertical space for one. - const note = rows.nth(1).locator(".pg-slot-chip-note"); - await expect(note).toContainText("same instance as"); - await expect(note.getByRole("button", { name: /taskId/ })).toBeVisible(); - await expect(rows.nth(1).locator(".pg-args-form")).toHaveCount(0); - // The first row's own value is untouched by the second selection. - await expect(firstForm.locator("textarea, input").first()).toHaveValue( - "Shared title", + /** Adds an instance of the resource labelled `label` from the section's own picker. */ + async function addResource(page: Page, scope: Locator, label: string) { + await scope.getByRole("button", { name: "＋ Add resource" }).click(); + await page.getByRole("menuitem", { name: label, exact: true }).click(); + } + + test("a resource added with no slot shows as unbound, and is sent with the run", async ({ + mount, + page, + }) => { + const bodies: unknown[] = []; + await page.route("**/api/**", route => { + if (route.request().method() === "POST") { + bodies.push(route.request().postDataJSON()); + } + return route.fulfill({ + json: { traceId: "t1", tracerProviderId: "p1", rootSpanId: "s1" }, + }); + }); + const component = await mount( + , ); + const resources = component.getByRole("region", { name: "Resources" }); + await addResource(page, resources, SEEDED_TASK.label); + + await expect(card(resources, "seededRootTask")).toContainText( + "not bound to a slot: runs for its side effects", + ); + + await component.getByRole("button", { name: "▶ Run" }).click(); + await expect + .poll(() => bodies.at(-1)) + .toMatchObject({ + resources: { seededRootTask: { uri: SEEDED_TASK.uri } }, + }); }); - test("the nested title argument offers Title generator as a source, and picking it shows a chip", async ({ + test("picking from New adds an instance and binds it; a second slot can reuse it", async ({ mount, page, }) => { const component = await mount( , ); + const rows = component.locator(".pg-exec-param"); + const resources = component.getByRole("region", { name: "Resources" }); + + // With no instances yet, the catalog is all there is: no New to open. await chooseSource( page, - component.locator(".pg-slot-source-wrap"), - PARAMETERIZED_SEEDED_TASK.label, + rows.nth(0).locator(".pg-slot-source-wrap"), + SEEDED_TASK.label, + ); + await expect(card(resources, "seededRootTask")).toContainText("→ taskId"); + + // Now the run's instance comes first, and the catalog sits under New. + await rows.nth(1).locator(".pg-slot-source-wrap").click(); + await expect(page.getByRole("menuitem")).toHaveText([ + "✓Custom", + "seededRootTask", + "New▸", + "= taskId", + ]); + await page.getByRole("menuitem", { name: "seededRootTask" }).click(); + + await expect(rows.nth(1).locator(".pg-slot-chip")).toContainText( + "◆ seededRootTask", + ); + // Still one instance, which both slots name. + await expect(resources.locator("[data-instance-card]")).toHaveCount(1); + await expect(card(resources, "seededRootTask")).toContainText( + "→ taskId, parentTaskId", ); - const form = component.locator(".pg-args-form").first(); - const titleTrigger = form.locator(".pg-slot-source-wrap").first(); - await expect(titleTrigger).toBeVisible(); - await chooseSource(page, titleTrigger, TITLE_GENERATOR.label); - await expect(form.locator(".pg-slot-chip").first()).toContainText( - TITLE_GENERATOR.label, + // Picking from New again is a second, separate instance. + await rows.nth(1).locator(".pg-slot-source-wrap").click(); + await page.getByRole("menuitem", { name: "New" }).click(); + await page + .getByRole("menuitem", { name: SEEDED_TASK.label, exact: true }) + .click(); + await expect(rows.nth(1).locator(".pg-slot-chip")).toContainText( + "◆ seededRootTask2", ); + await expect(card(resources, "seededRootTask")).toContainText("→ taskId"); }); - test("a prompt's own taskId slot and a different resource's own taskId argument don't collide just because they share a name", async ({ + test("clicking an instance's chip scrolls to and briefly highlights its card", async ({ mount, page, }) => { - // Regression: `SEEDED_RUN`'s own argument happens to be named `taskId`, - // the same as the prompt's own top-level parameter — the bare-name - // ownership check this replaced treated those as the same row, showing - // the form under *both* instead of one owning it and the other pointing - // back. See `ResourceArgsContext.path`. const component = await mount( , ); - - const rows = component.locator(".pg-exec-param"); - // Select seededTask for the prompt's own taskId slot first. await chooseSource( page, - rows.nth(0).locator(".pg-slot-source-wrap"), - PARAMETERIZED_SEEDED_TASK.label, + component.locator(".pg-slot-source-wrap"), + SEEDED_TASK.label, ); - // Then select seededRun for `run`, and seededTask again for *its* own - // `taskId` argument. + const target = card(component, "seededRootTask"); + await expect(target).not.toHaveClass(/pg-row-highlight/); + + await component + .locator(".pg-slot-chip") + .getByRole("button", { name: /seededRootTask/ }) + .click(); + await expect(target).toHaveClass(/pg-row-highlight/); + // The flash is temporary, not a lasting state change. + await expect(target).not.toHaveClass(/pg-row-highlight/, { + timeout: 2000, + }); + }); + + test("duplicating and then renaming an instance updates every chip that names it", async ({ + mount, + page, + }) => { + const props = { + functionParameters: [TASK_ID], + inputSources: sourcesFor({ taskId: [PARAMETERIZED_SEEDED_TASK.uri] }, [ + PARAMETERIZED_SEEDED_TASK, + ]), + promptId: "instances-rename-prompt", + }; + const component = await mount(); + const resources = component.getByRole("region", { name: "Resources" }); await chooseSource( page, - rows.nth(1).locator(".pg-slot-source-wrap"), - SEEDED_RUN.label, + component.locator(".pg-slot-source-wrap"), + PARAMETERIZED_SEEDED_TASK.label, + ); + await card(resources, "seededTask") + .locator("textarea, input") + .first() + .fill("Todo app"); + + // A duplicate copies the arguments, under the next free name. + await card(resources, "seededTask") + .getByRole("button", { name: "Duplicate" }) + .click(); + await expect( + card(resources, "seededTask2").locator("textarea, input").first(), + ).toHaveValue("Todo app"); + await expect(card(resources, "seededTask2")).toContainText( + "not bound to a slot", + ); + + // Renaming carries the slot's reference along with it. + await card(resources, "seededTask") + .getByRole("button", { name: /seededTask/ }) + .first() + .click(); + const nameInput = resources.getByLabel("Name for seededTask"); + await nameInput.fill("seededTask2"); + // A name that's taken is refused. + await expect(nameInput).toHaveAttribute("aria-invalid", "true"); + await nameInput.fill("root"); + await nameInput.press("Enter"); + await expect(component.locator(".pg-slot-chip")).toContainText("◆ root"); + await expect(card(resources, "root")).toContainText("→ taskId"); + + // And it all survives a remount. + await component.unmount(); + const remounted = await mount(); + await expect(remounted.locator(".pg-slot-chip")).toContainText("◆ root"); + await expect( + card(remounted, "seededTask2").locator("textarea, input").first(), + ).toHaveValue("Todo app"); + }); + + test("removing a referenced instance asks first, then clears the slot", async ({ + mount, + page, + }) => { + const component = await mount( + , ); - const runForm = rows.nth(1).locator(".pg-args-form").first(); await chooseSource( page, - runForm.locator(".pg-slot-source-wrap").first(), - PARAMETERIZED_SEEDED_TASK.label, + component.locator(".pg-slot-source-wrap"), + SEEDED_TASK.label, ); + const dialogs: string[] = []; + page.once("dialog", dialog => { + dialogs.push(dialog.message()); + return dialog.dismiss(); + }); + const remove = component.getByRole("button", { + name: "Remove seededRootTask", + }); + await remove.click(); + expect(dialogs).toEqual([ + "Remove 'seededRootTask'? taskId will be cleared.", + ]); + await expect(card(component, "seededRootTask")).toHaveCount(1); + + page.once("dialog", dialog => dialog.accept()); + await remove.click(); + await expect(card(component, "seededRootTask")).toHaveCount(0); + await expect(component.locator(".pg-slot-chip")).toHaveCount(0); + }); - // Exactly one editable seededTask form exists (title + status) — the - // prompt's own taskId slot owns it. - await expect(rows.nth(0).locator(".pg-args-form .pg-args-name")).toHaveText( - ["title", "status"], + test("a task seeded under another names it in its argument, and a cycle isn't offered", async ({ + mount, + page, + }) => { + const component = await mount( + , ); - // seededRun's own row shows only its own argument ("taskId"), whose chip - // says it's the same instance as the prompt's own taskId row — not a - // second title/status form nested inside it. - await expect(rows.nth(1).locator(".pg-args-name")).toHaveText(["taskId"]); - const nestedNote = runForm.locator(".pg-slot-chip-note"); - await expect(nestedNote).toContainText("same instance as"); + const resources = component.getByRole("region", { name: "Resources" }); + await addResource(page, resources, TREE_TASK.label); + await addResource(page, resources, TREE_TASK.label); + await expect(resources.locator("[data-instance-card]")).toHaveCount(2); + + // `task2`'s parent is `task`: another instance of the same resource. + const parentOf = (name: string) => + card(resources, name) + .locator(".pg-args-row") + .filter({ hasText: "parentId" }) + .locator(".pg-slot-source-wrap"); + await parentOf("task2").click(); + await page.getByRole("menuitem", { name: "task", exact: true }).click(); await expect( - nestedNote.getByRole("button", { name: /taskId/ }), - ).toBeVisible(); + card(resources, "task2").locator(".pg-slot-chip"), + ).toContainText("◆ task"); + await expect(card(resources, "task")).toContainText("→ task2.parentId"); + + // So `task` can't take `task2` as its own parent, nor itself: only a + // new instance is offered. + await parentOf("task").click(); + await expect(page.getByRole("menuitem")).not.toContainText(["task2"]); + await expect( + page.getByRole("menuitem", { name: "task", exact: true }), + ).toHaveCount(0); }); }); diff --git a/src/client/components/combined-inputs.test.ts b/src/client/components/combined-inputs.test.ts index db6de90..5dce832 100644 --- a/src/client/components/combined-inputs.test.ts +++ b/src/client/components/combined-inputs.test.ts @@ -253,7 +253,7 @@ function odinInput(): ExecutionInput { ): ExecutionInput => ({ kind: "object", properties: { - db: { kind: "resource", uri: ".evalution/playground/db.ts#db" }, + db: { kind: "instance", name: "db" }, workspaceId: { kind: "value", value: { kind: "primitive", value: "ws_internal_default" }, diff --git a/src/client/components/combined-inputs.ts b/src/client/components/combined-inputs.ts index b65b40e2ecee6bddf8b1aa061a9133b819d04376..2a271dcca291420242745515c39828632f935b69 100644 GIT binary patch delta 46 zcmewy@XmV!tK{Tq@_L&&rG!|)jC1na7$=7)MKNnyDNHUDlA0``yk_%7bvvg$Hos1$@g?(bvnIs}fs37VlOETV!c zO^8WBQwWA5C@{zdrXp)g%Q|rZ*=!7qD3^zhDhYoJI_Taqb}^F-qvOoU@DWcV5B5AaPw#!aus2czjmc5^U( zu$8l&az6>qiw6NI?X9IcvI3p?bDph}X5W4E2}}uc-DH?~?dUe}f0IsoZE4u9w#U0e zPx0aRJC`Hu;#Ez7mVM{;oO4Uq!#pL@RjwMk@{!$UCj)CS*F`Zy0G0gfkF1;A%NtiW zNr}?8PV5Pu6NkRk)}$)eWRgUBhu~Y0ZT2m*jFuH$*&gvPCjxh}La)u{OFf+syj9Wa#i$k2S_Mv!wdosblz7KJM#uZ!| SC(O_Iac?r@-MoGD_3j_QkMOVn diff --git a/src/client/components/dataset-grid.test.ts b/src/client/components/dataset-grid.test.ts index 9e01b82..d1b7db3 100644 --- a/src/client/components/dataset-grid.test.ts +++ b/src/client/components/dataset-grid.test.ts @@ -46,7 +46,7 @@ const FIELDS = [ field("2", "info", "Info"), ]; const SHAPE = { - "1": { keys: ["title", "owner"], resource: true }, + "1": { keys: ["title", "owner"] }, "2": { keys: ["title"] }, }; @@ -59,12 +59,16 @@ const info: ExecutionInput = { }, }; +/** An object grafting a resource instance in beside a typed-in value. */ const seeded: ExecutionInput = { - kind: "resource", - uri: "tasks.ts#seededTask", - args: { title: text("Milk"), owner: { kind: "resource", uri: "db.ts#db" } }, + kind: "object", + properties: { + title: text("Milk"), + owner: { kind: "instance", name: "db" }, + }, }; -const blank: ExecutionInput = { kind: "resource", uri: "tasks.ts#blank" }; +const blank: ExecutionInput = { kind: "object", properties: {} }; +const task: ExecutionInput = { kind: "instance", name: "root", output: "id" }; const row = ( cells: DatasetRow["cells"], @@ -98,10 +102,9 @@ describe("buildColumns", () => { expect(cols[1].type).toBe("Task"); }); - it("splits an expanded resource field into its head and one column per key", () => { + it("splits an expanded field into one column per key", () => { expect(columns(["1"]).map(c => [c.id, c.title, c.role, c.group])).toEqual([ ["0", "ticket", "whole", undefined], - ["1/", "task", "head", "task"], ["1/title", "title", "key", "task"], ["1/owner", "owner", "key", "task"], ["2", "info", "whole", "info"], @@ -148,9 +151,7 @@ describe("menuFieldId", () => { it("names the field for a field's own columns, and nothing for the rest", () => { const cols = columns(["1"]); expect(menuFieldId(byId(cols, "0"))).toBe("0"); - // An expanded resource field's head stands for the field... - expect(menuFieldId(byId(cols, "1/"))).toBe("1"); - // ...but the keys inside it, and the source, aren't fields. + // The keys inside an expanded field, and the source, aren't fields. expect(menuFieldId(byId(cols, "1/title"))).toBeUndefined(); expect(menuFieldId(byId(cols, "source"))).toBeUndefined(); expect(menuFieldId(undefined)).toBeUndefined(); @@ -164,17 +165,10 @@ describe("readPath", () => { expect(readPath(row({}), { fieldId: "1", key: "title" })).toBe("absent"); }); - it("reads a resource's argument and an object's property by key", () => { + it("reads an object's property by key", () => { expect( readPath(row({ "1": seeded }), { fieldId: "1", key: "title" }), ).toEqual(text("Milk")); - const object: ExecutionInput = { - kind: "object", - properties: { title: text("Eggs") }, - }; - expect( - readPath(row({ "1": object }), { fieldId: "1", key: "title" }), - ).toEqual(text("Eggs")); }); it("reads a typed-in object's property, wrapped as the value cell it would be", () => { @@ -186,7 +180,7 @@ describe("readPath", () => { ); }); - it("is n/a when the row's resource takes no such argument", () => { + it("is n/a when the row's cell has no such key", () => { expect(readPath(row({ "1": blank }), { fieldId: "1", key: "title" })).toBe( "n/a", ); @@ -201,9 +195,9 @@ describe("cellView", () => { expect(cellView(undefined, columns()[0])).toEqual({ kind: "loading" }); }); - it("previews a value, an empty cell, and a resource chip with its arguments", () => { + it("previews a value, an empty cell, and a resource instance as a chip", () => { const cols = columns(); - const r = row({ "0": text("Hi"), "1": seeded }); + const r = row({ "0": text("Hi"), "1": task }); // `ticket: string` is typed into in place, so it's an editor's view. expect(cellView(r, byId(cols, "0"))).toEqual({ kind: "edit", @@ -220,17 +214,13 @@ describe("cellView", () => { expect(cellView(row({}), byId(cols, "1"))).toEqual({ kind: "empty" }); expect(cellView(r, byId(cols, "1"))).toEqual({ kind: "chip", - text: '◆ seededTask(title: "Milk", owner: db)', + text: "◆ root.id", }); }); - it("names just the resource in an expanded head, with arguments in their own columns", () => { + it("splits an expanded object's keys into their own columns", () => { const cols = columns(["1"]); const r = row({ "1": seeded }); - expect(cellView(r, byId(cols, "1/"))).toEqual({ - kind: "chip", - text: "◆ seededTask", - }); expect(cellView(r, byId(cols, "1/title"))).toEqual({ kind: "text", text: '"Milk"', @@ -260,7 +250,7 @@ describe("cellView", () => { const object: ExecutionInput = { kind: "object", properties: { - db: { kind: "resource", uri: "db.ts#db" }, + db: { kind: "instance", name: "db" }, userId: text("u1"), }, }; diff --git a/src/client/components/dataset-grid.ts b/src/client/components/dataset-grid.ts index f60d0b3..7f67838 100644 --- a/src/client/components/dataset-grid.ts +++ b/src/client/components/dataset-grid.ts @@ -6,12 +6,11 @@ * it has, what each cell shows, and the paged cache rows are read through. * Pure and grid-agnostic — `DatasetView` maps it onto Glide Data Grid. * - * A column is a *path* into a row: a whole field, or one key inside it — a - * resource's argument, an object's property, or a property of a typed-in - * object value. The keys come from {@link DatasetRowsOverview}, found in the - * rows rather than the schema, and merged by name, so one `title` column - * spans every resource that takes a `title`. A path is also what sorting and - * filtering will name when they arrive. + * A column is a *path* into a row: a whole field, or one key inside it — an + * object's property, or a property of a typed-in object value. The keys come + * from {@link DatasetRowsOverview}, found in the rows rather than the schema, + * and merged by name. A path is also what sorting and filtering will name + * when they arrive. * * Some cells are typed into in place — see {@link editableCell}. An edit is * reported per cell and sent per row ({@link groupEdits}), and the page it @@ -32,13 +31,13 @@ import type { ExecutionInput, PropValue, } from "../../shared/types"; -import { previewCell, previewPropValue, resourceName } from "./dataset-preview"; +import { previewCell, previewPropValue } from "./dataset-preview"; import { DISCLOSURE_COLLAPSED, DISCLOSURE_EXPANDED } from "./grid-sprites"; /** Where a column reads from: a field, or one key one level inside it. */ export interface CellPath { fieldId: string; - /** A resource argument's or object property's name. */ + /** An object property's name. */ key?: string; } @@ -62,13 +61,8 @@ export type DatasetColumn = { base?: EditableBase; } & ( | { - /** - * `whole`: a field in one column. `head`: the column naming which - * resource fills an expanded field, beside one `key` column per - * argument. A field of objects has no head: its own cell would only - * ever read `{…}` next to the columns holding what's in it. - */ - role: "whole" | "head"; + /** A field in one column. */ + role: "whole"; path: CellPath; } | { role: "key"; path: Required } @@ -76,7 +70,7 @@ export type DatasetColumn = { | { role: "source" } ); -const WIDTHS = { whole: 240, head: 140, key: 160, source: 96 }; +const WIDTHS = { whole: 240, key: 160, source: 96 }; /** * A field's group header, which is its name: only fields with keys have one, @@ -131,17 +125,6 @@ export function buildColumns({ columns.push({ ...whole, group }); continue; } - if (fieldShape?.resource) { - columns.push({ - id: `${field.id}/`, - title: field.def.name, - type, - group, - role: "head", - path, - width: WIDTHS.head, - }); - } for (const key of fieldKeys) { columns.push({ id: `${field.id}/${key}`, @@ -170,9 +153,7 @@ export function buildColumns({ export function menuFieldId( column: DatasetColumn | undefined, ): string | undefined { - return column?.role === "whole" || column?.role === "head" - ? column.path.fieldId - : undefined; + return column?.role === "whole" ? column.path.fieldId : undefined; } /** @@ -213,8 +194,7 @@ export function groupHeader( /** * What a path reads in a row: the input there, `"absent"` when the row has - * nothing there, or `"n/a"` when the field's cell has no such key — a - * different resource, taking other arguments, say. + * nothing there, or `"n/a"` when the field's cell has no such key. * * A typed-in object's property is a `PropValue`, not an `ExecutionInput`; it * comes back wrapped as the value cell it would have been on its own. @@ -233,12 +213,7 @@ export function readPath( ? { kind: "value", value: value.properties[key] } : "n/a"; } - const inner = - cell.kind === "resource" - ? cell.args - : cell.kind === "object" - ? cell.properties - : undefined; + const inner = cell.kind === "object" ? cell.properties : undefined; return inner && Object.hasOwn(inner, key) ? inner[key] : "n/a"; } @@ -391,7 +366,10 @@ export function groupEdits(edits: readonly CellEdit[]): DatasetRowUpdate[] { return [...byRow.values()]; } -/** `row` with `update`'s cells set or cleared, as the server will store it. */ +/** + * `row` with `update`'s cells and resource instances set or cleared, as the + * server will store it. + */ export function applyUpdate( row: DatasetRow, update: DatasetRowUpdate, @@ -401,7 +379,18 @@ export function applyUpdate( if (cell === null) delete cells[fieldId]; else cells[fieldId] = cell; } - return { ...row, cells }; + if (!update.resources) return { ...row, cells }; + const resources = { ...row.resources }; + for (const [name, spec] of Object.entries(update.resources)) { + if (spec === null) delete resources[name]; + else resources[name] = spec; + } + const { resources: _, ...rest } = row; + return { + ...rest, + cells, + ...(Object.keys(resources).length > 0 && { resources }), + }; } /** How the grid should draw one cell. */ @@ -409,10 +398,10 @@ export type CellView = | { kind: "loading" } /** Nothing here: a sparse row. Drawn as a dim dash. */ | { kind: "empty" } - /** Not applicable: this row's resource doesn't take this argument. */ + /** Not applicable: this row's cell has no such key. */ | { kind: "n/a" } | { kind: "text"; text: string; tone?: "dim" | "link" } - /** A resource, drawn as a chip. */ + /** A resource instance, drawn as a chip. */ | { kind: "chip"; text: string } /** * A cell typed into in place (see {@link editableCell}): `value` is what @@ -434,7 +423,7 @@ function previewObject( return `{ ${entries .map( ([key, child]) => - `${key}: ${child.kind === "resource" ? `◆ ${previewCell(child)}` : previewCell(child)}`, + `${key}: ${child.kind === "instance" ? `◆ ${previewCell(child)}` : previewCell(child)}`, ) .join(", ")} }`; } @@ -451,22 +440,12 @@ function previewValueObject( } /** How an input is drawn in a column of `role`. */ -function inputView( - input: ExecutionInput, - role: "whole" | "head" | "key", -): CellView { +function inputView(input: ExecutionInput): CellView { switch (input.kind) { - case "resource": - return { - kind: "chip", - // An expanded field's head names the resource; the arguments have - // their own columns beside it. - text: `◆ ${role === "head" ? resourceName(input.uri) : previewCell(input)}`, - }; + case "instance": + return { kind: "chip", text: `◆ ${previewCell(input)}` }; case "object": - return role === "head" - ? { kind: "text", text: "{…}", tone: "dim" } - : { kind: "text", text: previewObject(input) }; + return { kind: "text", text: previewObject(input) }; default: return { kind: "text", @@ -504,7 +483,7 @@ export function cellView( } if (found === "absent") return { kind: "empty" }; if (found === "n/a") return { kind: "n/a" }; - return inputView(found, column.role); + return inputView(found); } } } diff --git a/src/client/components/dataset-preview.test.ts b/src/client/components/dataset-preview.test.ts index eb0f075..00742ae 100644 --- a/src/client/components/dataset-preview.test.ts +++ b/src/client/components/dataset-preview.test.ts @@ -27,17 +27,11 @@ describe("previewPropValue", () => { }); describe("previewCell", () => { - it("names a resource by its export, with an arguments summary", () => { + it("names a resource instance, and the output read from it", () => { + expect(previewCell({ kind: "instance", name: "db" })).toBe("db"); expect( - previewCell({ - kind: "resource", - uri: ".evalution/playground/tasks.ts#seededTask", - args: { - title: { kind: "value", value: { kind: "primitive", value: "Milk" } }, - owner: { kind: "resource", uri: "db.ts#db" }, - }, - }), - ).toBe('seededTask(title: "Milk", owner: db)'); + previewCell({ kind: "instance", name: "root", output: "taskId" }), + ).toBe("root.taskId"); }); it("truncates a long value", () => { @@ -53,7 +47,7 @@ describe("previewCell", () => { const cell = { kind: "object" as const, properties: { - db: { kind: "resource" as const, uri: "db.ts#db" }, + db: { kind: "instance" as const, name: "db" }, userId: { kind: "value" as const, value: { kind: "primitive" as const, value: "u1" }, diff --git a/src/client/components/dataset-preview.ts b/src/client/components/dataset-preview.ts index 071969b..ce232d5 100644 --- a/src/client/components/dataset-preview.ts +++ b/src/client/components/dataset-preview.ts @@ -58,18 +58,6 @@ export function resourceName(uri: string): string { return hash >= 0 ? uri.slice(hash + 1) : uri; } -/** A one-line summary of a resource's arguments: `title: "Milk", owner: db`. */ -export function previewArgs( - args: Record | undefined, -): string { - if (!args) return ""; - return truncate( - Object.entries(args) - .map(([name, input]) => `${name}: ${previewCell(input)}`) - .join(", "), - ); -} - /** A one-line preview of any cell. */ export function previewCell(input: ExecutionInput): string { switch (input.kind) { @@ -77,12 +65,10 @@ export function previewCell(input: ExecutionInput): string { return truncate(previewPropValue(input.value)); case "object": return "{…}"; - case "resource": { - const args = previewArgs(input.args); - return args - ? `${resourceName(input.uri)}(${args})` - : resourceName(input.uri); - } + case "instance": + return input.output === undefined + ? input.name + : `${input.name}.${input.output}`; case "dataset": return `= column ${input.field}`; case "input": diff --git a/src/client/components/dataset-row-editing.test.ts b/src/client/components/dataset-row-editing.test.ts index 6713819..812956d 100644 --- a/src/client/components/dataset-row-editing.test.ts +++ b/src/client/components/dataset-row-editing.test.ts @@ -10,11 +10,13 @@ import type { ResourceInfo, } from "../../shared/types"; import { + changesAnything, chosenResources, datasetInputSources, - rowCellChanges, + rowChanges, rowEditorState, } from "./dataset-row-editing"; +import { instanceUri } from "./pseudo-sources"; const def = (name: string, syntax: string): PropDefinition => ({ name, @@ -36,7 +38,6 @@ const SEEDED_ID: ResourceInfo = { scope: "run", parent: SEEDED.uri, }; -const BY_URI = new Map([SEEDED, SEEDED_ID].map(r => [r.uri, r])); const prompt = (overrides: Partial = {}): NormalizedPrompt => ({ @@ -85,62 +86,85 @@ describe("datasetInputSources", () => { }); }); -describe("rowCellChanges", () => { - const seededCell: ExecutionInput = { - kind: "resource", - uri: SEEDED_ID.uri, - args: { title: text("Buy milk") }, +describe("rowChanges", () => { + const taskCell: ExecutionInput = { + kind: "instance", + name: "task", + output: "id", }; + const task = { uri: SEEDED.uri, args: { title: text("Buy milk") } }; - it("commits nothing for an untouched row, however its cells were built", () => { + it("commits nothing for an untouched row, however it was built", () => { // Keys in another order than the editor would write them. - const cells: Record = { - "0": { - args: { title: text("Buy milk") }, - uri: SEEDED_ID.uri, - kind: "resource", - } as ExecutionInput, - "1": text("hello"), + const row = { + cells: { + "0": { output: "id", name: "task", kind: "instance" } as ExecutionInput, + "1": text("hello"), + }, + resources: { task: { args: task.args, uri: task.uri, receipt: "r" } }, }; - const state = rowEditorState(cells, BY_URI); - expect(rowCellChanges(["0", "1", "2"], state, cells, BY_URI)).toEqual({}); + const changes = rowChanges(["0", "1", "2"], rowEditorState(row), row); + expect(changes).toEqual({ cells: {} }); + expect(changesAnything(changes)).toBe(false); }); - it("folds a chosen resource and its arguments back into a cell", () => { - const cells = { "1": text("hello") }; - const state = rowEditorState(cells, BY_URI); - state.selections["0"] = { resources: { "": SEEDED_ID.uri } }; - // Arguments live under the root resource, as in the panel. - state.resourceArgs[SEEDED.uri] = { - title: { value: { kind: "primitive", value: "Buy milk" } }, + it("folds a chosen instance into a cell, and a new instance into resources", () => { + const row = { cells: { "1": text("hello") } }; + const state = rowEditorState(row); + state.selections["0"] = { resources: { "": instanceUri("task", "id") } }; + state.instances.task = { + uri: SEEDED.uri, + args: { title: { value: { kind: "primitive", value: "Buy milk" } } }, }; - expect(rowCellChanges(["0", "1"], state, cells, BY_URI)).toEqual({ - "0": seededCell, + expect(rowChanges(["0", "1"], state, row)).toEqual({ + cells: { "0": taskCell }, + resources: { task }, }); }); - it("clears a field emptied in its editor, and leaves other fields alone", () => { - const cells = { "0": seededCell, "1": text("hello") }; - const state = rowEditorState(cells, BY_URI); + it("clears a field emptied in its editor and removes a dropped instance, leaving the rest", () => { + const row = { + cells: { "0": taskCell, "1": text("hello") }, + resources: { task, other: { uri: SEEDED.uri } }, + }; + const state = rowEditorState(row); state.selections["1"] = { value: { kind: "primitive", value: "" } }; state.selections["0"] = {}; - expect(rowCellChanges(["1"], state, cells, BY_URI)).toEqual({ "1": null }); + delete state.instances.other; + expect(rowChanges(["1"], state, row)).toEqual({ + cells: { "1": null }, + resources: { other: null }, + }); }); }); describe("chosenResources", () => { - it("changes when a resource is chosen, not when a value is typed", () => { - const state = rowEditorState({ "0": text("a") }, BY_URI); + it("changes when a source is chosen or an instance renamed, not when a value is typed", () => { + const state = rowEditorState({ + cells: { "0": text("a") }, + resources: { task: { uri: SEEDED.uri } }, + }); const before = chosenResources(state); const typed = { ...state, selections: { "0": { value: { kind: "primitive", value: "ab" } } }, + instances: { + task: { + uri: SEEDED.uri, + args: { title: { value: { kind: "primitive", value: "x" } } }, + }, + }, } as typeof state; expect(chosenResources(typed)).toBe(before); const chosen = { ...state, - selections: { "0": { resources: { "": SEEDED.uri } } }, + selections: { "0": { resources: { "": instanceUri("task") } } }, }; expect(chosenResources(chosen)).not.toBe(before); + const renamed = { + ...state, + instances: { renamed: { uri: SEEDED.uri, args: {} } }, + }; + expect(chosenResources(renamed)).not.toBe(before); }); }); diff --git a/src/client/components/dataset-row-editing.ts b/src/client/components/dataset-row-editing.ts index f42b6bc..293402d 100644 --- a/src/client/components/dataset-row-editing.ts +++ b/src/client/components/dataset-row-editing.ts @@ -3,26 +3,33 @@ /** * Editing a dataset row in the details pane the way the execute panel edits - * its slots: a field's cell becomes a {@link SlotSelection}, resources and - * their arguments included, and folds back into a cell on commit. Pure. See - * `specs/datasets.md` §P.2. + * its slots: a field's cell becomes a {@link SlotSelection}, the row's + * resource instances become the Resources section's state, and both fold back + * into a row update on commit. Pure. See `specs/datasets.md` §P.2 and + * `specs/resource-instances.md` §G. */ import { committedCell } from "../../shared/dataset-cells"; import { matchKey } from "../../shared/dataset-fields"; import type { DatasetField, + DatasetRow, + DatasetRowUpdate, ExecutionInput, NormalizedPrompt, ResourceInfo, } from "../../shared/types"; import { fromExecutionInput, - type ResourceArgs, - resourceArgsFor, type Selections, + type SlotSelection, toExecutionInput, } from "./execution-input-state"; +import { + fromWireResources, + type InstanceSelections, + toWireResources, +} from "./run-resources-state"; /** The resources a dataset's fields can be filled from, borrowed from a prompt. */ export interface DatasetInputSources { @@ -76,66 +83,92 @@ export function datasetInputSources( }; } -/** A row's editor state: one selection per field id, and the shared arguments. */ +/** A row's editor state: one selection per field id, and the row's resource instances. */ export interface RowEditorState { selections: Selections; - resourceArgs: ResourceArgs; + instances: InstanceSelections; } -/** The editor state for a row's `cells`, as the panel restores its own. */ +/** The editor state for `row`, as the panel restores its own. */ export function rowEditorState( - cells: Record, - resourcesByUri: ReadonlyMap, + row: Pick, ): RowEditorState { const selections: Selections = {}; - const resourceArgs: ResourceArgs = {}; - for (const [fieldId, cell] of Object.entries(cells)) { - const recovered = fromExecutionInput(cell, resourcesByUri); - selections[fieldId] = recovered.selection; - Object.assign(resourceArgs, recovered.resourceArgs); + for (const [fieldId, cell] of Object.entries(row.cells)) { + selections[fieldId] = fromExecutionInput(cell); } - return { selections, resourceArgs }; + return { selections, instances: fromWireResources(row.resources) }; } /** - * The cells `state` would change, of the fields in `fieldIds`: field id → - * the new cell, or `null` to clear it. A field whose cell is unchanged has - * no entry, so an untouched row commits nothing. + * What `state` would change about `row`, of the fields in `fieldIds` and its + * resource instances: field id → the new cell, or `null` to clear it, and + * instance name → its new spec, or `null` to remove it. Anything unchanged + * has no entry, so an untouched row commits nothing. */ -export function rowCellChanges( +export function rowChanges( fieldIds: readonly string[], state: RowEditorState, - cells: Record, - resourcesByUri: ReadonlyMap, -): Record { - const resolveArgs = (uri: string) => - resourceArgsFor(uri, state.resourceArgs, resourcesByUri); - const changes: Record = {}; + row: Pick, +): Omit { + const cells: Record = {}; for (const fieldId of fieldIds) { - const input = toExecutionInput(state.selections[fieldId], resolveArgs); + const input = toExecutionInput(state.selections[fieldId]); const next = input?.kind === "value" ? committedCell(input.value) : (input ?? null); - if (canonical(next) !== canonical(cells[fieldId] ?? null)) { - changes[fieldId] = next; + if (canonical(next) !== canonical(row.cells[fieldId] ?? null)) { + cells[fieldId] = next; + } + } + + const before = row.resources ?? {}; + const after = toWireResources(state.instances) ?? {}; + const resources: NonNullable = {}; + for (const name of new Set([...Object.keys(before), ...Object.keys(after)])) { + const next = after[name] ?? null; + const { receipt: _, ...prev } = before[name] ?? { uri: "" }; + if (canonical(next) !== canonical(before[name] ? prev : null)) { + resources[name] = next; } } - return changes; + return { + cells, + ...(Object.keys(resources).length > 0 && { resources }), + }; } -/** Every resource chosen anywhere in `state`, as one comparable string. */ +/** Whether `changes` changes anything at all. */ +export function changesAnything( + changes: Omit, +): boolean { + return ( + Object.keys(changes.cells).length > 0 || + Object.keys(changes.resources ?? {}).length > 0 + ); +} + +/** + * Everything in `state` that isn't typed — every source chosen in a slot or + * an argument, and which instances exist under which names — as one + * comparable string. A change to it commits at once, since there's nothing + * more to type; anything else is a draft until blur. + */ export function chosenResources(state: RowEditorState): string { const resourcesOf = (selections: Selections) => Object.fromEntries( - Object.entries(selections).map(([k, s]) => [k, s.resources ?? {}]), + Object.entries(selections).flatMap(([k, s]: [string, SlotSelection]) => + s.resources && Object.keys(s.resources).length > 0 + ? [[k, s.resources]] + : [], + ), ); return canonical([ resourcesOf(state.selections), - Object.fromEntries( - Object.entries(state.resourceArgs).map(([uri, args]) => [ - uri, - resourcesOf(args), - ]), - ), + Object.entries(state.instances).map(([name, instance]) => [ + name, + instance.uri, + resourcesOf(instance.args), + ]), ]); } diff --git a/src/client/components/eval-bindings.test.ts b/src/client/components/eval-bindings.test.ts index 5a20e02..645ac88 100644 --- a/src/client/components/eval-bindings.test.ts +++ b/src/client/components/eval-bindings.test.ts @@ -17,7 +17,7 @@ const obj = (name: string, properties: PropDefinition[]): PropDefinition => ({ type: { kind: "object", syntax: `{…}`, properties }, }); -const db: ExecutionInput = { kind: "resource", uri: "db.ts#db" }; +const db: ExecutionInput = { kind: "instance", name: "db" }; const text = (value: string): ExecutionInput => ({ kind: "value", value: { kind: "primitive", value }, @@ -56,11 +56,17 @@ describe("prefillBindings", () => { stored: { functionInputs: { title: text("hi") }, executeInputs: { db }, + resources: { + db: { uri: "db.ts#db", receipt: "r" }, + unused: { uri: "db.ts#other" }, + }, }, }); + // The instance a taken binding names comes along; the rest don't. expect(inputs).toEqual({ functionInputs: { title: text("hi") }, executeInputs: { db }, + resources: { db: { uri: "db.ts#db" } }, }); }); diff --git a/src/client/components/execution-input-state.test.ts b/src/client/components/execution-input-state.test.ts index 5e2ff7c..c8af473 100644 --- a/src/client/components/execution-input-state.test.ts +++ b/src/client/components/execution-input-state.test.ts @@ -2,223 +2,95 @@ // Copyright (c) 2026 Alexander Corrado import { describe, expect, it } from "vitest"; -import type { ExecutionInput, ResourceInfo } from "../../shared/types"; +import type { ExecutionInput } from "../../shared/types"; import { fromExecutionInput, - type ResourceArgs, - resourceArgsFor, SELF, type SlotSelection, toExecutionInput, } from "./execution-input-state"; +import { columnUri, instanceUri } from "./pseudo-sources"; const strVal = (value: string): ExecutionInput => ({ kind: "value", value: { kind: "primitive", value }, }); -describe("toExecutionInput (specs/resource-arguments.md §C, §J)", () => { - it("emits an input byte-identical to before arguments existed for a resource with no arguments", () => { - const selection: SlotSelection = { resources: { [SELF]: "pg.ts#db" } }; - expect(toExecutionInput(selection)).toEqual({ - kind: "resource", - uri: "pg.ts#db", - }); - // Same with a `resolveArgs` present but reporting nothing for this uri. - expect(toExecutionInput(selection, () => undefined)).toEqual({ - kind: "resource", - uri: "pg.ts#db", - }); - }); - - it("attaches args built by resolveArgs onto the resource node", () => { - const selection: SlotSelection = { resources: { [SELF]: "pg.ts#seeded" } }; - const args = { title: strVal("Todo app") }; - expect(toExecutionInput(selection, () => args)).toEqual({ - kind: "resource", - uri: "pg.ts#seeded", - args, - }); +describe("toExecutionInput", () => { + it("references an instance, or one of its outputs, chosen for the slot", () => { + expect( + toExecutionInput({ resources: { [SELF]: instanceUri("db") } }), + ).toEqual({ kind: "instance", name: "db" }); + expect( + toExecutionInput({ + resources: { [SELF]: instanceUri("root", "taskId") }, + }), + ).toEqual({ kind: "instance", name: "root", output: "taskId" }); }); - it("attaches args to a resource nested inside an object", () => { + it("grafts an instance into one field of a typed-in object", () => { const selection: SlotSelection = { - resources: { db: "pg.ts#seeded" }, + value: { + kind: "object", + properties: { workspaceId: { kind: "primitive", value: "ws_1" } }, + }, + resources: { db: instanceUri("db") }, }; - const args = { title: strVal("x") }; - const result = toExecutionInput(selection, uri => - uri === "pg.ts#seeded" ? args : undefined, - ); - expect(result).toEqual({ + expect(toExecutionInput(selection)).toEqual({ kind: "object", - properties: { db: { kind: "resource", uri: "pg.ts#seeded", args } }, - }); - }); -}); - -describe("resourceArgsFor (§J)", () => { - const resourcesByUri = new Map([ - [ - "pg.ts#seeded", - { - uri: "pg.ts#seeded", - label: "Seeded task", - scope: "run", - parameters: [ - { - name: "title", - type: { kind: "primitive", syntax: "string", base: "string" }, - optional: false, - }, - { - name: "status", - type: { kind: "primitive", syntax: "string", base: "string" }, - optional: false, - }, - ], - }, - ], - [ - "pg.ts#titleGen", - { uri: "pg.ts#titleGen", label: "Title generator", scope: "run" }, - ], - ]); - - it("builds one ExecutionInput per declared parameter, in declaration order", () => { - const resourceArgs: ResourceArgs = { - "pg.ts#seeded": { - title: { value: { kind: "primitive", value: "Todo app" } }, - status: { value: { kind: "primitive", value: "triaged" } }, + properties: { + workspaceId: strVal("ws_1"), + db: { kind: "instance", name: "db" }, }, - }; - const args = resourceArgsFor("pg.ts#seeded", resourceArgs, resourcesByUri); - expect(Object.keys(args!)).toEqual(["title", "status"]); - expect(args!.title).toEqual(strVal("Todo app")); + }); }); - it("returns undefined for a resource with no declared parameters", () => { + it("leaves out a catalog URI, which only stands for something once adopted", () => { expect( - resourceArgsFor("pg.ts#titleGen", {}, resourcesByUri), + toExecutionInput({ resources: { [SELF]: "pg.ts#db" } }), ).toBeUndefined(); - }); - - it("returns undefined when no argument has anything typed in or chosen", () => { - expect(resourceArgsFor("pg.ts#seeded", {}, resourcesByUri)).toBeUndefined(); - }); - - it("resolves an argument that is itself a resource, recursively", () => { - const resourceArgs: ResourceArgs = { - "pg.ts#seeded": { - title: { resources: { [SELF]: "pg.ts#titleGen" } }, - }, - }; - const args = resourceArgsFor("pg.ts#seeded", resourceArgs, resourcesByUri); - expect(args!.title).toEqual({ kind: "resource", uri: "pg.ts#titleGen" }); + expect( + toExecutionInput({ + value: { kind: "primitive", value: "x" }, + resources: { db: "pg.ts#db" }, + }), + ).toEqual(strVal("x")); }); }); -describe("fromExecutionInput (§I, §J)", () => { - it("restores a stored selection made before arguments existed, unchanged", () => { - const input: ExecutionInput = { kind: "resource", uri: "pg.ts#db" }; - const recovered = fromExecutionInput(input); - expect(recovered.selection).toEqual({ resources: { [SELF]: "pg.ts#db" } }); - expect(recovered.resourceArgs).toEqual({}); - }); - - it("recovers a resource's args into the ResourceArgs map, keyed by its uri", () => { - const input: ExecutionInput = { - kind: "resource", - uri: "pg.ts#seeded", - args: { title: strVal("Todo app") }, - }; - const recovered = fromExecutionInput(input); - expect(recovered.selection).toEqual({ - resources: { [SELF]: "pg.ts#seeded" }, - }); - expect(recovered.resourceArgs["pg.ts#seeded"]).toEqual({ - title: { value: { kind: "primitive", value: "Todo app" } }, - }); - }); - - it("recovers args from a resource nested arbitrarily deep in an object", () => { +describe("fromExecutionInput", () => { + it("restores instance, column and value inputs as the panel chose them", () => { const input: ExecutionInput = { kind: "object", properties: { - list_tasks: { - kind: "object", - properties: { - db: { - kind: "resource", - uri: "pg.ts#seeded", - args: { title: strVal("x") }, - }, - }, - }, + db: { kind: "instance", name: "db" }, + title: { kind: "dataset", field: "0" }, + workspaceId: strVal("ws_1"), }, }; - const recovered = fromExecutionInput(input); - expect(recovered.selection.resources).toEqual({ - "list_tasks.db": "pg.ts#seeded", - }); - expect(recovered.resourceArgs["pg.ts#seeded"]).toEqual({ - title: { value: { kind: "primitive", value: "x" } }, - }); - }); - - it("recovers a resource argument that is itself a resource with its own args", () => { - const input: ExecutionInput = { - kind: "resource", - uri: "pg.ts#seeded", - args: { - owner: { - kind: "resource", - uri: "pg.ts#owner", - args: { name: strVal("Ada") }, - }, + const selection = fromExecutionInput(input); + expect(selection).toEqual({ + value: { + kind: "object", + properties: { workspaceId: { kind: "primitive", value: "ws_1" } }, }, - }; - const recovered = fromExecutionInput(input); - expect(recovered.resourceArgs["pg.ts#seeded"].owner).toEqual({ - resources: { [SELF]: "pg.ts#owner" }, + resources: { db: instanceUri("db"), title: columnUri("0") }, }); - expect(recovered.resourceArgs["pg.ts#owner"]).toEqual({ - name: { value: { kind: "primitive", value: "Ada" } }, + expect(toExecutionInput(selection)).toEqual({ + kind: "object", + properties: { + workspaceId: strVal("ws_1"), + db: { kind: "instance", name: "db" }, + title: { kind: "dataset", field: "0" }, + }, }); }); -}); - -describe("round trip (toExecutionInput ∘ resourceArgsFor, then fromExecutionInput)", () => { - it("restores a shared-by-uri binding losslessly", () => { - const resourcesByUri = new Map([ - [ - "pg.ts#seeded", - { - uri: "pg.ts#seeded", - label: "Seeded task", - scope: "run", - parameters: [ - { - name: "title", - type: { kind: "primitive", syntax: "string", base: "string" }, - optional: false, - }, - ], - }, - ], - ]); - const resourceArgs: ResourceArgs = { - "pg.ts#seeded": { - title: { value: { kind: "primitive", value: "Todo app" } }, - }, - }; - const selection: SlotSelection = { resources: { [SELF]: "pg.ts#seeded" } }; - const wire = toExecutionInput(selection, uri => - resourceArgsFor(uri, resourceArgs, resourcesByUri), - )!; - const recovered = fromExecutionInput(wire); - - expect(recovered.selection).toEqual(selection); - expect(recovered.resourceArgs).toEqual(resourceArgs); + it("restores an inline resource stored before instances existed as an empty slot", () => { + const legacy = { + kind: "resource", + uri: "pg.ts#db", + } as unknown as ExecutionInput; + expect(fromExecutionInput(legacy)).toEqual({}); }); }); diff --git a/src/client/components/execution-input-state.ts b/src/client/components/execution-input-state.ts index e46d3f9..a2e8f15 100644 --- a/src/client/components/execution-input-state.ts +++ b/src/client/components/execution-input-state.ts @@ -6,7 +6,7 @@ import type { InputLayout, NormalizedPrompt, PropValue, - ResourceInfo, + RunResources, } from "../../shared/types"; import { pseudoInput, pseudoUriOf } from "./pseudo-sources"; @@ -17,6 +17,8 @@ import { pseudoInput, pseudoUriOf } from "./pseudo-sources"; export interface StoredInputs { functionInputs?: Record; executeInputs?: Record; + /** The run's named resource instances. See `specs/resource-instances.md`. */ + resources?: RunResources; /** The user's explicit layout choice, by slot path. Absent until they touch the toggle. */ layout?: { functionSlots?: Record; @@ -51,15 +53,17 @@ export function readStoredInputs( * * Kept apart from {@link ExecutionInput} because editing has a state the wire * format has no reason to carry: a hand-edited value *and* a set of nested - * slots that have been handed to a resource instead. `toExecutionInput` folds + * slots that have been handed to a source instead. `toExecutionInput` folds * the two together at submit time. */ export interface SlotSelection { /** The value typed into the editor, if any. */ value?: PropValue; /** - * Resource chosen for a slot, keyed by the slot's dotted path relative to - * this top-level slot (`''` for the slot itself, `'db'` for a nested field). + * Source chosen for a slot — a pseudo-source URI naming one of the run's + * resource instances, a column, or another slot (see `pseudo-sources.ts`) + * — keyed by the slot's dotted path relative to this top-level slot (`''` + * for the slot itself, `'db'` for a nested field). * * Nested entries are what let a resource fill one field of an otherwise * hand-edited object — `toolsContext`'s `db` beside its typed-in ids. @@ -70,47 +74,35 @@ export interface SlotSelection { /** Editor state for one set of slots, keyed by slot (or argument) name. */ export type Selections = Record; -/** - * Argument editor state, held once per resource `uri` and shared by every - * selection of it — the panel-level counterpart of `ExecutionInput.args` - * (`specs/resource-arguments.md` §I, §J). - * - * Kept as a sibling map rather than per slot because arguments are a property - * of the *resource*, the same reason `ResourceInfo.parameters` is: picking - * `seededTask.taskId` for one slot and `seededTask.title` for another edits - * one set of arguments, not two. - */ -export type ResourceArgs = Record; - /** The path key used for the top-level slot itself. */ export const SELF = ""; /** * Fold a slot's editor state into the input to send. * - * A resource chosen for the slot itself wins outright. Otherwise nested + * A source chosen for the slot itself wins outright. Otherwise nested * choices are grafted onto the typed-in value, producing the `object` variant - * only where one is actually needed — a slot with no nested resources stays a + * only where one is actually needed — a slot with no nested sources stays a * plain `value`, which is also what makes it restore exactly on replay. * - * @param selection - This slot's own editor state. - * @param resolveArgs - Given a chosen resource's `uri`, returns the `args` it - * should carry (built from {@link ResourceArgs} — see - * {@link resourceArgsFor}), or `undefined` if it takes none. Recursive: an - * argument that is itself a resource gets its own `args` resolved the same - * way, which is how `resourceArgsFor` calls back into this function. + * A chosen URI that isn't a pseudo-source (a catalog resource, which a host + * turns into an instance as it's picked) stands for nothing on the wire and + * is left out. */ export function toExecutionInput( selection: SlotSelection | undefined, - resolveArgs?: (uri: string) => Record | undefined, ): ExecutionInput | undefined { if (!selection) return undefined; const resources = selection.resources ?? {}; const own = resources[SELF]; - if (own) return resourceInput(own, resolveArgs); + if (own) return pseudoInput(own); - const nested = Object.entries(resources).filter(([path]) => path !== SELF); + const nested = Object.entries(resources).flatMap(([path, uri]) => { + if (path === SELF) return []; + const input = pseudoInput(uri); + return input ? [[path, input] as const] : []; + }); if (nested.length === 0) { return selection.value ? { kind: "value", value: selection.value } @@ -119,81 +111,15 @@ export function toExecutionInput( // Build the object tree from the value, then overlay each nested choice at // its path. - const root: ExecutionInput = selection.value + let out: ExecutionInput = selection.value ? { kind: "value", value: selection.value } : { kind: "object", properties: {} }; - - let out: ExecutionInput = root; - for (const [path, uri] of nested) { - out = overlay(out, path.split("."), resourceInput(uri, resolveArgs)); + for (const [path, input] of nested) { + out = overlay(out, path.split("."), input); } return out; } -/** - * A `{ kind: "resource" }` node, with `args` attached when `resolveArgs` has - * any — or, for a column or prompt slot chosen as a source, the `dataset` or - * `input` node it stands for. - */ -function resourceInput( - uri: string, - resolveArgs?: (uri: string) => Record | undefined, -): ExecutionInput { - const pseudo = pseudoInput(uri); - if (pseudo) return pseudo; - const args = resolveArgs?.(uri); - return args ? { kind: "resource", uri, args } : { kind: "resource", uri }; -} - -/** - * Builds the `args` a reference to `uri` should carry: one entry per its - * `ResourceInfo.parameters`, taken from `resourceArgs[uri]` and folded - * through {@link toExecutionInput} the same as any other slot — recursively, - * so an argument that is itself a resource gets its own `args` the same way. - * - * Meant to be passed (partially applied over `resourceArgs` and - * `resourcesByUri`) as {@link toExecutionInput}'s `resolveArgs`. - * - * @param uri - The resource whose arguments to build. - * @param resourceArgs - The panel's whole argument-editor state. - * @param resourcesByUri - Every resource in scope, for parameter lookup. - */ -export function resourceArgsFor( - uri: string, - resourceArgs: ResourceArgs, - resourcesByUri: ReadonlyMap, -): Record | undefined { - // `uri` may name an output value (`seededTask.taskId`) rather than the - // resource itself — only the root carries `parameters`, and §I's "one - // binding per resource" means the args live (and are looked up) under the - // *root's* uri regardless of which of its values was actually picked. - const root = rootResourceUri(uri, resourcesByUri); - const params = resourcesByUri.get(root)?.parameters; - if (!params || params.length === 0) return undefined; - - const selections = resourceArgs[root] ?? {}; - const entries = params.flatMap(param => { - const input = toExecutionInput(selections[param.name], childUri => - resourceArgsFor(childUri, resourceArgs, resourcesByUri), - ); - return input ? [[param.name, input] as const] : []; - }); - return entries.length > 0 ? Object.fromEntries(entries) : undefined; -} - -/** - * `uri` itself, or the resource it's an output value of (`ResourceInfo.parent`) - * — the key {@link ResourceArgs} and `ResourceArgsContext.resourceSlots` / - * `.claimed` are all addressed by, so that choosing `seededTask.taskId` for - * one slot and `seededTask.title` for another shares one set of arguments. - */ -export function rootResourceUri( - uri: string, - resourcesByUri: ReadonlyMap, -): string { - return resourcesByUri.get(uri)?.parent ?? uri; -} - /** * Replace whatever sits at `path` inside `input` with `replacement`, * converting the `value` variant into the nestable `object` variant only along @@ -232,73 +158,26 @@ function propertiesOf(input: ExecutionInput): Record { return {}; } -/** What {@link fromExecutionInput} recovers from a stored or recorded input. */ -export interface RecoveredInput { - /** This slot's own editor state — the direct inverse of the old `fromExecutionInput` result. */ - selection: SlotSelection; - /** - * Argument editor state for every resource (at any depth) the input named - * with `args` — merge into the panel's {@link ResourceArgs}. Empty when - * nothing in the input had arguments. - */ - resourceArgs: ResourceArgs; -} - /** * Recover editor state from a stored or recorded input. * * The inverse of {@link toExecutionInput}, and lossless for what the panel * actually shows: a `value` comes back in its original `PropValue` form (a * template stays a template rather than the flattened string a materialized - * value would have left behind), a resource comes back as a chip, and a - * resource's `args` come back as {@link ResourceArgs} entries the caller - * merges into the panel's own map — restoring the shared-by-uri binding - * `specs/resource-arguments.md` §I describes, not a copy per reference. - * - * @param resourcesByUri - When given, a recovered resource's `args` are - * keyed by its *root* uri ({@link rootResourceUri}) rather than by - * whichever of its values the reference actually named — the same - * resolution {@link resourceArgsFor} does, and necessary for the same - * reason: `seededTask.taskId`'s recorded `args` belong to `seededTask`, - * not to a key nothing else will ever look up. Omit only when no resource - * in the tree could possibly be a value reference (e.g. recovering a - * resource's own already-rooted argument, as this function does for - * itself below). + * value would have left behind), and an instance, column or slot reference + * comes back as the chip it was chosen as. Anything else — an inline + * `resource` node stored before instances existed — comes back empty rather + * than failing. */ export function fromExecutionInput( input: ExecutionInput | undefined, - resourcesByUri?: ReadonlyMap, -): RecoveredInput { - if (!input) return { selection: {}, resourceArgs: {} }; +): SlotSelection { + if (!input) return {}; const resources: Record = {}; let value: PropValue | undefined; - const resourceArgs: ResourceArgs = {}; - - const captureArgs = ( - uri: string, - args: Record | undefined, - ) => { - if (!args) return; - const selections: Selections = {}; - for (const [paramName, argInput] of Object.entries(args)) { - const recovered = fromExecutionInput(argInput, resourcesByUri); - selections[paramName] = recovered.selection; - Object.assign(resourceArgs, recovered.resourceArgs); - } - const root = resourcesByUri ? rootResourceUri(uri, resourcesByUri) : uri; - // Last-writer-wins where two references to `root` disagree — harmless in - // practice (see the doc comment above) and never happens for a binding - // this panel itself produced, since §I holds one per uri. - resourceArgs[root] = selections; - }; - const walk = (node: ExecutionInput, prefix: string) => { switch (node.kind) { - case "resource": - resources[prefix] = node.uri; - captureArgs(node.uri, node.args); - break; case "value": if (prefix === SELF) value = node.value; else setAtPath(prefix, node.value); @@ -310,9 +189,13 @@ export function fromExecutionInput( break; case "dataset": case "input": - // A column or another slot comes back as the chip it was chosen as. + case "instance": + // A column, another slot or an instance comes back as the chip it + // was chosen as. resources[prefix] = pseudoUriOf(node)!; break; + default: + break; } }; @@ -337,10 +220,7 @@ export function fromExecutionInput( walk(input, SELF); return { - selection: { - ...(value ? { value } : {}), - ...(Object.keys(resources).length > 0 ? { resources } : {}), - }, - resourceArgs, + ...(value ? { value } : {}), + ...(Object.keys(resources).length > 0 ? { resources } : {}), }; } diff --git a/src/client/components/named-inputs.test.ts b/src/client/components/named-inputs.test.ts index 56319b2..1ebadd1 100644 --- a/src/client/components/named-inputs.test.ts +++ b/src/client/components/named-inputs.test.ts @@ -88,30 +88,6 @@ describe("toCells", () => { expect(skipped).toEqual([{ name: "verbose", reason: "no-match" }]); }); - it("strips receipts, at any depth", () => { - const { cells } = toCells( - [ - { - def: def("taskId"), - input: { - kind: "resource", - uri: TASK_URI, - receipt: { id: "tsk_abc123" }, - args: { - owner: { kind: "resource", uri: DB_URI, receipt: "r2" }, - }, - }, - }, - ], - fields(def("taskId")), - ); - expect(cells["0"]).toEqual({ - kind: "resource", - uri: TASK_URI, - args: { owner: { kind: "resource", uri: DB_URI } }, - }); - }); - it("treats a second input for an already-filled field as the same value", () => { const { cells, matched, skipped } = toCells( [ @@ -149,7 +125,7 @@ describe("fromTrace", () => { it("types execute inputs from the recorded execute definitions", () => { const ctx: ExecutionInput = { kind: "object", - properties: { db: { kind: "resource", uri: DB_URI }, userId: text("u") }, + properties: { db: { kind: "instance", name: "db" }, userId: text("u") }, }; const inputs = fromTrace({ id: "x", @@ -218,19 +194,24 @@ describe("fromTrace", () => { }); describe("toPanel", () => { - it("skips a resource cell whose uri isn't in the prompt's scope", () => { + it("drops an instance whose resource isn't in the prompt's scope, and skips what references it", () => { const p = prompt(); - const { functionInputs, skipped } = toPanel( + const { functionInputs, resources, skipped } = toPanel( [ { def: def("ticket"), input: text("hi") }, { def: def("db", "Db", "opaque"), - input: { kind: "resource", uri: "elsewhere.playground.ts#db" }, + input: { kind: "instance", name: "far" }, }, ], p, + { + far: { uri: "elsewhere.playground.ts#db" }, + near: { uri: DB_URI, receipt: "stripped" }, + }, ); expect(functionInputs).toEqual({ ticket: text("hi") }); + expect(resources).toEqual({ near: { uri: DB_URI } }); expect(skipped).toEqual([{ name: "db", reason: "resource-out-of-scope" }]); }); @@ -257,17 +238,13 @@ describe("round trip", () => { kind: "value", value: { kind: "template", value: ["Order ", { expr: "id" }] }, } as ExecutionInput, - db: { - kind: "resource", - uri: TASK_URI, - args: { title: text("Buy milk") }, - } as ExecutionInput, + db: { kind: "instance", name: "task" } as ExecutionInput, }, executeInputs: { toolsContext: { kind: "object", properties: { - db: { kind: "resource", uri: DB_URI }, + db: { kind: "instance", name: "db" }, userId: text("u1"), }, } as ExecutionInput, @@ -278,12 +255,16 @@ describe("round trip", () => { id: i.toString(36), ...f, })); + const resources = { + task: { uri: TASK_URI, args: { title: text("Buy milk") } }, + db: { uri: DB_URI }, + }; const { cells, skipped } = toCells(fromPanel(p, request), schema); expect(skipped).toEqual([]); - const row = { id: "r", cells, createdAt: 0 }; - const back = toPanel(fromRow(dataset(schema), row), p); - expect(back).toEqual({ ...request, skipped: [] }); + const row = { id: "r", cells, resources, createdAt: 0 }; + const back = toPanel(fromRow(dataset(schema), row), p, row.resources); + expect(back).toEqual({ ...request, resources, skipped: [] }); }); }); diff --git a/src/client/components/named-inputs.ts b/src/client/components/named-inputs.ts index 2917be8..2528ea5 100644 --- a/src/client/components/named-inputs.ts +++ b/src/client/components/named-inputs.ts @@ -24,6 +24,7 @@ import type { NormalizedPrompt, PromptID, PropDefinition, + RunResources, } from "../../shared/types"; /** An input together with the definition that says what it is. */ @@ -55,6 +56,8 @@ export interface SkippedInput { export interface PartialExecuteRequest { functionInputs: Record; executeInputs: Record; + /** The run's named resource instances, which slots may reference. */ + resources?: RunResources; } /** @@ -167,7 +170,9 @@ export function hasRecordedInputs(recorded: PromptID | undefined): boolean { /** * Cells for a row of a dataset with `fields`: each input lands in the field - * that matches it, receipts stripped. An input matching no field is skipped; + * that matches it. A cell references the row's resource instances by name; + * the instances themselves travel beside the cells, as the row's + * `resources` (receipts stripped — see {@link rowResources}). An input matching no field is skipped; * a second input matching an already-filled field is the same value arriving * twice (a function and an execute parameter of one name and type) and is * neither matched again nor skipped. @@ -191,22 +196,32 @@ export function toCells( continue; } if (field.id in cells) continue; - cells[field.id] = stripReceipts(input); + cells[field.id] = input; matched++; } return { cells, matched, skipped }; } -/** Every resource `uri` an input names, at any depth (arguments included). */ -function resourceUris(input: ExecutionInput): string[] { +/** + * The resource instances a dataset row keeps: `resources` without receipts, + * or `undefined` when there are none. A receipt makes a run reconstruct a + * *past* instance; a row is a recipe for new runs. + */ +export function rowResources( + resources: RunResources | undefined, +): RunResources | undefined { + return resources && Object.keys(resources).length > 0 + ? stripReceipts(resources) + : undefined; +} + +/** Every instance name an input references, at any depth. */ +function instanceNames(input: ExecutionInput): string[] { switch (input.kind) { - case "resource": - return [ - input.uri, - ...Object.values(input.args ?? {}).flatMap(resourceUris), - ]; + case "instance": + return [input.name]; case "object": - return Object.values(input.properties).flatMap(resourceUris); + return Object.values(input.properties).flatMap(instanceNames); default: return []; } @@ -215,17 +230,25 @@ function resourceUris(input: ExecutionInput): string[] { /** * Panel inputs for `prompt`: each input fills every slot it matches — a * function and an execute parameter of one name and type are one value, so - * both get it. An input naming a resource outside this prompt's - * `inputSources` is skipped rather than filled: the chip couldn't resolve, - * and would only fail at Run. + * both get it. `resources` come along, receipts stripped, except an instance + * of a resource outside this prompt's `inputSources`: it couldn't be created + * here, so it's dropped, and an input referencing it is skipped rather than + * filled — it would only fail at Run. */ export function toPanel( inputs: NamedInputs, prompt: NormalizedPrompt, + resources?: RunResources, ): PartialExecuteRequest & { skipped: SkippedInput[] } { const inScope = new Set( (prompt.inputSources?.resources ?? []).map(r => r.uri), ); + const kept: RunResources = {}; + const dropped = new Set(); + for (const [name, spec] of Object.entries(rowResources(resources) ?? {})) { + if (inScope.has(spec.uri)) kept[name] = spec; + else dropped.add(name); + } const functionInputs: Record = {}; const executeInputs: Record = {}; const skipped: SkippedInput[] = []; @@ -238,15 +261,19 @@ export function toPanel( skipped.push({ name: def.name, reason: "no-match" }); continue; } - if (resourceUris(input).some(uri => !inScope.has(uri))) { + if (instanceNames(input).some(name => dropped.has(name))) { skipped.push({ name: def.name, reason: "resource-out-of-scope" }); continue; } - const clean = stripReceipts(input); - if (fnDef) functionInputs[fnDef.name] = clean; - if (execDef) executeInputs[execDef.name] = clean; + if (fnDef) functionInputs[fnDef.name] = input; + if (execDef) executeInputs[execDef.name] = input; } - return { functionInputs, executeInputs, skipped }; + return { + functionInputs, + executeInputs, + ...(Object.keys(kept).length > 0 && { resources: kept }), + skipped, + }; } /** A schema for "New dataset" from a set of inputs: one field per distinct input. */ @@ -323,6 +350,8 @@ export type PanelFillSource = { description: string } & ( export interface PanelFill { functionInputs: Record; executeInputs: Record; + /** Resource instances the inputs reference, receipts stripped. */ + resources?: RunResources; /** Shown in the notice: "trace 3f2a1b2c…" / "Support tickets, row 4". */ from: PanelFillSource; /** What didn't fit, named in the notice. */ @@ -333,18 +362,27 @@ export interface PanelFill { let fillSeq = 0; -/** A {@link PanelFill} for `prompt` from `inputs`, with a fresh nonce. */ +/** + * A {@link PanelFill} for `prompt` from `inputs` and the resource instances + * they came with — a row's, or a trace's recorded ones — with a fresh nonce. + */ export function panelFill( inputs: NamedInputs, prompt: NormalizedPrompt, from: PanelFillSource, + resources?: RunResources, ): PanelFill { - const { functionInputs, executeInputs, skipped } = toPanel(inputs, prompt); + const { functionInputs, executeInputs, skipped, ...rest } = toPanel( + inputs, + prompt, + resources, + ); // Time-based as well as sequential, so a nonce minted after a reload never // collides with one an earlier page load already applied. return { functionInputs, executeInputs, + ...(rest.resources && { resources: rest.resources }), from, skipped, nonce: Date.now() * 1000 + (fillSeq++ % 1000), diff --git a/src/client/components/pseudo-sources.test.ts b/src/client/components/pseudo-sources.test.ts index 22844f7..a42cb02 100644 --- a/src/client/components/pseudo-sources.test.ts +++ b/src/client/components/pseudo-sources.test.ts @@ -7,13 +7,19 @@ import { fromExecutionInput, toExecutionInput } from "./execution-input-state"; import { checkParameterSources, columnUri, + describeInstanceSource, describePseudoSource, + INSTANCES_GROUP, inputUri, + instanceUri, NEW_COLUMN_URI, + NEW_GROUP, + parseInstanceUri, pseudoInput, + withInstanceSources, withPseudoSources, } from "./pseudo-sources"; -import { computeClaims } from "./resource-args-context"; +import { buildSourceTree } from "./source-tree"; const def = (name: string, syntax: string, base = syntax): PropDefinition => ({ name, @@ -35,8 +41,10 @@ describe("pseudo-source URIs", () => { for (const input of [ { kind: "dataset", field: "3" }, { kind: "input", half: "execute", path: "ctx.db" }, + { kind: "instance", name: "root" }, + { kind: "instance", name: "root", output: "taskId" }, ] as const) { - const { selection } = fromExecutionInput(input); + const selection = fromExecutionInput(input); expect(toExecutionInput(selection)).toEqual(input); } expect(pseudoInput(columnUri("a"))).toEqual({ @@ -47,24 +55,74 @@ describe("pseudo-source URIs", () => { expect(pseudoInput(NEW_COLUMN_URI)).toBeUndefined(); }); - it("are never claimed as a shared instance", () => { - const claims = computeClaims( - [ - { - path: "fn.a", - label: "a", - selection: { resources: { "": columnUri("0") } }, - }, - { - path: "fn.b", - label: "b", - selection: { resources: { "": columnUri("0") } }, - }, - ], - {}, - new Map(), + it("name an instance and an output of it", () => { + expect(parseInstanceUri(instanceUri("root"))).toEqual({ name: "root" }); + expect(parseInstanceUri(instanceUri("root", "taskId"))).toEqual({ + name: "root", + output: "taskId", + }); + expect(parseInstanceUri(columnUri("0"))).toBeUndefined(); + }); +}); + +describe("withInstanceSources", () => { + const catalog: ResourceInfo[] = [ + { uri: "t.ts#task", label: "Seeded task", scope: "run" }, + { + uri: "t.ts#task.id", + label: "Task ID", + scope: "run", + parent: "t.ts#task", + }, + { uri: "t.ts#other", label: "Other", scope: "run", group: ["Misc"] }, + ]; + const sources = { + resources: catalog, + functionSlots: { taskId: ["t.ts#task.id"], other: ["t.ts#other"] }, + executeSlots: {}, + }; + const instances = { root: { uri: "t.ts#task" }, child: { uri: "t.ts#task" } }; + + it("offers each instance first wherever its resource or output fits", () => { + const withInstances = withInstanceSources(sources, instances); + expect(withInstances.functionSlots.taskId).toEqual([ + instanceUri("root", "id"), + instanceUri("child", "id"), + "t.ts#task.id", + ]); + expect(withInstances.functionSlots.other).toEqual(["t.ts#other"]); + expect( + withInstances.resources.filter(r => r.group?.[0] === INSTANCES_GROUP), + ).toHaveLength(4); + }); + + it("lists instances first and the catalog under New, opened in place where no instance fits", () => { + const { resources, functionSlots } = withInstanceSources( + sources, + instances, ); - expect(claims.size).toBe(0); + const pick = (uris: string[]) => + buildSourceTree(resources, uris).map(n => n.label); + expect(pick(functionSlots.taskId!)).toEqual([INSTANCES_GROUP, NEW_GROUP]); + expect(pick(functionSlots.other!)).toEqual(["Other"]); + }); + + it("leaves sources alone for a run with no instances", () => { + expect(withInstanceSources(sources, {}).resources).toEqual(catalog); + }); + + it("describes a chip by the instance's resource, or says it's gone", () => { + const byUri = new Map(catalog.map(r => [r.uri, r])); + expect( + describeInstanceSource(instanceUri("root", "id"), instances, byUri), + ).toEqual({ label: "root.id", note: "Seeded task" }); + expect( + describeInstanceSource(instanceUri("gone"), instances, byUri), + ).toEqual({ + label: "gone", + note: "resource no longer in this run", + missing: true, + }); }); }); diff --git a/src/client/components/pseudo-sources.ts b/src/client/components/pseudo-sources.ts index c21c580..8d9fabe 100644 --- a/src/client/components/pseudo-sources.ts +++ b/src/client/components/pseudo-sources.ts @@ -2,11 +2,13 @@ // Copyright (c) 2026 Alexander Corrado /** - * Columns and prompt slots as sources: `ResourceInfo` entries with URIs the - * server never sees, grouped as **Columns** and **Prompt inputs**, so the + * Columns, prompt slots and the run's resource instances as sources: + * `ResourceInfo` entries with URIs the server never sees, grouped as + * **Columns**, **Prompt inputs** and **Resources in this run**, so the * existing picker, chips and slot matching offer them without knowing they - * aren't resources. `toExecutionInput` turns a chosen one into the `dataset` - * or `input` variant it stands for. See `specs/evals.md` §B.2 and §F. + * aren't resources. `toExecutionInput` turns a chosen one into the + * `dataset`, `input` or `instance` variant it stands for. See + * `specs/evals.md` §B.2 and §F, and `specs/resource-instances.md` §G. */ import type { DatasetField } from "../../dataset/dataset-types"; @@ -26,6 +28,7 @@ import type { const COLUMN_PREFIX = "@@column/"; const INPUT_PREFIX = "@@input/"; +const INSTANCE_PREFIX = "@@instance/"; /** The picker entry that adds a dataset column from the slot it's chosen for. */ export const NEW_COLUMN_URI = "@@new-column"; @@ -34,6 +37,10 @@ export const NEW_COLUMN_URI = "@@new-column"; export const COLUMNS_GROUP = "Dataset columns"; /** The group other prompt slots are listed under. */ export const PROMPT_INPUTS_GROUP = "Prompt inputs"; +/** The group the run's own resource instances are listed under, first. */ +export const INSTANCES_GROUP = "Resources in this run"; +/** The group the resource catalog moves under once the run has instances to list first. */ +export const NEW_GROUP = "New"; /** The pseudo-source URI for dataset field `field`. */ export function columnUri(field: string): string { @@ -45,6 +52,25 @@ export function inputUri(half: "function" | "execute", path: string): string { return `${INPUT_PREFIX}${half}/${path}`; } +/** The pseudo-source URI for the run's instance `name`, or its `output`. */ +export function instanceUri(name: string, output?: string): string { + return output === undefined + ? `${INSTANCE_PREFIX}${name}` + : `${INSTANCE_PREFIX}${name}/${output}`; +} + +/** The instance (and output) an {@link instanceUri} names, or `undefined` for any other URI. */ +export function parseInstanceUri( + uri: string, +): { name: string; output?: string } | undefined { + if (!uri.startsWith(INSTANCE_PREFIX)) return undefined; + const rest = uri.slice(INSTANCE_PREFIX.length); + const slash = rest.indexOf("/"); + return slash < 0 + ? { name: rest } + : { name: rest.slice(0, slash), output: rest.slice(slash + 1) }; +} + /** Whether `uri` names a pseudo-source rather than a resource. */ export function isPseudoUri(uri: string): boolean { return uri.startsWith("@@"); @@ -65,6 +91,8 @@ export function pseudoInput(uri: string): ExecutionInput | undefined { if (half !== "function" && half !== "execute") return undefined; return { kind: "input", half, path: rest.slice(slash + 1) }; } + const instance = parseInstanceUri(uri); + if (instance) return { kind: "instance", ...instance }; return undefined; } @@ -72,6 +100,7 @@ export function pseudoInput(uri: string): ExecutionInput | undefined { export function pseudoUriOf(input: ExecutionInput): string | undefined { if (input.kind === "dataset") return columnUri(input.field); if (input.kind === "input") return inputUri(input.half, input.path); + if (input.kind === "instance") return instanceUri(input.name, input.output); return undefined; } @@ -351,3 +380,143 @@ export function describePseudoSource( ? { note, warning: `${type.syntax} into ${slotType.syntax}` } : { note }; } + +/** A run's instances, as the picker needs them: by name, in listing order. */ +type InstanceUris = Readonly>; + +/** + * `resources` with the run's instances (and their outputs) listed first, as + * **Resources in this run**, and the catalog moved under **New** once there + * is any instance to list ahead of it. See {@link withInstanceSources}. + */ +export function withInstanceResources( + resources: readonly ResourceInfo[], + instances: InstanceUris, +): ResourceInfo[] { + const named = Object.entries(instances); + if (named.length === 0) return [...resources]; + const byUri = new Map(resources.map(r => [r.uri, r])); + const outputsOf = new Map(); + for (const r of resources) { + if (r.parent) { + outputsOf.set(r.parent, [...(outputsOf.get(r.parent) ?? []), r]); + } + } + const instanceEntries: ResourceInfo[] = named.flatMap(([name, { uri }]) => { + const root = byUri.get(uri); + const scope = root?.scope ?? "run"; + return [ + { + uri: instanceUri(name), + label: name, + scope, + group: [INSTANCES_GROUP], + // A static resource's instance is its value: the slot previews it. + ...(root?.value !== undefined && { value: root.value }), + }, + ...(outputsOf.get(uri) ?? []).map( + (output): ResourceInfo => ({ + uri: instanceUri(name, output.uri.slice(uri.length + 1)), + label: output.label, + scope, + group: [INSTANCES_GROUP], + parent: instanceUri(name), + ...(output.value !== undefined && { value: output.value }), + }), + ), + ]; + }); + return [ + ...instanceEntries, + ...resources.map(r => + isPseudoUri(r.uri) ? r : { ...r, group: [NEW_GROUP, ...(r.group ?? [])] }, + ), + ]; +} + +/** + * `slots` (slot path → offered URIs) with each run instance offered ahead of + * the catalog entries it stands for: an instance of `seededTask` wherever + * `seededTask` is offered, and its `taskId` wherever `seededTask.taskId` is. + * + * @param resources - The catalog, to tell an output's resource. + */ +export function withInstanceSlots( + slots: Readonly> | undefined, + resources: readonly ResourceInfo[], + instances: InstanceUris, +): Record { + const named = Object.entries(instances); + if (named.length === 0) return { ...slots }; + const parentOf = new Map( + resources.flatMap(r => (r.parent ? [[r.uri, r.parent] as const] : [])), + ); + return Object.fromEntries( + Object.entries(slots ?? {}).map(([path, offered]) => { + const fitting: string[] = []; + for (const [name, { uri }] of named) { + for (const candidate of offered) { + if (candidate === uri) fitting.push(instanceUri(name)); + else if (parentOf.get(candidate) === uri) { + fitting.push(instanceUri(name, candidate.slice(uri.length + 1))); + } + } + } + return [path, [...fitting, ...offered]]; + }), + ); +} + +/** + * `sources`, with the run's own instances offered first wherever their + * resource (or one of its outputs) fits — the picker's **Resources in this + * run** group — and the catalog moved under **New** once there is any + * instance to list ahead of it. Picking from **New** adds an instance (see + * `adoptCatalogPick`); picking from the instances group only references + * one. No new matching is needed: an instance of `seededTask` fits wherever + * `seededTask` does, and its `taskId` wherever `seededTask.taskId` does. + * + * @param instances - The run's instances, by name, in the order to list them. + */ +export function withInstanceSources( + sources: PromptInputSources | undefined, + instances: InstanceUris, +): PromptInputSources { + const resources = sources?.resources ?? []; + const extend = (slots: Record | undefined) => + withInstanceSlots(slots, resources, instances); + return { + resources: withInstanceResources(resources, instances), + functionSlots: extend(sources?.functionSlots), + executeSlots: extend(sources?.executeSlots), + ...(sources?.resourceSlots && { + resourceSlots: Object.fromEntries( + Object.entries(sources.resourceSlots).map(([uri, slots]) => [ + uri, + extend(slots), + ]), + ), + }), + }; +} + +/** + * What a chip for instance reference `uri` says: what it is, or that the run + * no longer has it. + */ +export function describeInstanceSource( + uri: string, + instances: InstanceUris, + resourcesByUri: ReadonlyMap, +): { label: string; note: string; missing?: boolean } | undefined { + const ref = parseInstanceUri(uri); + if (!ref) return undefined; + const instance = instances[ref.name]; + const label = + ref.output === undefined ? ref.name : `${ref.name}.${ref.output}`; + if (!instance) { + return { label, note: "resource no longer in this run", missing: true }; + } + const resource = resourcesByUri.get(instance.uri); + return { label, note: resource?.label ?? instance.uri }; +} diff --git a/src/client/components/resource-args-context.ts b/src/client/components/resource-args-context.ts deleted file mode 100644 index e39a2d6..0000000 --- a/src/client/components/resource-args-context.ts +++ /dev/null @@ -1,193 +0,0 @@ -// SPDX-License-Identifier: AGPL-3.0-only -// Copyright (c) 2026 Alexander Corrado - -import type { PropType, ResourceInfo } from "../../shared/types"; -import { - type ResourceArgs, - rootResourceUri, - type Selections, - type SlotSelection, -} from "./execution-input-state"; -import { isPseudoUri } from "./pseudo-sources"; - -/** - * Nesting cap on a resource's own argument form — a resource whose argument - * is a resource whose argument is a resource. `ResourceRegistry`'s cycle - * check (`specs/resource-arguments.md` §D) is the backstop for anything past - * this; this is purely about the panel not growing an unbounded column of - * nested forms. - */ -export const MAX_RESOURCE_ARG_DEPTH = 3; - -/** - * Who owns a resource's first selection: `path` is its identity (compared - * for equality — see {@link ResourceArgsContext.path}), `label` is what a - * later chip's "same instance as X above" note displays for it. - * - * The two are deliberately different things. Two rows can easily share a - * *label* — a prompt's own `taskId` parameter and some other resource's own - * `taskId` argument are both just named `"taskId"` — so identity has to come - * from each row's full position in the tree, not from what it's called. - */ -export interface ClaimOwner { - /** Unique path identifying the owning row — see {@link ResourceArgsContext.path}. */ - path: string; - /** Display name for the "same instance as X above" note. */ - label: string; -} - -/** - * Everything `SourceRow` needs to render a chosen resource's argument form — - * bundled into one object so it threads through `ExecutionInputEditor` and - * `CombinedInputEditor` as a single prop rather than five. See - * `specs/resource-arguments.md` §I, §J. - */ -export interface ResourceArgsContext { - /** The panel's whole argument-editor state, keyed by resource `uri`. */ - resourceArgs: ResourceArgs; - /** Replaces one resource's argument selections. */ - onResourceArgsChange: (uri: string, next: Selections) => void; - /** - * Resource `uri` → (argument slot path → URIs of sources that can fill - * it) — `PromptInputSources.resourceSlots`, unpacked. - */ - resourceSlots: Record>; - /** - * Resource `uri` → the row that owns its *first* selection — every other - * selection of that `uri` is the same instance, not a fresh one, and - * `SourceRow` says so on the chip instead of repeating an argument form - * (§I's "one binding per resource" rule) it wouldn't own anyway. Computed - * once per render by {@link computeClaims}, from the same selections that - * will render, rather than mutated as rows render: `SourceRow` only ever - * reads this, which is what keeps it safe under React StrictMode's - * double-invoked renders. - */ - claimed: ReadonlyMap; - /** - * This row's own unique position in the whole tree — `"fn.taskId"` for a - * top-level function parameter, `"fn.toolsContext.list_tasks.db"` for a - * nested field within one, `"fn.taskId.args.title"` for an argument of - * whatever `fn.taskId` resolved to. Compared against - * {@link ClaimOwner.path} to decide "is this row the owner?" — never - * against a bare name, which is what let two same-named rows (a prompt's - * own `taskId` and some resource's own `taskId` argument) collide. - */ - path: string; - /** How many resource-argument forms deep this row already is. */ - depth: number; - /** - * What a chip for a column or prompt-slot source says, and whether its - * type doesn't fit the slot's — see `pseudo-sources.ts`. Absent where no - * such source is offered. - */ - describePseudo?: ( - uri: string, - slotType: PropType, - ) => { note: string; warning?: string } | undefined; -} - -/** - * `context`, moved to the child at `pathSuffix` (appended with `.`) and one - * level deeper — for a nested field within the current slot, or an argument - * of the resource the current slot resolved to. - */ -export function nested( - context: ResourceArgsContext, - pathSuffix: string, -): ResourceArgsContext { - return { - ...context, - path: `${context.path}.${pathSuffix}`, - depth: context.depth + 1, - }; -} - -/** - * `context`, moved to a nested field *within the current slot* — a `db` - * field buried inside `toolsContext`, say. Only the path grows; `depth` - * doesn't, because this isn't argument nesting at all, just a different - * field of the same top-level selection. (Crossing into a *resource's own* - * arguments is what {@link nested} is for, and that's the one the depth cap - * applies to.) - */ -export function nestedField( - context: ResourceArgsContext, - pathSuffix: string, -): ResourceArgsContext { - return { ...context, path: `${context.path}.${pathSuffix}` }; -} - -/** - * Precomputes, for every resource selected anywhere in the panel, which row - * "owns" its first selection — the first one reached walking `entries` in - * order, and within a slot's own nested resource choices, in whatever order - * they appear there. Recurses into an owning resource's own argument - * selections (up to {@link MAX_RESOURCE_ARG_DEPTH}), so a resource chosen as - * *another* resource's argument gets claimed too — whether or not either one - * takes arguments itself: a resource with none still gets the same instance - * both times it's picked, and `SourceRow`'s chip says so. - * - * Pure — no rendering, no mutation of anything the caller didn't just - * allocate — so it can be recomputed on every render without StrictMode's - * double-invocation ever observing a half-claimed map. - * - * @param entries - Top-level slots in panel order, each with the unique - * `path` and display `label` `SourceRow` would render it with (a function - * or execute parameter's own path, e.g. `"fn.taskId"`). - */ -export function computeClaims( - entries: readonly { - path: string; - label: string; - selection: SlotSelection | undefined; - }[], - resourceArgs: ResourceArgs, - resourcesByUri: ReadonlyMap, -): ReadonlyMap { - const claimed = new Map(); - - const visit = ( - path: string, - label: string, - selection: SlotSelection | undefined, - depth: number, - ) => { - if (!selection?.resources || depth > MAX_RESOURCE_ARG_DEPTH) return; - for (const [relPath, uri] of Object.entries(selection.resources)) { - // A column or another slot isn't an instance anyone shares. - if (isPseudoUri(uri)) continue; - const root = rootResourceUri(uri, resourcesByUri); - if (claimed.has(root)) continue; - - // A nested field's own row is identified by its full path, and - // labelled by its own (leaf) name — the same as the `propDef.name` - // `SourceRow` renders it with, matching `ExecutionInputEditor`'s - // nested-slot plugin. - const ownPath = relPath === "" ? path : `${path}.${relPath}`; - const segments = relPath.split("."); - const ownLabel = relPath === "" ? label : segments[segments.length - 1]; - claimed.set(root, { path: ownPath, label: ownLabel }); - - // Only a resource that actually takes arguments has selections of its - // own worth descending into — one that doesn't is still claimed above, - // just with nothing beneath it to recurse into. - const info = resourcesByUri.get(root); - if (!info?.parameters || info.parameters.length === 0) continue; - const argSelections = resourceArgs[root] ?? {}; - for (const param of info.parameters) { - visit( - `${ownPath}.args.${param.name}`, - param.name, - argSelections[param.name], - depth + 1, - ); - } - } - }; - - for (const { path, label, selection } of entries) { - visit(path, label, selection, 0); - } - - return claimed; -} diff --git a/src/client/components/run-resources-state.test.ts b/src/client/components/run-resources-state.test.ts new file mode 100644 index 0000000..a3dbcce --- /dev/null +++ b/src/client/components/run-resources-state.test.ts @@ -0,0 +1,218 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { describe, expect, it } from "vitest"; +import type { ResourceInfo } from "../../shared/types"; +import { SELF, type Selections } from "./execution-input-state"; +import { instanceUri } from "./pseudo-sources"; +import { + adoptCatalogPick, + defaultInstanceName, + derivedDependencies, + duplicateInstance, + excludedFromArgs, + fromWireResources, + type InstanceSelections, + instanceNameProblem, + referencesTo, + removeInstance, + renameInstance, + retargetSelections, + toWireResources, +} from "./run-resources-state"; + +const catalog: ResourceInfo[] = [ + { uri: "pg.ts#db", label: "DB", scope: "server" }, + { + uri: "pg.ts#task", + label: "Task", + scope: "run", + dependencies: { db: "pg.ts#db" }, + }, + { + uri: "pg.ts#task.id", + label: "Task ID", + scope: "run", + parent: "pg.ts#task", + }, +]; +const byUri = new Map(catalog.map(r => [r.uri, r])); + +const ref = (name: string, output?: string) => ({ + resources: { [SELF]: instanceUri(name, output) }, +}); + +/** A root task, a child taking its id, and a slot bound to the child. */ +function hierarchy(): InstanceSelections { + return { + root: { + uri: "pg.ts#task", + args: { title: { value: { kind: "primitive", value: "Root" } } }, + }, + child: { uri: "pg.ts#task", args: { parentId: ref("root", "id") } }, + }; +} + +describe("wire round trip", () => { + it("folds arguments into inputs and back, dropping receipts and empty arguments", () => { + const instances: InstanceSelections = { + ...hierarchy(), + bare: { uri: "pg.ts#db", args: { unused: {} } }, + }; + const wire = toWireResources(instances)!; + expect(wire).toEqual({ + root: { + uri: "pg.ts#task", + args: { + title: { kind: "value", value: { kind: "primitive", value: "Root" } }, + }, + }, + child: { + uri: "pg.ts#task", + args: { parentId: { kind: "instance", name: "root", output: "id" } }, + }, + bare: { uri: "pg.ts#db" }, + }); + expect( + fromWireResources({ ...wire, root: { ...wire.root!, receipt: "r" } }), + ).toEqual({ ...hierarchy(), bare: { uri: "pg.ts#db", args: {} } }); + expect(toWireResources({})).toBeUndefined(); + }); +}); + +describe("names", () => { + it("defaults to the export name, numbered when taken", () => { + expect(defaultInstanceName("pg.ts#task", [])).toBe("task"); + expect(defaultInstanceName("pg.ts#task", ["task", "task2"])).toBe("task3"); + expect(defaultInstanceName("pg.ts#1st", [])).toBe("r_1st"); + }); + + it("says why a name can't be used", () => { + expect(instanceNameProblem("root", hierarchy())).toMatch(/already/); + expect(instanceNameProblem("root", hierarchy(), "root")).toBeUndefined(); + expect(instanceNameProblem("-x", {})).toMatch(/letters, digits/); + }); +}); + +describe("adoptCatalogPick", () => { + it("adds an instance for a pick, and references the output picked", () => { + const picked = adoptCatalogPick(hierarchy(), "pg.ts#task.id", byUri); + expect(picked.uri).toBe(instanceUri("task", "id")); + expect(picked.instances.task).toEqual({ uri: "pg.ts#task", args: {} }); + }); + + it("reuses a server-scoped resource's one instance", () => { + const once = adoptCatalogPick({}, "pg.ts#db", byUri); + const twice = adoptCatalogPick(once.instances, "pg.ts#db", byUri); + expect(twice.instances).toBe(once.instances); + expect(twice.uri).toBe(instanceUri("db")); + }); +}); + +describe("rename, remove, duplicate", () => { + it("renames in place, following references in arguments and slots, and nothing else", () => { + const renamed = renameInstance(hierarchy(), "root", "parent"); + expect(Object.keys(renamed)).toEqual(["parent", "child"]); + expect(renamed.child.args.parentId).toEqual(ref("parent", "id")); + + const slots: Selections = { + taskId: ref("root", "id"), + other: ref("child"), + typed: { value: { kind: "primitive", value: "x" } }, + }; + const retargeted = retargetSelections(slots, "root", "parent"); + expect(retargeted.taskId).toEqual(ref("parent", "id")); + expect(retargeted.other).toBe(slots.other); + expect(retargeted.typed).toBe(slots.typed); + expect(retargetSelections(slots, "nope", "x")).toBe(slots); + }); + + it("refuses a rename to a taken name", () => { + expect(() => renameInstance(hierarchy(), "child", "root")).toThrow( + /already/, + ); + }); + + it("clears every reference to a removed instance", () => { + const removed = removeInstance(hierarchy(), "root"); + expect(Object.keys(removed)).toEqual(["child"]); + expect(removed.child.args.parentId).toEqual({}); + expect( + retargetSelections({ taskId: ref("root", "id") }, "root", null), + ).toEqual({ taskId: {} }); + }); + + it("duplicates next to the original, with its arguments copied", () => { + const { instances, name } = duplicateInstance(hierarchy(), "child"); + expect(name).toBe("child2"); + expect(Object.keys(instances)).toEqual(["root", "child", "child2"]); + expect(instances.child2).toEqual(instances.child); + expect(instances.child2.args).not.toBe(instances.child.args); + }); +}); + +describe("references and cycles", () => { + it("lists the slots and arguments that reference an instance", () => { + expect( + referencesTo("root", hierarchy(), [ + { + selections: { + taskId: ref("root", "id"), + toolsContext: { + resources: { "list_tasks.rootTaskId": instanceUri("root", "id") }, + }, + }, + }, + ]), + ).toEqual([ + "taskId", + "toolsContext.list_tasks.rootTaskId", + "child.parentId", + ]); + expect(referencesTo("child", hierarchy(), [])).toEqual([]); + }); + + it("excludes the instance itself and everything that reaches it from its arguments", () => { + const instances: InstanceSelections = { + ...hierarchy(), + grandchild: { uri: "pg.ts#task", args: { parentId: ref("child", "id") } }, + }; + expect([...excludedFromArgs(instances, "root")].sort()).toEqual([ + "child", + "grandchild", + "root", + ]); + expect([...excludedFromArgs(instances, "grandchild")]).toEqual([ + "grandchild", + ]); + }); +}); + +describe("derivedDependencies", () => { + it("lists a dependency the run doesn't declare, with what uses it", () => { + expect(derivedDependencies(hierarchy(), byUri)).toEqual([ + { uri: "pg.ts#db", usedBy: ["root", "child"] }, + ]); + }); + + it("leaves out a dependency declared once, and flags one declared twice", () => { + expect( + derivedDependencies( + { ...hierarchy(), main: { uri: "pg.ts#db", args: {} } }, + byUri, + ), + ).toEqual([]); + expect( + derivedDependencies( + { + ...hierarchy(), + a: { uri: "pg.ts#db", args: {} }, + b: { uri: "pg.ts#db", args: {} }, + }, + byUri, + ), + ).toEqual([ + { uri: "pg.ts#db", usedBy: ["root", "child"], ambiguous: ["a", "b"] }, + ]); + }); +}); diff --git a/src/client/components/run-resources-state.ts b/src/client/components/run-resources-state.ts new file mode 100644 index 0000000..4aee276 --- /dev/null +++ b/src/client/components/run-resources-state.ts @@ -0,0 +1,388 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +/** + * A run's named resource instances, as the Resources section edits them — + * the client counterpart of `RunResources`, shared by the execute panel, the + * eval Inputs panel and dataset row details. Pure; no React. See + * `specs/resource-instances.md` §G, §H. + */ + +import { isValidInstanceName } from "../../shared/instance-names"; +import type { ResourceInfo, RunResources } from "../../shared/types"; +import { + fromExecutionInput, + type Selections, + type SlotSelection, + toExecutionInput, +} from "./execution-input-state"; +import { instanceUri, parseInstanceUri } from "./pseudo-sources"; + +/** One instance being edited: the resource it creates, and its argument editors by parameter name. */ +export interface InstanceState { + /** The resource's root `uri`. */ + uri: string; + /** Argument editor state, by parameter name. */ + args: Selections; +} + +/** + * The run's instances, by name, in the order the section lists them — + * insertion order, which a plain object keeps for names like these (none is + * an integer-like key, since a name can't start with a digit). + */ +export type InstanceSelections = Record; + +/** The wire form of `instances`, or `undefined` when there are none. */ +export function toWireResources( + instances: InstanceSelections, +): RunResources | undefined { + const entries = Object.entries(instances); + if (entries.length === 0) return undefined; + return Object.fromEntries( + entries.map(([name, { uri, args }]) => { + const wireArgs = Object.fromEntries( + Object.entries(args).flatMap(([param, selection]) => { + const input = toExecutionInput(selection); + return input ? [[param, input] as const] : []; + }), + ); + return [ + name, + Object.keys(wireArgs).length > 0 ? { uri, args: wireArgs } : { uri }, + ]; + }), + ); +} + +/** Editor state for a stored or recorded `resources`, receipts dropped. */ +export function fromWireResources( + resources: RunResources | undefined, +): InstanceSelections { + if (!resources || typeof resources !== "object") return {}; + const out: InstanceSelections = {}; + for (const [name, spec] of Object.entries(resources)) { + if (!spec || typeof spec.uri !== "string") continue; + out[name] = { + uri: spec.uri, + args: Object.fromEntries( + Object.entries(spec.args ?? {}).map(([param, input]) => [ + param, + fromExecutionInput(input), + ]), + ), + }; + } + return out; +} + +/** + * A free name for a new instance of `uri`: its export name (`seededTask`), + * or that with a number (`seededTask2`) when it's taken. + */ +export function defaultInstanceName( + uri: string, + taken: Iterable, +): string { + const used = new Set(taken); + const exported = uri.slice(uri.lastIndexOf("#") + 1).split(".")[0] ?? ""; + let base = exported.replace(/[^A-Za-z0-9_-]/g, "_"); + if (!isValidInstanceName(base)) base = `r_${base}`; + if (!used.has(base)) return base; + for (let n = 2; ; n++) { + if (!used.has(`${base}${n}`)) return `${base}${n}`; + } +} + +/** `instances` with a new, argument-less instance of `uri` added at the end. */ +export function addInstance( + instances: InstanceSelections, + uri: string, +): { instances: InstanceSelections; name: string } { + const name = defaultInstanceName(uri, Object.keys(instances)); + return { instances: { ...instances, [name]: { uri, args: {} } }, name }; +} + +/** + * The instance reference a catalog pick stands for: a server-scoped resource + * already in the run is reused — it has one instance however it's named — + * and anything else becomes a new instance, named by default. An output pick + * (`seededTask.taskId`) references that output of the new instance. + * + * @returns The instances to keep, and the pseudo-source URI to bind. + */ +export function adoptCatalogPick( + instances: InstanceSelections, + pickedUri: string, + resourcesByUri: ReadonlyMap, +): { instances: InstanceSelections; uri: string } { + const picked = resourcesByUri.get(pickedUri); + const rootUri = picked?.parent ?? pickedUri; + const output = picked?.parent + ? pickedUri.slice(picked.parent.length + 1) + : undefined; + if (resourcesByUri.get(rootUri)?.scope === "server") { + const existing = Object.entries(instances).find( + ([, i]) => i.uri === rootUri, + ); + if (existing) return { instances, uri: instanceUri(existing[0], output) }; + } + const added = addInstance(instances, rootUri); + return { instances: added.instances, uri: instanceUri(added.name, output) }; +} + +/** `selection`, with every reference to instance `from` pointed at `to` instead — or cleared, for `to: null`. */ +function retargetSelection( + selection: SlotSelection, + from: string, + to: string | null, +): SlotSelection { + if (!selection.resources) return selection; + let changed = false; + const resources: Record = {}; + for (const [path, uri] of Object.entries(selection.resources)) { + const ref = parseInstanceUri(uri); + if (ref?.name !== from) { + resources[path] = uri; + continue; + } + changed = true; + if (to !== null) resources[path] = instanceUri(to, ref.output); + } + if (!changed) return selection; + return Object.keys(resources).length > 0 + ? { ...selection, resources } + : (({ resources: _, ...rest }) => rest)(selection); +} + +/** `selections`, with every reference to instance `from` pointed at `to`, or cleared for `to: null`. Unchanged entries keep their identity. */ +export function retargetSelections( + selections: Selections, + from: string, + to: string | null, +): Selections { + let changed = false; + const out: Selections = {}; + for (const [key, selection] of Object.entries(selections)) { + const next = retargetSelection(selection, from, to); + if (next !== selection) changed = true; + out[key] = next; + } + return changed ? out : selections; +} + +/** + * `instances` with `from` renamed to `to` in place (keeping its position), + * and every argument referencing it pointed at the new name. The host + * applies {@link retargetSelections} to its own slots the same way. + * + * @throws When `to` is invalid or already taken. + */ +export function renameInstance( + instances: InstanceSelections, + from: string, + to: string, +): InstanceSelections { + if (from === to) return instances; + const problem = instanceNameProblem(to, instances, from); + if (problem) throw new Error(problem); + return Object.fromEntries( + Object.entries(instances).map(([name, state]) => [ + name === from ? to : name, + { ...state, args: retargetSelections(state.args, from, to) }, + ]), + ); +} + +/** Why `name` can't be given to an instance (`self` being the one renamed), or `undefined` when it can. */ +export function instanceNameProblem( + name: string, + instances: InstanceSelections, + self?: string, +): string | undefined { + if (!isValidInstanceName(name)) { + return "Use letters, digits, '_' and '-', not starting with a digit or '-'."; + } + if (name !== self && Object.hasOwn(instances, name)) { + return `There's already a resource named '${name}'.`; + } + return undefined; +} + +/** `instances` without `name`, and with every argument that referenced it cleared. */ +export function removeInstance( + instances: InstanceSelections, + name: string, +): InstanceSelections { + return Object.fromEntries( + Object.entries(instances) + .filter(([n]) => n !== name) + .map(([n, state]) => [ + n, + { ...state, args: retargetSelections(state.args, name, null) }, + ]), + ); +} + +/** `instances` with a copy of `name` — same resource, same arguments — right after it. */ +export function duplicateInstance( + instances: InstanceSelections, + name: string, +): { instances: InstanceSelections; name: string } { + const source = instances[name]; + if (!source) return { instances, name }; + const copy = defaultInstanceName( + name.replace(/\d+$/, "") || name, + Object.keys(instances), + ); + const out: InstanceSelections = {}; + for (const [n, state] of Object.entries(instances)) { + out[n] = state; + if (n === name) { + out[copy] = { uri: source.uri, args: structuredClone(source.args) }; + } + } + return { instances: out, name: copy }; +} + +/** Every place in `selection` that references instance `name`, as dotted paths below `prefix`. */ +function referencesIn( + selection: SlotSelection | undefined, + name: string, + prefix: string, +): string[] { + return Object.entries(selection?.resources ?? {}).flatMap(([path, uri]) => + parseInstanceUri(uri)?.name === name + ? [path === "" ? prefix : `${prefix}.${path}`] + : [], + ); +} + +/** + * What references instance `name` — each slot (`taskId`, + * `toolsContext.list_tasks.db`) and each other instance's argument + * (`child1.parentId`) — for the card's "→ taskId" line. + * + * @param slots - The host's slot selections, each labelled by how a + * reference into it should read (`""` for slots named bare). + */ +export function referencesTo( + name: string, + instances: InstanceSelections, + slots: readonly { prefix?: string; selections: Selections }[], +): string[] { + const refs: string[] = []; + for (const { prefix, selections } of slots) { + for (const [slot, selection] of Object.entries(selections)) { + refs.push( + ...referencesIn(selection, name, prefix ? `${prefix}.${slot}` : slot), + ); + } + } + for (const [other, state] of Object.entries(instances)) { + if (other === name) continue; + for (const [param, selection] of Object.entries(state.args)) { + refs.push(...referencesIn(selection, name, `${other}.${param}`)); + } + } + return refs; +} + +/** The instance names `state`'s arguments reference directly. */ +function argumentTargets(state: InstanceState): Set { + const names = new Set(); + for (const selection of Object.values(state.args)) { + for (const uri of Object.values(selection.resources ?? {})) { + const ref = parseInstanceUri(uri); + if (ref) names.add(ref.name); + } + } + return names; +} + +/** + * The instances `name`'s arguments may not reference, because creating them + * means creating `name` first: itself, and every instance whose arguments + * reach it. Offering those would only close a cycle. + */ +export function excludedFromArgs( + instances: InstanceSelections, + name: string, +): Set { + const excluded = new Set([name]); + let grew = true; + while (grew) { + grew = false; + for (const [other, state] of Object.entries(instances)) { + if (excluded.has(other)) continue; + if ([...argumentTargets(state)].some(t => excluded.has(t))) { + excluded.add(other); + grew = true; + } + } + } + return excluded; +} + +/** A dependency the run doesn't declare an instance of, shown below the instances. */ +export interface DerivedDependency { + /** The dependency's resource uri. */ + uri: string; + /** The instances (or other dependencies, by uri) whose resource needs it. */ + usedBy: string[]; + /** + * The instances the run declares of it when there are several — a + * dependency can't tell which to use, so the run would fail. Absent when + * there are none. + */ + ambiguous?: string[]; +} + +/** + * The code-wired dependencies of the run's instances that the run doesn't + * declare itself (`specs/resource-instances.md` §D): each is created with no + * arguments, once, and shared by everything that needs it. Followed + * transitively. A dependency the run declares exactly one instance of + * resolves to that instance and isn't listed; one declared more than once is + * listed with {@link DerivedDependency.ambiguous}. + */ +export function derivedDependencies( + instances: InstanceSelections, + resourcesByUri: ReadonlyMap, +): DerivedDependency[] { + const declaredBy = new Map(); + for (const [name, state] of Object.entries(instances)) { + declaredBy.set(state.uri, [...(declaredBy.get(state.uri) ?? []), name]); + } + + const found = new Map(); + const visit = (uri: string, user: string, seen: Set) => { + for (const dep of Object.values( + resourcesByUri.get(uri)?.dependencies ?? {}, + )) { + const declared = declaredBy.get(dep) ?? []; + if (declared.length === 1) continue; + let entry = found.get(dep); + if (!entry) { + entry = { + uri: dep, + usedBy: [], + ...(declared.length > 1 && { ambiguous: declared }), + }; + found.set(dep, entry); + } + if (!entry.usedBy.includes(user)) entry.usedBy.push(user); + if (declared.length === 0 && !seen.has(dep)) { + visit( + dep, + resourcesByUri.get(dep)?.label ?? dep, + new Set([...seen, dep]), + ); + } + } + }; + for (const [name, state] of Object.entries(instances)) { + visit(state.uri, name, new Set([state.uri])); + } + return [...found.values()]; +} diff --git a/src/client/components/source-context.ts b/src/client/components/source-context.ts new file mode 100644 index 0000000..d13d395 --- /dev/null +++ b/src/client/components/source-context.ts @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import type { PropType } from "../../shared/types"; + +/** What a chip for a pseudo-source says — see {@link SourceContext.describePseudo}. */ +export interface PseudoDescription { + /** The chip's own label, when it should differ from the picker entry's. */ + label?: string; + /** The note beneath the label. */ + note: string; + /** Set when the source's type doesn't fit the slot's: `"string into number"`. */ + warning?: string; + /** Set when the source no longer exists — a removed instance, say. */ + missing?: boolean; + /** Brings what the chip names into view: an instance's card. */ + onOpen?: () => void; +} + +/** + * What a slot's editor needs from its host beyond its own selection: how to + * describe a chip for a column, another slot or a resource instance, and how + * to turn a pick from the resource catalog into an instance of the run. + * Threaded unchanged through `ExecutionInputEditor` and `CombinedInputEditor` + * as one prop. See `specs/resource-instances.md` §G, §H. + */ +export interface SourceContext { + /** + * What a chip for pseudo-source `uri` says in a slot of `slotType`. + * Absent where no pseudo-source is offered. + */ + describePseudo?: ( + uri: string, + slotType: PropType, + ) => PseudoDescription | undefined; + /** + * Turns a pick from the resource catalog into a reference to one of the + * run's instances — adding one, usually — and returns that reference's + * pseudo-source URI, which is what the slot then holds. Absent where the + * host offers no catalog. + */ + adopt?: (uri: string) => string; +} diff --git a/src/client/components/source-tree.ts b/src/client/components/source-tree.ts index 21acc1c..30b3617 100644 --- a/src/client/components/source-tree.ts +++ b/src/client/components/source-tree.ts @@ -2,6 +2,7 @@ // Copyright (c) 2026 Alexander Corrado import type { ResourceInfo } from "../../shared/types"; +import { NEW_GROUP } from "./pseudo-sources"; /** * One entry in the nested source menu {@link SourcePicker} renders — a @@ -99,7 +100,15 @@ export function buildSourceTree( if (node) items.push({ group: root.group ?? [], node }); } - return groupTree(items); + const tree = groupTree(items); + // The catalog sits under **New** only to make room for the run's own + // instances above it; for a slot none of them fits, it's all there is, so + // it's opened in place rather than behind one more click. + const [only] = tree; + if (tree.length === 1 && only.kind === "group" && only.label === NEW_GROUP) { + return only.children ?? []; + } + return tree; } /** `["Tasks", "Regressions"]` + `"Task A"` → `"Tasks / Regressions / Task A"`. */ @@ -236,7 +245,9 @@ function finalize( } const children = finalize(entry.children); if (children.length === 0) continue; // every member pruned away - if (children.length === 1) { + // **New** stays a group however little is in it: a pick from it adds an + // instance, which must not read like the run's own instances beside it. + if (children.length === 1 && entry.label !== NEW_GROUP) { resolved.push(children[0]); // nothing left to disambiguate } else { resolved.push({ diff --git a/src/client/components/use-run-instances.ts b/src/client/components/use-run-instances.ts new file mode 100644 index 0000000..d75dbed --- /dev/null +++ b/src/client/components/use-run-instances.ts @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { useCallback, useRef, useState } from "react"; +import type { ResourceInfo } from "../../shared/types"; +import { + adoptCatalogPick, + type InstanceSelections, +} from "./run-resources-state"; + +/** What {@link useRunInstances} returns. */ +export interface RunInstances { + /** The run's instances, as of this render. */ + instances: InstanceSelections; + /** Replaces the instances, or derives them from the latest. */ + setInstances: ( + next: + | InstanceSelections + | ((latest: InstanceSelections) => InstanceSelections), + ) => void; + /** + * `SourceContext.adopt`: adds an instance for a catalog pick (or reuses a + * server-scoped one) and returns the reference to bind. + */ + adopt: (uri: string) => string; +} + +/** + * The run's instances as host state. Updates apply to the latest value + * synchronously rather than to the one a render saw, so a catalog pick — + * which adds an instance and binds a slot (or another instance's argument) + * in one gesture — never has one half overwrite the other. + * + * @param resourcesByUri - The catalog, for {@link adoptCatalogPick}. + * @param onSet - Called with every new value, for the host to persist. + */ +export function useRunInstances( + initial: InstanceSelections | (() => InstanceSelections), + resourcesByUri: ReadonlyMap, + onSet?: (next: InstanceSelections) => void, +): RunInstances { + const [instances, setState] = useState(initial); + const latest = useRef(instances); + const byUri = useRef(resourcesByUri); + byUri.current = resourcesByUri; + const onSetRef = useRef(onSet); + onSetRef.current = onSet; + + const setInstances = useCallback(next => { + const value = typeof next === "function" ? next(latest.current) : next; + if (value === latest.current) return; + latest.current = value; + setState(value); + onSetRef.current?.(value); + }, []); + + const adopt = useCallback( + (uri: string) => { + const picked = adoptCatalogPick(latest.current, uri, byUri.current); + setInstances(picked.instances); + return picked.uri; + }, + [setInstances], + ); + + return { instances, setInstances, adopt }; +} diff --git a/src/client/styles.css b/src/client/styles.css index c03cbbe..1d03779 100644 --- a/src/client/styles.css +++ b/src/client/styles.css @@ -1860,8 +1860,8 @@ pre { color: var(--panel-text-secondary); } -/* The "same instance as ▲ X" link inside a chip note — reset to read as - text, not a button, until hovered/focused. +/* An instance chip's name (`◆ root.taskId`), which opens the instance's + card — reset to read as text, not a button, until hovered/focused. The underline is a `border-bottom`, not `text-decoration`: a `
)} - {(catalog.length > 0 || Object.keys(instances).length > 0) && ( - - referencesTo(name, instances, [ - { selections: functionSelections }, - { selections: executeSelections }, - ]) - } - context={context} - /> - )} {renderSlots( prompt.functionParameters, functionSelections, @@ -730,6 +715,22 @@ function PlaygroundExecution({ )} + {(catalog.length > 0 || Object.keys(instances).length > 0) && ( + + referencesTo(name, instances, [ + { selections: functionSelections }, + { selections: executeSelections }, + ]) + } + context={context} + /> + )} + {broken.map(r => (
Playground module {r.uri} failed to load: {r.error} diff --git a/src/client/components/__fixtures__/evalView.pw.tsx b/src/client/components/__fixtures__/evalView.pw.tsx index 617f05a..3395afa 100644 --- a/src/client/components/__fixtures__/evalView.pw.tsx +++ b/src/client/components/__fixtures__/evalView.pw.tsx @@ -363,7 +363,8 @@ test("what to run against is always showing, in the scrolling body above Run", a await expect(options.getByLabel("Working tree")).toBeChecked(); await expect(options.getByLabel("Parallel workers")).toHaveValue("4"); // Set off from the inputs above by a rule, as the execute inputs are. - const rule = component.locator(".pg-exec-body > .pg-exec-section"); + // (the last divider; others separate the inputs and the resources) + const rule = component.locator(".pg-exec-body > .pg-exec-section").last(); expect( await rule.evaluate((el: Element) => getComputedStyle(el).borderTopWidth), ).toBe("1px"); From 3e9ec4076ba914a893a4c980638627d219bf5d77 Mon Sep 17 00:00:00 2001 From: Alex Corrado Date: Fri, 9 Oct 2026 10:10:28 +0100 Subject: [PATCH 08/12] Use gear/plus icons to hide away the Duplicate/Remove/Configure buttons --- src/client/components/PromptList.tsx | 18 +--- src/client/components/ResourcesSection.tsx | 99 +++++++++++++++---- .../__fixtures__/ResourcesSectionHarness.tsx | 49 +++++++++ .../__fixtures__/playgroundExecution.pw.tsx | 27 +++-- .../__fixtures__/resourcesSection.pw.tsx | 84 ++++++++++++++++ src/client/styles.css | 7 ++ 6 files changed, 241 insertions(+), 43 deletions(-) create mode 100644 src/client/components/__fixtures__/ResourcesSectionHarness.tsx create mode 100644 src/client/components/__fixtures__/resourcesSection.pw.tsx diff --git a/src/client/components/PromptList.tsx b/src/client/components/PromptList.tsx index a1fb7a1..01e4267 100644 --- a/src/client/components/PromptList.tsx +++ b/src/client/components/PromptList.tsx @@ -4,6 +4,7 @@ import { useEffect, useMemo, useRef, useState } from "react"; import type { NormalizedPrompt } from "../../shared/types"; import { requireProviderId } from "../utils"; +import { PlusIcon } from "./trace/icons.tsx"; interface PromptListProps { prompts: NormalizedPrompt[]; @@ -193,23 +194,6 @@ function SearchIcon() { ); } -function PlusIcon() { - return ( - - - - - ); -} - // --- Inline rename input --- function RenameInput({ diff --git a/src/client/components/ResourcesSection.tsx b/src/client/components/ResourcesSection.tsx index 259d54d..b2e01f4 100644 --- a/src/client/components/ResourcesSection.tsx +++ b/src/client/components/ResourcesSection.tsx @@ -2,6 +2,7 @@ // Copyright (c) 2026 Alexander Corrado import { useMemo, useState } from "react"; +import { createPortal } from "react-dom"; import type { PromptInputSources, ResourceInfo } from "../../shared/types"; import { ExecutionInputEditor } from "./ExecutionInputEditor"; import { describeInstanceSource, parseInstanceUri } from "./pseudo-sources"; @@ -17,6 +18,8 @@ import { } from "./run-resources-state"; import SourcePicker from "./SourcePicker"; import type { SourceContext } from "./source-context"; +import { useAnchoredPopover } from "./use-anchored-popover"; +import { PlusIcon } from "./trace/icons"; /** * A change to the run's instances that the host has to carry into its own @@ -166,10 +169,11 @@ export function ResourcesSection({ )}
@@ -181,7 +185,7 @@ export function ResourcesSection({ > {dep.ambiguous ? `⚠ ${dep.usedBy.join(", ")} can't tell which to use: ${dep.ambiguous.join(", ")}` - : `used by ${dep.usedBy.join(", ")}`} + : `implicit instance for ${dep.usedBy.join(", ")}`}
); @@ -195,6 +199,26 @@ function scopeLabel(info: ResourceInfo | undefined): string { return info?.scope === "server" ? "shared" : "per run"; } +/** A gear — the instance's actions menu. */ +function GearIcon() { + return ( + + ); +} + function InstanceCard({ name, instances, @@ -214,6 +238,14 @@ function InstanceCard({ }) { const instance = instances[name]; const [draftName, setDraftName] = useState(null); + const [menuOpen, setMenuOpen] = useState(false); + const closeMenu = () => setMenuOpen(false); + const { triggerRef, popoverRef, style } = + useAnchoredPopover({ + open: menuOpen, + onClose: closeMenu, + matchTriggerWidth: false, + }); const problem = draftName === null ? undefined @@ -292,25 +324,56 @@ function InstanceCard({ {resource && ` · ${scopeLabel(resource)}`} - {resource?.scope !== "server" && ( - - )} + {menuOpen && + createPortal( +
+ {resource?.scope !== "server" && ( + + )} + +
, + document.body, + )}
{problem &&
{problem}
} diff --git a/src/client/components/__fixtures__/ResourcesSectionHarness.tsx b/src/client/components/__fixtures__/ResourcesSectionHarness.tsx new file mode 100644 index 0000000..7f32005 --- /dev/null +++ b/src/client/components/__fixtures__/ResourcesSectionHarness.tsx @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { useState } from "react"; +import type { PromptInputSources, ResourceInfo } from "../../../shared/types"; +import { ResourcesSection } from "../ResourcesSection"; +import type { InstanceSelections } from "../run-resources-state"; +import "../../styles.css"; + +const CATALOG: ResourceInfo[] = [ + { uri: "db.ts#db", label: "db", scope: "run" }, + { uri: "cache.ts#cache", label: "cache", scope: "server" }, +]; + +const SOURCES: PromptInputSources = { + resources: CATALOG, + functionSlots: {}, + executeSlots: {}, +}; + +const INITIAL: InstanceSelections = { + db: { uri: "db.ts#db", args: {} }, + cache: { uri: "cache.ts#cache", args: {} }, +}; + +/** + * Mounts {@link ResourcesSection} with a `db` run resource and a `cache` + * server resource. Only `db` is referenced, so removing it asks first. The + * current instance names are in `[data-testid="names"]`. + */ +export function ResourcesSectionHarness() { + const [instances, setInstances] = useState(INITIAL); + return ( +
+ + setInstances(latest => + typeof next === "function" ? next(latest) : next, + ) + } + sources={SOURCES} + catalog={CATALOG} + referencedBy={name => (name === "db" ? ["taskId"] : [])} + /> + {Object.keys(instances).join(",")} +
+ ); +} diff --git a/src/client/components/__fixtures__/playgroundExecution.pw.tsx b/src/client/components/__fixtures__/playgroundExecution.pw.tsx index b1b9db2..82f7096 100644 --- a/src/client/components/__fixtures__/playgroundExecution.pw.tsx +++ b/src/client/components/__fixtures__/playgroundExecution.pw.tsx @@ -49,6 +49,20 @@ async function chooseNew(page: Page, trigger: Locator, label: string) { await page.getByRole("menuitem", { name: label, exact: true }).click(); } +/** + * Opens instance `instance`'s actions menu (the "⋯" on its card) and picks + * `item` from it. The menu is portal-rendered, so it's found via `page`. + */ +async function chooseInstanceAction( + page: Page, + scope: Locator, + instance: string, + item: string, +) { + await scope.getByRole("button", { name: `Actions for ${instance}` }).click(); + await page.getByRole("menu").getByRole("menuitem", { name: item }).click(); +} + async function mockExecute(page: Page) { await page.route("**/api/**", route => route.fulfill({ @@ -1269,9 +1283,7 @@ test.describe("resource instances (specs/resource-instances.md §G)", () => { .fill("Todo app"); // A duplicate copies the arguments, under the next free name. - await card(resources, "seededTask") - .getByRole("button", { name: "Duplicate" }) - .click(); + await chooseInstanceAction(page, resources, "seededTask", "Duplicate"); await expect( card(resources, "seededTask2").locator("textarea, input").first(), ).toHaveValue("Todo app"); @@ -1322,17 +1334,16 @@ test.describe("resource instances (specs/resource-instances.md §G)", () => { dialogs.push(dialog.message()); return dialog.dismiss(); }); - const remove = component.getByRole("button", { - name: "Remove seededRootTask", - }); - await remove.click(); + const remove = () => + chooseInstanceAction(page, component, "seededRootTask", "Remove"); + await remove(); expect(dialogs).toEqual([ "Remove 'seededRootTask'? taskId will be cleared.", ]); await expect(card(component, "seededRootTask")).toHaveCount(1); page.once("dialog", dialog => dialog.accept()); - await remove.click(); + await remove(); await expect(card(component, "seededRootTask")).toHaveCount(0); await expect(component.locator(".pg-slot-chip")).toHaveCount(0); }); diff --git a/src/client/components/__fixtures__/resourcesSection.pw.tsx b/src/client/components/__fixtures__/resourcesSection.pw.tsx new file mode 100644 index 0000000..30574ad --- /dev/null +++ b/src/client/components/__fixtures__/resourcesSection.pw.tsx @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { expect, test } from "@playwright/experimental-ct-react"; +import { ResourcesSectionHarness } from "./ResourcesSectionHarness"; + +test("an instance's actions sit behind a menu, not inline", async ({ + mount, + page, +}) => { + const component = await mount(); + const trigger = component.getByRole("button", { name: "Actions for db" }); + await expect(trigger).toHaveAttribute("aria-haspopup", "menu"); + await expect(trigger).toHaveAttribute("aria-expanded", "false"); + await expect( + component.getByRole("button", { name: "Duplicate" }), + ).toHaveCount(0); + await expect( + component.getByRole("button", { name: "Remove db" }), + ).toHaveCount(0); + + await trigger.click(); + await expect(trigger).toHaveAttribute("aria-expanded", "true"); + // The menu is portaled to the body, outside the mounted component. + const menu = page.getByRole("menu"); + await expect(menu.getByRole("menuitem")).toHaveText(["Duplicate", "Remove"]); +}); + +test("a server-scoped instance can be removed but not duplicated", async ({ + mount, + page, +}) => { + const component = await mount(); + await component.getByRole("button", { name: "Actions for cache" }).click(); + const menu = page.getByRole("menu"); + await expect(menu.getByRole("menuitem")).toHaveText(["Remove"]); +}); + +test("Duplicate adds a copy and closes the menu", async ({ mount, page }) => { + const component = await mount(); + await component.getByRole("button", { name: "Actions for db" }).click(); + await page + .getByRole("menu") + .getByRole("menuitem", { name: "Duplicate" }) + .click(); + + await expect(component.getByTestId("names")).toHaveText("db,db2,cache"); + await expect(page.getByRole("menu")).toHaveCount(0); +}); + +test("Remove clears the instance once confirmed", async ({ mount, page }) => { + const component = await mount(); + await component.getByRole("button", { name: "Actions for db" }).click(); + page.once("dialog", dialog => dialog.accept()); + await page + .getByRole("menu") + .getByRole("menuitem", { name: "Remove db" }) + .click(); + + await expect(component.getByTestId("names")).toHaveText("cache"); + await expect(page.getByRole("menu")).toHaveCount(0); +}); + +test("cancelling Remove keeps the instance", async ({ mount, page }) => { + const component = await mount(); + await component.getByRole("button", { name: "Actions for db" }).click(); + page.once("dialog", dialog => dialog.dismiss()); + await page + .getByRole("menu") + .getByRole("menuitem", { name: "Remove db" }) + .click(); + + await expect(component.getByTestId("names")).toHaveText("db,cache"); + await expect(page.getByRole("menu")).toHaveCount(0); +}); + +test("a click outside the open menu closes it", async ({ mount, page }) => { + const component = await mount(); + await component.getByRole("button", { name: "Actions for db" }).click(); + await expect(page.getByRole("menu")).toBeVisible(); + + await page.mouse.click(1, 1); + await expect(page.getByRole("menu")).toHaveCount(0); +}); diff --git a/src/client/styles.css b/src/client/styles.css index 1d03779..9c1e293 100644 --- a/src/client/styles.css +++ b/src/client/styles.css @@ -2139,6 +2139,13 @@ pre { border-color: var(--exec-border); } +/* The gear that opens an instance's actions menu. */ +.pg-resource-menu-trigger { + display: flex; + align-items: center; + padding: 2px 5px; +} + /* What references the instance (`→ taskId`), or that nothing does. */ .pg-resource-refs { font-size: 10.5px; From a4ce6a18e5c5dd94a253be1035623d4e2693ea4d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 09:38:46 +0000 Subject: [PATCH 09/12] Address review: parallel cycles, released leases, removed inputs - Registry: a wait-for graph catches a cycle between creations that started on separate paths (declared instances are acquired in parallel, and a code-wired dependency is invisible to findInputCycle), so the run fails instead of hanging. A lease refuses to create anything once released, so a creation still in flight can't make an undisposed duplicate. - The removed inline `resource` input is a 400 naming its slot (inputReferenceProblems), and fails at resolution rather than passing undefined when it sits in a dataset cell. - Filling the panel drops an instance whose arguments name a dropped one, transitively. - Eval row results snapshot the row's own resources (new nullable eval_row_results.row_resources column). - The eval Inputs panel shows a reference to a row-declared instance as "declared by each row" rather than as missing. - Reuse isValidInstanceName in the Turso provider, and one exported instanceNames walker. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- src/client/components/EvalView.tsx | 1 + src/client/components/ResourcesSection.tsx | 13 +- src/client/components/named-inputs.test.ts | 26 + src/client/components/named-inputs.ts | 31 +- src/client/components/pseudo-sources.test.ts | 6 + src/client/components/pseudo-sources.ts | 9 +- src/dataset/turso-dataset-provider.ts | 3 +- .../migration.sql | 1 + .../snapshot.json | 621 ++++++++++++++++++ src/eval/db/migrations/bundled.ts | 9 + src/eval/db/schema.ts | 2 + src/eval/eval-runner.test.ts | 26 +- src/eval/eval-runner.ts | 13 + src/eval/eval-types.ts | 6 + src/eval/turso-eval-provider.ts | 8 +- src/prompt/execution-inputs.test.ts | 27 + src/prompt/execution-inputs.ts | 8 + .../playground/resource-registry.test.ts | 93 +++ src/prompt/playground/resource-registry.ts | 91 ++- src/shared/eval-prefill.ts | 13 +- src/shared/input-references.ts | 51 +- 21 files changed, 1016 insertions(+), 42 deletions(-) create mode 100644 src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql create mode 100644 src/eval/db/migrations/20261009093329_soft_pandemic/snapshot.json diff --git a/src/client/components/EvalView.tsx b/src/client/components/EvalView.tsx index 3f8be74..c644385 100644 --- a/src/client/components/EvalView.tsx +++ b/src/client/components/EvalView.tsx @@ -454,6 +454,7 @@ function EvalView({ const context = instanceSourceContext({ instances: state.instances, catalogByUri: resourcesByUri, + rowsMayDeclare: true, adopt: uri => { const picked = adoptCatalogPick( stateRef.current.instances, diff --git a/src/client/components/ResourcesSection.tsx b/src/client/components/ResourcesSection.tsx index b2e01f4..09950bb 100644 --- a/src/client/components/ResourcesSection.tsx +++ b/src/client/components/ResourcesSection.tsx @@ -18,8 +18,8 @@ import { } from "./run-resources-state"; import SourcePicker from "./SourcePicker"; import type { SourceContext } from "./source-context"; -import { useAnchoredPopover } from "./use-anchored-popover"; import { PlusIcon } from "./trace/icons"; +import { useAnchoredPopover } from "./use-anchored-popover"; /** * A change to the run's instances that the host has to carry into its own @@ -426,21 +426,28 @@ export function instanceSourceContext({ catalogByUri, adopt, describeOther, + rowsMayDeclare, }: { instances: InstanceSelections; catalogByUri: ReadonlyMap; adopt: (uri: string) => string; describeOther?: SourceContext["describePseudo"]; + /** An eval's: a name it doesn't declare is one each row declares. See `describeInstanceSource`. */ + rowsMayDeclare?: boolean; }): SourceContext { return { adopt, describePseudo: (uri, slotType) => { - const instance = describeInstanceSource(uri, instances, catalogByUri); + const instance = describeInstanceSource(uri, instances, catalogByUri, { + rowsMayDeclare, + }); if (!instance) return describeOther?.(uri, slotType); const ref = parseInstanceUri(uri)!; + // Only an instance with a card here has anywhere to scroll to. + const hasCard = Object.hasOwn(instances, ref.name); return { ...instance, - ...(!instance.missing && { onOpen: () => scrollToInstance(ref.name) }), + ...(hasCard && { onOpen: () => scrollToInstance(ref.name) }), }; }, }; diff --git a/src/client/components/named-inputs.test.ts b/src/client/components/named-inputs.test.ts index 1ebadd1..db1bb03 100644 --- a/src/client/components/named-inputs.test.ts +++ b/src/client/components/named-inputs.test.ts @@ -215,6 +215,32 @@ describe("toPanel", () => { expect(skipped).toEqual([{ name: "db", reason: "resource-out-of-scope" }]); }); + it("also drops an in-scope instance whose arguments name a dropped one, and what names it", () => { + const { resources, skipped } = toPanel( + [ + { + def: def("db", "Db", "opaque"), + input: { kind: "instance", name: "grandchild" }, + }, + ], + prompt(), + { + root: { uri: "elsewhere.playground.ts#db" }, + child: { + uri: DB_URI, + args: { parent: { kind: "instance", name: "root", output: "id" } }, + }, + grandchild: { + uri: DB_URI, + args: { parent: { kind: "instance", name: "child" } }, + }, + near: { uri: DB_URI }, + }, + ); + expect(resources).toEqual({ near: { uri: DB_URI } }); + expect(skipped).toEqual([{ name: "db", reason: "resource-out-of-scope" }]); + }); + it("fills both a function and an execute parameter of one name and type", () => { const p = prompt({ functionParameters: [def("userId")], diff --git a/src/client/components/named-inputs.ts b/src/client/components/named-inputs.ts index 2528ea5..ee3e38f 100644 --- a/src/client/components/named-inputs.ts +++ b/src/client/components/named-inputs.ts @@ -15,6 +15,7 @@ import { fieldsForPrompt, matchKey, } from "../../shared/dataset-fields"; +import { instanceNames } from "../../shared/input-references"; import { jsonToPropValue } from "../../shared/json-prop-value"; import type { Dataset, @@ -215,25 +216,14 @@ export function rowResources( : undefined; } -/** Every instance name an input references, at any depth. */ -function instanceNames(input: ExecutionInput): string[] { - switch (input.kind) { - case "instance": - return [input.name]; - case "object": - return Object.values(input.properties).flatMap(instanceNames); - default: - return []; - } -} - /** * Panel inputs for `prompt`: each input fills every slot it matches — a * function and an execute parameter of one name and type are one value, so * both get it. `resources` come along, receipts stripped, except an instance * of a resource outside this prompt's `inputSources`: it couldn't be created - * here, so it's dropped, and an input referencing it is skipped rather than - * filled — it would only fail at Run. + * here, so it's dropped — with every instance whose arguments name it — and + * an input referencing a dropped one is skipped rather than filled; it would + * only fail at Run. */ export function toPanel( inputs: NamedInputs, @@ -249,6 +239,19 @@ export function toPanel( if (inScope.has(spec.uri)) kept[name] = spec; else dropped.add(name); } + // An instance whose arguments name a dropped one couldn't be created + // either: drop it too, and whatever names it in turn. + for (let changed = dropped.size > 0; changed; ) { + changed = false; + for (const [name, spec] of Object.entries(kept)) { + const args = Object.values(spec.args ?? {}); + if (args.some(arg => instanceNames(arg).some(n => dropped.has(n)))) { + delete kept[name]; + dropped.add(name); + changed = true; + } + } + } const functionInputs: Record = {}; const executeInputs: Record = {}; const skipped: SkippedInput[] = []; diff --git a/src/client/components/pseudo-sources.test.ts b/src/client/components/pseudo-sources.test.ts index a42cb02..4cce2ed 100644 --- a/src/client/components/pseudo-sources.test.ts +++ b/src/client/components/pseudo-sources.test.ts @@ -123,6 +123,12 @@ describe("withInstanceSources", () => { note: "resource no longer in this run", missing: true, }); + // In an eval, an undeclared name is one each row declares. + expect( + describeInstanceSource(instanceUri("task"), instances, byUri, { + rowsMayDeclare: true, + }), + ).toEqual({ label: "task", note: "declared by each row" }); }); }); diff --git a/src/client/components/pseudo-sources.ts b/src/client/components/pseudo-sources.ts index 8d9fabe..e5bcfc4 100644 --- a/src/client/components/pseudo-sources.ts +++ b/src/client/components/pseudo-sources.ts @@ -503,11 +503,16 @@ export function withInstanceSources( /** * What a chip for instance reference `uri` says: what it is, or that the run * no longer has it. + * + * @param options.rowsMayDeclare - In an eval, a name the eval doesn't + * declare is one each dataset row declares (`specs/resource-instances.md` + * §E), so it reads as that rather than as missing. */ export function describeInstanceSource( uri: string, instances: InstanceUris, resourcesByUri: ReadonlyMap, + options: { rowsMayDeclare?: boolean } = {}, ): { label: string; note: string; missing?: boolean } | undefined { const ref = parseInstanceUri(uri); if (!ref) return undefined; @@ -515,7 +520,9 @@ export function describeInstanceSource( const label = ref.output === undefined ? ref.name : `${ref.name}.${ref.output}`; if (!instance) { - return { label, note: "resource no longer in this run", missing: true }; + return options.rowsMayDeclare + ? { label, note: "declared by each row" } + : { label, note: "resource no longer in this run", missing: true }; } const resource = resourcesByUri.get(instance.uri); return { label, note: resource?.label ?? instance.uri }; diff --git a/src/dataset/turso-dataset-provider.ts b/src/dataset/turso-dataset-provider.ts index 070622d..8ebbbdb 100644 --- a/src/dataset/turso-dataset-provider.ts +++ b/src/dataset/turso-dataset-provider.ts @@ -12,6 +12,7 @@ import type { Database } from "@tursodatabase/sync"; import { and, asc, desc, eq, inArray, type SQL, sql } from "drizzle-orm"; import { drizzle } from "drizzle-orm/tursodatabase-sync"; import { matchKey } from "../shared/dataset-fields.ts"; +import { isValidInstanceName } from "../shared/instance-names.ts"; import type { ExecutionInput, PropDefinition, @@ -497,7 +498,7 @@ export class TursoDatasetProvider implements DatasetProvider { } } for (const name of Object.keys(resources ?? {})) { - if (!/^[A-Za-z_][A-Za-z0-9_-]*$/.test(name)) { + if (!isValidInstanceName(name)) { throw new DatasetValidationError( `"${name}" is not a valid resource name`, ); diff --git a/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql b/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql new file mode 100644 index 0000000..14c0077 --- /dev/null +++ b/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql @@ -0,0 +1 @@ +ALTER TABLE `eval_row_results` ADD `row_resources` text; \ No newline at end of file diff --git a/src/eval/db/migrations/20261009093329_soft_pandemic/snapshot.json b/src/eval/db/migrations/20261009093329_soft_pandemic/snapshot.json new file mode 100644 index 0000000..4252592 --- /dev/null +++ b/src/eval/db/migrations/20261009093329_soft_pandemic/snapshot.json @@ -0,0 +1,621 @@ +{ + "version": "7", + "dialect": "sqlite", + "id": "9f2b17e8-d36d-460c-9c18-a39b135ec0f7", + "prevIds": [ + "a420bb54-c6db-469c-9f51-ac737c395d75" + ], + "ddl": [ + { + "name": "eval_check_results", + "entityType": "tables" + }, + { + "name": "eval_row_results", + "entityType": "tables" + }, + { + "name": "eval_runs", + "entityType": "tables" + }, + { + "name": "evals", + "entityType": "tables" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "run_id", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "arm_id", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "row_id", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "0", + "generated": null, + "name": "sample", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "check_id", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "outcome", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "real", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "score", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "message", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "details", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "real", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "duration_ms", + "entityType": "columns", + "table": "eval_check_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "run_id", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "arm_id", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "row_id", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "0", + "generated": null, + "name": "sample", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "row_index", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "row_cells", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "row_resources", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "trace_provider_id", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "trace_id", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "version", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "variation", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "status", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "error", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "real", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "cost_usd", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "real", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "duration_ms", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "0", + "generated": null, + "name": "trace_incomplete", + "entityType": "columns", + "table": "eval_row_results" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "eval_id", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "definition", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "arms", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "start_version", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "0", + "generated": null, + "name": "dirty", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "status", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "0", + "generated": null, + "name": "drifted", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "concurrency", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "total", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "started_at", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "real", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "ended_at", + "entityType": "columns", + "table": "eval_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "name", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "prompt", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "dataset_provider_id", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "dataset_id", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "inputs", + "entityType": "columns", + "table": "evals" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "checks", + "entityType": "columns", + "table": "evals" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "created_at", + "entityType": "columns", + "table": "evals" + }, + { + "type": "real", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_at", + "entityType": "columns", + "table": "evals" + }, + { + "columns": [ + "run_id" + ], + "tableTo": "eval_runs", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_eval_check_results_run_id_eval_runs_id_fk", + "entityType": "fks", + "table": "eval_check_results" + }, + { + "columns": [ + "run_id" + ], + "tableTo": "eval_runs", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_eval_row_results_run_id_eval_runs_id_fk", + "entityType": "fks", + "table": "eval_row_results" + }, + { + "columns": [ + "eval_id" + ], + "tableTo": "evals", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_eval_runs_eval_id_evals_id_fk", + "entityType": "fks", + "table": "eval_runs" + }, + { + "columns": [ + "run_id", + "arm_id", + "row_id", + "sample", + "check_id" + ], + "nameExplicit": false, + "name": "eval_check_results_pk", + "entityType": "pks", + "table": "eval_check_results" + }, + { + "columns": [ + "run_id", + "arm_id", + "row_id", + "sample" + ], + "nameExplicit": false, + "name": "eval_row_results_pk", + "entityType": "pks", + "table": "eval_row_results" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "eval_runs_pk", + "table": "eval_runs", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "evals_pk", + "table": "evals", + "entityType": "pks" + }, + { + "columns": [ + { + "value": "trace_provider_id", + "isExpression": false + }, + { + "value": "trace_id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "idx_eval_row_results_trace", + "entityType": "indexes", + "table": "eval_row_results" + }, + { + "columns": [ + { + "value": "eval_id", + "isExpression": false + }, + { + "value": "started_at", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "idx_eval_runs_eval", + "entityType": "indexes", + "table": "eval_runs" + } + ], + "renames": [] +} \ No newline at end of file diff --git a/src/eval/db/migrations/bundled.ts b/src/eval/db/migrations/bundled.ts index 86104c2..27cb068 100644 --- a/src/eval/db/migrations/bundled.ts +++ b/src/eval/db/migrations/bundled.ts @@ -22,5 +22,14 @@ export const bundledMigrations: MigrationMeta[] = [ "\nCREATE INDEX `idx_eval_row_results_trace` ON `eval_row_results` (`trace_provider_id`,`trace_id`);", "\nCREATE INDEX `idx_eval_runs_eval` ON `eval_runs` (`eval_id`,`started_at`);" ] + }, + { + "name": "20261009093329_soft_pandemic", + "hash": "e8916ff94f63623ce05055ed9e6522a447a24dfeac99214e6ec4c2f1efc954be", + "folderMillis": 1791538409000, + "bps": true, + "sql": [ + "ALTER TABLE `eval_row_results` ADD `row_resources` text;" + ] } ]; diff --git a/src/eval/db/schema.ts b/src/eval/db/schema.ts index faba95f..9599948 100644 --- a/src/eval/db/schema.ts +++ b/src/eval/db/schema.ts @@ -72,6 +72,8 @@ export const evalRowResults = sqliteTable( rowIndex: integer("row_index").notNull(), /** JSON: the row's cells as run. */ rowCells: text("row_cells").notNull(), + /** JSON: the row's own resource instances as run, if it declared any. */ + rowResources: text("row_resources"), traceProviderId: text("trace_provider_id"), traceId: text("trace_id"), version: text("version"), diff --git a/src/eval/eval-runner.test.ts b/src/eval/eval-runner.test.ts index 438709f..532fb74 100644 --- a/src/eval/eval-runner.test.ts +++ b/src/eval/eval-runner.test.ts @@ -15,6 +15,7 @@ import type { ExecutionInput, NormalizedPrompt, PropDefinition, + RunResources, } from "../shared/types.ts"; import { MemoryTraceProvider } from "../trace/memory-trace-provider.ts"; import { runEvalMigrations } from "./db/migrate.ts"; @@ -135,6 +136,8 @@ const contains = (id: string, field: string): EvalCheck => ({ async function setUp( options: { rows?: Record[]; + /** Each row's own resource instances, by position. */ + rowResources?: (RunResources | undefined)[]; checks?: EvalCheck[]; inputs?: Record; provider?: Parameters[1]; @@ -161,7 +164,12 @@ async function setUp( { "0": text("cats"), "1": text("cats") }, { "0": text("dogs"), "1": text("birds") }, ] - ).map(cells => ({ cells })), + ).map((cells, i) => ({ + cells, + ...(options.rowResources?.[i] && { + resources: options.rowResources[i], + }), + })), ); const def = await evals.createEval({ name: "Answers", @@ -225,6 +233,22 @@ describe("EvalRunner", () => { }); }); + it("snapshots each row's own resources with its result, receipts stripped", async () => { + const { runner, evals, def } = await setUp({ + rowResources: [ + { seed: { uri: "x.playground.ts#seed", receipt: { id: "t1" } } }, + ], + }); + const run = await runner.start(evals, def.id); + await runner.finished(run.id); + + const { rows } = await evals.listResults(run.id); + expect(rows[0].rowResources).toEqual({ + seed: { uri: "x.playground.ts#seed" }, + }); + expect(rows[1].rowResources).toBeUndefined(); + }); + it("is idle once every run in flight has finished", async () => { const { runner, evals, def } = await setUp(); await runner.idle(); diff --git a/src/eval/eval-runner.ts b/src/eval/eval-runner.ts index f84c09e..554097e 100644 --- a/src/eval/eval-runner.ts +++ b/src/eval/eval-runner.ts @@ -30,6 +30,7 @@ import type { ResolvedPromptInputs, } from "../prompt/prompt-provider.ts"; import { runPrompt } from "../server/run-prompt.ts"; +import { stripReceipts } from "../shared/dataset-cells.ts"; import { evalProblems } from "../shared/eval-problems.ts"; import type { NormalizedPrompt, PromptRef } from "../shared/types.ts"; import type { TraceProvider } from "../trace/trace-provider.ts"; @@ -501,6 +502,7 @@ export class EvalRunner { ...key, rowIndex: job.rowIndex, rowCells: job.row.cells, + ...rowResourcesOf(job.row), ...(traceId && { traceProviderId: this.options.traceProviderId, traceId, @@ -671,6 +673,7 @@ export class EvalRunner { ...key, rowIndex: job.rowIndex, rowCells: job.row.cells, + ...rowResourcesOf(job.row), traceProviderId: this.options.traceProviderId, traceId, ...(version && { version }), @@ -800,3 +803,13 @@ interface RowResults { row: EvalRowResult; checks: EvalCheckResult[]; } + +/** + * A result's snapshot of `row`'s own resource instances, receipts stripped — + * so its `instance` cells still say what they named once the row changes. + */ +function rowResourcesOf(row: DatasetRow): Pick { + return row.resources && Object.keys(row.resources).length > 0 + ? { rowResources: stripReceipts(row.resources) } + : {}; +} diff --git a/src/eval/eval-types.ts b/src/eval/eval-types.ts index 528b944..1d978e6 100644 --- a/src/eval/eval-types.ts +++ b/src/eval/eval-types.ts @@ -202,6 +202,12 @@ export interface EvalRowResult { rowIndex: number; /** The row's cells as they were run — readable after the row is deleted. */ rowCells: Record; + /** + * The row's own resource instances as they were run, receipts stripped — + * what `rowCells`' `instance` references named then, however the row has + * changed since. See `specs/resource-instances.md` §C. + */ + rowResources?: RunResources; traceProviderId?: string; traceId?: string; /** The version the row ran against, on a clean tree. */ diff --git a/src/eval/turso-eval-provider.ts b/src/eval/turso-eval-provider.ts index e70e5b7..50a659e 100644 --- a/src/eval/turso-eval-provider.ts +++ b/src/eval/turso-eval-provider.ts @@ -10,7 +10,7 @@ import type { Database } from "@tursodatabase/sync"; import { and, desc, eq, inArray, isNotNull, sql } from "drizzle-orm"; import { drizzle } from "drizzle-orm/tursodatabase-sync"; -import type { ExecutionInput } from "../shared/types.ts"; +import type { ExecutionInput, RunResources } from "../shared/types.ts"; import type { PromptID } from "../trace/trace-types.ts"; import { evalCheckResults, @@ -96,6 +96,9 @@ function rowToRowResult(row: RowResultRow): EvalRowResult { sample: row.sample, rowIndex: row.rowIndex, rowCells: JSON.parse(row.rowCells) as Record, + ...(row.rowResources && { + rowResources: JSON.parse(row.rowResources) as RunResources, + }), ...(row.traceProviderId && { traceProviderId: row.traceProviderId }), ...(row.traceId && { traceId: row.traceId }), ...(row.version && { version: row.version }), @@ -516,6 +519,9 @@ export class TursoEvalProvider implements EvalProvider { sample: result.sample, rowIndex: result.rowIndex, rowCells: JSON.stringify(result.rowCells), + rowResources: result.rowResources + ? JSON.stringify(result.rowResources) + : null, traceProviderId: result.traceProviderId ?? null, traceId: result.traceId ?? null, version: result.version ?? null, diff --git a/src/prompt/execution-inputs.test.ts b/src/prompt/execution-inputs.test.ts index c9a4f33..1039027 100644 --- a/src/prompt/execution-inputs.test.ts +++ b/src/prompt/execution-inputs.test.ts @@ -263,6 +263,17 @@ describe("resolveExecutionInput", () => { }); }); +describe("resolveExecutionInput: unsupported kinds", () => { + it("fails on a kind it can't resolve rather than passing undefined", async () => { + await expect( + resolveExecutionInput({ + kind: "resource", + uri: "x#db", + } as unknown as ExecutionInput), + ).rejects.toThrow("Unsupported input kind 'resource'"); + }); +}); + describe("resolveExecutionInputs", () => { it("resolves both halves through one resolver, so a shared instance is created once", async () => { const { resolver, created } = fakeResolver(); @@ -676,6 +687,22 @@ describe("inputReferenceProblems", () => { expect(problems[1]).toMatch(/cycle: b → c → b/); }); + it("refuses the removed inline `resource` input, naming its slot", () => { + const problems = inputReferenceProblems( + { + functionInputs: {}, + executeInputs: { + db: { kind: "resource", uri: "x#db" } as unknown as ExecutionInput, + }, + }, + { functionParameters: [], executeParameters: [str("db")] }, + ); + expect(problems).toHaveLength(1); + expect(problems[0]).toMatch( + /^'execute:db' uses an inline 'resource' input/, + ); + }); + it("names an instance the run doesn't declare, unless rows may declare it", () => { const bindings: InputBindings = { functionInputs: { a: { kind: "instance", name: "db" } }, diff --git a/src/prompt/execution-inputs.ts b/src/prompt/execution-inputs.ts index 0a77462..e7f69fa 100644 --- a/src/prompt/execution-inputs.ts +++ b/src/prompt/execution-inputs.ts @@ -262,6 +262,14 @@ async function resolveWithin( if (!node) return undefined; return readPath(await recurse(node, [...chain, key]), rest); } + + default: + // A dataset cell isn't checked up front the way bindings are (see + // `inputReferenceProblems`), so one holding a removed kind — the old + // inline `resource` — fails here rather than passing `undefined`. + throw new Error( + `Unsupported input kind '${(input as { kind: string }).kind}'`, + ); } } diff --git a/src/prompt/playground/resource-registry.test.ts b/src/prompt/playground/resource-registry.test.ts index d384520..2675a99 100644 --- a/src/prompt/playground/resource-registry.test.ts +++ b/src/prompt/playground/resource-registry.test.ts @@ -1132,6 +1132,99 @@ describe("named instances (specs/resource-instances.md)", () => { ); }); + it("fails, rather than hanging, when instances acquired in parallel wait on each other", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + await lease.declare({ + a: { + uri: "x.playground.ts#task", + binding: titled("A", async () => ({ + parentId: await lease.acquire("b", "id"), + })), + }, + b: { + uri: "x.playground.ts#task", + binding: titled("B", async () => ({ + parentId: await lease.acquire("a", "id"), + })), + }, + }); + // Each starts on its own path, so neither's chain holds the other. + const results = await Promise.allSettled([ + lease.acquire("a"), + lease.acquire("b"), + ]); + expect(results.map(r => r.status)).toEqual(["rejected", "rejected"]); + expect(String((results[0] as PromiseRejectedResult).reason)).toMatch( + /cycle: .*→ .*→/i, + ); + await lease.release(); + }); + + it("fails, rather than hanging, on a cycle through a code-wired dependency", async () => { + const { registry: reg } = registry({ + [p("x.playground.ts")]: `${importHelper} + ${schemaHelper} + export const y = resource({ + inputs: { ref: str() }, + create: ({ ref }) => ({ value: { id: "y", ref } }), + }); + export const x = resource({ + inputs: { y }, + create: () => ({ value: { id: "x" } }), + });`, + }); + const lease = reg.lease(); + await lease.declare({ + x: { uri: "x.playground.ts#x" }, + y: { + uri: "x.playground.ts#y", + binding: { + resolve: async () => ({ ref: await lease.acquire("x", "id") }), + }, + }, + }); + const results = await Promise.allSettled([ + lease.acquire("x"), + lease.acquire("y"), + ]); + expect(results.map(r => r.status)).toEqual(["rejected", "rejected"]); + expect(String((results[0] as PromiseRejectedResult).reason)).toMatch( + /cycle/i, + ); + await lease.release(); + }); + + it("creates nothing once released, so a creation still in flight can't make a second root", async () => { + const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); + const lease = reg.lease(); + let releaseNow!: () => void; + const released = new Promise(resolve => { + releaseNow = resolve; + }); + await lease.declare({ + root: { uri: "x.playground.ts#task", binding: titled("Root") }, + child: { + uri: "x.playground.ts#task", + binding: titled("Child", async () => { + // The run ends (another instance failed, say) while this one is + // still resolving its arguments. + await released; + return { parentId: await lease.acquire("root", "id") }; + }), + }, + }); + await lease.acquire("root"); + const child = lease.acquire("child"); + await lease.release(); + releaseNow(); + await expect(child).rejects.toThrow(/already finished/); + expect((globalThis as any).__created.map((t: any) => t.title)).toEqual([ + "Root", + ]); + expect((globalThis as any).__disposed).toHaveLength(1); + }); + it("wires a code dependency to the run's one declared instance of it, arguments and all", async () => { const { registry: reg } = registry({ [p("x.playground.ts")]: tasksModule }); const lease = reg.lease(); diff --git a/src/prompt/playground/resource-registry.ts b/src/prompt/playground/resource-registry.ts index 1cf926c..814be0a 100644 --- a/src/prompt/playground/resource-registry.ts +++ b/src/prompt/playground/resource-registry.ts @@ -596,6 +596,51 @@ interface LeaseState { declared: Map; /** Receipts recorded so far, by instance name. */ receipts: Map; + /** Set by `release()`: nothing more may be created for this run. */ + released: boolean; + /** + * Which instance creations are waiting on which, by {@link nodeId}: the + * wait-for graph that catches a cycle `chain` can't — two creations that + * started on separate async paths (declared instances are acquired in + * parallel) and then each await the other's pending promise. + */ + waits: Map; + /** What each {@link nodeId} is called in a cycle error. */ + labels: Map; +} + +/** A stable id per resource object, for {@link nodeId}. */ +const resourceIds = new WeakMap, number>(); +let nextResourceId = 0; + +/** Identifies one instance — (resource, memo key) — in a lease's wait-for graph. */ +function nodeId(resource: Resource, key: string): string { + let id = resourceIds.get(resource); + if (id === undefined) { + id = nextResourceId++; + resourceIds.set(resource, id); + } + return `${id}:${key}`; +} + +/** A path from `from` to `to` along `waits`, both ends included; `undefined` if none. */ +function waitPath( + waits: ReadonlyMap, + from: string, + to: string, +): string[] | undefined { + const seen = new Set(); + const walk = (at: string): string[] | undefined => { + if (at === to) return [at]; + if (seen.has(at)) return undefined; + seen.add(at); + for (const next of waits.get(at) ?? []) { + const rest = walk(next); + if (rest) return [at, ...rest]; + } + return undefined; + }; + return walk(from); } /** @@ -814,8 +859,10 @@ export class ResourceRegistry { resetLocks: this.resetLocks.forLease(), declared: new Map(), receipts: new Map(), + released: false, + waits: new Map(), + labels: new Map(), }; - let released = false; const declare = async ( instances: Record, @@ -897,8 +944,8 @@ export class ResourceRegistry { resolveDeclared, receipts: () => Object.fromEntries(state.receipts), release: async () => { - if (released) return; - released = true; + if (state.released) return; + state.released = true; const pending = [...state.runInstances.values()].flatMap(byKey => [ ...byKey.values(), ]); @@ -987,6 +1034,12 @@ export class ResourceRegistry { .join(" → ")}`, ); } + // After `release()` the run's memo is gone: creating now would make a + // second instance of something already disposed, and nothing would + // dispose this one. + if (state.released) { + throw new Error(`Resource '${label}': the run has already finished`); + } const scope = target.scope ?? defaultScope(target); const cache = @@ -998,7 +1051,25 @@ export class ResourceRegistry { cache.set(target, byKey); } + // The creation this one is needed by, if any, now waits on it. + const self = nodeId(target, key); + const waiter = chain.at(-1); + const waiterId = waiter && nodeId(waiter.resource, waiter.key); + state.labels.set(self, label); + let pending = byKey.get(key); + if (pending && waiterId) { + // Joining a creation already under way: if it (transitively) waits on + // the one asking, neither would ever finish. + const loop = waitPath(state.waits, self, waiterId); + if (loop) { + throw new Error( + `Resource dependency cycle: ${[waiterId, ...loop] + .map(id => state.labels.get(id) ?? id) + .join(" → ")}`, + ); + } + } if (!pending) { const nextChain = [...chain, { resource: target, key, label }]; // Ambient for the duration of `create()` — including its `await @@ -1033,7 +1104,19 @@ export class ResourceRegistry { } } - const instance = await pending; + let instance: Instance; + if (waiterId) { + const edges = state.waits.get(waiterId) ?? []; + edges.push(self); + state.waits.set(waiterId, edges); + try { + instance = await pending; + } finally { + edges.splice(edges.indexOf(self), 1); + } + } else { + instance = await pending; + } if (scope === "server" && instance.reset) { // Sorted by the resource's own uri, which every lease agrees on — not diff --git a/src/shared/eval-prefill.ts b/src/shared/eval-prefill.ts index a949a19..471e869 100644 --- a/src/shared/eval-prefill.ts +++ b/src/shared/eval-prefill.ts @@ -9,6 +9,7 @@ import type { DatasetField } from "../dataset/dataset-types.ts"; import type { EvalCheck, EvalInputs } from "../eval/eval-types.ts"; +import { instanceNames } from "./input-references.ts"; import type { CheckInfo, ExecutionInput, @@ -131,18 +132,6 @@ function fillSlot( return current; } -/** Every instance name `input` references, at any depth. */ -function instanceNames(input: ExecutionInput): string[] { - switch (input.kind) { - case "instance": - return [input.name]; - case "object": - return Object.values(input.properties).flatMap(instanceNames); - default: - return []; - } -} - /** * `own`, plus every instance of `stored` that the bindings (or those * instances' own arguments) reference and `own` doesn't already declare — so diff --git a/src/shared/input-references.ts b/src/shared/input-references.ts index 1d8b66f..0d418f4 100644 --- a/src/shared/input-references.ts +++ b/src/shared/input-references.ts @@ -140,8 +140,36 @@ function locationKey(loc: RefLocation): string { : inputKey(loc.half, loc.path); } -/** Every `input` and `instance` reference inside `bindings`, with where it sits. */ -function findRefs(bindings: InputBindings): FoundRef[] { +/** The input kinds a run can resolve; anything else is refused up front. */ +const KNOWN_KINDS: ReadonlySet = new Set([ + "value", + "object", + "input", + "instance", + "dataset", +]); + +/** Every instance name `input` references, at any depth. */ +export function instanceNames(input: ExecutionInput): string[] { + switch (input.kind) { + case "instance": + return [input.name]; + case "object": + return Object.values(input.properties).flatMap(instanceNames); + default: + return []; + } +} + +/** + * Every `input` and `instance` reference inside `bindings`, with where it + * sits. A node of a kind no run can resolve — the removed inline `resource` + * — goes to `unsupported`, when given. + */ +function findRefs( + bindings: InputBindings, + unsupported?: { at: RefLocation; kind: string }[], +): FoundRef[] { const found: FoundRef[] = []; const walk = (node: ExecutionInput, at: RefLocation) => { switch (node.kind) { @@ -160,6 +188,9 @@ function findRefs(bindings: InputBindings): FoundRef[] { found.push({ at, target: { instance: node.name } }); break; default: + if (!KNOWN_KINDS.has(node.kind)) { + unsupported?.push({ at, kind: (node as { kind: string }).kind }); + } break; } }; @@ -277,9 +308,19 @@ export function inputReferenceProblems( const slots = slotPaths(signature); const instances = bindings.resources ?? {}; const problems: string[] = []; - for (const ref of findRefs(bindings)) { - const at = - "instance" in ref.at ? locationKey(ref.at) : `'${locationKey(ref.at)}'`; + const unsupported: { at: RefLocation; kind: string }[] = []; + const refs = findRefs(bindings, unsupported); + const where = (loc: RefLocation) => + "instance" in loc ? locationKey(loc) : `'${locationKey(loc)}'`; + for (const { at, kind } of unsupported) { + problems.push( + kind === "resource" + ? `${where(at)} uses an inline 'resource' input, which is no longer supported: declare the resource in 'resources' and reference it with { kind: "instance" }` + : `${where(at)} has an input of unknown kind '${kind}'`, + ); + } + for (const ref of refs) { + const at = where(ref.at); if ("instance" in ref.target) { if ( !options.undeclaredInstances && From bafa6b8afaaf264017f225840992eeec693cc113 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 09:40:17 +0000 Subject: [PATCH 10/12] Add the license header to the row_resources migration Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- .../db/migrations/20261009093329_soft_pandemic/migration.sql | 3 +++ src/eval/db/migrations/bundled.ts | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql b/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql index 14c0077..113de53 100644 --- a/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql +++ b/src/eval/db/migrations/20261009093329_soft_pandemic/migration.sql @@ -1 +1,4 @@ +-- SPDX-License-Identifier: AGPL-3.0-only +-- Copyright (c) 2026 Alexander Corrado + ALTER TABLE `eval_row_results` ADD `row_resources` text; \ No newline at end of file diff --git a/src/eval/db/migrations/bundled.ts b/src/eval/db/migrations/bundled.ts index 27cb068..70cf264 100644 --- a/src/eval/db/migrations/bundled.ts +++ b/src/eval/db/migrations/bundled.ts @@ -25,11 +25,11 @@ export const bundledMigrations: MigrationMeta[] = [ }, { "name": "20261009093329_soft_pandemic", - "hash": "e8916ff94f63623ce05055ed9e6522a447a24dfeac99214e6ec4c2f1efc954be", + "hash": "6f73eb4f63048e9178c61e72d82312f4130f10129875644305d7b4185d018465", "folderMillis": 1791538409000, "bps": true, "sql": [ - "ALTER TABLE `eval_row_results` ADD `row_resources` text;" + "-- SPDX\u002DLicense-Identifier: AGPL-3.0-only\n-- Copyright (c) 2026 Alexander Corrado\n\nALTER TABLE `eval_row_results` ADD `row_resources` text;" ] } ]; From 8dc68cba7fbda1c38ad496b784c36af2f7208137 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:52:58 +0000 Subject: [PATCH 11/12] Address re-review: honest eval chips, kind checks, dispose logs, MCP - A name an eval doesn't declare reads "not declared here: each row must declare it" rather than as if it were fine. - KNOWN_KINDS is checked against the ExecutionInput union with satisfies, and resolveWithin's default is a never exhaustiveness check. - Releasing a lease no longer logs a failed or refused create as a failed dispose. - get_eval_run shows each result's row resources beside its inputs. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- src/client/components/pseudo-sources.test.ts | 5 +- src/client/components/pseudo-sources.ts | 6 +- src/mcp/eval-tools.test.ts | 62 +++++++++++++++++++ src/mcp/eval-tools.ts | 19 +++++- src/prompt/execution-inputs.ts | 7 ++- .../playground/resource-registry.test.ts | 4 ++ src/prompt/playground/resource-registry.ts | 6 +- src/shared/input-references.ts | 16 ++--- 8 files changed, 110 insertions(+), 15 deletions(-) create mode 100644 src/mcp/eval-tools.test.ts diff --git a/src/client/components/pseudo-sources.test.ts b/src/client/components/pseudo-sources.test.ts index 4cce2ed..9ecf5c2 100644 --- a/src/client/components/pseudo-sources.test.ts +++ b/src/client/components/pseudo-sources.test.ts @@ -128,7 +128,10 @@ describe("withInstanceSources", () => { describeInstanceSource(instanceUri("task"), instances, byUri, { rowsMayDeclare: true, }), - ).toEqual({ label: "task", note: "declared by each row" }); + ).toEqual({ + label: "task", + note: "not declared here: each row must declare it", + }); }); }); diff --git a/src/client/components/pseudo-sources.ts b/src/client/components/pseudo-sources.ts index e5bcfc4..5bece4f 100644 --- a/src/client/components/pseudo-sources.ts +++ b/src/client/components/pseudo-sources.ts @@ -505,8 +505,8 @@ export function withInstanceSources( * no longer has it. * * @param options.rowsMayDeclare - In an eval, a name the eval doesn't - * declare is one each dataset row declares (`specs/resource-instances.md` - * §E), so it reads as that rather than as missing. + * declare may be one each dataset row declares (`specs/resource-instances.md` + * §E), so it reads as that requirement rather than as missing. */ export function describeInstanceSource( uri: string, @@ -521,7 +521,7 @@ export function describeInstanceSource( ref.output === undefined ? ref.name : `${ref.name}.${ref.output}`; if (!instance) { return options.rowsMayDeclare - ? { label, note: "declared by each row" } + ? { label, note: "not declared here: each row must declare it" } : { label, note: "resource no longer in this run", missing: true }; } const resource = resourcesByUri.get(instance.uri); diff --git a/src/mcp/eval-tools.test.ts b/src/mcp/eval-tools.test.ts new file mode 100644 index 0000000..2b15385 --- /dev/null +++ b/src/mcp/eval-tools.test.ts @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Copyright (c) 2026 Alexander Corrado + +import { describe, expect, it } from "vitest"; +import type { DatasetField } from "../dataset/dataset-types.ts"; +import { resultInputs } from "./eval-tools.ts"; + +const FIELDS: DatasetField[] = [ + { + id: "0", + def: { + name: "title", + type: { kind: "primitive", syntax: "string", base: "string" }, + optional: false, + }, + }, + { + id: "1", + def: { + name: "taskId", + type: { kind: "primitive", syntax: "string", base: "string" }, + optional: false, + }, + }, +]; + +describe("resultInputs", () => { + it("shows the row's resources as run beside the cells that name them", () => { + expect( + resultInputs( + { + rowCells: { + "0": { kind: "value", value: { kind: "primitive", value: "Hi" } }, + "1": { kind: "instance", name: "task", output: "id" }, + }, + rowResources: { + task: { + uri: "x.playground.ts#task", + args: { title: { kind: "dataset", field: "0" } }, + }, + }, + }, + FIELDS, + ), + ).toEqual({ + inputs: { + title: "Hi", + taskId: { kind: "instance", name: "task", output: "id" }, + }, + resources: { + task: { + uri: "x.playground.ts#task", + args: { title: expect.anything() }, + }, + }, + }); + }); + + it("leaves resources out for a row that declared none", () => { + expect(resultInputs({ rowCells: {} }, FIELDS)).toEqual({ inputs: {} }); + }); +}); diff --git a/src/mcp/eval-tools.ts b/src/mcp/eval-tools.ts index 25c0593..117fb8b 100644 --- a/src/mcp/eval-tools.ts +++ b/src/mcp/eval-tools.ts @@ -20,6 +20,7 @@ import type { EvalCheckResult, EvalDefinition, EvalInputs, + EvalRowResult, EvalRun, EvalRunSummary, EvalSummary, @@ -155,6 +156,22 @@ function describeBinding( } } +/** + * What a stored result ran with: its row's cells, and the row's own resource + * instances as they were then — which its `instance` cells name. + */ +export function resultInputs( + row: Pick, + fields: readonly DatasetField[], +): { inputs: Record; resources?: Record } { + return { + inputs: describeCells(row.rowCells, fields), + ...(row.rowResources && { + resources: describeResources(row.rowResources, fields), + }), + }; +} + /** A stored eval's resource instances as a tool shows them, their arguments described as bindings are. */ function describeResources( resources: RunResources, @@ -892,7 +909,7 @@ export function registerEvalTools( armId: row.armId, status: row.status, ...(row.error && { error: row.error }), - inputs: describeCells(row.rowCells, fields ?? []), + ...resultInputs(row, fields ?? []), ...(row.traceId && { traceProviderId: row.traceProviderId, traceId: row.traceId, diff --git a/src/prompt/execution-inputs.ts b/src/prompt/execution-inputs.ts index e7f69fa..645abfb 100644 --- a/src/prompt/execution-inputs.ts +++ b/src/prompt/execution-inputs.ts @@ -263,13 +263,16 @@ async function resolveWithin( return readPath(await recurse(node, [...chain, key]), rest); } - default: + default: { // A dataset cell isn't checked up front the way bindings are (see // `inputReferenceProblems`), so one holding a removed kind — the old // inline `resource` — fails here rather than passing `undefined`. + // Typed `never`: a new kind left unhandled above won't compile. + const unsupported: never = input; throw new Error( - `Unsupported input kind '${(input as { kind: string }).kind}'`, + `Unsupported input kind '${(unsupported as { kind: string }).kind}'`, ); + } } } diff --git a/src/prompt/playground/resource-registry.test.ts b/src/prompt/playground/resource-registry.test.ts index 2675a99..317e6f4 100644 --- a/src/prompt/playground/resource-registry.test.ts +++ b/src/prompt/playground/resource-registry.test.ts @@ -1216,9 +1216,13 @@ describe("named instances (specs/resource-instances.md)", () => { }); await lease.acquire("root"); const child = lease.acquire("child"); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); await lease.release(); releaseNow(); await expect(child).rejects.toThrow(/already finished/); + // Refused, not a dispose that failed: nothing to warn about. + expect(warn).not.toHaveBeenCalled(); + warn.mockRestore(); expect((globalThis as any).__created.map((t: any) => t.title)).toEqual([ "Root", ]); diff --git a/src/prompt/playground/resource-registry.ts b/src/prompt/playground/resource-registry.ts index 814be0a..60a7e2e 100644 --- a/src/prompt/playground/resource-registry.ts +++ b/src/prompt/playground/resource-registry.ts @@ -953,8 +953,12 @@ export class ResourceRegistry { state.resetLocks.releaseAll(); await Promise.all( pending.map(async p => { + // A create that failed (or was refused once released) left + // nothing to dispose; only a real dispose failure is worth a + // warning. + const instance = await Promise.resolve(p).catch(() => undefined); try { - await (await p).dispose?.(); + await instance?.dispose?.(); } catch (err) { console.warn("failed to dispose playground resource:", err); } diff --git a/src/shared/input-references.ts b/src/shared/input-references.ts index 0d418f4..a2e6517 100644 --- a/src/shared/input-references.ts +++ b/src/shared/input-references.ts @@ -141,13 +141,15 @@ function locationKey(loc: RefLocation): string { } /** The input kinds a run can resolve; anything else is refused up front. */ -const KNOWN_KINDS: ReadonlySet = new Set([ - "value", - "object", - "input", - "instance", - "dataset", -]); +const KNOWN_KINDS: ReadonlySet = new Set( + Object.keys({ + value: true, + object: true, + input: true, + instance: true, + dataset: true, + } satisfies Record), +); /** Every instance name `input` references, at any depth. */ export function instanceNames(input: ExecutionInput): string[] { From dfc084e1da0bc9856ddfa4ab0b75850a2c076b89 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 18:59:52 +0000 Subject: [PATCH 12/12] Share disposeAll between lease release and invalidate A server-scoped create that fails while the registry is invalidated is no longer logged as a failed dispose either. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BRiCr7FZZ7dZf9aQsxTUru --- .../playground/resource-registry.test.ts | 25 +++++++++++ src/prompt/playground/resource-registry.ts | 44 +++++++++---------- 2 files changed, 47 insertions(+), 22 deletions(-) diff --git a/src/prompt/playground/resource-registry.test.ts b/src/prompt/playground/resource-registry.test.ts index 317e6f4..0bcf33f 100644 --- a/src/prompt/playground/resource-registry.test.ts +++ b/src/prompt/playground/resource-registry.test.ts @@ -266,6 +266,31 @@ describe("resource lifecycle", () => { expect((globalThis as any).__serverDisposes).toBe(1); }); + it("doesn't call a server-scoped create that fails mid-invalidate a failed dispose", async () => { + const { registry: reg } = registry({ + [p("x.playground.ts")]: `${importHelper} + export const thing = resource({ + scope: "server", + create: () => new Promise((_, reject) => { + globalThis.__failThing = () => reject(new Error("boom")); + }), + });`, + }); + const lease = reg.lease(); + const thing = acquireUri(lease, "x.playground.ts#thing"); + await vi.waitFor(() => + expect((globalThis as any).__failThing).toBeDefined(), + ); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const invalidated = reg.invalidate(); + (globalThis as any).__failThing(); + await invalidated; + await expect(thing).rejects.toThrow("boom"); + expect(warn).not.toHaveBeenCalled(); + warn.mockRestore(); + await lease.release(); + }); + it("rejects a dependency cycle at resolution rather than overflowing", async () => { const { registry: reg } = registry({ [p("cycle.playground.ts")]: `${importHelper} diff --git a/src/prompt/playground/resource-registry.ts b/src/prompt/playground/resource-registry.ts index 60a7e2e..7ce6ec8 100644 --- a/src/prompt/playground/resource-registry.ts +++ b/src/prompt/playground/resource-registry.ts @@ -609,6 +609,26 @@ interface LeaseState { labels: Map; } +/** + * Disposes every instance in `pending`, settled or not. A create that failed + * (or was refused once its lease was released) left nothing to dispose, so + * only a real dispose failure is worth a warning. + */ +async function disposeAll( + pending: readonly (Instance | Promise)[], +): Promise { + await Promise.all( + pending.map(async p => { + const instance = await Promise.resolve(p).catch(() => undefined); + try { + await instance?.dispose?.(); + } catch (err) { + console.warn("failed to dispose playground resource:", err); + } + }), + ); +} + /** A stable id per resource object, for {@link nodeId}. */ const resourceIds = new WeakMap, number>(); let nextResourceId = 0; @@ -837,15 +857,7 @@ export class ResourceRegistry { ...byKey.values(), ]); this.serverInstances = new Map(); - await Promise.all( - pending.map(async p => { - try { - await (await p).dispose?.(); - } catch (err) { - console.warn("failed to dispose playground resource:", err); - } - }), - ); + await disposeAll(pending); } /** @@ -951,19 +963,7 @@ export class ResourceRegistry { ]); state.runInstances.clear(); state.resetLocks.releaseAll(); - await Promise.all( - pending.map(async p => { - // A create that failed (or was refused once released) left - // nothing to dispose; only a real dispose failure is worth a - // warning. - const instance = await Promise.resolve(p).catch(() => undefined); - try { - await instance?.dispose?.(); - } catch (err) { - console.warn("failed to dispose playground resource:", err); - } - }), - ); + await disposeAll(pending); }, }; }