Skip to content

Fix/tool calls default metric - #12

Merged
jay-m-dev merged 3 commits into
mainfrom
fix/tool-calls-default-metric
Sep 22, 2026
Merged

jay-m-dev merged 3 commits into
mainfrom
fix/tool-calls-default-metric

Conversation

@jay-m-dev

Copy link
Copy Markdown
Collaborator

This pull request updates the handling of built-in runtime metrics, especially contextual metrics like tool calls and critic gates, to ensure only metrics that can actually be produced by the current canvas are pre-selected or enabled by default. It also removes "tool_calls" from the list of recommended metrics for new experiments, and updates related logic and tests accordingly. Additionally, it improves how node run statuses are sourced and displayed in the protocol canvas, ensuring consistency between test and real runs.

Metrics selection and availability improvements:

  • Only metrics that the canvas can actually produce are pre-selected by default in the metrics editor, preventing users from saving metrics that can never report data. Contextual metrics (like tool_calls and critic_approvals) are now disabled and not pre-checked if the canvas lacks the necessary nodes. [1] [2] [3]
  • The checkbox for an unavailable metric is now only disabled if it is not already selected, allowing users to uncheck metrics that became unavailable after being selected.

Recommended metrics changes:

  • "tool_calls" is removed from the list of recommended runtime metrics, so new experiments will not include it by default unless the canvas supports it. Corresponding test expectations and catalog flags are updated. [1] [2] [3] [4] [5] [6]

Protocol canvas node run status consistency:

  • The protocol canvas now consistently uses the latest available node run data (from either a real run or a test run) for node badges and inspectors, ensuring UI consistency and correct display after page reloads or test runs. [1] [2] [3] [4] [5]

Metrics plan synchronization:

  • When saving metric selections, the logic now ensures that deselected metrics are fully removed from both the UI and the measurement plan, even if the design's metric list has changed, preventing potential errors on save.

Jay Moran added 3 commits September 22, 2026 14:50
An experiment is created before its canvas exists, so the default set can't
know whether a tool will be wired. On a canvas without one, Tool calls can
never measure anything. The metrics editor already offers it once a tool is
attached.
Deselecting a built-in only removed it from the plan if design_spec.metrics
also declared it. When that list was missing, the plan kept naming metrics
nothing declared and every save and publish failed with 422. Removal now
also goes by the plan's runtime outputs, and existing plan ids are reused.

An unavailable metric that is already selected can now be unchecked, and
the default 'Add metrics' selection skips tool and critic metrics the
canvas can't produce.
Node inspectors and badges read only the run held in page state, which a
reload re-seeds from list_protocol_runs -- and that list excludes test
runs. So a Test Run's Input/Output disappeared on return while its results
panel still showed it. Read the newest of the watched run and the latest
Test Run instead, as config findings already did.
@jay-m-dev
jay-m-dev merged commit 5225329 into main Sep 22, 2026
2 checks passed
@jay-m-dev
jay-m-dev deleted the fix/tool-calls-default-metric branch September 22, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant