Fix/tool calls default metric - #12
Merged
Merged
Conversation
added 3 commits
September 22, 2026 14:50
An experiment is created before its canvas exists, so the default set can't know whether a tool will be wired. On a canvas without one, Tool calls can never measure anything. The metrics editor already offers it once a tool is attached.
Deselecting a built-in only removed it from the plan if design_spec.metrics also declared it. When that list was missing, the plan kept naming metrics nothing declared and every save and publish failed with 422. Removal now also goes by the plan's runtime outputs, and existing plan ids are reused. An unavailable metric that is already selected can now be unchecked, and the default 'Add metrics' selection skips tool and critic metrics the canvas can't produce.
Node inspectors and badges read only the run held in page state, which a reload re-seeds from list_protocol_runs -- and that list excludes test runs. So a Test Run's Input/Output disappeared on return while its results panel still showed it. Read the newest of the watched run and the latest Test Run instead, as config findings already did.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates the handling of built-in runtime metrics, especially contextual metrics like tool calls and critic gates, to ensure only metrics that can actually be produced by the current canvas are pre-selected or enabled by default. It also removes "tool_calls" from the list of recommended metrics for new experiments, and updates related logic and tests accordingly. Additionally, it improves how node run statuses are sourced and displayed in the protocol canvas, ensuring consistency between test and real runs.
Metrics selection and availability improvements:
tool_callsandcritic_approvals) are now disabled and not pre-checked if the canvas lacks the necessary nodes. [1] [2] [3]Recommended metrics changes:
Protocol canvas node run status consistency:
Metrics plan synchronization: