feat(traces): superadmin source-code context for the evaluator#35
Merged
Conversation
…e enum The tool's topic param is constrained to an enum of real doc names, but the description and system prompt told the model to "pass an unknown topic to list available topics". The AI SDK rejects out-of-enum args before execute() runs, so the model never reached the listing fallback — it just got repeated validation errors and gave up. List the valid topics inline (in the description and prompt) so the model picks a valid enum value on the first call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nfig Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eTools Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Whitelist json-layout/json-layout in the github proxy and point explore_github at core/src/webmcp/tools/ — the MCP form tools backing the *_form form sub-agents (pageConfig_form, portalConfig_form). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Give superadmins a restricted GitHub source-exploration tool in the trace evaluator, and fix the
readArchitectureDoctool it complements.What changed:
readArchitectureDoc: itsenum-constrainedtopiccontradicted a description telling the model to "pass an unknown topic" to list topics, so the AI SDK rejected the call beforeexecuteand the model looped on validation errors. Valid topics are now listed inline and the contradictory instruction removed.GET /api/admin/github, scoped todata-fair/{agents,data-fair,portals}andjson-layout/json-layout, with an optionalGITHUB_TOKEN(unauthenticated fallback + a startup hint when unset).explore_githubevaluator tool calling that proxy, gated onadminMode. Its description points at each repo's starting paths (incl. json-layoutcore/src/webmcp/tools/, the MCP form tools behind the*_formsub-agents). Account admins keep the docs-only evaluator;readArchitectureDocis retained for everyone as the cheap first hop.Why: the evaluator's curated architecture docs are lossy; superadmins need source as ground truth (actual prompts, tool schemas, the assistant's tools in data-fair/portals, and the json-layout form-MCP tools) to make precise recommendations.
Heads-up: the UI
adminModegate must stay in lockstep with the server'sreqAdminMode— that parity is the access model.api/config/type/.type/is generated bybuild-types; regenerate it whenever the config schema changes.