feat: onboarding skill evals — single-turn + multi-turn with mocked tools - #62
Merged
Conversation
Generalize the multi-turn harness behind a SkillHarness interface (migration behavior unchanged) and add an onboarding toolset that mocks Bash (regex-routed auth/REST/gcloud responses), AskUserQuestion (regex-matched scripted answers), and the onboarding/warehouse MCP tools. Scenarios can script failure sequences via bash_responses and tool_responses. - 13 multi-turn scenarios: default entry, create-account happy + error paths (name taken, under-review verify vs fraud, work email), setup wizard (context schema before targeting, MCP retry, no redundant questions), invite batch, status, warehouse dispatch + BigQuery - 16 single-turn cases: routing, error interpretation, validation, email-derived suggestions, communication; scored by a Next-step verdict footer, content checks, a binary internal-leak denylist, and onboarding-tuned LLM judges - telemetry contract asserted: key acquired, events published, never mentioned in user-visible text - Braintrust experiments onboard-single-turn-v1 / onboard-multi-turn-v1 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new multi-turn evals caught models announcing 'let me send the final telemetry' at flow end — telemetry is the only action left to narrate there, so the transition-text habit beats the existing 'never mention it' rule. Add an explicit never-narrate rule to every skill's telemetry block; the BigQuery scenario passes 11/11 with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bespoyasov
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds automated quality evals for the onboarding skills (
onboard-confidence,setup-warehouse,setup-warehouse-bigquery), extending the eval stack that already covers the four migration skills. Results log to Braintrust asonboard-single-turn-v1/onboard-multi-turn-v1.How
Multi-turn harness generalized. The driver now selects a
SkillHarnessper scenario; migration scenarios are byte-for-byte unchanged. The onboarding harness mocks all three tool surfaces the skill uses:auth.py→ mock JWT, userinfo, availability checks, account creation, telemetry endpoints, gcloud/bq)Scenarios script failure sequences declaratively:
bash_responses(e.g. availability false-then-true) andtool_responses(e.g.getIdentityInfofailing until the user runs/mcp).Coverage. 13 multi-turn scenarios (default entry, create-account happy + 4 error paths incl. under-review verify-vs-fraud disambiguation, setup wizard incl. context-schema-before-targeting and MCP retry, invite batch, status, warehouse dispatch + BigQuery end-to-end) and 16 single-turn cases (routing, error interpretation, validation, email-derived name suggestions, communication rules). The telemetry contract is asserted: key acquired, events published, never mentioned to the user.
Scoring. Deterministic assertions (incl. new
tool_call_arg_not_contains) plus a conversation-level internal-leak LLM judge; single-turn adds a forcedNext step:verdict footer, content checks, a binary leak denylist, and onboarding-tuned judges.Findings
The evals caught a real defect on first run: models narrate "let me send the final telemetry" at flow end, violating the telemetry-transparency rule. Fixed by adding a Never narrate telemetry rule to every skill's telemetry block (second commit); the BigQuery scenario passes 11/11 with it.
Current scores: single-turn 100% on 5/6 scorers (NextStep 93.75%, one nondeterministic miss); multi-turn AssertionsPassed 99.3% pre-fix.
🤖 Generated with Claude Code