feat: onboarding skill evals — single-turn + multi-turn with mocked tools

evals/README.md

@@ -1,10 +1,12 @@
 # Skill Evals
 
 Automated quality evaluation for the migration skills (`migrate-optimizely`,
-`migrate-posthog`, `migrate-eppo`, `migrate-statsig`). Each eval sends a
-skill's full `SKILL.md` as the system prompt plus a source-platform flag
-definition, and scores the model's response on 8 dimensions. Results are
-logged to Braintrust (project **Confidence ai plugins** on
+`migrate-posthog`, `migrate-eppo`, `migrate-statsig`) and the onboarding
+skills (`onboard-confidence`, `setup-warehouse`, `setup-warehouse-bigquery`).
+Each single-turn eval sends a skill's full `SKILL.md` as the system prompt
+plus one case message and scores the model's response; multi-turn evals run
+scripted conversations against mocked tools. Results are logged to
+Braintrust (project **Confidence ai plugins** on
 `braintrust.spotifyinternal.com`).
 
 ## Run it with a Hendrix key (recommended)
@@ -85,6 +87,72 @@ it up automatically. Ground truth is hand-written; if the model disagrees
 with a case, check whether the case (or the skill) is wrong before assuming
 the model is.
 
+## Onboarding evals
+
+The onboarding skill acts through Bash (bundled `auth.py`, curl to REST
+endpoints), AskUserQuestion, and MCP tools — so its evals mock all three.
+
+### Single-turn (`onboard-confidence.eval.ts`, cases in `cases/onboard/`)
+
+Each case is one message (optionally with an `input.context` block holding a
+prior-state summary or a raw API error) answered with no tools. A footer
+(`lib/onboard-footer.ts`) forces a `Next step: <sub-command>.<step>` verdict
+line. Case schema:
+
+```yaml
+name: error-under-review-fraud
+tags: [error-interpretation]
+input:
+  context: |
+    (Conversation so far: ... the API responded: {"code":9,"message":"...flagged as suspicious."})
+  user_message: "So, is my account ready?"
+expected:
+  next_step_pattern: "^create-account"   # regex on the verdict line
+  response_includes: ["confidence-support@spotify.com"]  # all must appear
+  response_includes_any: ["flagged", "review"]           # at least one
+  response_excludes: ["verify your email", "code 9"]     # none in prose
+```
+
+| Scorer | Type | What it checks |
+|--------|------|----------------|
+| NextStep | deterministic | verdict line matches `next_step_pattern` |
+| ResponseContent | deterministic | includes / includes_any / excludes |
+| NoInternalLeak | deterministic | no Auth0 client IDs, JWTs, `Bearer`, org IDs, curl, telemetry mention (binary: any leak = 0) |
+| OnboardCommunication | LLM judge | plain-English status, no payloads/codes/internals |
+| OnboardEducateFirst | LLM judge | concept explained before asking for input |
+| OnboardStepTracker | LLM judge | step tracker present (cases tagged `interactive`) |
+
+### Multi-turn (`multi-turn/onboard-confidence.eval.ts`, cases in `cases/multi-turn/onboard/`)
+
+Scripted conversations against a mock harness (`multi-turn/onboard-tools.ts`):
+
+- **Bash** is regex-routed to canned outputs (auth script → mock JWT,
+  availability checks, account creation, telemetry endpoints, gcloud/bq).
+  Scenarios override per-command with `bash_responses` (consumed in order —
+  e.g. a 409 then a 200).
+- **AskUserQuestion** answers come from `ask_answers` — each entry's `match`
+  regex is tested against the question text, header, and option labels;
+  unmatched questions fall back to the first option with a warning.
+- **MCP tools** are mocked with in-memory state (clients, flags, warehouse);
+  `tool_responses` overrides any tool's results in order (e.g. a failing
+  `getIdentityInfo` before the user authenticates).
+- `skill:` selects the SKILL.md to load (`onboard-confidence`,
+  `setup-warehouse`, `setup-warehouse-bigquery`); `skills:` can list several
+  to concatenate (dispatcher + hand-off target).
+
+Telemetry is asserted, not skipped: happy-path scenarios check the telemetry
+key is acquired and events are published, and that telemetry is never
+mentioned in user-visible text. Scoring is `AssertionsPassed` (declarative
+assertions, including the new `tool_call_arg_not_contains`) plus a
+conversation-level `NoInternalLeak` LLM judge over all user-visible text.
+
+```bash
+npm run eval:onboard:local              # single-turn, Hendrix, no upload
+npm run eval:multi-turn:onboard:local   # multi-turn, Hendrix, no upload
+npm run eval:onboard                    # single-turn → Braintrust (onboard-single-turn-v1)
+npm run eval:multi-turn:onboard         # multi-turn → Braintrust (onboard-multi-turn-v1)
+```
+
 ## CI
 
 PR runs are **disabled** (each full run costs real tokens). The workflow

evals/cases/multi-turn/onboard/create-account-happy.yaml

@@ -0,0 +1,80 @@
+name: create-account-happy-path
+description: >
+  Full create-account flow: signup auth with the signup client ID, workspace
+  name suggestions derived from the signup email, region/auth choices via
+  AskUserQuestion, admin email pre-filled from signup (not re-asked), account
+  created, then re-auth with the regular client ID + org. Telemetry runs
+  silently; the token never appears in user-visible text.
+skill: onboard-confidence
+tags: [multi-turn, create-account, telemetry, happy-path]
+
+conversation:
+  - "I'd like to create a new Confidence account for my company."
+  - "Let's use the workspace name acme."
+  - "Display name: Acme Inc"
+  - "The suggested admin email is fine, and no login domain restrictions."
+  - "Great, that's all — thanks!"
+
+ask_answers:
+  - match: "workspace"
+    answer: "acme"
+  - match: "region|stored"
+    answer: "EU (Europe)"
+  - match: "authentication|google|password"
+    answer: "Google"
+  - match: "admin"
+    answer: "Use jane+test@acme.com"
+  - match: "domain"
+    answer: "No restrictions"
+
+assertions:
+  # Telemetry contract: key acquired, events published, never mentioned.
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "agentTelemetryKey"
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "events:publish"
+  - type: text_not_contains
+    pattern: "telemetry"
+    case_sensitive: false
+  # Signup auth with the signup client, then userinfo for the email.
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "82qMvwZvqd3t3S0gRDvs8R53TehQXSJY"
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "userinfo"
+  # Workspace suggestions derived from jane+test@acme.com.
+  - type: text_contains
+    pattern: "jane"
+    case_sensitive: false
+  # Availability checked before creation.
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "loginIdAvailability"
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "/v1/accounts"
+  # Admin email pre-filled from the signup email.
+  - type: text_contains
+    pattern: "jane+test@acme.com"
+  # Re-auth with the regular client for the org-scoped token.
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w"
+  # Workspace confirmed to the user; no internals leaked.
+  - type: text_contains
+    pattern: "acme"
+    case_sensitive: false
+  - type: text_not_contains
+    pattern: "eyJhbGciOiJub25l"
+  - type: text_not_contains
+    pattern: "org_mock123"

evals/cases/multi-turn/onboard/create-account-name-taken.yaml

@@ -0,0 +1,43 @@
+name: create-account-name-taken
+description: >
+  The chosen workspace name is taken (availability check returns false).
+  The skill must say so in plain English, suggest alternatives, re-check
+  the new name, and complete account creation.
+skill: onboard-confidence
+tags: [multi-turn, create-account, error-recovery]
+
+bash_responses:
+  - match: "loginIdAvailability"
+    responses:
+      - '{"available": false}'
+      - '{"available": true}'
+
+conversation:
+  - >
+    Create a Confidence account. Workspace name: acme. Display name:
+    Acme Inc. Region: EU. Login with Google. Admin email: jane+test@acme.com.
+    No domain restrictions.
+  - "OK, let's use acme-hq instead."
+  - "Yes, go ahead."
+
+ask_answers:
+  - match: "region|stored"
+    answer: "EU (Europe)"
+  - match: "authentication|google|password"
+    answer: "Google"
+
+assertions:
+  - type: text_contains
+    pattern: "taken|not available|unavailable|already in use"
+    regex: true
+    case_sensitive: false
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "acme-hq"
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "/v1/accounts"
+  - type: text_not_contains
+    pattern: "eyJhbGciOiJub25l"

evals/cases/multi-turn/onboard/create-account-under-review-fraud.yaml

@@ -0,0 +1,49 @@
+name: create-account-under-review-fraud
+description: >
+  Account creation returns code 9 with a "flagged as suspicious" message.
+  The skill must NOT assume email verification — it must parse the actual
+  message, point the user to support, and not auto-retry.
+skill: onboard-confidence
+tags: [multi-turn, create-account, error-recovery, under-review, fraud]
+
+bash_responses:
+  - match: "/v1/accounts"
+    responses:
+      - |-
+        {"code":9,"message":"Account creation is under review: your account has been flagged as suspicious."}
+        400
+
+conversation:
+  - >
+    Create a Confidence account. Workspace name: acme. Display name:
+    Acme Inc. Region: EU. Login with Google. Admin email: jane+test@acme.com.
+    No domain restrictions.
+  - "Hmm, okay. What should I do now?"
+
+ask_answers:
+  - match: "region|stored"
+    answer: "EU (Europe)"
+  - match: "authentication|google|password"
+    answer: "Google"
+  - match: "proceed|retry|try again|support"
+    answer: "Not right now — I'll wait"
+
+assertions:
+  - type: text_contains
+    pattern: "confidence-support@spotify.com"
+  - type: text_contains
+    pattern: "flagged|review"
+    regex: true
+    case_sensitive: false
+  # Must not misdiagnose as the email-verification case.
+  - type: text_not_contains
+    pattern: "verify your email"
+    case_sensitive: false
+  - type: text_not_contains
+    pattern: "verification link"
+    case_sensitive: false
+  - type: text_not_contains
+    pattern: "code 9"
+    case_sensitive: false
+  - type: text_not_contains
+    pattern: "eyJhbGciOiJub25l"

evals/cases/multi-turn/onboard/create-account-under-review-verify.yaml

@@ -0,0 +1,49 @@
+name: create-account-under-review-verify
+description: >
+  Account creation returns code 9 with a "verify your email" message. The
+  skill must ask the user to verify (this IS the email-verification case),
+  wait for confirmation, then run the retry loop and succeed. No error
+  codes shown to the user.
+skill: onboard-confidence
+tags: [multi-turn, create-account, error-recovery, under-review]
+
+bash_responses:
+  - match: "/v1/accounts"
+    responses:
+      - |-
+        {"code":9,"message":"Account is under review: please verify your email address before continuing."}
+        400
+      - |-
+        ATTEMPT 1: HTTP=200
+        {"name":"accounts/acme-mock","externalId":"ext-123","loginId":"acme","displayName":"Acme Inc"}
+        SUCCESS
+
+conversation:
+  - >
+    Create a Confidence account. Workspace name: acme. Display name:
+    Acme Inc. Region: EU. Login with Google. Admin email: jane+test@acme.com.
+    No domain restrictions.
+  - "Done — I clicked the verification link in my email."
+  - "Thanks!"
+
+ask_answers:
+  - match: "region|stored"
+    answer: "EU (Europe)"
+  - match: "authentication|google|password"
+    answer: "Google"
+
+assertions:
+  - type: text_contains
+    pattern: "verif"
+    case_sensitive: false
+  - type: text_contains
+    pattern: "acme"
+    case_sensitive: false
+  - type: tool_called_count
+    tool_name: Bash
+    min_count: 2
+  - type: text_not_contains
+    pattern: "code 9"
+    case_sensitive: false
+  - type: text_not_contains
+    pattern: "eyJhbGciOiJub25l"

evals/cases/multi-turn/onboard/create-account-work-email.yaml

@@ -0,0 +1,48 @@
+name: create-account-work-email-rejected
+description: >
+  The user supplies a free-provider admin email. Whether the skill catches
+  it upfront (the skill documents the work-email rule) or the API rejects
+  it, the user must get a plain-English explanation and the account must be
+  created once a work email is provided. No HTTP codes in prose.
+skill: onboard-confidence
+tags: [multi-turn, create-account, error-recovery]
+
+# Only reject requests that actually carry the free-provider email — a model
+# that catches the rule upfront and never sends it gets the default 200.
+bash_responses:
+  - match: 'jane@gmail\.com'
+    responses:
+      - |-
+        {"code":3,"message":"adminEmail must be a work email address; free email providers are not allowed"}
+        400
+
+conversation:
+  - >
+    Create a Confidence account. Workspace name: acme. Display name:
+    Acme Inc. Region: EU. Login with Google. Admin email: jane@gmail.com.
+    No domain restrictions.
+  - "Ah right — use jane@acme.com instead."
+  - "Perfect, thanks!"
+
+ask_answers:
+  - match: "region|stored"
+    answer: "EU (Europe)"
+  - match: "authentication|google|password"
+    answer: "Google"
+  # If the model catches the gmail address upfront and asks for a
+  # replacement before the user's second turn arrives:
+  - match: "admin|gmail|work email"
+    answer: "Use jane@acme.com"
+
+assertions:
+  - type: text_contains
+    pattern: "work email"
+    case_sensitive: false
+  - type: tool_call_arg_contains
+    tool_name: Bash
+    arg_name: command
+    pattern: "jane@acme.com"
+  - type: text_not_contains
+    pattern: "400"
+  - type: text_not_contains
+    pattern: "eyJhbGciOiJub25l"

evals/cases/multi-turn/onboard/invite-user-batch.yaml

@@ -0,0 +1,34 @@
+name: invite-user-batch
+description: >
+  Batch invitation with one invalid address. Valid emails get invited via
+  MCP; the invalid one is flagged in a summary. MCP is verified first.
+skill: onboard-confidence
+tags: [multi-turn, invite-user]
+
+conversation:
+  - "Invite alice@acme.com, bob@acme.com and charlie@invalid to my Confidence workspace."
+  - "Yes, send invitation emails."
+
+ask_answers:
+  - match: "invitation email|send"
+    answer: "Yes, send invitation emails"
+
+assertions:
+  - type: tool_called
+    tool_name: mcp__confidence_flags__getIdentityInfo
+  - type: tool_call_arg_contains
+    tool_name: mcp__confidence_flags__inviteUser
+    arg_name: email
+    pattern: "alice@acme.com"
+  - type: tool_call_arg_contains
+    tool_name: mcp__confidence_flags__inviteUser
+    arg_name: email
+    pattern: "bob@acme.com"
+  - type: text_contains
+    pattern: "alice@acme.com"
+  - type: text_contains
+    pattern: "bob@acme.com"
+  # The invalid address is surfaced (either caught locally or via the tool error).
+  - type: text_contains
+    pattern: "invalid"
+    case_sensitive: false