feat(evals): eval the server-side auth client guide in supabase.com/docs

[czenko]

Closes DOCS-1303

Problem

auth/server-side/creating-a-client carries the highest agent share of any task-driven page in the docs cohort, 35.0 percent on 22,918 views, and Server-side Auth is the densest label in feedback intake. Nothing measures whether the page gets an agent to a dashboard that will not hand itself to a forged cookie.

Solution

• Adds build-docs-005-server-side-auth, suite: regression, services: [gotrue, kong, postgrest].
• Seeds a Next.js 16 app: a sign-in form posting to POST /login, an unwired login route handler, and an unwired dashboard. The seed fixes the form field names, the login contract, and data-testid="viewer-email" on the dashboard.
• The one claim is measured behaviorally. a tampered session cookie does not get the dashboard signs a user in through the app, re-encodes the session with a second user's identity, and leaves the signature alone so the token no longer verifies. Reading stored session state accepts that cookie; verifying the token rejects it. It is gated on the render control passing first, so an app that renders nothing cannot clear it.
• Five runtime checks install, build, and serve the app inside the sandbox with ctx.exec. next is not a framework dependency, and the host-side build helpers relink the workspace's node_modules to the framework's own, so a host-side build cannot see what the seed declared.
• Three source checks cover the verifying call, the cookie adapter, and the deprecated helpers import.
• Checks are sourced from outside the page: feedback intake (FDBKIN-20970, FDBKIN-6870, FDBKIN-10628, FDBKIN-14804, FDBKIN-15214, FDBKIN-11678, FDBKIN-4573), Supacademy auth/troubleshooting/02-session-refresh-and-ssr-failure-patterns.md, and examples/prompts/nextjs-supabase-auth.md in the main repo.

The verifying-call check accepts getClaims() and getUser() both. DOCS-1303 and the scoping doc ask for getUser(); the page now recommends getClaims() and names it the way to protect pages. A check on either alone would report the page as broken for giving current advice, so the check names the class.

Two framework problems this surfaced, neither fixed here:

• ctx.stackStatus() requires API_URL, PUBLISHABLE_KEY and SECRET_KEY together and throws when any is missing. This eval needs the api url and one client key, so it reads supabase status directly and accepts PUBLISHABLE_KEY or ANON_KEY.
• discoverEvals reads PROMPT.md in every directory under evals/ and throws when one is absent, so a git-excluded solutions/ directory left behind by a branch switch breaks discovery for every eval in the repo.

Baseline

Six runs, sonnet-5 with and without skills, three each. 9/9 on all six.

The checks are not vacuous. Five fixtures were scored against the stack first and each failed what it was built to fail:

Fixture	Result	Failed
green	8/9	the guide-read check, which no fixture can pass
deprecated-cookie-methods	7/9	the cookie adapter check
auth-helpers-import	7/9	the deprecated import check
get-session-only	6/9	the tampered cookie rendered the intruder's email
dashboard-renders-nothing	5/9	all three runtime checks

get-session-only reads stored session state instead of verifying, and the forged cookie rendered another user's identity. The central check catches a real bypass; agents did not write that code.

Read the 9/9 as regression cover rather than benchmark signal. It says the page gets sonnet-5 to a dashboard that rejects a forged cookie. It does not say the page caused that: the guide-read check proves the page was opened, not that it was load-bearing, and a model that already knows the @supabase/ssr pattern produces the same output. Hardening is worth considering after DOCS-1313 lands, so the two changes do not confound each other.

Manual testing

1. Confirm the ports are free.

   docker ps --format '{{.Names}}\t{{.Ports}}' | grep -E '5432[12]->'
   

2. Confirm discovery and frontmatter.

   pnpm eval:dry -- --eval build-docs-005-server-side-auth
   

   One PLAN line per experiment, each reading stage=build suite=regression mode=local-stack.

3. Typecheck.

   pnpm typecheck
   

   Clean.

4. Lint the eval.

   ./node_modules/.bin/biome check evals/build-docs-005-server-side-auth
   

   No fixes applied.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 3, 2026 9:42pm UTC