feat(skills): add a planning skill for documentation evals

.claude/skills/docs-eval-planning/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: docs-eval-planning
+description: Plan a documentation eval for supabase/evals, where a docs guide is the subject under test. Use when asked to write, add, or design an eval for a Supabase docs guide, when a ticket asks for a deterministic eval on a page, or when deciding what a guide-under-test eval should check. Produces a plan, not files. Not for debugging a scorer, running an eval, or writing example solutions.
+---
+
+# Planning a documentation eval
+
+A documentation eval measures a page, not an agent. The prompt sends a user's request plus the page's
+url, and the checks say whether an agent that read the page produced working code. A gap in the page
+counts as a failure.
+
+That framing is the whole difficulty. The failure modes are collected in
+[references/flakiness.md](references/flakiness.md). Read it before designing checks, not after.
+
+## What this skill produces
+
+A plan. It does not write `PROMPT.md`, `EVAL.ts`, or a seed. Implementation and local scoring are a
+separate job.
+
+## The shape it takes
+
+Model a new eval on `build-docs-003-api-keys-guide` or later.
+
+`build-docs-NNN-<subject>`, named after the doc rather than the feature. **The id is permanent.** The
+published results series is keyed on it, so renaming breaks history. So does renaming a check.
+
+```
+evals/build-docs-NNN-<subject>/
+  PROMPT.md      frontmatter and the task the agent sees
+  EVAL.ts        the scorer
+  README.md      the design rationale, addressed to the next editor
+  <helper>.ts    flat beside EVAL.ts, never in a subdirectory
+  local/         the seed workspace
+```
+
+`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is required: it is the only
+place the stripped-word list and the constraints that must not be edited away live. Its sections:
+
+- **What this eval measures.** The page is the subject, not the agent.
+- **Do not reintroduce the vocabulary.** The stripped words, listed.
+- **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs.
+- **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which
+  ones make them mean something.
+- **The guide has to actually be read.**
+- **What this eval does not score.** Each entry with its reason. An unmeasured rule from the prompt is
+  named here.
+
+`EVAL.ts` is orchestration. Import named `check*` functions, assemble one flat array, return
+`{ passed: checks.every(c => c.passed), checks }`, and wrap it in a try/catch whose catch returns a
+single self-named failure check. Split the implementations into modules and keep the full check list
+declared in `EVAL.ts`. Comment why the phase order is what it is.
+
+Every page-under-test eval carries a check that the referenced page was read with content. Import
+`checkDocsGuideRead` from core if it has landed there; otherwise copy the most recent eval's. It
+resolves the url from the harness's docs result rather than the raw tool call, because a `search_docs`
+hit carries the guide's url in its result rather than its request.
+
+## Phase 0: read-only, and the repo rules
+
+**Write nothing in this phase or the next four.** Gathering, choosing the claim, drafting the prompt,
+and designing the checks are all reading. The output is a plan.
+
+Where the harness offers a planning mode that enforces read-only, enter it first and let it hold the
+line. In Claude Code that is `EnterPlanMode`. Where it does not, the discipline is yours to keep.
+
+Then read three things:
+
+1. `CONTRIBUTING.md`, which carries the repo's rules for any eval: suite selection, the folder shape,
+   the `motivation:` requirement, prompt discipline, scorer discipline, and how to refresh results in
+   CI. This skill defers to it and does not restate it. Where this skill goes further, it says so.
+2. The guide under test, as its `.md` variant.
+3. [references/flakiness.md](references/flakiness.md), before designing anything rather than after.
+
+## Phase 1: gather the evidence
+
+Work through [references/sources.md](references/sources.md). Run independent sources in parallel.
+
+**The checks come from outside the page.** A check list derived from the guide inherits the guide's
+blind spots, so every check passes and the score reports that the guide is fine when nobody looked.
+Build the list from the sources, then diff it against the page. Anything the sources treat as essential
+and the page omits is a candidate check, and it should fail a solution written from the page. That
+failure is the finding the paired docs ticket acts on.
+
+Several sources are internal to Supabase. If one you have access to is unauthorized, stop and ask for
+the connector rather than working around it, then record in the plan which sources were reachable.
+
+This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a
+thread, or a url. A finding with no source does not go in.
+
+## Phase 2: pick the one claim, and confirm it can be measured
+
+Name the single thing the guide has to transmit, in the user's terms rather than the product's.
+
+One claim, not three. An eval that measures several unrelated things reports which of them an agent
+got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is
+what makes a saturated check obvious later.
+
+A dense page yields ten candidates, all evidenced. Pick on consequence: the failure that costs a user
+the most, and among equals the one the page itself warns about. A page that carries a danger callout
+has already named its own worst case.
+
+**Then confirm the harness can measure it, before designing a single check.** A claim the harness
+cannot observe is not the claim, however well evidenced. Three questions, and read the code for the
+answers rather than assuming:
+
+- **Does the end state exist somewhere a scorer can reach?** A file in the workspace, a row in the
+  database, a response from a running process. If the correct answer is something the platform supplies
+  at deploy time, decide now where the seed lets it land.
+- **Can the scorer run whatever has to run?** A framework the eval seeds is installed and built inside
+  the sandbox with `ctx.exec`, not on the host: the host-side build helpers relink the workspace's
+  `node_modules` to the framework's, which does not carry an eval's own dependencies. Check whether any
+  existing eval runs the same toolchain, and treat a first as a risk to state in the plan.
+- **What is the time budget?** The agent's timeout does not bound the scorer, so a long install and
+  build costs wall-clock rather than the agent's budget. Per-command limits are yours to set.
+
+Write the answers into the plan. A feasibility problem found here is a design change; found in Phase 4
+it is a rewrite.
+
+## Phase 3: draft the prompt and the seed
+
+Work through [references/prompt-rules.md](references/prompt-rules.md).
+
+The prompt is a product request plus the page's url, written the way the user would write it, with the
+vocabulary that gives away the answer removed.
+
+The seed is where the detail goes. Write seed comments in product vocabulary, never the vocabulary the
+prompt strips: the endpoint the rest of the team builds against, the shapes a handler must return,
+which table the API serves. Pre-solve what belongs to another scenario and say so in the comment, so a
+mistake there cannot fail this eval for the wrong reason. Choose fixture values that a memorized answer
+gets wrong, so a whitelist check measures something. Seed a conflict where the subject allows one, so a
+single approach cannot satisfy every case.
+
+**Where the task contract lives is a real choice.** Put it in seed comments and gain a positive control
+the scorer can prove, at the cost of a discovery question. Leave it out and measure whether the agent
+derives it, which is harder and less observable. Decide deliberately and record which in the
+`README.md`.
+
+This phase ends with the prompt body, the list of stripped words, the seed, and the `motivation:`
+frontmatter.
+
+## Phase 4: design the checks
+
+Work through [references/check-rules.md](references/check-rules.md), in order. The first three rules
+matter most.
+
+Use a judge only when the artifact class is unbounded, meaning free prose or files whose shape you
+cannot predict. Use a deterministic check for anything a query or a file read settles. Scope the rubric
+to what the check is worth, state what not to grade, give the judge a tie-break, and make one claim per
+check. An eval passes only when every check passes, so an ambitious rubric on a secondary check fails
+the whole page.
+
+This phase ends with a table: the check name, what it proves, what it does when the object is absent,
+whether it reads files or runs code, and the evidence it came from.
+
+Then name the checks you expect to saturate, and write the prediction down. A check whose answer the
+seed labels costs an agent nothing and carries no signal. A plan that says which checks are cheap is
+honest about how much the eval measures.
+
+## Phase 5: write the plan
+
+Sections: context, failure-point inventory, the seed, the prompt, the check table, what is not scored,
+the expected-failure table, verification, the paired docs ticket, and Phase 6 as the closing step.
+
+**Everything in the prompt is either measured or named as out of scope.** A reviewer finds the rule
+that is neither.
+
+Verification means example solutions: one you believe is correct, plus a few carrying a single
+deliberate flaw each. Write down which checks you expect each to fail before running anything; that
+list is the test. A flaw usually trips several checks, so do not aim for exactly one failure each.
+**Never commit them.** `evals/*/solutions/` is a git exclusion, and you add it to `.git/info/exclude`
+before the first `git add` of an eval directory, because that file is local to your clone. A
+transcript-anchored check cannot be exercised by a solution, because no agent ran.
+
+## Phase 6: feed the catalog
+
+The plan's last step is to append whatever broke to [references/flakiness.md](references/flakiness.md)
+once the baseline lands, with the PR number.
+
+This is a plan step rather than advice, because the file is only worth having if it grows. Ask
+reviewers to write findings into it directly rather than leaving them in a review thread, where the
+next author does not look.
+
+## Content shape against end-to-end proof
+
+Two kinds of check, and a plan needs both.
+
+**A check that reads files** asserts content shape. It is cheap, stable, and has no opinion about
+whether anything runs. Every one of them passes for an agent that edited a config file and stopped.
+
+**A check that runs the code** asserts the outcome. It is what stops a decorative pass, and it is
+slower and more exposed to the environment.
+
+The table says which each check is, because the two fail for different reasons and a reviewer reading
+a failure needs to know which kind they are looking at.
+
+## Where the boundary sits
+
+Some claims are not this skill's to make. Whether the page carries runnable commands in fenced blocks,
+declares its environment variables and prerequisites, and ends with a verification step is a claim
+about the page's own markdown. That belongs to the page and to the project's authoring rubric. No
+tooling in this repo asserts it; the docs end-to-end suite is rendering, links, and accessibility.
+
+What belongs here is the claim about what an agent produces after reading the page. Mixing the two
+produces an eval that fails when someone reformats a code block.

.claude/skills/docs-eval-planning/references/check-rules.md

@@ -0,0 +1,105 @@
+# Designing the checks
+
+Ordered by what a violation costs. The first three matter most.
+
+## 1. The check must be observable in the end state
+
+If the correct answer is to let the platform supply a value, a correct agent writes nothing and the
+check reds correct work.
+
+**Before writing a check, decide where its answer lands.** Give the seed a place for it: an empty
+`.env.example`, a config file, a named file the contract points at. If there is nowhere for the answer
+to go, there is no check yet.
+
+## 2. Behavior over structure
+
+A check that reads where code sits fails correct refactorings. Count the effect instead.
+
+The same rule applies to static scans of source and config. A shell `export` prefix, a config value
+reached through a variable, a renamed environment variable, and a wrapper function around the call
+being matched each defeat one. Replace the static claim with a behavioral probe, or commit a
+counterexample as a local solution and confirm the check rejects it.
+
+Where a static check is the only option, name it for what it proves and say in the `README.md` that it
+is a source-level claim.
+
+## 3. Positive controls, and not ones a literal satisfies
+
+Every absence check needs a paired proof that the allowed path changes state.
+
+An absence check passes for an agent that built nothing, so the paired control is what makes it mean
+anything.
+
+**A control must not be satisfiable by a literal.** A control that greps a response for a seeded value
+passes a hardcoded value. Insert state with a run-scoped marker after the code was written, and require
+the marker back.
+
+## 4. Every call site qualifies, not the first
+
+Reading the first match passes a workspace that holds one correct call site and one wrong one. Range
+over every site.
+
+## 5. Write the absence semantics down, per check
+
+Decide what the check does when the object it inspects does not exist, put it in the table, then read
+the code to confirm it does that.
+
+A check on an object the prompt names fails when the object is missing. A check on a class of object
+the agent may not have created passes when there are none. A conditional check says so in its notes,
+so a not-applicable pass does not read as a real one.
+
+## 6. Whitelist, not blacklist
+
+Enumerating spellings cannot close a class; any list is incomplete by construction. Whitelist what the
+project was given and fail everything else.
+
+## 7. Name the check for what it proves
+
+Not for what you meant. A name that claims a class is false when the check proves only a few members
+of it.
+
+The published results series is keyed on check names, so renaming one breaks history. Get the name
+right before the first run.
+
+## 8. The environment must not decide the verdict
+
+Transport security, address family, and port availability can each fail correct work, and the eval then
+reports the environment rather than the page.
+
+Two ways out. Make the harness support the correct behavior, or declare the claim unscored in the
+`README.md` and say why.
+
+The same trap catches a check that scores something the platform decides rather than something the page
+says. A pinned CLI version or a runtime default is not the subject under test, and a check on one
+reports an upgrade as a docs change.
+
+## 9. Predict which checks saturate
+
+If the seed labels the answer, the check is free and carries no signal.
+
+Write the prediction into the plan, so a reviewer sees where the signal actually is.
+
+## 10. A blocked check fails, it never skips
+
+A missing artifact is the absence of evidence. Reporting it green hands a clean sheet to a run that
+produced nothing. Give the blocked path its own note saying why it did not run.
+
+## 11. Split the phases, and say why the order matters
+
+Static scans run before anything installs or writes. Snapshot state before running anything the agent
+wrote. A probe that mutates a fixture runs after every probe that reads it.
+
+Scope every marker to the run. A fixed literal leaks between runs.
+
+**Return errors instead of throwing.** A throwing setup step collapses the whole result to one check.
+Fold the error into a check of its own, so a failed step costs only the checks that needed it.
+
+## 12. Reach for a judge only when the artifact class is unbounded
+
+Free prose or arbitrary test files, not anything a query or a file read can settle.
+
+When you do: one claim per check, a checklist rubric rather than an adjective, an explicit statement of
+what not to grade, and a tie-break.
+
+Read the notes on a failing run before blaming the judge. A rubric that complains differently each run
+is usually tracking real variance in what the agent produced.