diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md new file mode 100644 index 00000000..377e27d9 --- /dev/null +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -0,0 +1,204 @@ +--- +name: docs-eval-planning +description: Plan a documentation eval for supabase/evals, where a docs guide is the subject under test. Use when asked to write, add, or design an eval for a Supabase docs guide, when a ticket asks for a deterministic eval on a page, or when deciding what a guide-under-test eval should check. Produces a plan, not files. Not for debugging a scorer, running an eval, or writing example solutions. +--- + +# Planning a documentation eval + +A documentation eval measures a page, not an agent. The prompt sends a user's request plus the page's +url, and the checks say whether an agent that read the page produced working code. A gap in the page +counts as a failure. + +That framing is the whole difficulty. The failure modes are collected in +[references/flakiness.md](references/flakiness.md). Read it before designing checks, not after. + +## What this skill produces + +A plan. It does not write `PROMPT.md`, `EVAL.ts`, or a seed. Implementation and local scoring are a +separate job. + +## The shape it takes + +Model a new eval on `build-docs-003-api-keys-guide` or later. + +`build-docs-NNN-`, named after the doc rather than the feature. **The id is permanent.** The +published results series is keyed on it, so renaming breaks history. So does renaming a check. + +``` +evals/build-docs-NNN-/ + PROMPT.md frontmatter and the task the agent sees + EVAL.ts the scorer + README.md the design rationale, addressed to the next editor + .ts flat beside EVAL.ts, never in a subdirectory + local/ the seed workspace +``` + +`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is required: it is the only +place the stripped-word list and the constraints that must not be edited away live. Its sections: + +- **What this eval measures.** The page is the subject, not the agent. +- **Do not reintroduce the vocabulary.** The stripped words, listed. +- **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs. +- **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which + ones make them mean something. +- **The guide has to actually be read.** +- **What this eval does not score.** Each entry with its reason. An unmeasured rule from the prompt is + named here. + +`EVAL.ts` is orchestration. Import named `check*` functions, assemble one flat array, return +`{ passed: checks.every(c => c.passed), checks }`, and wrap it in a try/catch whose catch returns a +single self-named failure check. Split the implementations into modules and keep the full check list +declared in `EVAL.ts`. Comment why the phase order is what it is. + +Every page-under-test eval carries a check that the referenced page was read with content. Import +`checkDocsGuideRead` from core if it has landed there; otherwise copy the most recent eval's. It +resolves the url from the harness's docs result rather than the raw tool call, because a `search_docs` +hit carries the guide's url in its result rather than its request. + +## Phase 0: read-only, and the repo rules + +**Write nothing in this phase or the next four.** Gathering, choosing the claim, drafting the prompt, +and designing the checks are all reading. The output is a plan. + +Where the harness offers a planning mode that enforces read-only, enter it first and let it hold the +line. In Claude Code that is `EnterPlanMode`. Where it does not, the discipline is yours to keep. + +Then read three things: + +1. `CONTRIBUTING.md`, which carries the repo's rules for any eval: suite selection, the folder shape, + the `motivation:` requirement, prompt discipline, scorer discipline, and how to refresh results in + CI. This skill defers to it and does not restate it. Where this skill goes further, it says so. +2. The guide under test, as its `.md` variant. +3. [references/flakiness.md](references/flakiness.md), before designing anything rather than after. + +## Phase 1: gather the evidence + +Work through [references/sources.md](references/sources.md). Run independent sources in parallel. + +**The checks come from outside the page.** A check list derived from the guide inherits the guide's +blind spots, so every check passes and the score reports that the guide is fine when nobody looked. +Build the list from the sources, then diff it against the page. Anything the sources treat as essential +and the page omits is a candidate check, and it should fail a solution written from the page. That +failure is the finding the paired docs ticket acts on. + +Several sources are internal to Supabase. If one you have access to is unauthorized, stop and ask for +the connector rather than working around it, then record in the plan which sources were reachable. + +This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a +thread, or a url. A finding with no source does not go in. + +## Phase 2: pick the one claim, and confirm it can be measured + +Name the single thing the guide has to transmit, in the user's terms rather than the product's. + +One claim, not three. An eval that measures several unrelated things reports which of them an agent +got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is +what makes a saturated check obvious later. + +A dense page yields ten candidates, all evidenced. Pick on consequence: the failure that costs a user +the most, and among equals the one the page itself warns about. A page that carries a danger callout +has already named its own worst case. + +**Then confirm the harness can measure it, before designing a single check.** A claim the harness +cannot observe is not the claim, however well evidenced. Three questions, and read the code for the +answers rather than assuming: + +- **Does the end state exist somewhere a scorer can reach?** A file in the workspace, a row in the + database, a response from a running process. If the correct answer is something the platform supplies + at deploy time, decide now where the seed lets it land. +- **Can the scorer run whatever has to run?** A framework the eval seeds is installed and built inside + the sandbox with `ctx.exec`, not on the host: the host-side build helpers relink the workspace's + `node_modules` to the framework's, which does not carry an eval's own dependencies. Check whether any + existing eval runs the same toolchain, and treat a first as a risk to state in the plan. +- **What is the time budget?** The agent's timeout does not bound the scorer, so a long install and + build costs wall-clock rather than the agent's budget. Per-command limits are yours to set. + +Write the answers into the plan. A feasibility problem found here is a design change; found in Phase 4 +it is a rewrite. + +## Phase 3: draft the prompt and the seed + +Work through [references/prompt-rules.md](references/prompt-rules.md). + +The prompt is a product request plus the page's url, written the way the user would write it, with the +vocabulary that gives away the answer removed. + +The seed is where the detail goes. Write seed comments in product vocabulary, never the vocabulary the +prompt strips: the endpoint the rest of the team builds against, the shapes a handler must return, +which table the API serves. Pre-solve what belongs to another scenario and say so in the comment, so a +mistake there cannot fail this eval for the wrong reason. Choose fixture values that a memorized answer +gets wrong, so a whitelist check measures something. Seed a conflict where the subject allows one, so a +single approach cannot satisfy every case. + +**Where the task contract lives is a real choice.** Put it in seed comments and gain a positive control +the scorer can prove, at the cost of a discovery question. Leave it out and measure whether the agent +derives it, which is harder and less observable. Decide deliberately and record which in the +`README.md`. + +This phase ends with the prompt body, the list of stripped words, the seed, and the `motivation:` +frontmatter. + +## Phase 4: design the checks + +Work through [references/check-rules.md](references/check-rules.md), in order. The first three rules +matter most. + +Use a judge only when the artifact class is unbounded, meaning free prose or files whose shape you +cannot predict. Use a deterministic check for anything a query or a file read settles. Scope the rubric +to what the check is worth, state what not to grade, give the judge a tie-break, and make one claim per +check. An eval passes only when every check passes, so an ambitious rubric on a secondary check fails +the whole page. + +This phase ends with a table: the check name, what it proves, what it does when the object is absent, +whether it reads files or runs code, and the evidence it came from. + +Then name the checks you expect to saturate, and write the prediction down. A check whose answer the +seed labels costs an agent nothing and carries no signal. A plan that says which checks are cheap is +honest about how much the eval measures. + +## Phase 5: write the plan + +Sections: context, failure-point inventory, the seed, the prompt, the check table, what is not scored, +the expected-failure table, verification, the paired docs ticket, and Phase 6 as the closing step. + +**Everything in the prompt is either measured or named as out of scope.** A reviewer finds the rule +that is neither. + +Verification means example solutions: one you believe is correct, plus a few carrying a single +deliberate flaw each. Write down which checks you expect each to fail before running anything; that +list is the test. A flaw usually trips several checks, so do not aim for exactly one failure each. +**Never commit them.** `evals/*/solutions/` is a git exclusion, and you add it to `.git/info/exclude` +before the first `git add` of an eval directory, because that file is local to your clone. A +transcript-anchored check cannot be exercised by a solution, because no agent ran. + +## Phase 6: feed the catalog + +The plan's last step is to append whatever broke to [references/flakiness.md](references/flakiness.md) +once the baseline lands, with the PR number. + +This is a plan step rather than advice, because the file is only worth having if it grows. Ask +reviewers to write findings into it directly rather than leaving them in a review thread, where the +next author does not look. + +## Content shape against end-to-end proof + +Two kinds of check, and a plan needs both. + +**A check that reads files** asserts content shape. It is cheap, stable, and has no opinion about +whether anything runs. Every one of them passes for an agent that edited a config file and stopped. + +**A check that runs the code** asserts the outcome. It is what stops a decorative pass, and it is +slower and more exposed to the environment. + +The table says which each check is, because the two fail for different reasons and a reviewer reading +a failure needs to know which kind they are looking at. + +## Where the boundary sits + +Some claims are not this skill's to make. Whether the page carries runnable commands in fenced blocks, +declares its environment variables and prerequisites, and ends with a verification step is a claim +about the page's own markdown. That belongs to the page and to the project's authoring rubric. No +tooling in this repo asserts it; the docs end-to-end suite is rendering, links, and accessibility. + +What belongs here is the claim about what an agent produces after reading the page. Mixing the two +produces an eval that fails when someone reformats a code block. diff --git a/.claude/skills/docs-eval-planning/references/check-rules.md b/.claude/skills/docs-eval-planning/references/check-rules.md new file mode 100644 index 00000000..e751f679 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/check-rules.md @@ -0,0 +1,105 @@ +# Designing the checks + +Ordered by what a violation costs. The first three matter most. + +## 1. The check must be observable in the end state + +If the correct answer is to let the platform supply a value, a correct agent writes nothing and the +check reds correct work. + +**Before writing a check, decide where its answer lands.** Give the seed a place for it: an empty +`.env.example`, a config file, a named file the contract points at. If there is nowhere for the answer +to go, there is no check yet. + +## 2. Behavior over structure + +A check that reads where code sits fails correct refactorings. Count the effect instead. + +The same rule applies to static scans of source and config. A shell `export` prefix, a config value +reached through a variable, a renamed environment variable, and a wrapper function around the call +being matched each defeat one. Replace the static claim with a behavioral probe, or commit a +counterexample as a local solution and confirm the check rejects it. + +Where a static check is the only option, name it for what it proves and say in the `README.md` that it +is a source-level claim. + +## 3. Positive controls, and not ones a literal satisfies + +Every absence check needs a paired proof that the allowed path changes state. + +An absence check passes for an agent that built nothing, so the paired control is what makes it mean +anything. + +**A control must not be satisfiable by a literal.** A control that greps a response for a seeded value +passes a hardcoded value. Insert state with a run-scoped marker after the code was written, and require +the marker back. + +## 4. Every call site qualifies, not the first + +Reading the first match passes a workspace that holds one correct call site and one wrong one. Range +over every site. + +## 5. Write the absence semantics down, per check + +Decide what the check does when the object it inspects does not exist, put it in the table, then read +the code to confirm it does that. + +A check on an object the prompt names fails when the object is missing. A check on a class of object +the agent may not have created passes when there are none. A conditional check says so in its notes, +so a not-applicable pass does not read as a real one. + +## 6. Whitelist, not blacklist + +Enumerating spellings cannot close a class; any list is incomplete by construction. Whitelist what the +project was given and fail everything else. + +## 7. Name the check for what it proves + +Not for what you meant. A name that claims a class is false when the check proves only a few members +of it. + +The published results series is keyed on check names, so renaming one breaks history. Get the name +right before the first run. + +## 8. The environment must not decide the verdict + +Transport security, address family, and port availability can each fail correct work, and the eval then +reports the environment rather than the page. + +Two ways out. Make the harness support the correct behavior, or declare the claim unscored in the +`README.md` and say why. + +The same trap catches a check that scores something the platform decides rather than something the page +says. A pinned CLI version or a runtime default is not the subject under test, and a check on one +reports an upgrade as a docs change. + +## 9. Predict which checks saturate + +If the seed labels the answer, the check is free and carries no signal. + +Write the prediction into the plan, so a reviewer sees where the signal actually is. + +## 10. A blocked check fails, it never skips + +A missing artifact is the absence of evidence. Reporting it green hands a clean sheet to a run that +produced nothing. Give the blocked path its own note saying why it did not run. + +## 11. Split the phases, and say why the order matters + +Static scans run before anything installs or writes. Snapshot state before running anything the agent +wrote. A probe that mutates a fixture runs after every probe that reads it. + +Scope every marker to the run. A fixed literal leaks between runs. + +**Return errors instead of throwing.** A throwing setup step collapses the whole result to one check. +Fold the error into a check of its own, so a failed step costs only the checks that needed it. + +## 12. Reach for a judge only when the artifact class is unbounded + +Free prose or arbitrary test files, not anything a query or a file read can settle. + +When you do: one claim per check, a checklist rubric rather than an adjective, an explicit statement of +what not to grade, and a tie-break. + +Read the notes on a failing run before blaming the judge. A rubric that complains differently each run +is usually tracking real variance in what the agent produced. diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md new file mode 100644 index 00000000..da9a5350 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -0,0 +1,217 @@ +# What has gone wrong before + +Every entry is from a run or a review, with the PR that recorded it. A citation per entry is what keeps +a plausible-sounding rule from creeping in. + +Add to this file. A documentation eval is not finished until whatever went wrong on it is here. + +## False green: the check passes broken work + +- **A static scan is defeated by aliasing, four ways.** In a `.env` file, `export VITE_ADMIN=` + slips past a key-name parse. A bundler config that computes `envPrefix` through a variable is + invisible to a literal read of the config. An environment variable renamed to an innocuous alias + defeats a name blocklist. A wrapper function defeats a match on the call inside it. #212 + **Fix.** Commit the counterexample as a local fixture, or replace the static claim with a behavioral + probe. +- **A literal match counts dead code as working code.** `.auth.signUp(` anywhere in client source + satisfied a check, including in code nothing reaches. #212 + **Fix.** Pair it with a check that proves the path runs, and name the check as a literal match. +- **A control that greps for a seeded value passes a hardcoded array.** A handler returning a literal + containing the seeded row's name cleared a read control. #257 + **Fix.** Insert a row with a run-scoped marker after the code was written and require the marker back. +- **Reading only the first call site.** Checking `sites[0]` cleared a workspace holding one correct + client and one constructed per request elsewhere. #257 + **Fix.** Range over every site. +- **Enumerating spellings cannot close a class.** Six key-name patterns still missed two more. #168 + **Fix.** Whitelist what the project was given and fail everything else. +- **The result is green when nothing was read.** `!error && leaked.length === 0` is true of an empty + result set. #168 + **Fix.** Assert the subject's own row is present, then that nobody else's is. +- **Repointing REFERENCE at a docs preview without matching the preview host.** The guide-read check + matches on `includes`, so a fetch of the published page satisfies a check meant to prove the preview + was read. #218 + **Fix.** Match the full preview host, not the path. +- **A probe asserting a request was refused passes on an error it caused itself.** An anonymous insert + sent `owner_id` and `title` against a schema whose `cadence` is `not null`, so Postgres rejected it + as a not-null violation and the check counted any error as a refusal. A wide-open table scored as + protected, and the fixture that caught it was the one predicted to fail. #264 + **Fix.** Send the whole contract row, so access control is the only thing left that can refuse it. + Confirm the row is absent as the superuser instead of inferring absence from the error, and treat any + code outside `42501` as could-not-measure rather than as a pass. + +## False red: the check fails correct work + +- **A structural read of where code sits.** Requiring the client constructor at module scope red a + factory called once at module scope, which is correct. #257 + **Fix.** Count the effect. One connection across repeated invocations proves the same thing and does + not care how the client was built. +- **The correct answer is not observable.** Agents left a connection string to the deploy, as the seed + told them to, so nothing in the workspace carried their choice and the central check red all six + runs. One of them named the right answer in a code comment. #257 + **Fix.** Give the choice somewhere to land in the seed. +- **The environment cannot support the correct behavior.** A handler setting `ssl: 'require'` could + not reach a local database that speaks no TLS, so both end-to-end checks red on a more correct + solution. #257 + **Fix.** Terminate TLS in front of the database, or declare the claim unscored. +- **The sandbox image made the task unwinnable.** `npm install -g` failed with `EACCES` because the + image ran as a user with no writable global prefix, so the agent could not complete step one. #108 + **Fix.** Change the image, and add a test that locks the behavior in. +- **A prompt ambiguity the checks resolve one way.** An agent gated an endpoint on being signed in, + which the prompt never ruled out, and it red in CI. #212 + **Fix.** Either the check accepts it, or the prompt says which it wants. +- **A design choice scored as a defect.** A probe requiring two rows in one response would fail an + agent that paginated. #212 + **Fix.** Name it in the README as a narrow edge, or probe in a way that tolerates it. +- **The check measures the platform, not the page.** Two checks scored which keys a runtime hands a + function, which the pinned CLI decides. Both were reverted. #212, #218 + **Fix.** Drop it. Keeping it means a CLI upgrade reports as a docs change. +- **The guide-read check drops non-apex hosts.** `isSupabaseApexUrl` accepts only `supabase.com`, so a + preview fetch never reaches `docsCalls` and the check reds a run that did read the page. #218 + **Fix.** Widen the host predicate when pointing an eval at an unmerged docs PR. +- **A regex over a tool call is too strict.** A plugin-install check missed the command because the + agent added `--yes`. #108 + **Fix.** Match the part that carries the meaning. +- **Extension-owned objects land in scope.** pgTAP creates views in `public` when installed there. #168 + **Fix.** Anti-join `pg_depend` on `deptype = 'e'`. + +## Nondeterminism + +- **Judge variance was misdiagnosed twice.** A rubric's complaint changed run to run. The first + response blamed reasoning effort and proposed a per-eval override; the second raised the global + default; the third proved the variance was never in the judge. It tracked whichever test suite the + agent wrote that run. #220, #222, #224 + **Fix.** Read the notes on the failing runs before blaming the judge. Then scope the rubric to what + the check is worth, drop every craft requirement, and give it a tie-break. +- **One ambitious check fails a whole eval.** An eval passes only when every check passes, so a bonus + check fails the whole page while every primary check passes. #224 + **Fix.** Scope a bonus check to a low bar, and rename it if the old name claimed more. +- **Retry layers multiply.** A pair-level retry wrapping a create-level retry produced 39 attempts over + 588 seconds for one unrecoverable credential error. At high concurrency a single bad secret makes + every pair thrash. #192 + **Fix.** Classify errors as retryable or not, and do not nest retries. +- **Stop-on-pass retries hide variance.** `attempts: 2` means attempt one failed. That can be agent + variance rather than a scorer defect, and the two read identically in a summary. #218 + **Fix.** Read the attempts, and treat one green run as a signal rather than proof. +- **Fixed marker literals leak between runs.** #168 + **Fix.** Scope markers to the run. + +## Crashes and contamination + +- **A throwing setup step collapses the result.** One uncaught insert took a 35-check result down to + one, visible only by comparing check counts against the previous commit. #168 + **Fix.** Return the error and fold it into a check of its own. +- **`ON CONFLICT` pins a schema the prompt invites the agent to change.** #168 + **Fix.** `INSERT ... SELECT ... WHERE NOT EXISTS`. +- **A probe mutates what a later check reads.** #168 + **Fix.** Snapshot the catalog first, run the agent's own suite last, and put a probe that rewrites a + fixture after every probe that reads it. +- **`export-results` merge never prunes.** It overwrites keys present in the new results and leaves + stale keys alive, so renaming an eval leaves the old row beside the new one and a skipped pair keeps + its last recorded verdict. #168, #108 + **Fix.** Pass the requested pairs and drop rows that were requested but produced nothing. +- **An empty `results/` overwrites the export with `[]`** rather than refusing. #228 +- **A bot pushed refreshed results onto a measurement branch**, where the results were measured against + a preview url rather than the published page. #218 + **Fix.** Do not refresh results from a scratch branch. + +## Environment and reproducibility + +- **Ports.** The range is 54321 to 54329. Another project's stack holding them blocked local runs + entirely, and one eval shipped for review having never run end to end because of it. #168, #206 + **Fix.** Preflight the ports. Both later evals open their testing steps with it. +- **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. + #228, #168 +- **A stale local `VERCEL_OIDC_TOKEN` shadows the real one** and every sandbox create fails. #192 +- **`ctx.stackStatus()` requires three values together** and throws when any is missing, so an eval + that needs only the api url and one client key cannot use it. Read `supabase status` directly and + accept `PUBLISHABLE_KEY` or `ANON_KEY`. #261 +- **A leftover git-excluded directory under `evals/` breaks discovery for the whole repo.** + `discoverEvals` reads `PROMPT.md` in every directory it finds and throws when one is absent, so a + `solutions/` directory left behind by a branch switch stops every eval from being discovered. + #261, #264 + **Fix.** Move the leftover aside. Discovery skipping a directory with no `PROMPT.md` is the real fix. + It has now happened on two consecutive evals, each time from switching onto the next eval's branch. +- **A sandbox can lose its database container mid-run.** One run of six reported + `No such container: supabase_db_` when the scorer read `supabase status`, which cost the + install check and every check downstream of it. The static checks still ran. Nothing in the eval + caused it, and a rerun did not reproduce it. #257 + **Fix.** None available at the eval's level. Report the run as lost rather than folding it into the + score, and read the surviving checks. This is the reason to run three or more times per experiment + rather than once. +- **`kong` does not come up on its own.** With `services: [gotrue, kong]`, `supabase status` reports no + `API_URL`. Adding `postgrest` brings it up. #261 +- **A framework the eval seeds is not installed on the host.** The host-side build and test helpers + relink the workspace's `node_modules` to the framework's own, which carries no eval dependencies, so + a scorer that calls them cannot import what the seed declared. Install, build, and serve inside the + sandbox with `ctx.exec` instead. #259 + **Fix.** Make the install its own check, so a failure costs only the checks that needed it. +- **A first-of-its-kind toolchain in a scorer is a risk, not a detail.** Before designing checks around + one, confirm an existing eval runs it, and say in the plan when none does. The agent's timeout does + not bound the scorer, so wall-clock rather than budget is the cost. #259 +- **An env-isolation assertion cannot be strict.** Vite sets `NODE_ENV` and macOS injects + `__CF_USER_TEXT_ENCODING` even with a controlled environment, so asserting an exact key set fails. + #188 +- **`pnpm format:check` fails on local artifacts that are nobody's diff.** Biome walks + `.claude/worktrees/`, where it hits broken symlinks, `.solution-runs/`, and the git-excluded + `evals/*/solutions/`, which is where a deliberate parse-error fixture lives. The failure reads as a + broken branch. #259 + **Fix.** Scope Biome to the paths you touched, or add those three to `biome.json`. + +## Metadata + +- **An optional dimension is a silent hole.** `interface:` was optional, so six scenarios never + declared one and fell out of any categorization on that axis. #228 + **Fix.** Require it, so the gap becomes a load-time error. +- **`motivation:` is required by `CONTRIBUTING.md` and is not in the schema**, so the preprocess step + drops it and two evals have none. #228 +- **Renaming breaks history.** The published series is keyed on the eval id and on check names. #168, + #230 + **Fix.** Get both right before the first run, and keep name strings byte-identical when refactoring. + +## Reading a result + +- **One lost run is not a data point.** A run that failed on infrastructure has no verdict, and + averaging it in reads as a page or scorer problem. Say how many runs were lost and why, then quote + the score over the runs that completed. #257 +- **Saturation is a finding, not a defect, when the checks are known to fail something.** An eval whose + every check passes on every run has measured that the page works, provided the fixtures prove the + checks can fail. Without that proof the same result is indistinguishable from a scorer that asserts + nothing. Score the fixtures before reading a full-marks baseline. #261 +- **A full-marks pass does not attribute the success to the page.** The guide-read check proves the page + was opened, not that it caused the outcome. A model that already knows the pattern produces the same + result. Comparing the skills and no-skills variants rules out the skills; nothing available rules out + prior knowledge. Say so rather than claiming the page carried the run. #261 +- **The prior-knowledge confound is now measured, and it is total.** One eval ran twice, once with the + reliance instruction missing and once with it present. Unread, agents scored every substantive check. + Read, agents scored every substantive check. Same eval, same model, same checks, and reading the page + changed nothing. Treat a page-under-test pass on a mainstream task as regression cover by default, and + ask for the unread arm before claiming the page carried anything. #264 +- **A missing reliance instruction produces a baseline that looks fine.** Every check but the guide-read + check passed, the run counts and durations were unremarkable, and only the empty `docs.calls` array + said the page had never been opened. The score was evidence about the model. #264 + **Fix.** Read `docs.calls` before reading the score. Empty means the run measured nothing about the + page, whatever the checks say. +- **Benign CLI chatter at the front of a note gets a real failure filed as infrastructure.** + `supabase status` writes a `Stopped services` line naming every service the eval does not start, plus + an upgrade notice. Both led the seed failure's note, and the `ERROR: relation ... does not exist` + underneath was read as a symptom of the stopped services. A genuine missing-table failure went into + the record as a lost run. #264 + **Fix.** Silence the status call's stderr, hoist `ERROR`, `FATAL`, `DETAIL` and `HINT` to the front of + the message, and check which services the eval actually starts before reading `Stopped services` as a + fault. + +## Process + +- **Strip deployment-url churn before diffing two previews.** Every internal link on a docs preview + carries the deployment hash, so a rebuild rewrites dozens of lines that say nothing. Substitute the + host out of both copies before comparing, or the real edit is buried. #257 +- **Verify your own predictions.** A PR body predicted a check would fail an agent following the page. + The reviewer found it passing locally and in CI and asked why. #212 +- **An unmeasured rule in the prompt is a defect.** A reviewer found a prompt rule with no check and no + README entry. #168 + **Fix.** Measure it or name it out of scope. +- **The harness can leak the answer.** A system-prompt addendum gave agents a head start on what they + should have discovered. #108, #241 +- **The scorer copied the guide's own AI prompt panel into `PROMPT.md`**, so the rules that spell out + the answer were the prompt. #168 + **Fix.** The prompt is a product request. The page is the thing being tested. diff --git a/.claude/skills/docs-eval-planning/references/prompt-rules.md b/.claude/skills/docs-eval-planning/references/prompt-rules.md new file mode 100644 index 00000000..f4a68598 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/prompt-rules.md @@ -0,0 +1,80 @@ +# Drafting the prompt + +The prompt is a product request plus the page's url. Nothing else. + +`CONTRIBUTING.md` already says prompts read as a real user's message, do not spoonfeed, and move +detail into seed data. Everything here is the strict reading of that for a page under test. + +## Strip the vocabulary that gives away the answer + +Write the request the way the user would write it, in product terms. Never name the mechanism, the +setting, or the concept the page teaches. + +The stripped words are the measurement. An agent that applies a practice only when the prompt names it +has not been served by the page. + +The list is the vocabulary the page's own answer is made of: the mechanism, its settings, and the +concepts that name them. Write it down, and check the prompt against it. + +**Say where the code runs and what it has to do. Never say how to reach the service.** The runtime and +the task are the user's own framing. The method is the answer. + +**List the stripped words in the eval's `README.md`.** They creep back otherwise, one loosening at a +time, and the next editor has no way to know they were deliberate. + +## Strip from the prompt, not from everything + +Fixture material a user would already have in front of them may carry the vocabulary. A seed that +reproduces what a dashboard or a generated file prints is more faithful than one that strips words the +user can already see. + +The line is the prompt. The prompt is the user's own words; the seed is the world the user is already +in. + +## Keep the reliance instruction + +End the request with this, verbatim: + +``` +Read the guide below before you start and rely on it for how to set this up, +rather than on what you already know. +``` + +Without it, a pass is evidence about the model rather than the page. The older, weaker form is "Read +this guide first and follow it". Use the sentence above. + +## The `REFERENCE` block + +Bare. No colon, no markdown link, no code fence. + +``` +REFERENCE +https://supabase.com/docs/guides/.md +``` + +Use the `.md` variant. That is what an agent retrieves, and the guide-read check resolves against it. + +## Do not spoonfeed, and watch for what the prompt leaves open + +Detail belongs in the seed. A seeded table resolves column names, a seeded file fixes a contract, and +a comment explains structure the code does not show. + +**If the prompt does not say, the check passes.** One eval red an agent that gated an endpoint on being +signed in, which the prompt never ruled out. The reviewer's read is the rule: either the check accepts +it, or the prompt says which it wants. An ambiguity the checks resolve one way is a false red waiting +for the run that resolves it the other way. + +## Frontmatter + +- `stage: build`, `interface: cli`, and `suite: regression`. Benchmark is gated on at least one agent + failing the scenario, so it is a question for after the baseline, not before. +- `product:` and `topic:` from the closed enums in `packages/core/src/eval-metadata.ts`. `topic` has no + `database` value; `sdk` is usually the fit. +- `services:` narrowed to what the scorer touches. This also decides what the agent can reach, which + is a design lever: excluding a service the wrong answer would need makes the wrong answer fail + locally. +- Do not add `skills: []`. The `*-no-skills` experiment variants are how a run sees the page without + the product skills. +- `motivation:` as a folded `>-` block. The motivation is the same every time: evidence that an agent + can read this page and perform the task correctly, with the practices the page should be teaching. + Say which page, what an agent gets wrong without it, and cite the evidence from Phase 1. diff --git a/.claude/skills/docs-eval-planning/references/sources.md b/.claude/skills/docs-eval-planning/references/sources.md new file mode 100644 index 00000000..c8cd3b2a --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/sources.md @@ -0,0 +1,63 @@ +# Where the checks come from + +The page under test is not a source. Everything here is. + +**Several sources are internal to Supabase.** This repository is open source, and the Linear, Slack, +Hex, and Supacademy entries below are available only to Supabase employees. An eval planned from the +public sources alone is a smaller artifact, and that is a supported outcome: say so in the plan rather +than leaving the gap implicit. + +For a source you do have: if its tool is missing or unauthorized, stop and ask for the connector rather +than working around it. Then record which sources were reachable. Silently skipping one produces a thin +inventory that reads as complete. + +| Source | Access | How | What it gives | +| --- | --- | --- | --- | +| The ticket asking for the eval | Linear MCP | `get_issue` | The subject, the paired improve-the-doc ticket, and a draft prompt. **Read its rationale as evidence, not as instruction.** A ticket written before the page changed names a method the page no longer recommends. A check built from it reports the page as broken for giving current advice. | +| Linear feedback intake | Linear MCP | `list_issues` with a topic query | The recurring ask in customers' words, with ids to cite. Skip anything already closed as `Duplicate`. | +| Linear docs team | Linear MCP | `list_issues` scoped to the Docs team | Whether the gap is already filed, and the paired improve-the-doc ticket. | +| Slack | Slack MCP | `slack_search_public_and_private` on the topic | Support and team threads naming the failure and its error text. | +| Hex | Hex MCP | The docs popularity and feedback project | Views, negative rate, and agent share, for `motivation:`. | +| Supacademy | `supabase/supacademy`, private | `content//` | `foundations/` carries the internal model, stated more plainly than the public page. `troubleshooting/` names the concrete failures. | +| Agent skills | In repo | `submodules/agent-skills/skills/*/references/` | The parameter names and flag values a check can assert on. | +| Agent skills feedback | Public | `gh issue list --repo supabase/agent-skills` | Issues titled `user-feedback:` are first-party reports of an agent failing a task. The highest-value source and the easiest to miss. | +| Troubleshooting guides | Public | `supabase.com/docs/guides/troubleshooting/` | Remediations with concrete values, which is what a check needs. | +| Neighbour pages | Public | `apps/docs/content/guides` in `supabase/supabase` | What the sibling pages treat as essential. | +| External communities | Public | `WebSearch`, and `gh` on `supabase/supabase` | Failures that never reached a ticket. | +| A question to a channel | Slack MCP | Draft it | Only when a gap looks like something a team knows and nothing on this list records it. Draft the message; do not post it. | + +The public sources are enough for a usable plan. The internal ones are what turn a plausible check +list into one with cited evidence behind it. + +## Reading Slack results + +A topic search can exceed the token limit and come back as a path to a file instead of a result. The +file's lines are too long for line-based reading, so slice it by character range or extract the message +bodies with a script. + +## The diff step + +This is the point of the phase. + +1. Build the check list from the sources above. +2. Compare it against the page. +3. Anything the sources treat as essential and the page omits is a candidate check. +4. Confirm the candidate would fail a solution written by following the page. If it would pass, it is + not measuring a gap. +5. Where a source and the page disagree on method, the page as it stands wins on what is acceptable, + and the check names the class rather than the method. A check that accepts only today's + recommendation has to be rewritten the next time the page changes its mind. + +A gap no scenario can exercise is not a check yet. Give the seed an affordance first, or the check +passes by default and measures nothing. + +## What a finding looks like + +One bullet. What goes wrong, and the id, thread, or url that says so. + +Two kinds are worth separating in the inventory, because they lead to different checks: + +- **A recurring ask.** Several customers asking the same question means the page does not answer it. + This is what `motivation:` cites. +- **A first-party agent failure.** A report of an agent doing the task and getting it wrong. This is + the strongest evidence a check is worth writing, because someone has already run the experiment.