From 008f8327d32dfd224b32444ca22c312d1170eed5 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Wed, 2 Sep 2026 16:25:45 -0700 Subject: [PATCH 01/11] feat(skills): add a planning skill for documentation evals Four build-docs evals exist and each was designed from scratch. The most recent cost three false reds and one false green, all of them mistakes an earlier author had already made. The skill plans a documentation eval and stops at the plan. Six phases: enter plan mode and read CONTRIBUTING.md, gather the evidence, pick the one claim the page has to transmit, draft the prompt, design the checks, write the plan, and feed the catalog once the baseline lands. Five references carry the catalogs. sources.md lists every evidence source and marks which need an MCP connector or private access, because an agent that silently skips a gated source produces a thin inventory that reads as complete. check-rules.md is ordered by what a violation costs, and the first three rules are the ones that have sunk a run. flakiness.md is mined from the review feedback on twenty-one PRs, with a citation per entry. It lives under .claude/skills/ rather than skills/, which holds product skills that experiments install into eval sandboxes. Closes DOCS-1228 --- .claude/skills/docs-eval-planning/SKILL.md | 119 +++++++++++++++ .../references/canonical-shape.md | 116 ++++++++++++++ .../references/check-rules.md | 120 +++++++++++++++ .../references/flakiness.md | 144 ++++++++++++++++++ .../references/prompt-rules.md | 84 ++++++++++ .../docs-eval-planning/references/sources.md | 54 +++++++ 6 files changed, 637 insertions(+) create mode 100644 .claude/skills/docs-eval-planning/SKILL.md create mode 100644 .claude/skills/docs-eval-planning/references/canonical-shape.md create mode 100644 .claude/skills/docs-eval-planning/references/check-rules.md create mode 100644 .claude/skills/docs-eval-planning/references/flakiness.md create mode 100644 .claude/skills/docs-eval-planning/references/prompt-rules.md create mode 100644 .claude/skills/docs-eval-planning/references/sources.md diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md new file mode 100644 index 00000000..af6c402d --- /dev/null +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -0,0 +1,119 @@ +--- +name: docs-eval-planning +description: Plan a documentation eval for supabase/evals, where a docs guide is the subject under test. Use when asked to write, add, or design an eval for a Supabase docs guide, when a ticket asks for a deterministic eval on a page, or when deciding what a guide-under-test eval should check. Produces a plan, not files. Not for debugging a scorer, running an eval, or writing example solutions. +--- + +# Planning a documentation eval + +A documentation eval measures a page, not an agent. The prompt sends a user's request plus the page's +url, and the checks say whether an agent that read the page produced working code. A gap in the page +counts as a failure. + +That framing is the whole difficulty. Four of these exist, and the ways they went wrong are collected +in [references/flakiness.md](references/flakiness.md). Read it before designing checks, not after. + +## What this skill produces + +A plan. It does not write `PROMPT.md`, `EVAL.ts`, or a seed. Implementation and local scoring are a +separate job. + +## Phase 0: plan mode and repo rules + +Call `EnterPlanMode` first. Everything in the gathering phase is read-only, and plan mode is what +enforces that. + +Then read, in this order: + +1. `CONTRIBUTING.md`. It carries the repo's rules for any eval: suite selection, the folder shape, the + `motivation:` requirement, prompt discipline, scorer discipline, and how to refresh results in CI. + This skill defers to it and does not restate it. Where this skill goes further, it says so. +2. The guide under test, as its `.md` variant. +3. [references/canonical-shape.md](references/canonical-shape.md), for the conventions the existing + evals settled. + +## Phase 1: gather the evidence + +Work through [references/sources.md](references/sources.md). Run independent sources in parallel. + +**The checks come from outside the page.** A check list derived from the guide inherits the guide's +blind spots, so every check passes and the score reports that the guide is fine when nobody looked. +Build the list from the sources, then diff it against the page. Anything the sources treat as essential +and the page omits is a candidate check, and it should fail a solution written from the page. That +failure is the finding the paired docs ticket acts on. + +Half the sources need an MCP connector. If one is missing or unauthorized, stop and ask for it rather +than working around it, then record in the plan which sources were reachable. An inventory built from +public sources alone is a weaker artifact and the plan has to say so. + +This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a +thread, or a url. A finding with no source does not go in. + +## Phase 2: pick the one claim + +Name the single thing the guide has to transmit, in the user's terms rather than the product's. + +One claim, not three. An eval that measures several unrelated things reports which of them an agent +got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is +what makes a saturated check obvious later. + +## Phase 3: draft the prompt + +Work through [references/prompt-rules.md](references/prompt-rules.md). + +The prompt is a product request plus the page's url, written the way the user would write it, with the +vocabulary that gives away the answer removed. Detail that a real user would have moves into the seed. + +This phase ends with the prompt body, the list of stripped words, and the `motivation:` frontmatter. + +## Phase 4: design the checks + +Work through [references/check-rules.md](references/check-rules.md), in order. The first three rules +are the ones that have cost the most. + +This phase ends with a table: the check name, what it proves, what it does when the object is absent, +whether it reads files or runs code, and the evidence it came from. + +Then predict which checks will saturate, and write the prediction down. On the most recent eval every +check but one came back 6/6, because the seed labelled the answer. A plan that says which checks are +cheap is honest about how much signal the eval carries. + +## Phase 5: write the plan + +Sections: context, failure-point inventory, the seed, the prompt, the check table, what is not scored, +the expected-failure table, verification, the paired docs ticket, and Phase 6 as the closing step. + +**Everything in the prompt is either measured or named as out of scope.** A reviewer finds the rule +that is neither. + +## Phase 6: feed the catalog + +The plan's last step is to append whatever broke to [references/flakiness.md](references/flakiness.md) +once the baseline lands, with the PR number. + +This is a plan step rather than advice, because the file is only worth having if it grows. Ask +reviewers to write findings into it directly rather than leaving them in a review thread where the +next author will not look. + +## Content shape against end-to-end proof + +Two kinds of check, and a plan needs both. + +**A check that reads files** asserts content shape. It is cheap, stable, and has no opinion about +whether anything runs. Every one of them passes for an agent that edited a config file and stopped. + +**A check that runs the code** asserts the outcome. It is what stops a decorative pass, and it is +slower and more exposed to the environment. + +Across the mature evals the ratio runs about six shape to four outcome. The table says which each +check is, because the two fail for different reasons and a reviewer reading a red needs to know which +kind they are looking at. + +## Where the boundary sits + +Some claims are not this skill's to make. Whether the page carries runnable commands in fenced blocks, +declares its environment variables and prerequisites, and ends with a verification step is a claim +about the page's own markdown. That belongs to the page and to the project's authoring rubric. No +tooling in this repo asserts it; the docs end-to-end suite is rendering, links, and accessibility. + +What belongs here is the claim about what an agent produces after reading the page. Mixing the two +produces an eval that fails when someone reformats a code block. diff --git a/.claude/skills/docs-eval-planning/references/canonical-shape.md b/.claude/skills/docs-eval-planning/references/canonical-shape.md new file mode 100644 index 00000000..b6d9b161 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/canonical-shape.md @@ -0,0 +1,116 @@ +# The shape a documentation eval takes + +Drawn from `build-docs-002-rls-guide`, `build-docs-003-api-keys-guide`, and +`build-docs-004-postgres-connection`. Where `002` differs it is usually the first of the genre rather +than an alternative, so `003` and `004` are the reference. + +`build-docs-001-homepage-quickstart` is a different genre. Its subject is a prompt string on the docs +homepage, not a page, so it has no REFERENCE block, no guide-read check, and no stripped vocabulary, +and its `skills: []`, `skipCliInstall: true`, and `projectRunning: false` follow from a prompt that +tells the agent to install its own tooling. Do not generalize from it. + +## Naming and files + +`build-docs-NNN-`, named after the doc rather than the feature. **The id is permanent.** The +published results series is keyed on it, so renaming breaks history. + +``` +evals/build-docs-NNN-/ + PROMPT.md frontmatter and the task the agent sees + EVAL.ts the scorer + README.md the design rationale, addressed to the next editor + .ts flat beside EVAL.ts, never in a subdirectory + local/ the seed workspace +``` + +`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is mandatory: it is the only +place the stripped-word list and the do-not-remove-this constraints live. + +## EVAL.ts is orchestration + +Sixty-seven to a hundred and twenty lines. Import named `check*` functions, assemble one flat array, +return `{ passed: checks.every(c => c.passed), checks }`, and wrap the whole thing in a try/catch whose +catch returns a single self-named failure check. + +**Split the implementations into modules and keep the full check list declared here.** One reviewer +asked for a 791-line scorer to be split; four PRs later another objected that splitting obscures the +list. Doing both satisfies both. + +Comment why the phase order is what it is. Every eval has an ordering constraint and every one of them +documents it. + +## The README section set + +From `003` and `004`: + +- **What this eval measures.** The page is the subject, not the agent. A gap in the page is a failure. +- **Do not reintroduce the vocabulary.** The stripped words, listed. +- **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs. +- **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which + ones make them mean something. +- **The guide has to actually be read.** +- **What this eval does not score.** Each entry with its reason. This is where an unmeasured rule from + the prompt gets named. + +## The seed + +`local/` only; none of these use `remote/`. + +Product vocabulary in seed comments, never the vocabulary the prompt strips. The comments carry the +contract the prompt cannot state: the endpoint the rest of the team builds against, the shapes a +handler must return, which table the API serves, and why something is pre-solved. + +Pre-solve what is a different scenario's subject and say so in the comment. Grants are pre-solved in +the API keys eval because working out grants is not the same question as working out which key goes +where, and a grant mistake would fail the eval for the wrong reason. + +**Defeat memorized answers.** The connection eval's fixture uses `aws-1`, because `aws-0` is what +agents recite from memory, so a whitelist only measures anything when the project is on something else. + +**Build in a conflict where you can.** The RLS eval seeds two apps so one access pattern cannot cover +both: `using (true)` is correct on the public feed and catastrophic on the private one. + +## One real authoring choice + +`002` puts the task contract nowhere and measures whether the agent derives it. `003` and `004` put it +in seed comments and buy a positive control the scorer can prove. + +Both are defensible and they measure different things. Deriving the contract is a harder, less +observable test. Giving it costs a discovery question and buys provability. Decide deliberately and +record which in the `README.md`. + +## Example solutions + +A reviewer spelled the method out on the first of these evals: one green solution you believe is +correct, plus a few carrying a single deliberate flaw each. Write down which checks you expect each to +fail *before* running anything; that list is the test. Do not aim for exactly one failure per bad +solution, since a real flaw usually trips several. + +**They are not committed.** Two PRs proposed committing them and both were closed, on the grounds that +it increases the volume of material to review per scenario. So `evals/*/solutions/` is a git exclusion +rather than a directory, and the practice survives locally. + +Two limits worth knowing before relying on them: + +- **`.git/info/exclude` is local-only and never committed.** A fresh clone does not have the exclusion. + Add it before the first `git add` of an eval directory. +- **A transcript-anchored check cannot be exercised by a solution at all**, because no agent ran. The + guide-read check fails on every solution by construction. + +## Judges + +`003` and `004` are fully deterministic. `002` uses one judge, for pgTAP test files, where the artifact +class is genuinely unbounded. + +If you need one: scope the rubric to what the check is worth, state explicitly what not to grade, and +give it a tie-break. An eval passes only when every check passes, so a bonus check with an ambitious +rubric can turn a whole page red while forty real checks are green. + +## The guide-read check + +Every page-under-test eval has one, and it is currently copy-pasted into three scorers. +`core/share-docs-guide-read-check` moves it into core as `checkDocsGuideRead`. Import it if that has +landed; otherwise copy `build-docs-004`'s and leave a comment pointing at the branch. + +It resolves the url from the harness's own docs result rather than the raw tool call, because a +`search_docs` hit carries the guide's url in its result rather than its request. diff --git a/.claude/skills/docs-eval-planning/references/check-rules.md b/.claude/skills/docs-eval-planning/references/check-rules.md new file mode 100644 index 00000000..6bcd3685 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/check-rules.md @@ -0,0 +1,120 @@ +# Designing the checks + +Ordered by what a violation costs. The first three have each sunk a run. + +## 1. The check must be observable in the end state + +If the correct answer is to let the platform supply a value, a correct agent writes nothing and the +check reds correct work. + +The connection eval scored 0/6 on its central check for exactly this. The seed told the agent the +deploy sets `DATABASE_URL`, so agents read it from the environment and recorded the string nowhere. +One run named the transaction pooler and port 6543 in its own code comment and still scored red. The +choice was right and unmeasurable. + +**Before writing a check, decide where its answer lands.** Give the seed a place for it: an empty +`.env.example`, a config file, a named file the contract points at. If there is nowhere for the answer +to go, there is no check yet. + +## 2. Behaviour over structure + +A check that reads *where* code sits reds correct refactorings. + +Reading whether a client was constructed inside a function red a factory called once at module scope, +which is correct. Counting how many connections open across repeated invocations does not care how the +client was built. + +The same rule kills static scans of source and config, which are defeated by four independent moves: +a shell `export` prefix in a `.env` file, a config value reached through a variable, a renamed +environment variable, and a wrapper function around the call being matched. A reviewer asked for the +same remedy four times on one PR: commit the counterexample as a fixture, or replace the static claim +with a behavioral probe. + +Where a static check is the only option, name it for what it proves and say in the `README.md` that it +is a source-level claim. + +## 3. Positive controls, and not ones a literal satisfies + +Every absence check needs a paired proof that the allowed path changes state. + +"No secret key in the bundle" and "not the direct connection" both pass for an agent that built +nothing. The paired control is what makes them mean anything. + +**A control must not be satisfiable by a literal.** A control that greps the response for a seeded +value passes a hardcoded array containing that value. Insert a row with a run-scoped marker *after the +code was written* and require the marker back. + +## 4. Every call site qualifies, not the first + +Reading `sites[0]` passes a workspace holding one correct client and one wrong one somewhere else. +Range over all of them. + +## 5. Write the absence semantics down, per check + +Decide what the check does when the object it inspects does not exist, put it in the table, then read +the code to confirm it does that. + +"RLS is enabled on `todos`" fails if `todos` is missing. "No client role can read a materialized view" +passes if the agent made none. A check that is conditional says so in its notes, so a not-applicable +pass does not read as a real one. + +## 6. Whitelist, not blacklist + +Enumerating spellings cannot close a class. Six key-name patterns still missed two more. A whitelist of +what the project was given fails everything else by construction. + +## 7. Name the check for what it proves + +Not for what you meant. "No write grant anywhere" proven by three privileges is a false name while a +fourth is still reachable. + +Names are also history: the published results series is keyed on them, so renaming a check breaks it. +Get the name right before the first run. + +## 8. The environment must not decide the verdict + +TLS, address family, and ports have each red correct work. + +A handler that sets `ssl: 'require'` cannot connect to a local database that speaks no TLS. A direct +connection is IPv6-only. A stale token shadows a real one. In each case the eval reports the +environment rather than the page. + +Two ways out. Make the harness support the correct behaviour, which is what terminating TLS in front of +the database does. Or declare the claim unscored in the `README.md` and say why. + +The same trap catches a check that measures the platform. Two checks were built, shipped, and reverted +for scoring which keys a runtime hands a function, which the pinned CLI decides. Keeping either means a +CLI upgrade reports as a docs change. + +## 9. Predict which checks will saturate + +If the seed labels the answer, the check is free. + +The connection eval's fixture names the three connection modes the way the dashboard does, so picking +the right one costs an agent nothing, and that check came back 6/6. Every check but one did. Write the +prediction into the plan, so a reviewer sees where the signal actually is. + +## 10. A blocked check fails, it never skips + +A missing artifact is the absence of evidence. Reporting it green hands a clean sheet to a run that +produced nothing. Give the blocked path its own note saying why it did not run. + +## 11. Split the phases, and say why the order matters + +Static scans run before anything installs or writes. Snapshot state before running anything the agent +wrote. A probe that mutates a fixture runs after every probe that reads it. + +Scope every marker to the run. A fixed literal leaks between runs. + +**Return errors instead of throwing.** A throwing setup step collapses a whole result to one check, and +the drop is only visible by comparing check counts against the previous commit. Fold the error into a +check of its own so a failed step costs only the checks that needed it. + +## 12. Reach for a judge only when the artifact class is unbounded + +Free prose or arbitrary test files, not anything a query or a file read can settle. + +When you do: one claim per check, a checklist rubric rather than an adjective, an explicit statement of +what not to grade, and a tie-break. And read the notes on a run that failed before blaming the judge. +Effort was raised twice on one rubric before someone checked and found the variance was in the agent's +output, not the judge. diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md new file mode 100644 index 00000000..576001d3 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -0,0 +1,144 @@ +# What has gone wrong before + +Every entry is from a run or a review, with the PR that recorded it. A citation per entry is what keeps +a plausible-sounding rule from creeping in. + +Add to this file. A documentation eval is not finished until whatever went wrong on it is here. + +## False green: the check passes broken work + +- **A static scan is defeated by aliasing, four ways.** In a `.env` file, `export VITE_ADMIN=` + slips past a key-name parse. A bundler config that computes `envPrefix` through a variable is + invisible to a literal read of the config. An environment variable renamed to an innocuous alias + defeats a name blocklist. A wrapper function defeats a match on the call inside it. #212 + **Fix.** Commit the counterexample as a local fixture, or replace the static claim with a behavioral + probe. +- **A literal match counts dead code as working code.** `.auth.signUp(` anywhere in client source + satisfied a check, including in code nothing reaches. #212 + **Fix.** Pair it with a check that proves the path runs, and name the check as a literal match. +- **A control that greps for a seeded value passes a hardcoded array.** A handler returning a literal + containing the seeded row's name cleared a read control. #257 + **Fix.** Insert a row with a run-scoped marker after the code was written and require the marker back. +- **Reading only the first call site.** Checking `sites[0]` cleared a workspace holding one correct + client and one constructed per request elsewhere. #257 + **Fix.** Range over every site. +- **Enumerating spellings cannot close a class.** Six key-name patterns still missed two more. #168 + **Fix.** Whitelist what the project was given and fail everything else. +- **The result is green when nothing was read.** `!error && leaked.length === 0` is true of an empty + result set. #168 + **Fix.** Assert the subject's own row is present, then that nobody else's is. +- **Repointing REFERENCE at a docs preview without matching the preview host.** The guide-read check + matches on `includes`, so a fetch of the published page satisfies a check meant to prove the preview + was read. #218 + **Fix.** Match the full preview host, not the path. + +## False red: the check fails correct work + +- **A structural read of where code sits.** Requiring the client constructor at module scope red a + factory called once at module scope, which is correct. #257 + **Fix.** Count the effect. One connection across repeated invocations proves the same thing and does + not care how the client was built. +- **The correct answer is not observable.** Agents left a connection string to the deploy, as the seed + told them to, so nothing in the workspace carried their choice and the central check red all six + runs. One of them named the right answer in a code comment. #257 + **Fix.** Give the choice somewhere to land in the seed. +- **The environment cannot support the correct behaviour.** A handler setting `ssl: 'require'` could + not reach a local database that speaks no TLS, so both end-to-end checks red on a more correct + solution. #257 + **Fix.** Terminate TLS in front of the database, or declare the claim unscored. +- **The sandbox image made the task unwinnable.** `npm install -g` failed with `EACCES` because the + image ran as a user with no writable global prefix, so the agent could not complete step one. #108 + **Fix.** Change the image, and add a test that locks the behaviour in. +- **A prompt ambiguity the checks resolve one way.** An agent gated an endpoint on being signed in, + which the prompt never ruled out, and it red in CI. #212 + **Fix.** Either the check accepts it, or the prompt says which it wants. +- **A design choice scored as a defect.** A probe requiring two rows in one response would fail an + agent that paginated. #212 + **Fix.** Name it in the README as a narrow edge, or probe in a way that tolerates it. +- **The check measures the platform, not the page.** Two checks scored which keys a runtime hands a + function, which the pinned CLI decides. Both were reverted. #212, #218 + **Fix.** Drop it. Keeping it means a CLI upgrade reports as a docs change. +- **The guide-read check drops non-apex hosts.** `isSupabaseApexUrl` accepts only `supabase.com`, so a + preview fetch never reaches `docsCalls` and the check reds a run that did read the page. #218 + **Fix.** Widen the host predicate when pointing an eval at an unmerged docs PR. +- **A regex over a tool call is too strict.** A plugin-install check missed the command because the + agent added `--yes`. #108 + **Fix.** Match the part that carries the meaning. +- **Extension-owned objects land in scope.** pgTAP creates views in `public` when installed there. #168 + **Fix.** Anti-join `pg_depend` on `deptype = 'e'`. + +## Nondeterminism + +- **Judge variance was misdiagnosed twice.** A rubric's complaint changed run to run. The first + response blamed reasoning effort and proposed a per-eval override; the second raised the global + default; the third proved the variance was never in the judge. It tracked whichever test suite the + agent wrote that run. #220, #222, #224 + **Fix.** Read the notes on the failing runs before blaming the judge. Then scope the rubric to what + the check is worth, drop every craft requirement, and give it a tie-break. +- **One ambitious check fails a whole eval.** An eval passes only when every check passes, so a bonus + check turned a page red while forty real checks were green. #224 + **Fix.** Scope a bonus check to a low bar, and rename it if the old name claimed more. +- **Retry layers multiply.** A pair-level retry wrapping a create-level retry produced 39 attempts over + 588 seconds for one unrecoverable credential error. At high concurrency a single bad secret makes + every pair thrash. #192 + **Fix.** Classify errors as retryable or not, and do not nest retries. +- **Stop-on-pass retries hide variance.** `attempts: 2` means attempt one failed. That can be agent + variance rather than a scorer defect, and the two read identically in a summary. #218 + **Fix.** Read the attempts, and treat one green run as a signal rather than proof. +- **Fixed marker literals leak between runs.** #168 + **Fix.** Scope markers to the run. + +## Crashes and contamination + +- **A throwing setup step collapses the result.** One uncaught insert took a 35-check result down to + one, visible only by comparing check counts against the previous commit. #168 + **Fix.** Return the error and fold it into a check of its own. +- **`ON CONFLICT` pins a schema the prompt invites the agent to change.** #168 + **Fix.** `INSERT ... SELECT ... WHERE NOT EXISTS`. +- **A probe mutates what a later check reads.** #168 + **Fix.** Snapshot the catalog first, run the agent's own suite last, and put a probe that rewrites a + fixture after every probe that reads it. +- **`export-results` merge never prunes.** It overwrites keys present in the new results and leaves + stale keys alive, so renaming an eval leaves the old row beside the new one and a skipped pair keeps + its last recorded verdict forever. Found twice, six weeks apart, by two reviewers. #168, #108 + **Fix.** Pass the requested pairs and drop rows that were requested but produced nothing. +- **An empty `results/` overwrites the export with `[]`.** Easy to do by accident. #228 +- **A bot pushed refreshed results onto a measurement branch**, where the results were measured against + a preview url rather than the published page. #218 + **Fix.** Do not refresh results from a scratch branch. + +## Environment and reproducibility + +- **Ports.** The range is 54321 to 54329. Another project's stack holding them blocked local runs + entirely, and one eval shipped for review having never run end to end because of it. #168, #206 + **Fix.** Preflight the ports. Both later evals open their testing steps with it. +- **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. The same + unattributed failure appears six weeks earlier. #228, #168 +- **A stale local `VERCEL_OIDC_TOKEN` shadows the real one** and every sandbox create fails. #192 +- **An env-isolation assertion cannot be strict.** Vite sets `NODE_ENV` and macOS injects + `__CF_USER_TEXT_ENCODING` even with a controlled environment, so asserting an exact key set fails. + #188 + +## Metadata + +- **An optional dimension is a silent hole.** `interface:` was optional, so six scenarios never + declared one and fell out of any categorization on that axis. #228 + **Fix.** Require it, so the gap becomes a load-time error. +- **`motivation:` is required by `CONTRIBUTING.md` and is not in the schema**, so the preprocess step + drops it and two evals have none. #228 +- **Renaming breaks history.** The published series is keyed on the eval id and on check names. #168, + #230 + **Fix.** Get both right before the first run, and keep name strings byte-identical when refactoring. + +## Process + +- **Verify your own predictions.** A PR body predicted a check would fail an agent following the page. + The reviewer found it passing locally and in CI and asked why. #212 +- **An unmeasured rule in the prompt is a defect.** A reviewer found a prompt rule with no check and no + README entry. #168 + **Fix.** Measure it or name it out of scope. +- **The harness can leak the answer.** A system-prompt addendum gave agents a head start on what they + should have discovered. #108, #241 +- **The scorer copied the guide's own AI prompt panel into `PROMPT.md`**, so the rules that spell out + the answer were the prompt. #168 + **Fix.** The prompt is a product request. The page is the thing being tested. diff --git a/.claude/skills/docs-eval-planning/references/prompt-rules.md b/.claude/skills/docs-eval-planning/references/prompt-rules.md new file mode 100644 index 00000000..f1611967 --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/prompt-rules.md @@ -0,0 +1,84 @@ +# Drafting the prompt + +The prompt is a product request plus the page's url. Nothing else. + +`CONTRIBUTING.md` already says prompts read as a real user's message, do not spoonfeed, and move +detail into seed data. Everything here is the strict reading of that for a page under test. + +## Strip the vocabulary that gives away the answer + +Write the request the way the user would write it, in product terms. Never name the mechanism, the +setting, or the concept the page teaches. + +The stripped words are the measurement. An agent that only writes secure policies when the prompt says +"security" has not been served by the guide. + +Worked examples: + +| Eval | Words the prompt never says | +| --- | --- | +| RLS guide | RLS, policy, security, role, tenant, test | +| API keys guide | key, publishable, secret, service role, server, expose, environment variable | +| Connecting to Postgres | pooler, pooling, transaction, session, port, prepared statement, connection string | + +**Say where the code runs. Never say how to reach the service.** "I'm deploying this to Vercel +serverless functions" is the user's own framing. "Use the transaction pooler" is the answer. + +**List the stripped words in the eval's `README.md`.** They creep back otherwise, one loosening at a +time, and the next editor has no way to know they were deliberate. + +## Strip from the prompt, not from everything + +Fixture material a user would already have on screen may carry the vocabulary. The connection eval's +seed prints "Transaction pooler" and the port numbers, because that is what the dashboard's Connect +panel prints, and pretending otherwise would be a stranger fixture than the real one. + +The line is the prompt. The prompt is the user's own words; the seed is the world the user is already +in. + +## Keep the reliance instruction + +End the request with this, verbatim: + +``` +Read the guide below before you start and rely on it for how to set this up, +rather than on what you already know. +``` + +Without it, a pass is evidence about the model rather than the page. The older, weaker form is "Read +this guide first and follow it". Use the sentence above. + +## The REFERENCE block + +Bare. No colon, no markdown link, no code fence. + +``` +REFERENCE +https://supabase.com/docs/guides/.md +``` + +Use the `.md` variant. That is what an agent retrieves, and the guide-read check resolves against it. + +## Do not spoonfeed, and watch for what the prompt leaves open + +Detail belongs in the seed. A seeded table resolves column names, a seeded file fixes a contract, and +a comment explains structure the code does not show. + +**If the prompt does not say, the check passes.** One eval red an agent that gated an endpoint on being +signed in, which the prompt never ruled out. The reviewer's read is the rule: either the check accepts +it, or the prompt says which it wants. An ambiguity the checks resolve one way is a false red waiting +for the run that resolves it the other way. + +## Frontmatter + +- `stage: build`, `interface: cli`, and `suite: regression`. Benchmark is gated on at least one agent + failing the scenario, so it is a question for after the baseline, not before. +- `product:` and `topic:` from the closed enums in `packages/core/src/eval-metadata.ts`. `topic` has no + `database` value; `sdk` is usually the fit. +- `services:` narrowed to what the scorer touches. This also decides what the agent can reach, which + is a design lever: excluding a service the wrong answer would need makes the wrong answer fail + locally. +- Do not add `skills: []`. The `*-no-skills` experiment variants are how a run sees the page without + the product skills. +- `motivation:` as a folded `>-` block, in three moves: what agents are pointed at this page for, the + blast radius of getting it wrong, and what the eval determines. Cite the evidence from Phase 1. diff --git a/.claude/skills/docs-eval-planning/references/sources.md b/.claude/skills/docs-eval-planning/references/sources.md new file mode 100644 index 00000000..93b9fa4c --- /dev/null +++ b/.claude/skills/docs-eval-planning/references/sources.md @@ -0,0 +1,54 @@ +# Where the checks come from + +The page under test is not a source. Everything here is. + +Half of these need an MCP connector or private-repo access. Try the source; if the tool is missing or +unauthorized, stop and ask for it. Then record in the plan which sources were reachable. Silently +skipping a gated source produces a thin inventory that reads as complete. + +| Source | Access | How | What it gives | +| --- | --- | --- | --- | +| Linear feedback intake | Linear MCP | `list_issues` with a topic query | The recurring ask in customers' words, with ids to cite. Skip anything already closed as `Duplicate`. | +| Linear docs team | Linear MCP | `list_issues` scoped to the Docs team | Whether the gap is already filed, and the paired improve-the-doc ticket. | +| Slack | Slack MCP | `slack_search_public_and_private` on the topic | Support and team threads naming the failure and its error text. | +| Hex | Hex MCP | The docs popularity and feedback project | Views, negative rate, and agent share, for `motivation:`. | +| Supacademy | `supabase/supacademy`, private | `content//` | `foundations/` carries the internal model, stated more plainly than the public page. `troubleshooting/` names the concrete failures. | +| Agent skills | In repo | `submodules/agent-skills/skills/*/references/` | The parameter names and flag values a check can assert on. | +| Agent skills feedback | Public | `gh issue list --repo supabase/agent-skills` | Issues titled `user-feedback:` are first-party reports of an agent failing a task. The highest-value source and the easiest to miss. | +| Troubleshooting guides | Public | `supabase.com/docs/guides/troubleshooting/` | Remediations with concrete values, which is what a check needs. | +| Neighbour pages | Public | `apps/docs/content/guides` in `supabase/supabase` | What the sibling pages treat as essential. | +| External communities | Public | `WebSearch`, and `gh` on `supabase/supabase` | Failures that never reached a ticket. | +| A question to a channel | Slack MCP | Draft it | Only when a gap looks like something a team knows and nothing on this list records it. Draft the message; do not post it. | + +The five public sources are enough for a usable plan. The gated ones are what turn a plausible check +list into one with issue ids behind it. + +## Reading Slack results + +A topic search can exceed the token limit and come back as a path to a file instead of a result. The +file's lines are too long for line-based reading, so slice it by character range or extract the message +bodies with a script. + +## The diff step + +This is the point of the phase. + +1. Build the check list from the sources above. +2. Compare it against the page. +3. Anything the sources treat as essential and the page omits is a candidate check. +4. Confirm the candidate would fail a solution written by following the page. If it would pass, it is + not measuring a gap. + +A gap no scenario can exercise is not a check yet. Give the seed an affordance first, or the check +passes by default and measures nothing. + +## What a finding looks like + +One bullet. What goes wrong, and the id, thread, or url that says so. + +Two kinds are worth separating in the inventory, because they lead to different checks: + +- **A recurring ask.** Several customers asking the same question means the page does not answer it. + This is what `motivation:` cites. +- **A first-party agent failure.** A report of an agent doing the task and getting it wrong. This is + the strongest evidence a check is worth writing, because someone has already run the experiment. From 018bde0c844ec6ca2cfbc992b326e3ffd7445f07 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Wed, 2 Sep 2026 16:26:23 -0700 Subject: [PATCH 02/11] docs(skills): record the format:check failure on local artifacts --- .claude/skills/docs-eval-planning/references/flakiness.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index 576001d3..898b1d81 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -118,6 +118,11 @@ Add to this file. A documentation eval is not finished until whatever went wrong - **An env-isolation assertion cannot be strict.** Vite sets `NODE_ENV` and macOS injects `__CF_USER_TEXT_ENCODING` even with a controlled environment, so asserting an exact key set fails. #188 +- **`pnpm format:check` fails on local artifacts that are nobody's diff.** Biome walks + `.claude/worktrees/`, where it hits broken symlinks, `.solution-runs/`, and the git-excluded + `evals/*/solutions/`, which is where a deliberate parse-error fixture lives. The failure reads as a + broken branch. #259 + **Fix.** Scope Biome to the paths you touched, or add those three to `biome.json`. ## Metadata From 364927432ce6e6e174ef19a6036b075cbc570c2f Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Wed, 2 Sep 2026 16:47:02 -0700 Subject: [PATCH 03/11] docs(skills): state the rules without the history behind them Applies review feedback. Every reference now states the decision rather than the case that produced it, per Google's timeless documentation guidance. canonical-shape.md drops the genre background, the scorer-structure history, and the reasoning behind not committing solutions. It says to model on build-docs-003 or later, and to never commit solutions. check-rules.md and prompt-rules.md drop their worked examples and state what to do. The motivation guidance now names the constant behind every one of these evals: evidence that an agent can read the page and do the task correctly. sources.md opens with the note that Linear, Slack, Hex, and Supacademy are internal to Supabase, since this repository is open source, and that planning from the public sources alone is a supported outcome. --- .claude/skills/docs-eval-planning/SKILL.md | 15 ++- .../references/canonical-shape.md | 103 +++++++----------- .../references/check-rules.md | 89 +++++++-------- .../references/flakiness.md | 8 +- .../references/prompt-rules.md | 28 ++--- .../docs-eval-planning/references/sources.md | 15 ++- 6 files changed, 111 insertions(+), 147 deletions(-) diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md index af6c402d..19d209e7 100644 --- a/.claude/skills/docs-eval-planning/SKILL.md +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -9,8 +9,8 @@ A documentation eval measures a page, not an agent. The prompt sends a user's re url, and the checks say whether an agent that read the page produced working code. A gap in the page counts as a failure. -That framing is the whole difficulty. Four of these exist, and the ways they went wrong are collected -in [references/flakiness.md](references/flakiness.md). Read it before designing checks, not after. +That framing is the whole difficulty. The failure modes are collected in +[references/flakiness.md](references/flakiness.md). Read it before designing checks, not after. ## What this skill produces @@ -73,9 +73,9 @@ are the ones that have cost the most. This phase ends with a table: the check name, what it proves, what it does when the object is absent, whether it reads files or runs code, and the evidence it came from. -Then predict which checks will saturate, and write the prediction down. On the most recent eval every -check but one came back 6/6, because the seed labelled the answer. A plan that says which checks are -cheap is honest about how much signal the eval carries. +Then predict which checks will saturate, and write the prediction down. A check whose answer the seed +labels costs an agent nothing and carries no signal. A plan that says which checks are cheap is honest +about how much the eval measures. ## Phase 5: write the plan @@ -104,9 +104,8 @@ whether anything runs. Every one of them passes for an agent that edited a confi **A check that runs the code** asserts the outcome. It is what stops a decorative pass, and it is slower and more exposed to the environment. -Across the mature evals the ratio runs about six shape to four outcome. The table says which each -check is, because the two fail for different reasons and a reviewer reading a red needs to know which -kind they are looking at. +The table says which each check is, because the two fail for different reasons and a reviewer reading +a failure needs to know which kind they are looking at. ## Where the boundary sits diff --git a/.claude/skills/docs-eval-planning/references/canonical-shape.md b/.claude/skills/docs-eval-planning/references/canonical-shape.md index b6d9b161..290e27fc 100644 --- a/.claude/skills/docs-eval-planning/references/canonical-shape.md +++ b/.claude/skills/docs-eval-planning/references/canonical-shape.md @@ -1,18 +1,11 @@ # The shape a documentation eval takes -Drawn from `build-docs-002-rls-guide`, `build-docs-003-api-keys-guide`, and -`build-docs-004-postgres-connection`. Where `002` differs it is usually the first of the genre rather -than an alternative, so `003` and `004` are the reference. - -`build-docs-001-homepage-quickstart` is a different genre. Its subject is a prompt string on the docs -homepage, not a page, so it has no REFERENCE block, no guide-read check, and no stripped vocabulary, -and its `skills: []`, `skipCliInstall: true`, and `projectRunning: false` follow from a prompt that -tells the agent to install its own tooling. Do not generalize from it. +Model a new eval on `build-docs-003-api-keys-guide` or later. ## Naming and files `build-docs-NNN-`, named after the doc rather than the feature. **The id is permanent.** The -published results series is keyed on it, so renaming breaks history. +published results series is keyed on it, so renaming breaks history. So does renaming a check. ``` evals/build-docs-NNN-/ @@ -23,94 +16,80 @@ evals/build-docs-NNN-/ local/ the seed workspace ``` -`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is mandatory: it is the only -place the stripped-word list and the do-not-remove-this constraints live. +`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is required: it is the only +place the stripped-word list and the constraints that must not be edited away live. ## EVAL.ts is orchestration -Sixty-seven to a hundred and twenty lines. Import named `check*` functions, assemble one flat array, -return `{ passed: checks.every(c => c.passed), checks }`, and wrap the whole thing in a try/catch whose -catch returns a single self-named failure check. +Import named `check*` functions, assemble one flat array, return +`{ passed: checks.every(c => c.passed), checks }`, and wrap the whole thing in a try/catch whose catch +returns a single self-named failure check. -**Split the implementations into modules and keep the full check list declared here.** One reviewer -asked for a 791-line scorer to be split; four PRs later another objected that splitting obscures the -list. Doing both satisfies both. +Split the implementations into modules and keep the full check list declared in `EVAL.ts`. -Comment why the phase order is what it is. Every eval has an ordering constraint and every one of them -documents it. +Comment why the phase order is what it is. ## The README section set -From `003` and `004`: - - **What this eval measures.** The page is the subject, not the agent. A gap in the page is a failure. - **Do not reintroduce the vocabulary.** The stripped words, listed. - **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs. - **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which ones make them mean something. - **The guide has to actually be read.** -- **What this eval does not score.** Each entry with its reason. This is where an unmeasured rule from - the prompt gets named. +- **What this eval does not score.** Each entry with its reason. An unmeasured rule from the prompt is + named here. ## The seed -`local/` only; none of these use `remote/`. +Use `local/`. -Product vocabulary in seed comments, never the vocabulary the prompt strips. The comments carry the -contract the prompt cannot state: the endpoint the rest of the team builds against, the shapes a -handler must return, which table the API serves, and why something is pre-solved. +Write seed comments in product vocabulary, never the vocabulary the prompt strips. The comments carry +the contract the prompt cannot state: the endpoint the rest of the team builds against, the shapes a +handler must return, which table the API serves. -Pre-solve what is a different scenario's subject and say so in the comment. Grants are pre-solved in -the API keys eval because working out grants is not the same question as working out which key goes -where, and a grant mistake would fail the eval for the wrong reason. +Pre-solve what belongs to another scenario and say so in the comment, so a mistake there cannot fail +this eval for the wrong reason. -**Defeat memorized answers.** The connection eval's fixture uses `aws-1`, because `aws-0` is what -agents recite from memory, so a whitelist only measures anything when the project is on something else. +Choose fixture values that a memorized answer gets wrong, so a whitelist check measures something. -**Build in a conflict where you can.** The RLS eval seeds two apps so one access pattern cannot cover -both: `using (true)` is correct on the public feed and catastrophic on the private one. +Seed a conflict where the subject allows one, so a single approach cannot satisfy every case. -## One real authoring choice +## Where the contract lives is a real choice -`002` puts the task contract nowhere and measures whether the agent derives it. `003` and `004` put it -in seed comments and buy a positive control the scorer can prove. +Put the task contract in seed comments and gain a positive control the scorer can prove, at the cost +of a discovery question. Leave it out and measure whether the agent derives it, which is harder and +less observable. -Both are defensible and they measure different things. Deriving the contract is a harder, less -observable test. Giving it costs a discovery question and buys provability. Decide deliberately and -record which in the `README.md`. +Decide deliberately and record which in the `README.md`. ## Example solutions -A reviewer spelled the method out on the first of these evals: one green solution you believe is -correct, plus a few carrying a single deliberate flaw each. Write down which checks you expect each to -fail *before* running anything; that list is the test. Do not aim for exactly one failure per bad -solution, since a real flaw usually trips several. +Write one solution you believe is correct, plus a few carrying a single deliberate flaw each. Write +down which checks you expect each to fail before running anything; that list is the test. A flaw +usually trips several checks, so do not aim for exactly one failure each. -**They are not committed.** Two PRs proposed committing them and both were closed, on the grounds that -it increases the volume of material to review per scenario. So `evals/*/solutions/` is a git exclusion -rather than a directory, and the practice survives locally. +**Never commit them.** `evals/*/solutions/` is a git exclusion. Add it to `.git/info/exclude` before +the first `git add` of an eval directory, because that file is local to your clone. -Two limits worth knowing before relying on them: - -- **`.git/info/exclude` is local-only and never committed.** A fresh clone does not have the exclusion. - Add it before the first `git add` of an eval directory. -- **A transcript-anchored check cannot be exercised by a solution at all**, because no agent ran. The - guide-read check fails on every solution by construction. +A transcript-anchored check cannot be exercised by a solution, because no agent ran. The guide-read +check fails on every solution. ## Judges -`003` and `004` are fully deterministic. `002` uses one judge, for pgTAP test files, where the artifact -class is genuinely unbounded. +Use a judge when the artifact class is unbounded, meaning free prose or files whose shape you cannot +predict. Use a deterministic check for anything a query or a file read settles. + +Scope the rubric to what the check is worth. State what not to grade, give the judge a tie-break, and +make one claim per check. -If you need one: scope the rubric to what the check is worth, state explicitly what not to grade, and -give it a tie-break. An eval passes only when every check passes, so a bonus check with an ambitious -rubric can turn a whole page red while forty real checks are green. +An eval passes only when every check passes, so an ambitious rubric on a secondary check fails the +whole page. ## The guide-read check -Every page-under-test eval has one, and it is currently copy-pasted into three scorers. -`core/share-docs-guide-read-check` moves it into core as `checkDocsGuideRead`. Import it if that has -landed; otherwise copy `build-docs-004`'s and leave a comment pointing at the branch. +Every page-under-test eval has one. Import `checkDocsGuideRead` from core if it has landed there; +otherwise copy the most recent eval's. -It resolves the url from the harness's own docs result rather than the raw tool call, because a +It resolves the url from the harness's docs result rather than the raw tool call, because a `search_docs` hit carries the guide's url in its result rather than its request. diff --git a/.claude/skills/docs-eval-planning/references/check-rules.md b/.claude/skills/docs-eval-planning/references/check-rules.md index 6bcd3685..7ebac423 100644 --- a/.claude/skills/docs-eval-planning/references/check-rules.md +++ b/.claude/skills/docs-eval-planning/references/check-rules.md @@ -1,34 +1,24 @@ # Designing the checks -Ordered by what a violation costs. The first three have each sunk a run. +Ordered by what a violation costs. The first three matter most. ## 1. The check must be observable in the end state If the correct answer is to let the platform supply a value, a correct agent writes nothing and the check reds correct work. -The connection eval scored 0/6 on its central check for exactly this. The seed told the agent the -deploy sets `DATABASE_URL`, so agents read it from the environment and recorded the string nowhere. -One run named the transaction pooler and port 6543 in its own code comment and still scored red. The -choice was right and unmeasurable. - **Before writing a check, decide where its answer lands.** Give the seed a place for it: an empty `.env.example`, a config file, a named file the contract points at. If there is nowhere for the answer to go, there is no check yet. ## 2. Behaviour over structure -A check that reads *where* code sits reds correct refactorings. - -Reading whether a client was constructed inside a function red a factory called once at module scope, -which is correct. Counting how many connections open across repeated invocations does not care how the -client was built. +A check that reads where code sits fails correct refactorings. Count the effect instead. -The same rule kills static scans of source and config, which are defeated by four independent moves: -a shell `export` prefix in a `.env` file, a config value reached through a variable, a renamed -environment variable, and a wrapper function around the call being matched. A reviewer asked for the -same remedy four times on one PR: commit the counterexample as a fixture, or replace the static claim -with a behavioral probe. +The same rule applies to static scans of source and config. A shell `export` prefix, a config value +reached through a variable, a renamed environment variable, and a wrapper function around the call +being matched each defeat one. Replace the static claim with a behavioral probe, or commit a +counterexample as a local solution and confirm the check rejects it. Where a static check is the only option, name it for what it proves and say in the `README.md` that it is a source-level claim. @@ -37,62 +27,57 @@ is a source-level claim. Every absence check needs a paired proof that the allowed path changes state. -"No secret key in the bundle" and "not the direct connection" both pass for an agent that built -nothing. The paired control is what makes them mean anything. +An absence check passes for an agent that built nothing, so the paired control is what makes it mean +anything. -**A control must not be satisfiable by a literal.** A control that greps the response for a seeded -value passes a hardcoded array containing that value. Insert a row with a run-scoped marker *after the -code was written* and require the marker back. +**A control must not be satisfiable by a literal.** A control that greps a response for a seeded value +passes a hardcoded value. Insert state with a run-scoped marker after the code was written, and require +the marker back. ## 4. Every call site qualifies, not the first -Reading `sites[0]` passes a workspace holding one correct client and one wrong one somewhere else. -Range over all of them. +Reading the first match passes a workspace that holds one correct call site and one wrong one. Range +over every site. ## 5. Write the absence semantics down, per check Decide what the check does when the object it inspects does not exist, put it in the table, then read the code to confirm it does that. -"RLS is enabled on `todos`" fails if `todos` is missing. "No client role can read a materialized view" -passes if the agent made none. A check that is conditional says so in its notes, so a not-applicable -pass does not read as a real one. +A check on an object the prompt names fails when the object is missing. A check on a class of object +the agent may not have created passes when there are none. A conditional check says so in its notes, +so a not-applicable pass does not read as a real one. ## 6. Whitelist, not blacklist -Enumerating spellings cannot close a class. Six key-name patterns still missed two more. A whitelist of -what the project was given fails everything else by construction. +Enumerating spellings cannot close a class; any list is incomplete by construction. Whitelist what the +project was given and fail everything else. ## 7. Name the check for what it proves -Not for what you meant. "No write grant anywhere" proven by three privileges is a false name while a -fourth is still reachable. +Not for what you meant. A name that claims a class while the check proves a few members of it is a +false name. -Names are also history: the published results series is keyed on them, so renaming a check breaks it. -Get the name right before the first run. +The published results series is keyed on check names, so renaming one breaks history. Get the name +right before the first run. ## 8. The environment must not decide the verdict -TLS, address family, and ports have each red correct work. +Transport security, address family, and port availability can each fail correct work, and the eval then +reports the environment rather than the page. -A handler that sets `ssl: 'require'` cannot connect to a local database that speaks no TLS. A direct -connection is IPv6-only. A stale token shadows a real one. In each case the eval reports the -environment rather than the page. +Two ways out. Make the harness support the correct behaviour, or declare the claim unscored in the +`README.md` and say why. -Two ways out. Make the harness support the correct behaviour, which is what terminating TLS in front of -the database does. Or declare the claim unscored in the `README.md` and say why. - -The same trap catches a check that measures the platform. Two checks were built, shipped, and reverted -for scoring which keys a runtime hands a function, which the pinned CLI decides. Keeping either means a -CLI upgrade reports as a docs change. +The same trap catches a check that scores something the platform decides rather than something the page +says. A pinned CLI version or a runtime default is not the subject under test, and a check on one +reports an upgrade as a docs change. ## 9. Predict which checks will saturate -If the seed labels the answer, the check is free. +If the seed labels the answer, the check is free and carries no signal. -The connection eval's fixture names the three connection modes the way the dashboard does, so picking -the right one costs an agent nothing, and that check came back 6/6. Every check but one did. Write the -prediction into the plan, so a reviewer sees where the signal actually is. +Write the prediction into the plan, so a reviewer sees where the signal actually is. ## 10. A blocked check fails, it never skips @@ -106,15 +91,15 @@ wrote. A probe that mutates a fixture runs after every probe that reads it. Scope every marker to the run. A fixed literal leaks between runs. -**Return errors instead of throwing.** A throwing setup step collapses a whole result to one check, and -the drop is only visible by comparing check counts against the previous commit. Fold the error into a -check of its own so a failed step costs only the checks that needed it. +**Return errors instead of throwing.** A throwing setup step collapses the whole result to one check. +Fold the error into a check of its own, so a failed step costs only the checks that needed it. ## 12. Reach for a judge only when the artifact class is unbounded Free prose or arbitrary test files, not anything a query or a file read can settle. When you do: one claim per check, a checklist rubric rather than an adjective, an explicit statement of -what not to grade, and a tie-break. And read the notes on a run that failed before blaming the judge. -Effort was raised twice on one rubric before someone checked and found the variance was in the agent's -output, not the judge. +what not to grade, and a tie-break. + +Read the notes on a failing run before blaming the judge. A rubric that complains differently each run +is usually tracking real variance in what the agent produced. diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index 898b1d81..b418f617 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -76,7 +76,7 @@ Add to this file. A documentation eval is not finished until whatever went wrong **Fix.** Read the notes on the failing runs before blaming the judge. Then scope the rubric to what the check is worth, drop every craft requirement, and give it a tie-break. - **One ambitious check fails a whole eval.** An eval passes only when every check passes, so a bonus - check turned a page red while forty real checks were green. #224 + check fails the whole page while every primary check passes. #224 **Fix.** Scope a bonus check to a low bar, and rename it if the old name claimed more. - **Retry layers multiply.** A pair-level retry wrapping a create-level retry produced 39 attempts over 588 seconds for one unrecoverable credential error. At high concurrency a single bad secret makes @@ -100,7 +100,7 @@ Add to this file. A documentation eval is not finished until whatever went wrong fixture after every probe that reads it. - **`export-results` merge never prunes.** It overwrites keys present in the new results and leaves stale keys alive, so renaming an eval leaves the old row beside the new one and a skipped pair keeps - its last recorded verdict forever. Found twice, six weeks apart, by two reviewers. #168, #108 + its last recorded verdict. #168, #108 **Fix.** Pass the requested pairs and drop rows that were requested but produced nothing. - **An empty `results/` overwrites the export with `[]`.** Easy to do by accident. #228 - **A bot pushed refreshed results onto a measurement branch**, where the results were measured against @@ -112,8 +112,8 @@ Add to this file. A documentation eval is not finished until whatever went wrong - **Ports.** The range is 54321 to 54329. Another project's stack holding them blocked local runs entirely, and one eval shipped for review having never run end to end because of it. #168, #206 **Fix.** Preflight the ports. Both later evals open their testing steps with it. -- **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. The same - unattributed failure appears six weeks earlier. #228, #168 +- **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. + #228, #168 - **A stale local `VERCEL_OIDC_TOKEN` shadows the real one** and every sandbox create fails. #192 - **An env-isolation assertion cannot be strict.** Vite sets `NODE_ENV` and macOS injects `__CF_USER_TEXT_ENCODING` even with a controlled environment, so asserting an exact key set fails. diff --git a/.claude/skills/docs-eval-planning/references/prompt-rules.md b/.claude/skills/docs-eval-planning/references/prompt-rules.md index f1611967..2442d5b2 100644 --- a/.claude/skills/docs-eval-planning/references/prompt-rules.md +++ b/.claude/skills/docs-eval-planning/references/prompt-rules.md @@ -10,28 +10,23 @@ detail into seed data. Everything here is the strict reading of that for a page Write the request the way the user would write it, in product terms. Never name the mechanism, the setting, or the concept the page teaches. -The stripped words are the measurement. An agent that only writes secure policies when the prompt says -"security" has not been served by the guide. +The stripped words are the measurement. An agent that applies a practice only when the prompt names it +has not been served by the page. -Worked examples: +The list is the vocabulary the page's own answer is made of: the mechanism, its settings, and the +concepts that name them. Write it down, and check the prompt against it. -| Eval | Words the prompt never says | -| --- | --- | -| RLS guide | RLS, policy, security, role, tenant, test | -| API keys guide | key, publishable, secret, service role, server, expose, environment variable | -| Connecting to Postgres | pooler, pooling, transaction, session, port, prepared statement, connection string | - -**Say where the code runs. Never say how to reach the service.** "I'm deploying this to Vercel -serverless functions" is the user's own framing. "Use the transaction pooler" is the answer. +**Say where the code runs and what it has to do. Never say how to reach the service.** The runtime and +the task are the user's own framing. The method is the answer. **List the stripped words in the eval's `README.md`.** They creep back otherwise, one loosening at a time, and the next editor has no way to know they were deliberate. ## Strip from the prompt, not from everything -Fixture material a user would already have on screen may carry the vocabulary. The connection eval's -seed prints "Transaction pooler" and the port numbers, because that is what the dashboard's Connect -panel prints, and pretending otherwise would be a stranger fixture than the real one. +Fixture material a user would already have in front of them may carry the vocabulary. A seed that +reproduces what a dashboard or a generated file prints is more faithful than one that strips words the +user can already see. The line is the prompt. The prompt is the user's own words; the seed is the world the user is already in. @@ -80,5 +75,6 @@ for the run that resolves it the other way. locally. - Do not add `skills: []`. The `*-no-skills` experiment variants are how a run sees the page without the product skills. -- `motivation:` as a folded `>-` block, in three moves: what agents are pointed at this page for, the - blast radius of getting it wrong, and what the eval determines. Cite the evidence from Phase 1. +- `motivation:` as a folded `>-` block. The motivation is the same every time: evidence that an agent + can read this page and perform the task correctly, with the practices the page should be teaching. + Say which page, what an agent gets wrong without it, and cite the evidence from Phase 1. diff --git a/.claude/skills/docs-eval-planning/references/sources.md b/.claude/skills/docs-eval-planning/references/sources.md index 93b9fa4c..90c24614 100644 --- a/.claude/skills/docs-eval-planning/references/sources.md +++ b/.claude/skills/docs-eval-planning/references/sources.md @@ -2,9 +2,14 @@ The page under test is not a source. Everything here is. -Half of these need an MCP connector or private-repo access. Try the source; if the tool is missing or -unauthorized, stop and ask for it. Then record in the plan which sources were reachable. Silently -skipping a gated source produces a thin inventory that reads as complete. +**Several sources are internal to Supabase.** This repository is open source, and the Linear, Slack, +Hex, and Supacademy entries below are available only to Supabase employees. An eval planned from the +public sources alone is a smaller artifact, and that is a supported outcome: say so in the plan rather +than leaving the gap implicit. + +For a source you do have: if its tool is missing or unauthorized, stop and ask for the connector rather +than working around it. Then record which sources were reachable. Silently skipping one produces a thin +inventory that reads as complete. | Source | Access | How | What it gives | | --- | --- | --- | --- | @@ -20,8 +25,8 @@ skipping a gated source produces a thin inventory that reads as complete. | External communities | Public | `WebSearch`, and `gh` on `supabase/supabase` | Failures that never reached a ticket. | | A question to a channel | Slack MCP | Draft it | Only when a gap looks like something a team knows and nothing on this list records it. Draft the message; do not post it. | -The five public sources are enough for a usable plan. The gated ones are what turn a plausible check -list into one with issue ids behind it. +The public sources are enough for a usable plan. The internal ones are what turn a plausible check +list into one with cited evidence behind it. ## Reading Slack results From b0714cdb1ba33558b54b471b3a28a6a3a3263907 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Wed, 2 Sep 2026 17:21:08 -0700 Subject: [PATCH 04/11] docs(skills): fold the shape conventions into SKILL.md canonical-shape.md read as a thin spec once its background came out. Its six items move into SKILL.md: the naming and id rule, the file layout, the EVAL.ts shape, the README section set, and the guide-read check go into a shape section before the phases; the seed rules and the contract choice join Phase 3, where the seed is drafted; judges join Phase 4; example solutions join Phase 5, where verification is planned. SKILL.md is 1,587 words, inside the range a skill body should hold. --- .claude/skills/docs-eval-planning/SKILL.md | 89 ++++++++++++++--- .../references/canonical-shape.md | 95 ------------------- 2 files changed, 74 insertions(+), 110 deletions(-) delete mode 100644 .claude/skills/docs-eval-planning/references/canonical-shape.md diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md index 19d209e7..7f3d8499 100644 --- a/.claude/skills/docs-eval-planning/SKILL.md +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -17,19 +17,53 @@ That framing is the whole difficulty. The failure modes are collected in A plan. It does not write `PROMPT.md`, `EVAL.ts`, or a seed. Implementation and local scoring are a separate job. +## The shape it takes + +Model a new eval on `build-docs-003-api-keys-guide` or later. + +`build-docs-NNN-`, named after the doc rather than the feature. **The id is permanent.** The +published results series is keyed on it, so renaming breaks history. So does renaming a check. + +``` +evals/build-docs-NNN-/ + PROMPT.md frontmatter and the task the agent sees + EVAL.ts the scorer + README.md the design rationale, addressed to the next editor + .ts flat beside EVAL.ts, never in a subdirectory + local/ the seed workspace +``` + +`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is required: it is the only +place the stripped-word list and the constraints that must not be edited away live. Its sections: + +- **What this eval measures.** The page is the subject, not the agent. +- **Do not reintroduce the vocabulary.** The stripped words, listed. +- **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs. +- **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which + ones make them mean something. +- **The guide has to actually be read.** +- **What this eval does not score.** Each entry with its reason. An unmeasured rule from the prompt is + named here. + +`EVAL.ts` is orchestration. Import named `check*` functions, assemble one flat array, return +`{ passed: checks.every(c => c.passed), checks }`, and wrap it in a try/catch whose catch returns a +single self-named failure check. Split the implementations into modules and keep the full check list +declared in `EVAL.ts`. Comment why the phase order is what it is. + +Every page-under-test eval carries a check that the referenced page was read with content. Import +`checkDocsGuideRead` from core if it has landed there; otherwise copy the most recent eval's. It +resolves the url from the harness's docs result rather than the raw tool call, because a `search_docs` +hit carries the guide's url in its result rather than its request. + ## Phase 0: plan mode and repo rules Call `EnterPlanMode` first. Everything in the gathering phase is read-only, and plan mode is what enforces that. -Then read, in this order: - -1. `CONTRIBUTING.md`. It carries the repo's rules for any eval: suite selection, the folder shape, the - `motivation:` requirement, prompt discipline, scorer discipline, and how to refresh results in CI. - This skill defers to it and does not restate it. Where this skill goes further, it says so. -2. The guide under test, as its `.md` variant. -3. [references/canonical-shape.md](references/canonical-shape.md), for the conventions the existing - evals settled. +Then read `CONTRIBUTING.md` and the guide under test, as its `.md` variant. `CONTRIBUTING.md` carries +the repo's rules for any eval: suite selection, the folder shape, the `motivation:` requirement, prompt +discipline, scorer discipline, and how to refresh results in CI. This skill defers to it and does not +restate it. Where this skill goes further, it says so. ## Phase 1: gather the evidence @@ -41,9 +75,8 @@ Build the list from the sources, then diff it against the page. Anything the sou and the page omits is a candidate check, and it should fail a solution written from the page. That failure is the finding the paired docs ticket acts on. -Half the sources need an MCP connector. If one is missing or unauthorized, stop and ask for it rather -than working around it, then record in the plan which sources were reachable. An inventory built from -public sources alone is a weaker artifact and the plan has to say so. +Several sources are internal to Supabase. If one you have access to is unauthorized, stop and ask for +the connector rather than working around it, then record in the plan which sources were reachable. This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a thread, or a url. A finding with no source does not go in. @@ -56,19 +89,38 @@ One claim, not three. An eval that measures several unrelated things reports whi got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is what makes a saturated check obvious later. -## Phase 3: draft the prompt +## Phase 3: draft the prompt and the seed Work through [references/prompt-rules.md](references/prompt-rules.md). The prompt is a product request plus the page's url, written the way the user would write it, with the -vocabulary that gives away the answer removed. Detail that a real user would have moves into the seed. +vocabulary that gives away the answer removed. + +The seed is where the detail goes. Write seed comments in product vocabulary, never the vocabulary the +prompt strips: the endpoint the rest of the team builds against, the shapes a handler must return, +which table the API serves. Pre-solve what belongs to another scenario and say so in the comment, so a +mistake there cannot fail this eval for the wrong reason. Choose fixture values that a memorized answer +gets wrong, so a whitelist check measures something. Seed a conflict where the subject allows one, so a +single approach cannot satisfy every case. -This phase ends with the prompt body, the list of stripped words, and the `motivation:` frontmatter. +**Where the task contract lives is a real choice.** Put it in seed comments and gain a positive control +the scorer can prove, at the cost of a discovery question. Leave it out and measure whether the agent +derives it, which is harder and less observable. Decide deliberately and record which in the +`README.md`. + +This phase ends with the prompt body, the list of stripped words, the seed, and the `motivation:` +frontmatter. ## Phase 4: design the checks Work through [references/check-rules.md](references/check-rules.md), in order. The first three rules -are the ones that have cost the most. +matter most. + +Use a judge only when the artifact class is unbounded, meaning free prose or files whose shape you +cannot predict. Use a deterministic check for anything a query or a file read settles. Scope the rubric +to what the check is worth, state what not to grade, give the judge a tie-break, and make one claim per +check. An eval passes only when every check passes, so an ambitious rubric on a secondary check fails +the whole page. This phase ends with a table: the check name, what it proves, what it does when the object is absent, whether it reads files or runs code, and the evidence it came from. @@ -85,6 +137,13 @@ the expected-failure table, verification, the paired docs ticket, and Phase 6 as **Everything in the prompt is either measured or named as out of scope.** A reviewer finds the rule that is neither. +Verification means example solutions: one you believe is correct, plus a few carrying a single +deliberate flaw each. Write down which checks you expect each to fail before running anything; that +list is the test. A flaw usually trips several checks, so do not aim for exactly one failure each. +**Never commit them.** `evals/*/solutions/` is a git exclusion, and you add it to `.git/info/exclude` +before the first `git add` of an eval directory, because that file is local to your clone. A +transcript-anchored check cannot be exercised by a solution, because no agent ran. + ## Phase 6: feed the catalog The plan's last step is to append whatever broke to [references/flakiness.md](references/flakiness.md) diff --git a/.claude/skills/docs-eval-planning/references/canonical-shape.md b/.claude/skills/docs-eval-planning/references/canonical-shape.md deleted file mode 100644 index 290e27fc..00000000 --- a/.claude/skills/docs-eval-planning/references/canonical-shape.md +++ /dev/null @@ -1,95 +0,0 @@ -# The shape a documentation eval takes - -Model a new eval on `build-docs-003-api-keys-guide` or later. - -## Naming and files - -`build-docs-NNN-`, named after the doc rather than the feature. **The id is permanent.** The -published results series is keyed on it, so renaming breaks history. So does renaming a check. - -``` -evals/build-docs-NNN-/ - PROMPT.md frontmatter and the task the agent sees - EVAL.ts the scorer - README.md the design rationale, addressed to the next editor - .ts flat beside EVAL.ts, never in a subdirectory - local/ the seed workspace -``` - -`CONTRIBUTING.md` makes the `README.md` optional. For a page under test it is required: it is the only -place the stripped-word list and the constraints that must not be edited away live. - -## EVAL.ts is orchestration - -Import named `check*` functions, assemble one flat array, return -`{ passed: checks.every(c => c.passed), checks }`, and wrap the whole thing in a try/catch whose catch -returns a single self-named failure check. - -Split the implementations into modules and keep the full check list declared in `EVAL.ts`. - -Comment why the phase order is what it is. - -## The README section set - -- **What this eval measures.** The page is the subject, not the agent. A gap in the page is a failure. -- **Do not reintroduce the vocabulary.** The stripped words, listed. -- **The seed carries the contract.** What the seed fixes, and what each fixed thing buys and costs. -- **Do not drop the positive controls.** Which checks pass for an agent that built nothing, and which - ones make them mean something. -- **The guide has to actually be read.** -- **What this eval does not score.** Each entry with its reason. An unmeasured rule from the prompt is - named here. - -## The seed - -Use `local/`. - -Write seed comments in product vocabulary, never the vocabulary the prompt strips. The comments carry -the contract the prompt cannot state: the endpoint the rest of the team builds against, the shapes a -handler must return, which table the API serves. - -Pre-solve what belongs to another scenario and say so in the comment, so a mistake there cannot fail -this eval for the wrong reason. - -Choose fixture values that a memorized answer gets wrong, so a whitelist check measures something. - -Seed a conflict where the subject allows one, so a single approach cannot satisfy every case. - -## Where the contract lives is a real choice - -Put the task contract in seed comments and gain a positive control the scorer can prove, at the cost -of a discovery question. Leave it out and measure whether the agent derives it, which is harder and -less observable. - -Decide deliberately and record which in the `README.md`. - -## Example solutions - -Write one solution you believe is correct, plus a few carrying a single deliberate flaw each. Write -down which checks you expect each to fail before running anything; that list is the test. A flaw -usually trips several checks, so do not aim for exactly one failure each. - -**Never commit them.** `evals/*/solutions/` is a git exclusion. Add it to `.git/info/exclude` before -the first `git add` of an eval directory, because that file is local to your clone. - -A transcript-anchored check cannot be exercised by a solution, because no agent ran. The guide-read -check fails on every solution. - -## Judges - -Use a judge when the artifact class is unbounded, meaning free prose or files whose shape you cannot -predict. Use a deterministic check for anything a query or a file read settles. - -Scope the rubric to what the check is worth. State what not to grade, give the judge a tie-break, and -make one claim per check. - -An eval passes only when every check passes, so an ambitious rubric on a secondary check fails the -whole page. - -## The guide-read check - -Every page-under-test eval has one. Import `checkDocsGuideRead` from core if it has landed there; -otherwise copy the most recent eval's. - -It resolves the url from the harness's docs result rather than the raw tool call, because a -`search_docs` hit carries the guide's url in its result rather than its request. From 2f17648a8b091725631fa3e121efcfb8b4129f3b Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Thu, 3 Sep 2026 08:30:59 -0700 Subject: [PATCH 05/11] docs(skills): close the gaps a dry run against a third guide exposed Running the skill on auth/server-side/creating-a-client found five things it does not say. Phase 2 now ends by confirming the harness can measure the claim, before any check is designed. A claim the harness cannot observe is not the claim, and finding that in Phase 4 is a rewrite rather than a design change. The three questions are where the end state lives, whether the scorer can run what has to run, and what the time budget is. Phase 2 also gives a basis for picking. A dense page yields ten evidenced candidates, so pick on consequence and prefer the failure the page itself warns about. sources.md adds the ticket as a source and says to read its rationale as evidence rather than instruction, because a ticket written before the page changed will name a method the page no longer recommends. The diff step now says the page wins on what is acceptable and the check names the class. flakiness.md gains the two entries behind the feasibility step: a framework the eval seeds is not on the host, and a first-of-its-kind toolchain in a scorer is a risk to state rather than a detail. Phase 0 reads flakiness.md, which the intro already asked for. --- .claude/skills/docs-eval-planning/SKILL.md | 34 ++++++++++++++++--- .../references/flakiness.md | 8 +++++ .../docs-eval-planning/references/sources.md | 4 +++ 3 files changed, 41 insertions(+), 5 deletions(-) diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md index 7f3d8499..1f0c8d97 100644 --- a/.claude/skills/docs-eval-planning/SKILL.md +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -60,10 +60,13 @@ hit carries the guide's url in its result rather than its request. Call `EnterPlanMode` first. Everything in the gathering phase is read-only, and plan mode is what enforces that. -Then read `CONTRIBUTING.md` and the guide under test, as its `.md` variant. `CONTRIBUTING.md` carries -the repo's rules for any eval: suite selection, the folder shape, the `motivation:` requirement, prompt -discipline, scorer discipline, and how to refresh results in CI. This skill defers to it and does not -restate it. Where this skill goes further, it says so. +Then read three things: + +1. `CONTRIBUTING.md`, which carries the repo's rules for any eval: suite selection, the folder shape, + the `motivation:` requirement, prompt discipline, scorer discipline, and how to refresh results in + CI. This skill defers to it and does not restate it. Where this skill goes further, it says so. +2. The guide under test, as its `.md` variant. +3. [references/flakiness.md](references/flakiness.md), before designing anything rather than after. ## Phase 1: gather the evidence @@ -81,7 +84,7 @@ the connector rather than working around it, then record in the plan which sourc This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a thread, or a url. A finding with no source does not go in. -## Phase 2: pick the one claim +## Phase 2: pick the one claim, and confirm it can be measured Name the single thing the guide has to transmit, in the user's terms rather than the product's. @@ -89,6 +92,27 @@ One claim, not three. An eval that measures several unrelated things reports whi got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is what makes a saturated check obvious later. +A dense page yields ten candidates, all evidenced. Pick on consequence: the failure that costs a user +the most, and among equals the one the page itself warns about. A page that carries a danger callout +has already named its own worst case. + +**Then confirm the harness can measure it, before designing a single check.** A claim the harness +cannot observe is not the claim, however well evidenced. Three questions, and read the code for the +answers rather than assuming: + +- **Does the end state exist somewhere a scorer can reach?** A file in the workspace, a row in the + database, a response from a running process. If the correct answer is something the platform supplies + at deploy time, decide now where the seed lets it land. +- **Can the scorer run whatever has to run?** A framework the eval seeds is installed and built inside + the sandbox with `ctx.exec`, not on the host: the host-side build helpers relink the workspace's + `node_modules` to the framework's, which does not carry an eval's own dependencies. Check whether any + existing eval runs the same toolchain, and treat a first as a risk to state in the plan. +- **What is the time budget?** The agent's timeout does not bound the scorer, so a long install and + build costs wall-clock rather than the agent's budget. Per-command limits are yours to set. + +Write the answers into the plan. A feasibility problem found here is a design change; found in Phase 4 +it is a rewrite. + ## Phase 3: draft the prompt and the seed Work through [references/prompt-rules.md](references/prompt-rules.md). diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index b418f617..b4e005d7 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -115,6 +115,14 @@ Add to this file. A documentation eval is not finished until whatever went wrong - **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. #228, #168 - **A stale local `VERCEL_OIDC_TOKEN` shadows the real one** and every sandbox create fails. #192 +- **A framework the eval seeds is not installed on the host.** The host-side build and test helpers + relink the workspace's `node_modules` to the framework's own, which carries no eval dependencies, so + a scorer that calls them cannot import what the seed declared. Install, build, and serve inside the + sandbox with `ctx.exec` instead. #259 + **Fix.** Make the install its own check, so a failure costs only the checks that needed it. +- **A first-of-its-kind toolchain in a scorer is a risk, not a detail.** Before designing checks around + one, confirm an existing eval runs it, and say in the plan when none does. The agent's timeout does + not bound the scorer, so wall-clock rather than budget is the cost. #259 - **An env-isolation assertion cannot be strict.** Vite sets `NODE_ENV` and macOS injects `__CF_USER_TEXT_ENCODING` even with a controlled environment, so asserting an exact key set fails. #188 diff --git a/.claude/skills/docs-eval-planning/references/sources.md b/.claude/skills/docs-eval-planning/references/sources.md index 90c24614..38cc3726 100644 --- a/.claude/skills/docs-eval-planning/references/sources.md +++ b/.claude/skills/docs-eval-planning/references/sources.md @@ -13,6 +13,7 @@ inventory that reads as complete. | Source | Access | How | What it gives | | --- | --- | --- | --- | +| The ticket asking for the eval | Linear MCP | `get_issue` | The subject, the paired improve-the-doc ticket, and a draft prompt. **Read its rationale as evidence, not as instruction.** A ticket written weeks before the page changed will name a method the page no longer recommends, and a check built from it reports the page as broken for giving current advice. | | Linear feedback intake | Linear MCP | `list_issues` with a topic query | The recurring ask in customers' words, with ids to cite. Skip anything already closed as `Duplicate`. | | Linear docs team | Linear MCP | `list_issues` scoped to the Docs team | Whether the gap is already filed, and the paired improve-the-doc ticket. | | Slack | Slack MCP | `slack_search_public_and_private` on the topic | Support and team threads naming the failure and its error text. | @@ -43,6 +44,9 @@ This is the point of the phase. 3. Anything the sources treat as essential and the page omits is a candidate check. 4. Confirm the candidate would fail a solution written by following the page. If it would pass, it is not measuring a gap. +5. Where a source and the page disagree on method, the page as it stands wins on what is acceptable, + and the check names the class rather than the method. A check that accepts only today's + recommendation has to be rewritten the next time the page changes its mind. A gap no scenario can exercise is not a check yet. Give the seed an affordance first, or the check passes by default and measures nothing. From 64c208419976a09b130ab0eca0c61dbae7a74c37 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Thu, 3 Sep 2026 14:44:07 -0700 Subject: [PATCH 06/11] docs(skills): record what the server-side auth eval run taught Four framework entries and two on reading a result. The saturation pair is the one that matters: a full-marks baseline is a finding when the fixtures prove the checks can fail, and it still does not attribute the success to the page. --- .../references/flakiness.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index b4e005d7..2cd63c5b 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -115,6 +115,15 @@ Add to this file. A documentation eval is not finished until whatever went wrong - **`pnpm check` needs `OPENAI_API_KEY`** and stops without it, which reads as a broken diff. #228, #168 - **A stale local `VERCEL_OIDC_TOKEN` shadows the real one** and every sandbox create fails. #192 +- **`ctx.stackStatus()` requires three values together** and throws when any is missing, so an eval + that needs only the api url and one client key cannot use it. Read `supabase status` directly and + accept `PUBLISHABLE_KEY` or `ANON_KEY`. #261 +- **A leftover git-excluded directory under `evals/` breaks discovery for the whole repo.** + `discoverEvals` reads `PROMPT.md` in every directory it finds and throws when one is absent, so a + `solutions/` directory left behind by a branch switch stops every eval from being discovered. #261 + **Fix.** Move the leftover aside. Discovery skipping a directory with no `PROMPT.md` is the real fix. +- **`kong` does not come up on its own.** With `services: [gotrue, kong]`, `supabase status` reports no + `API_URL`. Adding `postgrest` brings it up. #261 - **A framework the eval seeds is not installed on the host.** The host-side build and test helpers relink the workspace's `node_modules` to the framework's own, which carries no eval dependencies, so a scorer that calls them cannot import what the seed declared. Install, build, and serve inside the @@ -143,6 +152,17 @@ Add to this file. A documentation eval is not finished until whatever went wrong #230 **Fix.** Get both right before the first run, and keep name strings byte-identical when refactoring. +## Reading a result + +- **Saturation is a finding, not a defect, when the checks are known to fail something.** An eval whose + every check passes on every run has measured that the page works, provided the fixtures prove the + checks can fail. Without that proof the same result is indistinguishable from a scorer that asserts + nothing. Score the fixtures before reading a full-marks baseline. #261 +- **A full-marks pass does not attribute the success to the page.** The guide-read check proves the page + was opened, not that it caused the outcome. A model that already knows the pattern produces the same + result. Comparing the skills and no-skills variants rules out the skills; nothing available rules out + prior knowledge. Say so rather than claiming the page carried the run. #261 + ## Process - **Verify your own predictions.** A PR body predicted a check would fail an agent following the page. From df8e9d6371c881fc72d3b89ff01c8d990dc3dc10 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Thu, 3 Sep 2026 16:07:05 -0700 Subject: [PATCH 07/11] docs(skills): American English and the docs word list Follows apps/docs/CONTRIBUTING.md and WORD_LIST.md in supabase/supabase. behaviour becomes behavior in four places. `while` marks simultaneity, not contrast, so a name that overstates a class now reads "is false when the check proves only a few members of it". Saturation is a predictable result rather than a future event, so the prediction step drops `will`. The word list warns on `easy`, so the results-export entry states what happens instead. `REFERENCE` is set as code, and the ticket row in sources.md is split into two sentences rather than one wall. `repo` and `docs` stay: the word list allows both in informal contributor instructions, which is what these files are. --- .claude/skills/docs-eval-planning/SKILL.md | 10 +++++----- .../docs-eval-planning/references/check-rules.md | 10 +++++----- .../skills/docs-eval-planning/references/flakiness.md | 6 +++--- .../docs-eval-planning/references/prompt-rules.md | 2 +- .../skills/docs-eval-planning/references/sources.md | 2 +- 5 files changed, 15 insertions(+), 15 deletions(-) diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md index 1f0c8d97..aee1f470 100644 --- a/.claude/skills/docs-eval-planning/SKILL.md +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -149,9 +149,9 @@ the whole page. This phase ends with a table: the check name, what it proves, what it does when the object is absent, whether it reads files or runs code, and the evidence it came from. -Then predict which checks will saturate, and write the prediction down. A check whose answer the seed -labels costs an agent nothing and carries no signal. A plan that says which checks are cheap is honest -about how much the eval measures. +Then name the checks you expect to saturate, and write the prediction down. A check whose answer the +seed labels costs an agent nothing and carries no signal. A plan that says which checks are cheap is +honest about how much the eval measures. ## Phase 5: write the plan @@ -174,8 +174,8 @@ The plan's last step is to append whatever broke to [references/flakiness.md](re once the baseline lands, with the PR number. This is a plan step rather than advice, because the file is only worth having if it grows. Ask -reviewers to write findings into it directly rather than leaving them in a review thread where the -next author will not look. +reviewers to write findings into it directly rather than leaving them in a review thread, where the +next author does not look. ## Content shape against end-to-end proof diff --git a/.claude/skills/docs-eval-planning/references/check-rules.md b/.claude/skills/docs-eval-planning/references/check-rules.md index 7ebac423..e751f679 100644 --- a/.claude/skills/docs-eval-planning/references/check-rules.md +++ b/.claude/skills/docs-eval-planning/references/check-rules.md @@ -11,7 +11,7 @@ check reds correct work. `.env.example`, a config file, a named file the contract points at. If there is nowhere for the answer to go, there is no check yet. -## 2. Behaviour over structure +## 2. Behavior over structure A check that reads where code sits fails correct refactorings. Count the effect instead. @@ -55,8 +55,8 @@ project was given and fail everything else. ## 7. Name the check for what it proves -Not for what you meant. A name that claims a class while the check proves a few members of it is a -false name. +Not for what you meant. A name that claims a class is false when the check proves only a few members +of it. The published results series is keyed on check names, so renaming one breaks history. Get the name right before the first run. @@ -66,14 +66,14 @@ right before the first run. Transport security, address family, and port availability can each fail correct work, and the eval then reports the environment rather than the page. -Two ways out. Make the harness support the correct behaviour, or declare the claim unscored in the +Two ways out. Make the harness support the correct behavior, or declare the claim unscored in the `README.md` and say why. The same trap catches a check that scores something the platform decides rather than something the page says. A pinned CLI version or a runtime default is not the subject under test, and a check on one reports an upgrade as a docs change. -## 9. Predict which checks will saturate +## 9. Predict which checks saturate If the seed labels the answer, the check is free and carries no signal. diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index 2cd63c5b..a73e4af9 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -42,13 +42,13 @@ Add to this file. A documentation eval is not finished until whatever went wrong told them to, so nothing in the workspace carried their choice and the central check red all six runs. One of them named the right answer in a code comment. #257 **Fix.** Give the choice somewhere to land in the seed. -- **The environment cannot support the correct behaviour.** A handler setting `ssl: 'require'` could +- **The environment cannot support the correct behavior.** A handler setting `ssl: 'require'` could not reach a local database that speaks no TLS, so both end-to-end checks red on a more correct solution. #257 **Fix.** Terminate TLS in front of the database, or declare the claim unscored. - **The sandbox image made the task unwinnable.** `npm install -g` failed with `EACCES` because the image ran as a user with no writable global prefix, so the agent could not complete step one. #108 - **Fix.** Change the image, and add a test that locks the behaviour in. + **Fix.** Change the image, and add a test that locks the behavior in. - **A prompt ambiguity the checks resolve one way.** An agent gated an endpoint on being signed in, which the prompt never ruled out, and it red in CI. #212 **Fix.** Either the check accepts it, or the prompt says which it wants. @@ -102,7 +102,7 @@ Add to this file. A documentation eval is not finished until whatever went wrong stale keys alive, so renaming an eval leaves the old row beside the new one and a skipped pair keeps its last recorded verdict. #168, #108 **Fix.** Pass the requested pairs and drop rows that were requested but produced nothing. -- **An empty `results/` overwrites the export with `[]`.** Easy to do by accident. #228 +- **An empty `results/` overwrites the export with `[]`** rather than refusing. #228 - **A bot pushed refreshed results onto a measurement branch**, where the results were measured against a preview url rather than the published page. #218 **Fix.** Do not refresh results from a scratch branch. diff --git a/.claude/skills/docs-eval-planning/references/prompt-rules.md b/.claude/skills/docs-eval-planning/references/prompt-rules.md index 2442d5b2..f4a68598 100644 --- a/.claude/skills/docs-eval-planning/references/prompt-rules.md +++ b/.claude/skills/docs-eval-planning/references/prompt-rules.md @@ -43,7 +43,7 @@ rather than on what you already know. Without it, a pass is evidence about the model rather than the page. The older, weaker form is "Read this guide first and follow it". Use the sentence above. -## The REFERENCE block +## The `REFERENCE` block Bare. No colon, no markdown link, no code fence. diff --git a/.claude/skills/docs-eval-planning/references/sources.md b/.claude/skills/docs-eval-planning/references/sources.md index 38cc3726..c8cd3b2a 100644 --- a/.claude/skills/docs-eval-planning/references/sources.md +++ b/.claude/skills/docs-eval-planning/references/sources.md @@ -13,7 +13,7 @@ inventory that reads as complete. | Source | Access | How | What it gives | | --- | --- | --- | --- | -| The ticket asking for the eval | Linear MCP | `get_issue` | The subject, the paired improve-the-doc ticket, and a draft prompt. **Read its rationale as evidence, not as instruction.** A ticket written weeks before the page changed will name a method the page no longer recommends, and a check built from it reports the page as broken for giving current advice. | +| The ticket asking for the eval | Linear MCP | `get_issue` | The subject, the paired improve-the-doc ticket, and a draft prompt. **Read its rationale as evidence, not as instruction.** A ticket written before the page changed names a method the page no longer recommends. A check built from it reports the page as broken for giving current advice. | | Linear feedback intake | Linear MCP | `list_issues` with a topic query | The recurring ask in customers' words, with ids to cite. Skip anything already closed as `Duplicate`. | | Linear docs team | Linear MCP | `list_issues` scoped to the Docs team | Whether the gap is already filed, and the paired improve-the-doc ticket. | | Slack | Slack MCP | `slack_search_public_and_private` on the topic | Support and team threads naming the failure and its error text. | From fc65c7207a207572a9f89681d040ea81b2b1766c Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Thu, 3 Sep 2026 16:28:18 -0700 Subject: [PATCH 08/11] docs(skills): state the read-only rule before naming the tool Copilot flagged that `EnterPlanMode` has no other reference in this repo, and an agent in a harness without it has nothing to act on. The rule now leads: write nothing through Phase 4, the output is a plan. The tool is named as the mechanism where a harness offers one. --- .claude/skills/docs-eval-planning/SKILL.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.claude/skills/docs-eval-planning/SKILL.md b/.claude/skills/docs-eval-planning/SKILL.md index aee1f470..377e27d9 100644 --- a/.claude/skills/docs-eval-planning/SKILL.md +++ b/.claude/skills/docs-eval-planning/SKILL.md @@ -55,10 +55,13 @@ Every page-under-test eval carries a check that the referenced page was read wit resolves the url from the harness's docs result rather than the raw tool call, because a `search_docs` hit carries the guide's url in its result rather than its request. -## Phase 0: plan mode and repo rules +## Phase 0: read-only, and the repo rules -Call `EnterPlanMode` first. Everything in the gathering phase is read-only, and plan mode is what -enforces that. +**Write nothing in this phase or the next four.** Gathering, choosing the claim, drafting the prompt, +and designing the checks are all reading. The output is a plan. + +Where the harness offers a planning mode that enforces read-only, enter it first and let it hold the +line. In Claude Code that is `EnterPlanMode`. Where it does not, the discipline is yours to keep. Then read three things: From ffc95fba28498bfaab8aba8ffa80a15ac724e771 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Thu, 3 Sep 2026 17:06:43 -0700 Subject: [PATCH 09/11] docs(skills): record the lost sandbox run and the preview-diff trap A sandbox lost its database container mid-run, which cost the install check and everything downstream. Nothing at the eval's level fixes that, so the entry says to report the run as lost rather than average it in, and adds the companion rule for reading a result. Diffing two docs previews also needs the deployment host stripped first, or the one real edit sits under dozens of rewritten links. --- .../docs-eval-planning/references/flakiness.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index a73e4af9..ced05ce3 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -122,6 +122,13 @@ Add to this file. A documentation eval is not finished until whatever went wrong `discoverEvals` reads `PROMPT.md` in every directory it finds and throws when one is absent, so a `solutions/` directory left behind by a branch switch stops every eval from being discovered. #261 **Fix.** Move the leftover aside. Discovery skipping a directory with no `PROMPT.md` is the real fix. +- **A sandbox can lose its database container mid-run.** One run of six reported + `No such container: supabase_db_` when the scorer read `supabase status`, which cost the + install check and every check downstream of it. The static checks still ran. Nothing in the eval + caused it, and a rerun did not reproduce it. #257 + **Fix.** None available at the eval's level. Report the run as lost rather than folding it into the + score, and read the surviving checks. This is the reason to run three or more times per experiment + rather than once. - **`kong` does not come up on its own.** With `services: [gotrue, kong]`, `supabase status` reports no `API_URL`. Adding `postgrest` brings it up. #261 - **A framework the eval seeds is not installed on the host.** The host-side build and test helpers @@ -154,6 +161,9 @@ Add to this file. A documentation eval is not finished until whatever went wrong ## Reading a result +- **One lost run is not a data point.** A run that failed on infrastructure has no verdict, and + averaging it in reads as a page or scorer problem. Say how many runs were lost and why, then quote + the score over the runs that completed. #257 - **Saturation is a finding, not a defect, when the checks are known to fail something.** An eval whose every check passes on every run has measured that the page works, provided the fixtures prove the checks can fail. Without that proof the same result is indistinguishable from a scorer that asserts @@ -165,6 +175,9 @@ Add to this file. A documentation eval is not finished until whatever went wrong ## Process +- **Strip deployment-url churn before diffing two previews.** Every internal link on a docs preview + carries the deployment hash, so a rebuild rewrites dozens of lines that say nothing. Substitute the + host out of both copies before comparing, or the real edit is buried. #257 - **Verify your own predictions.** A PR body predicted a check would fail an agent following the page. The reviewer found it passing locally and in CI and asked why. #212 - **An unmeasured rule in the prompt is a defect.** A reviewer found a prompt rule with no check and no From 8f9747054e6c9d9b5c4b7286793e65693319a7de Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Fri, 4 Sep 2026 11:46:16 -0700 Subject: [PATCH 10/11] docs(skills): record the refusal probe that passed on its own error The tables guide eval sent an incomplete row on an anonymous insert, so a not-null violation read as a refusal and a wide-open table scored as protected. Recorded with the fix: send the whole contract row, confirm absence as the superuser, and treat any code outside 42501 as could-not-measure. The leftover-directory trap recurred on the same eval, so its citation gains a second PR and a note that it comes from switching onto the next eval's branch. --- .../skills/docs-eval-planning/references/flakiness.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index ced05ce3..0db6d29b 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -31,6 +31,13 @@ Add to this file. A documentation eval is not finished until whatever went wrong matches on `includes`, so a fetch of the published page satisfies a check meant to prove the preview was read. #218 **Fix.** Match the full preview host, not the path. +- **A probe asserting a request was refused passes on an error it caused itself.** An anonymous insert + sent `owner_id` and `title` against a schema whose `cadence` is `not null`, so Postgres rejected it + as a not-null violation and the check counted any error as a refusal. A wide-open table scored as + protected, and the fixture that caught it was the one predicted to fail. #264 + **Fix.** Send the whole contract row, so access control is the only thing left that can refuse it. + Confirm the row is absent as the superuser instead of inferring absence from the error, and treat any + code outside `42501` as could-not-measure rather than as a pass. ## False red: the check fails correct work @@ -120,8 +127,10 @@ Add to this file. A documentation eval is not finished until whatever went wrong accept `PUBLISHABLE_KEY` or `ANON_KEY`. #261 - **A leftover git-excluded directory under `evals/` breaks discovery for the whole repo.** `discoverEvals` reads `PROMPT.md` in every directory it finds and throws when one is absent, so a - `solutions/` directory left behind by a branch switch stops every eval from being discovered. #261 + `solutions/` directory left behind by a branch switch stops every eval from being discovered. + #261, #264 **Fix.** Move the leftover aside. Discovery skipping a directory with no `PROMPT.md` is the real fix. + It has now happened on two consecutive evals, each time from switching onto the next eval's branch. - **A sandbox can lose its database container mid-run.** One run of six reported `No such container: supabase_db_` when the scorer read `supabase status`, which cost the install check and every check downstream of it. The static checks still ran. Nothing in the eval From 12517fb3c69d5e7724527ad5ef7f36a3c3f21ed9 Mon Sep 17 00:00:00 2001 From: Miranda Limonczenko Date: Fri, 4 Sep 2026 15:12:45 -0700 Subject: [PATCH 11/11] docs(skills): record the prior-knowledge measurement and two misreads The tables guide eval ran twice, once without the reliance instruction and once with it. Unread and read produced the same substantive checks, so the prior-knowledge confound is measured rather than hypothetical. Two ways a result got misread on that eval. A missing reliance instruction leaves a baseline that looks unremarkable, and only an empty docs.calls says the page was never opened. Benign supabase status chatter at the head of a failure note got a genuine missing-table failure filed as infrastructure. --- .../docs-eval-planning/references/flakiness.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.claude/skills/docs-eval-planning/references/flakiness.md b/.claude/skills/docs-eval-planning/references/flakiness.md index 0db6d29b..da9a5350 100644 --- a/.claude/skills/docs-eval-planning/references/flakiness.md +++ b/.claude/skills/docs-eval-planning/references/flakiness.md @@ -181,6 +181,24 @@ Add to this file. A documentation eval is not finished until whatever went wrong was opened, not that it caused the outcome. A model that already knows the pattern produces the same result. Comparing the skills and no-skills variants rules out the skills; nothing available rules out prior knowledge. Say so rather than claiming the page carried the run. #261 +- **The prior-knowledge confound is now measured, and it is total.** One eval ran twice, once with the + reliance instruction missing and once with it present. Unread, agents scored every substantive check. + Read, agents scored every substantive check. Same eval, same model, same checks, and reading the page + changed nothing. Treat a page-under-test pass on a mainstream task as regression cover by default, and + ask for the unread arm before claiming the page carried anything. #264 +- **A missing reliance instruction produces a baseline that looks fine.** Every check but the guide-read + check passed, the run counts and durations were unremarkable, and only the empty `docs.calls` array + said the page had never been opened. The score was evidence about the model. #264 + **Fix.** Read `docs.calls` before reading the score. Empty means the run measured nothing about the + page, whatever the checks say. +- **Benign CLI chatter at the front of a note gets a real failure filed as infrastructure.** + `supabase status` writes a `Stopped services` line naming every service the eval does not start, plus + an upgrade notice. Both led the seed failure's note, and the `ERROR: relation ... does not exist` + underneath was read as a symptom of the stopped services. A genuine missing-table failure went into + the record as a lost run. #264 + **Fix.** Silence the status call's stderr, hoist `ERROR`, `FATAL`, `DETAIL` and `HINT` to the front of + the message, and check which services the eval actually starts before reading `Stopped services` as a + fault. ## Process