feat(skills): add a planning skill for documentation evals - #259
Open
czenko wants to merge 11 commits into
Open
Conversation
Four build-docs evals exist and each was designed from scratch. The most recent cost three false reds and one false green, all of them mistakes an earlier author had already made. The skill plans a documentation eval and stops at the plan. Six phases: enter plan mode and read CONTRIBUTING.md, gather the evidence, pick the one claim the page has to transmit, draft the prompt, design the checks, write the plan, and feed the catalog once the baseline lands. Five references carry the catalogs. sources.md lists every evidence source and marks which need an MCP connector or private access, because an agent that silently skips a gated source produces a thin inventory that reads as complete. check-rules.md is ordered by what a violation costs, and the first three rules are the ones that have sunk a run. flakiness.md is mined from the review feedback on twenty-one PRs, with a citation per entry. It lives under .claude/skills/ rather than skills/, which holds product skills that experiments install into eval sandboxes. Closes DOCS-1228
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Author
|
I plan to stress test this SKILL before merging by creating another Docs Eval. |
czenko
commented
Sep 2, 2026
czenko
left a comment
czenko
left a comment
Contributor
Author
There was a problem hiding this comment.
Feedback on my own PR 😁
Applies review feedback. Every reference now states the decision rather than the case that produced it, per Google's timeless documentation guidance. canonical-shape.md drops the genre background, the scorer-structure history, and the reasoning behind not committing solutions. It says to model on build-docs-003 or later, and to never commit solutions. check-rules.md and prompt-rules.md drop their worked examples and state what to do. The motivation guidance now names the constant behind every one of these evals: evidence that an agent can read the page and do the task correctly. sources.md opens with the note that Linear, Slack, Hex, and Supacademy are internal to Supabase, since this repository is open source, and that planning from the public sources alone is a supported outcome.
There was a problem hiding this comment.
🟡 Changes recommended
The new skill is committed under .claude/skills/, but the harness resolves/installable skills from skills/<name>, so it won’t be discoverable via the normal skills mechanism as-is.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new “docs-eval-planning” agent skill intended to standardize how documentation-page evals are planned in this repo, capturing prior failure modes and conventions so new evals don’t repeat avoidable mistakes.
Changes:
- Introduces a new planning skill (
SKILL.md) describing a phased workflow for designing docs eval prompts and checks. - Adds reference documents covering evidence sources, prompt authoring rules, check design rules, canonical eval shape, and a flakiness catalog.
File summaries
| File | Description |
|---|---|
| .claude/skills/docs-eval-planning/SKILL.md | Main skill entrypoint describing phases and how to produce a planning artifact. |
| .claude/skills/docs-eval-planning/references/sources.md | Catalog of evidence sources to derive checks from (internal + public). |
| .claude/skills/docs-eval-planning/references/prompt-rules.md | Rules for writing prompts that test the page rather than the model. |
| .claude/skills/docs-eval-planning/references/check-rules.md | Ordered checklist of check-design constraints and common traps. |
| .claude/skills/docs-eval-planning/references/flakiness.md | Curated list of previously observed false reds/greens and nondeterminism sources. |
| .claude/skills/docs-eval-planning/references/canonical-shape.md | Conventions for docs-eval folder shape, README content, and orchestration patterns. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
canonical-shape.md read as a thin spec once its background came out. Its six items move into SKILL.md: the naming and id rule, the file layout, the EVAL.ts shape, the README section set, and the guide-read check go into a shape section before the phases; the seed rules and the contract choice join Phase 3, where the seed is drafted; judges join Phase 4; example solutions join Phase 5, where verification is planned. SKILL.md is 1,587 words, inside the range a skill body should hold.
Running the skill on auth/server-side/creating-a-client found five things it does not say. Phase 2 now ends by confirming the harness can measure the claim, before any check is designed. A claim the harness cannot observe is not the claim, and finding that in Phase 4 is a rewrite rather than a design change. The three questions are where the end state lives, whether the scorer can run what has to run, and what the time budget is. Phase 2 also gives a basis for picking. A dense page yields ten evidenced candidates, so pick on consequence and prefer the failure the page itself warns about. sources.md adds the ticket as a source and says to read its rationale as evidence rather than instruction, because a ticket written before the page changed will name a method the page no longer recommends. The diff step now says the page wins on what is acceptable and the check names the class. flakiness.md gains the two entries behind the feasibility step: a framework the eval seeds is not on the host, and a first-of-its-kind toolchain in a scorer is a risk to state rather than a detail. Phase 0 reads flakiness.md, which the intro already asked for.
Four framework entries and two on reading a result. The saturation pair is the one that matters: a full-marks baseline is a finding when the fixtures prove the checks can fail, and it still does not attribute the success to the page.
Follows apps/docs/CONTRIBUTING.md and WORD_LIST.md in supabase/supabase. behaviour becomes behavior in four places. `while` marks simultaneity, not contrast, so a name that overstates a class now reads "is false when the check proves only a few members of it". Saturation is a predictable result rather than a future event, so the prediction step drops `will`. The word list warns on `easy`, so the results-export entry states what happens instead. `REFERENCE` is set as code, and the ticket row in sources.md is split into two sentences rather than one wall. `repo` and `docs` stay: the word list allows both in informal contributor instructions, which is what these files are.
Copilot flagged that `EnterPlanMode` has no other reference in this repo, and an agent in a harness without it has nothing to act on. The rule now leads: write nothing through Phase 4, the output is a plan. The tool is named as the mechanism where a harness offers one.
czenko
marked this pull request as ready for review
September 3, 2026 23:28
A sandbox lost its database container mid-run, which cost the install check and everything downstream. Nothing at the eval's level fixes that, so the entry says to report the run as lost rather than average it in, and adds the companion rule for reading a result. Diffing two docs previews also needs the deployment host stripped first, or the one real edit sits under dozens of rewritten links.
The tables guide eval sent an incomplete row on an anonymous insert, so a not-null violation read as a refusal and a wide-open table scored as protected. Recorded with the fix: send the whole contract row, confirm absence as the superuser, and treat any code outside 42501 as could-not-measure. The leftover-directory trap recurred on the same eval, so its citation gains a second PR and a note that it comes from switching onto the next eval's branch.
The tables guide eval ran twice, once without the reliance instruction and once with it. Unread and read produced the same substantive checks, so the prior-knowledge confound is measured rather than hypothetical. Two ways a result got misread on that eval. A missing reliance instruction leaves a baseline that looks unremarkable, and only an empty docs.calls says the page was never opened. Benign supabase status chatter at the head of a failure note got a genuine missing-table failure filed as infrastructure.
nrichers
reviewed
Sep 4, 2026
nrichers
left a comment
nrichers
left a comment
There was a problem hiding this comment.
Not a PR for me to approve but I did have a suggestion about softening the divide between open-source and Supabase. Very helpful skill - I think it could work for a broader audience.
When I tested the skill, it worked as expected: 🙌
I picked a real negative-feedback guide from obsidian-supabase → Priorities/03-Evidence based docs prioritization → the live prioritization manifest (docs-agent-skills): rank #17, BL-032, Storage/RLS, 11 linked FDBKIN issues. Followed the skill's phases (read CONTRIBUTING.md, the guide's .md variant, flakiness.md; gathered evidence via Linear MCP, GitHub agent-skills issues, and the live docs page; picked one claim; confirmed measurability):
The guide: storage/security/access-control.mdx ("Storage Access Control"). The one claim: RLS policies on storage.objects are completely irrelevant for reads once a bucket is marked public — a bucket's public URL bypasses RLS entirely — and the guide teaches elaborate RLS policies without ever warning that public-bucket status overrides all of them. Confirmed via Linear (FDBKIN-31412, "Improve security of default Storage policies in tutorials and docs," priority Urgent) plus its already-filed paired ticket STORAGE-478 ("Update Storage docs for RLS, listing, and public buckets" — created after "a recent incident where AI-generated setup appears to have added insecure policies"), and confirmed the tutorial's own copy-paste snippet (storage/quickstart.mdx) literally ships a policy named "Public Access" on bucket_id = 'public'. This gives the eval a behavioral check with no aliasing risk at all: query storage.buckets.public, and — the real proof — attempt an unauthenticated fetch of the agent-created object's public URL, which must fail if the bucket is truly private.
The skill worked well in practice: its phase ordering (evidence before claim, claim before checks) kept me from jumping straight to writing checks, and check-rules.md/flakiness.md directly shaped which check design I picked (behavioral over structural, given rule 2 and the aliasing bug I just reproduced in #257). One friction point worth relaying to the author: sources.md assumes Linear/Hex/Slack access that a fresh session doesn't have by default — I only got real evidence because Linear MCP happened to be reachable; the skill has no fallback guidance for "you got partial access, here's how much plan quality to expect."
Since the skill's own scope is "produces a plan, not files," I stopped there — want me to write up the full plan (prompt draft, seed spec, check table) as a proposal, or go further and actually build evals/build-docs-006-storage-access-control/?
|
|
||
| The page under test is not a source. Everything here is. | ||
|
|
||
| **Several sources are internal to Supabase.** This repository is open source, and the Linear, Slack, |
There was a problem hiding this comment.
Since this is an open-source repo, you could soften this language a bit and provide fallback guidance for open-source users.
Here's how I tackled a similar access issue for universe repo lookup in /pm-the-docs:
- "Cross-repo confirmation is required for everyone... that is a successful outcome, not a failure"
- Capability gate + flowchart
- OSS path checklist
- "Never treat missing universe access as a blocker or an incomplete Frame/Shape"
(As in, your new skill could likely determine if it has access and, even if it doesn't, still support planning for new evals for open-source users.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-1228
Problem
Four
build-docs-*evals exist and each was designed from scratch. The most recent cost three false reds and one false green, all of them mistakes an earlier author had already made or could have warned about. The only authoring guidance that exists is a five-step how-to buried inbuild-docs-001/README.md.Solution
.claude/skills/docs-eval-planning/:SKILL.mdplus four references.CONTRIBUTING.md, gather the evidence, pick the one claim the page has to transmit, draft the prompt, design the checks, write the plan, and feed the catalog once the baseline lands.references/sources.mdlists every evidence source and marks which need an MCP connector or private-repo access. The rule is to stop and ask for a missing connector rather than working around it, then record which sources were reachable, because an inventory built from public sources alone reads as complete when it is not.references/check-rules.mdis ordered by what a violation costs. The first three are observability, behavior over structure, and positive controls a literal cannot satisfy.references/flakiness.mdis mined from review feedback on twenty-one PRs, with a citation per entry. Nine of those PRs were outside the set I started from, including feat: add docs homepage quickstart eval #108, which carries aCHANGES_REQUESTEDreview, and feat: move eval execution from GitHub Actions to Vercel Sandboxes #192, where the CI reproducibility work happened.references/prompt-rules.mdcarries the prompt discipline. The file, frontmatter, and README conventions live inSKILL.mditself.Manual testing
Confirm the skill is discovered and its description reads correctly.
Confirm every relative link resolves and every path the skill names exists.
Typecheck.
Clean. The change is markdown only.
Confirm Biome does not process the skill, since it ships no markdown formatter.
No files were processed in the specified paths.