DO NOT MERGE: point build-docs-006 at the tables access-control preview - #265
Draft
czenko wants to merge 4 commits into
Draft
DO NOT MERGE: point build-docs-006 at the tables access-control preview#265czenko wants to merge 4 commits into
czenko wants to merge 4 commits into
Conversation
Scratch, not for merge. Three changes, all reverted rather than landed. PROMPT.md's REFERENCE points at the preview for supabase/supabase#50024. EVAL.ts matches the preview host rather than the path alone, so fetching the published page cannot satisfy the guide-read check. docs-results.ts accepts docs-git-*-supabase.vercel.app, without which the preview fetch never reaches docsCalls and the check fails a run that did read the page.
czenko
added
the
run-evals-changed
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. Scratch branch for measuring an unmerged docs revision. Every change here gets reverted, not landed.
Why
build-docs-006-database-tablesasks for a habit tracker database and points at Tables and Data. The published page never says to protect a table, so the central checks measure whether an agent gets to row level security unprompted.supabase/supabase#50024 adds a "Securing your tables" section: enabling row level security, a first policy, a two-access-class worked example, and a verification step. This branch points the eval at that PR's preview to see whether the revision closes the gap.
Changes
evals/build-docs-006-database-tables/PROMPT.md—REFERENCErepointed to the preview. One line.evals/build-docs-006-database-tables/EVAL.ts— the guide-read check matches the preview hostname, not the path alone. Without this a fetch of the published page satisfies the check, and an agent that ignored the preview reads as having used it.packages/core/src/docs-results.ts—isSupabaseApexUrlwidened to acceptdocs-git-*-supabase.vercel.app. Without it a preview fetch is dropped fromdocsCalls, so the guide-read check fails on a run that did read the page. Lookalike hosts are still rejected and the core suite passes unchanged, 114/114.What to look for
Read
the agent read the Tables and Data guide the prompt referencedfirst. If it fails, the agent never reached the preview and the rest of the run says nothing about the revision.The signal is the pair of catalog checks,
row level security is enabled on every table in the public schemaandevery table with row level security enabled carries at least one policy, plus the two behavioral probes that a wide-open schema fails:a signed-out visitor cannot read anyone's routinesanda signed-out visitor cannot create a routine.The positive controls say whether a green is real.
a signed-out visitor can browse the starter routine libraryandthe signed-in owner reads their own routines and nobody else'sboth fail for a schema that enables row level security and writes no policy.Running it
Label with
run-evals-changed. Onlybuild-docs-006-database-tableschanged, so that is the only eval that runs.