From 424877813a838c59b86b939e5bd85183da14db46 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Wed, 26 Aug 2026 12:28:02 +0200 Subject: [PATCH 1/3] docs: add pull request template Add a local PR template on top of the inherited supabase/.github default, adding How to Review and Verification sections tailored to the eval contribution flow (PROMPT.md / EVAL.ts / seed data, and refreshing results in CI). --- .github/PULL_REQUEST_TEMPLATE.md | 59 ++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 .github/PULL_REQUEST_TEMPLATE.md diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..67914a28 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,59 @@ +## What kind of change does this PR introduce? + +New eval, new experiment, scorer fix, framework change, docs update, ... + +## What is the current behavior? + +Please link any relevant issues here. + +## What is the new behavior? + +Feel free to include screenshots if it includes visual changes. + +## How to Review + + + +1. **Scenario** + - `evals//PROMPT.md` + - The task the agent sees and the cited `motivation:`. + +2. **Scorer** + - `evals//EVAL.ts` + - What end state it checks, and where it uses `judge()` versus deterministic checks. + +3. **Seed data** + - `evals//remote/` or `local/` + - What project or filesystem state the scenario seeds. + +**Review questions** + + + +- [ ] Does `motivation:` cite real evidence from the Supabase user journey? +- [ ] Does the scorer check end state and prefer deterministic checks over prescribing process? +- [ ] For a benchmark scenario, do results show agents legitimately failing rather than framework limitations? +- [ ] Is the scenario representative rather than over-indexed on a niche use case? + +## Verification + + + +Include refreshed results (from CI or the Vercel preview) for PRs with new or changed evals, so a reviewer can see results directly. + +## Additional context + +Add any other context or screenshots. From 569eb382088921c2ed6c31f6bc197ef180a10a1b Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Wed, 26 Aug 2026 12:30:38 +0200 Subject: [PATCH 2/3] docs: keep PR template body generic, scope eval specifics Make the visible How to Review body and review questions generic so experiment, framework, and docs PRs aren't forced into an eval-only shape. Keep the PROMPT.md/EVAL.ts/seed-data path and eval checks as optional guidance, and make the local sanity check plus refreshed CI results a visible verification requirement for eval changes per CONTRIBUTING.md. --- .github/PULL_REQUEST_TEMPLATE.md | 38 +++++++++++++++++--------------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 67914a28..7b7a8585 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -17,42 +17,44 @@ A reviewer should understand what this PR does within 15-30 seconds, then know w Non-trivial or multi-file PRs must fill in the reading path below. Trivial changes may delete this section when the review path is obvious. -- For a new or changed eval, start from `PROMPT.md` (the task and `motivation:`), then `EVAL.ts` (the scorer), then any `remote/` or `local/` seed data. -- For a new or changed experiment, start from the `experiments/` file and call out the `suite:`, skills, and MCP servers it configures. - Use bold numbered areas, each with one or two indented bullets: the exact files, then what to inspect. - Use exact repository-relative paths, and avoid line numbers while the branch is still changing. -- Don't repeat the PR purpose or the verification steps here. Keep the section scannable. ---> +- Keep it scannable. Don't repeat the PR purpose or the verification steps here. -1. **Scenario** - - `evals//PROMPT.md` - - The task the agent sees and the cited `motivation:`. +For a new or changed eval, a good path is `PROMPT.md` (task + `motivation:`) -> `EVAL.ts` (scorer) -> `remote/` or `local/` seed data. +For an experiment, point at the `experiments/` file and call out the `suite:`, skills, and MCP servers it configures. +--> -2. **Scorer** - - `evals//EVAL.ts` - - What end state it checks, and where it uses `judge()` versus deterministic checks. +1. **Review area** + - `path/to/file` + - What to follow or verify. -3. **Seed data** - - `evals//remote/` or `local/` - - What project or filesystem state the scenario seeds. +2. **Next review area** + - `path/to/next-file` + - What to follow or verify. **Review questions** +- [ ] Is the change scoped, and does the behavior match the description? +- [ ] Are edge cases and failure paths handled safely? + + ## Verification - + + +For new or changed evals (see [CONTRIBUTING.md](../CONTRIBUTING.md)): -Include refreshed results (from CI or the Vercel preview) for PRs with new or changed evals, so a reviewer can see results directly. +- [ ] Ran the eval locally to sanity check it completes without framework errors. +- [ ] Refreshed results in CI (`run-evals-changed` / `run-evals` labels, or the Refresh eval results workflow) and included them so a reviewer can see results directly. ## Additional context From db63410b5f512422d6fec5cda7b2e58ebd9f2e00 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Wed, 26 Aug 2026 12:31:57 +0200 Subject: [PATCH 3/3] docs: use absolute repo path for CONTRIBUTING link A relative ../CONTRIBUTING.md resolves from the PR page once GitHub copies the template into a PR body, so link the repo path instead. --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 7b7a8585..ab6813e3 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -51,7 +51,7 @@ For an experiment, point at the `experiments/` file and call out the `suite:`, s -For new or changed evals (see [CONTRIBUTING.md](../CONTRIBUTING.md)): +For new or changed evals (see [CONTRIBUTING.md](/supabase/evals/blob/main/CONTRIBUTING.md)): - [ ] Ran the eval locally to sanity check it completes without framework errors. - [ ] Refreshed results in CI (`run-evals-changed` / `run-evals` labels, or the Refresh eval results workflow) and included them so a reviewer can see results directly.