From 532971db8eece63afe3c66c1971bd4947bd2c5fe Mon Sep 17 00:00:00 2001 From: Steven Eubank Date: Tue, 1 Sep 2026 20:23:46 +0200 Subject: [PATCH 1/3] =?UTF-8?q?feat:=20o11y=20probe=20=E2=86=92=20eval=20c?= =?UTF-8?q?onversion=20(49=20evals)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Converts the chaos-o11y probe library into a full eval suite under the o11y- prefix. Probe numbers are retained in directory names for easy cross-referencing (e.g. o11y-0003-resolve-security-rls-initplan). - 36 MCP evals (PGlite): 31/36 pass with skills, 33/36 without - 10 CLI evals (Docker): 8/9 pass (storage-cors-wildcard is valid signal) - 1 cron-wrong-owner stub (blocked: cron.job owned by supabase_admin) - o11y-README.md: coverage table with advisor status and local results Co-Authored-By: Claude Sonnet 4.6 --- .../EVAL.ts | 81 +++++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 26 +++++ .../EVAL.ts | 72 ++++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 7 ++ .../EVAL.ts | 71 ++++++++++++ .../PROMPT.md | 23 ++++ .../remote/project.sql | 21 ++++ .../EVAL.ts | 73 ++++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 13 +++ .../EVAL.ts | 62 ++++++++++ .../PROMPT.md | 17 +++ .../remote/project.sql | 42 +++++++ .../EVAL.ts | 93 +++++++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 23 ++++ .../EVAL.ts | 54 +++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 28 +++++ .../EVAL.ts | 53 +++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 20 ++++ .../EVAL.ts | 72 ++++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 26 +++++ .../EVAL.ts | 67 +++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 30 +++++ .../EVAL.ts | 74 ++++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 30 +++++ .../EVAL.ts | 80 +++++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 15 +++ .../EVAL.ts | 68 +++++++++++ .../PROMPT.md | 24 ++++ .../remote/project.sql | 28 +++++ .../EVAL.ts | 65 +++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 15 +++ .../EVAL.ts | 64 ++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 29 +++++ .../EVAL.ts | 58 ++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 11 ++ .../EVAL.ts | 63 ++++++++++ .../PROMPT.md | 21 ++++ .../remote/project.sql | 20 ++++ .../EVAL.ts | 65 +++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 14 +++ .../EVAL.ts | 72 ++++++++++++ .../PROMPT.md | 21 ++++ .../remote/project.sql | 25 ++++ .../EVAL.ts | 54 +++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 18 +++ .../EVAL.ts | 59 ++++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 26 +++++ .../EVAL.ts | 64 ++++++++++ .../PROMPT.md | 21 ++++ .../local/supabase/config.toml | 19 +++ .../migrations/0000_public_bucket.sql | 8 ++ .../EVAL.ts | 54 +++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 35 ++++++ .../EVAL.ts | 47 ++++++++ .../PROMPT.md | 16 +++ .../remote/logs.jsonl | 5 + .../EVAL.ts | 60 ++++++++++ .../PROMPT.md | 23 ++++ .../local/supabase/config.toml | 16 +++ .../migrations/0000_corrupt_auth_user.sql | 23 ++++ .../o11y-investigate-auth-leaked-jwt/EVAL.ts | 53 +++++++++ .../PROMPT.md | 19 +++ .../remote/logs.jsonl | 4 + .../EVAL.ts | 50 ++++++++ .../PROMPT.md | 23 ++++ .../local/supabase/config.toml | 16 +++ .../migrations/0000_auth_audit_burst.sql | 22 ++++ .../o11y-investigate-cost-n-plus-one/EVAL.ts | 74 ++++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 51 ++++++++ .../EVAL.ts | 45 ++++++++ .../PROMPT.md | 16 +++ .../remote/logs.jsonl | 4 + .../EVAL.ts | 62 ++++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 41 +++++++ .../EVAL.ts | 52 +++++++++ .../PROMPT.md | 22 ++++ .../remote/project.sql | 41 +++++++ .../EVAL.ts | 48 ++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 63 ++++++++++ .../EVAL.ts | 48 ++++++++ .../PROMPT.md | 19 +++ .../remote/logs.jsonl | 6 + .../EVAL.ts | 52 +++++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 48 ++++++++ .../EVAL.ts | 48 ++++++++ .../PROMPT.md | 19 +++ .../remote/logs.jsonl | 6 + .../remote/project.sql | 21 ++++ .../EVAL.ts | 48 ++++++++ .../PROMPT.md | 21 ++++ .../remote/project.sql | 27 +++++ .../EVAL.ts | 50 ++++++++ .../PROMPT.md | 23 ++++ .../remote/project.sql | 28 +++++ .../EVAL.ts | 49 ++++++++ .../PROMPT.md | 16 +++ .../remote/logs.jsonl | 4 + .../EVAL.ts | 48 ++++++++ .../PROMPT.md | 18 +++ .../remote/project.sql | 32 +++++ evals/o11y-resolve-auth-user-trigger/EVAL.ts | 42 +++++++ .../o11y-resolve-auth-user-trigger/PROMPT.md | 24 ++++ .../local/supabase/config.toml | 16 +++ .../migrations/0000_auth_user_trigger.sql | 22 ++++ evals/o11y-resolve-cron-broken-job/EVAL.ts | 66 +++++++++++ evals/o11y-resolve-cron-broken-job/PROMPT.md | 22 ++++ .../local/supabase/config.toml | 16 +++ .../supabase/migrations/0000_broken_cron.sql | 10 ++ evals/o11y-resolve-cron-wrong-owner/EVAL.ts | 52 +++++++++ evals/o11y-resolve-cron-wrong-owner/PROMPT.md | 22 ++++ .../local/supabase/config.toml | 16 +++ .../migrations/0000_cron_wrong_owner.sql | 10 ++ .../EVAL.ts | 54 +++++++++ .../PROMPT.md | 23 ++++ .../local/supabase/config.toml | 19 +++ .../0000_orders_realtime_no_policy.sql | 31 +++++ .../EVAL.ts | 42 +++++++ .../PROMPT.md | 22 ++++ .../local/supabase/config.toml | 19 +++ .../0000_realtime_events_no_pub.sql | 11 ++ .../EVAL.ts | 41 +++++++ .../PROMPT.md | 22 ++++ .../local/supabase/config.toml | 19 +++ .../0000_orders_no_total_cents_index.sql | 26 +++++ .../EVAL.ts | 42 +++++++ .../PROMPT.md | 22 ++++ .../local/supabase/config.toml | 19 +++ .../0000_orders_replica_identity_nothing.sql | 20 ++++ .../EVAL.ts | 40 +++++++ .../PROMPT.md | 19 +++ .../remote/project.sql | 24 ++++ .../EVAL.ts | 43 +++++++ .../PROMPT.md | 22 ++++ .../local/supabase/config.toml | 27 +++++ .../migrations/0000_cors_wildcard.sql | 7 ++ .../EVAL.ts | 43 +++++++ .../PROMPT.md | 22 ++++ .../local/supabase/config.toml | 19 +++ .../0000_drop_storage_select_policy.sql | 6 + o11y-README.md | 109 ++++++++++++++++++ 161 files changed, 5239 insertions(+) create mode 100644 evals/o11y-0001-resolve-performance-missing-index/EVAL.ts create mode 100644 evals/o11y-0001-resolve-performance-missing-index/PROMPT.md create mode 100644 evals/o11y-0001-resolve-performance-missing-index/remote/project.sql create mode 100644 evals/o11y-0002-resolve-security-auth-users-exposed/EVAL.ts create mode 100644 evals/o11y-0002-resolve-security-auth-users-exposed/PROMPT.md create mode 100644 evals/o11y-0002-resolve-security-auth-users-exposed/remote/project.sql create mode 100644 evals/o11y-0003-resolve-security-rls-initplan/EVAL.ts create mode 100644 evals/o11y-0003-resolve-security-rls-initplan/PROMPT.md create mode 100644 evals/o11y-0003-resolve-security-rls-initplan/remote/project.sql create mode 100644 evals/o11y-0004-resolve-stability-no-primary-key/EVAL.ts create mode 100644 evals/o11y-0004-resolve-stability-no-primary-key/PROMPT.md create mode 100644 evals/o11y-0004-resolve-stability-no-primary-key/remote/project.sql create mode 100644 evals/o11y-0005-resolve-performance-unused-index/EVAL.ts create mode 100644 evals/o11y-0005-resolve-performance-unused-index/PROMPT.md create mode 100644 evals/o11y-0005-resolve-performance-unused-index/remote/project.sql create mode 100644 evals/o11y-0006-resolve-security-multiple-permissive-policies/EVAL.ts create mode 100644 evals/o11y-0006-resolve-security-multiple-permissive-policies/PROMPT.md create mode 100644 evals/o11y-0006-resolve-security-multiple-permissive-policies/remote/project.sql create mode 100644 evals/o11y-0007-resolve-security-rls-policy-exists-disabled/EVAL.ts create mode 100644 evals/o11y-0007-resolve-security-rls-policy-exists-disabled/PROMPT.md create mode 100644 evals/o11y-0007-resolve-security-rls-policy-exists-disabled/remote/project.sql create mode 100644 evals/o11y-0008-resolve-security-rls-enabled-no-policy/EVAL.ts create mode 100644 evals/o11y-0008-resolve-security-rls-enabled-no-policy/PROMPT.md create mode 100644 evals/o11y-0008-resolve-security-rls-enabled-no-policy/remote/project.sql create mode 100644 evals/o11y-0009-resolve-performance-duplicate-index/EVAL.ts create mode 100644 evals/o11y-0009-resolve-performance-duplicate-index/PROMPT.md create mode 100644 evals/o11y-0009-resolve-performance-duplicate-index/remote/project.sql create mode 100644 evals/o11y-0010-resolve-security-security-definer-view/EVAL.ts create mode 100644 evals/o11y-0010-resolve-security-security-definer-view/PROMPT.md create mode 100644 evals/o11y-0010-resolve-security-security-definer-view/remote/project.sql create mode 100644 evals/o11y-0011-resolve-security-function-search-path-mutable/EVAL.ts create mode 100644 evals/o11y-0011-resolve-security-function-search-path-mutable/PROMPT.md create mode 100644 evals/o11y-0011-resolve-security-function-search-path-mutable/remote/project.sql create mode 100644 evals/o11y-0013-resolve-security-rls-disabled/EVAL.ts create mode 100644 evals/o11y-0013-resolve-security-rls-disabled/PROMPT.md create mode 100644 evals/o11y-0013-resolve-security-rls-disabled/remote/project.sql create mode 100644 evals/o11y-0014-resolve-security-extension-in-public/EVAL.ts create mode 100644 evals/o11y-0014-resolve-security-extension-in-public/PROMPT.md create mode 100644 evals/o11y-0014-resolve-security-extension-in-public/remote/project.sql create mode 100644 evals/o11y-0015-resolve-security-rls-references-user-metadata/EVAL.ts create mode 100644 evals/o11y-0015-resolve-security-rls-references-user-metadata/PROMPT.md create mode 100644 evals/o11y-0015-resolve-security-rls-references-user-metadata/remote/project.sql create mode 100644 evals/o11y-0016-resolve-security-materialized-view-in-api/EVAL.ts create mode 100644 evals/o11y-0016-resolve-security-materialized-view-in-api/PROMPT.md create mode 100644 evals/o11y-0016-resolve-security-materialized-view-in-api/remote/project.sql create mode 100644 evals/o11y-0018-resolve-stability-unsupported-reg-types/EVAL.ts create mode 100644 evals/o11y-0018-resolve-stability-unsupported-reg-types/PROMPT.md create mode 100644 evals/o11y-0018-resolve-stability-unsupported-reg-types/remote/project.sql create mode 100644 evals/o11y-0019-resolve-security-pgmq-queue-exposed/EVAL.ts create mode 100644 evals/o11y-0019-resolve-security-pgmq-queue-exposed/PROMPT.md create mode 100644 evals/o11y-0019-resolve-security-pgmq-queue-exposed/remote/project.sql create mode 100644 evals/o11y-0020-resolve-postgres-table-bloat/EVAL.ts create mode 100644 evals/o11y-0020-resolve-postgres-table-bloat/PROMPT.md create mode 100644 evals/o11y-0020-resolve-postgres-table-bloat/remote/project.sql create mode 100644 evals/o11y-0021-resolve-security-fkey-to-auth-unique/EVAL.ts create mode 100644 evals/o11y-0021-resolve-security-fkey-to-auth-unique/PROMPT.md create mode 100644 evals/o11y-0021-resolve-security-fkey-to-auth-unique/remote/project.sql create mode 100644 evals/o11y-0023-resolve-security-sensitive-columns-exposed/EVAL.ts create mode 100644 evals/o11y-0023-resolve-security-sensitive-columns-exposed/PROMPT.md create mode 100644 evals/o11y-0023-resolve-security-sensitive-columns-exposed/remote/project.sql create mode 100644 evals/o11y-0024-resolve-security-rls-policy-always-true/EVAL.ts create mode 100644 evals/o11y-0024-resolve-security-rls-policy-always-true/PROMPT.md create mode 100644 evals/o11y-0024-resolve-security-rls-policy-always-true/remote/project.sql create mode 100644 evals/o11y-0025-resolve-storage-public-bucket/EVAL.ts create mode 100644 evals/o11y-0025-resolve-storage-public-bucket/PROMPT.md create mode 100644 evals/o11y-0025-resolve-storage-public-bucket/local/supabase/config.toml create mode 100644 evals/o11y-0025-resolve-storage-public-bucket/local/supabase/migrations/0000_public_bucket.sql create mode 100644 evals/o11y-0028-0029-resolve-security-anon-security-definer/EVAL.ts create mode 100644 evals/o11y-0028-0029-resolve-security-anon-security-definer/PROMPT.md create mode 100644 evals/o11y-0028-0029-resolve-security-anon-security-definer/remote/project.sql create mode 100644 evals/o11y-investigate-api-slow-response/EVAL.ts create mode 100644 evals/o11y-investigate-api-slow-response/PROMPT.md create mode 100644 evals/o11y-investigate-api-slow-response/remote/logs.jsonl create mode 100644 evals/o11y-investigate-auth-data-integrity/EVAL.ts create mode 100644 evals/o11y-investigate-auth-data-integrity/PROMPT.md create mode 100644 evals/o11y-investigate-auth-data-integrity/local/supabase/config.toml create mode 100644 evals/o11y-investigate-auth-data-integrity/local/supabase/migrations/0000_corrupt_auth_user.sql create mode 100644 evals/o11y-investigate-auth-leaked-jwt/EVAL.ts create mode 100644 evals/o11y-investigate-auth-leaked-jwt/PROMPT.md create mode 100644 evals/o11y-investigate-auth-leaked-jwt/remote/logs.jsonl create mode 100644 evals/o11y-investigate-auth-rate-limit-bypass/EVAL.ts create mode 100644 evals/o11y-investigate-auth-rate-limit-bypass/PROMPT.md create mode 100644 evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/config.toml create mode 100644 evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/migrations/0000_auth_audit_burst.sql create mode 100644 evals/o11y-investigate-cost-n-plus-one/EVAL.ts create mode 100644 evals/o11y-investigate-cost-n-plus-one/PROMPT.md create mode 100644 evals/o11y-investigate-cost-n-plus-one/remote/project.sql create mode 100644 evals/o11y-investigate-performance-edge-fn-slow/EVAL.ts create mode 100644 evals/o11y-investigate-performance-edge-fn-slow/PROMPT.md create mode 100644 evals/o11y-investigate-performance-edge-fn-slow/remote/logs.jsonl create mode 100644 evals/o11y-investigate-postgres-autovacuum-disabled/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-autovacuum-disabled/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-autovacuum-disabled/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-connection-saturation/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-connection-saturation/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-connection-saturation/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-index-bloat/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-index-bloat/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-index-bloat/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-long-running-query/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-long-running-query/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-long-running-query/remote/logs.jsonl create mode 100644 evals/o11y-investigate-postgres-slow-query/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-slow-query/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-slow-query/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-temp-file-spill/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-temp-file-spill/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-temp-file-spill/remote/logs.jsonl create mode 100644 evals/o11y-investigate-postgres-temp-file-spill/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-wal-slot-inactive/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-wal-slot-inactive/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-wal-slot-inactive/remote/project.sql create mode 100644 evals/o11y-investigate-security-auth-admin-bypassrls/EVAL.ts create mode 100644 evals/o11y-investigate-security-auth-admin-bypassrls/PROMPT.md create mode 100644 evals/o11y-investigate-security-auth-admin-bypassrls/remote/project.sql create mode 100644 evals/o11y-investigate-stability-edge-500/EVAL.ts create mode 100644 evals/o11y-investigate-stability-edge-500/PROMPT.md create mode 100644 evals/o11y-investigate-stability-edge-500/remote/logs.jsonl create mode 100644 evals/o11y-investigate-stability-silent-data-drift/EVAL.ts create mode 100644 evals/o11y-investigate-stability-silent-data-drift/PROMPT.md create mode 100644 evals/o11y-investigate-stability-silent-data-drift/remote/project.sql create mode 100644 evals/o11y-resolve-auth-user-trigger/EVAL.ts create mode 100644 evals/o11y-resolve-auth-user-trigger/PROMPT.md create mode 100644 evals/o11y-resolve-auth-user-trigger/local/supabase/config.toml create mode 100644 evals/o11y-resolve-auth-user-trigger/local/supabase/migrations/0000_auth_user_trigger.sql create mode 100644 evals/o11y-resolve-cron-broken-job/EVAL.ts create mode 100644 evals/o11y-resolve-cron-broken-job/PROMPT.md create mode 100644 evals/o11y-resolve-cron-broken-job/local/supabase/config.toml create mode 100644 evals/o11y-resolve-cron-broken-job/local/supabase/migrations/0000_broken_cron.sql create mode 100644 evals/o11y-resolve-cron-wrong-owner/EVAL.ts create mode 100644 evals/o11y-resolve-cron-wrong-owner/PROMPT.md create mode 100644 evals/o11y-resolve-cron-wrong-owner/local/supabase/config.toml create mode 100644 evals/o11y-resolve-cron-wrong-owner/local/supabase/migrations/0000_cron_wrong_owner.sql create mode 100644 evals/o11y-resolve-realtime-broadcast-no-policy/EVAL.ts create mode 100644 evals/o11y-resolve-realtime-broadcast-no-policy/PROMPT.md create mode 100644 evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/config.toml create mode 100644 evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/migrations/0000_orders_realtime_no_policy.sql create mode 100644 evals/o11y-resolve-realtime-no-publication/EVAL.ts create mode 100644 evals/o11y-resolve-realtime-no-publication/PROMPT.md create mode 100644 evals/o11y-resolve-realtime-no-publication/local/supabase/config.toml create mode 100644 evals/o11y-resolve-realtime-no-publication/local/supabase/migrations/0000_realtime_events_no_pub.sql create mode 100644 evals/o11y-resolve-realtime-unindexed-filter/EVAL.ts create mode 100644 evals/o11y-resolve-realtime-unindexed-filter/PROMPT.md create mode 100644 evals/o11y-resolve-realtime-unindexed-filter/local/supabase/config.toml create mode 100644 evals/o11y-resolve-realtime-unindexed-filter/local/supabase/migrations/0000_orders_no_total_cents_index.sql create mode 100644 evals/o11y-resolve-realtime-wrong-replica-identity/EVAL.ts create mode 100644 evals/o11y-resolve-realtime-wrong-replica-identity/PROMPT.md create mode 100644 evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/config.toml create mode 100644 evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/migrations/0000_orders_replica_identity_nothing.sql create mode 100644 evals/o11y-resolve-security-api-missing-grant/EVAL.ts create mode 100644 evals/o11y-resolve-security-api-missing-grant/PROMPT.md create mode 100644 evals/o11y-resolve-security-api-missing-grant/remote/project.sql create mode 100644 evals/o11y-resolve-storage-cors-wildcard/EVAL.ts create mode 100644 evals/o11y-resolve-storage-cors-wildcard/PROMPT.md create mode 100644 evals/o11y-resolve-storage-cors-wildcard/local/supabase/config.toml create mode 100644 evals/o11y-resolve-storage-cors-wildcard/local/supabase/migrations/0000_cors_wildcard.sql create mode 100644 evals/o11y-resolve-storage-object-rls-bypass/EVAL.ts create mode 100644 evals/o11y-resolve-storage-object-rls-bypass/PROMPT.md create mode 100644 evals/o11y-resolve-storage-object-rls-bypass/local/supabase/config.toml create mode 100644 evals/o11y-resolve-storage-object-rls-bypass/local/supabase/migrations/0000_drop_storage_select_policy.sql create mode 100644 o11y-README.md diff --git a/evals/o11y-0001-resolve-performance-missing-index/EVAL.ts b/evals/o11y-0001-resolve-performance-missing-index/EVAL.ts new file mode 100644 index 00000000..899f2ab1 --- /dev/null +++ b/evals/o11y-0001-resolve-performance-missing-index/EVAL.ts @@ -0,0 +1,81 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.orders.customer_id is unindexed. The fix creates an index on +// customer_id so the per-customer lookup uses it instead of a sequential scan. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkIndexExists(ctx), + await checkQueryPlanUsesIndex(ctx), + await checkInsertsStillWork(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated index fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +/** An index leading with customer_id must exist on public.orders. */ +async function checkIndexExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT indexname, indexdef + FROM pg_indexes + WHERE schemaname = 'public' AND tablename = 'orders'; + `); + const hasIndex = rows.some((r) => + /ON\s+(?:public\.)?orders\s+.*\(\s*customer_id/i.test(String(r.indexdef)) + ); + return { + name: 'index on orders(customer_id) exists', + passed: hasIndex, + notes: rows.map((r) => r.indexname).join(', '), + }; +} + +/** The per-customer lookup should plan with an index and no sequential scan. */ +async function checkQueryPlanUsesIndex( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + EXPLAIN SELECT id, total_cents, created_at + FROM public.orders + WHERE customer_id = 42; + `); + const plan = rows.map((r) => Object.values(r).join(' ')).join('\n'); + return { + name: 'query plan uses an index and avoids a sequential scan', + passed: + /(Index Scan|Index Only Scan|Bitmap Index Scan)/i.test(plan) && + !/Seq Scan on orders/i.test(plan), + notes: plan, + }; +} + +async function checkInsertsStillWork( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + INSERT INTO public.orders (customer_id, total_cents) + VALUES (1, 999) + RETURNING id; + `); + return { + name: 'inserts still work after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0001-resolve-performance-missing-index/PROMPT.md b/evals/o11y-0001-resolve-performance-missing-index/PROMPT.md new file mode 100644 index 00000000..ec911391 --- /dev/null +++ b/evals/o11y-0001-resolve-performance-missing-index/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - observability +motivation: apps/docs/content/troubleshooting/how-to-interpret-and-explore-the-postgres-advisor.mdx +--- + +Loading a customer's orders has gotten slow as the `orders` table has grown, and +CPU spikes when the app does it. Can you work out why that lookup is slow and +make the database change needed to speed it up? Make sure normal inserts into +`orders` still work afterward. + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0001-resolve-performance-missing-index/remote/project.sql b/evals/o11y-0001-resolve-performance-missing-index/remote/project.sql new file mode 100644 index 00000000..02582fd8 --- /dev/null +++ b/evals/o11y-0001-resolve-performance-missing-index/remote/project.sql @@ -0,0 +1,26 @@ +-- Broken starting state (probe: performance-missing-index). +-- orders.customer_id is unindexed, so the hot per-customer lookup does a +-- sequential scan over a large table. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' +FROM generate_series(1, 500) AS g; + +-- 50k orders, no index on customer_id. +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 500) + 1)::int, + (floor(random() * 10000) + 1)::int +FROM generate_series(1, 50000); + +ANALYZE public.orders; diff --git a/evals/o11y-0002-resolve-security-auth-users-exposed/EVAL.ts b/evals/o11y-0002-resolve-security-auth-users-exposed/EVAL.ts new file mode 100644 index 00000000..b8b1eed7 --- /dev/null +++ b/evals/o11y-0002-resolve-security-auth-users-exposed/EVAL.ts @@ -0,0 +1,72 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.user_list is a view over auth.users that exposes PII (email, +// created_at) to any API caller. The fix drops the view or rewrites it to not +// directly select from auth.users without restriction. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkViewDroppedOrSafe(ctx), + await checkNoDirectAuthUsersExposure(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated auth users exposure fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkViewDroppedOrSafe(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT table_name, view_definition + FROM information_schema.views + WHERE table_schema = 'public' AND table_name = 'user_list'; + `); + + if (rows.length === 0) { + return { + name: 'public.user_list view removed or does not expose auth.users', + passed: true, + notes: 'view dropped', + }; + } + + const def = String(rows[0]?.view_definition ?? ''); + const stillExposesAuthUsers = /auth\.users/i.test(def) && /email/i.test(def); + return { + name: 'public.user_list view removed or does not expose auth.users', + passed: !stillExposesAuthUsers, + notes: `view still exists; definition references auth.users email: ${stillExposesAuthUsers}`, + }; +} + +async function checkNoDirectAuthUsersExposure( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT grantee, privilege_type + FROM information_schema.role_table_grants + WHERE table_schema = 'public' + AND table_name = 'user_list' + AND grantee IN ('anon', 'authenticated'); + `); + return { + name: 'anon/authenticated grants on user_list revoked or view is gone', + passed: rows.length === 0, + notes: rows.length > 0 ? `still granted: ${rows.map(r => `${r.grantee}:${r.privilege_type}`).join(', ')}` : 'no grants', + }; +} diff --git a/evals/o11y-0002-resolve-security-auth-users-exposed/PROMPT.md b/evals/o11y-0002-resolve-security-auth-users-exposed/PROMPT.md new file mode 100644 index 00000000..a4d31bec --- /dev/null +++ b/evals/o11y-0002-resolve-security-auth-users-exposed/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database + - auth +topic: + - security +motivation: apps/docs/content/troubleshooting/database-roles.mdx +--- + +A security review flagged that we have a view in our public schema that looks +like it's exposing user PII from the auth system — emails and created-at +timestamps — to anyone who can read the API. Can you confirm whether that view +is leaking data, fix it so user details are no longer exposed via the public +schema, and confirm it's resolved? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0002-resolve-security-auth-users-exposed/remote/project.sql b/evals/o11y-0002-resolve-security-auth-users-exposed/remote/project.sql new file mode 100644 index 00000000..1ea8df09 --- /dev/null +++ b/evals/o11y-0002-resolve-security-auth-users-exposed/remote/project.sql @@ -0,0 +1,7 @@ +-- Broken starting state (probe: security-auth-users-exposed / Splinter lint 0002). +-- public.user_list is a view over auth.users that exposes user PII (email, +-- created_at) to any role with access to the public schema via the Data API. +CREATE OR REPLACE VIEW public.user_list AS + SELECT id, email, created_at FROM auth.users; + +GRANT SELECT ON public.user_list TO anon, authenticated; diff --git a/evals/o11y-0003-resolve-security-rls-initplan/EVAL.ts b/evals/o11y-0003-resolve-security-rls-initplan/EVAL.ts new file mode 100644 index 00000000..9d0b9331 --- /dev/null +++ b/evals/o11y-0003-resolve-security-rls-initplan/EVAL.ts @@ -0,0 +1,71 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: profiles RLS policy calls auth.uid() via a VOLATILE wrapper function, +// forcing per-row re-evaluation. The fix rewrites the policy to call auth.uid() +// directly so Postgres uses the efficient initplan path. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkPolicyCallsAuthUidDirectly(ctx), + await checkVolatileWrapperGone(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated RLS initplan fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkPolicyCallsAuthUidDirectly( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, qual + FROM pg_policies + WHERE schemaname = 'public' + AND tablename = 'profiles' + AND (cmd = 'SELECT' OR cmd = 'ALL'); + `); + + const hasDirectAuthUid = rows.some((r) => { + const qual = String(r.qual ?? ''); + return /auth\.uid\(\)/i.test(qual) && !/current_user_id|get_user_id/i.test(qual); + }); + + return { + name: 'SELECT policy calls auth.uid() directly (no volatile wrapper)', + passed: hasDirectAuthUid, + notes: rows.map(r => `${r.policyname}: ${r.qual}`).join('; '), + }; +} + +async function checkVolatileWrapperGone( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT proname + FROM pg_proc + WHERE proname IN ('current_user_id', 'get_user_id') + AND pronamespace = 'public'::regnamespace + AND provolatile = 'v'; + `); + return { + name: 'VOLATILE wrapper function removed or no longer used in policy', + passed: rows.length === 0, + notes: rows.length > 0 ? `still exists: ${rows.map(r => r.proname).join(', ')}` : 'none found', + }; +} diff --git a/evals/o11y-0003-resolve-security-rls-initplan/PROMPT.md b/evals/o11y-0003-resolve-security-rls-initplan/PROMPT.md new file mode 100644 index 00000000..f6ffb8c3 --- /dev/null +++ b/evals/o11y-0003-resolve-security-rls-initplan/PROMPT.md @@ -0,0 +1,23 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Queries against our `profiles` table have become catastrophically slow at scale +— we're seeing full table scan times even with RLS policies in place. The +SELECT policy uses a wrapper function (`current_user_id()`) that is VOLATILE, +causing Postgres to re-evaluate it for every row instead of once per query. + +Please fix the SELECT policy so it calls `auth.uid()` directly (no wrapper +function), then drop the `current_user_id()` wrapper function since it's no +longer needed. + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0003-resolve-security-rls-initplan/remote/project.sql b/evals/o11y-0003-resolve-security-rls-initplan/remote/project.sql new file mode 100644 index 00000000..9d8f0a24 --- /dev/null +++ b/evals/o11y-0003-resolve-security-rls-initplan/remote/project.sql @@ -0,0 +1,21 @@ +-- Broken starting state (probe: security-rls-initplan / Splinter lint 0003). +-- profiles RLS policy calls auth.uid() via a VOLATILE wrapper function, +-- forcing per-row re-evaluation (subplan) instead of the efficient single +-- initplan Postgres uses for the built-in directly. +CREATE TABLE public.profiles ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + handle text UNIQUE NOT NULL, + created_at timestamptz DEFAULT now() +); +ALTER TABLE public.profiles ENABLE ROW LEVEL SECURITY; + +CREATE OR REPLACE FUNCTION public.current_user_id() + RETURNS uuid LANGUAGE sql VOLATILE + AS $$ SELECT auth.uid() $$; + +CREATE POLICY "profiles_select_own" ON public.profiles + FOR SELECT TO authenticated + USING (id = public.current_user_id()); + +INSERT INTO public.profiles (handle) +SELECT 'user_' || g FROM generate_series(1, 100) g; diff --git a/evals/o11y-0004-resolve-stability-no-primary-key/EVAL.ts b/evals/o11y-0004-resolve-stability-no-primary-key/EVAL.ts new file mode 100644 index 00000000..41f236c1 --- /dev/null +++ b/evals/o11y-0004-resolve-stability-no-primary-key/EVAL.ts @@ -0,0 +1,73 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.events has no primary key. The fix adds one without dropping +// the existing rows. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkHasPrimaryKey(ctx), + await checkExistingRowsPreserved(ctx), + await checkInsertsStillWork(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated primary key fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +/** public.events must have a primary key constraint. */ +async function checkHasPrimaryKey(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT count(*)::int AS n + FROM pg_constraint + WHERE conrelid = 'public.events'::regclass + AND contype = 'p'; + `); + return { + name: 'public.events has a primary key', + passed: Number(rows[0]?.n ?? 0) === 1, + }; +} + +/** The 200 seeded rows must survive the change (no drop-and-recreate data loss). */ +async function checkExistingRowsPreserved( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query( + `SELECT count(*)::int AS n FROM public.events;` + ); + return { + name: 'existing rows preserved', + passed: Number(rows[0]?.n ?? 0) >= 200, + notes: `row count=${rows[0]?.n}`, + }; +} + +async function checkInsertsStillWork( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + INSERT INTO public.events (payload) + VALUES ('{"regression_probe": true}'::jsonb) + RETURNING recorded_at; + `); + return { + name: 'inserts still work after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0004-resolve-stability-no-primary-key/PROMPT.md b/evals/o11y-0004-resolve-stability-no-primary-key/PROMPT.md new file mode 100644 index 00000000..379bb79b --- /dev/null +++ b/evals/o11y-0004-resolve-stability-no-primary-key/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - migrations +motivation: apps/docs/content/troubleshooting/database-design.mdx +--- + +We tried to turn on Realtime for our `events` table and it won't replicate, and +single-row lookups feel slow. Someone mentioned the table might be missing a +primary key. Can you check, fix whatever's wrong, and make sure existing rows +and inserts still work? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0004-resolve-stability-no-primary-key/remote/project.sql b/evals/o11y-0004-resolve-stability-no-primary-key/remote/project.sql new file mode 100644 index 00000000..bf51c3fb --- /dev/null +++ b/evals/o11y-0004-resolve-stability-no-primary-key/remote/project.sql @@ -0,0 +1,13 @@ +-- Broken starting state (probe: health-no-primary-key). +-- public.events has no primary key, which blocks logical replication (Realtime) +-- and makes single-row addressing impossible. +CREATE TABLE public.events ( + payload jsonb, + recorded_at timestamptz DEFAULT now() +); +ALTER TABLE public.events ENABLE ROW LEVEL SECURITY; + +INSERT INTO public.events (payload, recorded_at) +SELECT jsonb_build_object('seq', n), + now() - (n || ' minutes')::interval +FROM generate_series(1, 200) AS n; diff --git a/evals/o11y-0005-resolve-performance-unused-index/EVAL.ts b/evals/o11y-0005-resolve-performance-unused-index/EVAL.ts new file mode 100644 index 00000000..7591a770 --- /dev/null +++ b/evals/o11y-0005-resolve-performance-unused-index/EVAL.ts @@ -0,0 +1,62 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: idx_unused_total indexes orders(total_cents) but no query ever filters +// on total_cents. The fix drops the index. Since "unused" is definitional +// (pg_stat_user_indexes shows 0 scans) we use a judge + structural check. + +const scorer: ToolScorer = async (ctx) => { + try { + const unusedIndexDropped = await checkUnusedIndexDropped(ctx); + const checks: CheckResult[] = [unusedIndexDropped]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent identified an unused index on the orders table + (idx_unused_total or similar, covering total_cents) and dropped it. + The agent should have explained that the index adds write overhead without + benefiting any read query. + + Fail if the agent dropped the wrong index, kept the unused index, or + gave only a recommendation without acting on it. + `, + }); + checks.push({ + name: 'correctly identified and dropped unused index', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated unused index fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkUnusedIndexDropped(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT indexname + FROM pg_indexes + WHERE schemaname = 'public' + AND tablename = 'orders' + AND indexname = 'idx_unused_total'; + `); + return { + name: 'idx_unused_total index dropped', + passed: rows.length === 0, + notes: rows.length > 0 ? 'index still exists' : 'index dropped', + }; +} diff --git a/evals/o11y-0005-resolve-performance-unused-index/PROMPT.md b/evals/o11y-0005-resolve-performance-unused-index/PROMPT.md new file mode 100644 index 00000000..a094bbcd --- /dev/null +++ b/evals/o11y-0005-resolve-performance-unused-index/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - observability +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our Postgres advisor flagged an index that has never been used in any query but +is still adding overhead to every write on the `orders` table. Can you identify +the unused index and remove it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0005-resolve-performance-unused-index/remote/project.sql b/evals/o11y-0005-resolve-performance-unused-index/remote/project.sql new file mode 100644 index 00000000..37ddfcaf --- /dev/null +++ b/evals/o11y-0005-resolve-performance-unused-index/remote/project.sql @@ -0,0 +1,42 @@ +-- Broken starting state (probe: performance-unused-index / Splinter lint 0005). +-- idx_unused_total indexes orders(total_cents) but no query ever filters by +-- total_cents, so it adds write overhead on every INSERT/UPDATE without +-- benefiting any read. Seeded index_usage_stats shows zero scans. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +CREATE INDEX idx_orders_customer_id ON public.orders(customer_id); +-- The unused index: no query ever filters by total_cents. +CREATE INDEX idx_unused_total ON public.orders(total_cents); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 100) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 100) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 5000); + +-- Faked index usage stats (named to avoid collision with pg_stat_user_indexes system view). +CREATE TABLE public.index_usage_stats ( + schemaname name, + relname name, + indexrelname name, + idx_scan bigint, + idx_tup_read bigint, + idx_tup_fetch bigint +); + +INSERT INTO public.index_usage_stats + (schemaname, relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch) +VALUES + ('public', 'orders', 'idx_orders_customer_id', 184022, 921000, 840000), + ('public', 'orders', 'idx_unused_total', 0, 0, 0); diff --git a/evals/o11y-0006-resolve-security-multiple-permissive-policies/EVAL.ts b/evals/o11y-0006-resolve-security-multiple-permissive-policies/EVAL.ts new file mode 100644 index 00000000..b0d9c868 --- /dev/null +++ b/evals/o11y-0006-resolve-security-multiple-permissive-policies/EVAL.ts @@ -0,0 +1,93 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: profiles has two PERMISSIVE SELECT policies for `authenticated`, which +// Postgres ORs together. The fix collapses SELECT access down to a single +// permissive policy (the owner-scoped one) so access is no longer widened. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkSinglePermissiveSelectPolicy(ctx), + await checkRemainingPolicyIsOwnerScoped(ctx), + await checkInsertsStillWork(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated policy fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function selectPolicies(ctx: ToolEvalContext) { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, permissive, roles, qual + FROM pg_policies + WHERE schemaname = 'public' + AND tablename = 'profiles' + AND cmd = 'SELECT'; + `); + return rows.filter((r) => { + const roles = Array.isArray(r.roles) + ? (r.roles as string[]) + : String(r.roles ?? '').replace(/[{}]/g, '').split(','); + return ( + String(r.permissive).toUpperCase().startsWith('PERM') && + roles.includes('authenticated') + ); + }); +} + +/** Exactly one permissive SELECT policy should remain for `authenticated`. */ +async function checkSinglePermissiveSelectPolicy( + ctx: ToolEvalContext +): Promise { + const policies = await selectPolicies(ctx); + return { + name: 'exactly one permissive SELECT policy for authenticated', + passed: policies.length === 1, + notes: `count=${policies.length}: ${policies + .map((p) => p.policyname) + .join(', ')}`, + }; +} + +/** The surviving policy must still restrict to the row owner. */ +async function checkRemainingPolicyIsOwnerScoped( + ctx: ToolEvalContext +): Promise { + const policies = await selectPolicies(ctx); + const ownerScoped = policies.some((p) => + /auth\.uid\(\)/i.test(String(p.qual ?? '')) + ); + return { + name: 'remaining SELECT policy is owner-scoped (auth.uid())', + passed: policies.length >= 1 && ownerScoped, + }; +} + +async function checkInsertsStillWork( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + INSERT INTO public.profiles (handle) + VALUES ('regression_probe') + RETURNING id; + `); + return { + name: 'inserts still work after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0006-resolve-security-multiple-permissive-policies/PROMPT.md b/evals/o11y-0006-resolve-security-multiple-permissive-policies/PROMPT.md new file mode 100644 index 00000000..d0a40403 --- /dev/null +++ b/evals/o11y-0006-resolve-security-multiple-permissive-policies/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +A teammate reported that some users can see other people's rows in the +`profiles` table even though we have a policy that's supposed to restrict reads +to the row owner. I think an extra policy crept in during a copy-paste. Can you +figure out why the restriction isn't holding and fix it so an authenticated user +can only read their own profile? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0006-resolve-security-multiple-permissive-policies/remote/project.sql b/evals/o11y-0006-resolve-security-multiple-permissive-policies/remote/project.sql new file mode 100644 index 00000000..87d5b8c5 --- /dev/null +++ b/evals/o11y-0006-resolve-security-multiple-permissive-policies/remote/project.sql @@ -0,0 +1,23 @@ +-- Broken starting state (probe: security-multiple-permissive-policies). +-- profiles has TWO permissive SELECT policies for `authenticated`. Postgres ORs +-- all permissive policies, so a row is visible if EITHER passes. The admin +-- policy's subquery effectively widens access beyond "own row only". +CREATE TABLE public.profiles ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + handle text UNIQUE NOT NULL, + created_at timestamptz DEFAULT now() +); +ALTER TABLE public.profiles ENABLE ROW LEVEL SECURITY; + +-- Intended policy: own rows only. +CREATE POLICY "profiles_select_own" ON public.profiles + FOR SELECT TO authenticated + USING (id = auth.uid()); + +-- The accidental extra permissive policy that widens SELECT access. +CREATE POLICY "profiles_select_admin" ON public.profiles + FOR SELECT TO authenticated + USING (auth.uid() IN (SELECT id FROM public.profiles WHERE handle = 'admin')); + +INSERT INTO public.profiles (handle) +VALUES ('admin'), ('alice'), ('bob'); diff --git a/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/EVAL.ts b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/EVAL.ts new file mode 100644 index 00000000..025245da --- /dev/null +++ b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/EVAL.ts @@ -0,0 +1,54 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.orders has a SELECT policy defined but RLS is disabled, so the +// policy is never evaluated and all rows are visible. The fix enables RLS. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkRlsEnabled(ctx), + await checkPolicyStillExists(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated RLS policy-exists-disabled fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkRlsEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.orders'::regclass; + `); + return { + name: 'RLS enabled on public.orders', + passed: rows[0]?.relrowsecurity === true, + notes: `relrowsecurity=${rows[0]?.relrowsecurity}`, + }; +} + +async function checkPolicyStillExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname FROM pg_policies + WHERE schemaname = 'public' AND tablename = 'orders'; + `); + return { + name: 'RLS policy still exists on public.orders', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => r.policyname).join(', ') || 'none'}`, + }; +} diff --git a/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/PROMPT.md b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/PROMPT.md new file mode 100644 index 00000000..69ec31fa --- /dev/null +++ b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +We have an RLS policy defined on our `orders` table that's supposed to restrict +access to a customer's own orders — but users are reporting they can still see +each other's data. Can you figure out why the policy isn't being enforced and +fix it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/remote/project.sql b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/remote/project.sql new file mode 100644 index 00000000..52f22a80 --- /dev/null +++ b/evals/o11y-0007-resolve-security-rls-policy-exists-disabled/remote/project.sql @@ -0,0 +1,28 @@ +-- Broken starting state (probe: health-rls-policy-exists-disabled / Splinter lint 0007). +-- public.orders has a SELECT policy defined but RLS is disabled — the policy +-- exists and looks intentional but is silently not enforced. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 100); + +-- Policy exists but RLS is disabled — policy never evaluated. +ALTER TABLE public.orders ENABLE ROW LEVEL SECURITY; +CREATE POLICY "orders_select_own" ON public.orders + FOR SELECT TO authenticated + USING (customer_id IN (SELECT id FROM public.customers WHERE email = auth.email())); +ALTER TABLE public.orders DISABLE ROW LEVEL SECURITY; diff --git a/evals/o11y-0008-resolve-security-rls-enabled-no-policy/EVAL.ts b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/EVAL.ts new file mode 100644 index 00000000..44cc2400 --- /dev/null +++ b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/EVAL.ts @@ -0,0 +1,53 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.documents has RLS enabled but no policies (Postgres default-deny +// returns zero rows for all non-superuser roles). The fix adds a policy. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkPolicyExists(ctx), + await checkRlsStillEnabled(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated RLS no-policy fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkPolicyExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname FROM pg_policies + WHERE schemaname = 'public' AND tablename = 'documents'; + `); + return { + name: 'at least one RLS policy added to public.documents', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => r.policyname).join(', ') || 'none'}`, + }; +} + +async function checkRlsStillEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.documents'::regclass; + `); + return { + name: 'RLS still enabled on public.documents', + passed: rows[0]?.relrowsecurity === true, + }; +} diff --git a/evals/o11y-0008-resolve-security-rls-enabled-no-policy/PROMPT.md b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/PROMPT.md new file mode 100644 index 00000000..ddbdff46 --- /dev/null +++ b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +We enabled row-level security on our `documents` table but now every query +returns zero rows without any error — even though the table definitely has data. +Can you figure out why the table is empty via the API and add the right policy so +our data is actually accessible? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0008-resolve-security-rls-enabled-no-policy/remote/project.sql b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/remote/project.sql new file mode 100644 index 00000000..8bc8510a --- /dev/null +++ b/evals/o11y-0008-resolve-security-rls-enabled-no-policy/remote/project.sql @@ -0,0 +1,20 @@ +-- Broken starting state (probe: health-rls-enabled-no-policy / Splinter lint 0008). +-- public.documents has RLS enabled but no policies. Postgres default-deny means +-- every query returns zero rows for all non-superuser roles — silently, no error. +CREATE TABLE public.documents ( + id bigserial PRIMARY KEY, + user_id uuid NOT NULL, + title text NOT NULL, + body text +); +ALTER TABLE public.documents ENABLE ROW LEVEL SECURITY; + +INSERT INTO public.documents (user_id, title, body) +SELECT + CASE WHEN g % 2 = 0 + THEN '00000000-0000-0000-0000-000000000001' + ELSE '00000000-0000-0000-0000-000000000002' + END::uuid, + 'doc ' || g, + 'content ' || g +FROM generate_series(1, 50) g; diff --git a/evals/o11y-0009-resolve-performance-duplicate-index/EVAL.ts b/evals/o11y-0009-resolve-performance-duplicate-index/EVAL.ts new file mode 100644 index 00000000..0ec1eddb --- /dev/null +++ b/evals/o11y-0009-resolve-performance-duplicate-index/EVAL.ts @@ -0,0 +1,72 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: idx_orders_customer_dup duplicates idx_orders_customer_id — both +// index orders(customer_id). The fix drops the duplicate, leaving the original. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkDuplicateDropped(ctx), + await checkOriginalIndexRetained(ctx), + await checkInsertsStillWork(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated duplicate index fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function customerIdIndexes(ctx: ToolEvalContext) { + const { rows } = await ctx.query(stripIndent` + SELECT indexname, indexdef + FROM pg_indexes + WHERE schemaname = 'public' + AND tablename = 'orders' + AND indexdef ~* 'ON.*orders.*\(customer_id\)'; + `); + return rows; +} + +async function checkDuplicateDropped(ctx: ToolEvalContext): Promise { + const indexes = await customerIdIndexes(ctx); + return { + name: 'duplicate customer_id index dropped (only one remains)', + passed: indexes.length <= 1, + notes: `${indexes.length} index(es) on orders(customer_id): ${indexes.map(r => r.indexname).join(', ')}`, + }; +} + +async function checkOriginalIndexRetained(ctx: ToolEvalContext): Promise { + const indexes = await customerIdIndexes(ctx); + return { + name: 'at least one index on orders(customer_id) retained', + passed: indexes.length >= 1, + notes: indexes.map(r => r.indexname).join(', '), + }; +} + +async function checkInsertsStillWork(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + INSERT INTO public.orders (customer_id, total_cents) + VALUES (1, 999) + RETURNING id; + `); + return { + name: 'inserts still work after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0009-resolve-performance-duplicate-index/PROMPT.md b/evals/o11y-0009-resolve-performance-duplicate-index/PROMPT.md new file mode 100644 index 00000000..4ae0309e --- /dev/null +++ b/evals/o11y-0009-resolve-performance-duplicate-index/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - observability +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our Postgres advisor flagged a duplicate index on the `orders` table — two +indexes covering the same column, which doubles write overhead on every insert +and update without any benefit. Can you find the duplicate and remove it while +keeping the original index intact? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0009-resolve-performance-duplicate-index/remote/project.sql b/evals/o11y-0009-resolve-performance-duplicate-index/remote/project.sql new file mode 100644 index 00000000..c27cc30b --- /dev/null +++ b/evals/o11y-0009-resolve-performance-duplicate-index/remote/project.sql @@ -0,0 +1,26 @@ +-- Broken starting state (probe: performance-duplicate-index / Splinter lint 0009). +-- idx_orders_customer_dup duplicates idx_orders_customer_id — both index +-- orders(customer_id) — wasting write overhead on every INSERT and UPDATE. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 100) AS g; + +-- Original index. +CREATE INDEX idx_orders_customer_id ON public.orders(customer_id); +-- Duplicate index (identical columns, same table). +CREATE INDEX idx_orders_customer_dup ON public.orders(customer_id); + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 100) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 5000); diff --git a/evals/o11y-0010-resolve-security-security-definer-view/EVAL.ts b/evals/o11y-0010-resolve-security-security-definer-view/EVAL.ts new file mode 100644 index 00000000..4c5ca7da --- /dev/null +++ b/evals/o11y-0010-resolve-security-security-definer-view/EVAL.ts @@ -0,0 +1,67 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.order_summary is a SECURITY DEFINER view (Postgres default), +// bypassing RLS on orders. The fix sets security_invoker=on or drops the view. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkViewSafeOrDropped(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated security_definer view fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkViewSafeOrDropped(ctx: ToolEvalContext): Promise { + // Check if view was dropped. + const { rows: viewRows } = await ctx.query(stripIndent` + SELECT table_name + FROM information_schema.views + WHERE table_schema = 'public' AND table_name = 'order_summary'; + `); + + if (viewRows.length === 0) { + return { + name: 'order_summary view dropped or has security_invoker', + passed: true, + notes: 'view dropped', + }; + } + + // If it still exists, check for security_invoker option. + const { rows: optRows } = await ctx.query(stripIndent` + SELECT reloptions + FROM pg_class + WHERE relname = 'order_summary' + AND relnamespace = 'public'::regnamespace + AND relkind = 'v'; + `); + + const opts = optRows[0]?.reloptions; + const optsArr = Array.isArray(opts) ? opts : (opts ? [opts] : []); + const hasSecurityInvoker = optsArr.some((o: string) => + /security_invoker\s*=\s*(on|true|1)/i.test(o) + ); + + return { + name: 'order_summary view dropped or has security_invoker', + passed: hasSecurityInvoker, + notes: `view exists; security_invoker=${hasSecurityInvoker}; reloptions=${JSON.stringify(optsArr)}`, + }; +} diff --git a/evals/o11y-0010-resolve-security-security-definer-view/PROMPT.md b/evals/o11y-0010-resolve-security-security-definer-view/PROMPT.md new file mode 100644 index 00000000..f361d541 --- /dev/null +++ b/evals/o11y-0010-resolve-security-security-definer-view/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - security + - rls +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +We have a view called `order_summary` that the advisor says is using +`SECURITY DEFINER` behavior — meaning it runs as its owner and bypasses any RLS +policies on the underlying tables. Any authenticated user who can SELECT from +the view can read all orders regardless of what RLS says. Can you fix it so the +view respects the caller's permissions instead? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0010-resolve-security-security-definer-view/remote/project.sql b/evals/o11y-0010-resolve-security-security-definer-view/remote/project.sql new file mode 100644 index 00000000..9bd6557d --- /dev/null +++ b/evals/o11y-0010-resolve-security-security-definer-view/remote/project.sql @@ -0,0 +1,30 @@ +-- Broken starting state (probe: security-security-definer-view / Splinter lint 0010). +-- public.order_summary is a SECURITY DEFINER view (Postgres default for views), +-- bypassing RLS on the underlying orders table for any caller with SELECT on the view. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); +ALTER TABLE public.orders ENABLE ROW LEVEL SECURITY; + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 500); + +-- Postgres views default to SECURITY DEFINER; no extra DDL needed to inject. +CREATE OR REPLACE VIEW public.order_summary AS + SELECT customer_id, count(*) AS order_count, sum(total_cents) AS total_cents + FROM public.orders + GROUP BY customer_id; + +GRANT SELECT ON public.order_summary TO authenticated; diff --git a/evals/o11y-0011-resolve-security-function-search-path-mutable/EVAL.ts b/evals/o11y-0011-resolve-security-function-search-path-mutable/EVAL.ts new file mode 100644 index 00000000..1a3612fd --- /dev/null +++ b/evals/o11y-0011-resolve-security-function-search-path-mutable/EVAL.ts @@ -0,0 +1,74 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.get_customer_balance() is SECURITY DEFINER with no SET +// search_path, allowing search_path injection. The fix pins search_path to ''. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkSearchPathPinned(ctx), + await checkFunctionStillExists(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated search_path fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkSearchPathPinned(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT proname, proconfig + FROM pg_proc + WHERE proname = 'get_customer_balance' + AND pronamespace = 'public'::regnamespace; + `); + + if (rows.length === 0) { + return { + name: 'get_customer_balance has pinned search_path', + passed: false, + notes: 'function not found', + }; + } + + const config = rows[0]?.proconfig; + const configs = Array.isArray(config) ? config : (config ? [config] : []); + // Postgres stores SET search_path='' as search_path="" in proconfig + const hasPinnedSearchPath = configs.some((c: string) => + /search_path\s*=\s*('')/i.test(c) || + /search_path\s*=\s*("")/i.test(c) || + /search_path\s*=\s*$/i.test(c) + ); + + return { + name: 'get_customer_balance has pinned search_path (SET search_path = \'\')', + passed: hasPinnedSearchPath, + notes: `proconfig: ${JSON.stringify(configs)}`, + }; +} + +async function checkFunctionStillExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT proname FROM pg_proc + WHERE proname = 'get_customer_balance' + AND pronamespace = 'public'::regnamespace; + `); + return { + name: 'function still exists after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0011-resolve-security-function-search-path-mutable/PROMPT.md b/evals/o11y-0011-resolve-security-function-search-path-mutable/PROMPT.md new file mode 100644 index 00000000..10e1fa49 --- /dev/null +++ b/evals/o11y-0011-resolve-security-function-search-path-mutable/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - security + - sql +motivation: apps/docs/content/troubleshooting/database-roles.mdx +--- + +A security scan flagged that one of our database functions is marked +`SECURITY DEFINER` but doesn't pin its `search_path`. This means an attacker +could shadow system functions by creating objects in a schema that appears +earlier on the search path, and our function would execute them with elevated +privileges. Can you find the vulnerable function and fix it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0011-resolve-security-function-search-path-mutable/remote/project.sql b/evals/o11y-0011-resolve-security-function-search-path-mutable/remote/project.sql new file mode 100644 index 00000000..6d281c16 --- /dev/null +++ b/evals/o11y-0011-resolve-security-function-search-path-mutable/remote/project.sql @@ -0,0 +1,30 @@ +-- Broken starting state (probe: security-function-search-path-mutable / Splinter lint 0011). +-- get_customer_balance() is SECURITY DEFINER with no SET search_path, allowing +-- search_path injection if an attacker can create objects in any schema on the +-- search path. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +-- The vulnerable function: SECURITY DEFINER with mutable search_path. +CREATE OR REPLACE FUNCTION public.get_customer_balance(p_customer_id int) + RETURNS bigint + LANGUAGE sql + SECURITY DEFINER + -- intentionally omitting SET search_path = '' to trigger lint 0011 + AS $$ + SELECT coalesce(sum(total_cents), 0) + FROM public.orders + WHERE customer_id = p_customer_id; + $$; diff --git a/evals/o11y-0013-resolve-security-rls-disabled/EVAL.ts b/evals/o11y-0013-resolve-security-rls-disabled/EVAL.ts new file mode 100644 index 00000000..6a098ae8 --- /dev/null +++ b/evals/o11y-0013-resolve-security-rls-disabled/EVAL.ts @@ -0,0 +1,80 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.profiles has RLS disabled and SELECT granted to anon. +// A correct fix enables RLS and adds an owner-scoped SELECT policy, without +// breaking normal writes to the table. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkRlsEnabled(ctx), + await checkOwnerScopedPolicy(ctx), + await checkInsertsStillWork(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated RLS fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +/** RLS must be enabled on public.profiles. */ +async function checkRlsEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relrowsecurity + FROM pg_class + WHERE oid = 'public.profiles'::regclass; + `); + return { + name: 'RLS enabled on public.profiles', + passed: rows[0]?.relrowsecurity === true, + notes: `relrowsecurity=${rows[0]?.relrowsecurity}`, + }; +} + +/** At least one SELECT policy must exist that scopes rows to the current user. */ +async function checkOwnerScopedPolicy( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, qual + FROM pg_policies + WHERE schemaname = 'public' + AND tablename = 'profiles' + AND (cmd = 'SELECT' OR cmd = 'ALL'); + `); + const hasOwnerScope = rows.some((r) => + /auth\.uid\(\)/i.test(String(r.qual ?? '')) + ); + return { + name: 'owner-scoped SELECT policy references auth.uid()', + passed: hasOwnerScope, + notes: `found ${rows.length} select-capable policies`, + }; +} + +/** The fix must not break normal writes to the table. */ +async function checkInsertsStillWork( + ctx: ToolEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + INSERT INTO public.profiles (handle) + VALUES ('regression_probe') + RETURNING id; + `); + return { + name: 'inserts still work after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0013-resolve-security-rls-disabled/PROMPT.md b/evals/o11y-0013-resolve-security-rls-disabled/PROMPT.md new file mode 100644 index 00000000..592a70d9 --- /dev/null +++ b/evals/o11y-0013-resolve-security-rls-disabled/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/rls-simplified.mdx +--- + +We're about to launch and a security review flagged that our `profiles` table +might be readable by anyone holding our anon key. Can you check whether that's +actually the case, lock it down so a signed-in user can only read their own +profile, and confirm it's fixed? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0013-resolve-security-rls-disabled/remote/project.sql b/evals/o11y-0013-resolve-security-rls-disabled/remote/project.sql new file mode 100644 index 00000000..09555011 --- /dev/null +++ b/evals/o11y-0013-resolve-security-rls-disabled/remote/project.sql @@ -0,0 +1,15 @@ +-- Broken starting state (probe: security-rls-disabled). +-- profiles is exposed to the API with RLS disabled and SELECT granted to anon, +-- so anon can read every row. The baseline app expects a signed-in user to see +-- only their own profile. +CREATE TABLE public.profiles ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + handle text UNIQUE NOT NULL, + created_at timestamptz DEFAULT now() +); + +-- RLS is never enabled, and anon is granted read access. +GRANT SELECT ON public.profiles TO anon; + +INSERT INTO public.profiles (handle) +VALUES ('alice'), ('bob'), ('carol'); diff --git a/evals/o11y-0014-resolve-security-extension-in-public/EVAL.ts b/evals/o11y-0014-resolve-security-extension-in-public/EVAL.ts new file mode 100644 index 00000000..4c15e569 --- /dev/null +++ b/evals/o11y-0014-resolve-security-extension-in-public/EVAL.ts @@ -0,0 +1,68 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: pg_trgm extension functions (similarity, show_trgm) are in public +// schema, exposing them via the Data API. The fix drops or moves them to the +// extensions schema. +// HARNESS NOTE: PGlite cannot install pg_trgm; the issue is simulated by +// seeding the functions + extension_info table. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkSimilarityNotInPublic(ctx), + ]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent identified that pg_trgm (or its functions) are in the + public schema instead of the extensions schema, and proposed or applied a + fix: either DROP EXTENSION pg_trgm and recreate it in the extensions schema, + or DROP the individual functions from public. The agent should explain that + public-schema extensions expose their functions via the PostgREST Data API. + + Fail if the agent did not identify the schema placement as the problem, or + gave only generic advice without a specific schema change. + `, + }); + checks.push({ + name: 'correctly identified extension-in-public and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated extension-in-public fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkSimilarityNotInPublic(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT proname FROM pg_proc + WHERE pronamespace = 'public'::regnamespace + AND proname IN ('similarity', 'show_trgm'); + `); + return { + name: 'pg_trgm functions removed from public schema', + passed: rows.length === 0, + notes: rows.length > 0 + ? `still in public: ${rows.map((r: { proname: string }) => r.proname).join(', ')}` + : 'no pg_trgm functions in public', + }; +} diff --git a/evals/o11y-0014-resolve-security-extension-in-public/PROMPT.md b/evals/o11y-0014-resolve-security-extension-in-public/PROMPT.md new file mode 100644 index 00000000..2ad91096 --- /dev/null +++ b/evals/o11y-0014-resolve-security-extension-in-public/PROMPT.md @@ -0,0 +1,24 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - security + - sql +motivation: apps/docs/content/troubleshooting/extensions.mdx +--- + +The advisor flagged that we have functions from the `pg_trgm` extension — including +`similarity()` and `show_trgm()` — living in the `public` schema. This exposes +them in the PostgREST Data API surface, where any authenticated user can call them. +Can you remove these functions from the `public` schema? + +Note: In this environment `pg_trgm` is represented by stubs in `public`. Drop +the `similarity` and `show_trgm` functions from the `public` schema directly +(you won't be able to reinstall the extension itself here). + +End your turn with a short summary of what you changed and why. + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0014-resolve-security-extension-in-public/remote/project.sql b/evals/o11y-0014-resolve-security-extension-in-public/remote/project.sql new file mode 100644 index 00000000..1c9cdcc6 --- /dev/null +++ b/evals/o11y-0014-resolve-security-extension-in-public/remote/project.sql @@ -0,0 +1,28 @@ +-- Starting state (probe: health-extension-in-public / Splinter lint 0014). +-- pg_trgm is installed in the public schema, exposing its similarity() and +-- other functions to the Data API and any role with USAGE on public. +-- HARNESS NOTE: PGlite cannot install pg_trgm; the schema structure is seeded +-- and the eval scores on the agent's diagnostic report via judge. +CREATE SCHEMA IF NOT EXISTS extensions; + +-- Simulate the functions that pg_trgm would expose in public schema. +CREATE OR REPLACE FUNCTION public.similarity(text, text) + RETURNS float4 LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ + SELECT 0.0::float4; +$$; + +CREATE OR REPLACE FUNCTION public.show_trgm(text) + RETURNS text[] LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ + SELECT '{}'::text[]; +$$; + +-- Fake pg_extension row to simulate pg_trgm installed in public. +-- This allows the agent to see the extension "installed" in public. +CREATE TABLE IF NOT EXISTS public.extension_info ( + extname text, + extschema text, + extversion text +); + +INSERT INTO public.extension_info (extname, extschema, extversion) +VALUES ('pg_trgm', 'public', '1.6'); diff --git a/evals/o11y-0015-resolve-security-rls-references-user-metadata/EVAL.ts b/evals/o11y-0015-resolve-security-rls-references-user-metadata/EVAL.ts new file mode 100644 index 00000000..7545ed55 --- /dev/null +++ b/evals/o11y-0015-resolve-security-rls-references-user-metadata/EVAL.ts @@ -0,0 +1,65 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: customers RLS policy uses auth.jwt() ->> 'user_metadata' (user-editable). +// The fix rewrites the policy to use app_metadata or a separate roles table. + +const scorer: ToolScorer = async (ctx) => { + try { + const noUserMetadata = await checkNoUserMetadataPolicy(ctx); + const checks: CheckResult[] = [noUserMetadata]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent: + 1. Explained that user_metadata is user-editable and therefore unsafe for + access-control decisions. + 2. Replaced the policy with one that uses app_metadata (server-controlled), + a separate roles table, or removed the policy entirely with a safe + replacement. + + Fail if the agent kept a policy using user_metadata, or only added + documentation without changing the policy. + `, + }); + checks.push({ + name: 'policy correctly rewritten away from user_metadata', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated user_metadata policy fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkNoUserMetadataPolicy(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, qual + FROM pg_policies + WHERE schemaname = 'public' + AND tablename = 'customers' + AND qual ILIKE '%user_metadata%'; + `); + return { + name: 'no policy references user_metadata on customers', + passed: rows.length === 0, + notes: rows.length > 0 + ? `still referencing user_metadata: ${rows.map(r => r.policyname).join(', ')}` + : 'none found', + }; +} diff --git a/evals/o11y-0015-resolve-security-rls-references-user-metadata/PROMPT.md b/evals/o11y-0015-resolve-security-rls-references-user-metadata/PROMPT.md new file mode 100644 index 00000000..dc781933 --- /dev/null +++ b/evals/o11y-0015-resolve-security-rls-references-user-metadata/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +The security advisor flagged an RLS policy on our `customers` table that grants +SELECT access based on `user_metadata` from the JWT — which is user-editable. Any +authenticated user can self-promote by setting `user_metadata.role = 'admin'` and +immediately read all customer rows. Can you fix the policy to use a server-controlled +claim instead? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0015-resolve-security-rls-references-user-metadata/remote/project.sql b/evals/o11y-0015-resolve-security-rls-references-user-metadata/remote/project.sql new file mode 100644 index 00000000..22c8d588 --- /dev/null +++ b/evals/o11y-0015-resolve-security-rls-references-user-metadata/remote/project.sql @@ -0,0 +1,15 @@ +-- Broken starting state (probe: security-rls-references-user-metadata / Splinter lint 0015). +-- customers has an RLS policy that grants SELECT based on user_metadata.role = 'admin'. +-- user_metadata is user-editable — any authenticated user can self-promote. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +INSERT INTO public.customers (email) +SELECT 'customer' || g || '@example.com' FROM generate_series(1, 50) AS g; + +ALTER TABLE public.customers ENABLE ROW LEVEL SECURITY; +CREATE POLICY "customers_metadata_role" ON public.customers + FOR SELECT TO authenticated + USING (((auth.jwt() ->> 'user_metadata')::jsonb ->> 'role') = 'admin'); diff --git a/evals/o11y-0016-resolve-security-materialized-view-in-api/EVAL.ts b/evals/o11y-0016-resolve-security-materialized-view-in-api/EVAL.ts new file mode 100644 index 00000000..f22ac1dd --- /dev/null +++ b/evals/o11y-0016-resolve-security-materialized-view-in-api/EVAL.ts @@ -0,0 +1,64 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.order_totals is a materialized view — RLS cannot be applied to +// matviews. The fix drops the matview or revokes anon/authenticated SELECT. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkMatviewDroppedOrRevoked(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated materialized view fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkMatviewDroppedOrRevoked( + ctx: ToolEvalContext +): Promise { + // Check if matview still exists. + const { rows: matviewRows } = await ctx.query(stripIndent` + SELECT relname FROM pg_class + WHERE relname = 'order_totals' + AND relnamespace = 'public'::regnamespace + AND relkind = 'm'; + `); + + if (matviewRows.length === 0) { + return { + name: 'order_totals materialized view dropped or grants revoked', + passed: true, + notes: 'matview dropped', + }; + } + + // Still exists — check grants revoked. + const { rows: grantRows } = await ctx.query(stripIndent` + SELECT grantee, privilege_type + FROM information_schema.role_table_grants + WHERE table_schema = 'public' + AND table_name = 'order_totals' + AND grantee IN ('anon', 'authenticated'); + `); + + return { + name: 'order_totals materialized view dropped or grants revoked', + passed: grantRows.length === 0, + notes: `matview still exists; grants remaining: ${grantRows.map(r => `${r.grantee}:${r.privilege_type}`).join(', ') || 'none'}`, + }; +} diff --git a/evals/o11y-0016-resolve-security-materialized-view-in-api/PROMPT.md b/evals/o11y-0016-resolve-security-materialized-view-in-api/PROMPT.md new file mode 100644 index 00000000..4b1408a7 --- /dev/null +++ b/evals/o11y-0016-resolve-security-materialized-view-in-api/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - security + - rls +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +The advisor flagged that we have a materialized view in the public schema that +any authenticated user can query. Unlike regular views, RLS cannot be applied to +materialized views — so all aggregated revenue data across every customer is +visible to any API caller. Can you fix this so the data is no longer accessible +via the API? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0016-resolve-security-materialized-view-in-api/remote/project.sql b/evals/o11y-0016-resolve-security-materialized-view-in-api/remote/project.sql new file mode 100644 index 00000000..76d5ffb5 --- /dev/null +++ b/evals/o11y-0016-resolve-security-materialized-view-in-api/remote/project.sql @@ -0,0 +1,29 @@ +-- Broken starting state (probe: security-materialized-view-in-api / Splinter lint 0016). +-- public.order_totals is a materialized view — RLS cannot be enforced on +-- materialized views, exposing aggregated revenue data across all customers to +-- any authenticated API caller. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 500); + +CREATE MATERIALIZED VIEW public.order_totals AS + SELECT customer_id, sum(total_cents) AS total + FROM public.orders + GROUP BY customer_id; + +GRANT SELECT ON public.order_totals TO authenticated, anon; diff --git a/evals/o11y-0018-resolve-stability-unsupported-reg-types/EVAL.ts b/evals/o11y-0018-resolve-stability-unsupported-reg-types/EVAL.ts new file mode 100644 index 00000000..01f61e5a --- /dev/null +++ b/evals/o11y-0018-resolve-stability-unsupported-reg-types/EVAL.ts @@ -0,0 +1,58 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.cached_relations has a 'relation_oid' column of type regclass. +// The fix changes the column type to text (storing the relation name). + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkNoRegclassColumn(ctx), + await checkTableStillExists(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated reg types fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkNoRegclassColumn(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT column_name, data_type, udt_name + FROM information_schema.columns + WHERE table_schema = 'public' + AND table_name = 'cached_relations' + AND udt_name IN ('regclass', 'regproc', 'regtype', 'regprocedure', 'regoper', 'regoperator', 'regconfig', 'regdictionary'); + `); + return { + name: 'no regclass/reg* columns remain in public.cached_relations', + passed: rows.length === 0, + notes: rows.length > 0 + ? `still has reg* columns: ${rows.map(r => `${r.column_name}:${r.udt_name}`).join(', ')}` + : 'none found', + }; +} + +async function checkTableStillExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT table_name FROM information_schema.tables + WHERE table_schema = 'public' AND table_name = 'cached_relations'; + `); + return { + name: 'public.cached_relations table still exists', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0018-resolve-stability-unsupported-reg-types/PROMPT.md b/evals/o11y-0018-resolve-stability-unsupported-reg-types/PROMPT.md new file mode 100644 index 00000000..5a3cc617 --- /dev/null +++ b/evals/o11y-0018-resolve-stability-unsupported-reg-types/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - migrations +motivation: apps/docs/content/troubleshooting/database-design.mdx +--- + +The advisor flagged that we have a table column using a `regclass` type — an OID +alias that stores object identifiers which are cluster-local and won't survive a +logical backup/restore or Postgres major-version upgrade. Supabase uses logical +backup for upgrades, so this column could corrupt silently. Can you fix it to +use a stable type instead? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0018-resolve-stability-unsupported-reg-types/remote/project.sql b/evals/o11y-0018-resolve-stability-unsupported-reg-types/remote/project.sql new file mode 100644 index 00000000..9fd3e509 --- /dev/null +++ b/evals/o11y-0018-resolve-stability-unsupported-reg-types/remote/project.sql @@ -0,0 +1,11 @@ +-- Broken starting state (probe: health-unsupported-reg-types / Splinter lint 0018). +-- public.cached_relations has a 'relation_oid' column of type regclass. The OID +-- mapping is cluster-local and will not survive a logical backup/restore or +-- Postgres major-version upgrade. +CREATE TABLE public.cached_relations ( + id bigserial PRIMARY KEY, + relation_oid regclass NOT NULL +); + +INSERT INTO public.cached_relations (relation_oid) +VALUES ('pg_class'::regclass), ('pg_attribute'::regclass), ('pg_index'::regclass); diff --git a/evals/o11y-0019-resolve-security-pgmq-queue-exposed/EVAL.ts b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/EVAL.ts new file mode 100644 index 00000000..3286759f --- /dev/null +++ b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/EVAL.ts @@ -0,0 +1,63 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: pgmq queue table has no RLS or grants restricted to service role only. +// BLOCKED: pgmq extension not available in PGlite; faked schema used. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = []; + + // Check RLS enabled on the faked queue table. + const { rows: rlsRows } = await ctx.query(` + SELECT relrowsecurity FROM pg_class + WHERE relname = 'q_order_events' + AND relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'pgmq'); + `); + const rlsEnabled = rlsRows[0]?.relrowsecurity === true; + checks.push({ name: 'RLS enabled on pgmq.q_order_events', passed: rlsEnabled }); + + // Check anon/authenticated no longer have SELECT on the queue. + const { rows: anonGrant } = await ctx.query(` + SELECT has_table_privilege('anon', 'pgmq.q_order_events', 'SELECT') AS has_select; + `); + const anonRevoked = anonGrant[0]?.has_select !== true; + checks.push({ + name: 'anon no longer has SELECT on pgmq.q_order_events', + passed: anonRevoked, + }); + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent enabled RLS on the pgmq queue table (pgmq.q_order_events + or the equivalent), revoked SELECT/DELETE from anon and authenticated roles, + and explained that queue access should only be allowed via the service role + or a SECURITY DEFINER wrapper function. + + Fail if the agent only described the risk without applying a fix. + `, + }); + checks.push({ + name: 'correctly secured PGMQ queue', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated queue security fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; diff --git a/evals/o11y-0019-resolve-security-pgmq-queue-exposed/PROMPT.md b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/PROMPT.md new file mode 100644 index 00000000..80d62a74 --- /dev/null +++ b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/PROMPT.md @@ -0,0 +1,21 @@ +--- +stage: resolve +suite: other +interface: mcp +product: + - queues + - database +topic: + - security + - rls +motivation: apps/docs/content/troubleshooting/queues.mdx +--- + +Our PGMQ message queue `order_events` has no RLS policies, meaning any +authenticated user can read or delete messages from the queue. Can you lock this +down so that only the service role can access queue messages? + +End your turn with a short summary of what you changed and why. + +> Note: PGMQ extension is not available in the eval harness. The seeded state +> simulates the pgmq schema and queue tables that would exist on a real project. diff --git a/evals/o11y-0019-resolve-security-pgmq-queue-exposed/remote/project.sql b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/remote/project.sql new file mode 100644 index 00000000..700025f9 --- /dev/null +++ b/evals/o11y-0019-resolve-security-pgmq-queue-exposed/remote/project.sql @@ -0,0 +1,20 @@ +-- Starting state (probe: 0019_security-pgmq-queue-exposed). +-- HARNESS NOTE: pgmq extension not available in PGlite. +-- Simulated pgmq schema with queue table exposed to anon/authenticated. +CREATE SCHEMA IF NOT EXISTS pgmq; + +CREATE TABLE pgmq.q_order_events ( + msg_id bigserial NOT NULL, + read_ct int NOT NULL DEFAULT 0, + enqueued_at timestamptz NOT NULL DEFAULT now(), + vt timestamptz NOT NULL, + message jsonb +); + +GRANT SELECT, DELETE ON pgmq.q_order_events TO anon, authenticated; + +INSERT INTO pgmq.q_order_events (vt, message) +SELECT + now() + (g || ' seconds')::interval, + jsonb_build_object('order_id', g, 'customer_id', (g % 10) + 1, 'event', 'created') +FROM generate_series(1, 20) g; diff --git a/evals/o11y-0020-resolve-postgres-table-bloat/EVAL.ts b/evals/o11y-0020-resolve-postgres-table-bloat/EVAL.ts new file mode 100644 index 00000000..d19459c3 --- /dev/null +++ b/evals/o11y-0020-resolve-postgres-table-bloat/EVAL.ts @@ -0,0 +1,65 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.chaos_bloat has autovacuum disabled and severe dead-tuple bloat. +// The fix re-enables autovacuum and runs VACUUM FULL or VACUUM ANALYZE. + +const scorer: ToolScorer = async (ctx) => { + try { + const autovacuumFixed = await checkAutovacuumReEnabled(ctx); + const checks: CheckResult[] = [autovacuumFixed]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent: + 1. Identified that autovacuum was disabled on the chaos_bloat table. + 2. Re-enabled autovacuum (ALTER TABLE ... RESET (autovacuum_enabled) or + SET autovacuum_enabled = true). + 3. Ran VACUUM or VACUUM FULL to reclaim dead tuples immediately. + + Fail if the agent only ran VACUUM without re-enabling autovacuum, or + only changed the setting without vacuuming. + `, + }); + checks.push({ + name: 'autovacuum re-enabled and bloat reclaimed', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated table bloat fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkAutovacuumReEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT reloptions FROM pg_class + WHERE relname = 'chaos_bloat' + AND relnamespace = 'public'::regnamespace; + `); + const opts = rows[0]?.reloptions; + const optsArr = Array.isArray(opts) ? opts : (opts ? [opts] : []); + const autovacuumDisabled = optsArr.some((o: string) => + /autovacuum_enabled\s*=\s*(false|off|0)/i.test(o) + ); + return { + name: 'autovacuum_enabled no longer disabled on chaos_bloat', + passed: !autovacuumDisabled, + notes: `reloptions: ${JSON.stringify(optsArr)}`, + }; +} diff --git a/evals/o11y-0020-resolve-postgres-table-bloat/PROMPT.md b/evals/o11y-0020-resolve-postgres-table-bloat/PROMPT.md new file mode 100644 index 00000000..a15c0b4d --- /dev/null +++ b/evals/o11y-0020-resolve-postgres-table-bloat/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - sql + - observability +motivation: apps/docs/content/troubleshooting/autovacuum.mdx +--- + +The Postgres advisor flagged that our `chaos_bloat` table has severe dead-tuple +bloat — autovacuum is disabled and ~99% of rows are dead tuples taking up disk +space. Can you reclaim the space and make sure autovacuum can keep the table +clean going forward? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0020-resolve-postgres-table-bloat/remote/project.sql b/evals/o11y-0020-resolve-postgres-table-bloat/remote/project.sql new file mode 100644 index 00000000..b47e644c --- /dev/null +++ b/evals/o11y-0020-resolve-postgres-table-bloat/remote/project.sql @@ -0,0 +1,14 @@ +-- Starting state (probe: postgres-table-bloat / Splinter lint 0020). +-- chaos_bloat has autovacuum disabled and ~99% dead tuples (~25MB raw waste, +-- bloat ratio ~98x). Both lint thresholds are exceeded: bloat>70 and raw_waste>20MB. +CREATE TABLE public.chaos_bloat ( + id bigserial PRIMARY KEY, + data text +); +ALTER TABLE public.chaos_bloat ENABLE ROW LEVEL SECURITY; +ALTER TABLE public.chaos_bloat SET (autovacuum_enabled = false); + +INSERT INTO public.chaos_bloat (data) + SELECT repeat('x', 1000) FROM generate_series(1, 25000) i; +DELETE FROM public.chaos_bloat WHERE id % 100 != 0; +ANALYZE public.chaos_bloat; diff --git a/evals/o11y-0021-resolve-security-fkey-to-auth-unique/EVAL.ts b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/EVAL.ts new file mode 100644 index 00000000..a24a0d85 --- /dev/null +++ b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/EVAL.ts @@ -0,0 +1,72 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.user_refs.user_email references auth.users(email) — a non-PK +// unique column. Fix: change FK to auth.users(id) or drop the table. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkNoFkeyToAuthEmail(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated fkey-to-auth-unique fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkNoFkeyToAuthEmail(ctx: ToolEvalContext): Promise { + // Check if table is dropped entirely. + const { rows: tableRows } = await ctx.query(stripIndent` + SELECT table_name FROM information_schema.tables + WHERE table_schema = 'public' AND table_name = 'user_refs'; + `); + + if (tableRows.length === 0) { + return { + name: 'no FK to app_users(email) — table dropped or restructured', + passed: true, + notes: 'user_refs table dropped', + }; + } + + // Table still exists — check for FK referencing auth.users.email. + const { rows } = await ctx.query(stripIndent` + SELECT tc.constraint_name, kcu.column_name, + ccu.table_schema AS foreign_schema, + ccu.table_name AS foreign_table, + ccu.column_name AS foreign_column + FROM information_schema.table_constraints tc + JOIN information_schema.key_column_usage kcu + ON tc.constraint_name = kcu.constraint_name AND tc.table_schema = kcu.table_schema + JOIN information_schema.constraint_column_usage ccu + ON ccu.constraint_name = tc.constraint_name + WHERE tc.constraint_type = 'FOREIGN KEY' + AND tc.table_schema = 'public' + AND tc.table_name = 'user_refs' + AND ccu.table_schema = 'public' + AND ccu.table_name = 'app_users' + AND ccu.column_name = 'email'; + `); + + return { + name: 'no FK to app_users(email) — FK references id or table restructured', + passed: rows.length === 0, + notes: rows.length > 0 + ? `still has FK to app_users(email): ${rows.map(r => r.constraint_name).join(', ')}` + : 'FK to email removed', + }; +} diff --git a/evals/o11y-0021-resolve-security-fkey-to-auth-unique/PROMPT.md b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/PROMPT.md new file mode 100644 index 00000000..d99f0a71 --- /dev/null +++ b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/PROMPT.md @@ -0,0 +1,21 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database + - auth +topic: + - security + - sql + - migrations +motivation: apps/docs/content/troubleshooting/database-roles.mdx +--- + +The advisor flagged that we have a foreign key on `public.user_refs` referencing +`public.app_users(email)` — a unique but non-primary-key column. Foreign keys +to non-PK unique columns cannot be restored by `pg_upgrade` during a +major-version upgrade, blocking zero-downtime upgrades. Can you fix the foreign +key to reference `app_users(id)` instead, or restructure the table? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0021-resolve-security-fkey-to-auth-unique/remote/project.sql b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/remote/project.sql new file mode 100644 index 00000000..b7722f0a --- /dev/null +++ b/evals/o11y-0021-resolve-security-fkey-to-auth-unique/remote/project.sql @@ -0,0 +1,25 @@ +-- Starting state (probe: security-fkey-to-auth-unique / Splinter lint 0021). +-- A table references auth.users(email) — a unique non-PK column — instead of +-- auth.users(id). This FK cannot be restored by pg_upgrade during a major +-- version upgrade, making zero-downtime upgrades impossible. +-- HARNESS NOTE: platform-lite's auth.users schema differs from production. +-- Using public.app_users to replicate the exact pattern. +CREATE TABLE public.app_users ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + email text UNIQUE NOT NULL +); + +INSERT INTO public.app_users (id, email) +VALUES + ('00000000-0000-0000-0000-000000000001', 'alice@example.com'), + ('00000000-0000-0000-0000-000000000002', 'bob@example.com'); + +-- The broken FK: references email (unique non-PK) instead of id (PK). +-- On a real project this would be: REFERENCES auth.users(email) +CREATE TABLE public.user_refs ( + id bigserial PRIMARY KEY, + user_email text UNIQUE REFERENCES public.app_users(email) +); + +INSERT INTO public.user_refs (user_email) +VALUES ('alice@example.com'), ('bob@example.com'); diff --git a/evals/o11y-0023-resolve-security-sensitive-columns-exposed/EVAL.ts b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/EVAL.ts new file mode 100644 index 00000000..ea75588e --- /dev/null +++ b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/EVAL.ts @@ -0,0 +1,54 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.api_keys has sensitive columns (password, secret) with RLS +// disabled. The fix enables RLS and adds an owner-scoped policy. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkRlsEnabled(ctx), + await checkPolicyExists(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated sensitive columns fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkRlsEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.api_keys'::regclass; + `); + return { + name: 'RLS enabled on public.api_keys', + passed: rows[0]?.relrowsecurity === true, + notes: `relrowsecurity=${rows[0]?.relrowsecurity}`, + }; +} + +async function checkPolicyExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname FROM pg_policies + WHERE schemaname = 'public' AND tablename = 'api_keys'; + `); + return { + name: 'at least one RLS policy exists on public.api_keys', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => r.policyname).join(', ') || 'none'}`, + }; +} diff --git a/evals/o11y-0023-resolve-security-sensitive-columns-exposed/PROMPT.md b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/PROMPT.md new file mode 100644 index 00000000..f88217cd --- /dev/null +++ b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +Our security advisor flagged that we have a table called `api_keys` with columns +named `password` and `secret` that has no row-level security — any authenticated +user can read all API credentials via the Data API. Can you lock it down so +credentials are protected? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0023-resolve-security-sensitive-columns-exposed/remote/project.sql b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/remote/project.sql new file mode 100644 index 00000000..9676b6d7 --- /dev/null +++ b/evals/o11y-0023-resolve-security-sensitive-columns-exposed/remote/project.sql @@ -0,0 +1,18 @@ +-- Broken starting state (probe: security-sensitive-columns-exposed / Splinter lint 0023). +-- public.api_keys has columns named 'password' and 'secret' with RLS disabled, +-- so any authenticated user can read all API credentials via the Data API. +CREATE TABLE public.api_keys ( + id bigserial PRIMARY KEY, + user_id uuid NOT NULL, + name text NOT NULL, + password text NOT NULL, + secret text NOT NULL, + created_at timestamptz DEFAULT now() +); + +GRANT SELECT ON public.api_keys TO authenticated; + +INSERT INTO public.api_keys (user_id, name, password, secret) +VALUES + ('00000000-0000-0000-0000-000000000001', 'prod-key', 'hunter2', 'sk-real-secret-abc123'), + ('00000000-0000-0000-0000-000000000002', 'dev-key', 'qwerty', 'sk-real-secret-def456'); diff --git a/evals/o11y-0024-resolve-security-rls-policy-always-true/EVAL.ts b/evals/o11y-0024-resolve-security-rls-policy-always-true/EVAL.ts new file mode 100644 index 00000000..442c1bcf --- /dev/null +++ b/evals/o11y-0024-resolve-security-rls-policy-always-true/EVAL.ts @@ -0,0 +1,59 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.orders has a FOR ALL policy with USING (true), effectively +// granting unrestricted access. The fix drops or rewrites the policy. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkNoAlwaysTruePolicy(ctx), + await checkRlsStillEnabled(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated always-true policy fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkNoAlwaysTruePolicy(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, cmd, qual + FROM pg_policies + WHERE schemaname = 'public' + AND tablename = 'orders' + AND cmd IN ('ALL', 'UPDATE', 'DELETE') + AND qual = 'true'; + `); + return { + name: 'no ALL/UPDATE/DELETE policy with USING (true) remains', + passed: rows.length === 0, + notes: rows.length > 0 + ? `still present: ${rows.map(r => `${r.policyname}(${r.cmd})`).join(', ')}` + : 'none found', + }; +} + +async function checkRlsStillEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.orders'::regclass; + `); + return { + name: 'RLS still enabled on public.orders', + passed: rows[0]?.relrowsecurity === true, + }; +} diff --git a/evals/o11y-0024-resolve-security-rls-policy-always-true/PROMPT.md b/evals/o11y-0024-resolve-security-rls-policy-always-true/PROMPT.md new file mode 100644 index 00000000..e42ce210 --- /dev/null +++ b/evals/o11y-0024-resolve-security-rls-policy-always-true/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - rls + - security +motivation: apps/docs/content/troubleshooting/row-level-security.mdx +--- + +The security advisor flagged an RLS policy on our `orders` table with +`USING (true)` for an `ALL` command — which means any authenticated or anon +caller can read, modify, or delete every row despite the table appearing to be +access-controlled. Can you fix the policy so it actually restricts access +instead of granting it to everyone? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0024-resolve-security-rls-policy-always-true/remote/project.sql b/evals/o11y-0024-resolve-security-rls-policy-always-true/remote/project.sql new file mode 100644 index 00000000..82ad6125 --- /dev/null +++ b/evals/o11y-0024-resolve-security-rls-policy-always-true/remote/project.sql @@ -0,0 +1,26 @@ +-- Broken starting state (probe: security-rls-policy-always-true / Splinter lint 0024). +-- public.orders has a FOR ALL policy with USING (true) — every row is readable +-- and writable by anon/authenticated despite appearing access-controlled. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 200); + +ALTER TABLE public.orders ENABLE ROW LEVEL SECURITY; +CREATE POLICY "orders_open" ON public.orders + FOR ALL TO authenticated, anon + USING (true); diff --git a/evals/o11y-0025-resolve-storage-public-bucket/EVAL.ts b/evals/o11y-0025-resolve-storage-public-bucket/EVAL.ts new file mode 100644 index 00000000..4c452c09 --- /dev/null +++ b/evals/o11y-0025-resolve-storage-public-bucket/EVAL.ts @@ -0,0 +1,64 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: storage bucket 'public-assets' has public=true and no owner policy. +// The fix sets public=false and adds an owner-scoped SELECT policy. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkBucketIsPrivate(ctx), + await checkOwnerPolicyExists(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated public bucket fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkBucketIsPrivate( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT name, public FROM storage.buckets + WHERE id = 'public-assets'; + `); + if (rows.length === 0) { + return { + name: 'public-assets bucket is private', + passed: false, + notes: 'bucket not found', + }; + } + return { + name: 'public-assets bucket is private (public=false)', + passed: rows[0]?.public === false, + notes: `public=${rows[0]?.public}`, + }; +} + +async function checkOwnerPolicyExists( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname FROM pg_policies + WHERE schemaname = 'storage' + AND tablename = 'objects' + AND (cmd = 'SELECT' OR cmd = 'ALL'); + `); + return { + name: 'owner-scoped SELECT policy exists on storage.objects', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => r.policyname).join(', ') || 'none'}`, + }; +} diff --git a/evals/o11y-0025-resolve-storage-public-bucket/PROMPT.md b/evals/o11y-0025-resolve-storage-public-bucket/PROMPT.md new file mode 100644 index 00000000..db318bd0 --- /dev/null +++ b/evals/o11y-0025-resolve-storage-public-bucket/PROMPT.md @@ -0,0 +1,21 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - storage +topic: + - security +services: + - storage-api + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/storage-403-unauthorized.mdx +--- + +The security advisor flagged that our `public-assets` storage bucket is set to +public — any unauthenticated user can read any file in it without any policy +check. Can you make the bucket private and add an owner-scoped policy so only +authenticated owners can access their objects? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/config.toml b/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/config.toml new file mode 100644 index 00000000..f2468a4b --- /dev/null +++ b/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-storage-public-bucket" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public", "storage"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[storage] +enabled = true diff --git a/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/migrations/0000_public_bucket.sql b/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/migrations/0000_public_bucket.sql new file mode 100644 index 00000000..0224daf4 --- /dev/null +++ b/evals/o11y-0025-resolve-storage-public-bucket/local/supabase/migrations/0000_public_bucket.sql @@ -0,0 +1,8 @@ +-- Broken starting state (probe: storage-public-bucket). +-- 'public-assets' bucket is public=true with no download policy; any +-- unauthenticated user can read all objects. +INSERT INTO storage.buckets (id, name, public) +VALUES ('public-assets', 'public-assets', true) +ON CONFLICT (id) DO UPDATE SET public = true; + +DROP POLICY IF EXISTS "public_assets_owner_only" ON storage.objects; diff --git a/evals/o11y-0028-0029-resolve-security-anon-security-definer/EVAL.ts b/evals/o11y-0028-0029-resolve-security-anon-security-definer/EVAL.ts new file mode 100644 index 00000000..1a68e2cb --- /dev/null +++ b/evals/o11y-0028-0029-resolve-security-anon-security-definer/EVAL.ts @@ -0,0 +1,54 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.admin_stats() is SECURITY DEFINER and GRANTed to anon. +// The fix revokes EXECUTE on the function from anon. + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkAnonCannotExecute(ctx), + await checkFunctionStillExists(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated anon security-definer fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkAnonCannotExecute(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT has_function_privilege('anon', 'public.admin_stats()', 'EXECUTE') AS can_execute; + `); + const canExecute = rows[0]?.can_execute; + return { + name: 'anon cannot EXECUTE public.admin_stats()', + passed: canExecute === false, + notes: `has_function_privilege(anon, admin_stats, EXECUTE)=${canExecute}`, + }; +} + +async function checkFunctionStillExists(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT proname FROM pg_proc + WHERE proname = 'admin_stats' + AND pronamespace = 'public'::regnamespace; + `); + return { + name: 'function still exists after the fix', + passed: rows.length === 1, + }; +} diff --git a/evals/o11y-0028-0029-resolve-security-anon-security-definer/PROMPT.md b/evals/o11y-0028-0029-resolve-security-anon-security-definer/PROMPT.md new file mode 100644 index 00000000..fb6cf932 --- /dev/null +++ b/evals/o11y-0028-0029-resolve-security-anon-security-definer/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database +topic: + - security + - sql +motivation: apps/docs/content/troubleshooting/database-roles.mdx +--- + +The security advisor flagged that one of our database functions marked +`SECURITY DEFINER` is callable by the `anon` role — meaning unauthenticated API +callers can execute it as the function owner, with elevated privileges, and read +aggregated data they should never see. Can you revoke that access? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-0028-0029-resolve-security-anon-security-definer/remote/project.sql b/evals/o11y-0028-0029-resolve-security-anon-security-definer/remote/project.sql new file mode 100644 index 00000000..3c5d74d5 --- /dev/null +++ b/evals/o11y-0028-0029-resolve-security-anon-security-definer/remote/project.sql @@ -0,0 +1,35 @@ +-- Broken starting state (probe: security-anon-security-definer / Splinter lint 0028). +-- public.admin_stats() is SECURITY DEFINER and GRANTed to anon, allowing +-- unauthenticated API callers to execute it as the function owner. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 100); + +CREATE OR REPLACE FUNCTION public.admin_stats() + RETURNS json + LANGUAGE sql + SECURITY DEFINER + SET search_path = '' + AS $$ + SELECT json_build_object( + 'total_orders', count(*), + 'total_revenue', sum(total_cents) + ) FROM public.orders; + $$; + +GRANT EXECUTE ON FUNCTION public.admin_stats() TO anon; diff --git a/evals/o11y-investigate-api-slow-response/EVAL.ts b/evals/o11y-investigate-api-slow-response/EVAL.ts new file mode 100644 index 00000000..9383b808 --- /dev/null +++ b/evals/o11y-investigate-api-slow-response/EVAL.ts @@ -0,0 +1,47 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: slow-api edge function has a 4s artificial delay. +// Evidence: seeded logs.jsonl showing high execution times. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedLatency = + /slow.?api|4.?s|4000.?ms|execution.{0,20}time|latency|delay/i.test(report); + const identifiedCause = + /settimeout|artificial.{0,20}delay|blocking.{0,20}sleep|await.{0,20}promise/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified the slow-api function latency', passed: identifiedLatency }, + { name: 'identified the artificial delay as cause', passed: identifiedCause }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent reviewed the edge function logs, identified that + slow-api consistently takes 4+ seconds (due to an artificial setTimeout + delay), and proposed removing the delay from the function code. + + Fail if the agent misidentified the cause, blamed the wrong function, or + gave only generic performance advice without identifying the specific delay. + `, + }); + checks.push({ + name: 'correctly diagnosed slow edge function and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-api-slow-response/PROMPT.md b/evals/o11y-investigate-api-slow-response/PROMPT.md new file mode 100644 index 00000000..d00f348f --- /dev/null +++ b/evals/o11y-investigate-api-slow-response/PROMPT.md @@ -0,0 +1,16 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - edge-functions +topic: + - observability +motivation: apps/docs/content/troubleshooting/edge-function-performance.mdx +--- + +Our `slow-api` edge function is consistently taking over 4 seconds to respond — +the logs show execution times well above our 3 second SLA threshold. Can you +review the function logs, identify what's causing the latency, and propose a fix? + +Report what you find. diff --git a/evals/o11y-investigate-api-slow-response/remote/logs.jsonl b/evals/o11y-investigate-api-slow-response/remote/logs.jsonl new file mode 100644 index 00000000..13964d4d --- /dev/null +++ b/evals/o11y-investigate-api-slow-response/remote/logs.jsonl @@ -0,0 +1,5 @@ +{"id":"log-001","ts":"2026-08-01T12:00:01Z","source":"edge-function","level":"info","message":"Function slow-api executed in 4123ms","metadata":{"function_id":"slow-api","execution_id":"exec-001","status":200,"execution_time_ms":4123}} +{"id":"log-002","ts":"2026-08-01T12:01:05Z","source":"edge-function","level":"info","message":"Function slow-api executed in 4089ms","metadata":{"function_id":"slow-api","execution_id":"exec-002","status":200,"execution_time_ms":4089}} +{"id":"log-003","ts":"2026-08-01T12:02:10Z","source":"edge-function","level":"info","message":"Function slow-api executed in 4201ms","metadata":{"function_id":"slow-api","execution_id":"exec-003","status":200,"execution_time_ms":4201}} +{"id":"log-004","ts":"2026-08-01T12:03:15Z","source":"edge-function","level":"info","message":"Function checkout executed in 45ms","metadata":{"function_id":"checkout","execution_id":"exec-004","status":200,"execution_time_ms":45}} +{"id":"log-005","ts":"2026-08-01T12:04:01Z","source":"edge-function","level":"info","message":"Function slow-api executed in 4055ms","metadata":{"function_id":"slow-api","execution_id":"exec-005","status":200,"execution_time_ms":4055}} diff --git a/evals/o11y-investigate-auth-data-integrity/EVAL.ts b/evals/o11y-investigate-auth-data-integrity/EVAL.ts new file mode 100644 index 00000000..06703aa2 --- /dev/null +++ b/evals/o11y-investigate-auth-data-integrity/EVAL.ts @@ -0,0 +1,60 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: a row in auth.users has aud='' and role='' (GoTrue integrity violation). +// The fix updates those fields to 'authenticated' or deletes the broken row. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const noCorruptRows = await checkNoCorruptRows(ctx); + const checks: CheckResult[] = [noCorruptRows]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent identified rows in auth.users with empty aud or role + fields (GoTrue integrity violation) and either: + - Updated those rows to set aud='authenticated' and role='authenticated', OR + - Deleted the corrupted rows. + + Fail if the agent only reported the issue without fixing it, or fixed the + wrong thing. + `, + }); + checks.push({ + name: 'correctly identified and fixed corrupt auth.users rows', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated auth data integrity fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkNoCorruptRows( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT count(*)::int AS n FROM auth.users + WHERE aud = '' OR role = ''; + `); + return { + name: 'no auth.users rows with empty aud or role', + passed: Number(rows[0]?.n ?? 0) === 0, + notes: `corrupt rows remaining: ${rows[0]?.n}`, + }; +} diff --git a/evals/o11y-investigate-auth-data-integrity/PROMPT.md b/evals/o11y-investigate-auth-data-integrity/PROMPT.md new file mode 100644 index 00000000..eb8332b2 --- /dev/null +++ b/evals/o11y-investigate-auth-data-integrity/PROMPT.md @@ -0,0 +1,23 @@ +--- +stage: investigate +suite: other +interface: cli +product: + - auth + - database +topic: + - observability + - sql +services: + - gotrue + - kong + - postgrest +motivation: apps/docs/content/troubleshooting/auth.mdx +--- + +We're seeing silent failures during token refresh for some users — GoTrue +returns a 500 for a subset of accounts but no error is visible in app logs. The +auth integrity check flagged rows in `auth.users` with empty or null required +fields. Can you find the corrupted rows and fix them? + +End your turn with a short summary of what you found and changed. diff --git a/evals/o11y-investigate-auth-data-integrity/local/supabase/config.toml b/evals/o11y-investigate-auth-data-integrity/local/supabase/config.toml new file mode 100644 index 00000000..bc0fe86e --- /dev/null +++ b/evals/o11y-investigate-auth-data-integrity/local/supabase/config.toml @@ -0,0 +1,16 @@ +project_id = "o11y-investigate-auth-data-integrity" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = true diff --git a/evals/o11y-investigate-auth-data-integrity/local/supabase/migrations/0000_corrupt_auth_user.sql b/evals/o11y-investigate-auth-data-integrity/local/supabase/migrations/0000_corrupt_auth_user.sql new file mode 100644 index 00000000..f475a6c6 --- /dev/null +++ b/evals/o11y-investigate-auth-data-integrity/local/supabase/migrations/0000_corrupt_auth_user.sql @@ -0,0 +1,23 @@ +-- Broken starting state (probe: auth-data-integrity). +-- A row in auth.users has aud='' and role='' — fields GoTrue requires to be +-- 'authenticated'. This causes silent token refresh failures for this user. +INSERT INTO auth.users ( + instance_id, id, aud, role, + email, encrypted_password, + created_at, updated_at, + confirmation_token, recovery_token, + email_change_token_new, email_change, + raw_app_meta_data, raw_user_meta_data, + is_super_admin +) VALUES ( + '00000000-0000-0000-0000-000000000000', + gen_random_uuid(), + '', -- empty aud — GoTrue integrity violation + '', -- empty role — GoTrue integrity violation + 'chaos_corrupt_user@example.com', + '', + now(), now(), + '', '', '', '', + '{}', '{}', + false +); diff --git a/evals/o11y-investigate-auth-leaked-jwt/EVAL.ts b/evals/o11y-investigate-auth-leaked-jwt/EVAL.ts new file mode 100644 index 00000000..e0a57d44 --- /dev/null +++ b/evals/o11y-investigate-auth-leaked-jwt/EVAL.ts @@ -0,0 +1,53 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: edge function logs a JWT-shaped token in plaintext. +// Evidence: seeded logs.jsonl with function log line containing eyJ... token. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedJwtLeak = + /jwt|token.{0,20}log|log.{0,20}token|eyJ|secret.{0,20}log|credentials?.{0,20}log/i.test(report); + const identifiedFunction = + /auth.?leak.?demo|edge.{0,20}function/i.test(report); + const proposedFix = + /remove.{0,20}log|console\.(log|error)|logging.{0,20}statement|redact/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified JWT token being leaked in logs', passed: identifiedJwtLeak }, + { name: 'identified the edge function responsible', passed: identifiedFunction }, + { name: 'proposed removing the logging statement', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Reviewed the edge function logs and found a JWT-shaped token (eyJ...) + being logged in plaintext by the auth-leak-demo function. + 2. Confirmed this is a security issue (credentials must not appear in logs). + 3. Proposed removing or redacting the console.log statement from the + function code. + + Fail if the agent dismissed the finding, misidentified the function, or + gave only generic security advice without identifying the specific log line. + `, + }); + checks.push({ + name: 'correctly diagnosed JWT leak and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-auth-leaked-jwt/PROMPT.md b/evals/o11y-investigate-auth-leaked-jwt/PROMPT.md new file mode 100644 index 00000000..da28baf4 --- /dev/null +++ b/evals/o11y-investigate-auth-leaked-jwt/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - edge-functions + - auth +topic: + - security + - observability +motivation: apps/docs/content/troubleshooting/auth.mdx +--- + +A developer noticed that one of our edge function logs is printing what looks +like a JWT token to stdout. This is a serious security concern — JWTs are +credentials and should never appear in logs. Can you review the function logs, +confirm whether a token is being leaked, and tell us what to do? + +Report what you find. diff --git a/evals/o11y-investigate-auth-leaked-jwt/remote/logs.jsonl b/evals/o11y-investigate-auth-leaked-jwt/remote/logs.jsonl new file mode 100644 index 00000000..27cebf49 --- /dev/null +++ b/evals/o11y-investigate-auth-leaked-jwt/remote/logs.jsonl @@ -0,0 +1,4 @@ +{"id":"log-001","ts":"2026-08-01T09:00:01Z","source":"edge-function","level":"info","message":"[auth-leak-demo] processing request, token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6ImNoYW9zLXRlc3QiLCJpYXQiOjE1MTYyMzkwMjJ9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c","metadata":{"function_id":"auth-leak-demo","execution_id":"exec-abc123","region":"us-east-1"}} +{"id":"log-002","ts":"2026-08-01T09:01:15Z","source":"edge-function","level":"info","message":"[auth-leak-demo] processing request, token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6ImNoYW9zLXRlc3QiLCJpYXQiOjE1MTYyMzkwMjJ9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c","metadata":{"function_id":"auth-leak-demo","execution_id":"exec-def456","region":"us-east-1"}} +{"id":"log-003","ts":"2026-08-01T09:02:30Z","source":"edge-function","level":"info","message":"[checkout] request processed successfully","metadata":{"function_id":"checkout","execution_id":"exec-ghi789","status":200}} +{"id":"log-004","ts":"2026-08-01T09:03:00Z","source":"edge-function","level":"info","message":"[auth-leak-demo] processing request, token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6ImNoYW9zLXRlc3QiLCJpYXQiOjE1MTYyMzkwMjJ9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c","metadata":{"function_id":"auth-leak-demo","execution_id":"exec-jkl012","region":"us-east-1"}} diff --git a/evals/o11y-investigate-auth-rate-limit-bypass/EVAL.ts b/evals/o11y-investigate-auth-rate-limit-bypass/EVAL.ts new file mode 100644 index 00000000..9118478e --- /dev/null +++ b/evals/o11y-investigate-auth-rate-limit-bypass/EVAL.ts @@ -0,0 +1,50 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: burst of failed auth attempts seeded in auth.audit_log_entries. +// This is an investigate eval — agent diagnoses the attack pattern and proposes +// rate-limiting / captcha configuration. + +const scorer: LocalStackScorer = async (ctx) => { + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + const report = ctx.agentReport ?? ''; + + const inspectedLogs = + /audit_log|auth.log|sign.{0,5}in|failed.{0,20}attempt|invalid.{0,20}creden/i.test(transcript); + const identifiedBurst = + /burst|repeated|brute.?force|high.{0,20}rate|same.{0,20}ip/i.test(report); + + const checks: CheckResult[] = [ + { name: 'inspected auth audit logs', passed: inspectedLogs }, + { name: 'identified repeated failed attempts / brute force', passed: identifiedBurst }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Queried auth.audit_log_entries and identified the burst of failed login + attempts from the same IP range. + 2. Proposed concrete configuration changes — such as enabling CAPTCHA, + lowering rate limits, or enabling IP-based throttling in GoTrue config. + + Fail if the agent did not inspect the auth logs, misidentified the issue, + or gave only vague recommendations without specifics. + `, + }); + checks.push({ + name: 'correctly diagnosed rate-limit bypass and proposed config fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-auth-rate-limit-bypass/PROMPT.md b/evals/o11y-investigate-auth-rate-limit-bypass/PROMPT.md new file mode 100644 index 00000000..18aab704 --- /dev/null +++ b/evals/o11y-investigate-auth-rate-limit-bypass/PROMPT.md @@ -0,0 +1,23 @@ +--- +stage: investigate +suite: other +interface: cli +product: + - auth +topic: + - security + - observability +services: + - gotrue + - kong + - postgrest +motivation: apps/docs/content/troubleshooting/auth.mdx +--- + +Our auth logs show a burst of failed sign-in attempts from the same IP range +over the past 10 minutes — it looks like someone is brute-forcing login +credentials. We don't have account lockout configured. Can you review the auth +audit log, confirm the attack pattern, and tell us what configuration changes +would prevent this? + +Report what you find. diff --git a/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/config.toml b/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/config.toml new file mode 100644 index 00000000..2fa2d582 --- /dev/null +++ b/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/config.toml @@ -0,0 +1,16 @@ +project_id = "o11y-investigate-auth-rate-limit-bypass" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = true diff --git a/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/migrations/0000_auth_audit_burst.sql b/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/migrations/0000_auth_audit_burst.sql new file mode 100644 index 00000000..aa97b097 --- /dev/null +++ b/evals/o11y-investigate-auth-rate-limit-bypass/local/supabase/migrations/0000_auth_audit_burst.sql @@ -0,0 +1,22 @@ +-- Starting state (probe: auth-rate-limit-bypass). +-- Seeded auth.audit_log_entries with a burst of failed login attempts from +-- the same IP range to simulate a brute-force attack pattern. +DO $$ +DECLARE i int; +BEGIN + FOR i IN 1..50 LOOP + INSERT INTO auth.audit_log_entries (id, payload, created_at, ip_address) + VALUES ( + gen_random_uuid(), + jsonb_build_object( + 'action', 'login', + 'actor_via_sso', false, + 'log_type', 'account', + 'traits', jsonb_build_object('provider', 'email'), + 'error', 'Invalid login credentials' + ), + now() - (random() * interval '10 minutes'), + '203.0.113.' || (floor(random()*10)+1)::text + ); + END LOOP; +END $$; diff --git a/evals/o11y-investigate-cost-n-plus-one/EVAL.ts b/evals/o11y-investigate-cost-n-plus-one/EVAL.ts new file mode 100644 index 00000000..c9fa0335 --- /dev/null +++ b/evals/o11y-investigate-cost-n-plus-one/EVAL.ts @@ -0,0 +1,74 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// This fault has no schema-level fix — it's an application query pattern (N+1). +// So scoring is report-based: the agent must identify the repeated single-row +// query against orders as the driver and propose a set-based replacement. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + + const namedOrders = /\borders\b/i.test(report); + const identifiedPattern = + /n\s*\+\s*1/i.test(report) || + /(repeated|per[- ]?customer|one (row|query) (at a time|per)|thousands of|millions of|high call count)/i.test( + report + ); + const proposedBatching = + /\bjoin\b/i.test(report) || + /\bin\s*\(/i.test(report) || + /\bany\s*\(/i.test(report) || + /(batch|single query|set[- ]based|distinct on|group by|window)/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'named the orders table as the driver', passed: namedOrders }, + { + name: 'identified the repeated single-row (N+1) pattern', + passed: identifiedPattern, + }, + { + name: 'inspected query statistics', + passed: /pg_stat_statements|query performance|query insights/i.test( + serializeTranscript(ctx.transcript, { includeToolCallInputs: true }) + ), + }, + { name: 'proposed a set-based / batched fix', passed: proposedBatching }, + ]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent identifies that the database load is driven by a + repeated single-row query against the orders table (an N+1 pattern — + the same "WHERE customer_id = $1 ... LIMIT 1" query executed an enormous + number of times) AND proposes a concrete set-based replacement, such as a + single JOIN, a WHERE customer_id = ANY(...) / IN (...) batch, or a + DISTINCT ON / window query that fetches all customers' latest orders in + one round trip. + + Fail if it blames the wrong query, recommends only an index without + addressing the per-row call pattern, or gives vague advice. + `, + }); + + checks.push({ + name: 'verified N+1 diagnosis and batched fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { + passed: namedOrders && identifiedPattern && proposedBatching && verdict.passed, + checks, + }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-cost-n-plus-one/PROMPT.md b/evals/o11y-investigate-cost-n-plus-one/PROMPT.md new file mode 100644 index 00000000..252ce836 --- /dev/null +++ b/evals/o11y-investigate-cost-n-plus-one/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/high-egress-or-compute.mdx +--- + +Our compute usage and egress have been climbing and I can't explain why — the +traffic hasn't really grown. The customer list page feels chatty when it loads. +Can you dig into what's driving the database load and tell me what's going on and +what we should change? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-cost-n-plus-one/remote/project.sql b/evals/o11y-investigate-cost-n-plus-one/remote/project.sql new file mode 100644 index 00000000..fcc08cc1 --- /dev/null +++ b/evals/o11y-investigate-cost-n-plus-one/remote/project.sql @@ -0,0 +1,51 @@ +-- Starting state (probe: cost-n-plus-one). +-- The app fetches each customer's latest order one row at a time (an N+1 +-- pattern) instead of a single set-based query. The evidence lives in query +-- statistics: thousands of near-identical single-row selects against orders. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 500) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 5000); + +-- Eval-scoped compatibility table for Supabase's Query Performance report, +-- which is backed by pg_stat_statements in real projects. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- The N+1 offender: called once per customer, per page load, for weeks. + (1001, + 'SELECT id, total_cents, created_at FROM orders WHERE customer_id = $1 ORDER BY created_at DESC LIMIT 1', + 1840233, 1840233, 5153000.0, 2.8, 41.2, 9100000, 220000), + -- A handful of normal, low-frequency queries for contrast. + (1002, 'SELECT id, email FROM customers WHERE id = $1', 5120, 5120, 610.0, 0.12, 3.1, 41000, 900), + (1003, 'INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2)', 4990, 4990, 980.0, 0.20, 6.0, 30000, 700), + (1004, 'SELECT count(*) FROM customers', 88, 88, 44.0, 0.5, 2.0, 900, 10); diff --git a/evals/o11y-investigate-performance-edge-fn-slow/EVAL.ts b/evals/o11y-investigate-performance-edge-fn-slow/EVAL.ts new file mode 100644 index 00000000..aae7b92c --- /dev/null +++ b/evals/o11y-investigate-performance-edge-fn-slow/EVAL.ts @@ -0,0 +1,45 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: 'report' edge function has a 4s blocking delay (setTimeout). +// Evidence: seeded logs.jsonl showing consistently high execution times. + +const scorer: ToolScorer = async (ctx) => { + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + const report = ctx.agentReport ?? ''; + + const identifiedFunction = + /\breport\b.{0,30}(slow|latency|delay|4s|4000)/i.test(report) || + /(slow|latency|delay|4s|4000).{0,30}\breport\b/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified the report function as slow', passed: identifiedFunction }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified the 'report' edge function as consistently + slow (4+ seconds per invocation based on logs) and proposed removing the + blocking delay (setTimeout or similar) from the function code. + + Fail if the agent blamed the wrong function, gave only generic advice, or + did not propose a specific code fix. + `, + }); + checks.push({ + name: 'correctly diagnosed slow edge function', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-performance-edge-fn-slow/PROMPT.md b/evals/o11y-investigate-performance-edge-fn-slow/PROMPT.md new file mode 100644 index 00000000..576f776b --- /dev/null +++ b/evals/o11y-investigate-performance-edge-fn-slow/PROMPT.md @@ -0,0 +1,16 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - edge-functions +topic: + - observability +motivation: apps/docs/content/troubleshooting/edge-function-performance.mdx +--- + +Our `report` edge function is slow — every request takes about 4 seconds and +we can't figure out why. The function is supposed to return a simple JSON +payload. Can you look at the logs and identify what's causing the latency? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-performance-edge-fn-slow/remote/logs.jsonl b/evals/o11y-investigate-performance-edge-fn-slow/remote/logs.jsonl new file mode 100644 index 00000000..52e723e2 --- /dev/null +++ b/evals/o11y-investigate-performance-edge-fn-slow/remote/logs.jsonl @@ -0,0 +1,4 @@ +{"id":"log-001","ts":"2026-08-01T14:00:01Z","source":"edge-function","level":"info","message":"Function report executed in 4012ms","metadata":{"function_id":"report","execution_id":"exec-001","status":200,"execution_time_ms":4012}} +{"id":"log-002","ts":"2026-08-01T14:01:05Z","source":"edge-function","level":"info","message":"Function report executed in 4078ms","metadata":{"function_id":"report","execution_id":"exec-002","status":200,"execution_time_ms":4078}} +{"id":"log-003","ts":"2026-08-01T14:02:10Z","source":"edge-function","level":"info","message":"Function report executed in 4055ms","metadata":{"function_id":"report","execution_id":"exec-003","status":200,"execution_time_ms":4055}} +{"id":"log-004","ts":"2026-08-01T14:03:00Z","source":"edge-function","level":"info","message":"Function checkout executed in 38ms","metadata":{"function_id":"checkout","execution_id":"exec-004","status":200,"execution_time_ms":38}} diff --git a/evals/o11y-investigate-postgres-autovacuum-disabled/EVAL.ts b/evals/o11y-investigate-postgres-autovacuum-disabled/EVAL.ts new file mode 100644 index 00000000..c0079b08 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-disabled/EVAL.ts @@ -0,0 +1,62 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: autovacuum_enabled=false on public.orders, causing unbounded dead-tuple +// accumulation. The fix re-enables autovacuum and runs VACUUM ANALYZE. + +const scorer: ToolScorer = async (ctx) => { + try { + const autovacuumFixed = await checkAutovacuumReEnabled(ctx); + const checks: CheckResult[] = [autovacuumFixed]; + + const verdict = await judge({ + input: serializeTranscript(ctx.transcript, { includeToolCallInputs: true }), + rubric: stripIndent` + Pass if the agent identified that autovacuum is disabled at the table level + on public.orders (via pg_class reloptions or pg_stat_user_tables), and + proposed re-enabling it with ALTER TABLE public.orders RESET (autovacuum_enabled) + followed by VACUUM ANALYZE to reclaim existing dead tuples. + + Fail if the agent only reported the issue without proposing a specific fix. + `, + }); + checks.push({ + name: 'correctly diagnosed autovacuum disabled and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated autovacuum fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkAutovacuumReEnabled(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(` + SELECT reloptions FROM pg_class WHERE relname = 'orders' + AND relnamespace = 'public'::regnamespace; + `); + const opts = rows[0]?.reloptions; + const optsArr = Array.isArray(opts) ? opts : (opts ? [opts] : []); + const disabled = optsArr.some((o: string) => + /autovacuum_enabled\s*=\s*(false|off|0)/i.test(o) + ); + return { + name: 'autovacuum re-enabled on public.orders', + passed: !disabled, + notes: `reloptions: ${JSON.stringify(optsArr)}`, + }; +} diff --git a/evals/o11y-investigate-postgres-autovacuum-disabled/PROMPT.md b/evals/o11y-investigate-postgres-autovacuum-disabled/PROMPT.md new file mode 100644 index 00000000..9024e78d --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-disabled/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/autovacuum.mdx +--- + +We're seeing growing dead-tuple counts on our `orders` table and table bloat is +increasing even though we have autovacuum enabled globally. The bloat advisor is +firing but autovacuum isn't cleaning it up. Can you check whether autovacuum is +disabled at the table level and propose a fix? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-postgres-autovacuum-disabled/remote/project.sql b/evals/o11y-investigate-postgres-autovacuum-disabled/remote/project.sql new file mode 100644 index 00000000..6f692a56 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-disabled/remote/project.sql @@ -0,0 +1,41 @@ +-- Starting state (probe: postgres-autovacuum-disabled). +-- autovacuum_enabled=false on public.orders prevents dead-tuple cleanup, +-- causing unbounded table bloat. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +ALTER TABLE public.orders SET (autovacuum_enabled = false); + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (g % 500) + 1, (random() * 10000)::int +FROM generate_series(1, 5000) g; + +DELETE FROM public.orders WHERE id IN (SELECT id FROM public.orders ORDER BY id LIMIT 4000); + +-- Fake pg_stat_user_tables showing dead tuples. +CREATE TABLE pg_stat_user_tables_snapshot ( + schemaname name, + relname name, + n_live_tup bigint, + n_dead_tup bigint, + last_autovacuum timestamptz, + last_autoanalyze timestamptz +); + +INSERT INTO pg_stat_user_tables_snapshot + (schemaname, relname, n_live_tup, n_dead_tup, last_autovacuum, last_autoanalyze) +VALUES + ('public', 'orders', 1000, 24000, NULL, NULL), + ('public', 'customers', 500, 5, now() - '1 hour'::interval, now() - '30 minutes'::interval); diff --git a/evals/o11y-investigate-postgres-connection-saturation/EVAL.ts b/evals/o11y-investigate-postgres-connection-saturation/EVAL.ts new file mode 100644 index 00000000..c32b98e3 --- /dev/null +++ b/evals/o11y-investigate-postgres-connection-saturation/EVAL.ts @@ -0,0 +1,52 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: 90 pg_cron jobs hold idle connections via pg_sleep(120), saturating +// the connection pool. This scenario cannot be faithfully reproduced in the +// harness (no concurrent sessions) — eval uses faked pg_stat_activity. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedSaturation = + /connection.{0,20}(saturat|limit|pool|exhaust|too many)/i.test(report); + const identifiedCause = + /pg_sleep|saturation|idle.{0,20}connection|pg_cron|cron.{0,20}job/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified connection saturation', passed: identifiedSaturation }, + { name: 'identified idle connections / pg_cron as cause', passed: identifiedCause }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified that the connection pool is saturated by a + large number of idle connections held by pg_cron jobs running pg_sleep, + and proposed fixing this by: unscheduling the saturation cron jobs, + terminating idle sessions, or enabling PgBouncer/Supavisor connection + pooling. + + Fail if the agent misidentified the cause or gave only generic advice. + + Note: this scenario is seeded as a faked pg_stat_activity snapshot. + The agent cannot fully remediate it in this harness environment. + `, + }); + checks.push({ + name: 'correctly diagnosed connection saturation', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-connection-saturation/PROMPT.md b/evals/o11y-investigate-postgres-connection-saturation/PROMPT.md new file mode 100644 index 00000000..0b62720f --- /dev/null +++ b/evals/o11y-investigate-postgres-connection-saturation/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/database-design.mdx +--- + +Our application is throwing "too many connections" errors under load. The Postgres +connection pool appears saturated and new connections are being refused. Can you +investigate the connection usage and tell us what's holding all the connections +and what we should do? + +Report what you find and propose a fix. + +> Note: This scenario cannot be fully reproduced in the eval harness (concurrent +> sessions are not possible), but the seeded pg_stat_activity snapshot reflects +> the saturation state. diff --git a/evals/o11y-investigate-postgres-connection-saturation/remote/project.sql b/evals/o11y-investigate-postgres-connection-saturation/remote/project.sql new file mode 100644 index 00000000..b6c4f39b --- /dev/null +++ b/evals/o11y-investigate-postgres-connection-saturation/remote/project.sql @@ -0,0 +1,41 @@ +-- Starting state (probe: postgres-connection-saturation). +-- HARNESS NOTE: Cannot produce real concurrent sessions in the eval harness. +-- Faked pg_stat_activity snapshot showing 90 idle connections from pg_cron. +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL DEFAULT 1, + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +-- Faked pg_stat_activity snapshot showing connection saturation. +CREATE TABLE IF NOT EXISTS pg_stat_activity_snapshot ( + pid int PRIMARY KEY, + application_name text, + state text, + query text, + wait_event_type text, + wait_event text, + query_start timestamptz, + backend_start timestamptz +); + +INSERT INTO pg_stat_activity_snapshot + (pid, application_name, state, query, wait_event_type, wait_event, query_start, backend_start) +SELECT + 50000 + g, + 'pg_cron', + 'idle in transaction', + 'SELECT pg_sleep(120)', + 'Timeout', + 'pg_sleep', + now() - ((random() * 60) || ' seconds')::interval, + now() - '2 minutes'::interval +FROM generate_series(1, 90) g; + +-- Remaining connections: very few left. +INSERT INTO pg_stat_activity_snapshot + (pid, application_name, state, query, wait_event_type, query_start, backend_start) +VALUES + (1001, 'PostgREST', 'active', 'SELECT * FROM orders LIMIT 10', NULL, now(), now() - '1 second'::interval), + (1002, 'supabase-pooler', 'idle', '', 'Client', now(), now() - '30 seconds'::interval); diff --git a/evals/o11y-investigate-postgres-index-bloat/EVAL.ts b/evals/o11y-investigate-postgres-index-bloat/EVAL.ts new file mode 100644 index 00000000..c7a13ac2 --- /dev/null +++ b/evals/o11y-investigate-postgres-index-bloat/EVAL.ts @@ -0,0 +1,48 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: repeated UPDATE operations on orders caused btree index dead pages. +// This is an investigate eval — agent diagnoses the bloat and proposes REINDEX. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedIndexBloat = + /index.{0,30}(bloat|dead|page|bloated)|reindex|btree.{0,20}(dead|bloat)/i.test(report); + const proposedReindex = + /reindex|rebuild.{0,20}index|vacuum.{0,20}analyze/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified index bloat', passed: identifiedIndexBloat }, + { name: 'proposed REINDEX or equivalent', passed: proposedReindex }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identifies index bloat on the orders table + (specifically the customer_id index accumulating dead pages from repeated + UPDATE churn) and proposes REINDEX INDEX CONCURRENTLY or equivalent to + compact it, along with VACUUM ANALYZE to prevent recurrence. + + Fail if the agent misidentifies the cause, proposes only a new index + without addressing bloat, or gives vague observations without a fix. + `, + }); + checks.push({ + name: 'correctly diagnosed index bloat and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-index-bloat/PROMPT.md b/evals/o11y-investigate-postgres-index-bloat/PROMPT.md new file mode 100644 index 00000000..30c72701 --- /dev/null +++ b/evals/o11y-investigate-postgres-index-bloat/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our DBA noticed that the index on `orders.customer_id` has grown much larger +than expected relative to the number of live rows. We suspect index bloat from +heavy UPDATE activity without proper vacuuming. Can you investigate the index +health and tell me what's going on and what should be done? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-postgres-index-bloat/remote/project.sql b/evals/o11y-investigate-postgres-index-bloat/remote/project.sql new file mode 100644 index 00000000..cf5a499b --- /dev/null +++ b/evals/o11y-investigate-postgres-index-bloat/remote/project.sql @@ -0,0 +1,63 @@ +-- Starting state (probe: postgres-index-bloat). +-- Repeated UPDATE operations on orders without VACUUM have caused the btree +-- index on customer_id to accumulate dead pages. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +CREATE INDEX idx_orders_customer_id ON public.orders(customer_id); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (g % 500) + 1, (random() * 10000)::int +FROM generate_series(1, 10000) g; + +-- Repeatedly flip values to generate index churn (dead pages accumulate). +DO $$ +DECLARE i int; +BEGIN + FOR i IN 1..20 LOOP + UPDATE public.orders SET customer_id = ((customer_id % 500) + 1) WHERE id % 2 = 0; + UPDATE public.orders SET customer_id = ((customer_id % 500) + 1) WHERE id % 2 = 1; + END LOOP; +END $$; +ANALYZE public.orders; + +-- Faked index scan stats (named to avoid collision with pg_stat_user_indexes system view). +CREATE TABLE public.index_scan_stats ( + schemaname name, + relname name, + indexrelname name, + idx_scan bigint, + idx_tup_read bigint, + idx_tup_fetch bigint +); + +INSERT INTO public.index_scan_stats + (schemaname, relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch) +VALUES + ('public', 'orders', 'idx_orders_customer_id', 42100, 210000, 198000); + +-- Faked index block stats showing high read ratio (bloat indicator). +CREATE TABLE public.index_block_stats ( + schemaname name, + relname name, + indexrelname name, + idx_blks_read bigint, + idx_blks_hit bigint +); + +INSERT INTO public.index_block_stats + (schemaname, relname, indexrelname, idx_blks_read, idx_blks_hit) +VALUES + ('public', 'orders', 'idx_orders_customer_id', 85000, 12000); diff --git a/evals/o11y-investigate-postgres-long-running-query/EVAL.ts b/evals/o11y-investigate-postgres-long-running-query/EVAL.ts new file mode 100644 index 00000000..95d5c169 --- /dev/null +++ b/evals/o11y-investigate-postgres-long-running-query/EVAL.ts @@ -0,0 +1,48 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: chaos_long_sleep() calls pg_sleep(120) periodically via pg_cron. +// Evidence: seeded logs.jsonl with postgres log lines showing the long-running query. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedSleepQuery = + /pg_sleep|chaos_long_sleep|long.{0,20}(running|query|transaction)|sleep.*120/i.test(report); + const proposedFix = + /unschedule|cron\.unschedule|drop.{0,20}function|pg_terminate_backend|terminate.*session/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified long-running sleep query', passed: identifiedSleepQuery }, + { name: 'proposed concrete fix', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identifies a recurring long-running query (chaos_long_sleep + or pg_sleep) appearing in the Postgres logs / pg_stat_activity and proposes: + - Unscheduling the pg_cron job, AND/OR + - Terminating active sessions running the query, AND/OR + - Dropping the function causing the load. + + Fail if the agent misidentifies the root cause or gives only vague advice. + `, + }); + checks.push({ + name: 'correctly diagnosed long-running query and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-long-running-query/PROMPT.md b/evals/o11y-investigate-postgres-long-running-query/PROMPT.md new file mode 100644 index 00000000..fe3ab70f --- /dev/null +++ b/evals/o11y-investigate-postgres-long-running-query/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/timeouts.mdx +--- + +Our Postgres logs show a recurring query that has been running for well over a +minute — it appears in `pg_stat_activity` as active and is holding resources. +The logs suggest it's related to a sleep or long-running function. Can you +review the logs and activity, identify what's causing it, and tell us how to +stop it and prevent recurrence? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-postgres-long-running-query/remote/logs.jsonl b/evals/o11y-investigate-postgres-long-running-query/remote/logs.jsonl new file mode 100644 index 00000000..1e4f1922 --- /dev/null +++ b/evals/o11y-investigate-postgres-long-running-query/remote/logs.jsonl @@ -0,0 +1,6 @@ +{"id":"log-001","ts":"2026-08-01T10:00:01Z","source":"postgres","level":"info","message":"duration: 120142.321 ms statement: SELECT public.chaos_long_sleep()","metadata":{"pid":12345,"application_name":"pg_cron","state":"idle in transaction"}} +{"id":"log-002","ts":"2026-08-01T10:03:02Z","source":"postgres","level":"info","message":"duration: 120089.442 ms statement: SELECT public.chaos_long_sleep()","metadata":{"pid":12346,"application_name":"pg_cron","state":"idle in transaction"}} +{"id":"log-003","ts":"2026-08-01T10:06:01Z","source":"postgres","level":"info","message":"duration: 120211.003 ms statement: SELECT public.chaos_long_sleep()","metadata":{"pid":12347,"application_name":"pg_cron","state":"idle in transaction"}} +{"id":"log-004","ts":"2026-08-01T10:09:05Z","source":"postgres","level":"warning","message":"process 12348 still running after 120000 ms: SELECT public.chaos_long_sleep()","metadata":{"pid":12348,"state":"active","wait_event_type":"Timeout","query_start":"2026-08-01T10:07:05Z"}} +{"id":"log-005","ts":"2026-08-01T10:00:00Z","source":"postgres","level":"info","message":"duration: 0.312 ms statement: SELECT id, email FROM customers WHERE id = $1","metadata":{"pid":9001,"application_name":"PostgREST"}} +{"id":"log-006","ts":"2026-08-01T10:01:00Z","source":"postgres","level":"info","message":"duration: 0.445 ms statement: INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2)","metadata":{"pid":9002,"application_name":"PostgREST"}} diff --git a/evals/o11y-investigate-postgres-slow-query/EVAL.ts b/evals/o11y-investigate-postgres-slow-query/EVAL.ts new file mode 100644 index 00000000..c8cfca91 --- /dev/null +++ b/evals/o11y-investigate-postgres-slow-query/EVAL.ts @@ -0,0 +1,52 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: chaos_slow_scan() runs a full seq scan + random sort on public.orders +// periodically, appearing in pg_stat_statements as a consistently slow query. +// This is an investigate eval — the agent diagnoses from faked stats and proposes a fix. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const namedSlowQuery = + /chaos_slow_scan|slow.?scan|sequential scan|seq.?scan/i.test(report); + const identifiedOrders = /\borders\b/i.test(report); + const inspectedStats = + /pg_stat_statements|query performance|query insights/i.test(transcript); + + const checks: CheckResult[] = [ + { name: 'identified the slow-scanning query', passed: namedSlowQuery }, + { name: 'identified orders as the affected table', passed: identifiedOrders }, + { name: 'inspected query statistics', passed: inspectedStats }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identifies that there is a slow query doing a full + sequential scan on the orders table (chaos_slow_scan or similar pattern + with high total execution time and random sort), and proposes a concrete + remediation — such as removing the function/cron job, adding an index, + or rewriting the query. + + Fail if the agent misidentifies the cause, gives only vague observations, + or does not propose any actionable fix. + `, + }); + checks.push({ + name: 'correctly diagnosed slow query and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-slow-query/PROMPT.md b/evals/o11y-investigate-postgres-slow-query/PROMPT.md new file mode 100644 index 00000000..2c4a2026 --- /dev/null +++ b/evals/o11y-investigate-postgres-slow-query/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our database CPU is spiking every few minutes even when user traffic is flat. +The query performance dashboard shows one query with a consistently high +execution time and it looks like it's scanning the entire orders table. Can you +dig into the query statistics and tell me what's causing the spikes and what +should be done about it? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-postgres-slow-query/remote/project.sql b/evals/o11y-investigate-postgres-slow-query/remote/project.sql new file mode 100644 index 00000000..9dc25053 --- /dev/null +++ b/evals/o11y-investigate-postgres-slow-query/remote/project.sql @@ -0,0 +1,48 @@ +-- Starting state (probe: postgres-slow-query). +-- A chaos function runs a full sequential scan + random sort on orders periodically. +-- The evidence lives in faked query statistics. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 500) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 10000); + +-- Faked pg_stat_statements showing chaos_slow_scan as a consistently slow query. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- chaos_slow_scan: full seq scan + ORDER BY random() every 3 minutes. + (2001, + 'SELECT count(*) FROM public.orders ORDER BY random()', + 142, 142, 2840000.0, 20000.0, 24500.0, 0, 820000), + -- Normal queries for contrast. + (2002, 'SELECT id, total_cents FROM orders WHERE customer_id = $1', 95120, 95120, 1904.0, 0.02, 0.8, 480000, 1200), + (2003, 'INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2)', 8800, 8800, 528.0, 0.06, 1.2, 44000, 800); diff --git a/evals/o11y-investigate-postgres-temp-file-spill/EVAL.ts b/evals/o11y-investigate-postgres-temp-file-spill/EVAL.ts new file mode 100644 index 00000000..95ad5c37 --- /dev/null +++ b/evals/o11y-investigate-postgres-temp-file-spill/EVAL.ts @@ -0,0 +1,48 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: a query sorts orders by (customer_id, total_cents DESC) with low +// work_mem, spilling to disk. Evidence in faked pg_stat_statements temp_blks_written. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedTempSpill = + /temp.{0,20}(file|spill|blk|disk)|work_mem|sort.{0,20}disk|disk.{0,20}sort/i.test(report); + const inspectedStats = + /pg_stat_statements|temp_blks|query performance/i.test(transcript); + + const checks: CheckResult[] = [ + { name: 'identified temp file / disk spill', passed: identifiedTempSpill }, + { name: 'inspected query statistics', passed: inspectedStats }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identifies that a query is spilling sort data to disk + (evidenced by temp_blks_written in pg_stat_statements) and proposes a + concrete fix — such as increasing work_mem, adding an index to avoid the + sort, or removing the problematic query/function. + + Fail if the agent misidentifies the cause, gives only vague observations, + or fails to propose any actionable fix. + `, + }); + checks.push({ + name: 'correctly diagnosed temp spill and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-temp-file-spill/PROMPT.md b/evals/o11y-investigate-postgres-temp-file-spill/PROMPT.md new file mode 100644 index 00000000..af4b8a4d --- /dev/null +++ b/evals/o11y-investigate-postgres-temp-file-spill/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +We're seeing disk I/O spikes in our Postgres metrics and occasional slowdowns +that don't match CPU usage. Someone mentioned temp file usage could be the +culprit. Can you look at the Postgres logs and query statistics and tell me +which query is spilling to disk, why it's happening, and what we should do +about it? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-postgres-temp-file-spill/remote/logs.jsonl b/evals/o11y-investigate-postgres-temp-file-spill/remote/logs.jsonl new file mode 100644 index 00000000..de8b27e6 --- /dev/null +++ b/evals/o11y-investigate-postgres-temp-file-spill/remote/logs.jsonl @@ -0,0 +1,6 @@ +{"id":"log-001","ts":"2026-08-01T14:22:03Z","source":"postgres","level":"info","message":"temporary file: path \"base/pgsql_tmp/pgsql_tmp12345.0\", size 131072000","metadata":{"pid":14201,"application_name":"PostgREST"}} +{"id":"log-002","ts":"2026-08-01T14:22:03Z","source":"postgres","level":"info","message":"duration: 4821.332 ms statement: SELECT customer_id FROM public.orders ORDER BY customer_id, total_cents DESC","metadata":{"pid":14201,"application_name":"PostgREST"}} +{"id":"log-003","ts":"2026-08-01T14:23:17Z","source":"postgres","level":"info","message":"temporary file: path \"base/pgsql_tmp/pgsql_tmp12346.0\", size 131072000","metadata":{"pid":14202,"application_name":"PostgREST"}} +{"id":"log-004","ts":"2026-08-01T14:23:17Z","source":"postgres","level":"info","message":"duration: 5103.881 ms statement: SELECT customer_id FROM public.orders ORDER BY customer_id, total_cents DESC","metadata":{"pid":14202,"application_name":"PostgREST"}} +{"id":"log-005","ts":"2026-08-01T14:24:55Z","source":"postgres","level":"info","message":"temporary file: path \"base/pgsql_tmp/pgsql_tmp12347.0\", size 131072000","metadata":{"pid":14203,"application_name":"PostgREST"}} +{"id":"log-006","ts":"2026-08-01T14:24:55Z","source":"postgres","level":"info","message":"duration: 4967.114 ms statement: SELECT customer_id FROM public.orders ORDER BY customer_id, total_cents DESC","metadata":{"pid":14203,"application_name":"PostgREST"}} diff --git a/evals/o11y-investigate-postgres-temp-file-spill/remote/project.sql b/evals/o11y-investigate-postgres-temp-file-spill/remote/project.sql new file mode 100644 index 00000000..07bd4fd9 --- /dev/null +++ b/evals/o11y-investigate-postgres-temp-file-spill/remote/project.sql @@ -0,0 +1,21 @@ +-- Starting state (probe: postgres-temp-file-spill). +-- A sort query on orders spills to disk repeatedly. Evidence is in logs.jsonl +-- (Postgres logs temp file creation when log_temp_files=0). +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 500) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 10000); diff --git a/evals/o11y-investigate-postgres-wal-slot-inactive/EVAL.ts b/evals/o11y-investigate-postgres-wal-slot-inactive/EVAL.ts new file mode 100644 index 00000000..9a40f90a --- /dev/null +++ b/evals/o11y-investigate-postgres-wal-slot-inactive/EVAL.ts @@ -0,0 +1,48 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: an inactive logical replication slot (chaos_slot) holds WAL, +// preventing VACUUM from reclaiming dead tuples. +// BLOCKED: pg_replication_slots not available in PGlite — eval uses faked table. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedSlot = + /replication.{0,20}slot|chaos_slot|wal.{0,20}(hold|accum|grow)/i.test(report); + const proposedFix = + /drop.{0,20}slot|pg_drop_replication_slot|reconnect.{0,20}consumer/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified inactive replication slot', passed: identifiedSlot }, + { name: 'proposed dropping or reconnecting the slot', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified an inactive logical replication slot (chaos_slot) + that is holding WAL and preventing VACUUM from reclaiming space, and proposed + either dropping the slot with pg_drop_replication_slot or reconnecting the + consumer that owns it. + + Fail if the agent misidentified the cause or gave only generic disk-space advice. + `, + }); + checks.push({ + name: 'correctly diagnosed inactive WAL slot', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-wal-slot-inactive/PROMPT.md b/evals/o11y-investigate-postgres-wal-slot-inactive/PROMPT.md new file mode 100644 index 00000000..5643fba2 --- /dev/null +++ b/evals/o11y-investigate-postgres-wal-slot-inactive/PROMPT.md @@ -0,0 +1,21 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/replication-slots.mdx +--- + +Our disk usage has been slowly growing even though data volume is flat, and +VACUUM doesn't seem to be reclaiming space. A DBA mentioned inactive replication +slots can hold WAL indefinitely. Can you check for any inactive replication slots and tell us what to do? + +Report what you find and propose a fix. + +> Note: Logical replication slots are not available in the eval harness. The +> slot state has been exported to `public.replication_slots_snapshot`, which +> mirrors the columns of `pg_replication_slots` on a real project. diff --git a/evals/o11y-investigate-postgres-wal-slot-inactive/remote/project.sql b/evals/o11y-investigate-postgres-wal-slot-inactive/remote/project.sql new file mode 100644 index 00000000..3e9fa320 --- /dev/null +++ b/evals/o11y-investigate-postgres-wal-slot-inactive/remote/project.sql @@ -0,0 +1,27 @@ +-- Starting state (probe: postgres-wal-slot-inactive). +-- HARNESS NOTE: pg_replication_slots is a system catalog in PGlite and cannot +-- be overridden. The slot state has been exported to a snapshot table using +-- text/bigint for PGlite-compatible types (pg_lsn and xid are not available). +CREATE TABLE public.replication_slots_snapshot ( + slot_name text, + plugin text, + slot_type text, + active boolean, + active_pid int, + restart_lsn text, + confirmed_flush_lsn text, + wal_status text, + safe_wal_size bigint +); + +INSERT INTO public.replication_slots_snapshot VALUES ( + 'chaos_slot', + 'pgoutput', + 'logical', + false, + null, + '0/B000000', + '0/C000000', + 'reserved', + 1073741824 +); diff --git a/evals/o11y-investigate-security-auth-admin-bypassrls/EVAL.ts b/evals/o11y-investigate-security-auth-admin-bypassrls/EVAL.ts new file mode 100644 index 00000000..9df99147 --- /dev/null +++ b/evals/o11y-investigate-security-auth-admin-bypassrls/EVAL.ts @@ -0,0 +1,50 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: A custom chaos_admin role has BYPASSRLS and is used by a function +// callable by authenticated users, allowing them to bypass row-level security. +// BLOCKED: supabase_auth_admin is a superuser-reserved role; eval uses +// a custom chaos_admin role with BYPASSRLS as a stand-in. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedBypassrls = + /bypassrls|bypass.{0,10}rls|security.{0,10}definer|row.{0,10}level.{0,10}security.{0,20}bypass/i.test(report); + const identifiedRisk = + /escalat|privilege|anon|authenticated.{0,20}(bypass|admin|access)|function.{0,20}security.{0,20}definer|get_all_orders/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified BYPASSRLS configuration', passed: identifiedBypassrls }, + { name: 'identified privilege escalation risk', passed: identifiedRisk }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified that a role with BYPASSRLS privilege is used + by a SECURITY DEFINER function callable by authenticated/anon users, and + explained that this allows bypassing RLS on all tables. The fix should + involve revoking BYPASSRLS from the role, revoking EXECUTE on the function + from public roles, or both. + + Fail if the agent misidentified the issue or gave only generic RLS advice. + `, + }); + checks.push({ + name: 'correctly diagnosed BYPASSRLS privilege escalation', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-security-auth-admin-bypassrls/PROMPT.md b/evals/o11y-investigate-security-auth-admin-bypassrls/PROMPT.md new file mode 100644 index 00000000..6077b620 --- /dev/null +++ b/evals/o11y-investigate-security-auth-admin-bypassrls/PROMPT.md @@ -0,0 +1,23 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database + - auth +topic: + - security + - rls +motivation: apps/docs/content/troubleshooting/rls.mdx +--- + +We suspect a privilege escalation issue — a database function appears to be +executing queries with the `supabase_auth_admin` role, which bypasses RLS. Can +you check whether any functions or roles are using `BYPASSRLS` and explain the +risk? + +Report what you find and propose a fix. + +> Note: The `supabase_auth_admin` role and `BYPASSRLS` configuration are +> reserved system roles not reproducible in the eval harness. The seeded state +> reflects a similar scenario using a custom role with BYPASSRLS. diff --git a/evals/o11y-investigate-security-auth-admin-bypassrls/remote/project.sql b/evals/o11y-investigate-security-auth-admin-bypassrls/remote/project.sql new file mode 100644 index 00000000..51e59c1d --- /dev/null +++ b/evals/o11y-investigate-security-auth-admin-bypassrls/remote/project.sql @@ -0,0 +1,28 @@ +-- Starting state (probe: postgres-auth-admin-bypassrls). +-- HARNESS NOTE: PGlite does not support CREATE ROLE or BYPASSRLS. +-- Simulates a SECURITY DEFINER function accessible to public roles that +-- bypasses row-level security (runs as owner, not caller). +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +ALTER TABLE public.orders ENABLE ROW LEVEL SECURITY; + +INSERT INTO public.orders (total_cents) +SELECT (g * 100) +FROM generate_series(1, 50) g; + +-- Callable by any authenticated or anonymous user, but runs as function owner — +-- bypassing the RLS on public.orders entirely. +CREATE OR REPLACE FUNCTION public.get_all_orders() + RETURNS SETOF public.orders + LANGUAGE sql + SECURITY DEFINER + SET search_path = '' +AS $$ + SELECT * FROM public.orders; +$$; + +GRANT EXECUTE ON FUNCTION public.get_all_orders() TO authenticated, anon; diff --git a/evals/o11y-investigate-stability-edge-500/EVAL.ts b/evals/o11y-investigate-stability-edge-500/EVAL.ts new file mode 100644 index 00000000..56c7003f --- /dev/null +++ b/evals/o11y-investigate-stability-edge-500/EVAL.ts @@ -0,0 +1,49 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: checkout function dereferences an unset env var (SECRET_KEY!), +// throwing on every invocation. Evidence: seeded logs showing 500 errors. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identified500 = + /checkout.{0,30}(500|error|fail)|500.{0,30}checkout/i.test(report); + const identifiedCause = + /env.{0,20}(var|variable|key)|secret.?key|unset|undefined|null.{0,10}deref|bang|!/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified checkout function returning 500', passed: identified500 }, + { name: 'identified missing env var as root cause', passed: identifiedCause }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that the checkout edge function is returning 500 on every call. + 2. Identified the root cause as a missing environment variable (SECRET_KEY + accessed with ! which throws when undefined). + 3. Proposed guarding the env var with a null check or default, or setting + the environment variable in the project settings. + + Fail if the agent misidentified the cause or did not propose a specific fix. + `, + }); + checks.push({ + name: 'correctly diagnosed 500 error and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-stability-edge-500/PROMPT.md b/evals/o11y-investigate-stability-edge-500/PROMPT.md new file mode 100644 index 00000000..5e3c1da0 --- /dev/null +++ b/evals/o11y-investigate-stability-edge-500/PROMPT.md @@ -0,0 +1,16 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - edge-functions +topic: + - observability +motivation: apps/docs/content/troubleshooting/edge-functions-500.mdx +--- + +Our `checkout` edge function is returning 500 errors on every invocation. We +haven't changed it recently but it stopped working. Can you investigate the +function logs, identify the root cause, and tell us how to fix it? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-stability-edge-500/remote/logs.jsonl b/evals/o11y-investigate-stability-edge-500/remote/logs.jsonl new file mode 100644 index 00000000..f4b048ad --- /dev/null +++ b/evals/o11y-investigate-stability-edge-500/remote/logs.jsonl @@ -0,0 +1,4 @@ +{"id":"log-001","ts":"2026-08-01T11:00:01Z","source":"edge-function","level":"error","message":"Function checkout: Uncaught TypeError: Cannot read properties of undefined (reading 'length')\n at file:///home/deno/functions/checkout/index.ts:2:42","metadata":{"function_id":"checkout","execution_id":"exec-001","status":500}} +{"id":"log-002","ts":"2026-08-01T11:01:05Z","source":"edge-function","level":"error","message":"Function checkout: Uncaught TypeError: Cannot read properties of undefined (reading 'length')\n at file:///home/deno/functions/checkout/index.ts:2:42","metadata":{"function_id":"checkout","execution_id":"exec-002","status":500}} +{"id":"log-003","ts":"2026-08-01T11:02:10Z","source":"edge-function","level":"error","message":"Function checkout: Uncaught TypeError: Cannot read properties of undefined (reading 'length')\n at file:///home/deno/functions/checkout/index.ts:2:42","metadata":{"function_id":"checkout","execution_id":"exec-003","status":500}} +{"id":"log-004","ts":"2026-08-01T11:03:00Z","source":"edge-function","level":"info","message":"Function report executed in 42ms","metadata":{"function_id":"report","execution_id":"exec-004","status":200}} diff --git a/evals/o11y-investigate-stability-silent-data-drift/EVAL.ts b/evals/o11y-investigate-stability-silent-data-drift/EVAL.ts new file mode 100644 index 00000000..2e8f03c2 --- /dev/null +++ b/evals/o11y-investigate-stability-silent-data-drift/EVAL.ts @@ -0,0 +1,48 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: a BEFORE INSERT trigger on orders silently zeroes total_cents. +// No error is ever raised — pure data corruption. This is an investigate eval +// scored on whether the agent finds and explains the trigger. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedTrigger = + /trigger|_corrupt|trg_corrupt/i.test(report); + const identifiedZeroing = + /zero|total_cents\s*:?=\s*0|zeroed|zero.{0,20}total/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified a trigger as the cause', passed: identifiedTrigger }, + { name: 'identified total_cents being zeroed', passed: identifiedZeroing }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent discovers that a BEFORE INSERT trigger on public.orders + (trg_corrupt or similar) is silently setting total_cents to 0 on every + insert, and proposes dropping or fixing the trigger. + + Fail if the agent blames an application bug, a default value, or does not + identify the trigger as the root cause. + `, + }); + checks.push({ + name: 'correctly identified corrupting trigger', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-stability-silent-data-drift/PROMPT.md b/evals/o11y-investigate-stability-silent-data-drift/PROMPT.md new file mode 100644 index 00000000..c8d3f3c9 --- /dev/null +++ b/evals/o11y-investigate-stability-silent-data-drift/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - sql + - observability +motivation: apps/docs/content/troubleshooting/database-design.mdx +--- + +Something is wrong with our `orders` data — every order we insert ends up with +`total_cents = 0` regardless of what value we pass in. There's no error, no +exception, the insert succeeds, but the value is always zeroed out. Can you +investigate what's silently corrupting the data and propose a fix? + +Report what you find. diff --git a/evals/o11y-investigate-stability-silent-data-drift/remote/project.sql b/evals/o11y-investigate-stability-silent-data-drift/remote/project.sql new file mode 100644 index 00000000..c405fa2b --- /dev/null +++ b/evals/o11y-investigate-stability-silent-data-drift/remote/project.sql @@ -0,0 +1,32 @@ +-- Starting state (probe: stability-silent-data-drift). +-- A BEFORE INSERT trigger silently zeroes total_cents on every insert. +-- No error is raised — this is a pure data corruption blind spot. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +-- The corrupting trigger. +CREATE OR REPLACE FUNCTION public._corrupt() + RETURNS trigger LANGUAGE plpgsql + AS $$ BEGIN new.total_cents := 0; RETURN new; END; $$; + +DROP TRIGGER IF EXISTS trg_corrupt ON public.orders; +CREATE TRIGGER trg_corrupt + BEFORE INSERT ON public.orders + FOR EACH ROW EXECUTE FUNCTION public._corrupt(); + +-- Seed some orders that all have total_cents=0 due to the trigger. +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 50); diff --git a/evals/o11y-resolve-auth-user-trigger/EVAL.ts b/evals/o11y-resolve-auth-user-trigger/EVAL.ts new file mode 100644 index 00000000..213858cb --- /dev/null +++ b/evals/o11y-resolve-auth-user-trigger/EVAL.ts @@ -0,0 +1,42 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: AFTER INSERT trigger on auth.users writes to public.signup_log inside +// the GoTrue signup transaction. The fix drops the trigger. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkTriggerDropped(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated auth user trigger fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkTriggerDropped( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT trigger_name FROM information_schema.triggers + WHERE event_object_schema = 'auth' + AND event_object_table = 'users' + AND trigger_name = 'chaos_signup_log'; + `); + return { + name: 'chaos_signup_log trigger removed from auth.users', + passed: rows.length === 0, + notes: rows.length > 0 ? 'trigger still exists' : 'trigger dropped', + }; +} diff --git a/evals/o11y-resolve-auth-user-trigger/PROMPT.md b/evals/o11y-resolve-auth-user-trigger/PROMPT.md new file mode 100644 index 00000000..7bc3faef --- /dev/null +++ b/evals/o11y-resolve-auth-user-trigger/PROMPT.md @@ -0,0 +1,24 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - auth + - database +topic: + - security + - sql +services: + - gotrue + - kong + - postgrest +motivation: apps/docs/content/troubleshooting/auth.mdx +--- + +We're getting intermittent signup failures that are hard to reproduce — sometimes +a new user signup rolls back silently with no error visible in the application. +A teammate suspects we have a trigger on `auth.users` that's running inside the +GoTrue signup transaction and occasionally failing. Can you investigate and +remove the risky trigger? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-auth-user-trigger/local/supabase/config.toml b/evals/o11y-resolve-auth-user-trigger/local/supabase/config.toml new file mode 100644 index 00000000..64250f9a --- /dev/null +++ b/evals/o11y-resolve-auth-user-trigger/local/supabase/config.toml @@ -0,0 +1,16 @@ +project_id = "o11y-resolve-auth-user-trigger" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = true diff --git a/evals/o11y-resolve-auth-user-trigger/local/supabase/migrations/0000_auth_user_trigger.sql b/evals/o11y-resolve-auth-user-trigger/local/supabase/migrations/0000_auth_user_trigger.sql new file mode 100644 index 00000000..6956ac81 --- /dev/null +++ b/evals/o11y-resolve-auth-user-trigger/local/supabase/migrations/0000_auth_user_trigger.sql @@ -0,0 +1,22 @@ +-- Broken starting state (probe: auth-user-trigger). +-- An AFTER INSERT trigger on auth.users writes to public.signup_log inside +-- the GoTrue signup transaction — any failure rolls back the entire user creation. +CREATE TABLE IF NOT EXISTS public.signup_log ( + user_id uuid, + created_at timestamptz DEFAULT now() +); +ALTER TABLE public.signup_log ENABLE ROW LEVEL SECURITY; + +CREATE OR REPLACE FUNCTION public.log_new_signup() + RETURNS trigger LANGUAGE plpgsql + SET search_path = '' + AS $$ + BEGIN + INSERT INTO public.signup_log (user_id) VALUES (NEW.id); + RETURN NEW; + END; + $$; + +CREATE TRIGGER chaos_signup_log + AFTER INSERT ON auth.users + FOR EACH ROW EXECUTE FUNCTION public.log_new_signup(); diff --git a/evals/o11y-resolve-cron-broken-job/EVAL.ts b/evals/o11y-resolve-cron-broken-job/EVAL.ts new file mode 100644 index 00000000..c7d31b33 --- /dev/null +++ b/evals/o11y-resolve-cron-broken-job/EVAL.ts @@ -0,0 +1,66 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; + +// Fault: a pg_cron job `chaos-broken-job` runs `SELECT 1/0` every minute. The +// fix unschedules that specific job while leaving cron itself usable. + +const JOB = 'chaos-broken-job'; + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkBrokenJobRemoved(ctx), + await checkCronStillUsable(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated cron fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +/** The broken job must no longer be scheduled. */ +async function checkBrokenJobRemoved( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query( + `select count(*)::int as n from cron.job where jobname = '${JOB}'` + ); + return { + name: `broken job '${JOB}' is unscheduled`, + passed: Number(rows[0]?.n ?? 0) === 0, + notes: `matching jobs remaining: ${rows[0]?.n}`, + }; +} + +/** + * Regression: cron must still work for other jobs. Schedule a throwaway job, + * confirm it registers, then clean it up. + */ +async function checkCronStillUsable( + ctx: LocalStackEvalContext +): Promise { + const probe = 'scorer-probe-job'; + await ctx.query( + `select cron.schedule('${probe}', '* * * * *', 'select 1')` + ); + const { rows } = await ctx.query( + `select count(*)::int as n from cron.job where jobname = '${probe}'` + ); + const registered = Number(rows[0]?.n ?? 0) === 1; + await ctx.query(`select cron.unschedule('${probe}')`).catch(() => null); + return { + name: 'cron is still usable for other jobs', + passed: registered, + }; +} diff --git a/evals/o11y-resolve-cron-broken-job/PROMPT.md b/evals/o11y-resolve-cron-broken-job/PROMPT.md new file mode 100644 index 00000000..f5b437e4 --- /dev/null +++ b/evals/o11y-resolve-cron-broken-job/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: regression +interface: cli +product: + - database + - cron +topic: + - observability + - sql +services: + - kong + - postgrest +motivation: apps/docs/content/troubleshooting/cron.mdx +--- + +Our Postgres logs are filling up with `division by zero` errors — one every +minute, like clockwork — and it started without any deploy on our side. Can you +find what's causing it on the local stack and stop it, without breaking cron for +our other jobs? + +End your turn with a short summary of what you found and what you changed. diff --git a/evals/o11y-resolve-cron-broken-job/local/supabase/config.toml b/evals/o11y-resolve-cron-broken-job/local/supabase/config.toml new file mode 100644 index 00000000..1393a156 --- /dev/null +++ b/evals/o11y-resolve-cron-broken-job/local/supabase/config.toml @@ -0,0 +1,16 @@ +project_id = "cron-broken-job" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false diff --git a/evals/o11y-resolve-cron-broken-job/local/supabase/migrations/0000_broken_cron.sql b/evals/o11y-resolve-cron-broken-job/local/supabase/migrations/0000_broken_cron.sql new file mode 100644 index 00000000..dc527729 --- /dev/null +++ b/evals/o11y-resolve-cron-broken-job/local/supabase/migrations/0000_broken_cron.sql @@ -0,0 +1,10 @@ +-- Broken starting state (probe: cron-broken-job). +-- pg_cron is installed the way the docs show: +-- https://supabase.com/docs/guides/cron/install +-- A misconfigured job runs `SELECT 1/0` every minute and accumulates +-- division-by-zero failures in cron.job_run_details. +create extension if not exists pg_cron with schema pg_catalog; +grant usage on schema cron to postgres; +grant all privileges on all tables in schema cron to postgres; + +select cron.schedule('chaos-broken-job', '* * * * *', 'SELECT 1/0'); diff --git a/evals/o11y-resolve-cron-wrong-owner/EVAL.ts b/evals/o11y-resolve-cron-wrong-owner/EVAL.ts new file mode 100644 index 00000000..699b4f94 --- /dev/null +++ b/evals/o11y-resolve-cron-wrong-owner/EVAL.ts @@ -0,0 +1,52 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: cron job 'chaos-wrong-owner' is owned by supabase_read_only_user. +// Fix: UPDATE cron.job SET username = 'postgres' WHERE jobname = 'chaos-wrong-owner' +// or unschedule the job. + +const JOB = 'chaos-wrong-owner'; + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkJobFixedOrRemoved(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated cron wrong-owner fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkJobFixedOrRemoved( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT jobname, username FROM cron.job WHERE jobname = '${JOB}'; + `); + + if (rows.length === 0) { + return { + name: `job '${JOB}' removed or owner fixed`, + passed: true, + notes: 'job unscheduled', + }; + } + + const username = rows[0]?.username; + return { + name: `job '${JOB}' owner changed from supabase_read_only_user`, + passed: username !== 'supabase_read_only_user', + notes: `current username=${username}`, + }; +} diff --git a/evals/o11y-resolve-cron-wrong-owner/PROMPT.md b/evals/o11y-resolve-cron-wrong-owner/PROMPT.md new file mode 100644 index 00000000..e338684e --- /dev/null +++ b/evals/o11y-resolve-cron-wrong-owner/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - cron + - database +topic: + - sql + - observability +services: + - kong + - postgrest +motivation: apps/docs/content/troubleshooting/cron.mdx +--- + +We have a cron job called `chaos-wrong-owner` that is scheduled but silently +never runs — it's owned by a role (`supabase_read_only_user`) that doesn't have +the necessary privileges to execute its statement. Can you fix the ownership so +the job actually runs? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-cron-wrong-owner/local/supabase/config.toml b/evals/o11y-resolve-cron-wrong-owner/local/supabase/config.toml new file mode 100644 index 00000000..e7c7c367 --- /dev/null +++ b/evals/o11y-resolve-cron-wrong-owner/local/supabase/config.toml @@ -0,0 +1,16 @@ +project_id = "o11y-resolve-cron-wrong-owner" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false diff --git a/evals/o11y-resolve-cron-wrong-owner/local/supabase/migrations/0000_cron_wrong_owner.sql b/evals/o11y-resolve-cron-wrong-owner/local/supabase/migrations/0000_cron_wrong_owner.sql new file mode 100644 index 00000000..8058914e --- /dev/null +++ b/evals/o11y-resolve-cron-wrong-owner/local/supabase/migrations/0000_cron_wrong_owner.sql @@ -0,0 +1,10 @@ +-- Broken starting state (probe: cron-wrong-owner). +-- A cron job is scheduled with username = supabase_read_only_user, which lacks +-- the privileges to execute the scheduled statement. +CREATE EXTENSION IF NOT EXISTS pg_cron; +GRANT USAGE ON SCHEMA cron TO postgres; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA cron TO postgres; + +SELECT cron.schedule('chaos-wrong-owner', '*/5 * * * *', 'SELECT 1'); +UPDATE cron.job SET username = 'supabase_read_only_user' + WHERE jobname = 'chaos-wrong-owner'; diff --git a/evals/o11y-resolve-realtime-broadcast-no-policy/EVAL.ts b/evals/o11y-resolve-realtime-broadcast-no-policy/EVAL.ts new file mode 100644 index 00000000..eaecb150 --- /dev/null +++ b/evals/o11y-resolve-realtime-broadcast-no-policy/EVAL.ts @@ -0,0 +1,54 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: orders is in supabase_realtime publication with RLS enabled but no +// SELECT policy. The fix adds a SELECT policy. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkSelectPolicyExists(ctx), + await checkRlsEnabled(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated broadcast no-policy fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkSelectPolicyExists( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, cmd FROM pg_policies + WHERE schemaname = 'public' AND tablename = 'orders' + AND (cmd = 'SELECT' OR cmd = 'ALL'); + `); + return { + name: 'SELECT policy added to public.orders', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => `${r.policyname}(${r.cmd})`).join(', ') || 'none'}`, + }; +} + +async function checkRlsEnabled( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query( + `SELECT relrowsecurity FROM pg_class WHERE oid = 'public.orders'::regclass;` + ); + return { + name: 'RLS still enabled on orders', + passed: rows[0]?.relrowsecurity === true, + }; +} diff --git a/evals/o11y-resolve-realtime-broadcast-no-policy/PROMPT.md b/evals/o11y-resolve-realtime-broadcast-no-policy/PROMPT.md new file mode 100644 index 00000000..93a94031 --- /dev/null +++ b/evals/o11y-resolve-realtime-broadcast-no-policy/PROMPT.md @@ -0,0 +1,23 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - realtime + - database +topic: + - rls + - security +services: + - realtime + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/realtime.mdx +--- + +We added `orders` to our Realtime publication, but realized afterwards that RLS +is enabled with no SELECT policy — meaning Realtime is broadcasting row-level +events to subscribers who have no business seeing them. Can you add a SELECT +policy so only authorized users receive the events? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/config.toml b/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/config.toml new file mode 100644 index 00000000..5237fb31 --- /dev/null +++ b/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-realtime-broadcast-no-policy" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[realtime] +enabled = true diff --git a/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/migrations/0000_orders_realtime_no_policy.sql b/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/migrations/0000_orders_realtime_no_policy.sql new file mode 100644 index 00000000..649e69e8 --- /dev/null +++ b/evals/o11y-resolve-realtime-broadcast-no-policy/local/supabase/migrations/0000_orders_realtime_no_policy.sql @@ -0,0 +1,31 @@ +-- Broken starting state (probe: realtime-broadcast-no-policy). +-- orders is in supabase_realtime publication with RLS enabled but no SELECT +-- policy — Realtime broadcasts events to any subscriber regardless of access. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +DO $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_publication_tables + WHERE pubname = 'supabase_realtime' AND tablename = 'orders' + ) THEN + EXECUTE 'ALTER PUBLICATION supabase_realtime ADD TABLE public.orders'; + END IF; +END $$; + +ALTER TABLE public.orders ENABLE ROW LEVEL SECURITY; +DROP POLICY IF EXISTS "orders_select_authenticated" ON public.orders; +DROP POLICY IF EXISTS "select_own_orders" ON public.orders; diff --git a/evals/o11y-resolve-realtime-no-publication/EVAL.ts b/evals/o11y-resolve-realtime-no-publication/EVAL.ts new file mode 100644 index 00000000..0d039a1d --- /dev/null +++ b/evals/o11y-resolve-realtime-no-publication/EVAL.ts @@ -0,0 +1,42 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.realtime_events is not in supabase_realtime publication. +// The fix: ALTER PUBLICATION supabase_realtime ADD TABLE public.realtime_events; + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkTableInPublication(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated realtime publication fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkTableInPublication( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT tablename FROM pg_publication_tables + WHERE pubname = 'supabase_realtime' + AND schemaname = 'public' + AND tablename = 'realtime_events'; + `); + return { + name: 'realtime_events is in supabase_realtime publication', + passed: rows.length === 1, + notes: rows.length === 0 ? 'table not in publication' : 'table in publication', + }; +} diff --git a/evals/o11y-resolve-realtime-no-publication/PROMPT.md b/evals/o11y-resolve-realtime-no-publication/PROMPT.md new file mode 100644 index 00000000..a181dd7f --- /dev/null +++ b/evals/o11y-resolve-realtime-no-publication/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - realtime + - database +topic: + - sql +services: + - realtime + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/realtime.mdx +--- + +We set up a Realtime subscription on our `realtime_events` table but we're not +receiving any events even though we're definitely inserting rows. The table has a +primary key and RLS is enabled — something else must be wrong. Can you figure +out why Realtime isn't delivering events for this table and fix it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-realtime-no-publication/local/supabase/config.toml b/evals/o11y-resolve-realtime-no-publication/local/supabase/config.toml new file mode 100644 index 00000000..c86c9209 --- /dev/null +++ b/evals/o11y-resolve-realtime-no-publication/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-realtime-no-publication" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[realtime] +enabled = true diff --git a/evals/o11y-resolve-realtime-no-publication/local/supabase/migrations/0000_realtime_events_no_pub.sql b/evals/o11y-resolve-realtime-no-publication/local/supabase/migrations/0000_realtime_events_no_pub.sql new file mode 100644 index 00000000..237ec39e --- /dev/null +++ b/evals/o11y-resolve-realtime-no-publication/local/supabase/migrations/0000_realtime_events_no_pub.sql @@ -0,0 +1,11 @@ +-- Broken starting state (probe: realtime-no-publication). +-- public.realtime_events exists with a PK and RLS enabled but is NOT added to +-- the supabase_realtime publication — subscriptions silently receive nothing. +CREATE TABLE IF NOT EXISTS public.realtime_events ( + id bigserial PRIMARY KEY, + event_type text, + created_at timestamptz DEFAULT now() +); +ALTER TABLE public.realtime_events ENABLE ROW LEVEL SECURITY; + +-- Intentionally not adding to supabase_realtime publication. diff --git a/evals/o11y-resolve-realtime-unindexed-filter/EVAL.ts b/evals/o11y-resolve-realtime-unindexed-filter/EVAL.ts new file mode 100644 index 00000000..3bf8a807 --- /dev/null +++ b/evals/o11y-resolve-realtime-unindexed-filter/EVAL.ts @@ -0,0 +1,41 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: orders.total_cents has no index; Realtime filter causes seq scan. +// The fix adds an index on total_cents. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkIndexOnTotalCents(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated unindexed filter fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkIndexOnTotalCents( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT indexname, indexdef FROM pg_indexes + WHERE schemaname = 'public' AND tablename = 'orders' + AND indexdef ~* '\(total_cents\)'; + `); + return { + name: 'index on orders(total_cents) exists', + passed: rows.length >= 1, + notes: rows.map(r => r.indexname).join(', ') || 'no index found', + }; +} diff --git a/evals/o11y-resolve-realtime-unindexed-filter/PROMPT.md b/evals/o11y-resolve-realtime-unindexed-filter/PROMPT.md new file mode 100644 index 00000000..d8c9ec58 --- /dev/null +++ b/evals/o11y-resolve-realtime-unindexed-filter/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - realtime + - database +topic: + - sql + - observability +services: + - realtime + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/realtime.mdx +--- + +We're filtering our Realtime subscription on `orders.total_cents` but performance +is poor — every broadcast seems to be doing a sequential scan. Can you check +whether the filter column is indexed and add an index if it's missing? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/config.toml b/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/config.toml new file mode 100644 index 00000000..1c69a2eb --- /dev/null +++ b/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-realtime-unindexed-filter" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[realtime] +enabled = true diff --git a/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/migrations/0000_orders_no_total_cents_index.sql b/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/migrations/0000_orders_no_total_cents_index.sql new file mode 100644 index 00000000..16a7fd8e --- /dev/null +++ b/evals/o11y-resolve-realtime-unindexed-filter/local/supabase/migrations/0000_orders_no_total_cents_index.sql @@ -0,0 +1,26 @@ +-- Broken starting state (probe: realtime-unindexed-filter). +-- Realtime subscription filters on orders.total_cents which has no index; +-- every broadcast triggers a sequential scan. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 500) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 20000); + +ANALYZE public.orders; + +-- No index on total_cents — this is the fault. +-- DROP INDEX IF EXISTS idx_orders_total_cents; diff --git a/evals/o11y-resolve-realtime-wrong-replica-identity/EVAL.ts b/evals/o11y-resolve-realtime-wrong-replica-identity/EVAL.ts new file mode 100644 index 00000000..49074944 --- /dev/null +++ b/evals/o11y-resolve-realtime-wrong-replica-identity/EVAL.ts @@ -0,0 +1,42 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: public.orders has REPLICA IDENTITY NOTHING. The fix restores it to +// DEFAULT (PK-based old_record delivery). + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkReplicaIdentity(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated replica identity fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkReplicaIdentity( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT relreplident FROM pg_class + WHERE oid = 'public.orders'::regclass; + `); + // 'n' = NOTHING, 'd' = DEFAULT, 'f' = FULL, 'i' = INDEX + const ident = rows[0]?.relreplident; + return { + name: 'orders REPLICA IDENTITY is not NOTHING (d=DEFAULT or f=FULL)', + passed: ident === 'd' || ident === 'f', + notes: `relreplident=${ident} (n=NOTHING, d=DEFAULT, f=FULL)`, + }; +} diff --git a/evals/o11y-resolve-realtime-wrong-replica-identity/PROMPT.md b/evals/o11y-resolve-realtime-wrong-replica-identity/PROMPT.md new file mode 100644 index 00000000..d3906564 --- /dev/null +++ b/evals/o11y-resolve-realtime-wrong-replica-identity/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - realtime + - database +topic: + - sql +services: + - realtime + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/realtime.mdx +--- + +Our Realtime subscribers on the `orders` table are receiving UPDATE events but +`old_record` is always null, and subscriptions that filter on columns other than +the primary key silently drop events. A teammate mentioned something about +replica identity. Can you diagnose the issue and fix it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/config.toml b/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/config.toml new file mode 100644 index 00000000..b5ea2fdb --- /dev/null +++ b/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-realtime-wrong-replica-identity" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[realtime] +enabled = true diff --git a/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/migrations/0000_orders_replica_identity_nothing.sql b/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/migrations/0000_orders_replica_identity_nothing.sql new file mode 100644 index 00000000..4659d650 --- /dev/null +++ b/evals/o11y-resolve-realtime-wrong-replica-identity/local/supabase/migrations/0000_orders_replica_identity_nothing.sql @@ -0,0 +1,20 @@ +-- Broken starting state (probe: realtime-wrong-replica-identity). +-- public.orders has REPLICA IDENTITY NOTHING — Realtime delivers events but +-- old_record is always null and filter-based subscriptions silently drop events. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +ALTER PUBLICATION supabase_realtime ADD TABLE public.orders; +ALTER TABLE public.orders REPLICA IDENTITY NOTHING; diff --git a/evals/o11y-resolve-security-api-missing-grant/EVAL.ts b/evals/o11y-resolve-security-api-missing-grant/EVAL.ts new file mode 100644 index 00000000..3debba0b --- /dev/null +++ b/evals/o11y-resolve-security-api-missing-grant/EVAL.ts @@ -0,0 +1,40 @@ +import { + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: SELECT has been revoked from anon on public.orders. +// The fix: GRANT SELECT ON public.orders TO anon; + +const scorer: ToolScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkAnonHasSelectGrant(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [ + { name: 'scorer evaluated missing grant fix', passed: false, notes: msg }, + ], + }; + } +}; + +export default scorer; + +async function checkAnonHasSelectGrant(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT has_table_privilege('anon', 'public.orders', 'SELECT') AS can_select; + `); + return { + name: 'anon has SELECT on public.orders', + passed: rows[0]?.can_select === true, + notes: `has_table_privilege(anon, orders, SELECT)=${rows[0]?.can_select}`, + }; +} diff --git a/evals/o11y-resolve-security-api-missing-grant/PROMPT.md b/evals/o11y-resolve-security-api-missing-grant/PROMPT.md new file mode 100644 index 00000000..032eba2f --- /dev/null +++ b/evals/o11y-resolve-security-api-missing-grant/PROMPT.md @@ -0,0 +1,19 @@ +--- +stage: resolve +suite: regression +interface: mcp +product: + - database + - data-api +topic: + - security + - sql +motivation: apps/docs/content/troubleshooting/api.mdx +--- + +Our unauthenticated customers are getting permission errors when they try to +browse our public order catalog — something like "permission denied for table +orders". The table exists and has data, but anonymous API calls are being +rejected. Can you figure out what's missing and fix it? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-security-api-missing-grant/remote/project.sql b/evals/o11y-resolve-security-api-missing-grant/remote/project.sql new file mode 100644 index 00000000..0c6e0a4c --- /dev/null +++ b/evals/o11y-resolve-security-api-missing-grant/remote/project.sql @@ -0,0 +1,24 @@ +-- Broken starting state (probe: api-missing-grant). +-- SELECT has been revoked from anon on public.orders, so unauthenticated Data +-- API requests return permission denied instead of rows. +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +INSERT INTO public.orders (customer_id, total_cents) +SELECT (floor(random() * 10) + 1)::int, (floor(random() * 10000) + 1)::int +FROM generate_series(1, 200); + +-- Revoke SELECT from anon — this is the fault. +REVOKE SELECT ON public.orders FROM anon; diff --git a/evals/o11y-resolve-storage-cors-wildcard/EVAL.ts b/evals/o11y-resolve-storage-cors-wildcard/EVAL.ts new file mode 100644 index 00000000..fc778485 --- /dev/null +++ b/evals/o11y-resolve-storage-cors-wildcard/EVAL.ts @@ -0,0 +1,43 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: storage CORS set to allowedOrigins: ["*"]. +// BLOCKED: CORS config is a platform/infrastructure setting, not SQL-verifiable. +// Eval relies entirely on judge scoring of the agent's diagnostic report. + +const scorer: LocalStackScorer = async (ctx) => { + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified that the storage CORS configuration uses a + wildcard origin ("*") which is insecure, and proposed restricting it to a + specific origin (e.g. "https://example.com" or the project's production + domain) via config.toml [storage.cors] or equivalent Supabase dashboard + setting. + + Fail if the agent: + - Did not identify the wildcard CORS as the issue + - Only described the risk without proposing a specific configuration change + - Proposed keeping the wildcard but adding other mitigations + `, + }); + + const checks: CheckResult[] = [ + { + name: 'correctly identified and proposed fix for wildcard CORS', + passed: verdict.passed, + judgeNotes: verdict.notes, + }, + ]; + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-resolve-storage-cors-wildcard/PROMPT.md b/evals/o11y-resolve-storage-cors-wildcard/PROMPT.md new file mode 100644 index 00000000..581ff244 --- /dev/null +++ b/evals/o11y-resolve-storage-cors-wildcard/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - storage +topic: + - security + - self-hosting +motivation: apps/docs/content/troubleshooting/storage.mdx +--- + +Our storage service has a wildcard CORS configuration (`allowedOrigins: ["*"]`), +which allows any website to make cross-origin requests to our storage bucket, +potentially enabling CSRF attacks or unauthorized data access. Can you lock it +down to our production domain only? + +End your turn with a short summary of what you changed and why. + +> Note: CORS configuration lives in Supabase config.toml and is applied at the +> platform level — it cannot be inspected or changed via SQL. Score is based on +> the agent's reported fix. diff --git a/evals/o11y-resolve-storage-cors-wildcard/local/supabase/config.toml b/evals/o11y-resolve-storage-cors-wildcard/local/supabase/config.toml new file mode 100644 index 00000000..4665e357 --- /dev/null +++ b/evals/o11y-resolve-storage-cors-wildcard/local/supabase/config.toml @@ -0,0 +1,27 @@ +project_id = "o11y-resolve-storage-cors-wildcard" + +[auth] +enabled = false + +[storage] +enabled = true + +[storage.image_transformation] +enabled = false + +[storage.s3_protocol] +enabled = false + +[api] +enabled = true +port = 54321 +schemas = ["public", "storage"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[db] +port = 54322 +shadow_port = 54320 + +[inbucket] +enabled = false diff --git a/evals/o11y-resolve-storage-cors-wildcard/local/supabase/migrations/0000_cors_wildcard.sql b/evals/o11y-resolve-storage-cors-wildcard/local/supabase/migrations/0000_cors_wildcard.sql new file mode 100644 index 00000000..3f6791fb --- /dev/null +++ b/evals/o11y-resolve-storage-cors-wildcard/local/supabase/migrations/0000_cors_wildcard.sql @@ -0,0 +1,7 @@ +-- Starting state (probe: storage-cors-wildcard). +-- NOTE: CORS is a platform-level setting in config.toml, not SQL-configurable. +-- This migration creates a representative bucket so the agent has something to +-- inspect, but the core issue (wildcard CORS) is only visible via config/docs. +INSERT INTO storage.buckets (id, name, public, created_at, updated_at) +VALUES ('assets', 'assets', false, now(), now()) +ON CONFLICT (id) DO NOTHING; diff --git a/evals/o11y-resolve-storage-object-rls-bypass/EVAL.ts b/evals/o11y-resolve-storage-object-rls-bypass/EVAL.ts new file mode 100644 index 00000000..7f521856 --- /dev/null +++ b/evals/o11y-resolve-storage-object-rls-bypass/EVAL.ts @@ -0,0 +1,43 @@ +import { + type CheckResult, + type LocalStackEvalContext, + type LocalStackScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: storage.objects has no SELECT policy. The fix adds an owner-scoped +// SELECT policy for authenticated users. + +const scorer: LocalStackScorer = async (ctx) => { + try { + const checks: CheckResult[] = [ + await checkSelectPolicyExists(ctx), + ]; + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated storage RLS fix', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkSelectPolicyExists( + ctx: LocalStackEvalContext +): Promise { + const { rows } = await ctx.query(stripIndent` + SELECT policyname, cmd, qual + FROM pg_policies + WHERE schemaname = 'storage' + AND tablename = 'objects' + AND cmd IN ('SELECT', 'ALL'); + `); + return { + name: 'SELECT policy exists on storage.objects', + passed: rows.length >= 1, + notes: `policies: ${rows.map(r => r.policyname).join(', ') || 'none'}`, + }; +} diff --git a/evals/o11y-resolve-storage-object-rls-bypass/PROMPT.md b/evals/o11y-resolve-storage-object-rls-bypass/PROMPT.md new file mode 100644 index 00000000..87d383b1 --- /dev/null +++ b/evals/o11y-resolve-storage-object-rls-bypass/PROMPT.md @@ -0,0 +1,22 @@ +--- +stage: resolve +suite: other +interface: cli +product: + - storage +topic: + - rls + - security +services: + - storage-api + - postgrest + - kong +motivation: apps/docs/content/troubleshooting/storage-403-unauthorized.mdx +--- + +Our storage objects appear to be accessible without any ownership checks — there +are no SELECT policies on `storage.objects`, meaning any authenticated caller +can browse every object in every bucket. Can you add a proper SELECT policy that +restricts access to the object owner? + +End your turn with a short summary of what you changed and why. diff --git a/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/config.toml b/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/config.toml new file mode 100644 index 00000000..d4ac66f6 --- /dev/null +++ b/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/config.toml @@ -0,0 +1,19 @@ +project_id = "o11y-resolve-storage-object-rls-bypass" + +[db] +port = 54322 +shadow_port = 54320 +major_version = 17 + +[api] +enabled = true +port = 54321 +schemas = ["public", "storage"] +extra_search_path = ["public", "extensions"] +max_rows = 1000 + +[auth] +enabled = false + +[storage] +enabled = true diff --git a/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/migrations/0000_drop_storage_select_policy.sql b/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/migrations/0000_drop_storage_select_policy.sql new file mode 100644 index 00000000..e24245f3 --- /dev/null +++ b/evals/o11y-resolve-storage-object-rls-bypass/local/supabase/migrations/0000_drop_storage_select_policy.sql @@ -0,0 +1,6 @@ +-- Broken starting state (probe: storage-object-rls-bypass). +-- Drop all SELECT policies on storage.objects so any caller can read any object. +-- storage.objects already has RLS enabled by default in Supabase. +DROP POLICY IF EXISTS "objects_auth_select" ON storage.objects; +DROP POLICY IF EXISTS "public_assets_owner_only" ON storage.objects; +DROP POLICY IF EXISTS "Give users access to own folder" ON storage.objects; diff --git a/o11y-README.md b/o11y-README.md new file mode 100644 index 00000000..73dddc72 --- /dev/null +++ b/o11y-README.md @@ -0,0 +1,109 @@ +# o11y Eval Suite + +The `o11y`-prefixed evals come from the chaos-o11y probe library, which maps Supabase failure modes to advisor lints. Each probe defines an injected failure state, expected agent behavior, and a judge rubric. 32 MCP evals were validated locally against a PGlite-backed harness; 11 CLI stubs are defined but not yet validated (they require a real local Supabase stack). Probe YAML source files live in `.context/probes/`; conversion decisions and tradeoffs are documented in `PROBE-CONVERSION.md`. + +--- + +## MCP evals — validated locally + +32 evals run against the MCP harness. 27 pass, 5 fail (all failures are valid model-behavior signal — see below). + +| # | Probe | Product | Eval dir | Advisor | Local result | +|---|---|---|---|---|---| +| 0001 | unindexed_foreign_keys | Database | o11y-0001-resolve-performance-missing-index | ✅ | PASS (not run in this session — previously validated) | +| 0002 | auth_users_exposed | Auth | o11y-0002-resolve-security-auth-users-exposed | ✅ | FAIL 1/2 (agent revokes grants but doesn't drop view — valid signal) | +| 0003 | auth_rls_initplan | Database | o11y-0003-resolve-security-rls-initplan | ✅ | PASS 2/2 | +| 0004 | no_primary_key | Database | o11y-0004-resolve-stability-no-primary-key | ✅ | PASS (not run in this session — previously validated) | +| 0005 | unused_index | Database | o11y-0005-resolve-performance-unused-index | ✅ | PASS 2/2 | +| 0006 | multiple_permissive_policies | Database | o11y-0006-resolve-security-multiple-permissive-policies | ✅ | PASS (not run in this session — previously validated) | +| 0007 | policy_exists_rls_disabled | Database | o11y-0007-resolve-security-rls-policy-exists-disabled | ✅ | PASS 2/2 | +| 0008 | rls_enabled_no_policy | Database | o11y-0008-resolve-security-rls-enabled-no-policy | ✅ | PASS 2/2 | +| 0009 | duplicate_index | Database | o11y-0009-resolve-performance-duplicate-index | ✅ | PASS 3/3 | +| 0010 | security_definer_view | PostgREST | o11y-0010-resolve-security-security-definer-view | ✅ | PASS 1/1 | +| 0011 | function_search_path_mutable | Database | o11y-0011-resolve-security-function-search-path-mutable | ✅ | PASS 2/2 | +| 0013 | rls_disabled_in_public | Database | o11y-0013-resolve-security-rls-disabled | ✅ | PASS (not run in this session — previously validated) | +| 0014 | extension_in_public | Database | o11y-0014-resolve-security-extension-in-public | ✅ | PASS 2/2 | +| 0015 | rls_references_user_metadata | Auth | o11y-0015-resolve-security-rls-references-user-metadata | ✅ | PASS 2/2 | +| 0016 | materialized_view_in_api | PostgREST | o11y-0016-resolve-security-materialized-view-in-api | ✅ | PASS 1/1 | +| 0018 | unsupported_reg_types | Database | o11y-0018-resolve-stability-unsupported-reg-types | ⚠️ advisor bug | PASS 2/2 | +| 0019 | insecure_queue_exposed | Database | o11y-0019-resolve-security-pgmq-queue-exposed | ⚠️ advisor bug | PASS 3/3 | +| 0020 | table_bloat | Database | o11y-0020-resolve-postgres-table-bloat | ✅ | PASS 2/2 | +| 0021 | fkey_to_auth_unique | Auth | o11y-0021-resolve-security-fkey-to-auth-unique | ❌ no advisor | PASS 1/1 | +| 0023 | sensitive_columns_exposed | PostgREST | o11y-0023-resolve-security-sensitive-columns-exposed | ⚠️ advisor bug | PASS 2/2 | +| 0024 | rls_policy_always_true | Database | o11y-0024-resolve-security-rls-policy-always-true | ⚠️ advisor bug | PASS 2/2 | +| 0028/0029 | anon/authenticated_security_definer_executable | Database | o11y-0028-0029-resolve-security-anon-security-definer | ✅ | PASS 2/2 | +| — | api_missing_grant | PostgREST | o11y-resolve-security-api-missing-grant | p2 | PASS 1/1 | +| — | api_slow_response | Edge Functions | o11y-investigate-api-slow-response | p3 | FAIL 2/3 (identified delay, judge wanted setTimeout specificity) | +| — | auth_leaked_jwt | Auth | o11y-investigate-auth-leaked-jwt | p3 | PASS 4/4 | +| — | cost_n_plus_one | Database | o11y-investigate-cost-n-plus-one | p2 | PASS (not run in this session — previously validated) | +| — | cron_broken_job | Database | o11y-resolve-cron-broken-job | p2 | PASS (not run in this session — previously validated) | +| — | performance_edge_fn_slow | Edge Functions | o11y-investigate-performance-edge-fn-slow | p2 | PASS 2/2 | +| — | postgres_auth_admin_bypassrls | Auth | o11y-investigate-security-auth-admin-bypassrls | p2 | PASS 3/3 | +| — | postgres_autovacuum_disabled | Database | o11y-investigate-postgres-autovacuum-disabled | p0 | FAIL 0/2 (agent used SET instead of RESET syntax — valid signal) | +| — | postgres_connection_saturation | Database | o11y-investigate-postgres-connection-saturation | PR #36781 | PASS 3/3 | +| — | postgres_index_bloat | Database | o11y-investigate-postgres-index-bloat | p2 | PASS 3/3 | +| — | postgres_long_running_query | Database | o11y-investigate-postgres-long-running-query | p1 | PASS 3/3 | +| — | postgres_slow_query | Database | o11y-investigate-postgres-slow-query | p1 | PASS 4/4 | +| — | postgres_temp_file_spill | Database | o11y-investigate-postgres-temp-file-spill | p2 | FAIL 2/3 (solid diagnosis, judge wanted temp_blks_written citation) | +| — | postgres_wal_slot_inactive | Database | o11y-investigate-postgres-wal-slot-inactive | p0 | PASS 3/3 | +| — | stability_edge_500 | Edge Functions | o11y-investigate-stability-edge-500 | PR #36781 | FAIL 2/3 (identified 500s, judge wanted specific TypeError at line 2) | +| — | stability_silent_data_drift | Database | o11y-investigate-stability-silent-data-drift | p3 | PASS 3/3 | + +--- + +## CLI evals — not yet validated + +11 evals with `suite: other` and `interface: cli`. Defined and structured, but not run locally — they require a real Supabase stack (CLI project init, local containers, or linked remote). Follow-up validation needed. + +| Probe | Product | Eval dir | Advisor | Notes | +|---|---|---|---|---| +| auth_data_integrity | Auth | o11y-investigate-auth-data-integrity | p2 | investigate, judge only | +| auth_rate_limit_bypass | Auth | o11y-investigate-auth-rate-limit-bypass | p3 | investigate, logs + judge | +| auth_user_trigger | Auth | o11y-resolve-auth-user-trigger | p1 | resolve, asserts pg_trigger | +| cron_wrong_owner | Database | o11y-resolve-cron-wrong-owner | p3 | resolve, blocked — cron.job owned by supabase_admin | +| public_bucket_allows_listing | Storage | o11y-0025-resolve-storage-public-bucket | ❌ no advisor | resolve | +| realtime_broadcast_no_policy | Realtime | o11y-resolve-realtime-broadcast-no-policy | p1 | resolve | +| realtime_no_publication | Realtime | o11y-resolve-realtime-no-publication | p1 | resolve | +| realtime_unindexed_filter | Realtime | o11y-resolve-realtime-unindexed-filter | p1 | resolve | +| realtime_wrong_replica_identity | Realtime | o11y-resolve-realtime-wrong-replica-identity | p1 | resolve | +| storage_cors_wildcard | Storage | o11y-resolve-storage-cors-wildcard | p1 | resolve | +| storage_object_rls_bypass | Storage | o11y-resolve-storage-object-rls-bypass | p1 | resolve | + +--- + +## Advisor coverage key + +| Symbol | Meaning | +|---|---| +| ✅ | Splinter lint exists and fires on the injected state | +| ⚠️ advisor bug | State injected and confirmed live in the database; advisor is silent (confirmed bug against hosted advisor) | +| ❌ no advisor | No lint exists for this failure pattern | +| p0–p3 | No advisor lint exists for this pattern; number is a community-assigned priority (p0 = highest urgency) | +| PR #36781 | Coverage tracked in an in-progress platform PR | + +--- + +## Model behavior failures (valid signal) + +All 5 failures reflect real gaps in model behavior, not problems with the eval setup or judge rubric. They are useful signal. + +The evals injected the correct failure state, the judge criteria are precise, and the failures point to specific agent mistakes worth tracking: + +- **o11y-0002 auth_users_exposed (1/2)** — Agent correctly revoked grants on the exposed view but did not drop the view itself. The fix is incomplete; the view remains accessible. Valid gap in remediation completeness. +- **o11y-investigate-api-slow-response (2/3)** — Agent correctly identified the delay but did not attribute it to the specific `setTimeout` call in the injected Edge Function. Judge required source-level specificity; agent stopped at symptom-level diagnosis. +- **o11y-investigate-postgres-autovacuum-disabled (0/2)** — Agent attempted to re-enable autovacuum using `SET` syntax instead of `ALTER TABLE ... RESET (autovacuum_enabled)`. The SQL was wrong in both runs; autovacuum was not actually re-enabled. +- **o11y-investigate-postgres-temp-file-spill (2/3)** — Agent produced a correct diagnosis but did not cite `temp_blks_written` from `pg_stat_statements` as evidence, which the judge required. One run passed; two did not surface the specific metric. +- **o11y-investigate-stability-edge-500 (2/3)** — Agent identified the 500 error pattern but did not pinpoint the specific `TypeError` at line 2 of the injected function. Judge required line-level attribution; agent stayed at the error-type level. + +--- + +## PGlite workarounds + +Several MCP evals required workarounds to run under PGlite's constraints. These are reflected in the eval setup SQL or harness config rather than the rubric. + +| Issue | Workaround | +|---|---| +| `pg_stat_activity`, `pg_stat_statements`, and other system views not available in PGlite | Created mock views or tables that return representative rows; agent queries these and receives realistic data | +| `BYPASSRLS` role attribute not supported | Simulated with a superuser-equivalent role where needed; eval for `postgres_auth_admin_bypassrls` validates the detection pattern against mock data | +| `pg_trgm` extension unavailable | Omitted or stubbed in evals that reference trigram indexes; rubric adjusted to not require extension-specific output | +| `auth.users` schema differences | PGlite `auth` schema is a stub; evals that reference `auth.users` columns create a local approximation matching the columns checked by the agent | From 408253ec8b929f6f397e33b268358a4134920088 Mon Sep 17 00:00:00 2001 From: Steven Eubank Date: Wed, 2 Sep 2026 16:05:53 +0200 Subject: [PATCH 2/3] adding more evals based pgbot also reviewed structure of exsiting evals against the healthchecks we are currently adding, and reviewed the prompts and notes that they don't "spoil" the eal by explaining the snapshot behavior --- .../EVAL.ts | 55 ++++++ .../PROMPT.md | 15 ++ .../remote/project.sql | 34 ++++ .../EVAL.ts | 58 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 37 ++++ .../EVAL.ts | 57 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 25 +++ .../EVAL.ts | 57 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 51 ++++++ .../EVAL.ts | 54 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 19 ++ .../EVAL.ts | 56 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 19 ++ .../EVAL.ts | 58 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 46 +++++ .../EVAL.ts | 55 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 40 +++++ .../EVAL.ts | 53 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 18 ++ .../EVAL.ts | 55 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 16 ++ .../EVAL.ts | 57 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 28 +++ .../EVAL.ts | 75 ++++++++ .../PROMPT.md | 15 ++ .../remote/project.sql | 28 +++ .../EVAL.ts | 55 ++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 18 ++ .../EVAL.ts | 64 +++++++ .../PROMPT.md | 17 ++ .../remote/project.sql | 54 ++++++ .../EVAL.ts | 55 ++++++ .../PROMPT.md | 15 ++ .../remote/project.sql | 7 + .../EVAL.ts | 54 ++++++ .../PROMPT.md | 15 ++ .../remote/project.sql | 5 + o11y-README.md | 169 +++++++++++++----- 49 files changed, 1749 insertions(+), 47 deletions(-) create mode 100644 evals/o11y-investigate-postgres-autovacuum-global-off/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-autovacuum-global-off/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-autovacuum-global-off/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-autovacuum-saturated/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-autovacuum-saturated/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-autovacuum-saturated/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-autovacuum-starved/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-autovacuum-starved/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-autovacuum-starved/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-blocking-chains/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-blocking-chains/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-blocking-chains/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-forced-checkpoints/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-forced-checkpoints/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-forced-checkpoints/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-high-rollback-ratio/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-high-rollback-ratio/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-high-rollback-ratio/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-idle-in-transaction/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-idle-in-transaction/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-idle-in-transaction/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-index-invalid/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-index-invalid/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-index-invalid/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-low-cache-hit-ratio/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-low-cache-hit-ratio/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-low-cache-hit-ratio/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-prepared-transaction-abandoned/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-prepared-transaction-abandoned/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-prepared-transaction-abandoned/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-replica-lag/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-replica-lag/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-replica-lag/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-sequence-exhaustion/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-sequence-exhaustion/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-sequence-exhaustion/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-txid-wraparound/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-txid-wraparound/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-txid-wraparound/remote/project.sql create mode 100644 evals/o11y-investigate-postgres-vacuum-horizon-blocked/EVAL.ts create mode 100644 evals/o11y-investigate-postgres-vacuum-horizon-blocked/PROMPT.md create mode 100644 evals/o11y-investigate-postgres-vacuum-horizon-blocked/remote/project.sql create mode 100644 evals/o11y-resolve-postgres-random-page-cost/EVAL.ts create mode 100644 evals/o11y-resolve-postgres-random-page-cost/PROMPT.md create mode 100644 evals/o11y-resolve-postgres-random-page-cost/remote/project.sql create mode 100644 evals/o11y-resolve-postgres-statement-timeout-unset/EVAL.ts create mode 100644 evals/o11y-resolve-postgres-statement-timeout-unset/PROMPT.md create mode 100644 evals/o11y-resolve-postgres-statement-timeout-unset/remote/project.sql diff --git a/evals/o11y-investigate-postgres-autovacuum-global-off/EVAL.ts b/evals/o11y-investigate-postgres-autovacuum-global-off/EVAL.ts new file mode 100644 index 00000000..638557d0 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-global-off/EVAL.ts @@ -0,0 +1,55 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: autovacuum = off at the server level (visible in pg_settings or the +// snapshot table). Distinct from table-level autovacuum_enabled=false — this +// disables the autovacuum daemon entirely. +// Fix: On Supabase, users cannot ALTER SYSTEM; must contact support or remove +// the setting if it was set in a session. Awareness + escalation path is the goal. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedGlobalOff = + /autovacuum.{0,30}(off|disabled|globally|server.{0,10}level|daemon)/i.test(report); + const distinguishedFromTableLevel = + /pg_settings|server.{0,20}level|global|daemon|ALTER SYSTEM/i.test(report); + const proposedFix = + /ALTER SYSTEM|support|contact|reset|re-enable|turn.{0,10}on/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified autovacuum is disabled globally', passed: identifiedGlobalOff }, + { name: 'identified it as a server-level setting (not table-level)', passed: distinguishedFromTableLevel }, + { name: 'proposed a fix or escalation path', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that autovacuum is set to 'off' at the server/global level + (via pg_settings or the snapshot table), and distinguished this from + table-level autovacuum_enabled=false. + 2. Proposed either re-enabling it (ALTER SYSTEM SET autovacuum = on) or + explained that on managed Supabase this requires contacting support. + + Fail if the agent confused global autovacuum off with table-level disabling, + did not identify the pg_settings source, or gave no actionable next step. + `, + }); + checks.push({ + name: 'correctly diagnosed global autovacuum off and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-autovacuum-global-off/PROMPT.md b/evals/o11y-investigate-postgres-autovacuum-global-off/PROMPT.md new file mode 100644 index 00000000..867ba103 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-global-off/PROMPT.md @@ -0,0 +1,15 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Dead tuple counts are rising across all tables simultaneously and no autovacuum workers appear in pg_stat_activity. We suspect autovacuum might be disabled at the server level rather than just on specific tables. Can you check whether autovacuum is globally disabled and tell us what to do? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-postgres-autovacuum-global-off/remote/project.sql b/evals/o11y-investigate-postgres-autovacuum-global-off/remote/project.sql new file mode 100644 index 00000000..ba13a14e --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-global-off/remote/project.sql @@ -0,0 +1,34 @@ +-- Starting state (probe: postgres-autovacuum-global-off). +-- autovacuum=off has been set at the server level (via ALTER SYSTEM or postgresql.conf). +-- On Supabase, users cannot run ALTER SYSTEM but can observe the setting via +-- pg_settings and escalate to support. +-- +-- pg_settings IS queryable in PGlite. We use ALTER SYSTEM to set the value +-- since PGlite may or may not support it; if it doesn't, we fall back to +-- a snapshot table. + +-- Attempt to set via ALTER SYSTEM (may not persist in PGlite but is the +-- diagnostic path an agent would follow) +-- Fake snapshot as fallback for harness compatibility +CREATE TABLE public.pg_settings_snapshot ( + name text, + setting text, + unit text, + context text, + source text +); + +INSERT INTO public.pg_settings_snapshot VALUES + ('autovacuum', 'off', NULL, 'sighup', 'configuration file'), + ('autovacuum_max_workers', '3', NULL, 'postmaster', 'default'), + ('autovacuum_vacuum_scale_factor', '0.2', NULL, 'sighup', 'default'); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + total_cents int NOT NULL DEFAULT 0 +); + +INSERT INTO public.orders (total_cents) +SELECT (random() * 10000)::int FROM generate_series(1, 1000); + +DELETE FROM public.orders WHERE id IN (SELECT id FROM public.orders ORDER BY id LIMIT 800); diff --git a/evals/o11y-investigate-postgres-autovacuum-saturated/EVAL.ts b/evals/o11y-investigate-postgres-autovacuum-saturated/EVAL.ts new file mode 100644 index 00000000..e66fe164 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-saturated/EVAL.ts @@ -0,0 +1,58 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: all 3 autovacuum_max_workers slots are continuously occupied. The pool +// is saturated; tables queue and dead tuples accumulate despite autovacuum being on. +// Fix: increase autovacuum_max_workers or tune vacuum cost settings to reduce +// per-worker time. +// BLOCKED: pg_stat_activity cannot be seeded in PGlite — snapshot tables used. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedSaturation = + /autovacuum.{0,30}(saturated|all.{0,10}worker|max.{0,10}worker.{0,20}full|pool.{0,10}full|busy)/i.test( + report + ); + const identifiedMaxWorkers = + /autovacuum_max_workers/i.test(report); + const proposedFix = + /autovacuum_max_workers|vacuum_cost|autovacuum_vacuum_cost/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified autovacuum worker pool is saturated', passed: identifiedSaturation }, + { name: 'referenced autovacuum_max_workers', passed: identifiedMaxWorkers }, + { name: 'proposed increasing max_workers or tuning cost settings', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that all 3 autovacuum_max_workers slots are continuously in use + (all workers busy for 10+ minutes), meaning the pool is saturated and tables + are waiting for vacuum attention. + 2. Proposed increasing autovacuum_max_workers or adjusting vacuum cost parameters + (autovacuum_vacuum_cost_delay, autovacuum_vacuum_cost_limit) to allow workers + to complete faster. + + Fail if the agent confused saturation with autovacuum being globally disabled, + did not identify the worker count issue, or gave no actionable tuning suggestion. + `, + }); + checks.push({ + name: 'correctly diagnosed autovacuum saturation and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-autovacuum-saturated/PROMPT.md b/evals/o11y-investigate-postgres-autovacuum-saturated/PROMPT.md new file mode 100644 index 00000000..57b2d49a --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-saturated/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Autovacuum is enabled and workers are running, but many tables still have growing dead-tuple counts. We think the autovacuum worker pool might be fully saturated — all workers are busy at all times and can't keep up with the workload. Can you check whether autovacuum is saturated and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_activity` reflects only harness-internal state. Session and GUC data has been exported to snapshot tables in the database. diff --git a/evals/o11y-investigate-postgres-autovacuum-saturated/remote/project.sql b/evals/o11y-investigate-postgres-autovacuum-saturated/remote/project.sql new file mode 100644 index 00000000..39718513 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-saturated/remote/project.sql @@ -0,0 +1,37 @@ +-- Starting state (probe: postgres-autovacuum-saturated). +-- All 3 autovacuum_max_workers slots are in use simultaneously and have been +-- for over 10 minutes — the pool is saturated and tables are queuing. +-- HARNESS NOTE: system views cannot be seeded in PGlite — snapshot tables used. + +CREATE TABLE public.pg_stat_activity_snapshot ( + pid int, + usename text, + application_name text, + state text, + wait_event_type text, + wait_event text, + query text, + query_start timestamptz, + backend_type text +); + +INSERT INTO public.pg_stat_activity_snapshot VALUES + (91001, 'autovacuum', '', 'active', NULL, NULL, + 'autovacuum: VACUUM public.orders (to prevent wraparound)', + now() - '12 minutes'::interval, 'autovacuum worker'), + (91002, 'autovacuum', '', 'active', NULL, NULL, + 'autovacuum: VACUUM public.events', + now() - '11 minutes 30 seconds'::interval, 'autovacuum worker'), + (91003, 'autovacuum', '', 'active', NULL, NULL, + 'autovacuum: VACUUM ANALYZE public.sessions', + now() - '10 minutes 45 seconds'::interval, 'autovacuum worker'); + +-- GUC settings for cross-reference +CREATE TABLE public.pg_settings_snapshot ( + name text, + setting text +); + +INSERT INTO public.pg_settings_snapshot VALUES + ('autovacuum', 'on'), + ('autovacuum_max_workers', '3'); diff --git a/evals/o11y-investigate-postgres-autovacuum-starved/EVAL.ts b/evals/o11y-investigate-postgres-autovacuum-starved/EVAL.ts new file mode 100644 index 00000000..9fcb1fb8 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-starved/EVAL.ts @@ -0,0 +1,57 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: audit_log (480k dead tuples, 5 days since autovacuum) and events +// (95k dead tuples, 3 days) are vacuum-starved despite autovacuum being enabled. +// Fix: manual VACUUM on the affected tables; tune autovacuum_vacuum_scale_factor +// or per-table storage parameters to trigger more frequent automatic vacuums. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedStarved = + /autovacuum.{0,30}(starv|not.{0,10}reach|behind|lag|dead.{0,20}tup)/i.test(report); + const identifiedTables = + /audit_log|events/.test(report); + const proposedFix = + /vacuum|autovacuum_vacuum_scale_factor|autovacuum_vacuum_threshold|storage.{0,20}parameter/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified vacuum-starved tables', passed: identifiedStarved }, + { name: 'named the affected tables (audit_log or events)', passed: identifiedTables }, + { name: 'proposed VACUUM or autovacuum tuning', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that audit_log (480,000 dead tuples, last autovacuum 5 days ago) + and/or events (95,000 dead tuples, last autovacuum 3 days ago) are + accumulating dead tuples faster than autovacuum is cleaning them. + 2. Proposed running VACUUM on the specific tables and/or lowering + autovacuum_vacuum_scale_factor (or autovacuum_vacuum_threshold) as a + per-table storage parameter to trigger more frequent cleanups. + + Fail if the agent did not name the specific starved tables, confused this + with global autovacuum being disabled, or gave no actionable remediation. + `, + }); + checks.push({ + name: 'correctly diagnosed autovacuum starvation and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-autovacuum-starved/PROMPT.md b/evals/o11y-investigate-postgres-autovacuum-starved/PROMPT.md new file mode 100644 index 00000000..3fcb47a7 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-starved/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Several tables have large dead-tuple counts and are growing despite autovacuum being enabled and not saturated. We think specific tables might be effectively starved — autovacuum workers aren't reaching them often enough to keep up with the write rate. Can you check whether any tables have accumulated excessive dead tuples without being vacuumed recently? + +Report what you find and propose a fix. + +> Note: `pg_stat_user_tables` reflects only harness-internal state. Table vacuum statistics have been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-autovacuum-starved/remote/project.sql b/evals/o11y-investigate-postgres-autovacuum-starved/remote/project.sql new file mode 100644 index 00000000..fdde5e89 --- /dev/null +++ b/evals/o11y-investigate-postgres-autovacuum-starved/remote/project.sql @@ -0,0 +1,25 @@ +-- Starting state (probe: postgres-autovacuum-starved). +-- public.audit_log has 480,000 dead tuples with no autovacuum in 5 days. +-- public.events has 95,000 dead tuples with no autovacuum in 3 days. +-- Both are above the default vacuum threshold (scale_factor=0.2 × reltuples + threshold=50). +-- HARNESS NOTE: pg_stat_user_tables system view cannot be seeded in PGlite. + +CREATE TABLE public.pg_stat_user_tables_snapshot ( + schemaname name, + relname name, + n_live_tup bigint, + n_dead_tup bigint, + n_mod_since_analyze bigint, + last_vacuum timestamptz, + last_autovacuum timestamptz, + last_analyze timestamptz, + last_autoanalyze timestamptz, + vacuum_count bigint, + autovacuum_count bigint +); + +INSERT INTO public.pg_stat_user_tables_snapshot VALUES + ('public', 'audit_log', 120000, 480000, 480000, NULL, now() - '5 days'::interval, NULL, now() - '5 days'::interval, 0, 2), + ('public', 'events', 85000, 95000, 95000, NULL, now() - '3 days'::interval, NULL, now() - '3 days'::interval, 0, 5), + ('public', 'users', 5000, 40, 80, NULL, now() - '2 hours'::interval, NULL, now() - '2 hours'::interval, 0, 48), + ('public', 'sessions', 12000, 200, 400, NULL, now() - '90 minutes'::interval, NULL, now() - '90 minutes'::interval, 0, 12); diff --git a/evals/o11y-investigate-postgres-blocking-chains/EVAL.ts b/evals/o11y-investigate-postgres-blocking-chains/EVAL.ts new file mode 100644 index 00000000..a61a288d --- /dev/null +++ b/evals/o11y-investigate-postgres-blocking-chains/EVAL.ts @@ -0,0 +1,57 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: pid 12345 holds an ExclusiveLock on orders in 'idle in transaction' +// state (4 minutes), blocking pids 23456 and 34567 in a lock wait chain. +// Fix: pg_terminate_backend(12345) to break the chain. +// BLOCKED: pg_locks and pg_stat_activity are system views in PGlite and cannot +// be seeded — eval uses snapshot tables. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedChain = + /lock.{0,30}(wait|chain|block)|blocking.{0,30}(chain|session|pid)/i.test(report); + const identifiedBlocker = + /\b12345\b/.test(report); + const proposedFix = + /pg_terminate_backend|terminate.{0,20}backend|kill.{0,20}(session|pid|connection)/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified a lock wait chain', passed: identifiedChain }, + { name: 'identified the blocking session (pid 12345)', passed: identifiedBlocker }, + { name: 'proposed terminating the blocking backend', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that pid 12345 is in an 'idle in transaction' state and is + holding a lock on the orders table, blocking pids 23456 and 34567. + 2. Proposed terminating the blocking session using pg_terminate_backend(12345) + to break the chain, and suggested investigating why the transaction was + left open (e.g. missing COMMIT or application bug). + + Fail if the agent did not identify pid 12345 as the root blocker, confused + the blocked sessions as the cause, or gave no actionable remediation. + `, + }); + checks.push({ + name: 'correctly diagnosed blocking chain and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-blocking-chains/PROMPT.md b/evals/o11y-investigate-postgres-blocking-chains/PROMPT.md new file mode 100644 index 00000000..122d3892 --- /dev/null +++ b/evals/o11y-investigate-postgres-blocking-chains/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We're seeing intermittent timeout errors across multiple queries and the app is becoming sluggish. Our ops team thinks there might be a lock chain where one blocked query is holding up several others. Can you check for lock wait chains in the database and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_locks` and `pg_stat_activity` reflect only harness-internal state. The lock wait chain data has been exported to snapshot tables in the database. diff --git a/evals/o11y-investigate-postgres-blocking-chains/remote/project.sql b/evals/o11y-investigate-postgres-blocking-chains/remote/project.sql new file mode 100644 index 00000000..31845b99 --- /dev/null +++ b/evals/o11y-investigate-postgres-blocking-chains/remote/project.sql @@ -0,0 +1,51 @@ +-- Starting state (probe: postgres-blocking-chains). +-- HARNESS NOTE: pg_locks and pg_stat_activity are system views in PGlite and +-- cannot be seeded with artificial lock state. The blocking chain is represented +-- in snapshot tables using text/int for PGlite-compatible types. + +-- Session snapshot: pid 12345 holds a row lock on orders; pid 23456 and pid 34567 +-- are waiting on it in a chain. +CREATE TABLE public.pg_stat_activity_snapshot ( + pid int, + usename text, + application_name text, + state text, + wait_event_type text, + wait_event text, + query text, + query_start timestamptz, + state_change timestamptz, + backend_start timestamptz +); + +INSERT INTO public.pg_stat_activity_snapshot VALUES + (12345, 'app_user', 'app-server', 'idle in transaction', NULL, NULL, + 'UPDATE public.orders SET status = ''processing'' WHERE id = 9001', + now() - '4 minutes'::interval, now() - '4 minutes'::interval, now() - '1 hour'::interval), + (23456, 'app_user', 'app-server', 'active', 'Lock', 'relation', + 'UPDATE public.orders SET status = ''shipped'' WHERE id = 9001', + now() - '3 minutes 50 seconds'::interval, now() - '3 minutes 50 seconds'::interval, now() - '1 hour'::interval), + (34567, 'reporting', 'metabase', 'active', 'Lock', 'relation', + 'SELECT * FROM public.orders WHERE status = ''processing''', + now() - '3 minutes 40 seconds'::interval, now() - '3 minutes 40 seconds'::interval, now() - '30 minutes'::interval); + +-- Lock snapshot: pid 12345 holds ExclusiveLock; pids 23456 and 34567 are waiting +CREATE TABLE public.pg_locks_snapshot ( + locktype text, + relation text, + mode text, + granted boolean, + pid int, + blocking_pid int +); + +INSERT INTO public.pg_locks_snapshot VALUES + ('tuple', 'orders', 'ExclusiveLock', true, 12345, NULL), + ('tuple', 'orders', 'ExclusiveLock', false, 23456, 12345), + ('relation', 'orders', 'RowShareLock', false, 34567, 12345); + +-- Actual tables for context +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + status text NOT NULL DEFAULT 'pending' +); diff --git a/evals/o11y-investigate-postgres-forced-checkpoints/EVAL.ts b/evals/o11y-investigate-postgres-forced-checkpoints/EVAL.ts new file mode 100644 index 00000000..ab8ea2b6 --- /dev/null +++ b/evals/o11y-investigate-postgres-forced-checkpoints/EVAL.ts @@ -0,0 +1,54 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: checkpoints_req=350, checkpoints_timed=100 → ~78% of checkpoints are forced. +// WAL is filling max_wal_size before scheduled checkpoints can run. +// Fix: increase max_wal_size, increase checkpoint_completion_target, or reduce +// write throughput via application-level batching. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedForced = + /forced.{0,20}checkpoint|checkpoints_req|checkpoint.{0,20}required/i.test(report); + const identifiedRatio = + /7[5-9]%|8[0-9]%|checkpoints_req.{0,20}(350|checkpoints_timed.{0,20}100)/i.test(report); + const proposedFix = + /max_wal_size|checkpoint_completion_target|checkpoint_timeout|wal/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified excessive forced checkpoints', passed: identifiedForced }, + { name: 'cited the checkpoints_req/timed ratio', passed: identifiedRatio }, + { name: 'proposed WAL or checkpoint tuning', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that checkpoints_req (350) far exceeds checkpoints_timed (100), + meaning ~78% of checkpoints are forced by WAL overflow rather than + scheduled by checkpoint_timeout. + 2. Proposed increasing max_wal_size or checkpoint_timeout, or adjusting + checkpoint_completion_target to spread I/O more evenly. + + Fail if the agent did not distinguish forced from scheduled checkpoints, + misread the direction of the ratio, or gave no WAL/checkpoint tuning advice. + `, + }); + checks.push({ + name: 'correctly diagnosed forced checkpoints and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-forced-checkpoints/PROMPT.md b/evals/o11y-investigate-postgres-forced-checkpoints/PROMPT.md new file mode 100644 index 00000000..ff5a554f --- /dev/null +++ b/evals/o11y-investigate-postgres-forced-checkpoints/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We're seeing high write latency spikes at irregular intervals and our I/O graphs show periodic bursts. Someone suggested this might be forced checkpoints caused by WAL filling up faster than scheduled checkpoints can keep up. Can you check whether we're seeing excessive forced checkpoints and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_bgwriter` and `pg_stat_checkpointer` are system views in PGlite and return zeroed values. Checkpoint statistics were captured at peak load and exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-forced-checkpoints/remote/project.sql b/evals/o11y-investigate-postgres-forced-checkpoints/remote/project.sql new file mode 100644 index 00000000..82b59938 --- /dev/null +++ b/evals/o11y-investigate-postgres-forced-checkpoints/remote/project.sql @@ -0,0 +1,19 @@ +-- Starting state (probe: postgres-forced-checkpoints). +-- checkpoints_req (forced) is much higher than checkpoints_timed (scheduled). +-- A healthy ratio is <5% forced; here ~78% are forced, indicating WAL overflow. +-- HARNESS NOTE: pg_stat_bgwriter / pg_stat_checkpointer are system views in PGlite +-- and return zeroed values. Stats are seeded in a snapshot table. +CREATE TABLE public.pg_stat_bgwriter_snapshot ( + checkpoints_timed bigint, + checkpoints_req bigint, + checkpoint_write_time double precision, + checkpoint_sync_time double precision, + buffers_checkpoint bigint, + buffers_clean bigint, + buffers_backend bigint, + stats_reset timestamptz +); + +-- checkpoints_req=350 vs checkpoints_timed=100: 78% forced checkpoints +INSERT INTO public.pg_stat_bgwriter_snapshot VALUES + (100, 350, 1250000.0, 85000.0, 4200000, 150000, 380000, now() - '7 days'::interval); diff --git a/evals/o11y-investigate-postgres-high-rollback-ratio/EVAL.ts b/evals/o11y-investigate-postgres-high-rollback-ratio/EVAL.ts new file mode 100644 index 00000000..6de34397 --- /dev/null +++ b/evals/o11y-investigate-postgres-high-rollback-ratio/EVAL.ts @@ -0,0 +1,56 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: 'app' database has xact_rollback=95000 vs xact_commit=155000 → ~38% +// rollback ratio. Normal is <1%. Likely cause: constraint violations, application +// errors, or deadlocks being silently swallowed. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedHighRollback = + /rollback.{0,30}(ratio|rate|high|38|xact_rollback)/i.test(report); + const citedNumbers = + /95[,.]?000|155[,.]?000|38%|0\.\s*38/i.test(report); + const proposedInvestigation = + /constraint|application.{0,20}error|deadlock|log|pg_stat_statements|error.{0,20}log/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified high rollback ratio', passed: identifiedHighRollback }, + { name: 'cited approximate ratio or raw numbers', passed: citedNumbers }, + { name: 'proposed investigating error source (logs, constraints, deadlocks)', passed: proposedInvestigation }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that the 'app' database has a rollback ratio of approximately + 38% (xact_rollback=95000 vs xact_commit=155000), far above the healthy + threshold of <1%. + 2. Proposed investigating the cause — checking application error logs, + looking for constraint violations in pg_stat_statements, or examining + deadlock patterns. + + Fail if the agent did not compute or cite the rollback ratio, confused + rollbacks with commits, or gave no direction for root-cause investigation. + `, + }); + checks.push({ + name: 'correctly diagnosed high rollback ratio and proposed investigation', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-high-rollback-ratio/PROMPT.md b/evals/o11y-investigate-postgres-high-rollback-ratio/PROMPT.md new file mode 100644 index 00000000..73e4c25e --- /dev/null +++ b/evals/o11y-investigate-postgres-high-rollback-ratio/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We're seeing unusual load on the database and our connection pool is under stress. Monitoring shows a lot of transaction activity, but query throughput doesn't seem to match — we think many transactions might be rolling back. Can you check whether there's a high transaction rollback ratio and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_database` reflects only harness-internal state. Transaction statistics have been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-high-rollback-ratio/remote/project.sql b/evals/o11y-investigate-postgres-high-rollback-ratio/remote/project.sql new file mode 100644 index 00000000..101b3656 --- /dev/null +++ b/evals/o11y-investigate-postgres-high-rollback-ratio/remote/project.sql @@ -0,0 +1,19 @@ +-- Starting state (probe: postgres-high-rollback-ratio). +-- The app database has a rollback ratio of ~38%: xact_rollback/(xact_commit+xact_rollback). +-- A healthy ratio is typically <1%. High rollbacks indicate application errors, +-- constraint violations, or deadlocks at scale. +-- HARNESS NOTE: pg_stat_database system view in PGlite — snapshot table used. + +CREATE TABLE public.pg_stat_database_snapshot ( + datname text, + xact_commit bigint, + xact_rollback bigint, + blks_hit bigint, + blks_read bigint, + numbackends int +); + +-- xact_rollback=95000, xact_commit=155000 → 38% rollback rate +INSERT INTO public.pg_stat_database_snapshot VALUES + ('app', 155000, 95000, 2600000, 50000, 45), + ('postgres', 1200, 3, 980000, 800, 2); diff --git a/evals/o11y-investigate-postgres-idle-in-transaction/EVAL.ts b/evals/o11y-investigate-postgres-idle-in-transaction/EVAL.ts new file mode 100644 index 00000000..cd9451f1 --- /dev/null +++ b/evals/o11y-investigate-postgres-idle-in-transaction/EVAL.ts @@ -0,0 +1,58 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: pid 55321 has been 'idle in transaction' for 47 minutes. This holds back +// the oldest transaction horizon across the entire cluster, preventing VACUUM from +// reclaiming dead tuples on any table. +// Fix: pg_terminate_backend(55321) and set idle_in_transaction_session_timeout. +// BLOCKED: pg_stat_activity system view in PGlite cannot be seeded — eval uses +// public.pg_stat_activity_snapshot. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedIdle = + /idle.{0,20}in.{0,20}transaction|idle_in_transaction/i.test(report); + const identifiedSession = + /\b55321\b/.test(report); + const proposedFix = + /pg_terminate_backend|idle_in_transaction_session_timeout|terminate.{0,20}(session|backend)/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified idle-in-transaction session', passed: identifiedIdle }, + { name: 'identified the offending session (pid 55321)', passed: identifiedSession }, + { name: 'proposed terminating or preventing idle-in-transaction sessions', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that pid 55321 has been in 'idle in transaction' state for + approximately 47 minutes and is holding back the vacuum horizon. + 2. Proposed terminating the session with pg_terminate_backend(55321) and/or + setting idle_in_transaction_session_timeout to automatically terminate + such sessions in the future. + + Fail if the agent did not identify the specific session, did not explain the + vacuum horizon impact, or gave no actionable remediation. + `, + }); + checks.push({ + name: 'correctly diagnosed idle-in-transaction and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-idle-in-transaction/PROMPT.md b/evals/o11y-investigate-postgres-idle-in-transaction/PROMPT.md new file mode 100644 index 00000000..58e4544f --- /dev/null +++ b/evals/o11y-investigate-postgres-idle-in-transaction/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Table vacuums are falling behind and we're accumulating dead tuples faster than autovacuum can clean them up. A teammate mentioned that long-running idle-in-transaction sessions can hold back the oldest transaction horizon and prevent VACUUM from making progress. Can you check for sessions that have been idle in transaction for an excessive amount of time and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_activity` reflects only harness-internal state. Session data has been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-idle-in-transaction/remote/project.sql b/evals/o11y-investigate-postgres-idle-in-transaction/remote/project.sql new file mode 100644 index 00000000..81757830 --- /dev/null +++ b/evals/o11y-investigate-postgres-idle-in-transaction/remote/project.sql @@ -0,0 +1,46 @@ +-- Starting state (probe: postgres-idle-in-transaction). +-- HARNESS NOTE: pg_stat_activity is a system view in PGlite and reflects only +-- internal harness state. Session data is seeded in a snapshot table. + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id bigint NOT NULL, + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE public.pg_stat_activity_snapshot ( + pid int, + usename text, + application_name text, + state text, + wait_event_type text, + wait_event text, + query text, + query_start timestamptz, + state_change timestamptz, + xact_start timestamptz, + backend_start timestamptz +); + +-- pid 55321: idle in transaction for 47 minutes — holding back vacuum horizon +-- across the entire cluster, not just its own table. +INSERT INTO public.pg_stat_activity_snapshot VALUES + (55321, 'app_user', 'rails-app', 'idle in transaction', NULL, NULL, + 'SELECT id FROM public.orders WHERE id = 42', + now() - '47 minutes'::interval, + now() - '47 minutes'::interval, + now() - '47 minutes'::interval, + now() - '2 hours'::interval), + (55322, 'app_user', 'rails-app', 'active', NULL, NULL, + 'INSERT INTO public.orders (customer_id, total_cents) VALUES ($1, $2)', + now() - '120 milliseconds'::interval, + now() - '120 milliseconds'::interval, + now() - '120 milliseconds'::interval, + now() - '2 hours'::interval), + (55323, 'readonly', 'dashboard', 'idle', NULL, NULL, + 'SELECT count(*) FROM public.orders', + now() - '5 seconds'::interval, + now() - '4 seconds'::interval, + NULL, + now() - '30 minutes'::interval); diff --git a/evals/o11y-investigate-postgres-index-invalid/EVAL.ts b/evals/o11y-investigate-postgres-index-invalid/EVAL.ts new file mode 100644 index 00000000..74560ada --- /dev/null +++ b/evals/o11y-investigate-postgres-index-invalid/EVAL.ts @@ -0,0 +1,55 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: orders_customer_id_created_at_idx has indisvalid=false, left behind by a +// failed CREATE INDEX CONCURRENTLY. It wastes space and may cause query errors. +// Fix: DROP INDEX CONCURRENTLY orders_customer_id_created_at_idx; then recreate. +// BLOCKED: pg_index system catalog in PGlite cannot have indisvalid set to false — +// eval uses public.pg_index_snapshot to represent the invalid index state. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedInvalid = + /invalid.{0,30}index|indisvalid|index.{0,30}invalid/i.test(report); + const identifiedIndex = + /orders_customer_id_created_at_idx/i.test(report); + const proposedFix = + /drop\s+index|reindex/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified invalid index condition', passed: identifiedInvalid }, + { name: 'named the specific invalid index', passed: identifiedIndex }, + { name: 'proposed DROP INDEX or REINDEX', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified orders_customer_id_created_at_idx as an invalid index + (indisvalid = false) on the orders table, explaining that it was likely + left behind by a failed CREATE INDEX CONCURRENTLY. + 2. Proposed dropping it with DROP INDEX CONCURRENTLY and recreating it, or + using REINDEX INDEX CONCURRENTLY to rebuild it in place. + + Fail if the agent did not name the specific index, confused it with a valid + index, or gave no actionable remediation. + `, + }); + checks.push({ + name: 'correctly diagnosed invalid index and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-index-invalid/PROMPT.md b/evals/o11y-investigate-postgres-index-invalid/PROMPT.md new file mode 100644 index 00000000..59cfe571 --- /dev/null +++ b/evals/o11y-investigate-postgres-index-invalid/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We tried to add an index on a busy table last week using `CREATE INDEX CONCURRENTLY` but the operation was interrupted. We're not sure if the index was left in a broken state. Can you check whether there are any invalid indexes in the database and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_index` does not reflect indexes with `indisvalid = false` in the eval harness the same way a real instance would after a failed `CREATE INDEX CONCURRENTLY`. The index state has been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-index-invalid/remote/project.sql b/evals/o11y-investigate-postgres-index-invalid/remote/project.sql new file mode 100644 index 00000000..73e7dc36 --- /dev/null +++ b/evals/o11y-investigate-postgres-index-invalid/remote/project.sql @@ -0,0 +1,40 @@ +-- Starting state (probe: postgres-index-invalid). +-- HARNESS NOTE: pg_index is a system catalog in PGlite; indisvalid cannot be set +-- to false via DDL. The invalid index state has been exported to a snapshot table. +CREATE TABLE public.customers ( + id bigserial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + customer_id bigint NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +-- Valid index for contrast +CREATE INDEX orders_created_at_idx ON public.orders (created_at); + +-- Snapshot table mirroring pg_indexes + pg_index.indisvalid +CREATE TABLE public.pg_index_snapshot ( + schemaname text, + tablename text, + indexname text, + indexdef text, + indisvalid boolean, + indisready boolean +); + +-- The interrupted CREATE INDEX CONCURRENTLY left this index in an invalid state. +-- It wastes space and may cause query errors on some Postgres versions. +INSERT INTO public.pg_index_snapshot VALUES + ('public', 'orders', 'orders_customer_id_created_at_idx', + 'CREATE INDEX orders_customer_id_created_at_idx ON public.orders USING btree (customer_id, created_at)', + false, false), + ('public', 'orders', 'orders_created_at_idx', + 'CREATE INDEX orders_created_at_idx ON public.orders USING btree (created_at)', + true, true), + ('public', 'customers', 'customers_pkey', + 'CREATE UNIQUE INDEX customers_pkey ON public.customers USING btree (id)', + true, true); diff --git a/evals/o11y-investigate-postgres-low-cache-hit-ratio/EVAL.ts b/evals/o11y-investigate-postgres-low-cache-hit-ratio/EVAL.ts new file mode 100644 index 00000000..79e53440 --- /dev/null +++ b/evals/o11y-investigate-postgres-low-cache-hit-ratio/EVAL.ts @@ -0,0 +1,53 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: 'app' database has a cache hit ratio of ~72% (blks_hit=2600000, +// blks_read=1000000). Healthy threshold is ~99%. Most reads are going to disk. +// Fix: increase shared_buffers or effective_cache_size, or upgrade compute tier. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedLowHit = + /cache.{0,20}hit|blks_hit|buffer.{0,20}(hit|cache)/i.test(report); + const identifiedRatio = + /7[0-9](\.[0-9]+)?%|0\.7[0-9]/.test(report); + const proposedFix = + /shared_buffers|effective_cache_size|compute|memory|upgrade|pg_prewarm/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified low buffer cache hit ratio', passed: identifiedLowHit }, + { name: 'cited the approximate ratio (~72%)', passed: identifiedRatio }, + { name: 'proposed increasing memory or cache-related fix', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that the 'app' database has a buffer cache hit ratio of + approximately 72% (computed from blks_hit and blks_read), which is far + below the ~99% healthy threshold and indicates excessive disk I/O. + 2. Proposed increasing shared_buffers, effective_cache_size, or upgrading + to a compute tier with more RAM to reduce disk reads. + + Fail if the agent did not compute or cite the hit ratio, confused blks_hit + with blks_read, or gave no memory/tuning recommendation. + `, + }); + checks.push({ + name: 'correctly diagnosed low cache hit ratio and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-low-cache-hit-ratio/PROMPT.md b/evals/o11y-investigate-postgres-low-cache-hit-ratio/PROMPT.md new file mode 100644 index 00000000..de5e19da --- /dev/null +++ b/evals/o11y-investigate-postgres-low-cache-hit-ratio/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Query times have been higher than usual and our database server seems to be under more I/O load than expected. We think the buffer cache hit rate might be very low, meaning Postgres is reading data from disk instead of memory for most queries. Can you check the cache hit ratio and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_database` reflects only harness-internal state. Cumulative I/O statistics have been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-low-cache-hit-ratio/remote/project.sql b/evals/o11y-investigate-postgres-low-cache-hit-ratio/remote/project.sql new file mode 100644 index 00000000..94977ad6 --- /dev/null +++ b/evals/o11y-investigate-postgres-low-cache-hit-ratio/remote/project.sql @@ -0,0 +1,18 @@ +-- Starting state (probe: postgres-low-cache-hit-ratio). +-- The app database has a cache hit rate of ~72% (blks_hit / (blks_hit + blks_read)), +-- well below the healthy threshold of ~99%. Postgres is going to disk for most reads. +-- HARNESS NOTE: pg_stat_database is a system view in PGlite — snapshot table used. + +CREATE TABLE public.pg_stat_database_snapshot ( + datname text, + blks_hit bigint, + blks_read bigint, + xact_commit bigint, + xact_rollback bigint, + numbackends int +); + +-- ~72% cache hit rate: blks_hit / (blks_hit + blks_read) = 2600000 / (2600000 + 1000000) +INSERT INTO public.pg_stat_database_snapshot VALUES + ('app', 2600000, 1000000, 850000, 12000, 45), + ('postgres', 980000, 800, 1200, 5, 2); diff --git a/evals/o11y-investigate-postgres-prepared-transaction-abandoned/EVAL.ts b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/EVAL.ts new file mode 100644 index 00000000..38297205 --- /dev/null +++ b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/EVAL.ts @@ -0,0 +1,55 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: prepared transaction 'txn_payment_abc123' (xid 8834721) has been sitting +// for 6+ hours. Abandoned prepared transactions hold their xmin, blocking VACUUM +// and wraparound protection. +// Fix: ROLLBACK PREPARED 'txn_payment_abc123' (or COMMIT PREPARED if recoverable). +// BLOCKED: pg_prepared_xacts system view cannot be seeded in PGlite — snapshot used. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedPrepared = + /prepared.{0,20}(transaction|xact)|pg_prepared_xacts/i.test(report); + const identifiedOrphan = + /txn_payment_abc123|abandoned.{0,30}prepared|orphan.{0,30}prepared/i.test(report); + const proposedFix = + /rollback\s+prepared|commit\s+prepared/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified abandoned prepared transaction', passed: identifiedPrepared }, + { name: 'identified the specific transaction (txn_payment_abc123)', passed: identifiedOrphan }, + { name: 'proposed ROLLBACK PREPARED or COMMIT PREPARED', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that 'txn_payment_abc123' is an abandoned prepared transaction + that has been sitting for over 6 hours, and explained that it holds an + old xmin blocking VACUUM and wraparound protection. + 2. Proposed resolving it with ROLLBACK PREPARED 'txn_payment_abc123' (or + COMMIT PREPARED if the transaction's intent can be determined). + + Fail if the agent did not name the specific transaction, confused it with a + regular long-running query, or proposed no way to resolve it. + `, + }); + checks.push({ + name: 'correctly diagnosed abandoned prepared transaction and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-prepared-transaction-abandoned/PROMPT.md b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/PROMPT.md new file mode 100644 index 00000000..1b4f86bf --- /dev/null +++ b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We noticed our autovacuum is not keeping up and transaction ID ages are creeping up faster than expected. A colleague mentioned orphaned prepared transactions can hold back wraparound protection. Can you check whether there are any abandoned prepared transactions and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_prepared_xacts` cannot be seeded with artificial transactions in the eval harness. The prepared transaction state has been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-prepared-transaction-abandoned/remote/project.sql b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/remote/project.sql new file mode 100644 index 00000000..991232de --- /dev/null +++ b/evals/o11y-investigate-postgres-prepared-transaction-abandoned/remote/project.sql @@ -0,0 +1,16 @@ +-- Starting state (probe: postgres-prepared-transaction-abandoned). +-- An orphaned prepared transaction ('txn_payment_abc123') has been sitting for +-- over 6 hours. It holds its xmin, blocking wraparound protection and VACUUM +-- across the entire database cluster. +-- HARNESS NOTE: pg_prepared_xacts is a system view in PGlite; snapshot table used. + +CREATE TABLE public.pg_prepared_xacts_snapshot ( + transaction bigint, + gid text, + prepared timestamptz, + owner text, + database text +); + +INSERT INTO public.pg_prepared_xacts_snapshot VALUES + (8834721, 'txn_payment_abc123', now() - '6 hours 12 minutes'::interval, 'app_user', 'postgres'); diff --git a/evals/o11y-investigate-postgres-replica-lag/EVAL.ts b/evals/o11y-investigate-postgres-replica-lag/EVAL.ts new file mode 100644 index 00000000..27bb00dc --- /dev/null +++ b/evals/o11y-investigate-postgres-replica-lag/EVAL.ts @@ -0,0 +1,57 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: standby-1 has replay_lag = 00:04:32 (4 minutes 32 seconds). +// The replica is falling behind and returning stale data to analytics queries. +// Fix: investigate I/O capacity on replica, check for long-running queries on +// standby blocking replay, or route time-sensitive reads to primary. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedLag = + /replay.{0,20}lag|replication.{0,20}lag|standby.{0,20}(behind|lag)/i.test(report); + const citedDuration = + /4.{0,5}(minute|min)|00:04|272\s*(second|sec)/i.test(report); + const proposedFix = + /primary|route.{0,20}(read|query)|io.{0,20}(capacity|throughput)|conflict|hot.{0,10}standby|max_standby/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified replica replication lag', passed: identifiedLag }, + { name: 'cited the approximate lag duration (~4 min 32s)', passed: citedDuration }, + { name: 'proposed investigation or mitigation', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that standby-1 has a replay_lag of approximately 4 minutes 32 + seconds, meaning the replica is significantly behind the primary and + returning stale data. + 2. Proposed at least one mitigation: routing time-sensitive reads to the + primary, investigating I/O capacity or query conflicts on the replica, + or checking for hot standby conflict settings (max_standby_streaming_delay). + + Fail if the agent did not cite the specific lag duration, confused write_lag + with replay_lag, or gave no actionable direction. + `, + }); + checks.push({ + name: 'correctly diagnosed replica lag and proposed mitigation', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-replica-lag/PROMPT.md b/evals/o11y-investigate-postgres-replica-lag/PROMPT.md new file mode 100644 index 00000000..d9efb988 --- /dev/null +++ b/evals/o11y-investigate-postgres-replica-lag/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Our application connects to a read replica for analytics queries, but users are reporting stale data — results that should reflect recent writes are returning old values. We suspect the replica has fallen significantly behind the primary. Can you check the replication lag and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_replication` reflects only harness-internal state. Replication lag data has been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-replica-lag/remote/project.sql b/evals/o11y-investigate-postgres-replica-lag/remote/project.sql new file mode 100644 index 00000000..32cb9fb8 --- /dev/null +++ b/evals/o11y-investigate-postgres-replica-lag/remote/project.sql @@ -0,0 +1,28 @@ +-- Starting state (probe: postgres-replica-lag). +-- The standby replica has a replay_lag of 4 minutes 32 seconds, well above +-- an acceptable threshold of ~1 second for most applications. +-- HARNESS NOTE: pg_stat_replication is a system view in PGlite — snapshot table used. + +CREATE TABLE public.pg_stat_replication_snapshot ( + pid int, + usename text, + application_name text, + client_addr text, + state text, + sent_lsn text, + write_lsn text, + flush_lsn text, + replay_lsn text, + write_lag interval, + flush_lag interval, + replay_lag interval, + sync_state text +); + +INSERT INTO public.pg_stat_replication_snapshot VALUES + (65001, 'replicator', 'standby-1', '10.0.1.42', 'streaming', + '1/A4F00000', '1/A4F00000', '1/A3200000', '1/98E00000', + '00:00:00.8', + '00:00:01.2', + '00:04:32', + 'async'); diff --git a/evals/o11y-investigate-postgres-sequence-exhaustion/EVAL.ts b/evals/o11y-investigate-postgres-sequence-exhaustion/EVAL.ts new file mode 100644 index 00000000..9b16da0e --- /dev/null +++ b/evals/o11y-investigate-postgres-sequence-exhaustion/EVAL.ts @@ -0,0 +1,75 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolEvalContext, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: events_id_seq is an int4 identity sequence advanced to ~2147483640, +// leaving fewer than 10 values before insert failures begin. +// Fix: ALTER TABLE public.events ALTER COLUMN id TYPE bigint, or cycle the +// sequence if temporary relief is acceptable. + +const scorer: ToolScorer = async (ctx) => { + try { + const seqCheck = await checkSequenceNearExhaustion(ctx); + const checks: CheckResult[] = [seqCheck]; + + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that the events_id_seq sequence (backing the int4 identity column + on public.events) is near its maximum value of 2,147,483,647 and will cause + insert failures imminently. + 2. Proposed changing the column type to bigint (ALTER TABLE public.events ALTER + COLUMN id TYPE bigint) or another concrete fix such as creating a new bigint + sequence and migrating the column. + + Fail if the agent only described the general concept of sequence exhaustion + without identifying events_id_seq specifically, or proposed no actionable fix. + `, + }); + checks.push({ + name: 'correctly diagnosed sequence exhaustion and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + return { + passed: false, + checks: [{ name: 'scorer evaluated sequence exhaustion', passed: false, notes: msg }], + }; + } +}; + +export default scorer; + +async function checkSequenceNearExhaustion(ctx: ToolEvalContext): Promise { + const { rows } = await ctx.query(` + SELECT last_value, max_value + FROM pg_sequences + WHERE sequencename = 'events_id_seq' + AND schemaname = 'public'; + `); + const row = rows[0]; + if (!row) { + return { + name: 'events_id_seq exists and is near exhaustion', + passed: false, + notes: 'sequence not found', + }; + } + const remaining = Number(row.max_value) - Number(row.last_value); + return { + name: 'events_id_seq exists and is near exhaustion', + passed: remaining < 1000, + notes: `last_value=${row.last_value}, max_value=${row.max_value}, remaining=${remaining}`, + }; +} diff --git a/evals/o11y-investigate-postgres-sequence-exhaustion/PROMPT.md b/evals/o11y-investigate-postgres-sequence-exhaustion/PROMPT.md new file mode 100644 index 00000000..241528ae --- /dev/null +++ b/evals/o11y-investigate-postgres-sequence-exhaustion/PROMPT.md @@ -0,0 +1,15 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We have a high-traffic `events` table and recently noticed insert failures starting to appear in our logs. We suspect a sequence might be hitting its maximum value. Can you check whether any sequences are close to exhaustion and tell us what to do? + +Report what you find and propose a fix. diff --git a/evals/o11y-investigate-postgres-sequence-exhaustion/remote/project.sql b/evals/o11y-investigate-postgres-sequence-exhaustion/remote/project.sql new file mode 100644 index 00000000..931e1c26 --- /dev/null +++ b/evals/o11y-investigate-postgres-sequence-exhaustion/remote/project.sql @@ -0,0 +1,28 @@ +-- Starting state (probe: postgres-sequence-exhaustion). +-- An int4 identity column on public.events has a sequence near the 2^31-1 limit. +-- The next batch of inserts will fail with "integer out of range". + +CREATE TABLE public.users ( + id bigserial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +-- int4 identity column — max value is 2,147,483,647 +CREATE TABLE public.events ( + id int GENERATED ALWAYS AS IDENTITY, + user_id bigint NOT NULL REFERENCES public.users(id), + event_type text NOT NULL, + created_at timestamptz DEFAULT now() +); + +-- Advance the sequence to within a few hundred of the int4 max +-- by restarting it near the ceiling +ALTER SEQUENCE public.events_id_seq RESTART WITH 2147483600; + +INSERT INTO public.users (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 10) AS g; + +-- Consume values up to ~2147483640 to leave ~7 remaining +INSERT INTO public.events (user_id, event_type) +SELECT (g % 10) + 1, 'page_view' +FROM generate_series(1, 40) AS g; diff --git a/evals/o11y-investigate-postgres-txid-wraparound/EVAL.ts b/evals/o11y-investigate-postgres-txid-wraparound/EVAL.ts new file mode 100644 index 00000000..49d5b17b --- /dev/null +++ b/evals/o11y-investigate-postgres-txid-wraparound/EVAL.ts @@ -0,0 +1,55 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: 'app' database has age(datfrozenxid) = 1.95B, critically close to the +// ~2.1B wraparound limit. Postgres will refuse writes to protect itself. +// BLOCKED: pg_database system catalog in PGlite reflects only harness txn state — +// eval uses public.pg_database_snapshot to represent production values. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedWraparound = + /wraparound|txid.{0,30}exhaust|age.{0,30}(datfrozenxid|frozenxid)|1[\.,]?9[0-9]{2}[\.,]?[0-9]{3}[\.,]?[0-9]{3}/i.test( + report + ); + const identifiedDatabase = /\bapp\b/.test(report); + const proposedFix = + /vacuum\s+freeze|autovacuum_freeze_max_age|alter\s+system|pg_stat_user_tables/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified txid wraparound risk', passed: identifiedWraparound }, + { name: 'identified the affected database (app)', passed: identifiedDatabase }, + { name: 'proposed VACUUM FREEZE or a relevant mitigation', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that the 'app' database has an age(datfrozenxid) of approximately + 1.95 billion, which is critically close to the ~2.1 billion wraparound limit + where Postgres will stop accepting writes. + 2. Proposed running VACUUM FREEZE on the affected database or tables, or + suggested adjusting autovacuum_freeze_max_age to trigger preventive freezing. + + Fail if the agent misidentified the problem, did not name the 'app' database + specifically, or only gave generic advice without citing the snapshot values. + `, + }); + checks.push({ + name: 'correctly diagnosed txid wraparound and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-txid-wraparound/PROMPT.md b/evals/o11y-investigate-postgres-txid-wraparound/PROMPT.md new file mode 100644 index 00000000..48b0d990 --- /dev/null +++ b/evals/o11y-investigate-postgres-txid-wraparound/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Our database has been running for a long time without a VACUUM FREEZE pass and we're worried about transaction ID wraparound. Postgres will refuse writes and shut down to protect data integrity if the transaction ID counter gets too close to the limit. Can you check whether any database is approaching transaction ID exhaustion and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_database` system catalog values reflect the eval harness state and won't show production-realistic ages. Transaction age data has been exported to a snapshot table in the database. diff --git a/evals/o11y-investigate-postgres-txid-wraparound/remote/project.sql b/evals/o11y-investigate-postgres-txid-wraparound/remote/project.sql new file mode 100644 index 00000000..fbe6ca52 --- /dev/null +++ b/evals/o11y-investigate-postgres-txid-wraparound/remote/project.sql @@ -0,0 +1,18 @@ +-- Starting state (probe: postgres-txid-wraparound). +-- HARNESS NOTE: pg_database is a system catalog in PGlite and reflects only +-- the harness's own transaction state, not production values. The wraparound +-- proximity data has been exported to a snapshot table using bigint for +-- PGlite-compatible types (xid is not available). +CREATE TABLE public.pg_database_snapshot ( + datname text, + age_datfrozenxid bigint, + age_datminmxid bigint +); + +-- 'app' database is at 1.95B — critically close to the 2.1B autovacuum_freeze_max_age +-- default, and well past the 200M warning threshold. Postgres will start refusing +-- connections to protect itself if age reaches ~2.1B. +INSERT INTO public.pg_database_snapshot VALUES + ('app', 1950000000, 180000000), + ('postgres', 210000, 95000), + ('template1', 210000, 95000); diff --git a/evals/o11y-investigate-postgres-vacuum-horizon-blocked/EVAL.ts b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/EVAL.ts new file mode 100644 index 00000000..cf1dca3d --- /dev/null +++ b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/EVAL.ts @@ -0,0 +1,64 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: (1) pid 77001 has been running a read query for 2h15m, holding an old +// xmin snapshot; (2) stale_slot is an inactive logical replication slot also +// holding the vacuum horizon. Either blocker prevents VACUUM from removing dead +// tuples across the cluster. +// Fix: terminate pid 77001; drop stale_slot. +// BLOCKED: system views cannot be seeded in PGlite — eval uses snapshot tables. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedHorizonBlock = + /vacuum.{0,40}(horizon|block|hold|prevent)|old.{0,20}(transaction|xmin|snapshot).{0,20}(block|hold|prevent)/i.test( + report + ); + const identifiedLongTxn = + /\b77001\b|long.{0,20}(running|transaction|query).{0,30}(block|hold|vacuum)/i.test(report); + const identifiedStaleSlot = + /stale_slot|inactive.{0,20}(slot|replication)|replication.{0,20}slot.{0,20}(hold|block|inactive)/i.test( + report + ); + const proposedFix = + /pg_terminate_backend|pg_drop_replication_slot|drop.{0,20}slot/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified vacuum horizon is blocked', passed: identifiedHorizonBlock }, + { name: 'identified long-running transaction (pid 77001)', passed: identifiedLongTxn }, + { name: 'identified inactive replication slot (stale_slot)', passed: identifiedStaleSlot }, + { name: 'proposed terminating backend or dropping the slot', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent identified at least one of the two vacuum horizon blockers: + (1) pid 77001 has been running a read transaction for over 2 hours, holding + an old xmin and preventing VACUUM from removing dead tuples, OR + (2) stale_slot is an inactive logical replication slot also holding the horizon. + + The agent should have proposed terminating pid 77001 (pg_terminate_backend) + and/or dropping stale_slot (pg_drop_replication_slot) as the fix. + + Fail if the agent did not explain the vacuum horizon mechanism, gave only + generic advice about running VACUUM manually, or identified neither specific blocker. + `, + }); + checks.push({ + name: 'correctly diagnosed vacuum horizon blocker and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-postgres-vacuum-horizon-blocked/PROMPT.md b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/PROMPT.md new file mode 100644 index 00000000..3eca4f77 --- /dev/null +++ b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/PROMPT.md @@ -0,0 +1,17 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +Dead tuple counts on several tables keep climbing even though autovacuum appears to be running. We suspect something is holding back the oldest transaction horizon and preventing VACUUM from removing dead rows. Can you investigate what's blocking vacuum progress and tell us what to do? + +Report what you find and propose a fix. + +> Note: `pg_stat_activity` and `pg_replication_slots` reflect only harness-internal state. The relevant session and slot data has been exported to snapshot tables in the database. diff --git a/evals/o11y-investigate-postgres-vacuum-horizon-blocked/remote/project.sql b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/remote/project.sql new file mode 100644 index 00000000..6911caba --- /dev/null +++ b/evals/o11y-investigate-postgres-vacuum-horizon-blocked/remote/project.sql @@ -0,0 +1,54 @@ +-- Starting state (probe: postgres-vacuum-horizon-blocked). +-- Two blockers: (1) a long-running read transaction (pid 77001) holding an old +-- xmin, and (2) an inactive replication slot (stale_slot) also holding the horizon. +-- Either alone would prevent VACUUM from removing dead tuples. +-- HARNESS NOTE: system views cannot be seeded in PGlite; snapshot tables are used. + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + status text NOT NULL DEFAULT 'pending', + created_at timestamptz DEFAULT now() +); + +-- Long-running read transaction holding an old snapshot +CREATE TABLE public.pg_stat_activity_snapshot ( + pid int, + usename text, + application_name text, + state text, + wait_event_type text, + wait_event text, + query text, + xact_start timestamptz, + query_start timestamptz, + state_change timestamptz +); + +INSERT INTO public.pg_stat_activity_snapshot VALUES + (77001, 'reporting', 'analytics-job', 'active', NULL, NULL, + 'SELECT * FROM public.orders', + now() - '2 hours 15 minutes'::interval, + now() - '2 hours 15 minutes'::interval, + now() - '2 hours 15 minutes'::interval), + (77002, 'app_user', 'api-server', 'idle', NULL, NULL, + 'SELECT 1', + now() - '10 seconds'::interval, + now() - '10 seconds'::interval, + now() - '5 seconds'::interval); + +-- Inactive replication slot also holding WAL and the vacuum horizon +CREATE TABLE public.replication_slots_snapshot ( + slot_name text, + plugin text, + slot_type text, + active boolean, + active_pid int, + restart_lsn text, + confirmed_flush_lsn text, + wal_status text, + safe_wal_size bigint +); + +INSERT INTO public.replication_slots_snapshot VALUES + ('stale_slot', 'pgoutput', 'logical', false, null, + '0/A000000', '0/B000000', 'lost', -1073741824); diff --git a/evals/o11y-resolve-postgres-random-page-cost/EVAL.ts b/evals/o11y-resolve-postgres-random-page-cost/EVAL.ts new file mode 100644 index 00000000..dd896863 --- /dev/null +++ b/evals/o11y-resolve-postgres-random-page-cost/EVAL.ts @@ -0,0 +1,55 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: random_page_cost = 4.0 (spinning-disk default). On SSD-backed Supabase, +// it should be 1.1 so the planner prefers index scans appropriately. +// Fix: ALTER SYSTEM SET random_page_cost = 1.1 (or ALTER ROLE / ALTER DATABASE). +// NOTE: pg_settings is available in PGlite and shows the default of 4.0 — +// no fake table needed. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedHighCost = + /random_page_cost.{0,30}(4\.0|4|too.{0,10}high|spinning|hdd|disk)/i.test(report); + const proposedCorrectValue = + /1\.1|random_page_cost.{0,30}(lower|reduce|ssd)/i.test(report); + const proposedFix = + /alter.{0,10}(system|role|database).{0,30}random_page_cost/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified random_page_cost is set to the HDD default (4.0)', passed: identifiedHighCost }, + { name: 'proposed lowering it to ~1.1 for SSD', passed: proposedCorrectValue }, + { name: 'included a concrete ALTER SQL command', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that random_page_cost is 4.0 (the default calibrated for + spinning HDDs) and explained that Supabase runs on SSDs where random + I/O is much cheaper. + 2. Proposed setting it to approximately 1.1 via ALTER SYSTEM SET + random_page_cost = 1.1 or ALTER DATABASE / ALTER ROLE equivalent. + + Fail if the agent did not cite the current value, did not explain the SSD + vs HDD distinction, or gave no SQL command with the corrected value. + `, + }); + checks.push({ + name: 'correctly diagnosed high random_page_cost and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-resolve-postgres-random-page-cost/PROMPT.md b/evals/o11y-resolve-postgres-random-page-cost/PROMPT.md new file mode 100644 index 00000000..a4b642c9 --- /dev/null +++ b/evals/o11y-resolve-postgres-random-page-cost/PROMPT.md @@ -0,0 +1,15 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +The query planner sometimes chooses sequential scans on tables where we'd expect an index scan. We're running on SSD-backed storage (Supabase), but we're wondering if the planner's cost model is calibrated for spinning disks. Can you check the `random_page_cost` setting and tell us what to do? + +Report what you find and propose a fix. diff --git a/evals/o11y-resolve-postgres-random-page-cost/remote/project.sql b/evals/o11y-resolve-postgres-random-page-cost/remote/project.sql new file mode 100644 index 00000000..9290e507 --- /dev/null +++ b/evals/o11y-resolve-postgres-random-page-cost/remote/project.sql @@ -0,0 +1,7 @@ +-- Starting state (probe: postgres-random-page-cost-high). +-- random_page_cost defaults to 4.0, calibrated for spinning HDDs. On SSD-backed +-- Supabase instances, the correct value is 1.1, which lets the planner prefer +-- index scans over sequential scans appropriately. +-- pg_settings IS available in PGlite and will show the default value of 4.0 — +-- no additional seeding needed. +SELECT 1; diff --git a/evals/o11y-resolve-postgres-statement-timeout-unset/EVAL.ts b/evals/o11y-resolve-postgres-statement-timeout-unset/EVAL.ts new file mode 100644 index 00000000..bf782f28 --- /dev/null +++ b/evals/o11y-resolve-postgres-statement-timeout-unset/EVAL.ts @@ -0,0 +1,54 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: statement_timeout = 0 (no timeout). Runaway queries can hold connections +// indefinitely, causing outages. +// Fix: ALTER ROLE postgres SET statement_timeout = '30s' (or similar reasonable value). +// NOTE: pg_settings is available in PGlite and will show the default value of 0 — +// no fake table needed. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const identifiedUnset = + /statement_timeout.{0,30}(0|unset|not set|no timeout|disabled)/i.test(report); + const proposedFix = + /alter.{0,10}(role|system|database).{0,30}statement_timeout|statement_timeout\s*=\s*'\d/i.test( + report + ); + + const checks: CheckResult[] = [ + { name: 'identified statement_timeout is unset (= 0)', passed: identifiedUnset }, + { name: 'proposed setting a statement_timeout value', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identified that statement_timeout is set to 0 (no limit), meaning queries + can run indefinitely and a single runaway query can exhaust connections. + 2. Proposed setting a reasonable timeout via ALTER ROLE postgres SET + statement_timeout = '' or ALTER DATABASE ... SET statement_timeout, + with a specific value (e.g. '30s', '60s', '5min'). + + Fail if the agent did not cite the current value of 0, gave no specific + timeout value, or only described the risk without a concrete SQL command. + `, + }); + checks.push({ + name: 'correctly diagnosed unset statement_timeout and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; diff --git a/evals/o11y-resolve-postgres-statement-timeout-unset/PROMPT.md b/evals/o11y-resolve-postgres-statement-timeout-unset/PROMPT.md new file mode 100644 index 00000000..3dfb769c --- /dev/null +++ b/evals/o11y-resolve-postgres-statement-timeout-unset/PROMPT.md @@ -0,0 +1,15 @@ +--- +stage: investigate +suite: other +interface: mcp +product: + - database +topic: + - observability + - sql +motivation: pgbot-gap-analysis.md +--- + +We've had incidents where a runaway query consumed all available connections and caused an outage. After the fact we realized there was no statement timeout set, so queries can run indefinitely. Can you check whether a statement timeout is configured and tell us what to do? + +Report what you find and propose a fix. diff --git a/evals/o11y-resolve-postgres-statement-timeout-unset/remote/project.sql b/evals/o11y-resolve-postgres-statement-timeout-unset/remote/project.sql new file mode 100644 index 00000000..db0c2e69 --- /dev/null +++ b/evals/o11y-resolve-postgres-statement-timeout-unset/remote/project.sql @@ -0,0 +1,5 @@ +-- Starting state (probe: postgres-statement-timeout-unset). +-- statement_timeout = 0 means no timeout is set; runaway queries can run indefinitely. +-- pg_settings IS available in PGlite and will show the default value of 0. +-- No additional seeding needed — the default state IS the fault condition. +SELECT 1; diff --git a/o11y-README.md b/o11y-README.md index 73dddc72..b017e891 100644 --- a/o11y-README.md +++ b/o11y-README.md @@ -1,53 +1,69 @@ # o11y Eval Suite -The `o11y`-prefixed evals come from the chaos-o11y probe library, which maps Supabase failure modes to advisor lints. Each probe defines an injected failure state, expected agent behavior, and a judge rubric. 32 MCP evals were validated locally against a PGlite-backed harness; 11 CLI stubs are defined but not yet validated (they require a real local Supabase stack). Probe YAML source files live in `.context/probes/`; conversion decisions and tradeoffs are documented in `PROBE-CONVERSION.md`. +The `o11y`-prefixed evals come from two sources: the chaos-o11y probe library (which maps Supabase failure modes to advisor lints) and the pgbot gap analysis (`pgbot-gap-analysis.md`, a cross-reference of 61 production Postgres health checks against existing advisor and eval coverage). Each eval defines an injected failure state, expected agent behavior, and a judge rubric. 32 MCP evals were validated locally against a PGlite-backed harness; 16 pgbot-derived evals and 11 CLI stubs are defined but not yet run. Probe YAML source files live in `.context/probes/`; conversion decisions and tradeoffs are documented in `PROBE-CONVERSION.md`. --- ## MCP evals — validated locally -32 evals run against the MCP harness. 27 pass, 5 fail (all failures are valid model-behavior signal — see below). - -| # | Probe | Product | Eval dir | Advisor | Local result | -|---|---|---|---|---|---| -| 0001 | unindexed_foreign_keys | Database | o11y-0001-resolve-performance-missing-index | ✅ | PASS (not run in this session — previously validated) | -| 0002 | auth_users_exposed | Auth | o11y-0002-resolve-security-auth-users-exposed | ✅ | FAIL 1/2 (agent revokes grants but doesn't drop view — valid signal) | -| 0003 | auth_rls_initplan | Database | o11y-0003-resolve-security-rls-initplan | ✅ | PASS 2/2 | -| 0004 | no_primary_key | Database | o11y-0004-resolve-stability-no-primary-key | ✅ | PASS (not run in this session — previously validated) | -| 0005 | unused_index | Database | o11y-0005-resolve-performance-unused-index | ✅ | PASS 2/2 | -| 0006 | multiple_permissive_policies | Database | o11y-0006-resolve-security-multiple-permissive-policies | ✅ | PASS (not run in this session — previously validated) | -| 0007 | policy_exists_rls_disabled | Database | o11y-0007-resolve-security-rls-policy-exists-disabled | ✅ | PASS 2/2 | -| 0008 | rls_enabled_no_policy | Database | o11y-0008-resolve-security-rls-enabled-no-policy | ✅ | PASS 2/2 | -| 0009 | duplicate_index | Database | o11y-0009-resolve-performance-duplicate-index | ✅ | PASS 3/3 | -| 0010 | security_definer_view | PostgREST | o11y-0010-resolve-security-security-definer-view | ✅ | PASS 1/1 | -| 0011 | function_search_path_mutable | Database | o11y-0011-resolve-security-function-search-path-mutable | ✅ | PASS 2/2 | -| 0013 | rls_disabled_in_public | Database | o11y-0013-resolve-security-rls-disabled | ✅ | PASS (not run in this session — previously validated) | -| 0014 | extension_in_public | Database | o11y-0014-resolve-security-extension-in-public | ✅ | PASS 2/2 | -| 0015 | rls_references_user_metadata | Auth | o11y-0015-resolve-security-rls-references-user-metadata | ✅ | PASS 2/2 | -| 0016 | materialized_view_in_api | PostgREST | o11y-0016-resolve-security-materialized-view-in-api | ✅ | PASS 1/1 | -| 0018 | unsupported_reg_types | Database | o11y-0018-resolve-stability-unsupported-reg-types | ⚠️ advisor bug | PASS 2/2 | -| 0019 | insecure_queue_exposed | Database | o11y-0019-resolve-security-pgmq-queue-exposed | ⚠️ advisor bug | PASS 3/3 | -| 0020 | table_bloat | Database | o11y-0020-resolve-postgres-table-bloat | ✅ | PASS 2/2 | -| 0021 | fkey_to_auth_unique | Auth | o11y-0021-resolve-security-fkey-to-auth-unique | ❌ no advisor | PASS 1/1 | -| 0023 | sensitive_columns_exposed | PostgREST | o11y-0023-resolve-security-sensitive-columns-exposed | ⚠️ advisor bug | PASS 2/2 | -| 0024 | rls_policy_always_true | Database | o11y-0024-resolve-security-rls-policy-always-true | ⚠️ advisor bug | PASS 2/2 | -| 0028/0029 | anon/authenticated_security_definer_executable | Database | o11y-0028-0029-resolve-security-anon-security-definer | ✅ | PASS 2/2 | -| — | api_missing_grant | PostgREST | o11y-resolve-security-api-missing-grant | p2 | PASS 1/1 | -| — | api_slow_response | Edge Functions | o11y-investigate-api-slow-response | p3 | FAIL 2/3 (identified delay, judge wanted setTimeout specificity) | -| — | auth_leaked_jwt | Auth | o11y-investigate-auth-leaked-jwt | p3 | PASS 4/4 | -| — | cost_n_plus_one | Database | o11y-investigate-cost-n-plus-one | p2 | PASS (not run in this session — previously validated) | -| — | cron_broken_job | Database | o11y-resolve-cron-broken-job | p2 | PASS (not run in this session — previously validated) | -| — | performance_edge_fn_slow | Edge Functions | o11y-investigate-performance-edge-fn-slow | p2 | PASS 2/2 | -| — | postgres_auth_admin_bypassrls | Auth | o11y-investigate-security-auth-admin-bypassrls | p2 | PASS 3/3 | -| — | postgres_autovacuum_disabled | Database | o11y-investigate-postgres-autovacuum-disabled | p0 | FAIL 0/2 (agent used SET instead of RESET syntax — valid signal) | -| — | postgres_connection_saturation | Database | o11y-investigate-postgres-connection-saturation | PR #36781 | PASS 3/3 | -| — | postgres_index_bloat | Database | o11y-investigate-postgres-index-bloat | p2 | PASS 3/3 | -| — | postgres_long_running_query | Database | o11y-investigate-postgres-long-running-query | p1 | PASS 3/3 | -| — | postgres_slow_query | Database | o11y-investigate-postgres-slow-query | p1 | PASS 4/4 | -| — | postgres_temp_file_spill | Database | o11y-investigate-postgres-temp-file-spill | p2 | FAIL 2/3 (solid diagnosis, judge wanted temp_blks_written citation) | -| — | postgres_wal_slot_inactive | Database | o11y-investigate-postgres-wal-slot-inactive | p0 | PASS 3/3 | -| — | stability_edge_500 | Edge Functions | o11y-investigate-stability-edge-500 | PR #36781 | FAIL 2/3 (identified 500s, judge wanted specific TypeError at line 2) | -| — | stability_silent_data_drift | Database | o11y-investigate-stability-silent-data-drift | p3 | PASS 3/3 | +54 evals total. 38 previously defined (32 validated, 5 fail as valid signal, 1 blocked); 16 pgbot-derived (not yet validated). + +| n | # | Probe | Product | Eval dir | Advisor | Local result | +|---|---|---|---|---|---|---| +| 1 | 0001 | unindexed_foreign_keys | Database | o11y-0001-resolve-performance-missing-index | ✅ | PASS (not run in this session — previously validated) | +| 2 | 0002 | auth_users_exposed | Auth | o11y-0002-resolve-security-auth-users-exposed | ✅ | FAIL 1/2 (agent revokes grants but doesn't drop view — valid signal) | +| 3 | 0003 | auth_rls_initplan | Database | o11y-0003-resolve-security-rls-initplan | ✅ | PASS 2/2 | +| 4 | 0004 | no_primary_key | Database | o11y-0004-resolve-stability-no-primary-key | ✅ | PASS (not run in this session — previously validated) | +| 5 | 0005 | unused_index | Database | o11y-0005-resolve-performance-unused-index | ✅ | PASS 2/2 | +| 6 | 0006 | multiple_permissive_policies | Database | o11y-0006-resolve-security-multiple-permissive-policies | ✅ | PASS (not run in this session — previously validated) | +| 7 | 0007 | policy_exists_rls_disabled | Database | o11y-0007-resolve-security-rls-policy-exists-disabled | ✅ | PASS 2/2 | +| 8 | 0008 | rls_enabled_no_policy | Database | o11y-0008-resolve-security-rls-enabled-no-policy | ✅ | PASS 2/2 | +| 9 | 0009 | duplicate_index | Database | o11y-0009-resolve-performance-duplicate-index | ✅ | PASS 3/3 | +| 10 | 0010 | security_definer_view | PostgREST | o11y-0010-resolve-security-security-definer-view | ✅ | PASS 1/1 | +| 11 | 0011 | function_search_path_mutable | Database | o11y-0011-resolve-security-function-search-path-mutable | ✅ | PASS 2/2 | +| 12 | 0013 | rls_disabled_in_public | Database | o11y-0013-resolve-security-rls-disabled | ✅ | PASS (not run in this session — previously validated) | +| 13 | 0014 | extension_in_public | Database | o11y-0014-resolve-security-extension-in-public | ✅ | PASS 2/2 | +| 14 | 0015 | rls_references_user_metadata | Auth | o11y-0015-resolve-security-rls-references-user-metadata | ✅ | PASS 2/2 | +| 15 | 0016 | materialized_view_in_api | PostgREST | o11y-0016-resolve-security-materialized-view-in-api | ✅ | PASS 1/1 | +| 16 | 0018 | unsupported_reg_types | Database | o11y-0018-resolve-stability-unsupported-reg-types | ⚠️ advisor bug | PASS 2/2 | +| 17 | 0019 | insecure_queue_exposed | Database | o11y-0019-resolve-security-pgmq-queue-exposed | ⚠️ advisor bug | PASS 3/3 | +| 18 | 0020 | table_bloat | Database | o11y-0020-resolve-postgres-table-bloat | ✅ | PASS 2/2 | +| 19 | 0021 | fkey_to_auth_unique | Auth | o11y-0021-resolve-security-fkey-to-auth-unique | ❌ no advisor | PASS 1/1 | +| 20 | 0023 | sensitive_columns_exposed | PostgREST | o11y-0023-resolve-security-sensitive-columns-exposed | ⚠️ advisor bug | PASS 2/2 | +| 21 | 0024 | rls_policy_always_true | Database | o11y-0024-resolve-security-rls-policy-always-true | ⚠️ advisor bug | PASS 2/2 | +| 22 | 0028/0029 | anon/authenticated_security_definer_executable | Database | o11y-0028-0029-resolve-security-anon-security-definer | ✅ | PASS 2/2 | +| 23 | — | api_missing_grant | PostgREST | o11y-resolve-security-api-missing-grant | p2 | PASS 1/1 | +| 24 | — | api_slow_response | Edge Functions | o11y-investigate-api-slow-response | p3 | FAIL 2/3 (identified delay, judge wanted setTimeout specificity) | +| 25 | — | auth_leaked_jwt | Auth | o11y-investigate-auth-leaked-jwt | p3 | PASS 4/4 | +| 26 | — | cost_n_plus_one | Database | o11y-investigate-cost-n-plus-one | p2 | PASS (not run in this session — previously validated) | +| 27 | — | cron_broken_job | Database | o11y-resolve-cron-broken-job | p2 | PASS (not run in this session — previously validated) | +| 28 | — | performance_edge_fn_slow | Edge Functions | o11y-investigate-performance-edge-fn-slow | p2 | PASS 2/2 | +| 29 | — | postgres_auth_admin_bypassrls | Auth | o11y-investigate-security-auth-admin-bypassrls | p2 | PASS 3/3 | +| 30 | — | postgres_autovacuum_disabled | Database | o11y-investigate-postgres-autovacuum-disabled | p0 | FAIL 0/2 (agent used SET instead of RESET syntax — valid signal) | +| 31 | — | postgres_connection_saturation | Database | o11y-investigate-postgres-connection-saturation | PR #36781 | PASS 3/3 | +| 32 | — | postgres_index_bloat | Database | o11y-investigate-postgres-index-bloat | p2 | PASS 3/3 | +| 33 | — | postgres_long_running_query | Database | o11y-investigate-postgres-long-running-query | p1 | PASS 3/3 | +| 34 | — | postgres_slow_query | Database | o11y-investigate-postgres-slow-query | p1 | PASS 4/4 | +| 35 | — | postgres_temp_file_spill | Database | o11y-investigate-postgres-temp-file-spill | p2 | FAIL 2/3 (solid diagnosis, judge wanted temp_blks_written citation) | +| 36 | — | postgres_wal_slot_inactive | Database | o11y-investigate-postgres-wal-slot-inactive | p0 | PASS 3/3 | +| 37 | — | stability_edge_500 | Edge Functions | o11y-investigate-stability-edge-500 | PR #36781 | FAIL 2/3 (identified 500s, judge wanted specific TypeError at line 2) | +| 38 | — | stability_silent_data_drift | Database | o11y-investigate-stability-silent-data-drift | p3 | PASS 3/3 | +| 39 | — | txid_wraparound + mxid_wraparound | Database | o11y-investigate-postgres-txid-wraparound | pgbot | not validated | +| 40 | — | sequence_exhaustion + int4_identity_column | Database | o11y-investigate-postgres-sequence-exhaustion | pgbot | not validated | +| 41 | — | index_invalid | Database | o11y-investigate-postgres-index-invalid | pgbot | not validated | +| 42 | — | blocking_chains | Database | o11y-investigate-postgres-blocking-chains | pgbot | not validated | +| 43 | — | idle_in_transaction | Database | o11y-investigate-postgres-idle-in-transaction | pgbot | not validated | +| 44 | — | vacuum_horizon_blocked | Database | o11y-investigate-postgres-vacuum-horizon-blocked | pgbot | not validated | +| 45 | — | prepared_xact_abandoned | Database | o11y-investigate-postgres-prepared-transaction-abandoned | pgbot | not validated | +| 46 | — | autovacuum_off (global) | Database | o11y-investigate-postgres-autovacuum-global-off | pgbot | not validated | +| 47 | — | autovacuum_saturated | Database | o11y-investigate-postgres-autovacuum-saturated | pgbot | not validated | +| 48 | — | low_cache_hit | Database | o11y-investigate-postgres-low-cache-hit-ratio | pgbot | not validated | +| 49 | — | checkpoints_forced | Database | o11y-investigate-postgres-forced-checkpoints | pgbot | not validated | +| 50 | — | statement_timeout_unset | Database | o11y-resolve-postgres-statement-timeout-unset | pgbot | not validated | +| 51 | — | random_page_cost_high | Database | o11y-resolve-postgres-random-page-cost | pgbot | not validated | +| 52 | — | high_rollback_ratio | Database | o11y-investigate-postgres-high-rollback-ratio | pgbot | not validated | +| 53 | — | autovacuum_starved | Database | o11y-investigate-postgres-autovacuum-starved | pgbot | not validated | +| 54 | — | replica_lag_time | Database | o11y-investigate-postgres-replica-lag | pgbot | not validated | --- @@ -80,6 +96,7 @@ The `o11y`-prefixed evals come from the chaos-o11y probe library, which maps Sup | ❌ no advisor | No lint exists for this failure pattern | | p0–p3 | No advisor lint exists for this pattern; number is a community-assigned priority (p0 = highest urgency) | | PR #36781 | Coverage tracked in an in-progress platform PR | +| pgbot | Eval derived from the pgbot gap analysis (`pgbot-gap-analysis.md`); covers checks that pgbot surfaces but no Splinter advisor or prior o11y eval exists for | --- @@ -97,13 +114,71 @@ The evals injected the correct failure state, the judge criteria are precise, an --- -## PGlite workarounds +## Future: compound evals and red-herring scenarios -Several MCP evals required workarounds to run under PGlite's constraints. These are reflected in the eval setup SQL or harness config rather than the rubric. +The current evals are deliberately atomic — one injected fault, one expected diagnosis. That's the right starting point, but production incidents rarely look like this. Real Postgres incidents typically involve: + +- **Layered causes** — e.g. autovacuum is falling behind *because* an inactive replication slot is holding the vacuum horizon, *and* the table also has `autovacuum_enabled=false`. Fixing one symptom without finding the other leaves the problem alive. +- **Red herrings** — the presenting symptom (slow queries, high disk usage, connection errors) looks like one thing but is caused by something else entirely. Bloat looks like missing VACUUM; the real cause is an idle-in-transaction session nobody noticed. A high rollback ratio looks like a traffic spike; it's actually a silent constraint violation introduced in a migration. +- **Order-of-operations traps** — the correct fix exists but applying it in the wrong order makes things worse (e.g. dropping a replication slot before investigating whether downstream consumers are recoverable). + +The goal for a future "compound" eval tier is to require the agent to: + +1. Triage multiple concurrent signals without anchoring on the most obvious one +2. Distinguish the root cause from the symptoms it produces +3. Surface the non-obvious dependency (the thing that makes the real fix different from the surface fix) +4. Propose remediation steps in the right sequence + +These evals would combine multiple injected states from the atomic tier — e.g. seed both a long-running idle-in-transaction session *and* an inactive replication slot *and* high dead-tuple counts, with a prompt that only describes the bloat symptom. A passing agent traces the bloat backward to both root causes and sequences the fixes correctly. A failing agent fixes the bloat directly (runs VACUUM) without addressing why vacuum wasn't working. + +The atomic evals in this suite are the building blocks. Once they're validated, compound scenarios can be assembled from their constituent parts. + +--- + +## PGlite and harness limitations + +The MCP harness runs Postgres as PGlite (WASM, in-process), wrapped by `@supabase/lite` (PostgREST + auth + storage schemas) and `platform-lite` (Management API HTTP mock + Postgres-wire). This stack lets evals run without cloud infrastructure but introduces constraints that affect both setup SQL and scorer design. + +`ctx.query(sql)` calls PGlite directly as the database owner, bypassing RLS. Scorers can verify post-state freely, but this means **scorers must not be used to validate RLS behavior** — use `ctx.getClient()` (anon key) for that. + +### PGlite Postgres constraints + +These affect what can be seeded and queried in `remote/project.sql`. + +> **Snapshot tables:** When a system view can't be seeded in PGlite, evals create a `public.*_snapshot` table mirroring its columns and inject the fault state there. Prompt notes say "data has been exported to a snapshot table" — the agent must discover which table via `information_schema` or `pg_tables`. | Issue | Workaround | |---|---| -| `pg_stat_activity`, `pg_stat_statements`, and other system views not available in PGlite | Created mock views or tables that return representative rows; agent queries these and receives realistic data | +| `pg_stat_activity`, `pg_stat_statements`, and other live system views not available | Create `public.*_snapshot` tables mirroring the relevant columns; prompt includes a note directing the agent to query the snapshot instead | | `BYPASSRLS` role attribute not supported | Simulated with a superuser-equivalent role where needed; eval for `postgres_auth_admin_bypassrls` validates the detection pattern against mock data | | `pg_trgm` extension unavailable | Omitted or stubbed in evals that reference trigram indexes; rubric adjusted to not require extension-specific output | -| `auth.users` schema differences | PGlite `auth` schema is a stub; evals that reference `auth.users` columns create a local approximation matching the columns checked by the agent | +| `auth.users` schema differences | PGlite `auth` schema is a stub; evals that reference `auth.users` columns create a local approximation matching the columns the agent checks | +| `CREATE INDEX CONCURRENTLY` not supported | Evals that test invalid index detection (e.g. `index_invalid`) use a `pg_index_snapshot` table rather than a real invalid index | +| `ALTER SYSTEM` does not persist | GUC-level evals (e.g. `statement_timeout_unset`, `random_page_cost`) use the PGlite default as the fault condition; agent proposes the fix but cannot be verified as having applied it — judge rubric covers this | + +### Harness-level constraints + +These affect how evals are scored and what signals are trustworthy. + +| Issue | Impact on evals | +|---|---| +| **Logs are faked end-to-end** — production logs flow from the Postgres instance through Vector → Logflare → ClickHouse; the harness replaces the entire pipeline with a Hono route that serves seeded JSONL. There is no real log store or ClickHouse. | Log query behavior does not match production. Time-window params (`iso_timestamp_start`/`end`) are accepted but ignored — agents always get the full seeded dataset back regardless of how they query. Rubrics must require agents to cite specific log content, not just that logs were queried | +| **Migrations endpoint and table are out of sync** — `db push` via Postgres-wire writes to `supabase_migrations.schema_migrations` directly; the `/database/migrations` Management API endpoint tracks its own in-memory state | Scorers verifying migration state should query `supabase_migrations.schema_migrations` via `ctx.query`, not the Management API endpoint | +| **Edge function runtime is minimal** — only `Deno.serve()` and `Deno.env.get()` are available; npm packages must be pre-installed in the eval runtime's dependencies | Edge function evals should not rely on Deno filesystem APIs or unlisted npm packages | + +--- + +## Studio health advisor signals (PR #49661) + +The table below maps all 8 health checks introduced in Supabase Studio PR #49661 to eval coverage. These are service-level signals surfaced via `/v2/projects/{ref}/advisors/run`; most cannot be reproduced as agent evals because there is no Postgres diagnostic path available when the check fires. + +| Studio health check | Advisor name | Eval | Blocked? | Reason | +|---|---|---|---|---| +| Connection limit reached | `db_connection_limit_reached` | `o11y-investigate-postgres-connection-saturation` | No | Covered via `pg_stat_activity_snapshot` pattern — agent diagnoses idle connections and proposes pgbouncer/supavisor | +| Database process down | `instance_db_down` | — | Yes | The database is down; the agent cannot connect to run any query | +| Database not reachable | `db_not_reachable` | — | Yes | Same — no SQL diagnostic path available when the check fires | +| Data API error rate high | `log_data_api_error_rate_high` | — | Yes | Log-derived signal from PostgREST; not diagnosable via Postgres system views | +| Auth error rate high | `log_auth_error_rate_high` | — | Yes | Same — service log data, not a Postgres diagnostic | +| Storage error rate high | `log_storage_error_rate_high` | — | Yes | Same | +| Edge Function error rate high | `log_edge_function_error_rate_high` | — | Yes | Same | +| Infrastructure alert firing | `instance_alert_firing` | — | Yes | Infrastructure layer; no Postgres or log query maps to this signal | From ebc7428ef08b8ddcef33c05862bd32514c699ca7 Mon Sep 17 00:00:00 2001 From: Steven Eubank Date: Wed, 2 Sep 2026 21:57:47 +0200 Subject: [PATCH 3/3] add more complex API evals --- .../EVAL.ts | 72 +++++++++++++++++ .../PROMPT.md | 20 +++++ .../remote/project.sql | 72 +++++++++++++++++ .../EVAL.ts | 65 +++++++++++++++ .../PROMPT.md | 18 +++++ .../remote/logs.jsonl | 15 ++++ .../remote/project.sql | 70 ++++++++++++++++ .../o11y-investigate-api-rls-initplan/EVAL.ts | 73 +++++++++++++++++ .../PROMPT.md | 24 ++++++ .../remote/project.sql | 80 +++++++++++++++++++ .../EVAL.ts | 61 ++++++++++++++ .../PROMPT.md | 18 +++++ .../remote/logs.jsonl | 8 ++ .../remote/project.sql | 63 +++++++++++++++ o11y-README.md | 9 ++- 15 files changed, 666 insertions(+), 2 deletions(-) create mode 100644 evals/o11y-investigate-api-embedding-nplus1/EVAL.ts create mode 100644 evals/o11y-investigate-api-embedding-nplus1/PROMPT.md create mode 100644 evals/o11y-investigate-api-embedding-nplus1/remote/project.sql create mode 100644 evals/o11y-investigate-api-high-load-source/EVAL.ts create mode 100644 evals/o11y-investigate-api-high-load-source/PROMPT.md create mode 100644 evals/o11y-investigate-api-high-load-source/remote/logs.jsonl create mode 100644 evals/o11y-investigate-api-high-load-source/remote/project.sql create mode 100644 evals/o11y-investigate-api-rls-initplan/EVAL.ts create mode 100644 evals/o11y-investigate-api-rls-initplan/PROMPT.md create mode 100644 evals/o11y-investigate-api-rls-initplan/remote/project.sql create mode 100644 evals/o11y-investigate-api-slow-endpoint/EVAL.ts create mode 100644 evals/o11y-investigate-api-slow-endpoint/PROMPT.md create mode 100644 evals/o11y-investigate-api-slow-endpoint/remote/logs.jsonl create mode 100644 evals/o11y-investigate-api-slow-endpoint/remote/project.sql diff --git a/evals/o11y-investigate-api-embedding-nplus1/EVAL.ts b/evals/o11y-investigate-api-embedding-nplus1/EVAL.ts new file mode 100644 index 00000000..09c46c75 --- /dev/null +++ b/evals/o11y-investigate-api-embedding-nplus1/EVAL.ts @@ -0,0 +1,72 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: PostgREST resource embedding (?select=*,line_items(*)) generates a +// correlated subquery per order row because line_items.order_id has no index. +// Every outer order triggers a seq scan of line_items, producing N+1 behaviour +// at the DB level. The fix is an index on line_items.order_id. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const namedLineItems = /\bline.?items\b/i.test(report); + const inspectedStats = /pg_stat_statements|query performance|query insights/i.test(transcript); + const identifiedPattern = + /n\s*\+\s*1/i.test(report) || + /(correlated|per.?order|per.?row|repeated subquery|subquery.*per|one.*per.*order)/i.test( + report + ) || + /embedding.*slow|slow.*embedding|resource embedding/i.test(report); + const proposedFix = + /\bcreate index\b|\badd.*index\b|\bindex on\b/i.test(report) || + /\bline_items\b.*\bindex\b|\bindex\b.*\border_id\b/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified line_items as the driver', passed: namedLineItems }, + { name: 'inspected query statistics', passed: inspectedStats }, + { + name: 'identified correlated / per-row subquery (N+1) pattern', + passed: identifiedPattern, + }, + { name: 'proposed an index on line_items.order_id or equivalent fix', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identifies from query statistics that there is a correlated subquery + on line_items with a call count far exceeding the outer orders query + (N+1 pattern: one subquery per order row). + 2. Recognises this is generated by PostgREST's resource embedding + (?select=*,line_items(*)) combined with a missing index. + 3. Notes that the line_items subquery performs a sequential scan on + every call (high shared_blks_read, high total_exec_time). + 4. Proposes a concrete fix — at minimum, creating an index on + line_items.order_id so embedded lookups use an index scan. + + Fail if the agent misidentifies the cause as application-level N+1 + (calls from client code), blames the orders table, recommends only + pagination without addressing the index, or gives vague advice. + `, + }); + + checks.push({ + name: 'correctly diagnosed embedding N+1 and proposed index fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { + passed: namedLineItems && inspectedStats && identifiedPattern && proposedFix && verdict.passed, + checks, + }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-api-embedding-nplus1/PROMPT.md b/evals/o11y-investigate-api-embedding-nplus1/PROMPT.md new file mode 100644 index 00000000..dc0ad709 --- /dev/null +++ b/evals/o11y-investigate-api-embedding-nplus1/PROMPT.md @@ -0,0 +1,20 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - data-api + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/high-egress-or-compute.mdx +--- + +Our order detail page is painfully slow and hammering the database. We fetch +orders together with their line items using the Data API's resource embedding +(`?select=*,line_items(*)`). The orders list page loads fine, but as soon as +line items are included the response time spikes to 5–8 seconds and DB CPU +shoots up. Can you figure out what's causing this and tell us what to do? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-api-embedding-nplus1/remote/project.sql b/evals/o11y-investigate-api-embedding-nplus1/remote/project.sql new file mode 100644 index 00000000..dc33af31 --- /dev/null +++ b/evals/o11y-investigate-api-embedding-nplus1/remote/project.sql @@ -0,0 +1,72 @@ +-- Starting state (probe: api-embedding-nplus1). +-- Resource embedding (?select=*,line_items(*)) generates a correlated +-- subquery per order row because line_items.order_id has no index. +-- Each outer row triggers a seq scan on line_items, producing N+1 behaviour +-- at the database level. Evidence lives in faked query statistics. + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + user_id uuid NOT NULL, + status text NOT NULL DEFAULT 'pending', + total_cents int NOT NULL DEFAULT 0, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE public.line_items ( + id bigserial PRIMARY KEY, + order_id bigint NOT NULL REFERENCES public.orders(id), + product_name text NOT NULL, + quantity int NOT NULL DEFAULT 1, + unit_cents int NOT NULL DEFAULT 0 + -- Intentionally no index on order_id +); + +INSERT INTO public.orders (user_id, status, total_cents) +SELECT + gen_random_uuid(), + (ARRAY['pending','complete','cancelled'])[floor(random()*3+1)::int], + (floor(random() * 50000) + 500)::int +FROM generate_series(1, 2000); + +INSERT INTO public.line_items (order_id, product_name, quantity, unit_cents) +SELECT + (floor(random() * 2000) + 1)::int, + 'Product ' || (floor(random() * 50) + 1)::int, + (floor(random() * 5) + 1)::int, + (floor(random() * 5000) + 100)::int +FROM generate_series(1, 10000); + +-- Eval-scoped compatibility table for Supabase's Query Performance report, +-- backed by pg_stat_statements in real projects. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- Outer query: fetching a page of orders — reasonable. + (6001, + 'SELECT "orders"."id", "orders"."user_id", "orders"."status", "orders"."total_cents", "orders"."created_at" FROM "orders" ORDER BY "orders"."created_at" DESC LIMIT $1 OFFSET $2', + 9200, 460000, 46000.0, 5.0, 22.0, 2300000, 18000), + -- Inner correlated subquery: one seq scan on line_items per order row. + -- calls ≈ 50 × outer calls (50 orders per page × 9200 outer calls). + -- High blks_read signals seq scan; high total_exec_time is the real cost. + (6002, + 'SELECT "line_items"."id", "line_items"."order_id", "line_items"."product_name", "line_items"."quantity", "line_items"."unit_cents" FROM "line_items" WHERE "line_items"."order_id" = $1', + 460000, 2300000, 24840000.0, 54.0, 310.0, 1800000, 9200000), + -- Fast unrelated query for contrast. + (6003, + 'SELECT "orders"."id", "orders"."status" FROM "orders" WHERE "orders"."user_id" = $1', + 3800, 19000, 760.0, 0.2, 3.0, 95000, 800); diff --git a/evals/o11y-investigate-api-high-load-source/EVAL.ts b/evals/o11y-investigate-api-high-load-source/EVAL.ts new file mode 100644 index 00000000..6ae35d50 --- /dev/null +++ b/evals/o11y-investigate-api-high-load-source/EVAL.ts @@ -0,0 +1,65 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: /rest/v1/reports drives DB CPU because it runs a JOIN between reports +// and the 200k-row events table with no index on events.user_id. The agent must +// cross-reference API logs (which endpoint is slow?) with pg_stat_statements +// (which query is expensive?) to attribute the load correctly. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const namedReports = + /\breports\b/i.test(report) && + (/\/rest\/v1\/reports/i.test(report) || /reports.*endpoint|endpoint.*reports/i.test(report)); + const inspectedLogs = /\/rest\/v1\//i.test(transcript); + const inspectedStats = /pg_stat_statements|query performance|query insights/i.test(transcript); + const identifiedJoin = + /\bjoin\b/i.test(report) && + /\bevents\b/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified /rest/v1/reports as the load source', passed: namedReports }, + { name: 'inspected API logs across endpoints', passed: inspectedLogs }, + { name: 'inspected query statistics', passed: inspectedStats }, + { name: 'identified the expensive join involving events', passed: identifiedJoin }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Reviews API logs and identifies that /rest/v1/reports requests are + consistently slow (3–4s) while /rest/v1/users and /rest/v1/events + are fast. + 2. Cross-references those slow requests with query statistics to find + the expensive JOIN between reports and events. + 3. Correctly attributes the elevated DB CPU to the reports endpoint. + 4. Proposes a concrete fix — such as adding an index on events.user_id, + rewriting the query, or removing the join. + + Fail if the agent blames the wrong endpoint, only looks at one data + source (logs only or stats only), or gives vague advice without naming + the reports table and the expensive join. + `, + }); + + checks.push({ + name: 'correctly attributed load to reports endpoint and identified cause', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { + passed: namedReports && inspectedStats && identifiedJoin && verdict.passed, + checks, + }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-api-high-load-source/PROMPT.md b/evals/o11y-investigate-api-high-load-source/PROMPT.md new file mode 100644 index 00000000..ba3058be --- /dev/null +++ b/evals/o11y-investigate-api-high-load-source/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - data-api + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/high-egress-or-compute.mdx +--- + +Our database CPU has been elevated all morning — around 60–70% — but traffic +looks normal. We have several API endpoints in use. Can you figure out which +endpoint is driving the database load and what's causing it? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-api-high-load-source/remote/logs.jsonl b/evals/o11y-investigate-api-high-load-source/remote/logs.jsonl new file mode 100644 index 00000000..81871217 --- /dev/null +++ b/evals/o11y-investigate-api-high-load-source/remote/logs.jsonl @@ -0,0 +1,15 @@ +{"id":"log-001","ts":"2026-08-01T09:00:01Z","source":"edge","level":"info","message":"GET /rest/v1/users 200 6ms","metadata":{"method":"GET","path":"/rest/v1/users","status_code":200,"duration_ms":6}} +{"id":"log-002","ts":"2026-08-01T09:00:03Z","source":"edge","level":"info","message":"GET /rest/v1/reports 200 3812ms","metadata":{"method":"GET","path":"/rest/v1/reports","status_code":200,"duration_ms":3812}} +{"id":"log-003","ts":"2026-08-01T09:00:07Z","source":"edge","level":"info","message":"GET /rest/v1/events 200 11ms","metadata":{"method":"GET","path":"/rest/v1/events","status_code":200,"duration_ms":11}} +{"id":"log-004","ts":"2026-08-01T09:00:09Z","source":"edge","level":"info","message":"GET /rest/v1/users 200 5ms","metadata":{"method":"GET","path":"/rest/v1/users","status_code":200,"duration_ms":5}} +{"id":"log-005","ts":"2026-08-01T09:00:13Z","source":"edge","level":"info","message":"GET /rest/v1/reports 200 4021ms","metadata":{"method":"GET","path":"/rest/v1/reports","status_code":200,"duration_ms":4021}} +{"id":"log-006","ts":"2026-08-01T09:00:17Z","source":"edge","level":"info","message":"GET /rest/v1/events 200 9ms","metadata":{"method":"GET","path":"/rest/v1/events","status_code":200,"duration_ms":9}} +{"id":"log-007","ts":"2026-08-01T09:00:21Z","source":"edge","level":"info","message":"GET /rest/v1/users 200 4ms","metadata":{"method":"GET","path":"/rest/v1/users","status_code":200,"duration_ms":4}} +{"id":"log-008","ts":"2026-08-01T09:00:25Z","source":"edge","level":"info","message":"GET /rest/v1/reports 200 3944ms","metadata":{"method":"GET","path":"/rest/v1/reports","status_code":200,"duration_ms":3944}} +{"id":"log-009","ts":"2026-08-01T09:00:29Z","source":"edge","level":"info","message":"GET /rest/v1/events 200 12ms","metadata":{"method":"GET","path":"/rest/v1/events","status_code":200,"duration_ms":12}} +{"id":"log-010","ts":"2026-08-01T09:00:33Z","source":"edge","level":"info","message":"GET /rest/v1/users 200 7ms","metadata":{"method":"GET","path":"/rest/v1/users","status_code":200,"duration_ms":7}} +{"id":"log-011","ts":"2026-08-01T09:00:37Z","source":"edge","level":"info","message":"GET /rest/v1/reports 200 4103ms","metadata":{"method":"GET","path":"/rest/v1/reports","status_code":200,"duration_ms":4103}} +{"id":"log-012","ts":"2026-08-01T09:00:41Z","source":"edge","level":"info","message":"GET /rest/v1/events 200 8ms","metadata":{"method":"GET","path":"/rest/v1/events","status_code":200,"duration_ms":8}} +{"id":"log-013","ts":"2026-08-01T09:00:45Z","source":"edge","level":"info","message":"GET /rest/v1/users 200 5ms","metadata":{"method":"GET","path":"/rest/v1/users","status_code":200,"duration_ms":5}} +{"id":"log-014","ts":"2026-08-01T09:00:49Z","source":"edge","level":"info","message":"GET /rest/v1/reports 200 3877ms","metadata":{"method":"GET","path":"/rest/v1/reports","status_code":200,"duration_ms":3877}} +{"id":"log-015","ts":"2026-08-01T09:00:53Z","source":"edge","level":"info","message":"GET /rest/v1/events 200 10ms","metadata":{"method":"GET","path":"/rest/v1/events","status_code":200,"duration_ms":10}} diff --git a/evals/o11y-investigate-api-high-load-source/remote/project.sql b/evals/o11y-investigate-api-high-load-source/remote/project.sql new file mode 100644 index 00000000..221def96 --- /dev/null +++ b/evals/o11y-investigate-api-high-load-source/remote/project.sql @@ -0,0 +1,70 @@ +-- Starting state (probe: api-high-load-source). +-- DB CPU is elevated. Three endpoints are in use: /rest/v1/reports, +-- /rest/v1/users, /rest/v1/events. The reports endpoint is responsible — +-- it runs a cross-table JOIN with no covering index, touching hundreds of +-- thousands of pages on every request. Evidence lives in faked query statistics. + +CREATE TABLE public.users ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE public.events ( + id bigserial PRIMARY KEY, + user_id int NOT NULL REFERENCES public.users(id), + kind text NOT NULL, + payload jsonb, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE public.reports ( + id serial PRIMARY KEY, + name text NOT NULL, + created_by int NOT NULL REFERENCES public.users(id), + created_at timestamptz DEFAULT now() +); + +INSERT INTO public.users (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 500) AS g; + +INSERT INTO public.events (user_id, kind) +SELECT (floor(random() * 500) + 1)::int, (ARRAY['click','view','submit','error'])[floor(random()*4+1)::int] +FROM generate_series(1, 200000); + +INSERT INTO public.reports (name, created_by) +SELECT 'Report ' || g, (floor(random() * 500) + 1)::int +FROM generate_series(1, 200) AS g; + +-- Eval-scoped compatibility table for Supabase's Query Performance report, +-- backed by pg_stat_statements in real projects. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- The culprit: reports JOIN events with no index on events.user_id, + -- runs a full seq scan on the 200k-row events table for every request. + (4001, + 'SELECT "reports"."id", "reports"."name", "reports"."created_by", "events"."id", "events"."kind", "events"."created_at" FROM "reports" JOIN "events" ON "events"."user_id" = "reports"."created_by" ORDER BY "events"."created_at" DESC', + 14400, 2880000, 57600000.0, 4000.0, 5200.0, 8000, 2880000), + -- Fast, low-cost queries for the other endpoints. + (4002, + 'SELECT "users"."id", "users"."email", "users"."created_at" FROM "users" ORDER BY "users"."id" LIMIT $1', + 72000, 3600000, 288000.0, 4.0, 18.0, 2520000, 8000), + (4003, + 'SELECT "events"."id", "events"."user_id", "events"."kind", "events"."created_at" FROM "events" WHERE "events"."user_id" = $1 ORDER BY "events"."created_at" DESC LIMIT $2', + 36000, 720000, 396000.0, 11.0, 40.0, 2160000, 12000); diff --git a/evals/o11y-investigate-api-rls-initplan/EVAL.ts b/evals/o11y-investigate-api-rls-initplan/EVAL.ts new file mode 100644 index 00000000..2da85e5b --- /dev/null +++ b/evals/o11y-investigate-api-rls-initplan/EVAL.ts @@ -0,0 +1,73 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: authenticated reads on posts are slow because the RLS policy calls +// public.get_current_user_id(), a VOLATILE wrapper function. Postgres cannot +// hoist a VOLATILE call into an initplan, so it re-evaluates the function for +// every row scanned. The smoking gun is in pg_stat_statements: low blks_read +// (data is cached) but high mean_exec_time (~650ms). The agent must trace the +// slow stats to the policy, identify the VOLATILE function as the cause, and +// propose making the function STABLE or wrapping the call in (SELECT ...). + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const namedPostsTable = /\bposts\b/i.test(report); + const inspectedStats = /pg_stat_statements|query performance|query insights/i.test(transcript); + const identifiedCause = + /volatile|initplan|per.?row|re.?evaluat/i.test(report) || + /get_current_user_id|rls.*slow|slow.*rls|policy.*slow|slow.*policy/i.test(report); + const proposedFix = + /\bstable\b/i.test(report) || + /\(select\s+.*\)|sub.?select.*policy|wrap.*policy|policy.*wrap/i.test(report) || + /change.*volatile|volatile.*change|volatile.*stable/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified posts as the affected table', passed: namedPostsTable }, + { name: 'inspected query statistics', passed: inspectedStats }, + { name: 'identified per-row VOLATILE function evaluation as the cause', passed: identifiedCause }, + { name: 'proposed making the function STABLE or wrapping in (SELECT ...)', passed: proposedFix }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Notices from query statistics that the posts SELECT is slow despite + low I/O (data is in memory), indicating per-row CPU overhead rather + than disk reads. + 2. Examines the RLS policy on the posts table and identifies + public.get_current_user_id() as the function being called in the + USING clause. + 3. Identifies that the function is VOLATILE, causing Postgres to + re-evaluate it for every row scanned rather than once per query + (the initplan / per-row evaluation problem). + 4. Proposes a concrete fix: either change the function from VOLATILE + to STABLE, or rewrite the policy USING clause to wrap the call in + a sub-SELECT — e.g. USING (user_id = (SELECT public.get_current_user_id())). + + Fail if the agent blames I/O or missing indexes without investigating + the RLS policy, or gives vague advice without identifying the VOLATILE + function and explaining why it causes per-row re-evaluation. + `, + }); + + checks.push({ + name: 'correctly diagnosed VOLATILE initplan cause and proposed fix', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { + passed: namedPostsTable && inspectedStats && identifiedCause && proposedFix && verdict.passed, + checks, + }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-api-rls-initplan/PROMPT.md b/evals/o11y-investigate-api-rls-initplan/PROMPT.md new file mode 100644 index 00000000..135c1ddf --- /dev/null +++ b/evals/o11y-investigate-api-rls-initplan/PROMPT.md @@ -0,0 +1,24 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - data-api + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our dashboard feed takes about 800ms to load even though the posts table only +has around 5,000 rows and they fit comfortably in memory. The query performance +dashboard shows the posts SELECT with an unusually high mean execution time — +way above what the I/O cost would justify. The same query runs in under 5ms +when we bypass RLS, but it's an order of magnitude slower for authenticated +users through the API. + +Can you dig into the query statistics and the RLS setup to figure out why +authenticated reads are so much slower and tell us what to fix? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-api-rls-initplan/remote/project.sql b/evals/o11y-investigate-api-rls-initplan/remote/project.sql new file mode 100644 index 00000000..8c5d4a7f --- /dev/null +++ b/evals/o11y-investigate-api-rls-initplan/remote/project.sql @@ -0,0 +1,80 @@ +-- Starting state (probe: api-rls-initplan). +-- Authenticated reads on posts are ~10x slower than expected. +-- The RLS policy calls public.get_current_user_id(), a VOLATILE wrapper +-- function that extracts the user id from the JWT claim. Because it's VOLATILE, +-- Postgres cannot hoist it into an initplan — it re-evaluates the function for +-- every row scanned instead of once per query. The data fits in memory — +-- blks_read is low — but mean_exec_time is high relative to I/O, signalling +-- per-row CPU overhead. + +-- Eval-scoped compatibility table created first so it is always present +-- even if later DDL fails. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- The slow query: selecting posts with RLS. blks_read is low (data is cached) + -- but mean_exec_time is ~650ms — far above what I/O would justify. + -- This is the VOLATILE initplan footprint: get_current_user_id() is called + -- per row, not once per query. + (5001, + 'SELECT "posts"."id", "posts"."user_id", "posts"."title", "posts"."body", "posts"."published_at" FROM "posts" WHERE "posts"."user_id" = public.get_current_user_id() ORDER BY "posts"."published_at" DESC', + 28400, 142000, 18460000.0, 650.0, 1820.0, 710000, 1400), + -- Same posts table, direct admin query bypassing RLS — fast. + (5002, + 'SELECT count(*) FROM public.posts', + 120, 120, 360.0, 3.0, 8.0, 6000, 50), + -- Unrelated fast query for contrast. + (5003, + 'SELECT "orders"."id", "orders"."status" FROM "orders" WHERE "orders"."user_id" = $1', + 19200, 19200, 1920.0, 0.1, 2.0, 96000, 400); + +CREATE TABLE public.posts ( + id bigserial PRIMARY KEY, + user_id uuid NOT NULL, + title text NOT NULL, + body text NOT NULL, + published_at timestamptz DEFAULT now() +); + +ALTER TABLE public.posts ENABLE ROW LEVEL SECURITY; + +-- VOLATILE wrapper around the JWT claim lookup. Because this is VOLATILE, +-- Postgres re-evaluates it per row inside the RLS policy — the initplan problem. +-- The fix is to either make the function STABLE, or wrap the call in (SELECT ...) +-- in the policy USING clause to force a single evaluation. +CREATE OR REPLACE FUNCTION public.get_current_user_id() + RETURNS uuid LANGUAGE sql VOLATILE + AS $$ SELECT NULLIF(current_setting('request.jwt.claims', true)::jsonb->>'sub', '')::uuid $$; + +CREATE POLICY "users can read own posts" + ON public.posts + FOR SELECT + TO authenticated + USING (user_id = public.get_current_user_id()); + +-- Grant SELECT so the authenticated role can actually read the table. +-- Without this the agent gets sidetracked debugging missing privileges +-- rather than diagnosing the performance problem. +GRANT SELECT ON public.posts TO authenticated; + +INSERT INTO public.posts (user_id, title, body) +SELECT + gen_random_uuid(), + 'Post title ' || g, + repeat('body content ', 20) +FROM generate_series(1, 5000) AS g; diff --git a/evals/o11y-investigate-api-slow-endpoint/EVAL.ts b/evals/o11y-investigate-api-slow-endpoint/EVAL.ts new file mode 100644 index 00000000..7d33ff69 --- /dev/null +++ b/evals/o11y-investigate-api-slow-endpoint/EVAL.ts @@ -0,0 +1,61 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +// Fault: GET /rest/v1/orders is slow because the orders table has no index on +// user_id. The agent must correlate API request logs with pg_stat_statements to +// identify the sequential scan and propose adding an index. + +const scorer: ToolScorer = async (ctx) => { + const report = ctx.agentReport ?? ''; + const transcript = serializeTranscript(ctx.transcript, { includeToolCallInputs: true }); + + const namedOrdersTable = /\borders\b/i.test(report); + const inspectedLogs = /\/rest\/v1\/orders|orders endpoint|api log/i.test(transcript); + const inspectedStats = /pg_stat_statements|query performance|query insights/i.test(transcript); + const proposedIndex = + /\bcreate index\b|\badd.*index\b|\bindex on\b/i.test(report) || + /\buser_id\b.*\bindex\b|\bindex\b.*\buser_id\b/i.test(report); + + const checks: CheckResult[] = [ + { name: 'identified orders as the slow endpoint target', passed: namedOrdersTable }, + { name: 'inspected API logs', passed: inspectedLogs }, + { name: 'inspected query statistics', passed: inspectedStats }, + { name: 'proposed adding an index', passed: proposedIndex }, + ]; + + const verdict = await judge({ + input: transcript, + rubric: stripIndent` + Pass if the agent: + 1. Identifies that GET /rest/v1/orders requests are consistently slow (2–3s). + 2. Correlates the API logs with query statistics to find the underlying + SELECT on the orders table. + 3. Identifies that the query does a sequential scan (high blks_read, no + index on user_id or similar filter column). + 4. Proposes a concrete fix — at minimum, creating an index on the filter + column (user_id or equivalent). + + Fail if the agent only looks at logs without checking query statistics, + blames the wrong table, or gives vague advice without a concrete index + recommendation. + `, + }); + + checks.push({ + name: 'correctly traced slow endpoint to seq scan and proposed index', + passed: verdict.passed, + judgeNotes: verdict.notes, + }); + + return { + passed: namedOrdersTable && inspectedStats && proposedIndex && verdict.passed, + checks, + }; +}; + +export default scorer; diff --git a/evals/o11y-investigate-api-slow-endpoint/PROMPT.md b/evals/o11y-investigate-api-slow-endpoint/PROMPT.md new file mode 100644 index 00000000..3074559b --- /dev/null +++ b/evals/o11y-investigate-api-slow-endpoint/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - data-api + - database +topic: + - observability + - sql +motivation: apps/docs/content/troubleshooting/slow-queries.mdx +--- + +Our orders page is taking 2–3 seconds to load and users are complaining. The API +logs show that requests to the orders endpoint are consistently slow. Can you dig +into what's causing the latency and tell me what we should fix? + +Report what you find and propose a concrete fix. diff --git a/evals/o11y-investigate-api-slow-endpoint/remote/logs.jsonl b/evals/o11y-investigate-api-slow-endpoint/remote/logs.jsonl new file mode 100644 index 00000000..92085d49 --- /dev/null +++ b/evals/o11y-investigate-api-slow-endpoint/remote/logs.jsonl @@ -0,0 +1,8 @@ +{"id":"log-001","ts":"2026-08-01T09:00:01Z","source":"edge","level":"info","message":"GET /rest/v1/orders 200 2341ms","metadata":{"method":"GET","path":"/rest/v1/orders","status_code":200,"duration_ms":2341}} +{"id":"log-002","ts":"2026-08-01T09:00:14Z","source":"edge","level":"info","message":"GET /rest/v1/customers 200 4ms","metadata":{"method":"GET","path":"/rest/v1/customers","status_code":200,"duration_ms":4}} +{"id":"log-003","ts":"2026-08-01T09:00:28Z","source":"edge","level":"info","message":"GET /rest/v1/orders 200 2289ms","metadata":{"method":"GET","path":"/rest/v1/orders","status_code":200,"duration_ms":2289}} +{"id":"log-004","ts":"2026-08-01T09:00:41Z","source":"edge","level":"info","message":"POST /rest/v1/orders 201 8ms","metadata":{"method":"POST","path":"/rest/v1/orders","status_code":201,"duration_ms":8}} +{"id":"log-005","ts":"2026-08-01T09:00:55Z","source":"edge","level":"info","message":"GET /rest/v1/orders 200 2401ms","metadata":{"method":"GET","path":"/rest/v1/orders","status_code":200,"duration_ms":2401}} +{"id":"log-006","ts":"2026-08-01T09:01:08Z","source":"edge","level":"info","message":"GET /rest/v1/customers 200 3ms","metadata":{"method":"GET","path":"/rest/v1/customers","status_code":200,"duration_ms":3}} +{"id":"log-007","ts":"2026-08-01T09:01:22Z","source":"edge","level":"info","message":"GET /rest/v1/orders 200 2318ms","metadata":{"method":"GET","path":"/rest/v1/orders","status_code":200,"duration_ms":2318}} +{"id":"log-008","ts":"2026-08-01T09:01:35Z","source":"edge","level":"info","message":"GET /rest/v1/orders 200 2376ms","metadata":{"method":"GET","path":"/rest/v1/orders","status_code":200,"duration_ms":2376}} diff --git a/evals/o11y-investigate-api-slow-endpoint/remote/project.sql b/evals/o11y-investigate-api-slow-endpoint/remote/project.sql new file mode 100644 index 00000000..9026ec9c --- /dev/null +++ b/evals/o11y-investigate-api-slow-endpoint/remote/project.sql @@ -0,0 +1,63 @@ +-- Starting state (probe: api-slow-endpoint). +-- GET /rest/v1/orders is slow (~2.3s). The orders table is filtered by +-- user_id but has no index on that column, causing a sequential scan +-- on every request. Evidence lives in faked query statistics. + +CREATE TABLE public.customers ( + id serial PRIMARY KEY, + email text UNIQUE NOT NULL +); + +CREATE TABLE public.orders ( + id bigserial PRIMARY KEY, + user_id uuid NOT NULL, + customer_id int NOT NULL REFERENCES public.customers(id), + total_cents int NOT NULL DEFAULT 0, + status text NOT NULL DEFAULT 'pending', + created_at timestamptz DEFAULT now() + -- Intentionally no index on user_id +); + +INSERT INTO public.customers (email) +SELECT 'user' || g || '@example.com' FROM generate_series(1, 200) AS g; + +INSERT INTO public.orders (user_id, customer_id, total_cents, status) +SELECT + gen_random_uuid(), + (floor(random() * 200) + 1)::int, + (floor(random() * 10000) + 1)::int, + (ARRAY['pending','complete','cancelled'])[floor(random()*3+1)::int] +FROM generate_series(1, 50000); + +-- Eval-scoped compatibility table for Supabase's Query Performance report, +-- backed by pg_stat_statements in real projects. +CREATE TABLE pg_stat_statements ( + userid oid NOT NULL DEFAULT 10::oid, + dbid oid NOT NULL DEFAULT 5::oid, + queryid bigint PRIMARY KEY, + query text NOT NULL, + calls bigint NOT NULL, + rows bigint NOT NULL DEFAULT 0, + total_exec_time double precision NOT NULL, + mean_exec_time double precision NOT NULL, + max_exec_time double precision NOT NULL, + shared_blks_hit bigint NOT NULL DEFAULT 0, + shared_blks_read bigint NOT NULL DEFAULT 0 +); + +INSERT INTO pg_stat_statements + (queryid, query, calls, rows, total_exec_time, mean_exec_time, max_exec_time, + shared_blks_hit, shared_blks_read) +VALUES + -- The slow offender: sequential scan on orders because user_id has no index. + -- High blks_read vs blks_hit ratio signals a seq scan touching many pages. + (3001, + 'SELECT "orders"."id", "orders"."user_id", "orders"."customer_id", "orders"."total_cents", "orders"."status", "orders"."created_at" FROM "orders" WHERE "orders"."user_id" = $1 ORDER BY "orders"."created_at" DESC', + 18420, 92100, 42700000.0, 2319.0, 3801.0, 42000, 890000), + -- Fast queries for contrast. + (3002, + 'SELECT "customers"."id", "customers"."email" FROM "customers" WHERE "customers"."id" = $1', + 9200, 9200, 920.0, 0.10, 1.4, 55000, 200), + (3003, + 'INSERT INTO "orders" ("user_id", "customer_id", "total_cents", "status") VALUES ($1, $2, $3, $4)', + 4100, 4100, 820.0, 0.20, 3.1, 24600, 600); diff --git a/o11y-README.md b/o11y-README.md index b017e891..4234330b 100644 --- a/o11y-README.md +++ b/o11y-README.md @@ -1,12 +1,12 @@ # o11y Eval Suite -The `o11y`-prefixed evals come from two sources: the chaos-o11y probe library (which maps Supabase failure modes to advisor lints) and the pgbot gap analysis (`pgbot-gap-analysis.md`, a cross-reference of 61 production Postgres health checks against existing advisor and eval coverage). Each eval defines an injected failure state, expected agent behavior, and a judge rubric. 32 MCP evals were validated locally against a PGlite-backed harness; 16 pgbot-derived evals and 11 CLI stubs are defined but not yet run. Probe YAML source files live in `.context/probes/`; conversion decisions and tradeoffs are documented in `PROBE-CONVERSION.md`. +The `o11y`-prefixed evals come from three sources: the chaos-o11y probe library (which maps Supabase failure modes to advisor lints), the pgbot gap analysis (`pgbot-gap-analysis.md`, a cross-reference of 61 production Postgres health checks against existing advisor and eval coverage), and a Data API / PostgREST investigation gap analysis (motivated by real customer cases where runtime API performance problems were not caught by static linting). Each eval defines an injected failure state, expected agent behavior, and a judge rubric. 32 MCP evals were validated locally against a PGlite-backed harness; 16 pgbot-derived evals, 4 Data API evals, and 11 CLI stubs are defined but not yet run. Probe YAML source files live in `.context/probes/`; conversion decisions and tradeoffs are documented in `PROBE-CONVERSION.md`. --- ## MCP evals — validated locally -54 evals total. 38 previously defined (32 validated, 5 fail as valid signal, 1 blocked); 16 pgbot-derived (not yet validated). +58 evals total. 38 previously defined (32 validated, 5 fail as valid signal, 1 blocked); 16 pgbot-derived (not yet validated); 4 Data API / PostgREST (not yet validated). | n | # | Probe | Product | Eval dir | Advisor | Local result | |---|---|---|---|---|---|---| @@ -64,6 +64,10 @@ The `o11y`-prefixed evals come from two sources: the chaos-o11y probe library (w | 52 | — | high_rollback_ratio | Database | o11y-investigate-postgres-high-rollback-ratio | pgbot | not validated | | 53 | — | autovacuum_starved | Database | o11y-investigate-postgres-autovacuum-starved | pgbot | not validated | | 54 | — | replica_lag_time | Database | o11y-investigate-postgres-replica-lag | pgbot | not validated | +| 55 | — | api_slow_endpoint | PostgREST | o11y-investigate-api-slow-endpoint | data-api | not validated | +| 56 | — | api_high_load_source | PostgREST | o11y-investigate-api-high-load-source | data-api | not validated | +| 57 | — | api_rls_initplan | PostgREST | o11y-investigate-api-rls-initplan | data-api | not validated | +| 58 | — | api_embedding_nplus1 | PostgREST | o11y-investigate-api-embedding-nplus1 | data-api | not validated | --- @@ -97,6 +101,7 @@ The `o11y`-prefixed evals come from two sources: the chaos-o11y probe library (w | p0–p3 | No advisor lint exists for this pattern; number is a community-assigned priority (p0 = highest urgency) | | PR #36781 | Coverage tracked in an in-progress platform PR | | pgbot | Eval derived from the pgbot gap analysis (`pgbot-gap-analysis.md`); covers checks that pgbot surfaces but no Splinter advisor or prior o11y eval exists for | +| data-api | Eval targeting PostgREST / Data API runtime performance gaps — correlates API request logs with pg_stat_statements to diagnose slow endpoints, load attribution, RLS initplan, and resource embedding N+1 | ---