diff --git a/.agents/skills/writing-evals/SKILL.md b/.agents/skills/writing-evals/SKILL.md index 06cfd02ec133..f27747bb113f 100644 --- a/.agents/skills/writing-evals/SKILL.md +++ b/.agents/skills/writing-evals/SKILL.md @@ -48,7 +48,9 @@ async def eval_my_thing(ctx: EvalContext) -> None: - One suite = one Braintrust experiment. Bundle related cases into one suite; split only when the scorecard would be heterogeneous (self-skipping scorers stretch a shared scorer list across mildly different cases, but a scorecard where half the scorers skip half the cases is a sign to split). - `experiment_name` is the Braintrust history key — renaming it resets cross-run comparison. Existing sandboxed suites end in `-cli` because the MCP server serves the `cli` surface. -- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust and PostHog, including available input, output, and expected values. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written. PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md). +- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when `offline_suite` is configured. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written. Set `send_to_posthog=True` with `offline_suite` to publish only to PostHog, or `send_to_posthog=False` to disable that destination independently. These options also apply to one-shot suites. Ordinary SDK and trace clients still honor `OPT_OUT_CAPTURE`. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md). +- To enroll a suite in PostHog reporting, register its scorer kinds and immutable version UUIDs in the owning `offline_config.py` catalog, then pass `offline_suite=offline_suite("/::", metrics=[...])` to the runner. Use the PostHog AI catalog for built-in suites and the Slack app catalog for Slack suites. List every explicit scorer's `_name()` in `metrics`; the catalog adds `exit_code_zero` only for sandboxed suites. For one-shot suites, pass `suite_kind=SUITE_KIND`. The enrollment test discovers catalog-using modules and checks every suite function against its scorer list. +- Uploading and tagging require `POSTHOG_OFFLINE_EVAL_API_KEY` to be a personal key with `offline_evaluation_ingestion:write` and `evaluation:write`. Ingestion-only personal or project secret keys can upload without tags. Upload success is recorded before the separate, bounded tag update, so a tagging failure or interruption keeps the results link. ## One-shot suites @@ -135,7 +137,7 @@ class MyCheck(Scorer): Rules that apply to both: -- The harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected. +- The sandboxed harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected. One-shot suites do not receive it. - Reuse the other generic scorers first: `NoToolCall`, `RequiredToolCall`, and `LastToolCallNot` (all in `products.posthog_ai.eval_harness.scorers`). - Always parse logs via `LogParser.cached(...)` so a case's log is parsed once, not once per scorer. - Count only successful tool calls (`is_error=False`) — the agent is free to attempt and fail. diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md index 4f13ce74d02c..450f46879499 100644 --- a/docs/internal/ai-offline-evaluation-reporting.md +++ b/docs/internal/ai-offline-evaluation-reporting.md @@ -1,7 +1,7 @@ # Offline evaluation result reporting The evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust. -The SQL suite also publishes those results to PostHog's offline experiment API when configured. +Public suites under `products/posthog_ai/evals/` and `products/slack_app/evals/` also publish those results to PostHog's offline experiment API when configured. The Signals pytest suites reuse the same publisher for their completed results. Agent and judge trace capture remains available. Reporting does not run the agent or scorers again. @@ -9,9 +9,11 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../ ## Capture settings -Evaluation result uploads to Braintrust and PostHog share the `no_send_logs` setting. +`no_send_logs` controls Braintrust uploads and, by default, PostHog result uploads. `SandboxedPublicEval` sets `no_send_logs=False` and uploads to Braintrust, plus PostHog when the suite has offline publishing configured. -`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service; local logs are still written. +`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written. +Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled. +Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites. Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`. The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events. @@ -33,25 +35,27 @@ The scoring phase includes time waiting for the experiment results, rather than A failed final metadata upload produces a warning and keeps the completed results available locally. A scorer failure marks the case as an error. Statistics still include scores from other successful scorers. -## SQL offline experiment pilot +## Configured offline experiment suites -The `sql/eval_sql::eval_sql` suite (`sandboxed-sql-cli`) opts into the offline experiment publisher. -Other public suites continue reporting scores to Braintrust until they opt into offline publishing. +Configured public PostHog AI and Slack app suites opt into the offline experiment publisher, including both sandboxed and one-shot suites. +The 11 experiment-skill suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog. After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call. -`no_send_logs=True` suppresses this upload too. +Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too. -The [SQL suite definition](../../products/posthog_ai/evals/sql/eval_sql.py) pins destination project `2` and the seven immutable scorer version UUIDs. +The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and [Slack app catalog](../../products/slack_app/evals/offline_config.py) pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags. +Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric. +The SQL suite retains its seven existing scorer versions. Provide the API key through the environment before running the suite: -| Variable | Purpose | -| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | -| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal or project secret API key with `offline_evaluation_ingestion:write`. A public project token cannot authenticate uploads. | -| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. | +| Variable | Purpose | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal API key with `offline_evaluation_ingestion:write` and `evaluation:write` to upload and tag results. Ingestion-only personal or project secret keys can upload without tags. A public project token cannot authenticate uploads. | +| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. | The publisher normalizes the cloud ingestion hosts (`us.i.posthog.com`, `eu.i.posthog.com`) and the legacy `app.posthog.com` host to their app origins. Uploads, result links, and trace-destination checks use that same origin. -The suite uses these existing scorers: +The SQL suite uses these existing scorers: | Metric name | Scorer kind | | ------------------------------------ | -------------------------------- | @@ -72,15 +76,32 @@ export POSTHOG_OFFLINE_EVAL_API_KEY='your-secret-key' hogli evals sql/eval_sql::eval_sql ``` -Without an API key, the SQL run reports that offline uploads are disabled. +Without an API key, each enrolled suite reports that offline uploads are disabled. Keys containing whitespace are rejected before evaluation starts; remove any spaces, tabs, or pasted trailing newlines. Invalid configuration produces a warning and skips this destination; Braintrust and local reporting continue. An upload failure also warns without changing the evaluation's score gate. The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status. +### Scorer and experiment tags + +Score definitions and experiment runs use the same team and component tags: + +| Suites | Tags | +| -------------------- | ------------------------------------------------------------------------------------------- | +| PostHog AI | `team:surfaces`, `component:posthog-ai/` (for example `sql` or `product-analytics`) | +| Slack app | `team:surfaces`, `component:slack-app/` (for example `followup-classifier`) | +| Signals grouping | `team:self-driving`, `component:signals/grouping` | +| Signals scout safety | `team:self-driving`, `component:signals/scout-safety` | + +`OfflineEvalSuite.tags` supplies the experiment tags. Delivery records the successful upload and results link once the experiment has completed, then calls the publisher's `tag()` method to update tags through the Python SDK's `experiment.set_tags(...)`. Tagging has its own five-second deadline outside the upload deadline. Tags are mutable metadata, separate from the immutable creation declaration and scorer versions. + +To upload **and tag** experiments, `POSTHOG_OFFLINE_EVAL_API_KEY` must be a personal API key with both `offline_evaluation_ingestion:write` and `evaluation:write`. An ingestion-only personal or project secret key can still upload results, but the tag update will warn. Tagging failures, timeouts, and shutdown interruptions retain the successful upload status and results link and do not change the evaluation score gate. + +Each prepared experiment's tag update is attempted once per publisher, including when delivery pauses and resumes. A failed or interrupted replacement is not repeated automatically, because it may already have succeeded or someone may have edited the tags afterward. Inspect the experiment's tags before manually retrying a replacement. + ### CI configuration -CI jobs running this suite need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment. +CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment. Map the dedicated `ANTHROPIC_OFFLINE_EVALS_API_KEY` and `OPENAI_OFFLINE_EVALS_API_KEY` Actions secrets to `LLM_GATEWAY_ANTHROPIC_API_KEY` and `OPENAI_API_KEY` for the harness and judges. Publishing failures remain warnings, so check the final harness upload status even when a CI job is green. The harness sets the SDK's `run_source` to `ci` when `CI` is set, including manual Actions runs, and to `local` otherwise. @@ -124,6 +145,7 @@ Existing local logs, enabled case exports, and Braintrust results remain availab Both `SandboxedEval` and `OneShotEval` accept `offline_suite=OfflineEvalSuite(...)`. Each suite declares its destination `project_id`, selects at least one metric in `scorer_kinds`, and pins the corresponding immutable version UUIDs in `scorer_versions`. These fields belong to `OfflineEvalSuite` in code. Scorer version UUIDs must be distinct for the suite's enrolled metrics. +The AI and Slack catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics. Numeric scores can use the domain of their configured scorer version. The API validates bounds and steps; the harness does not restrict every numeric metric to `[0, 1]`. `OfflineEvalSuite.experiment_options` accepts the SDK's `ExperimentOptions` for external dataset identifiers, `prompt_version`, model/application versions, and `run_source`. @@ -185,7 +207,7 @@ Runs appear in the offline experiments UI. ## Postgres experiment ingestion The project API accepts offline experiment results behind the `ai-observability-offline-evaluations` feature flag. -The configured SQL pilot calls this API to publish completed scorer results. +The configured public suites call this API to publish completed scorer results. Experiments and their items, results, and payloads belong to the exact project/environment in the request path. Scorers and hosted datasets must belong to that same environment. diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.md index 87a88875a9c1..755d2b98c1ca 100644 --- a/products/posthog_ai/eval_harness/README.md +++ b/products/posthog_ai/eval_harness/README.md @@ -387,22 +387,25 @@ The summary includes the trial ID, the agent trace ID when available, phase dura Case names and `--eval` filters stay the same. `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when the suite has offline publishing configured. -`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written. -PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients. +`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service by default; local logs are still written. +Both sandboxed and one-shot suites can set `send_to_posthog=True` with `offline_suite` to publish only to PostHog while Braintrust uploads remain disabled, or `send_to_posthog=False` to disable PostHog uploads independently. +The 11 experiment-skill suites use this PostHog-only mode. +PostHog result uploads default to following `no_send_logs`, independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md) for capture settings and scope. ### SQL offline experiment pilot The `sql/eval_sql::eval_sql` suite can also send its seven scorer results to PostHog's offline experiment API. It publishes the same completed case results through the Python SDK and keeps sending to Braintrust. -Other public suites report scores to Braintrust until they opt into offline publishing. -Agent and judge trace capture remains available, and `no_send_logs=True` suppresses offline result uploads. +PostHog AI and Slack app public suites also opt into offline publishing with their own tagged scorer versions. +Agent and judge trace capture remains available. `no_send_logs=True` suppresses offline result uploads unless `send_to_posthog=True` is explicitly set. -Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal or project secret key with `offline_evaluation_ingestion:write` and access to project `2`. +Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged. The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code. `POSTHOG_OFFLINE_EVAL_HOST` defaults to `https://us.posthog.com`. -The [pilot configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#sql-offline-experiment-pilot) lists the metrics and explains local and CI key configuration. +The [suite configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#configured-offline-experiment-suites) lists the metrics and explains local and CI key configuration. Delivery runs in a bounded background queue after local results are saved. Prepared uploads stay in memory, with up to two uploads running concurrently, at most 32 queued or active uploads, and a 60-second deadline per active upload. +A completed upload records its results link before tagging begins. Tagging has a separate five-second deadline; failures, timeouts, and shutdown interruptions only warn and preserve the successful upload status. Unfinished uploads pause before infrastructure teardown. After teardown, the harness retries the same prepared objects and identities with fresh deadlines, waiting up to the full 60-second upload deadline for delivery to finish. Requests use a 5-second timeout and up to two retries within that overall deadline. Terminal API failures trigger a best-effort attempt to mark the experiment failed, limited to 5 seconds; pauses and interruptions leave it resumable. Both sandboxed and one-shot suites can opt in through `offline_suite`. Rich scorer recording runs only when offline publishing is configured and uploads are allowed. diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.py index 663c38261065..6f9d5047f92a 100644 --- a/products/posthog_ai/eval_harness/base.py +++ b/products/posthog_ai/eval_harness/base.py @@ -166,12 +166,14 @@ def __init__( no_send_logs: bool, engine: EvalEngine | None = None, offline_suite: OfflineEvalSuite | None = None, + send_to_posthog: bool | None = None, ) -> None: self.experiment_name = experiment_name self.cases = cases self.ctx = ctx self.is_public = is_public self.no_send_logs = no_send_logs + self.send_to_posthog = not no_send_logs if send_to_posthog is None else send_to_posthog self.offline_suite = offline_suite self.offline_publisher: OfflineEvalPublisher | None = None self.started_at = datetime.now(UTC) @@ -394,7 +396,7 @@ async def _finalize(self, result: ExperimentResult) -> None: await self._enqueue_offline_upload(result) async def _enqueue_offline_upload(self, result: ExperimentResult) -> None: - if self.offline_publisher is None or self.no_send_logs: + if self.offline_publisher is None or not self.send_to_posthog: return try: upload = await asyncio.to_thread( @@ -471,7 +473,7 @@ async def _report_results(self, result: ExperimentResult) -> None: async def run(self) -> ExperimentResult: eval_cases = self._build_eval_cases() - if self.offline_suite is not None and not self.no_send_logs: + if self.offline_suite is not None and self.send_to_posthog: try: settings = OfflineEvalSettings.from_env(self.offline_suite) if settings is not None: @@ -607,6 +609,7 @@ def __init__( is_public: bool, no_send_logs: bool, offline_suite: OfflineEvalSuite | None = None, + send_to_posthog: bool | None = None, ) -> None: if any(isinstance(scorer, ExitCodeZero) for scorer in scorers): raise ValueError("ExitCodeZero is added by the sandboxed eval harness; remove it from scorers") @@ -618,6 +621,7 @@ def __init__( is_public=is_public, no_send_logs=no_send_logs, offline_suite=offline_suite, + send_to_posthog=send_to_posthog, ) # Narrow the infra-backed optionals once: they are None only when the # harness didn't boot sandbox infrastructure for this run. @@ -864,6 +868,7 @@ async def SandboxedEval( is_public: bool = False, no_send_logs: bool = True, offline_suite: OfflineEvalSuite | None = None, + send_to_posthog: bool | None = None, ) -> ExperimentResult: """Run a sandboxed agent evaluation suite via Braintrust. @@ -888,6 +893,7 @@ async def SandboxedEval( is_public=is_public, no_send_logs=no_send_logs, offline_suite=offline_suite, + send_to_posthog=send_to_posthog, ) return await run.run() diff --git a/products/posthog_ai/eval_harness/harness/offline_delivery.py b/products/posthog_ai/eval_harness/harness/offline_delivery.py index ab8eb84d9ab9..232d31a8c0c7 100644 --- a/products/posthog_ai/eval_harness/harness/offline_delivery.py +++ b/products/posthog_ai/eval_harness/harness/offline_delivery.py @@ -66,6 +66,7 @@ async def _deliver(self, job: _PendingUpload) -> None: await self._report_failed(job, reason) else: await self._reporter.record_posthog_offline_result(job.upload.experiment_name, "uploaded", url=url) + await job.publisher.tag(job.upload) def _finished(self, task: asyncio.Task[None]) -> None: self._tasks.pop(task, None) diff --git a/products/posthog_ai/eval_harness/offline_results.py b/products/posthog_ai/eval_harness/offline_results.py index 5cfe9e1dcad9..02d4bc5014f9 100644 --- a/products/posthog_ai/eval_harness/offline_results.py +++ b/products/posthog_ai/eval_harness/offline_results.py @@ -4,6 +4,7 @@ import re import json import asyncio +import logging from collections import Counter from collections.abc import Mapping from contextlib import suppress @@ -30,6 +31,15 @@ from .engines.types import CaseResult, ExperimentResult, ScorerResult +logger = logging.getLogger(__name__) +TAG_TIMEOUT_SECONDS = 5.0 + + +@frozen +class OfflineScorer: + kind: ScorerKind + version: UUID + @frozen class OfflineEvalSuite: @@ -37,8 +47,21 @@ class OfflineEvalSuite: project_id: int scorer_kinds: Mapping[str, ScorerKind] scorer_versions: Mapping[str, UUID] + tags: tuple[str, ...] = () experiment_options: ExperimentOptions = field(default_factory=lambda: ExperimentOptions()) + @classmethod + def from_scorers( + cls, *, key: str, project_id: int, scorers: Mapping[str, OfflineScorer], tags: tuple[str, ...] + ) -> OfflineEvalSuite: + return cls( + key=key, + project_id=project_id, + scorer_kinds={metric: scorer.kind for metric, scorer in scorers.items()}, + scorer_versions={metric: scorer.version for metric, scorer in scorers.items()}, + tags=tags, + ) + class OfflineUploadError(Exception): @staticmethod @@ -163,6 +186,7 @@ class OfflineEvalUpload: experiment_name: str experiment: ExperimentOptions results: tuple[EvaluationResult, ...] + tags: tuple[str, ...] = () class OfflineEvalPublisher: @@ -176,6 +200,7 @@ def __init__( self.settings = settings self.suite = suite self.application_trace_ids_enabled = application_trace_ids_enabled + self._attempted_tag_updates: set[str] = set() def _item_payload( self, @@ -377,19 +402,54 @@ def prepare( experiment_name=experiment_name, experiment=experiment, results=tuple(scores), + tags=self.suite.tags, ) + def _client(self) -> AsyncOfflineEvaluations: + return AsyncOfflineEvaluations( + host=self.settings.host, + project_id=self.settings.project_id, + secret_key=self.settings.api_key.get_secret_value(), + timeout=self.settings.request_timeout, + max_retries=self.settings.max_retries, + ) + + async def tag(self, upload: OfflineEvalUpload) -> None: + if upload.host != self.settings.host or upload.project_id != self.settings.project_id: + raise OfflineUploadError("Upload destination differs from the configured host/project") + experiment_id = str(upload.experiment["id"]) + if not upload.tags or experiment_id in self._attempted_tag_updates: + return + self._attempted_tag_updates.add(experiment_id) + try: + async with asyncio.timeout(TAG_TIMEOUT_SECONDS), self._client() as client: + await client.resume_experiment(experiment_id).set_tags(upload.tags) + except EvaluationAPIError as error: + logger.warning( + "PostHog offline results uploaded, but tagging failed for experiment %s (status=%s, code=%s). " + "Tagging requires a personal key with evaluation:write. Tag updates are not retried.", + experiment_id, + error.status, + OfflineUploadError._safe_token(error.code, path=False), + ) + except TimeoutError: + logger.warning( + "PostHog offline results uploaded, but tagging timed out for experiment %s. " + "Tag updates are not retried.", + experiment_id, + ) + except asyncio.CancelledError: + logger.warning( + "PostHog offline tag update interrupted for experiment %s; inspect its tags before retrying.", + experiment_id, + ) + raise + async def publish(self, upload: OfflineEvalUpload) -> str: if upload.host != self.settings.host or upload.project_id != self.settings.project_id: raise OfflineUploadError("Upload destination differs from the configured host/project") try: - async with AsyncOfflineEvaluations( - host=self.settings.host, - project_id=self.settings.project_id, - secret_key=self.settings.api_key.get_secret_value(), - timeout=self.settings.request_timeout, - max_retries=self.settings.max_retries, - ) as client: + async with self._client() as client: experiment = await client.create_experiment(name=upload.experiment_name, **upload.experiment) try: await experiment.upload_results(upload.results) diff --git a/products/posthog_ai/eval_harness/one_shot.py b/products/posthog_ai/eval_harness/one_shot.py index d7b79a8dfa6f..ec3219a59ad3 100644 --- a/products/posthog_ai/eval_harness/one_shot.py +++ b/products/posthog_ai/eval_harness/one_shot.py @@ -52,6 +52,7 @@ def __init__( no_send_logs: bool, task_fn: OneShotTaskFn, offline_suite: OfflineEvalSuite | None = None, + send_to_posthog: bool | None = None, ) -> None: super().__init__( experiment_name=experiment_name, @@ -61,6 +62,7 @@ def __init__( is_public=is_public, no_send_logs=no_send_logs, offline_suite=offline_suite, + send_to_posthog=send_to_posthog, ) self._task_fn = task_fn @@ -114,6 +116,7 @@ async def OneShotEval( is_public: bool = False, no_send_logs: bool = True, offline_suite: OfflineEvalSuite | None = None, + send_to_posthog: bool | None = None, ) -> ExperimentResult: """Run a one-shot evaluation suite via Braintrust. @@ -132,6 +135,7 @@ async def OneShotEval( no_send_logs=no_send_logs, task_fn=task, offline_suite=offline_suite, + send_to_posthog=send_to_posthog, ) return await run.run() @@ -140,4 +144,4 @@ async def OneShotEval( """One-shot evaluation whose Braintrust experiment is publicly accessible.""" OneShotPrivateEval = partial(OneShotEval, is_public=False, no_send_logs=True) -"""One-shot evaluation that is not accessible publicly; local logs are the record.""" +"""One-shot evaluation with Braintrust uploads disabled; PostHog publishing is opt-in.""" diff --git a/products/posthog_ai/eval_harness/test/offline_api.py b/products/posthog_ai/eval_harness/test/offline_api.py index f1e75ada44bf..df16412d148e 100644 --- a/products/posthog_ai/eval_harness/test/offline_api.py +++ b/products/posthog_ai/eval_harness/test/offline_api.py @@ -13,10 +13,14 @@ def __init__(self) -> None: self.results: dict[tuple[str, str], object] = {} self.experiment: dict[str, object] = {} self.status = "uploading" + self.tags: list[str] = [] def respond(self, request: httpx.Request) -> httpx.Response: self.requests.append(request) body = json.loads(request.content) + if request.method == "PATCH": + self.tags = body["tags"] + return httpx.Response(200, json={"id": self.experiment["id"], "tags": self.tags}) now = str(self.experiment.get("started_at", body.get("started_at"))) if request.url.path.endswith("/upload/"): item_receipts = [] diff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery.py b/products/posthog_ai/eval_harness/test/test_offline_delivery.py index db7746051d82..a0f07f641c16 100644 --- a/products/posthog_ai/eval_harness/test/test_offline_delivery.py +++ b/products/posthog_ai/eval_harness/test/test_offline_delivery.py @@ -15,6 +15,7 @@ from posthoganalytics.ai.evaluations import EvaluationItem, EvaluationResult from pydantic import SecretStr +from products.posthog_ai.eval_harness import offline_results from products.posthog_ai.eval_harness.engines.types import EvalSummary from products.posthog_ai.eval_harness.harness import reporting from products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDelivery @@ -198,6 +199,61 @@ async def respond(request: httpx.Request) -> httpx.Response: assert "403" in output +@pytest.mark.asyncio +@pytest.mark.parametrize("shutdown", ["pause", "deadline", "interrupt", "tag_timeout"]) +async def test_completed_upload_stays_successful_when_tagging_is_interrupted( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + caplog: pytest.LogCaptureFixture, + shutdown: str, +) -> None: + monkeypatch.setattr(offline_results.logger, "disabled", False) + caplog.set_level("WARNING", logger=offline_results.__name__) + tag_started = asyncio.Event() + release_tag = asyncio.Event() + requests: list[httpx.Request] = [] + server = OfflineServer() + + async def respond(request: httpx.Request) -> httpx.Response: + requests.append(request) + if request.method == "PATCH": + tag_started.set() + await release_tag.wait() + return server.respond(request) + + publisher = _publisher(monkeypatch, respond) + reporter = ProgressReporter(total_suites=1) + await reporter.record_summary( + "experiment-1", EvalSummary(engine_name="braintrust", experiment_name="experiment-1", scores={}) + ) + delivery = OfflineEvalDelivery(reporter) + if shutdown == "tag_timeout": + monkeypatch.setattr(offline_results, "TAG_TIMEOUT_SECONDS", 0) + assert delivery.submit(publisher, replace(_upload(1), tags=("team:example",))) + if shutdown != "tag_timeout": + await asyncio.wait_for(tag_started.wait(), timeout=5) + if shutdown == "pause": + await delivery.pause() + release_tag.set() + elif shutdown == "deadline": + await delivery.drain(timeout=0) + else: + await delivery.cancel() + await delivery.drain() + + reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0) + output = capsys.readouterr().out + assert server.status == "completed" + assert sum(request.url.path.endswith("/complete/") for request in requests) == 1 + assert output.count("POSTHOG OFFLINE") == 1 + assert "PostHog: https://posthog.example.com/project/123/" in output + assert "upload failed" not in output + if shutdown == "tag_timeout": + assert "tagging timed out" in caplog.text + elif shutdown != "pause": + assert "tag update interrupted" in caplog.text + + @pytest.mark.asyncio @pytest.mark.parametrize("shutdown", ["deadline", "interrupt", "before_start", "paused"]) async def test_shutdown_cancels_active_and_waiting_uploads_and_reports_unsaved_results( diff --git a/products/posthog_ai/eval_harness/test/test_offline_results.py b/products/posthog_ai/eval_harness/test/test_offline_results.py index 83436608c9e5..4e3ae913157c 100644 --- a/products/posthog_ai/eval_harness/test/test_offline_results.py +++ b/products/posthog_ai/eval_harness/test/test_offline_results.py @@ -39,10 +39,10 @@ STARTED_AT = datetime(2026, 1, 1, tzinfo=UTC) -def _publisher(*, link_traces: bool = False) -> OfflineEvalPublisher: +def _publisher(*, link_traces: bool = False, suite: OfflineEvalSuite = SUITE) -> OfflineEvalPublisher: return OfflineEvalPublisher( OfflineEvalSettings(project_id=123, api_key=SecretStr("test-only-token"), scorer_versions=SCORER_VERSIONS), - SUITE, + suite, application_trace_ids_enabled=link_traces, ) @@ -368,6 +368,58 @@ def respond(request: httpx.Request) -> httpx.Response: assert url.endswith(f"/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}") +@parameterized.expand( + [ + ("success", None), + ("forbidden", 403), + ("server_error", 503), + ("lost_response", "lost_response"), + ("interrupted", "interrupted"), + ] +) +def test_publish_tags_completed_results_without_repeating_mutable_updates( + _name: str, tag_error: int | str | None +) -> None: + publisher = _publisher(suite=replace(SUITE, tags=("team:example", "component:example"))) + upload = _prepare(publisher, [_case()]) + server = OfflineServer() + tag_requests: list[httpx.Request] = [] + + def respond(request: httpx.Request) -> httpx.Response: + if request.method == "PATCH": + tag_requests.append(request) + assert server.status == "completed" + if isinstance(tag_error, int): + return httpx.Response(tag_error) + response = server.respond(request) + if tag_error == "lost_response": + raise httpx.ReadError("Lost tag acknowledgment", request=request) + if tag_error == "interrupted": + raise asyncio.CancelledError + return response + return server.respond(request) + + with patch("httpx.AsyncClient", new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond))): + assert asyncio.run(publisher.publish(upload)).endswith(EXPERIMENT_ID) + if tag_error == "interrupted": + with pytest.raises(asyncio.CancelledError): + asyncio.run(publisher.tag(upload)) + else: + asyncio.run(publisher.tag(upload)) + assert server.status == "completed" + assert len(server.results) == 2 + assert "tags" not in server.experiment + assert server.tags == ([] if isinstance(tag_error, int) else list(upload.tags)) + + server.tags = ["edited-after-upload"] + asyncio.run(publisher.publish(upload)) + asyncio.run(publisher.tag(upload)) + + assert len(tag_requests) == 1 + assert json.loads(tag_requests[0].content) == {"tags": list(upload.tags)} + assert server.tags == ["edited-after-upload"] + + @parameterized.expand( [ ("bad_request", 400), diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py index 19d29c050f55..bf1d6a95cd7a 100644 --- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py +++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py @@ -7,6 +7,7 @@ import asyncio import logging import warnings +import importlib import subprocess from collections.abc import Iterator from contextlib import contextmanager @@ -41,15 +42,16 @@ ) from products.posthog_ai.eval_harness.harness import reporting from products.posthog_ai.eval_harness.harness.cli import SkillDelivery +from products.posthog_ai.eval_harness.harness.discovery import EvalSuite, discover_suites from products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDelivery from products.posthog_ai.eval_harness.harness.reporting import ProgressReporter, SuiteRunResult +from products.posthog_ai.eval_harness.harness.requirements import SuiteKind from products.posthog_ai.eval_harness.harness.transcript import RunTranscript from products.posthog_ai.eval_harness.harness.trial_stats import ScorerTrialStats from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite from products.posthog_ai.eval_harness.one_shot import OneShotEval from products.posthog_ai.eval_harness.scorers import ExitCodeZero from products.posthog_ai.eval_harness.test.offline_api import OfflineServer -from products.posthog_ai.evals.sql import eval_sql as sql_eval from products.tasks.backend.facade.agents import CustomPromptSandboxContext @@ -379,17 +381,22 @@ def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatc @pytest.mark.parametrize( - "suite_kind,enrolled,no_send_logs,configuration,http_status,delivery_state", + "suite_kind,enrolled,no_send_logs,send_to_posthog,posthog_enabled,configuration,http_status,delivery_state", [ - ("sandboxed", True, False, "valid", 200, "ready"), - ("sandboxed", True, False, "valid", 403, "ready"), - ("sandboxed", True, False, "invalid", 200, "ready"), - ("sandboxed", True, False, "absent", 200, "ready"), - ("sandboxed", True, True, "valid", 200, "ready"), - ("sandboxed", False, False, "valid", 200, "ready"), - ("one-shot", True, False, "valid", 200, "ready"), - ("sandboxed", True, False, "valid", 200, "unavailable"), - ("sandboxed", True, False, "valid", 200, "closed"), + ("sandboxed", True, False, None, True, "valid", 200, "ready"), + ("sandboxed", True, False, None, True, "valid", 403, "ready"), + ("sandboxed", True, False, None, True, "invalid", 200, "ready"), + ("sandboxed", True, False, None, True, "absent", 200, "ready"), + ("sandboxed", True, True, None, False, "valid", 200, "ready"), + ("sandboxed", False, False, None, True, "valid", 200, "ready"), + ("one-shot", True, False, None, True, "valid", 200, "ready"), + ("sandboxed", True, False, None, True, "valid", 200, "unavailable"), + ("sandboxed", True, False, None, True, "valid", 200, "closed"), + ("sandboxed", True, True, True, True, "valid", 200, "ready"), + ("one-shot", True, True, True, True, "valid", 200, "ready"), + ("sandboxed", True, False, False, False, "valid", 200, "ready"), + ("sandboxed", True, True, True, True, "absent", 200, "ready"), + ("sandboxed", False, True, True, True, "valid", 200, "ready"), ], ) @pytest.mark.asyncio @@ -400,6 +407,8 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo suite_kind: Literal["sandboxed", "one-shot"], enrolled: bool, no_send_logs: bool, + send_to_posthog: bool | None, + posthog_enabled: bool, configuration: Literal["valid", "invalid", "absent"], http_status: int, delivery_state: Literal["ready", "unavailable", "closed"], @@ -473,6 +482,7 @@ async def respond(request: httpx.Request) -> httpx.Response: ctx=ctx, is_public=not no_send_logs, no_send_logs=no_send_logs, + send_to_posthog=send_to_posthog, offline_suite=OfflineEvalSuite( key="example/suite", project_id=123, @@ -494,14 +504,14 @@ async def respond(request: httpx.Request) -> httpx.Response: engine.run_experiment.assert_awaited_once() engine_spec = engine.run_experiment.call_args.args[0] assert engine_spec.no_send_logs is no_send_logs - assert engine_spec.record_scorer_results is (enrolled and not no_send_logs and configuration == "valid") + assert engine_spec.record_scorer_results is (enrolled and posthog_enabled and configuration == "valid") assert reporter.mean_score() == 1.0 reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0) output = capsys.readouterr().out assert "Braintrust: https://braintrust.example.com/experiment/test" in output assert "exit_code_zero: 100.0%" in output assert not (tmp_path / "offline-pilot" / "latest" / "posthog-offline-upload.json").exists() - if enrolled and not no_send_logs and configuration == "valid" and delivery_state == "ready": + if enrolled and posthog_enabled and configuration == "valid" and delivery_state == "ready": declaration = json.loads(requests[0].content) assert declaration["name"] == "offline-pilot" assert declaration["expected_item_count"] == 1 @@ -530,34 +540,69 @@ async def respond(request: httpx.Request) -> httpx.Response: ) assert reason in output assert "Results were not saved for retry." in output - elif enrolled and not no_send_logs: + elif enrolled and posthog_enabled: assert "PostHog offline: disabled:" in output assert "POSTHOG_OFFLINE_EVAL_" in output else: assert "POSTHOG OFFLINE" not in output +@pytest.mark.parametrize( + "discovered", + [ + suite + for suite in discover_suites() + if suite.fn.__module__.startswith(("products.posthog_ai.evals.", "products.slack_app.evals.")) + and hasattr(importlib.import_module(suite.fn.__module__), "offline_suite") + ], + ids=lambda suite: suite.id, +) @pytest.mark.asyncio -async def test_sql_suite_enrolls_every_scorer_in_offline_publishing(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_suite_enrolls_every_scorer_in_offline_publishing( + monkeypatch: pytest.MonkeyPatch, discovered: EvalSuite +) -> None: + domain, module_name, suite_kind = discovered.domain, discovered.module_name, discovered.kind + module = importlib.import_module(discovered.fn.__module__) + if domain == "taxonomic_filter": + monkeypatch.setattr(module, "system_one_configured", lambda: True) + if module_name == "eval_event_match": + monkeypatch.setattr(module, "EVENT_MATCH_MODEL", MagicMock()) + else: + monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock()) run = AsyncMock() - monkeypatch.setattr(sql_eval, "SandboxedPublicEval", run) + runner = "SandboxedPrivateEval" if domain == "experiments" else "SandboxedPublicEval" + if suite_kind == SuiteKind.ONE_SHOT: + runner = "OneShotPublicEval" + monkeypatch.setattr(module, runner, run) - await sql_eval.eval_sql(MagicMock()) + await discovered.fn(MagicMock(skill_delivery="bundled")) run.assert_awaited_once() suite = run.call_args.kwargs["offline_suite"] - assert suite.key == "sql/eval_sql::eval_sql" - assert suite.scorer_kinds == { - "exit_code_zero": "boolean", - "no_persistent_insight_save": "boolean", - "execute_sql_called": "boolean", - "answer_tool_not_typed_query": "boolean", - "querying_posthog_data_skill_loaded": "boolean", - "sql_schema_alignment": "numeric", - "sql_result_message_alignment": "numeric", - } + assert suite.key == discovered.id + component = ( + f"slack-app/{module_name.removeprefix('eval_').replace('_', '-')}" + if domain == "slack_app" + else f"posthog-ai/{domain.replace('_', '-')}" + ) + assert suite.tags == ("team:surfaces", f"component:{component}") + if domain == "experiments": + assert run.call_args.kwargs["send_to_posthog"] is True + assert set(suite.scorer_kinds.values()) == {"boolean"} + elif domain == "sql": + assert suite.scorer_kinds == { + "exit_code_zero": "boolean", + "no_persistent_insight_save": "boolean", + "execute_sql_called": "boolean", + "answer_tool_not_typed_query": "boolean", + "querying_posthog_data_skill_loaded": "boolean", + "sql_schema_alignment": "numeric", + "sql_result_message_alignment": "numeric", + } configured_names = {scorer._name() for scorer in run.call_args.kwargs["scorers"]} - assert set(suite.scorer_kinds) == {"exit_code_zero", *configured_names} + if suite_kind == SuiteKind.SANDBOXED: + configured_names.add("exit_code_zero") + assert set(suite.scorer_kinds) == set(suite.scorer_versions) == configured_names @pytest.mark.parametrize( diff --git a/products/posthog_ai/evals/ci/eval_basic.py b/products/posthog_ai/evals/ci/eval_basic.py index fad97bc84cb5..f7edf05de098 100644 --- a/products/posthog_ai/evals/ci/eval_basic.py +++ b/products/posthog_ai/evals/ci/eval_basic.py @@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.evals.cli_mcp.scorers import CalledTargetTool, SurfacedGeneratedAppUrl +from products.posthog_ai.evals.offline_config import offline_suite async def eval_bugfix(ctx: EvalContext) -> None: @@ -37,6 +38,10 @@ async def eval_bugfix(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-bugfix-cli", + offline_suite=offline_suite( + "ci/eval_basic::eval_bugfix", + metrics=[], + ), cases=cases, scorers=[], ctx=ctx, @@ -86,6 +91,10 @@ async def eval_app_link_generation(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-app-link-generation-cli", + offline_suite=offline_suite( + "ci/eval_basic::eval_app_link_generation", + metrics=["called_target_tool", "surfaced_generated_app_url"], + ), cases=cases, scorers=[CalledTargetTool(), SurfacedGeneratedAppUrl()], ctx=ctx, diff --git a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py index 492d8f3e323d..d58fb71145fc 100644 --- a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py +++ b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py @@ -53,6 +53,7 @@ ExecBeforeRender, RenderedEntityUi, ) +from products.posthog_ai.evals.offline_config import offline_suite # Renderable experiment tools (detail + results). The agent may render the detail # view (``experiment-get``) or a results view for a status/results question. Two @@ -96,6 +97,10 @@ async def eval_renders_entity_ui(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-renders-entity-ui-cli", + offline_suite=offline_suite( + "cli_mcp/eval_render_ui::eval_renders_entity_ui", + metrics=["rendered_entity_ui", "exec_before_render"], + ), cases=cases, scorers=[RenderedEntityUi(), ExecBeforeRender()], ctx=ctx, @@ -127,6 +132,10 @@ async def eval_no_render_for_query_results(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-no-render-for-query-cli", + offline_suite=offline_suite( + "cli_mcp/eval_render_ui::eval_no_render_for_query_results", + metrics=["called_target_tool", "did_not_render_ui", "did_not_cite_raw_sql_after_typed_query"], + ), cases=cases, scorers=[CalledTargetTool(), DidNotRenderUi(), DidNotCiteRawSqlAfterTypedQuery()], ctx=ctx, diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py index 116b50f68d69..7126dda1e16f 100644 --- a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py +++ b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py @@ -18,6 +18,7 @@ skill_distribution_expectations, ) from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues +from products.posthog_ai.evals.offline_config import offline_suite from products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN, CustomPromptSandboxContext @@ -173,6 +174,18 @@ async def eval_skill_distribution(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-skill-distribution-cli", + offline_suite=offline_suite( + "cli_mcp/eval_skill_distribution::eval_skill_distribution", + metrics=[ + "skill_search_first", + "expected_skill_discovered", + "expected_skill_loaded", + "excluded_skill_not_loaded", + "skill_loaded_before_tool", + "no_bundled_skill_bypass", + "no_exec_skill_bypass", + ], + ), cases=cases, scorers=[ SkillSearchFirst(), diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py index 8031426b0a00..db1a4eb61e0f 100644 --- a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py +++ b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py @@ -27,6 +27,7 @@ PROJECT_SKILL_REFERENCE_PATH, seed_project_skill, ) +from products.posthog_ai.evals.offline_config import offline_suite from products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN @@ -138,6 +139,20 @@ async def eval_skill_usage(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-skill-usage-cli", + offline_suite=offline_suite( + "cli_mcp/eval_skill_usage::eval_skill_usage", + metrics=[ + "skill_search_first", + "expected_skill_discovered", + "expected_skill_loaded", + "skill_loaded_before_tool", + "no_bundled_skill_bypass", + "no_exec_skill_bypass", + "expected_reference_pulled", + "search_recovery_after_zero_hit", + "skill_answer_correctness", + ], + ), cases=cases, scorers=[ SkillSearchFirst(), diff --git a/products/posthog_ai/evals/cli_mcp/eval_workflow.py b/products/posthog_ai/evals/cli_mcp/eval_workflow.py index 8f02a6ff7a34..688be90b6858 100644 --- a/products/posthog_ai/evals/cli_mcp/eval_workflow.py +++ b/products/posthog_ai/evals/cli_mcp/eval_workflow.py @@ -39,6 +39,7 @@ UsedJsonOutputFormat, VerifiedEventBeforeQuery, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_typo_recovery(ctx: EvalContext) -> None: @@ -69,6 +70,10 @@ async def eval_typo_recovery(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-typo-recovery-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_typo_recovery", + metrics=["recovered_to_correct_tool"], + ), cases=cases, scorers=[RecoveredToCorrectTool()], ctx=ctx, @@ -102,6 +107,10 @@ async def eval_schema_drilldown(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-schema-drilldown-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_schema_drilldown", + metrics=["called_target_tool", "drilled_into_schema"], + ), cases=cases, scorers=[CalledTargetTool(), DrilledIntoSchema()], ctx=ctx, @@ -124,6 +133,10 @@ async def eval_search_first(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-search-first-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_search_first", + metrics=["called_target_tool", "preferred_search_over_tools"], + ), cases=cases, scorers=[CalledTargetTool(), PreferredSearchOverTools()], ctx=ctx, @@ -154,6 +167,10 @@ async def eval_info_before_call(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-info-before-call-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_info_before_call", + metrics=["called_target_tool", "info_before_call"], + ), cases=cases, scorers=[CalledTargetTool(), InfoBeforeCall()], ctx=ctx, @@ -190,6 +207,10 @@ async def eval_json_for_post_processing(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-json-postprocess-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_json_for_post_processing", + metrics=["called_target_tool", "used_json_output_format", "ran_python_post_processing"], + ), cases=cases, scorers=[ CalledTargetTool(), @@ -216,6 +237,10 @@ async def eval_verify_event_before_query(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cli-mcp-verify-event-cli", + offline_suite=offline_suite( + "cli_mcp/eval_workflow::eval_verify_event_before_query", + metrics=["called_target_tool", "verified_event_before_query"], + ), cases=cases, scorers=[CalledTargetTool(), VerifiedEventBeforeQuery()], ctx=ctx, diff --git a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py index a525ff96d299..7b434dd3a7cf 100644 --- a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py +++ b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py @@ -27,6 +27,7 @@ CohortFromQueryUsed, QueryTargetsActorColumn, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_cohort_from_query(ctx: EvalContext) -> None: @@ -59,6 +60,10 @@ async def eval_cohort_from_query(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-cohorts-from-query-cli", + offline_suite=offline_suite( + "cohorts/eval_cohort_from_query::eval_cohort_from_query", + metrics=["cohort_from_query_used", "query_targets_actor_column", "no_uuid_batching"], + ), cases=cases, scorers=[ CohortFromQueryUsed(), diff --git a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py index 262a3f131759..136827813293 100644 --- a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py +++ b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py @@ -58,6 +58,7 @@ TYPE_NEEDLE_TABLE, VIEW_NEEDLE_NAME, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_dw_discovery(ctx: EvalContext) -> None: @@ -110,6 +111,10 @@ async def eval_dw_discovery(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-warehouse-discovery-cli", + offline_suite=offline_suite( + "data_warehouse/eval_information_schema::eval_dw_discovery", + metrics=["information_schema_queried", "agentic_search_used", "needle_table_identified"], + ), cases=cases, scorers=[ InformationSchemaQueried(), @@ -172,6 +177,16 @@ async def eval_dw_value_retrieval(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-warehouse-retrieval-cli", + offline_suite=offline_suite( + "data_warehouse/eval_information_schema::eval_dw_value_retrieval", + metrics=[ + "information_schema_queried", + "information_schema_before_answer", + "needle_value_retrieved", + "answer_query_ran", + "warehouse_answer_correctness", + ], + ), cases=cases, scorers=[ InformationSchemaQueried(), @@ -228,6 +243,15 @@ async def eval_dw_relationships(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-warehouse-relationships-cli", + offline_suite=offline_suite( + "data_warehouse/eval_information_schema::eval_dw_relationships", + metrics=[ + "information_schema_queried", + "relationship_discovery", + "join_path_traversed", + "warehouse_answer_correctness", + ], + ), cases=cases, scorers=[ InformationSchemaQueried(), @@ -271,6 +295,16 @@ async def eval_dw_table_relevancy(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-warehouse-relevancy-cli", + offline_suite=offline_suite( + "data_warehouse/eval_information_schema::eval_dw_table_relevancy", + metrics=[ + "information_schema_queried", + "agentic_search_used", + "needle_table_identified", + "stale_table_avoided", + "warehouse_answer_correctness", + ], + ), cases=cases, scorers=[ InformationSchemaQueried(), diff --git a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py index e5f7fc38c973..70965ed042db 100644 --- a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py +++ b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py @@ -33,6 +33,7 @@ IssueIdMatchesTarget, ) from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues +from products.posthog_ai.evals.offline_config import offline_suite def _events_case( @@ -107,6 +108,10 @@ async def eval_events_sampling(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-error-tracking-events-sampling-cli", + offline_suite=offline_suite( + "error_tracking/eval_events_sampling::eval_events_sampling", + metrics=["no_error_tracking_write", "events_tool_used", "events_args_alignment", "issue_id_matches_target"], + ), cases=cases, scorers=[ NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"), diff --git a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py index a9dceb789cbe..3e82533e9c17 100644 --- a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py +++ b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py @@ -40,6 +40,7 @@ IssueInputAlignment, ) from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues +from products.posthog_ai.evals.offline_config import offline_suite def _drilldown_case( @@ -121,6 +122,15 @@ async def eval_issue_drilldown(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-error-tracking-issue-drilldown-cli", + offline_suite=offline_suite( + "error_tracking/eval_issue_drilldown::eval_issue_drilldown", + metrics=[ + "no_error_tracking_write", + "issue_drilldown_order", + "issue_id_matches_target", + "issue_input_alignment", + ], + ), cases=cases, scorers=[ NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"), diff --git a/products/posthog_ai/evals/error_tracking/eval_issues_list.py b/products/posthog_ai/evals/error_tracking/eval_issues_list.py index b133a2e24c32..a016b2a02cd5 100644 --- a/products/posthog_ai/evals/error_tracking/eval_issues_list.py +++ b/products/posthog_ai/evals/error_tracking/eval_issues_list.py @@ -39,6 +39,7 @@ IssuesListToolUsed, ) from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues +from products.posthog_ai.evals.offline_config import offline_suite def _list_case( @@ -86,6 +87,10 @@ async def eval_issues_list(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-error-tracking-issues-list-cli", + offline_suite=offline_suite( + "error_tracking/eval_issues_list::eval_issues_list", + metrics=["no_error_tracking_write", "issues_list_tool_used", "issues_list_input_alignment"], + ), cases=cases, scorers=[ NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"), diff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py index 450164d638f1..9e7b72f954a1 100644 --- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py +++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py @@ -39,6 +39,7 @@ seed_running_experiment, seed_uneven_split_experiment, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_bias_uneven_split(ctx: EvalContext) -> None: @@ -107,6 +108,11 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-diagnose-bias-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_bias_uneven_split::eval_bias_uneven_split", + metrics=["diagnosis_group", "surfaces_all_findings"], + ), cases=cases, scorers=[ CitesDiagnosticGroup(), diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.py index 493d53c8e9e6..b7a6e5e93a0e 100644 --- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py +++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py @@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import RequiredToolCall +from products.posthog_ai.evals.offline_config import offline_suite async def eval_create_control_variant(ctx: EvalContext) -> None: @@ -56,6 +57,11 @@ async def eval_create_control_variant(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-create-control-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_create_control_variant::eval_create_control_variant", + metrics=["experiment_created_successfully"], + ), cases=cases, scorers=[ # The single outcome scorer: did experiment-create return without error? diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.py index a6374b4ebca3..9e962c140071 100644 --- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py +++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py @@ -52,6 +52,7 @@ seed_inactive_flag_experiment, seed_running_experiment, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_empty_experiment(ctx: EvalContext) -> None: @@ -139,6 +140,11 @@ async def eval_empty_experiment(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-diagnose-empty-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_empty_experiment::eval_empty_experiment", + metrics=["diagnosis_group"], + ), cases=cases, scorers=[ CitesDiagnosticGroup(), diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py index 98b786ab6e29..262879511506 100644 --- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py +++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py @@ -50,6 +50,7 @@ seed_day_old_experiment, seed_running_experiment, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_interpretation_traps(ctx: EvalContext) -> None: @@ -161,6 +162,11 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-diagnose-interpretation-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_interpretation_traps::eval_interpretation_traps", + metrics=["diagnosis_group", "advises_against_shipping"], + ), cases=cases, scorers=[ CitesDiagnosticGroup(), diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py index f1d0f956b849..e085458bcad5 100644 --- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py +++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py @@ -40,16 +40,10 @@ RecommendsShipVariant, ) from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment +from products.posthog_ai.evals.offline_config import offline_suite -# Cases are bundled into one suite function rather than one function per case -# because the harness wraps a list of cases in a single SandboxedPrivateEval -# call: one Braintrust experiment, with concurrency bounded by the harness's -# global sandbox semaphore and per-case filtering via --eval. Splitting into -# one function per case would create N separate Braintrust experiments and break -# cross-case comparison. Every existing sandboxed eval (eval_funnel, -# eval_retention, eval_insight_retrieval, eval_rollout_skill) follows this same -# single-function-multiple-cases shape. +# Keep the cases in one experiment so their scores can be compared together. async def eval_lifecycle_skills(ctx: EvalContext) -> None: cases: list[SandboxedEvalCase] = [ # Case 1: Ship variant must require confirmation. @@ -91,6 +85,17 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-lifecycle-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_lifecycle_skills::eval_lifecycle_skills", + metrics=[ + "resolved_experiment_by_name", + "no_unconfirmed_ship", + "asked_for_confirmation", + "duplicate_unique_flag_key", + "recommends_ship_variant", + ], + ), cases=cases, scorers=[ # Agent must look up the named experiment. diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py index 5a3d26b5003b..5b4cac746250 100644 --- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py +++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py @@ -25,6 +25,7 @@ validate_retention_metric, ) from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment +from products.posthog_ai.evals.offline_config import offline_suite async def eval_metric_schema_discovery(ctx: EvalContext) -> None: @@ -53,6 +54,11 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-metric-schema-discovery-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_metric_schema_discovery::eval_metric_schema_discovery", + metrics=["called_experiment_update", "first_update_metric_shape"], + ), cases=cases, scorers=[ RequiredToolCall(required={"experiment-update"}, name="called_experiment_update"), diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.py index 36baf69749ef..7db3ca178df9 100644 --- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py +++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py @@ -61,6 +61,7 @@ seed_paused_and_resumed_experiment, seed_running_experiment, ) +from products.posthog_ai.evals.offline_config import offline_suite async def eval_midrun_changes(ctx: EvalContext) -> None: @@ -152,6 +153,11 @@ async def eval_midrun_changes(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_midrun_changes::eval_midrun_changes", + metrics=["diagnosis_group", "does_not_recommend_edit", "advises_against_shipping"], + ), cases=cases, scorers=[ CitesDiagnosticGroup(), diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py index 852cfa4fdf1c..4604a3d058e9 100644 --- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py +++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py @@ -43,6 +43,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroup from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment +from products.posthog_ai.evals.offline_config import offline_suite async def eval_numbers_vs_sql(ctx: EvalContext) -> None: @@ -154,6 +155,11 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_numbers_vs_sql::eval_numbers_vs_sql", + metrics=["diagnosis_group"], + ), cases=cases, scorers=[ CitesDiagnosticGroup(), diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.py index fc646a7b4c51..59170914ec04 100644 --- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py +++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py @@ -8,8 +8,9 @@ Maps to stress-test finding #7 (skills rarely auto-load), specifically the variant-split rows (2.5, 5.3, 5.4, 5.5) in STEP_1_HARNESS.md. Uses -``SandboxedPrivateEval`` so it runs without a Braintrust API key — -local logs and PostHog ``$ai_trace`` events still emit. +``SandboxedPrivateEval`` with ``send_to_posthog=True`` to publish results +only to PostHog. The harness still requires a Braintrust API key for its +evaluation engine, but Braintrust result uploads remain disabled. To run: flox activate -- bash -c "set -a; source .env; set +a; python -m products.posthog_ai.eval_harness.harness eval_rollout_skill" @@ -21,6 +22,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.retrieval.scorers import SkillLoaded SKILL_NAME = "configuring-experiment-rollout" @@ -39,6 +41,11 @@ async def eval_rollout_skill(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-rollout-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_rollout_skill::eval_rollout_skill", + metrics=["skill_loaded"], + ), cases=cases, scorers=[ SkillLoaded(skill_name=SKILL_NAME), diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py index b1a8cc250afc..178b7f8bc1d8 100644 --- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py +++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py @@ -13,6 +13,7 @@ from products.posthog_ai.eval_harness.scorers import RequiredToolCall from products.posthog_ai.evals.experiments.scorers import SharedMetricValidationVerdict from products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_count +from products.posthog_ai.evals.offline_config import offline_suite async def eval_shared_metric_validation(ctx: EvalContext) -> None: @@ -50,6 +51,11 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-shared-metric-validation-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_shared_metric_validation::eval_shared_metric_validation", + metrics=["loaded_shared_metric", "shared_metric_validation_verdict"], + ), cases=cases, scorers=[ # Agent must actually read the seeded metric — either via list diff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py index d34379afeb26..9d58348b98a5 100644 --- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py +++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py @@ -40,6 +40,7 @@ from products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall from products.posthog_ai.evals.experiments.scorers import AskedForConfirmation from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment +from products.posthog_ai.evals.offline_config import offline_suite # Event name chosen to be obviously fictional so the agent can't plausibly # justify it as a typo of a real PostHog event in the seeded team. @@ -66,6 +67,11 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None: await SandboxedPrivateEval( experiment_name="sandboxed-experiments-unknown-events-cli", + send_to_posthog=True, + offline_suite=offline_suite( + "experiments/eval_unknown_events_confirmation::eval_unknown_events_confirmation", + metrics=["resolved_experiment_by_name", "no_silent_update_with_unknown_event", "asked_for_confirmation"], + ), cases=cases, scorers=[ # Agent must look up the named experiment first. diff --git a/products/posthog_ai/evals/offline_config.py b/products/posthog_ai/evals/offline_config.py new file mode 100644 index 000000000000..4479074fb0ec --- /dev/null +++ b/products/posthog_ai/evals/offline_config.py @@ -0,0 +1,225 @@ +from collections.abc import Sequence +from uuid import UUID + +from products.posthog_ai.eval_harness.harness.requirements import SuiteKind +from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer + +_SCORERS: dict[str, dict[str, OfflineScorer]] = { + "ci": { + "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557d-d3f2-7807-88ff-12d4c4b2679c")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-ca2a-75c8-b3a4-5b9a0a405a64")), + "surfaced_generated_app_url": OfflineScorer( + kind="boolean", version=UUID("01a1557d-dda3-782a-a2cb-dac316f6e7d6") + ), + }, + "cli_mcp": { + "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-090d-700e-8bc7-584c54a089fc")), + "did_not_cite_raw_sql_after_typed_query": OfflineScorer( + kind="boolean", version=UUID("01a1557e-1c0f-7302-96f5-ccdb2eed2b27") + ), + "did_not_render_ui": OfflineScorer(kind="boolean", version=UUID("01a1557e-1330-7ce0-a05a-914503a66cb0")), + "drilled_into_schema": OfflineScorer(kind="boolean", version=UUID("01a1557e-fd96-783d-8de2-dc4ddb5d6468")), + "excluded_skill_not_loaded": OfflineScorer( + kind="boolean", version=UUID("01a1557e-aca4-7826-8388-c66db0633772") + ), + "exec_before_render": OfflineScorer(kind="boolean", version=UUID("01a1557d-fb44-769a-b5cf-6d96a971182d")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-e737-7f9a-bb69-3dae3e58df7c")), + "expected_reference_pulled": OfflineScorer( + kind="boolean", version=UUID("01a1557e-d5f9-7e74-8369-2e4627c02a4a") + ), + "expected_skill_discovered": OfflineScorer( + kind="boolean", version=UUID("01a1557e-354f-726f-8e3e-c04ad776e8a7") + ), + "expected_skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1557e-a3f3-72c2-899f-3d5d9c903b39")), + "info_before_call": OfflineScorer(kind="boolean", version=UUID("01a1557f-1350-7eb4-8821-1b38bb2d8325")), + "no_bundled_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-c286-7545-a3ba-14a8e354cafe")), + "no_exec_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-cb02-7a20-a775-8dd932e4d07c")), + "preferred_search_over_tools": OfflineScorer( + kind="boolean", version=UUID("01a1557f-0679-7a2e-9cc6-a0559e513dcd") + ), + "ran_python_post_processing": OfflineScorer( + kind="boolean", version=UUID("01a1557f-85a2-7380-94ff-6d5de79a68a1") + ), + "recovered_to_correct_tool": OfflineScorer( + kind="boolean", version=UUID("01a1557e-f33b-7e99-9408-f3548316c76f") + ), + "rendered_entity_ui": OfflineScorer(kind="boolean", version=UUID("01a1557d-f13a-7eda-8447-e54d0ea0e7b1")), + "search_recovery_after_zero_hit": OfflineScorer( + kind="boolean", version=UUID("01a1557e-dedb-70fc-a4a4-c8be5f7c4939") + ), + "skill_answer_correctness": OfflineScorer(kind="boolean", version=UUID("01a1557e-e9b1-79b4-82c4-3639a73a9d66")), + "skill_loaded_before_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-b9d5-7cb3-9136-2a8176f7a8f3")), + "skill_search_first": OfflineScorer(kind="boolean", version=UUID("01a1557e-2b07-7fd6-95ed-e1f5cc459aff")), + "used_json_output_format": OfflineScorer(kind="boolean", version=UUID("01a1557f-7ce7-7ee9-8a62-0d219935a7cf")), + "verified_event_before_query": OfflineScorer( + kind="boolean", version=UUID("01a1557f-8e04-7c10-a084-3bcd95fd5fe8") + ), + }, + "cohorts": { + "cohort_from_query_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-a2a9-7cd5-a647-24fbc34fdaca")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-974a-79e8-a589-a41e4ff8bf2d")), + "no_uuid_batching": OfflineScorer(kind="boolean", version=UUID("01a1557f-bbad-7dcc-b140-d5c1fe69a9d2")), + "query_targets_actor_column": OfflineScorer( + kind="boolean", version=UUID("01a1557f-b048-7ebe-8db4-2d89aa79787e") + ), + }, + "data_warehouse": { + "agentic_search_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-db0e-76f5-ade3-847fb199d132")), + "answer_query_ran": OfflineScorer(kind="boolean", version=UUID("01a15580-3330-7033-a26b-bd7241292083")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-c598-76ff-8312-e8261f7357c2")), + "information_schema_before_answer": OfflineScorer( + kind="boolean", version=UUID("01a1557f-efac-7e78-b26d-3c45cab69449") + ), + "information_schema_queried": OfflineScorer( + kind="boolean", version=UUID("01a1557f-d0c7-7e57-9845-8c4b1f4d3004") + ), + "join_path_traversed": OfflineScorer(kind="boolean", version=UUID("01a15580-55a9-77a1-9815-f898ba72b96a")), + "needle_table_identified": OfflineScorer(kind="boolean", version=UUID("01a1557f-e4dc-75a1-9606-24647018c5c5")), + "needle_value_retrieved": OfflineScorer(kind="boolean", version=UUID("01a15580-29c9-7d6e-afa2-c650a1958662")), + "relationship_discovery": OfflineScorer(kind="boolean", version=UUID("01a15580-452b-7ccd-9a91-d87dace78770")), + "stale_table_avoided": OfflineScorer(kind="boolean", version=UUID("01a15580-5f0c-7e20-baca-eb544aba6950")), + "warehouse_answer_correctness": OfflineScorer( + kind="numeric", version=UUID("01a15580-3c17-77bc-af79-dfb04a075815") + ), + }, + "error_tracking": { + "events_args_alignment": OfflineScorer(kind="boolean", version=UUID("01a15580-83eb-7edb-88ed-e3a278c1bad6")), + "events_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15580-7acc-7e5b-bb9c-6e670b54e66d")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15580-67d1-7a2c-bdfe-d4de0553020c")), + "issue_drilldown_order": OfflineScorer(kind="boolean", version=UUID("01a15580-9762-7200-b518-2da4b1d854ce")), + "issue_id_matches_target": OfflineScorer(kind="boolean", version=UUID("01a15580-8e80-70df-b21a-15945ed85bdb")), + "issue_input_alignment": OfflineScorer(kind="boolean", version=UUID("01a15581-0aec-75b5-b0e7-e371e5853164")), + "issues_list_input_alignment": OfflineScorer( + kind="boolean", version=UUID("01a15581-1e2c-7ae1-9522-0f54a9c6e4a9") + ), + "issues_list_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15581-148b-74d7-93d7-8c3110ab2a95")), + "no_error_tracking_write": OfflineScorer(kind="boolean", version=UUID("01a15580-7158-7c13-856a-ff7669b5e0f2")), + }, + "experiments": { + "advises_against_shipping": OfflineScorer(kind="boolean", version=UUID("01a1558e-f848-7def-b3d8-970df097d0e5")), + "asked_for_confirmation": OfflineScorer(kind="boolean", version=UUID("01a1558f-1274-77da-9d79-2e410a63cbbc")), + "called_experiment_update": OfflineScorer(kind="boolean", version=UUID("01a1558f-1ac0-7176-8fbe-8730bbd9074a")), + "diagnosis_group": OfflineScorer(kind="boolean", version=UUID("01a1558f-2481-7ab2-95a5-a9f4e7e087eb")), + "does_not_recommend_edit": OfflineScorer(kind="boolean", version=UUID("01a1558f-314c-7d2f-bf87-43b94f4917ff")), + "duplicate_unique_flag_key": OfflineScorer( + kind="boolean", version=UUID("01a1558f-3b7c-7347-8fd2-a2eba94f5723") + ), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1558f-4447-7a87-b21c-4418a4152cc0")), + "experiment_created_successfully": OfflineScorer( + kind="boolean", version=UUID("01a1558f-55a3-7584-8fae-39efa8c6b11e") + ), + "first_update_metric_shape": OfflineScorer( + kind="boolean", version=UUID("01a1558f-5ee1-77eb-b837-843569f351b3") + ), + "loaded_shared_metric": OfflineScorer(kind="boolean", version=UUID("01a1558f-69dc-7efe-adfe-1debc1acbfa0")), + "no_silent_update_with_unknown_event": OfflineScorer( + kind="boolean", version=UUID("01a1558f-71f0-74a5-aec8-a03922367ecc") + ), + "no_unconfirmed_ship": OfflineScorer(kind="boolean", version=UUID("01a1558f-7d13-7a20-9035-2aa9cdddb4ac")), + "recommends_ship_variant": OfflineScorer(kind="boolean", version=UUID("01a1558f-8617-7a83-87a2-27f3adf7d54a")), + "resolved_experiment_by_name": OfflineScorer( + kind="boolean", version=UUID("01a1558f-8e45-7480-b669-ae4c5d655366") + ), + "shared_metric_validation_verdict": OfflineScorer( + kind="boolean", version=UUID("01a1558f-9a54-7743-905c-bb10f59c3264") + ), + "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1558f-a299-7e75-938e-07c2ff9c747f")), + "surfaces_all_findings": OfflineScorer(kind="boolean", version=UUID("01a1558f-ad52-7b81-8a93-d7e602bf0125")), + }, + "product_analytics": { + "answer_tool_not_execute_sql": OfflineScorer( + kind="boolean", version=UUID("01a15581-3926-7c14-a907-e994586bd216") + ), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-2721-7343-9ee8-d4cb29adbe1a")), + "first_relevant_tool": OfflineScorer(kind="boolean", version=UUID("01a15581-9357-7433-89c5-b66294e872d5")), + "first_relevant_tool_t1": OfflineScorer(kind="boolean", version=UUID("01a15581-bca9-7af2-a9e1-fd29a55c7291")), + "first_relevant_tool_t2": OfflineScorer(kind="boolean", version=UUID("01a15581-c7a2-7706-9d55-f2bd6640b817")), + "first_relevant_tool_t3": OfflineScorer(kind="boolean", version=UUID("01a15581-d0cf-7b8a-ae3f-558eb875af9c")), + "first_relevant_tool_t4": OfflineScorer(kind="boolean", version=UUID("01a15581-db3d-7eb7-8184-ef777d8900c1")), + "funnel_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-4c67-7321-bad6-608fbbe18564")), + "funnel_time_range_relevancy": OfflineScorer( + kind="boolean", version=UUID("01a15581-5652-79d8-a3b0-48be756495eb") + ), + "insight_shape": OfflineScorer(kind="boolean", version=UUID("01a15581-e3df-7304-b06e-8ae4ad58ccf5")), + "no_persistent_insight_save": OfflineScorer( + kind="boolean", version=UUID("01a15581-2fc3-7c68-b349-d44a670b232d") + ), + "providing_insights_skill_loaded": OfflineScorer( + kind="boolean", version=UUID("01a15581-9c84-7f36-837a-e20bbb99ac17") + ), + "retention_schema_alignment": OfflineScorer( + kind="numeric", version=UUID("01a15581-5faa-7855-b46a-90ff034dae82") + ), + "retention_time_range_relevancy": OfflineScorer( + kind="boolean", version=UUID("01a15581-6891-7f9b-bbb4-199b8b2edec0") + ), + "saved_insight": OfflineScorer(kind="boolean", version=UUID("01a15581-ee89-7754-b0dd-6012efeaa1c4")), + "schema_discovery_order": OfflineScorer(kind="boolean", version=UUID("01a15581-7429-7fff-85a7-9c223feddee6")), + "trends_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-a78c-7d03-bb2d-de43ab247f81")), + "trends_time_range_relevancy": OfflineScorer( + kind="boolean", version=UUID("01a15581-b135-7a70-ae09-b3d0e93bc894") + ), + "verified_event_exists": OfflineScorer(kind="boolean", version=UUID("01a15581-41af-7c8a-bf72-5e7c4ef63b0b")), + }, + "retrieval": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-f842-7336-8237-f5a8bf96d317")), + "info_before_execute_sql": OfflineScorer(kind="boolean", version=UUID("01a15582-4169-7666-8583-809f548f5ee7")), + "information_schema_before_sql": OfflineScorer( + kind="boolean", version=UUID("01a15582-3921-7c7f-9066-b2e9a767cd33") + ), + "lookup_id_in_output": OfflineScorer(kind="boolean", version=UUID("01a15582-2caf-725e-b7f0-ac9a02218b09")), + "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a15582-0134-7aaa-b171-52e5fb222bfa")), + }, + "sql": { + "answer_tool_not_typed_query": OfflineScorer( + kind="boolean", version=UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00") + ), + "execute_sql_called": OfflineScorer(kind="boolean", version=UUID("01a11725-20aa-78c3-af96-1aae34bca429")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d")), + "no_persistent_insight_save": OfflineScorer( + kind="boolean", version=UUID("01a11725-17f6-7673-93e9-4d525fa121ab") + ), + "querying_posthog_data_skill_loaded": OfflineScorer( + kind="boolean", version=UUID("01a11725-54c1-71e0-b606-132d5f4cb339") + ), + "sql_result_message_alignment": OfflineScorer( + kind="numeric", version=UUID("01a11725-6940-7f49-b979-11b3d6521393") + ), + "sql_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a11725-5da7-7dae-ab09-1c0beaa77224")), + }, + "surveys": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-4a50-75e1-a17a-3d483bb95586")), + "no_forbidden_survey_side_effects": OfflineScorer( + kind="boolean", version=UUID("01a15582-5371-7f81-8356-53532c73fac2") + ), + "survey_create_alignment": OfflineScorer(kind="numeric", version=UUID("01a15582-711a-7b7a-bd93-061a83ea909d")), + "survey_create_outcome": OfflineScorer(kind="boolean", version=UUID("01a15582-5d5a-786a-965f-ee2075db057d")), + "survey_create_returned_id": OfflineScorer( + kind="boolean", version=UUID("01a15582-6656-7fb2-b67d-6b918a8363e0") + ), + "survey_id_in_final_message": OfflineScorer( + kind="boolean", version=UUID("01a15582-7cea-7b35-9632-1fd1c3e22185") + ), + }, + "taxonomic_filter": { + "event_match_found": OfflineScorer(kind="boolean", version=UUID("01a15582-8f96-76a8-a6b0-9ad9c932f12a")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-85bb-71aa-b4c8-dc008015ab30")), + "no_wrong_event_match": OfflineScorer(kind="boolean", version=UUID("01a15582-9ce5-711e-b612-3b0be41c2669")), + "no_wrong_switch": OfflineScorer(kind="boolean", version=UUID("01a15582-b862-766e-b1ee-6664b70a9a93")), + "search_intent_match": OfflineScorer(kind="boolean", version=UUID("01a15582-a5dd-7f46-8681-5f0dd77b22d4")), + "switch_when_needed": OfflineScorer(kind="boolean", version=UUID("01a15582-aef0-748b-8282-c92f901762b5")), + }, +} + + +def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite: + domain = key.split("/", 1)[0] + scorers = _SCORERS[domain] + component = domain.replace("_", "-") + enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics + return OfflineEvalSuite.from_scorers( + key=key, + project_id=2, + scorers={metric: scorers[metric] for metric in enrolled_metrics}, + tags=("team:surfaces", f"component:posthog-ai/{component}"), + ) diff --git a/products/posthog_ai/evals/product_analytics/eval_funnel.py b/products/posthog_ai/evals/product_analytics/eval_funnel.py index 2bf6c71adc29..7c21408e64c8 100644 --- a/products/posthog_ai/evals/product_analytics/eval_funnel.py +++ b/products/posthog_ai/evals/product_analytics/eval_funnel.py @@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall, RequiredToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import ( INSIGHT_WRITE_TOOLS, FunnelSchemaAlignment, @@ -587,6 +588,16 @@ async def eval_funnel(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-funnels", + offline_suite=offline_suite( + "product_analytics/eval_funnel::eval_funnel", + metrics=[ + "no_persistent_insight_save", + "answer_tool_not_execute_sql", + "verified_event_exists", + "funnel_schema_alignment", + "funnel_time_range_relevancy", + ], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), diff --git a/products/posthog_ai/evals/product_analytics/eval_retention.py b/products/posthog_ai/evals/product_analytics/eval_retention.py index ee77fff5b70f..9cfb1d1f797a 100644 --- a/products/posthog_ai/evals/product_analytics/eval_retention.py +++ b/products/posthog_ai/evals/product_analytics/eval_retention.py @@ -8,6 +8,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import ( INSIGHT_WRITE_TOOLS, RetentionSchemaAlignment, @@ -116,6 +117,15 @@ async def eval_retention(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-retention-cli", + offline_suite=offline_suite( + "product_analytics/eval_retention::eval_retention", + metrics=[ + "no_persistent_insight_save", + "answer_tool_not_execute_sql", + "retention_schema_alignment", + "retention_time_range_relevancy", + ], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), diff --git a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py index 3c4f9b5bb735..c0d1c1522856 100644 --- a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py +++ b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py @@ -22,6 +22,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, SchemaDiscoveryOrder @@ -84,6 +85,10 @@ async def eval_schema_discovery(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-schema-discovery-cli", + offline_suite=offline_suite( + "product_analytics/eval_schema_discovery::eval_schema_discovery", + metrics=["no_persistent_insight_save", "answer_tool_not_execute_sql", "schema_discovery_order"], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), diff --git a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py index ecf2747c183f..c024a7fd52c5 100644 --- a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py +++ b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py @@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import NoToolCall from products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS ANALYSIS_QUERY_TOOLS = frozenset( @@ -119,6 +120,10 @@ async def eval_sql_shaped_typed_query_routing(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-sql-shaped-typed-query-routing-cli", + offline_suite=offline_suite( + "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_typed_query_routing", + metrics=["no_persistent_insight_save", "first_relevant_tool"], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), @@ -150,6 +155,10 @@ async def eval_sql_shaped_sql_controls(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-sql-shaped-sql-controls-cli", + offline_suite=offline_suite( + "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_sql_controls", + metrics=["no_persistent_insight_save", "first_relevant_tool"], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), diff --git a/products/posthog_ai/evals/product_analytics/eval_trends.py b/products/posthog_ai/evals/product_analytics/eval_trends.py index 89d9d73e2289..1fa92c1989f4 100644 --- a/products/posthog_ai/evals/product_analytics/eval_trends.py +++ b/products/posthog_ai/evals/product_analytics/eval_trends.py @@ -18,6 +18,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import ( INSIGHT_WRITE_TOOLS, TrendsSchemaAlignment, @@ -301,6 +302,16 @@ async def eval_trends(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-trends", + offline_suite=offline_suite( + "product_analytics/eval_trends::eval_trends", + metrics=[ + "no_persistent_insight_save", + "answer_tool_not_execute_sql", + "providing_insights_skill_loaded", + "trends_schema_alignment", + "trends_time_range_relevancy", + ], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py index d02003e698e6..d4c51ef0e614 100644 --- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py +++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py @@ -33,6 +33,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import NoToolCall from products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS from .eval_sql_shaped_routing import ANALYSIS_QUERY_TOOLS @@ -106,6 +107,16 @@ async def eval_typed_runner_no_lock_in_control(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-typed-runner-no-lock-in-control-cli", + offline_suite=offline_suite( + "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_no_lock_in_control", + metrics=[ + "no_persistent_insight_save", + "first_relevant_tool_t1", + "first_relevant_tool_t2", + "first_relevant_tool_t3", + "first_relevant_tool_t4", + ], + ), cases=cases, scorers=_lock_in_scorers(turn2_expected="query-trends"), ctx=ctx, @@ -140,6 +151,16 @@ async def eval_typed_runner_lock_in(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-typed-runner-lock-in-cli", + offline_suite=offline_suite( + "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_lock_in", + metrics=[ + "no_persistent_insight_save", + "first_relevant_tool_t1", + "first_relevant_tool_t2", + "first_relevant_tool_t3", + "first_relevant_tool_t4", + ], + ), cases=cases, scorers=_lock_in_scorers(turn2_expected="execute-sql"), ctx=ctx, diff --git a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py index b9b670559824..2073fc7b27bf 100644 --- a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py +++ b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py @@ -20,6 +20,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, InsightShape TIME_SERIES_DISPLAYS = ["ActionsLineGraph", "ActionsBar", "ActionsAreaGraph", "ActionsUnstackedBar"] @@ -104,6 +105,10 @@ async def eval_visualization_choice(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-visualization-choice-cli", + offline_suite=offline_suite( + "product_analytics/eval_visualization_choice::eval_visualization_choice", + metrics=["no_persistent_insight_save", "insight_shape"], + ), cases=cases, scorers=[ NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"), @@ -151,6 +156,10 @@ async def eval_saved_insight_display(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-saved-insight-display-cli", + offline_suite=offline_suite( + "product_analytics/eval_visualization_choice::eval_saved_insight_display", + metrics=["saved_insight", "insight_shape"], + ), cases=cases, scorers=[RequiredToolCall(["insight-create"], name="saved_insight"), InsightShape()], ctx=ctx, diff --git a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py index ef9a878e98c1..4c4ca5bfa3f3 100644 --- a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py +++ b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py @@ -23,6 +23,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.retrieval.scorers import LookupIdInOutput, SkillLoaded SKILL_NAME = "querying-posthog-data" @@ -72,6 +73,10 @@ async def eval_insight_retrieval(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-retrieval-cli", + offline_suite=offline_suite( + "retrieval/eval_insight_retrieval::eval_insight_retrieval", + metrics=["skill_loaded", "lookup_id_in_output"], + ), cases=cases, scorers=[ SkillLoaded(skill_name=SKILL_NAME), diff --git a/products/posthog_ai/evals/retrieval/eval_system_table_search.py b/products/posthog_ai/evals/retrieval/eval_system_table_search.py index fcb2dc80d646..adde6d5e19bd 100644 --- a/products/posthog_ai/evals/retrieval/eval_system_table_search.py +++ b/products/posthog_ai/evals/retrieval/eval_system_table_search.py @@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.retrieval.scorers import InfoCalledBeforeTool, InformationSchemaBeforeSql @@ -73,6 +74,10 @@ async def eval_system_table_search(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-system-table-search-cli", + offline_suite=offline_suite( + "retrieval/eval_system_table_search::eval_system_table_search", + metrics=["information_schema_before_sql", "info_before_execute_sql"], + ), cases=cases, scorers=[ InformationSchemaBeforeSql(), diff --git a/products/posthog_ai/evals/sql/eval_sql.py b/products/posthog_ai/evals/sql/eval_sql.py index 12d78f18dd8b..cea50c41769a 100644 --- a/products/posthog_ai/evals/sql/eval_sql.py +++ b/products/posthog_ai/evals/sql/eval_sql.py @@ -10,13 +10,11 @@ from __future__ import annotations -from uuid import UUID - from products.posthog_ai.eval_harness.base import SandboxedPublicEval from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext -from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS from products.posthog_ai.evals.retrieval.scorers import SkillLoaded from products.posthog_ai.evals.sql.scorers import AnswerQueryRan, SQLResultMessageAlignment, SQLSchemaAlignment @@ -375,27 +373,16 @@ async def eval_sql(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-sql-cli", - offline_suite=OfflineEvalSuite( - key="sql/eval_sql::eval_sql", - project_id=2, - scorer_kinds={ - "exit_code_zero": "boolean", - "no_persistent_insight_save": "boolean", - "execute_sql_called": "boolean", - "answer_tool_not_typed_query": "boolean", - "querying_posthog_data_skill_loaded": "boolean", - "sql_schema_alignment": "numeric", - "sql_result_message_alignment": "numeric", - }, - scorer_versions={ - "exit_code_zero": UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d"), - "no_persistent_insight_save": UUID("01a11725-17f6-7673-93e9-4d525fa121ab"), - "execute_sql_called": UUID("01a11725-20aa-78c3-af96-1aae34bca429"), - "answer_tool_not_typed_query": UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00"), - "querying_posthog_data_skill_loaded": UUID("01a11725-54c1-71e0-b606-132d5f4cb339"), - "sql_schema_alignment": UUID("01a11725-5da7-7dae-ab09-1c0beaa77224"), - "sql_result_message_alignment": UUID("01a11725-6940-7f49-b979-11b3d6521393"), - }, + offline_suite=offline_suite( + "sql/eval_sql::eval_sql", + metrics=[ + "no_persistent_insight_save", + "execute_sql_called", + "answer_tool_not_typed_query", + "querying_posthog_data_skill_loaded", + "sql_schema_alignment", + "sql_result_message_alignment", + ], ), cases=cases, scorers=[ diff --git a/products/posthog_ai/evals/surveys/eval_surveys.py b/products/posthog_ai/evals/surveys/eval_surveys.py index 187a36a4eb5f..7214ac6a247c 100644 --- a/products/posthog_ai/evals/surveys/eval_surveys.py +++ b/products/posthog_ai/evals/surveys/eval_surveys.py @@ -10,6 +10,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import NoToolCall from products.posthog_ai.eval_harness.seeders.survey import seed_survey_feature_flags +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS from products.posthog_ai.evals.surveys.scorers import ( SURVEY_FORBIDDEN_WRITE_TOOLS, @@ -270,6 +271,16 @@ async def eval_surveys(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-surveys-cli", + offline_suite=offline_suite( + "surveys/eval_surveys::eval_surveys", + metrics=[ + "no_forbidden_survey_side_effects", + "survey_create_outcome", + "survey_create_returned_id", + "survey_create_alignment", + "survey_id_in_final_message", + ], + ), cases=cases, scorers=[ NoToolCall( diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py index 493605b721a5..721285f7f3a4 100644 --- a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py +++ b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py @@ -36,6 +36,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.harness.requirements import SuiteKind from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.taxonomic_filter.scorers import EVENT_MATCH_KEY, EventMatchFound, NoWrongEventMatch SUITE_KIND = SuiteKind.ONE_SHOT @@ -103,6 +104,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="taxonomic-filter-event-match", + offline_suite=offline_suite( + "taxonomic_filter/eval_event_match::eval_event_match", + metrics=["event_match_found", "no_wrong_event_match"], + suite_kind=SUITE_KIND, + ), cases=[*DESCRIBED_CASES, *NO_MATCH_CASES], scorers=[EventMatchFound(), NoWrongEventMatch()], task=task, diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py index 9f78585527d6..cae15518e0c1 100644 --- a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py +++ b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py @@ -46,6 +46,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.harness.requirements import SuiteKind from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval +from products.posthog_ai.evals.offline_config import offline_suite from products.posthog_ai.evals.taxonomic_filter.scorers import ( SEARCH_INTENT_KEY, NoWrongSwitch, @@ -357,6 +358,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="ml-inference-search-intent", + offline_suite=offline_suite( + "taxonomic_filter/eval_search_intent::eval_search_intent", + metrics=["search_intent_match", "switch_when_needed", "no_wrong_switch"], + suite_kind=SUITE_KIND, + ), cases=[*WRONG_TAB_CASES, *RIGHT_TAB_CASES, *PARTIAL_CASES, *CONTEXT_CASES], scorers=[SearchIntentMatch(), SwitchWhenNeeded(), NoWrongSwitch()], task=task, diff --git a/products/signals/backend/test/test_eval_reporting.py b/products/signals/backend/test/test_eval_reporting.py index 8469247a7885..98631c355c22 100644 --- a/products/signals/backend/test/test_eval_reporting.py +++ b/products/signals/backend/test/test_eval_reporting.py @@ -127,12 +127,22 @@ def fail_network(request: httpx.Request) -> httpx.Response: assert asyncio.run(recorder.publish()) is None -def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pytest.CaptureFixture[str]) -> None: +@pytest.mark.parametrize("failed_action", ["upload", "tag"]) +def test_upload_failure_is_reported_without_hiding_completed_scores( + capsys: pytest.CaptureFixture[str], failed_action: str +) -> None: + server = OfflineServer() + + def respond(request: httpx.Request) -> httpx.Response: + if failed_action == "upload" or request.method == "PATCH": + return httpx.Response(403) + return server.respond(request) + with ( patch.dict(os.environ, {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token"}), patch( "httpx.AsyncClient", - new=partial(httpx.AsyncClient, transport=httpx.MockTransport(lambda request: httpx.Response(403))), + new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond)), ), ): recorder = SignalsEvalRecorder(SCOUT_SAFETY_SUITE) @@ -141,6 +151,14 @@ def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pyte item_name="case", metrics=[EvalMetric(name="correct_classification", score=0)], ) - assert asyncio.run(recorder.publish()) is None - assert "upload failed" in capsys.readouterr().err + url = asyncio.run(recorder.publish()) + output = capsys.readouterr().err + if failed_action == "upload": + assert url is None + assert "upload failed" in output + else: + assert server.status == "completed" + assert url is not None and recorder.experiment_id in url + assert f"uploaded: {url}" in output + assert "upload failed" not in output assert recorder.cases[0].scores["scout_safety_correct_classification"] == 0 diff --git a/products/signals/eval/offline.py b/products/signals/eval/offline.py index 43c9532f5e0d..f6054ab6e271 100644 --- a/products/signals/eval/offline.py +++ b/products/signals/eval/offline.py @@ -135,4 +135,5 @@ async def publish(self) -> str | None: self._report(f"upload failed: {reason}. Results were not saved for retry.") return None self._report(f"uploaded: {url}") + await self.publisher.tag(upload) return url diff --git a/products/signals/eval/offline_config.py b/products/signals/eval/offline_config.py index f829a71a0b00..19a1d5d390eb 100644 --- a/products/signals/eval/offline_config.py +++ b/products/signals/eval/offline_config.py @@ -4,6 +4,7 @@ GROUPING_SUITE = OfflineEvalSuite( key="signals/grouping-e2e", + tags=("team:self-driving", "component:signals/grouping"), project_id=2, scorer_kinds={ "signal_safety_correct_classification": "boolean", @@ -45,6 +46,7 @@ SCOUT_SAFETY_SUITE = OfflineEvalSuite( key="signals/scout-safety", + tags=("team:self-driving", "component:signals/scout-safety"), project_id=2, scorer_kinds={ "scout_safety_correct_classification": "boolean", diff --git a/products/slack_app/evals/eval_followup_classifier.py b/products/slack_app/evals/eval_followup_classifier.py index 7fdcdb78e84d..4526e841d3e4 100644 --- a/products/slack_app/evals/eval_followup_classifier.py +++ b/products/slack_app/evals/eval_followup_classifier.py @@ -26,6 +26,7 @@ from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval from products.slack_app.backend.services.slack_messages import SlackThreadMessage from products.slack_app.backend.temporal.activities import classifiers +from products.slack_app.evals.offline_config import offline_suite from products.slack_app.evals.scorers import FOLLOWUP_KEY, FollowupRoutingMatch, NoUnaskedWake SUITE_KIND = SuiteKind.ONE_SHOT @@ -239,6 +240,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="slack-app-followup-classifier", + offline_suite=offline_suite( + "slack_app/eval_followup_classifier::eval_followup_classifier", + metrics=["followup_routing", "no_unasked_wake"], + suite_kind=SUITE_KIND, + ), cases=[*DIRECTED_CASES, *CHATTER_CASES], scorers=[FollowupRoutingMatch(), NoUnaskedWake()], task=task, diff --git a/products/slack_app/evals/eval_model_classifier.py b/products/slack_app/evals/eval_model_classifier.py index 49b128f92ed3..d56f185160a6 100644 --- a/products/slack_app/evals/eval_model_classifier.py +++ b/products/slack_app/evals/eval_model_classifier.py @@ -30,6 +30,7 @@ from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval from products.slack_app.backend.services.model_catalogue import ModelChoice from products.slack_app.backend.temporal.activities import classifiers +from products.slack_app.evals.offline_config import offline_suite from products.slack_app.evals.scorers import MODEL_OVERRIDE_KEY, ModelOverrideMatch, NoUnaskedOverride SUITE_KIND = SuiteKind.ONE_SHOT @@ -275,6 +276,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="slack-app-model-classifier", + offline_suite=offline_suite( + "slack_app/eval_model_classifier::eval_model_classifier", + metrics=["model_override_match", "no_unasked_override"], + suite_kind=SUITE_KIND, + ), cases=[*INSTRUCTION_CASES, *UNDERSPECIFIED_CASES, *SUBJECT_MATTER_CASES], scorers=[ModelOverrideMatch(), NoUnaskedOverride()], task=task, diff --git a/products/slack_app/evals/eval_posthog_first.py b/products/slack_app/evals/eval_posthog_first.py index 87a7216df952..2d9ea513269c 100644 --- a/products/slack_app/evals/eval_posthog_first.py +++ b/products/slack_app/evals/eval_posthog_first.py @@ -28,6 +28,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase from products.posthog_ai.eval_harness.harness.context import EvalContext from products.posthog_ai.eval_harness.scorers import RequiredToolCall +from products.slack_app.evals.offline_config import offline_suite from products.slack_app.evals.scorers import ( DOCS_SEARCH_TOOL_NAME, AnsweredAboutPostHogProduct, @@ -62,6 +63,10 @@ async def eval_posthog_first(ctx: EvalContext) -> None: await SandboxedPublicEval( experiment_name="sandboxed-slack-posthog-first-cli", + offline_suite=offline_suite( + "slack_app/eval_posthog_first::eval_posthog_first", + metrics=["docs_searched", "posthog_product_answer", "no_third_party_recommendation"], + ), cases=cases, scorers=[ RequiredToolCall({DOCS_SEARCH_TOOL_NAME}, name="docs_searched"), diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.py index cfc917583ea0..c806af1dce15 100644 --- a/products/slack_app/evals/eval_project_classifier.py +++ b/products/slack_app/evals/eval_project_classifier.py @@ -47,6 +47,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKind from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval from products.slack_app.backend.temporal.activities import classifiers +from products.slack_app.evals.offline_config import offline_suite from products.slack_app.evals.scorers import PROJECT_ROUTE_KEY, NoUnaskedProjectSwitch, ProjectRouteMatch SUITE_KIND = SuiteKind.ONE_SHOT @@ -531,6 +532,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="slack-app-project-classifier", + offline_suite=offline_suite( + "slack_app/eval_project_classifier::eval_project_classifier", + metrics=["project_route_match", "no_unasked_project_switch"], + suite_kind=SUITE_KIND, + ), cases=[ *ROUTING_CASES, *CROWDED_ROUTING_CASES, diff --git a/products/slack_app/evals/eval_reply_suggestions.py b/products/slack_app/evals/eval_reply_suggestions.py index eb80eca083e9..4c0ee6837a62 100644 --- a/products/slack_app/evals/eval_reply_suggestions.py +++ b/products/slack_app/evals/eval_reply_suggestions.py @@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKind from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval from products.slack_app.backend.logic import reply_suggestions +from products.slack_app.evals.offline_config import offline_suite from products.slack_app.evals.scorers import REPLY_OFFER_KEY, NoUnaskedButtons, ReplyOfferMatch, SuggestedRepliesFit SUITE_KIND = SuiteKind.ONE_SHOT @@ -206,6 +207,11 @@ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict: await OneShotPublicEval( experiment_name="slack-app-reply-suggestions", + offline_suite=offline_suite( + "slack_app/eval_reply_suggestions::eval_reply_suggestions", + metrics=["reply_offer", "no_unasked_buttons", "suggested_replies_fit"], + suite_kind=SUITE_KIND, + ), cases=[*WAITING_CASES, *BARE_CASES], scorers=[ReplyOfferMatch(), NoUnaskedButtons(), SuggestedRepliesFit()], task=task, diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.py new file mode 100644 index 000000000000..b86cc92224c9 --- /dev/null +++ b/products/slack_app/evals/offline_config.py @@ -0,0 +1,52 @@ +from collections.abc import Sequence +from uuid import UUID + +from products.posthog_ai.eval_harness.harness.requirements import SuiteKind +from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer + +_SCORERS: dict[str, dict[str, OfflineScorer]] = { + "followup_classifier": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-9b63-776a-91f3-113ab57c025a")), + "followup_routing": OfflineScorer(kind="boolean", version=UUID("01a1557c-d1ca-7c68-96eb-5406fd8dde02")), + "no_unasked_wake": OfflineScorer(kind="boolean", version=UUID("01a1557c-dd52-78d0-a1f1-1854076c09ac")), + }, + "model_classifier": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-e620-710d-82d2-de63990000f1")), + "model_override_match": OfflineScorer(kind="boolean", version=UUID("01a1557c-ee78-7412-9b90-2ecf80591aa0")), + "no_unasked_override": OfflineScorer(kind="boolean", version=UUID("01a1557c-fd45-7a93-8852-b64bcf13eeb8")), + }, + "posthog_first": { + "docs_searched": OfflineScorer(kind="boolean", version=UUID("01a1557d-0f40-7999-9f55-d969442fe434")), + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-0682-75c3-bb88-5a620ba6ce5d")), + "no_third_party_recommendation": OfflineScorer( + kind="boolean", version=UUID("01a1557d-2542-7805-a791-6c3ac5f875b2") + ), + "posthog_product_answer": OfflineScorer(kind="boolean", version=UUID("01a1557d-1aa6-7fb4-ae3b-2fab4be4c44e")), + }, + "project_classifier": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-2ef4-7ee8-9f53-ae73633d593d")), + "no_unasked_project_switch": OfflineScorer( + kind="boolean", version=UUID("01a1557d-415e-78ff-8bcb-1e6cc67b77ac") + ), + "project_route_match": OfflineScorer(kind="boolean", version=UUID("01a1557d-37df-756c-a5fb-73edde0b1561")), + }, + "reply_suggestions": { + "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-49a7-7417-aa9f-4dbe00ec7b47")), + "no_unasked_buttons": OfflineScorer(kind="boolean", version=UUID("01a1557d-5b16-755e-ab88-b824e5103a97")), + "reply_offer": OfflineScorer(kind="boolean", version=UUID("01a1557d-5266-7853-a431-6213cfba1b5a")), + "suggested_replies_fit": OfflineScorer(kind="boolean", version=UUID("01a1557d-bcc0-7a96-8d22-4d6e91998d5d")), + }, +} + + +def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite: + domain = key.split("/", 1)[1].split("::", 1)[0].removeprefix("eval_") + scorers = _SCORERS[domain] + component = domain.replace("_", "-") + enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics + return OfflineEvalSuite.from_scorers( + key=key, + project_id=2, + scorers={metric: scorers[metric] for metric in enrolled_metrics}, + tags=("team:surfaces", f"component:slack-app/{component}"), + ) diff --git a/pyproject.toml b/pyproject.toml index 80340aa70da3..65749b80b297 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -105,7 +105,7 @@ dependencies = [ "paramiko~=3.5.0", "pillow==12.3.0", "protobuf~=6.31.1", - "posthoganalytics==7.67.1", + "posthoganalytics==7.73.0", "polars==1.37.1", "psycopg2-binary==2.9.11", "psycopg[binary]==3.2.10", diff --git a/uv.lock b/uv.lock index a1acc6bad859..390c4454d298 100644 --- a/uv.lock +++ b/uv.lock @@ -6252,7 +6252,7 @@ requires-dist = [ { name = "playwright", specifier = "~=1.60.0" }, { name = "polars", specifier = "==1.37.1" }, { name = "posthog-hogland", specifier = "==0.4.1" }, - { name = "posthoganalytics", specifier = "==7.67.1" }, + { name = "posthoganalytics", specifier = "==7.73.0" }, { name = "protobuf", specifier = "~=6.31.1" }, { name = "psycopg", extras = ["binary"], specifier = "==3.2.10" }, { name = "psycopg2-binary", specifier = "==2.9.11" }, @@ -6419,7 +6419,7 @@ wheels = [ [[package]] name = "posthoganalytics" -version = "7.67.1" +version = "7.73.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "backoff" }, @@ -6427,9 +6427,9 @@ dependencies = [ { name = "requests" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/76/ff/1b18c85d904340607c70284c9d867df588c22c18d615e2ef67741126f64c/posthoganalytics-7.67.1.tar.gz", hash = "sha256:c861483738eb6da05eeaac3540f2d9f6622849e7a8bd3d05a4a0221a32abc120", size = 752005, upload-time = "2026-10-12T14:17:21.915Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f4/8e/30cf3b4ce18f2224799e7fe38d147e4d8b6d3a54ef9998234100253180e6/posthoganalytics-7.73.0.tar.gz", hash = "sha256:6ea816666f116f132a4ae4e5fb3e36240e4f7de7c2a8afed12d2a258c227912a", size = 764745, upload-time = "2026-10-19T19:21:34.253Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/3f/d4/48706a5d0c2304f299a4564b86ce220eae3de6ab087a2bca9669644b8e3a/posthoganalytics-7.67.1-py3-none-any.whl", hash = "sha256:29a721c2e9ca383ed35deb736b390744e68ad9d2bf69b4bf08ebbc4487db97a4", size = 884481, upload-time = "2026-10-12T14:17:19.659Z" }, + { url = "https://files.pythonhosted.org/packages/66/01/d336d90c2b615b3db08864f004841d60bee2ebd4930cd1c2d761b30bc32a/posthoganalytics-7.73.0-py3-none-any.whl", hash = "sha256:a1ab79a6abf9d8d655d9e7e37fe8fe3a10781415037cf199c516a6013cc9a905", size = 899662, upload-time = "2026-10-19T19:21:32.037Z" }, ] [[package]]