PostHog analytics for the CLI and install scripts

[NejcS]

Purpose

Capture two product-analytics events from the codeplain CLI — one when a user finishes installing, one when a render ends — in the same PostHog project the web app already uses, so a user's signup, payment, install and renders appear on one timeline.

Changes and design decisions

Identity is the email address. The web app already calls posthog.identify(user.email), so CLI events keyed on email land on the same PostHog person as signup, plan selection and payment. No new endpoint was needed: POST /status and /connection_check already return the email. One PostHog project, shared with the web app — that is what makes the linking automatic.

Two events.

• cli_render_finished — sent from a single call site in main()'s finally, placed after print_exit_summary so a slow network can never delay the summary the user is waiting on. Properties: render_id, outcome (succeeded / cancelled / crashed / failed), rendered_functionalities, render_time_seconds, module_count, client_version, os, headless, the three *_script_provided flags, and error_type.
• cli_installed — sent at the end of both install scripts, and also on the failed-verification path with install_verified: false. Both installers already call /status to verify the API key, so they read the email out of that response rather than making an extra call.

The PostHog token is not committed. analytics.py and both install scripts carry __POSTHOG_PROJECT_TOKEN__ / __POSTHOG_CAPTURE_URL__ placeholders. The two publish workflows substitute them from a POSTHOG_PROJECT_TOKEN secret and a POSTHOG_CAPTURE_URL variable — publish-to-pypi before uv build, publish-install-script before the R2 upload. Both fail the publish if either value is missing or malformed, rather than shipping an artifact that silently reports nothing.

Whether substitution happened is decided by a positive check — token starts with phc_, URL starts with https:// — not by sniffing for the placeholder, so a sed change cannot quietly disable the check. CODEPLAIN_POSTHOG_PROJECT_TOKEN and CODEPLAIN_POSTHOG_CAPTURE_URL let a source checkout emit events without cutting a release; a clone with neither set reports nothing.

Substitution writes through a temp file and copies back with cat rather than using sed -i, which is GNU-only and would have been unverifiable outside the runner. This also preserves install.sh's 755 mode.

Consent is shared with crash reporting. CODEPLAIN_TELEMETRY in {0, false, off} disables analytics and Sentry together. One deliberate asymmetry: the CLI keeps its "off outside a production install" default, while the installers default to on — running an installer is a real user install.

Privacy. No spec content, file paths or error messages are sent. A failure is described by its exception class name alone. error_type is dropped when the outcome is cancelled, since a cancel also unwinds through an exception and the type would be noise.

Known gap, deliberate: installs that never verify an API key, and renders that never resolve the email, send nothing. The CLI mints no anonymous ids anywhere, so those users are invisible until they configure a key. This was a conscious choice, not an oversight — flagging it because it is the one place the funnel has a hole.

Alternatives considered and rejected. Proxying events through the Codeplain API scored best on every axis — no token client-side at all, identity guaranteed server-side, a kill-switch over already-installed CLIs — but needs a new endpoint and a coordinated deploy, so it was deferred. Serving the token from /status would keep it out of the artifacts but still hand it to the client. Keeping it hardcoded was rejected outright.

Breaking changes

None for users.

One action required before the next release: set the POSTHOG_PROJECT_TOKEN secret and the POSTHOG_CAPTURE_URL variable in the repository settings. Until they exist, publish-to-pypi and publish-install-script will fail — by design, so a release cannot silently ship dead analytics.

Testing

• 24 new unit tests for the analytics module and 3 for the CLI wiring, covering outcome classification, the opt-out, the missing-email path, the unconfigured-checkout path, six half-substituted config permutations, the short timeout, and that a capture failure never raises.
• Publish-time injection verified by extracting the real steps out of the YAML and running them against copies of the three files. Substituted copies resolve the config and send; source copies send nothing; the source scripts do send once the env override is supplied. No placeholder survives, and install.sh keeps mode 755.
• All four CI guard paths tested — missing secret, wrong token prefix, missing variable, non-https URL — each exits 1 with a specific ::error::.
• Email extraction tested against real /status body shapes in both shells, including a 401 body, an unreachable API and a non-JSON body. organization_owner_email never matches. Both installers survive set -u with the optional-step flags unset.
• install.ps1 parses cleanly under pwsh and stays pure ASCII.
• The capture endpoint was probed with a bogus token to confirm the path resolves; nothing was written.
• Full suite: 415 passed. black, isort, flake8 clean; mypy clean on the changed files.

Not verified: no live event has been sent with the real token, since that would write into production PostHog. Smoke test after merge and release, or from source with CODEPLAIN_POSTHOG_* set. Note PostHog answers HTTP 200 even for an invalid token, so the PostHog UI is the only real confirmation.