Skip to content

feat(analytics): ask about the email provider after install - #2

Draft
Silthus wants to merge 7 commits into
feat/detected-email-providersfrom
feat/email-provider-offer
Draft

Silthus wants to merge 7 commits into
feat/detected-email-providersfrom
feat/email-provider-offer

Conversation

@Silthus

@Silthus Silthus commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Stacked on PostHog#1493 (base feat/detected-email-providers). Fork-only slice 18a of the Wizard email work; it mirrors no upstream PR yet.

Problem

The Wizard's scan finds the email service a developer's app sends with, such as Resend. Today nothing asks them about it, so we can't tell how many of them would send email from PostHog instead, or with what intent.

With this PR, an interactive install that found an email provider can ask one question after the outro: "Your app sends email with Resend. PostHog can send it too, triggered by the events you already track. Want to try it?" The answer goes to analytics and onto the person. The question only shows while the flag wizard-email-provider-offer is on for the run, and the flag starts at 0%.

Spec slice 18 reuses this prompt for its migrate resend offer: the "Yes" screen already has a typed slot for the migrate command, which stays empty until slice 16's migrate family exists.

Impact

Pays into Grow Workflows through Started orgs (orgs that create their first workflow). Measured by the funnel Wizard email provider offer in the internal project: offer shown, answered (by answer), then a Workflows page view or an email sender created within 7 days, broken down by providers and answer. It counts production builds only, and only runs where wizard-email-provider-offer is on. Expected move: small, up. The first job of this PR is the intent read; the "Yes" link is the conversion path. Run cost: n/a.

Merge Danger

Note

🔵 Merge risk: low · one-way door (merge-risk rubric)

  • Released Wizards write wizard_email_offer_answer and wizard_email_offer_answered_at onto real person profiles once the flag is on. A revert stops new writes; values already set stay until a $unset.
  • A new flow step sits after the outro of every interactive run. It shows only when the flag is on, the run succeeded, sharing is on, and the scan found an email provider. The flag starts at 0%, so merging changes nothing a user sees.
  • Ctrl-C and process signals now unmount the TUI first, then wait for the analytics flush before they exit. That path is shared by every interactive run.

Changes

  • Flag (WIZARD_EMAIL_PROVIDER_OFFER_FLAG_KEY in src/shared/constants.ts): wizard-email-provider-offer joins the Wizard's closed flag set, WIZARD_FLAG_KEYS. The run already evaluates that set once after login and caches it. The offer reads the cached snapshot, so it adds no flag request.
    • Only the value 'true' asks. A flag that is off, missing from the snapshot, or never evaluated means no question.
    • A failed flag request already leaves an empty snapshot, so it means no question too. The run never waits on the flag or fails because of it.
  • When it asks (emailProviderOffer in src/programs/posthog-integration/email-provider-offer.ts): the flag is on, the run is interactive and not --ci, scan results may be shared (sharing granted and telemetry on), the outro reports success, and the scan found at least one email provider. Otherwise it returns nothing and the step never shows. Headless runs never mount the TUI, so they never reach it.
  • The question (EmailProviderOfferScreen, a new email-provider-offer step between the outro and the MCP step):
    • It names up to two providers: "with Resend", "with Resend and SendGrid", or "with Resend, SendGrid and others".
    • "Yes, show me how": shows "Set up email in Workflows: <app host>/project/<id>/workflows?source=wizard_cli" and moves on at the next key. The link stays on screen only.
    • "Not now" moves on at once. So does choice 3: "No, I'm happy with Resend" for one provider, or "No, I'm happy with my current setup" for several.
    • Esc, or Ctrl-C while the question is up, counts as skipped.
    • It asks once per run; a second answer or an unmount after an answer sends nothing.
  • Events, through the existing wizardCapture helper:
    • wizard: email provider offer shown with providers: every detected kind, such as ["Resend", "SendGrid"], even when the prompt names only two.
    • wizard: email provider offer answered with providers, answer (yes, not_now, no, skipped), and $set of wizard_email_offer_answer and wizard_email_offer_answered_at. The run is logged in by then, so the $set lands on the identified person.
  • Ctrl-C and signals deliver the answer (src/tui/run.ts): after a successful run the run report has already gone out, so the interrupt's analytics.shutdown('cancelled') returned at once and the process could exit before the skipped event left. The signal handler now unmounts the TUI first, which captures skipped for an open question, then awaits the shutdown and a flush before it exits. The screen-exit path already used that order.
  • Provider list: detectedEmailProviders(scan) in email-providers.ts now backs both feat(analytics): set detected email providers on the person PostHog/wizard#1493's person property and this offer, so the two can't drift. The kind list became a typed tuple, so EmailProviderKind keys the migrate command map.

Test plan

Lane gate on 324302a: pnpm typecheck green, pnpm lint 0 errors (564 existing warnings), pnpm vitest run 258 files and 3838 tests green. CI: the fork runs no workflows, so this PR has no checks; Michael accepted that gap, and the lane gate is the check.

Test seams:

  • emailProviderOffer(session, flags) over a real scan session (email-provider-offer.test.ts):
    • The flag on gives the offer with the Workflows link and source=wizard_cli.
    • The flag off, missing from the snapshot, or with no snapshot gives no offer.
    • No offer for --ci, telemetry off, sharing declined, a failed install, no login, and a scan with no email provider.
    • The copy for one, two and three providers, with providers keeping all of them.
  • Analytics.getAllFlagsForWizard (analytics.test.ts): a rejected flag request caches a snapshot with no flags, which the offer reads as off.
  • EmailProviderOfferScreen rendered in the real store with @utils/analytics mocked (EmailProviderOfferScreen.test.tsx): the exact prompt copy and the shown event; not_now, no and Esc; Yes shows the link and the next key moves on; one answer only; an unmount without an answer sends skipped.
  • runTui cancellation (run.test.ts):
    • An Ink Ctrl-C resolves 130 only after the analytics flush finishes.
    • A SIGINT, SIGTERM or SIGHUP unmounts the TUI before that final flush.
    • After a successful run, an Ink Ctrl-C flushes the answer the unmount captures before it resolves 130. This case fails on the earlier flush-then-unmount order.
  • Flow, frame, store and harness registries pick up the new step and flag key (flows.test.ts, frames.test.tsx snapshots at 80x28 and 120x40, store-invariants.test.ts, flags.test.ts, the e2e action registry). The analytics mocks in four flow tests and the harness test gained the flag read.

Proofs per qa-swarm PROOF.md, saved on 324302a. That commit adds only the test above, so the captures from a71b308 exercise the same source.

  • Live run (saved): the real TUI in a PTY on a Next.js fixture, after the real scan and detection, a real login through Analytics, and the real flag evaluation against the internal project.

    • Flag off: the flag evaluated to false at 0%. The outro went straight to exit, with no offer screen and no offer events.

      Flag off: after the outro the run exits

    • Flag on, two providers: the Wizard's dev-only flag override turned the flag on, on a fixture with resend and @sendgrid/mail. The prompt read "Your app sends email with Resend and SendGrid." and choice 3 read "No, I'm happy with my current setup". "Yes" showed the Workflows link, and shown and answered carried ["Resend", "SendGrid"] with answer: yes.

      Flag on, two providers: the question

      After Yes: the Workflows link

    • Each answer, flag on, one provider. Each sent shown then answered to the internal project, and the person got wizard_email_offer_answer:

      keys after the outro screen answer sent
      Enter on "Yes, show me how" the Workflows link, then exit on the next key yes
      Down, Enter exit not_now
      Down, Down, Enter exit no
      Esc exit skipped
      Ctrl-C exit 130 skipped

      One provider, choice 3 selected

    • Ctrl-C after a successful run, on 324302a: the harness sent the real success shutdown before the question, as a successful run does. Ctrl-C then ran the host's signal order: unmount, the cancelled shutdown (a no-op by then), the flush, and exit 130 with nothing after it. The internal project received setup wizard finished (success), then shown, then answered with skipped, and the person got wizard_email_offer_answer. So the flush alone delivers an answer captured after the run already reported its end.

      After a successful run: the question before Ctrl-C

  • CI case (saved): flag on and session.ci set. The outro goes straight to exit, with no offer screen and no offer events.

  • Prod shape (gap, accepted by standing rule until 2026-11-02): how many real runs find an email provider stays unknown until read-only production access is set up.

  • UI walk (saved): PNG captures of the real TUI at 100x30, shown above: flag off, two providers, the link, one provider, the CI run, and the question after a successful run.

  • Post-deploy watch: owed after merge.

  • Gap: no LLM agent install ran, since this box has no gateway key. The harness arranges the successful run and outro; the scan, detection, login, flag evaluation, TUI and analytics client are the real ones. For the same reason, the live Ctrl-C runs the host's steps in its order but not the runTui function itself: the repo's real-host route needs --ci, which hides the question. The run.test.ts cases drive runTui itself.

Decisions

  • 👤 Ask about the email provider in the Wizard, ahead of migrate? Yes: one question after the install outro, measured, never in CI or headless runs or with sharing off.
  • 👤 What does the question say? The prompt and three choices as Michael wrote them; Esc and Ctrl-C count as skipped.
  • 👤 What gates the question? A dedicated flag, wizard-email-provider-offer, at 0%. Not the Workflows launch flag: that one stays on the PostHog org until launch and would hide the question from the developers it measures.
  • 👤 What happens when the flag can't be read? No question, and the run carries on.
  • 👤 With several providers, what does the prompt name? Up to two: "Resend and SendGrid", or "Resend, SendGrid and others". Choice 3 then reads "No, I'm happy with my current setup". It replaced the agent's first-provider-only copy.
  • 👤 Does the empty migrate command slot stay? Yes, as the typed seam slice 16 fills; the review's call to remove it was rejected.
  • 👍 Where does the "Yes" link go? On screen only, before the MCP step, not in the exit summary.
  • 🤖 How does the Wizard read the flag? From the run's cached flag snapshot, by adding the key to the closed flag set, so no new request.
  • 🤖 What counts as reaching Workflows in the funnel? A page view under /project/<id>/workflows or an email integration created, within 7 days; production builds only.

LLM context

Built by Claude Code (Opus 5.5) as lane wiz-offer of the Wizard email map. Michael answered the open questions on 2026-10-26.

Involvement: no grilling rounds, no prototypes, one cross-family review (GPT-6.1 Sol, round 1: one high finding fixed, one low finding rejected), final reads of each later change, and two fixes from the proof judge (unmount before the final flush on a signal, and a test and live run for Ctrl-C after a successful run). Live-run and UI-walk proof captured.

🤖 Generated with Claude Code

Silthus and others added 7 commits October 26, 2026 10:31
An interactive install run whose scan found an email provider now asks
one question after the outro: try sending email with PostHog Workflows?
The answer goes out as `wizard: email provider offer answered` and is
set on the person. Yes shows a link to Workflows in the project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant