Repository navigation
feat(email): send transactional emails through workflows behind a flag - #122019
Conversation
The transactional-emails-via-workflows flag picks the recipients and templates that send through the Workflows transactional email API. Everything else, and any send Workflows rejects, still goes through Customer.io. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 20d9dd00-fcbd-4c83-a5c3-68a47fd9be9a
|
😎 Merged successfully - details. |
🤖 CI report
|
| File | Comment lines | Added lines |
|---|---|---|
posthog/email.py |
28 | 303 |
posthog/test/test_email.py |
6 | 271 |
This check does not block merging. It updates on every push and clears when the share drops.
…ls per template A failed send only reached error tracking, and the exception carries no template name, so failures could not be counted per template. - `transactional email failed` fires when a send raises, with `template_name`, `error_type` and `error`. - `transactional email workflows rejected` fires when Workflows refuses a send and Customer.io sends it instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
…only password_reset - `_queue_workflows_email` takes the same arguments as posthog-python's `Emails.send` and raises an error with the same `code` and `status_code`, so the SDK can replace one function body once it ships. - Sends go to `/api/projects/@current/...`, because the project secret API key names the project. `WORKFLOWS_TRANSACTIONAL_EMAIL_PROJECT_ID` is gone. - The fallback rule reads `code` and `status_code`: a refused connection or a send that queued nothing falls back to Customer.io, a timeout or an in-flight 409 does not. - The request timeout drops to 10 seconds, because the caller holds the MessagingRecord row lock while it waits. - `WORKFLOWS_TEMPLATE_ID_MAP` keeps only `password_reset`, for a single production test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
- Workflows sends use `POSTHOG_TRANSACTIONAL_EMAIL_API_KEY` and `POSTHOG_HOST`, the settings `posthog/transactional_email.py` already reads, so no second key needs provisioning. `WORKFLOWS_TRANSACTIONAL_EMAIL_API_KEY` and `_API_URL` are gone. - HTTP email is available only when Customer.io is configured again. The shared key alone no longer moves an instance off SMTP, so an environment that set it for another feature keeps sending email. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
`WORKFLOWS_TRANSACTIONAL_EMAIL_TEMPLATE_IDS` only let a local stack point at its own copies of the templates. Production and dev both send through the US project's templates, so `WORKFLOWS_TEMPLATE_ID_MAP` is the only source of template IDs. The feature flag decides whether a template sends through Workflows; the map only says which template to send. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
|
Risk: No findings This delta is a single follow-up fix: a pinned Workflows retry now passes through the API-key and HTTPS-host guards instead of bypassing them, so a Workflows-attempted email is never resent with the secret key over an insecure or unconfigured path. The prior finding is resolved; no new security issues were introduced. Sentinel reviewed |
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
| # timeout, Workflows can have queued the email. | ||
| return err.code in ("connection_error", "not_queued") | ||
| # A 409 means an earlier attempt with the same idempotency key is still in flight and can still send. | ||
| return err.status_code != 409 |
There was a problem hiding this comment.
P1 · Exclude uncertain server failures from fallback
When CDP queues an email but its response or subsequent metrics flush fails, the transactional API returns 503. _workflows_error_allows_fallback treats that as a confirmed rejection and sends through Customer.io too, producing duplicate emails despite the idempotency header. Restrict fallback to responses that guarantee nothing was queued.
There was a problem hiding this comment.
Fixed in df9900e. A 5xx (and 408/422) no longer falls back; the error reaches Celery, which retries with the same idempotency key.
| except requests.ConnectionError as err: | ||
| raise WorkflowsEmailError(str(err), code="connection_error") from err |
There was a problem hiding this comment.
P1 · Distinguish refused connections from post-send resets
When Workflows accepts the request and the connection resets before returning headers, requests raises ConnectionError too. _queue_workflows_email labels every such exception fallback-safe, so _send_one sends a duplicate through Customer.io. Only classify confirmed connection-establishment failures as safe; preserve uncertain outcomes for other connection errors.
There was a problem hiding this comment.
Fixed in df9900e. Only a ConnectionError with no connection made (NewConnectionError or ConnectTimeout) falls back. A ProtocolError after the request went out is treated as uncertain and retried with the same key.
| posthoganalytics.capture( | ||
| distinct_id=dest.get("distinct_id") or dest["raw_email"], | ||
| event=event, | ||
| properties={"recipient_email": dest["raw_email"], **properties}, | ||
| ) |
There was a problem hiding this comment.
P2 · Flush the new failure events from Celery workers
When _send_email runs in a Celery worker that exits before the global client's background flush, the new failure and rejection events disappear, undermining per-template rollout monitoring. _capture_email_event uses that global client despite the Celery requirement in AGENTS.md; use ph_scoped_capture to flush these events.
There was a problem hiding this comment.
Fixed in df9900e. All send events in _send_via_http go through ph_scoped_capture, which flushes when the batch finishes.
| except WorkflowsEmailError as err: | ||
| if not _workflows_error_allows_fallback(err): | ||
| raise |
There was a problem hiding this comment.
P2 · Propagate transient Workflows failures to Celery retries
When a Workflows read timeout occurs before enqueue, this branch raises into _send_via_http, which catches the exception and returns normally. Celery marks the task successful instead of applying EMAIL_TASK_KWARGS retries, permanently dropping the password-reset email and any remaining batch recipients. Propagate transient failures and retry with the same idempotency key.
There was a problem hiding this comment.
Fixed in df9900e. An uncertain WorkflowsEmailError is re-raised after the failure event, so EMAIL_TASK_KWARGS retries it with the same idempotency key. Recipients with sent_at are skipped on retry.
| response = requests.post( | ||
| # The project secret API key names the project, so `@current` resolves to it. | ||
| f"{settings.POSTHOG_HOST.rstrip('/')}/api/projects/@current/messaging_templates/{template_id}/send/", |
There was a problem hiding this comment.
P2 · Preserve HTTPS validation and disable redirects for sensitive sends
When POSTHOG_HOST uses HTTP or returns a 307/308 redirect, _queue_workflows_email sends password-reset data over plaintext or forwards it to the redirect target. Unlike posthog/transactional_email.py, this client neither requires an HTTPS origin nor disables redirects. Preserve those transport safeguards before sending secrets.
There was a problem hiding this comment.
Fixed in df9900e. Workflows sends use allow_redirects=False, and a non-HTTPS POSTHOG_HOST outside DEBUG keeps the email on Customer.io.
![parameterai[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
- A 5xx no longer falls back to Customer.io. The send API also returns 503 when the workflow service queued the email but failed to answer. - A connection error falls back only when no connection was made. A reset after the request went out can follow a queued email. - 408 and 422 no longer fall back either. Redirects do, because they come before anything is queued. - An uncertain Workflows outcome now reaches Celery, which retries with the same idempotency key. Workflows answers a retry with the queued email instead of sending again, and recipients already sent are skipped. - Send events go through a scoped client that flushes before the Celery task ends. - Workflows sends refuse redirects and need an HTTPS POSTHOG_HOST outside DEBUG, because the request carries the secret key and the email's contents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
…s test case mypy rejects `None` as the pool argument of `urllib3.exceptions.MaxRetryError`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: b07b2dac-4768-46c8-8930-75b8c8aecc31
|
/trunk cancel |
a-lider
left a comment
a-lider
left a comment
There was a problem hiding this comment.
LGTM, thanks for pushing this!
Left a few questions to double-check.
| except WorkflowsEmailError as err: | ||
| if not _workflows_error_allows_fallback(err): | ||
| raise |
There was a problem hiding this comment.
Seems unlikely, but could Workflows queue the email and time out before responding?
If so, the retry might send a duplicate email through Customer.io
There was a problem hiding this comment.
Good catch. A timeout already skips the fallback: it raises into the Celery retry, which posts again with the same Idempotency-Key, and the send API replays the first send. But the retry could still route differently: the flag changes between attempts, the flag check fails, or the retry gets a fallback-eligible error such as a refused connection.
Fixed in b5b85ef: after any attempt Workflows may have queued, the send is pinned to Workflows in Redis for 24h under its idempotency key. A retry skips the flag and never falls back, so the worst case is a failed email rather than a duplicate. test_retry_after_an_uncertain_workflows_send_never_uses_customer_io covers the flag-off and refused-connection retries.
| body = response.json() | ||
| return {"message_id": body.get("message_id"), "replayed": bool(body.get("replayed", False))} |
There was a problem hiding this comment.
A 202 without message_id reaches not_queued below and falls back to Customer.io, but Workflows may have already queued the email.
There was a problem hiding this comment.
Agreed. The send API always returns one, but a 202 means it may be queued, so falling back was wrong. Fixed in b5b85ef: a 202 without a message_id, or with a body that isn't JSON, now retries with the same key and pins the send to Workflows. Added a case to the routing test.
While checking this I found that synchronous senders such as password reset ran Celery's retries inline, so a Workflows timeout could hold the request for about 40s. They now make one attempt and hand an uncertain one to a worker.
| "template_name": template_name, | ||
| "campaign_key": campaign_key, | ||
| "provider": provider, | ||
| **provider_response, |
There was a problem hiding this comment.
This is probably for another PR, but it seems message_id is missing from $workflows_email_sent and $workflows_email_delivered. Without it we can't match sends to deliveries and track for example the average delivery time between transactional email triggered and $workflows_email_sent.
I think this is one of the metrics we need to monitor closely during rollout. We got a burst of support tickets in the past like "Why didn't I receive a verification code?" after login when delivery took too long. More than 10-20 sec is already quite a lot for login.
There was a problem hiding this comment.
Agreed, and the login-code point is a great one. That change is on the Workflows side, so it'll be a separate PR. Triggered-to-sent latency per template will be a gate before the auth templates roll out.
- Pin a send to Workflows after an attempt that may have queued it, so a retry skips the flag and never falls back. - Treat a 202 without a message_id as possibly queued instead of falling back. - Synchronous sends make one Workflows attempt and hand an uncertain one to a worker, instead of eager retries inside the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: a5c9c2a4-4563-4d98-a4b6-ef13d86f4473
A pinned retry now skips only the flag. Without the transactional email key or an https host it sends through Customer.io, so a rollback by settings cannot post the email to an unconfigured or insecure host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: a5c9c2a4-4563-4d98-a4b6-ef13d86f4473
Problem
posthog/email.py) send only through Customer.io, and PostHog is moving them to Workflows.Changes
posthog/email.pycan now send through Workflows instead of Customer.io, per recipient, when thetransactional-emails-via-workflowsflag matches. Without a match, nothing changes.password_reset, the single entry inWORKFLOWS_TEMPLATE_ID_MAP, for one production test. A template in the map still uses Customer.io until its flag rule matches.emailandtransactional_email_template. Its release conditions are one staff rule plus one rule per template, whose rollout % is that template's share of recipients.message_id, a timeout, or a reset after sendingIdempotency-Keyfrom the campaign key and recipient, so a retry gets back the queued email instead of a second one.POSTHOG_HOSToutside DEBUG.customer.*attributes travel inmessage_data, because templates read onlyvariables.*.transactional email triggeredgains aproviderproperty. New eventstransactional email failedandtransactional email workflows rejectedcarrytemplate_name, because the exception error tracking receives has none.Emails.send(posthog-python#1093), and its error carries the samecodeandstatus_code. Moving to the SDK replaces one function body.POSTHOG_TRANSACTIONAL_EMAIL_API_KEYandPOSTHOG_HOST, the settings of the shared client inposthog/transactional_email.py, and post to/api/projects/@current/.... The PR adds no settings.How did you test this code?
Test rationale: one parameterized test in
posthog/test/test_email.pycovers each row of the table above, plus flag off. Each case asserts the provider called, the events captured and theMessagingRecordoutcome. It also checks the payload: unescaped values, thecustomerattributes, the idempotency header and the@currentURL. The refused-connection, reset-after-send, 503 and timeout cases guard which failures may fall back, the split the SDK move must keep. The uncertain cases assert the error reaches Celery. The closest existing test,test_send_via_http_success, covers only the Customer.io path.Two more tests follow review.
test_retry_after_an_uncertain_workflows_send_stays_on_workflows_while_it_is_configuredfails if a retry after a timeout reaches Customer.io when the flag turns off or Workflows refuses the connection, or if it still posts to Workflows after the key is removed.test_inline_send_hands_an_uncertain_workflows_send_to_a_workerfails if a request-path send retries inline.End to end on a local stack, against the first version of this PR: that version routed three canary templates with its own settings. Each fired through its real call site, once to an
@posthog.comuser and once to an@example.comuser.@posthog.com@example.compassword_resetPOST /api/reset/provider=workflowsapproval_requestedsend_approval_requested_notificationprovider=workflowsproxy_provisionedactivity_send_proxy_created_emailprovider=workflows@example.comsends failed with "Missing Customer.io API key", which shows the routing.Hogflix & O'Reilly's Appand a requester nameRory O'Neil & Samrender once, with no&or'in the HTML or the text part.password_reset. The unit tests cover them.password_resetsend to one test account checks them after deploy.Screenshots from maildev
Release status
Docs update
None.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: Claude Code, Opus 5.5 (
claude-opus-5-5)/writing-tests,/writing-code-comments,/writing-pr-descriptions.WORKFLOWS_TRANSACTIONAL_EMAIL_*settings. Review narrowed it topassword_reset, moved it onto the existing shared key and host, and dropped a per-instance template ID override that only local testing used.message_idrule and the worker handoff for request-path sends.message_idon the$workflows_email_*events is a separate Workflows change.🤖 Generated with Claude Code
Created with PostHog Desktop