[Silthus]

Problem

Sandbox email must stop when SES pauses the shared sender, including test sends and sends waiting for a daily allowance.

Supersedes PostHog#111820 (left open as the original draft), together with the preceding schema layer.

Stack: layer 9 of 12, stacked on #452. Next: #454.

This fork PR moves to an upstream PostHog/posthog PR after review.

Warning

🟡 Merge risk: medium · one-way door (merge-risk rubric)

• A stale pause decision could allow a real email; an incorrect decision could block every sandbox sender.
• The worker reads primary storage, refreshes its cache, honors invalidation, and fails closed on lookup errors.

Changes

• The sandbox sender is offered only when Django also has SES_SANDBOX_TENANT_NAME. Without it, the provisioning endpoint answers 404, so no sandbox sender exists whose pauses Django cannot mirror.
• Sandbox sends skip with guidance to verify an own sender while the shared SES tenant is disabled.
• Delivery rechecks pauses after claiming a daily allowance and returns unused claims.
• SES notifications and the existing sweep mirror authoritative tenant state under a dedicated row lock.
• Committed changes invalidate worker caches. Own senders retain their existing pause behavior.
• The worker retries a failed tenant state read before it skips sandbox sends, so a short database blip does not pause them.

Before:

flowchart LR
    A[Sandbox email] --> B[Recipients and caps] --> C[SES]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000
    class A phYellow
    class B phBlue
    class C phRed

Loading

After:

flowchart LR
    A[SES notification or sweep] --> B[Shared tenant state]
    B --> C[Worker pause gate]
    D[Sandbox email] --> C --> E[Recipients and caps] --> F[Pause recheck] --> G[SES]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000
    class C,E,F phBlue
    class A,G phRed
    class D phYellow
    class B phGray

Loading

How did you test this code?

Current evidence, 2026-10-24, head 5fc313471fb.

Owed proofs:

• Live run: rerun at the stack tip 8ac8853cc82, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass. 12 of 13 recorded cases match the run at fa33d78fb36. Case 2c (an unverified member on To) now shows the verify-your-email guidance with reason recipient_unverified, and makes no SES request. For this layer: case 4 marks the tenant row DISABLED. Sandbox sends skip with the paused guidance, and own senders still send for master-shaped and new data.
• Cross-version: live-run case 6, carried from fa33d78fb36 and rerun at 8ac8853cc82 with the same result. A tip worker on a database without migration 0037 skips sandbox sends (it fails closed, with no SES request), while own-sender sends go out.

Changes in this layer since the 2026-10-19 heads (restacked onto fork master 448d802: no conflict in this layer):

• Round 1 fixed: the tenant state loader retries a failed read (sandbox-sender-state.service.test.ts).
• Round 1 fixed: tenant events ignore a change that only touches the dropped reputation impact (only_the_impact_changed case).
• No change in this layer since round 2.
• Final read: GPT-6.1 Sol read the G4 layer range-diffs and found nothing.
• Round 4 (R2-11, UI-F3): the provisioning check reads SES_SANDBOX_FROM_ADDRESS with the tenant setting, and the blocked reason union gains recipient_unverified. Carried from feat(workflows): provision a managed sandbox email sender #426 and feat(workflows): restrict sandbox email to verified org members #434; Python 93 and Node 234 tests passed in this layer.
• Final read 2: GPT-6.1 Sol read the round 4 range-diffs per group and found nothing.

Local gate at the stack tip 8ac8853cc82, shared by every layer

• Python suites for workflows, messaging, CDP validation and integrations: 3007 passed.
• Node CDP messaging, managers, rate limiter, hogflows and cdp-api: 42 suites, 1157 passed.
• Frontend Jest for the 17 changed suites: 292 passed.
• mypy: clean. Node tsc and frontend tsgo: clean.
• ruff, oxfmt, prettier, eslint and oxlint: clean on the 15 files changed in round 4.
• Per-layer suites during the round 4 fold: feat(workflows): restrict sandbox email to verified org members #434 Node 201 passed, feat(workflows): cap sandbox sends and refund ses throttles #441 222, feat(workflows): test templates with the sandbox sender #451 frontend 27, feat(workflows): honor shared sandbox tenant pauses #453 Python 93 and Node 234, feat(workflows): offer a switch from the sandbox sender #456 banner 15.
• Review: round 1, round 2, final read 1 and final read 2 per group. Every finding is fixed or recorded as a follow-up. Both final reads found nothing.

Test seams: SES notification/sweep persistence and post-commit invalidation; the worker's tenant state loader; EmailService pause checks and claim refunds.

Test rationale: Existing pause cases cover cached and pending-read invalidations, delayed claims and own-sender exclusion. Constructor fixtures now include the telemetry dependency introduced below; database-free content tests still pass.

Gaps: real AWS SES delivery and the AWS-side tenant pause (every proof uses a local SES fake or an unreachable SES), clustered Valkey, the production data shape (prod-shape gap until production read access on 2026-11-02), and fork CI, which fails during setup: path selection fails without a GitHub App client ID, and the Security aggregate rejects its fork-skipped path-filter job. Application CI is a verification gap.

Earlier evidence (superseded heads)

Round 3 heads, 2026-10-23. This layer's head: e6d96e87702. Stack tip fa33d78fb36.

• Live run: run at the stack tip fa33d78fb36, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass, identical to the earlier run. For this layer: case 4 marks the tenant row DISABLED. Sandbox sends skip with the paused guidance, and own senders still send for master-shaped and new data.
• Local gate at fa33d78fb36: Python 3007 passed; Node CDP 42 suites, 1157 passed; email service suites at the final heads of feat(workflows): restrict sandbox email to verified org members #434, feat(workflows): cap sandbox sends and refund ses throttles #441, feat(workflows): test templates with the sandbox sender #451 and feat(workflows): honor shared sandbox tenant pauses #453 passed; frontend Jest 17 changed suites, 290 passed; mypy, Node tsc and frontend tsgo clean; ruff, oxfmt, oxlint, prettier and eslint clean on the files changed since round 2. Final read 1 found nothing.

Whole-stack QA swarm, 2026-10-19. Reviewed tip cfee7e148ae. This layer's head: 70d2ce15b01.

• Live run at b9bd346ed0e (this stack's cdp-api against a local SES fake, workflow test-run endpoint): a sandbox send to a verified member went out from the fixed identity through the sandbox tenant and configuration set. The step's From name, From address and Reply-To were ignored with a log line. A non-member on CC and an unverified member were blocked with the clear message. A paused tenant state skipped sandbox sends while own-sender sends continued.
• Shadow diff: an own-sender send produced the same SES request and the same stored message on master and on this stack.
• Cross-version: a worker on master reports a sandbox send as sent without calling SES. See the deploy order in feat(workflows): provision a managed sandbox email sender #426's merge risk.
• Checks at the tip (Node suites at 1a0fe3663c2; later fixes touch no Node code): Node CDP messaging, members and rate limiter suites (including email-tracking.service.test.ts, which now runs), the cdp-api suite, the Python sandbox, tenant, tier, reputation and validation suites, the frontend Jest suites of this stack, ruff, and the strict pre-push preflight: pass.
• Fixed in this layer: provisioning requires SES_SANDBOX_TENANT_NAME, as the spec's settings rule says, and the internal doc lists all three settings. The new "tenant name unset" API case failed before the fix.
• Gaps: real AWS SES delivery, clustered Valkey, and the production data shape (no production read access yet). Fork CI still fails during setup.

Rebased onto master 734468db6f9 (2026-10-19), head 2735de37b56.

• Drift: No master change touches this layer's files beyond the shared drift below it.
• Adapted: No conflicts. The layer applies unchanged.
• This layer: test_ses_tenant_state.py, test_ses_tenant_events.py, and the Node sandbox suites pass at the tip (below).
• Gaps at modernize time: email-tracking.service.test.ts did not run then. It runs now; see the QA swarm block above. Fork CI still fails during setup, as noted below. Real AWS delivery and clustered Valkey remain unchecked.

Checks at the stack tip

Stack tip b9bd346ed0e (#456), shared by every layer:

• Python, 354 passed: sandbox sender, sender validation, CDP validation, reputation, SES tenant state and events, sending tier.
• Node CDP messaging, managers, and rate limiter: 17 suites, 599 passed.
• Frontend Jest, 37 suites, 487 passed. Five suites timed out under host load in the batch run and passed when rerun alone.
• Frontend tsgo: no errors. Node tsgo: no errors in CDP. The 23 errors it reports are all in session replay files that need the unbuilt Rust @posthog/replay-anonymizer module.
• ruff, oxfmt, oxlint, prettier, eslint, product:lint (workflows, cdp), IDOR coverage, hogli build:openapi (no diff), and the strict pre-push preflight: pass.
• Review: GPT-6.1 Sol read every conflict resolution in the stack and found no issues.

The delayed-claim fixture now installs its clock before constructing the membership service, so pause recovery and authorization use the same time.

Observed delayed-claim clock correction

Combined cap/pause cases: pause recovery failed with clocks constructed in different orders
email.service.test.ts -t 'rechecks a pause after a delayed cap claim': 1 passed, 211 skipped

Observed targeted verification after QA fixes

test_ses_tenant_state.py sandbox cases: 7 passed
SES provider cases: 3 passed
Sandbox sender content file: 27 passed
EmailService sandbox pause gate: 9 passed, 201 skipped
ci:preflight --strict --against workflows-sandbox/tenant-state-schema: failures 0

Observed combined-stack gate

Node.js typecheck and build: exit 0
Frontend typecheck and build: exit 0
Success: no issues found in 23388 source files
product:lint workflows: All checks passed
ci:preflight --strict: failures 0

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Release status

• This change is behind a feature flag and is not available to users

Docs update

Updates the existing email sending tiers document with sandbox pause behavior, worker invalidation, and shared Django/worker tenant configuration.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Decisions

Question	Answer	Who	Where
Provision without the tenant setting?	No: the endpoint answers 404 until all three Django settings are set, per the spec	🤖 Agent	QA handoff
Unused reputation_impact column (G4-3)	Dropped before migration 0037 merges. The pause gate reads the SES sending status, which carries the managed reputation pause	🤖 Agent	decision comment

Involvement: restack onto fork master 448d802 on 2026-10-23; round 1 with GPT-6.1 Sol lanes plus Claude Opus 5.5 fresh eyes for group G4; round 2 on the fix diff; final read 1 by GPT-6.1 Sol per group, with no findings; proofs captured; final read 2 by GPT-6.1 Sol on the round 4 decisions, with no findings; the round 4 live run and a UI walk of the two changed surfaces. Before that: the 2026-10-19 whole-stack QA swarm and the modernize rebase (see Earlier evidence). No review bots run on the fork.

Agent: Codex with GPT-6.1 Sol (gpt-6.1-sol) built the original layers, orchestrated by Claude Code with Claude Opus 5.5 (claude-opus-5-5). Restack, review fixes and this description refresh: Claude Opus 5.5 (claude-opus-5-5). Review: GPT-6.1 Sol (gpt-6.1-sol). Fresh eyes: Claude Opus 5.5 (claude-opus-5-5).

Skills: tdd, pr, writing-pr-descriptions, stacking-prs, merge-risk, qa-swarm, writing-dataclasses, running-ci-preflight, and writing-voice. The description refresh used pr-description, writing-pr-descriptions, writing-voice and merge-risk.

The additive table ships below this layer. The provider extraction preserves master's separate SES metrics client. Public source and invented fixtures only.

🤖 Generated with Claude Code

[Silthus]

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decision lines for this PR. The Decisions table in the description links each row here.

• 🤖 Unused reputation_impact column (G4-3) Dropped before migration 0037 merges. The pause gate reads the SES sending status, which carries the managed reputation pause

[Silthus]

Replaced by the re-sliced fork stack: #608. Tracked in Silthus/personal#71.