[Silthus]

Problem

Sandbox email must not earn a project's own sending tier, and accepted sends need a distinct delivery signal.

Supersedes PostHog#111904 (left open as the original draft).

Stack: layer 10 of 12, stacked on #453. Next: #455.

This fork PR moves to an upstream PostHog/posthog PR after review.

Warning

🟡 Merge risk: medium · one-way door (merge-risk rubric)

• Tier history controls real sending limits. Delivered analytics events persist after a revert.
• Signed tracking codes and the sandbox configuration set scope delivery capture; focused history tests protect own sends.

Changes

• Sandbox sends no longer count toward tier use or promotion. Their bounces and complaints still count.
• Sandbox delivery notifications capture recipient counts for the signed project, including test sends.
• Existing email metrics remain intact. Daily history subtracts sandbox sends per day and floors the result at zero.
• The sending allowance panel leaves out sandbox sends too, as the tier history does.

Before:

flowchart LR
    A[All sends] --> B[Tier history]
    C[SES delivery] --> D[Email delivery metrics]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000
    class A,C phYellow
    class B,D phGray

Loading

After:

flowchart LR
    A[Own sends] --> B[Tier history]
    C[Signed sandbox delivery] --> D[Email delivery metrics]
    C --> E[Sandbox delivered event]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000
    class A,C phYellow
    class B,D,E phGray

Loading

How did you test this code?

Current evidence, 2026-10-24, head dafe929b5d8.

Owed proofs:

• Live run: rerun at the stack tip 8ac8853cc82, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass. 12 of 13 recorded cases match the run at fa33d78fb36. Case 2c (an unverified member on To) now shows the verify-your-email guidance with reason recipient_unverified, and makes no SES request. No case covers SES delivery notifications or tier history, which need real SES feedback; the webhook and tier tests cover them.
• Prod shape: gap, accepted by Michael on 2026-10-24 until production read access on 2026-11-02. Until then it is unknown how many app metric totals take each changed branch.

Changes in this layer since the 2026-10-19 heads (restacked onto fork master 448d802: no conflict in this layer):

• Round 1 fixed: the sending allowance panel subtracts email_sandbox_sent, as the tier sweep does (test_sending_allowance_leaves_out_sandbox_sends).
• Round 2 fixed: the allowance test fake now honors the requested metric names, so a query that drops the sandbox metric fails the test (test_sending_allowance_leaves_out_sandbox_sends, mutation checked red then green).
• Final read: GPT-6.1 Sol read the G4 layer range-diffs and found nothing.
• Round 4 (UI-F3): the sandbox outcome type carries the new recipient_unverified reason from feat(workflows): restrict sandbox email to verified org members #434. Type only; delivery history is unchanged.
• Final read 2: GPT-6.1 Sol read the round 4 range-diffs per group and found nothing.

Local gate at the stack tip 8ac8853cc82, shared by every layer

• Python suites for workflows, messaging, CDP validation and integrations: 3007 passed.
• Node CDP messaging, managers, rate limiter, hogflows and cdp-api: 42 suites, 1157 passed.
• Frontend Jest for the 17 changed suites: 292 passed.
• mypy: clean. Node tsc and frontend tsgo: clean.
• ruff, oxfmt, prettier, eslint and oxlint: clean on the 15 files changed in round 4.
• Per-layer suites during the round 4 fold: feat(workflows): restrict sandbox email to verified org members #434 Node 201 passed, feat(workflows): cap sandbox sends and refund ses throttles #441 222, feat(workflows): test templates with the sandbox sender #451 frontend 27, feat(workflows): honor shared sandbox tenant pauses #453 Python 93 and Node 234, feat(workflows): offer a switch from the sandbox sender #456 banner 15.
• Review: round 1, round 2, final read 1 and final read 2 per group. Every finding is fixed or recorded as a follow-up. Both final reads found nothing.

Test seams: SES webhook attribution, ClickHouse-backed tier recomputation and the sending allowance API.

Test rationale: Existing signed-delivery cases distinguish sandbox from own sends and count recipient copies. Tier-history cases cover per-day subtraction, retained feedback and promotion/decay.

Gaps: real AWS SES delivery and the AWS-side tenant pause (every proof uses a local SES fake or an unreachable SES), clustered Valkey, the production data shape (prod-shape gap until production read access on 2026-11-02), and fork CI, which fails during setup: path selection fails without a GitHub App client ID, and the Security aggregate rejects its fork-skipped path-filter job. Application CI is a verification gap.

Earlier evidence (superseded heads)

Round 3 heads, 2026-10-23. This layer's head: 370462bc02a. Stack tip fa33d78fb36.

• Live run: run at the stack tip fa33d78fb36, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass, identical to the earlier run. No case covers SES delivery notifications or tier history, which need real SES feedback; the webhook and tier tests cover them.
• Local gate at fa33d78fb36: Python 3007 passed; Node CDP 42 suites, 1157 passed; email service suites at the final heads of feat(workflows): restrict sandbox email to verified org members #434, feat(workflows): cap sandbox sends and refund ses throttles #441, feat(workflows): test templates with the sandbox sender #451 and feat(workflows): honor shared sandbox tenant pauses #453 passed; frontend Jest 17 changed suites, 290 passed; mypy, Node tsc and frontend tsgo clean; ruff, oxfmt, oxlint, prettier and eslint clean on the files changed since round 2. Final read 1 found nothing.

Whole-stack QA swarm, 2026-10-19. Reviewed tip cfee7e148ae. This layer's head: f90a613d5b6.

• Live run at b9bd346ed0e (this stack's cdp-api against a local SES fake, workflow test-run endpoint): a sandbox send to a verified member went out from the fixed identity through the sandbox tenant and configuration set. The step's From name, From address and Reply-To were ignored with a log line. A non-member on CC and an unverified member were blocked with the clear message. A paused tenant state skipped sandbox sends while own-sender sends continued.
• Shadow diff: an own-sender send produced the same SES request and the same stored message on master and on this stack.
• Cross-version: a worker on master reports a sandbox send as sent without calling SES. See the deploy order in feat(workflows): provision a managed sandbox email sender #426's merge risk.
• Checks at the tip (Node suites at 1a0fe3663c2; later fixes touch no Node code): Node CDP messaging, members and rate limiter suites (including email-tracking.service.test.ts, which now runs), the cdp-api suite, the Python sandbox, tenant, tier, reputation and validation suites, the frontend Jest suites of this stack, ruff, and the strict pre-push preflight: pass.
• Gaps: real AWS SES delivery, clustered Valkey, and the production data shape (no production read access yet). Fork CI still fails during setup.

Rebased onto master 734468db6f9 (2026-10-19), head d28e139a268.

• Drift: No master change touches this layer's files beyond the shared drift below it.
• Adapted: No conflicts. The layer applies unchanged.
• This layer: test_email_sending_tier.py passes at the tip (below).
• Gaps at modernize time: email-tracking.service.test.ts did not run then. It runs now; see the QA swarm block above. Fork CI still fails during setup, as noted below. Real AWS delivery and clustered Valkey remain unchecked.

Checks at the stack tip

Stack tip b9bd346ed0e (#456), shared by every layer:

• Python, 354 passed: sandbox sender, sender validation, CDP validation, reputation, SES tenant state and events, sending tier.
• Node CDP messaging, managers, and rate limiter: 17 suites, 599 passed.
• Frontend Jest, 37 suites, 487 passed. Five suites timed out under host load in the batch run and passed when rerun alone.
• Frontend tsgo: no errors. Node tsgo: no errors in CDP. The 23 errors it reports are all in session replay files that need the unbuilt Rust @posthog/replay-anonymizer module.
• ruff, oxfmt, oxlint, prettier, eslint, product:lint (workflows, cdp), IDOR coverage, hogli build:openapi (no diff), and the strict pre-push preflight: pass.
• Review: GPT-6.1 Sol read every conflict resolution in the stack and found no issues.

Observed targeted verification after QA fixes

Email tracking sandbox delivery cases: 6 passed, 41 skipped
TestSandboxSendsInTierHistory: 6 passed
ci:preflight --strict --against workflows-sandbox/tenant-pause: failures 0

Observed combined-stack gate

Node.js typecheck and build: exit 0
Frontend typecheck and build: exit 0
Success: no issues found in 23388 source files
product:lint workflows: All checks passed
ci:preflight --strict: failures 0

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Release status

• This change is behind a feature flag and is not available to users

Docs update

Updates the existing email sending tiers document with sandbox subtraction and delivery capture configuration.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Decisions

Question	Answer	Who	Where
Prod-shape proofs before production read access	Accepted as gaps until production read access on 2026-11-02	👤 Human	decision comment

Involvement: restack onto fork master 448d802 on 2026-10-23; round 1 with GPT-6.1 Sol lanes plus Claude Opus 5.5 fresh eyes for group G4; round 2 on the fix diff; final read 1 by GPT-6.1 Sol per group, with no findings; proofs captured; final read 2 by GPT-6.1 Sol on the round 4 decisions, with no findings; the round 4 live run and a UI walk of the two changed surfaces. Before that: the 2026-10-19 whole-stack QA swarm and the modernize rebase (see Earlier evidence). No review bots run on the fork.

Agent: Codex with GPT-6.1 Sol (gpt-6.1-sol) built the original layers, orchestrated by Claude Code with Claude Opus 5.5 (claude-opus-5-5). Restack, review fixes and this description refresh: Claude Opus 5.5 (claude-opus-5-5). Review: GPT-6.1 Sol (gpt-6.1-sol). Fresh eyes: Claude Opus 5.5 (claude-opus-5-5).

Skills: tdd, pr, writing-pr-descriptions, stacking-prs, merge-risk, qa-swarm, running-ci-preflight, and writing-voice. The description refresh used pr-description, writing-pr-descriptions, writing-voice and merge-risk.

The shared sender instance supplies webhook configuration. Public source and invented fixtures only.

🤖 Generated with Claude Code

[Silthus]

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decision lines for this PR. The Decisions table in the description links each row here.

• 👤 Prod-shape proofs before production read access Accepted as gaps until production read access on 2026-11-02.

[Silthus]

Replaced by the re-sliced fork stack: #609 and #610. Tracked in Silthus/personal#71.