[Silthus]

Problem

After #434, sandbox email reaches only verified members of the organization, but nothing limits how much of it goes out. A workflow on a busy event can send thousands of sandbox emails to the same few teammates. That fills their inboxes and spends the shared sender's sending quota, which other new teams need to try email.

Sandbox sends need a small daily allowance per project and per organization.

Supersedes PostHog#111690 (left open as the original draft)

Supersedes PostHog#112071 (closed). Includes its cap-token refund, claim-once, and late-return fixes; the route layer includes its empty-settings fix.

Stack: layer 5 of 12, stacked on #434. Next: #450.

This fork PR moves to an upstream PostHog/posthog PR after review.

Caution

🔴 Merge risk: high · one-way door (merge-risk rubric)

• Atomic claims and refunds control real email volume. A lost limit can permit deliveries that a revert cannot recall.
• The worker switch defaults off; shared Valkey buckets fail closed, and refund markers prevent duplicate or delayed replenishment.

Changes

• With sandbox delivery enabled, CDP API opens only the SES Valkey pool, not the frequency cap pool. With it off, CDP API wiring matches master.
• A claim UUID identifies each allowance request. Replayed Valkey commands return the original grant or denial without spending twice.
• Sandbox sends claim project and organization-wide recipient allowances, including editor test sends through CDP API.
• Cap keys and final membership checks use one organization snapshot, valid for at most one minute.
• An expired or blocked final check returns the allowance before skipping delivery.
• Exhausted allowances skip delivery with guidance to verify a sender; unavailable limits fail closed.
• SES throttle responses return the allowance before the existing retry. Duplicate returns and commands arriving after five minutes add no tokens.
• Allowances refill continuously over 24 hours. Recipient buckets span projects within one organization; project spending counts every recipient copy.

Before:

flowchart LR
    Send[Sandbox send] --> Members[Final membership check] --> SES[SES delivery]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    class Send phYellow;
    class Members phBlue;
    class SES phRed;

Loading

After:

flowchart LR
    Send[Worker or API test send] --> Members[Bounded organization snapshot] --> Claim[Atomic sandbox claim] --> Recheck[Membership check with same snapshot] --> SES[SES delivery]
    Claim --> Skip[Skip when denied]
    SES -->|Throttle| Return[Deduplicated expiring return] --> Retry[Existing delivery retry]
    Retry --> Members
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    class Send,Skip phYellow;
    class Members,Claim,Recheck,Return,Retry phBlue;
    class SES phRed;

Loading

Known follow-ups (found in the UI walk, not fixed here):

• A sandbox test send to a non-member on a domain without mail servers shows the hard-bounce reason, not the members-only reason, because the mail-server check runs first. The send is refused either way.

How did you test this code?

Current evidence, 2026-10-24, head 64578999614.

Owed proofs:

• Live run: rerun at the stack tip 8ac8853cc82, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass. 12 of 13 recorded cases match the run at fa33d78fb36. Case 2c (an unverified member on To) now shows the verify-your-email guidance with reason recipient_unverified, and makes no SES request. For this layer: case 3 sets a project daily cap of 1. The first send goes out; the second skips with cap guidance and no SES request.
• Shadow diff: carried from the stack tip fa33d78fb36, because round 4 does not touch the own-sender path or the base worker. Own-sender SES requests from fork master 448d802 and that tip are identical across 7 cases (stored flows, editor drafts, a custom From name, unverified and missing senders), apart from the per-run signatures of tracking tokens. One intended difference: an editor test run with an empty recipient reports skipped instead of success, because feat(workflows): test templates with the sandbox sender #451 reports skipped email steps in test runs. Live sends are unchanged.

Changes in this layer since the 2026-10-19 heads (restacked onto fork master 448d802: the cdp-api dependencies in server.ts combine master's idempotency Valkey pool with this layer's sandbox email pools):

• Round 1 fixed: cdp-api opens only the email validation pool it needs, not the frequency cap pool, and the emailValidationValkey doc no longer says it is null on cdp-api. Wiring and docs; the existing suites cover it.
• Round 2 fixed: the Valkey pool comments in cdp-services.ts and server.ts now tell the email workers' pools apart from the SES-only pool on sandbox-enabled cdp-api. Docs only.
• Final read: GPT-6.1 Sol read the G2 layer range-diffs and found nothing.
• Round 4 (UI-F3): the daily cap's block copy table carries the new recipient_unverified reason from feat(workflows): restrict sandbox email to verified org members #434. Type and wiring only; the cap behavior is unchanged (222 tests passed in this layer's Node suites).
• Final read 2: GPT-6.1 Sol read the round 4 range-diffs per group and found nothing.

Local gate at the stack tip 8ac8853cc82, shared by every layer

• Python suites for workflows, messaging, CDP validation and integrations: 3007 passed.
• Node CDP messaging, managers, rate limiter, hogflows and cdp-api: 42 suites, 1157 passed.
• Frontend Jest for the 17 changed suites: 292 passed.
• mypy: clean. Node tsc and frontend tsgo: clean.
• ruff, oxfmt, prettier, eslint and oxlint: clean on the 15 files changed in round 4.
• Per-layer suites during the round 4 fold: feat(workflows): restrict sandbox email to verified org members #434 Node 201 passed, feat(workflows): cap sandbox sends and refund ses throttles #441 222, feat(workflows): test templates with the sandbox sender #451 frontend 27, feat(workflows): honor shared sandbox tenant pauses #453 Python 93 and Node 234, feat(workflows): offer a switch from the sandbox sender #456 banner 15.
• Review: round 1, round 2, final read 1 and final read 2 per group. Every finding is fixed or recorded as a follow-up. Both final reads found nothing.

Test seams: RateLimiterService against local Redis and EmailService through SES, membership expiry and refunds.

Test rationale: The replay and cross-project cases extend existing cap coverage. Post-claim membership checks refund a send blocked after a stalled claim; the extra pre-claim membership check is removed.

Gaps: real AWS SES delivery and the AWS-side tenant pause (every proof uses a local SES fake or an unreachable SES), clustered Valkey, the production data shape (prod-shape gap until production read access on 2026-11-02), and fork CI, which fails during setup: path selection fails without a GitHub App client ID, and the Security aggregate rejects its fork-skipped path-filter job. Application CI is a verification gap.

Earlier evidence (superseded heads)

Round 3 heads, 2026-10-23. This layer's head: 9c0f75432bd. Stack tip fa33d78fb36.

• Live run: run at the stack tip fa33d78fb36, which contains this layer. The stack's cdp-api sent through the workflow test-run endpoint to a local SES fake that records each request. 9 of 9 cases pass, identical to the earlier run. For this layer: case 3 sets a project daily cap of 1. The first send goes out; the second skips with cap guidance and no SES request.
• Local gate at fa33d78fb36: Python 3007 passed; Node CDP 42 suites, 1157 passed; email service suites at the final heads of feat(workflows): restrict sandbox email to verified org members #434, feat(workflows): cap sandbox sends and refund ses throttles #441, feat(workflows): test templates with the sandbox sender #451 and feat(workflows): honor shared sandbox tenant pauses #453 passed; frontend Jest 17 changed suites, 290 passed; mypy, Node tsc and frontend tsgo clean; ruff, oxfmt, oxlint, prettier and eslint clean on the files changed since round 2. Final read 1 found nothing.

Whole-stack QA swarm, 2026-10-19. Reviewed tip cfee7e148ae. This layer's head: 27d1e529411.

• Live run at b9bd346ed0e (this stack's cdp-api against a local SES fake, workflow test-run endpoint): a sandbox send to a verified member went out from the fixed identity through the sandbox tenant and configuration set. The step's From name, From address and Reply-To were ignored with a log line. A non-member on CC and an unverified member were blocked with the clear message. A paused tenant state skipped sandbox sends while own-sender sends continued.
• Shadow diff: an own-sender send produced the same SES request and the same stored message on master and on this stack.
• Cross-version: a worker on master reports a sandbox send as sent without calling SES. See the deploy order in feat(workflows): provision a managed sandbox email sender #426's merge risk.
• Checks at the tip (Node suites at 1a0fe3663c2; later fixes touch no Node code): Node CDP messaging, members and rate limiter suites (including email-tracking.service.test.ts, which now runs), the cdp-api suite, the Python sandbox, tenant, tier, reputation and validation suites, the frontend Jest suites of this stack, ruff, and the strict pre-push preflight: pass.
• Gaps: real AWS SES delivery, clustered Valkey, and the production data shape (no production read access yet). Fork CI still fails during setup.

Rebased onto master 734468db6f9 (2026-10-19), head 2d22d29e4a7.

• Drift: Master renamed withEmailValidationValkey to withEmailActionValkeys in server.ts. It now grants the frequency cap pool too.
• Adapted: Sandbox-enabled cdp-api now takes withEmailActionValkeys. With the sandbox switch off, cdp-api wiring matches master.
• This layer: Node messaging, managers, and rate-limiter suites at this layer: 11 suites, 325 passed.
• Gaps at modernize time: email-tracking.service.test.ts did not run then. It runs now; see the QA swarm block above. Fork CI still fails during setup, as noted below. Real AWS delivery and clustered Valkey remain unchecked.

Checks at the stack tip

Stack tip b9bd346ed0e (#456), shared by every layer:

• Python, 354 passed: sandbox sender, sender validation, CDP validation, reputation, SES tenant state and events, sending tier.
• Node CDP messaging, managers, and rate limiter: 17 suites, 599 passed.
• Frontend Jest, 37 suites, 487 passed. Five suites timed out under host load in the batch run and passed when rerun alone.
• Frontend tsgo: no errors. Node tsgo: no errors in CDP. The 23 errors it reports are all in session replay files that need the unbuilt Rust @posthog/replay-anonymizer module.
• ruff, oxfmt, oxlint, prettier, eslint, product:lint (workflows, cdp), IDOR coverage, hogli build:openapi (no diff), and the strict pre-push preflight: pass.
• Review: GPT-6.1 Sol read every conflict resolution in the stack and found no issues.

The organization transfer and stalled-claim cases extend the existing delivery seam. Independently aged caches must share one authorization snapshot.

Observed organization snapshot verification

Warm-cache transfer: failed before shared snapshot; passed afterward
Organization expiry during cap claim: failed before preserving expiry; passed afterward
email.service.test.ts -t 'daily caps': 21 passed, 181 skipped
rate-limiter.service.test.ts -t 'shares recipient allowances': 1 passed, 25 skipped

Observed targeted verification after QA fixes

Replayed-claim case: failed before dedupe; passed afterward
Cross-project recipient case: failed before org scope; passed afterward
Focused replay, organization scope and refund cases: 4 passed
email.service.test.ts -t 'daily caps': 19 passed, 181 skipped

Observed combined-stack gate

Node.js typecheck and build: exit 0
Frontend typecheck and build: exit 0
Success: no issues found in 23388 source files
product:lint workflows: All checks passed
ci:preflight --strict: failures 0

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Release status

• This change is behind a feature flag and is not available to users

Docs update

Provisioning uses workflows-sandbox-sender; delivery uses WORKFLOWS_SANDBOX_SENDER_ENABLED, off by default.

The existing workflow email sending tiers document now covers sandbox allowances, inline test sends, and throttle refunds.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Decisions

Question	Answer	Who	Where
Gate order for a non-member on a domain without mail servers (UI-F1)	Keep the mail-server check first. The send is refused either way; reordering the gates is a follow-up with its own proof	🤖 Agent	decision comment
Test-only cleanup of the recipient and cap tests (G2-4)	Moved to a test-only follow-up PR on the stack tip. Folding it into #434 and #441 meant rewriting each later layer's tests by hand	🤖 Agent	decision comment
Limiter-owned refund receipt (G2-8)	Follow-up refactor. The sender-made claim ID has no known defect	🤖 Agent	decision comment
Footer organization ID after a project moves organizations (G2-13)	Rejected. The ID can lag up to 90 seconds, it affects attribution only, and the send decision uses the fresh snapshot	🤖 Agent	decision comment

Involvement: restack onto fork master 448d802 on 2026-10-23; round 1 with GPT-6.1 Sol lanes plus Claude Opus 5.5 fresh eyes for group G2; round 2 on the fix diff; final read 1 by GPT-6.1 Sol per group, with no findings; proofs captured; final read 2 by GPT-6.1 Sol on the round 4 decisions, with no findings; the round 4 live run and a UI walk of the two changed surfaces. Before that: the 2026-10-19 whole-stack QA swarm and the modernize rebase (see Earlier evidence). No review bots run on the fork.

Agent: Codex with GPT-6.1 Sol (gpt-6.1-sol) built the original layers, orchestrated by Claude Code with Claude Opus 5.5 (claude-opus-5-5). Restack, review fixes and this description refresh: Claude Opus 5.5 (claude-opus-5-5). Review: GPT-6.1 Sol (gpt-6.1-sol). Fresh eyes: Claude Opus 5.5 (claude-opus-5-5).

Skills: tdd, pr, stacking-prs, writing-pr-descriptions, merge-risk, qa-swarm, writing-tests, writing-code-comments, using-redis-token-buckets, running-ci-preflight, and writing-voice. The description refresh used pr-description, writing-pr-descriptions, writing-voice and merge-risk.

The rebuild adapts the original allowance code to master's SES retry path and preserves recipient normalization and UTM handling. Sources are public repository changes; test addresses are invented.

🤖 Generated with Claude Code

[Silthus]

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decision lines for this PR. The Decisions table in the description links each row here.

• 🤖 Gate order for a non-member on a domain without mail servers (UI-F1) Keep the mail-server check first. The send is refused either way; reordering the gates is a follow-up with its own proof
• 🤖 Test-only cleanup of the recipient and cap tests (G2-4) Moved to a test-only follow-up PR on the stack tip. Folding it into feat(workflows): restrict sandbox email to verified org members #434 and feat(workflows): cap sandbox sends and refund ses throttles #441 meant rewriting each later layer's tests by hand
• 🤖 Limiter-owned refund receipt (G2-8) Follow-up refactor. The sender-made claim ID has no known defect
• 🤖 Footer organization ID after a project moves organizations (G2-13) Rejected. The ID can lag up to 90 seconds, it affects attribution only, and the send decision uses the fresh snapshot

[Silthus]

Replaced by the re-sliced fork stack: #604. Tracked in Silthus/personal#71.