[Silthus]

Problem

All sandbox sends go out through one shared PostHog SES tenant.
When AWS pauses that tenant, for example over its reputation, the worker keeps trying to send through it.
This PR makes every sandbox send skip while the shared tenant is paused, or while its state cannot be read.

Part of Silthus/personal#199 (sandbox delivery). Stacked on #607, which adds the table this PR reads.
Replaces the worker half of #605. Originally #452 and #453.

Stack

• Slice V3 of the sandbox sender rollout: a paused shared SES tenant stops sandbox sends, and sandbox sends leave the sending tier. Slice order: feat(workflows): mirror the sandbox ses tenant state in django #607 → feat(workflows): skip sandbox sends while the shared ses tenant is paused #608 → feat(workflows): subtract sandbox sends from the sending tier #609 → feat(workflows): record sandbox deliveries as their own metric #610.
• Base: workflows-sandbox-delivery/tenant-pause-mirror (feat(workflows): mirror the sandbox ses tenant state in django #607). Lands after feat(workflows): mirror the sandbox ses tenant state in django #607 and every PR below it.
• Production lines: 99 (tests, stories, generated types, migrations and docs excluded).
• Rebuilt on 2026-10-28 on the current master, in the new slice order. No change to its own lines.
• Lane gate on 35ce76039dd: Node typecheck and lint clean, touched Node suites 264 passed. Review: GPT-6.1 Sol through a delegated review of the restack (each PR's diff, its conflict resolutions, the slice seams and the save gate), no findings.
• Proof on 35ce76039dd, 2026-10-28: live run below. Proof gate: fails only on the fork-only ci gap (exit 3); every other check passes.

Impact

Grow Workflows, activation rate: the pause gate is a condition for enabling the sandbox sender in production.
No production team sees this yet: WORKFLOWS_SANDBOX_SENDER_ENABLED stays off.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

• Only sends through a sandbox integration read the new state. Own-sender sends are unchanged.
• A sandbox send fails closed when the state read fails.
• Safe to deploy alone: it needs feat(workflows): mirror the sandbox ses tenant state in django #607's migration deployed first. If it ran before the migration, the state read would fail and sandbox sends would skip. No other send changes, and the sandbox switch is off.

Production lines: 99 changed (git diff --numstat against #607, tests and docs excluded).

Changes

• Every sandbox send, test sends included, skips while the shared sandbox tenant is DISABLED, or when the worker cannot read its state. The run log and a workflows sandbox email blocked event with reason paused record the skip.
• The worker checks the pause first, and again after the daily cap claim. When the second check blocks, it returns the claim.
• A tenant Django has never synced counts as open. SES enforces its own state either way.
• The worker drops its cached state on reload-sandbox-sender-state.

How did you test this code?

Live run on the slice tip

• Live run of the V3 tip 7e95863f7a0 on a PostHog preview. Stubs: a local SES endpoint that records each call, and example.com seeded as a valid MX domain; a local SNS signing certificate, so a signed delivery webhook verifies; the trigger event produced straight onto the CDP events topic; personhog pointed at the persons database.

• SES tenant DISABLED, then the Django sync: the mirror row reads DISABLED, and the next sandbox send is skipped with "the sandbox sender is paused right now". ENABLED and a sync: the next send goes out. The worker picked up each change without a restart.

• A live workflow send on the sandbox sender recorded email_sent and email_sandbox_sent once each, the pair the sending tier subtracts.

• A signed SES Delivery notification for that message on the sandbox configuration set returned 200 and captured workflows sandbox email delivered. The same envelope with a forged signature returned 403.

• Lane gate on HEAD: tsc passes, and the pre-push hogli ci:preflight --strict passes. The lane classifier puts this PR on the Node side only, and it has no migration.

• Red: on feat(workflows): mirror the sandbox ses tenant state in django #607's code, 6 pause gate cases of email.service.test.ts fail.

• Green on HEAD: email.service.test.ts, sandbox-email-sender.test.ts, rate-limiter.service.test.ts and sandbox-sender-state.service.test.ts, 255 passed.

• Test seams:

  • EmailService.executeSendEmail with a sandbox integration and a stubbed sender state: skip, run log, event, and the returned claim.
  • SandboxSenderStateService against the test Postgres and the reload message.

• Proofs owed (fleet proof gate):

  • Review: GPT-6.1 Sol, 1 round, no findings.
  • Live run: passed at the stack tip (69afe3e4b1f), recorded on every delivery head. One real email worker process (cdp-cyclotron-worker-email) on scratch Postgres, with real Redis, Kafka and ClickHouse: a member received the email, a non-member was skipped, a paused tenant was skipped, the daily team cap held, and each delivered send recorded email_sandbox_sent, which the tier's own ClickHouse read then left out. Stubbed: the SES endpoint (a local HTTP stub behind the real SESv2 client), the MX verdict for the reserved example.com domain, and the tenant-state rows the Django sync would write.
  • Merge-queue lanes: CI computes this check against master, so it reads as mixed until the Node layers below land. Land the stack bottom-first, not with one trunk merge on the top layer.
  • Prod shape: gap, accepted until read-only production access arrives. The sandbox tenant is not configured in production.
  • CI: gap. Fork CI cannot run on Silthus/posthog.

Test rationale: the pause gate block drives real sends, because the regressions that matter are a send that goes out through a paused tenant, or a claim kept after the pause blocked delivery.

Release status

• This change is behind a feature flag and is not available to users

Docs update

docs/internal/workflows-email-sending-tiers.md documents the pause gate.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (implementation). Review by Codex, GPT-6.1 Sol.

Split out of #605 and #606 so each PR passes upstream's merge rules: a migration does not share a PR with nodejs/, and Python does not share a PR with Node. The code is carved by file from those reviewed heads without changes; the stack's tip has the same tree as #606.
Skills: tdd, /writing-tests, pr-description, merge-risk.
Public artifact: all test data is invented.

Decisions

Question	Answer	Who	Where
Slice order of the sandbox sender stack	Option B: the Channels row moves into V1 so a team has a sender to test from; V1's worker PRs land first	👍 Accepted default	map
How to split #605 and #606	Four fresh single-commit branches off #604, in deploy order: Django mirror, worker gate, Python tier, worker metric; #605 and #606 close	🤖 Agent	ticket
How a mixed-lane stack lands	Bottom-first; CI checks lanes against master, so each layer passes once the layers below it land	🤖 Agent	ticket
Where the docs lines go	Each half documents its own behavior; the tip's doc matches #606	🤖 Agent	ticket
Pause check order	Before the member check, and again after the cap claim	🤖 Agent	ticket
Involvement: no grilling rounds; design pass carried from #605 and #606 (the code is unchanged); review (GPT-6.1 Sol, 1 round, no findings); proof captured as test output and a live run.

🤖 Generated with Claude Code