Skip to content

feat(workflows): skip sandbox sends while the shared ses tenant is paused - #605

Closed
Silthus wants to merge 2 commits into
workflows-sandbox-delivery/daily-capsfrom
workflows-sandbox-delivery/tenant-pause
Closed

Silthus wants to merge 2 commits into
workflows-sandbox-delivery/daily-capsfrom
workflows-sandbox-delivery/tenant-pause

Conversation

@Silthus

@Silthus Silthus commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Note

Replaced by #607 (Django mirror) and #608 (worker pause gate). This PR mixed lanes that upstream's merge rules keep apart, so it was split in two. Same code, carved by file.

Problem

All sandbox sends go out through one shared PostHog SES tenant.
When AWS pauses that tenant, for example over its reputation, the worker keeps trying to send through it.
This PR makes every sandbox send skip while the shared tenant is paused, or while its state cannot be read.

Part of Silthus/personal#199 (sandbox delivery, slice 2 of 3 there). Stacked on #604.
Replaces #452 and #453.

Impact

Grow Workflows, activation rate: the pause gate is a condition for enabling the sandbox sender in production.
No production team sees this yet: WORKFLOWS_SANDBOX_SENDER_ENABLED stays off.

Merge Danger

Warning

🟡 Merge risk: medium · one-way door (merge-risk rubric)

  • New table workflows_sandboxsendertenantstate (migration 0039). It is a plain CREATE TABLE with no foreign keys, so the migration takes no lock on an existing table.
  • The SES tenant-state webhook and the daily reconciliation now also sync the tenant named by SES_SANDBOX_TENANT_NAME. With the setting empty, they do nothing new.
  • The team tenant sync moved its reputation lookup into get_tenant_sending_state. Team tenant tests pin the same results.
  • Only sends through a sandbox integration read the new state. A sandbox send fails closed when the read fails.

Production lines: 292 changed (git diff --numstat against #604; tests, docs and the migration excluded).

Changes

  • Every sandbox send, test sends included, skips while the shared sandbox tenant is DISABLED, or when the worker cannot read its state. The run log and a workflows sandbox email blocked event with reason paused record the skip.
  • The worker checks the pause first, and again after the daily cap claim. When the second check blocks, it returns the claim.
  • A tenant Django has never synced counts as open. SES enforces its own state either way.
  • Django mirrors the sandbox tenant into SandboxSenderTenantState:
    • from the SES tenant-state webhook, through a Celery task;
    • from the daily reconciliation, beside the team tenants. A failed sandbox sync does not stop the team sweep.
  • A committed state change sends reload-sandbox-sender-state to the workers, which drop their cached state.
  • The sandbox tenant never emails project admins. It belongs to no team.
  • SES_SANDBOX_TENANT_NAME is a new Django setting. Set the same value on Django web, Celery, and the worker.

How did you test this code?

  • Lane gate on HEAD: tsc passes. eslint, oxlint, prettier and ruff pass on the changed files. makemigrations --check reports no changes, and sqlmigrate workflows 0039 shows one CREATE TABLE. semgrep --config .semgrep/rules/security/ reports nothing on the lines this PR adds.
  • Red: on feat(workflows): cap sandbox sends per project and recipient each day #604's code, sandbox-sender-state.service.test.ts, the pause gate block of email.service.test.ts, and test_ses_tenant_state.py cannot load, because the state service and model do not exist.
  • Green on HEAD: email.service.test.ts, sandbox-email-sender.test.ts, rate-limiter.service.test.ts, sandbox-sender-state.service.test.ts, and the Django suites test_ses_tenant_state.py, test_ses_tenant_events.py, test_email_sending_tier.py and test_email_reputation.py.
  • Test seams:
    • EmailService.executeSendEmail with a sandbox integration and a stubbed sender state: skip, run log, event, and the returned claim.
    • SandboxSenderStateService against the test Postgres and the reload message.
    • sync_sandbox_tenant_state, the tenant-state webhook, and reconcile_ses_tenant_states, with the SES client mocked.
  • Proofs owed (fleet proof gate):
    • Review: GPT-6.1 Sol, 1 round, no findings.
    • Live run: gap. Not run within the lane's time cap; owed before the sandbox switch goes on.
    • Prod shape: gap, accepted until read-only production access arrives. The sandbox tenant is not configured in production.
    • CI: gap. Fork CI cannot run on Silthus/posthog.

Test rationale: the pause gate block drives real sends, because the regressions that matter are a send that goes out through a paused tenant, or a claim kept after the pause blocked delivery. The Django tests cover the webhook and sweep paths that write the mirror, including a failed sandbox sync that must not stop the team sweep.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

docs/internal/workflows-email-sending-tiers.md documents the pause gate, the mirror, and SES_SANDBOX_TENANT_NAME.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (implementation). Design pass and review by Codex, GPT-6.1 Sol.

Ported from the reviewed source PRs above onto the caps PR as a wayfinder ticket lane. Skills: tdd, /writing-tests, /django-migrations, pr-description, merge-risk, qa-swarm design pass.
No duplicate: this replaces the source fork PRs named above, which stay open until the stack exists.
Public artifact: all test data is invented.

Decisions

Question Answer Who Where
Where SES_SANDBOX_TENANT_NAME lives This PR adds the one setting line the Django mirror needs; whichever of this PR and the sandbox product PR lands second keeps the product PR's settings block 🤖 Agent ticket
Migration number 0039, after master's 0038 🤖 Agent ticket
Pause check order Before the member check, and again after the cap claim 🤖 Agent ticket

Involvement: no grilling rounds; design pass (1 lane, 1 minor finding fixed: the reload uses transaction.on_commit(..., robust=True)); review (GPT-6.1 Sol, 1 round, no findings); proof captured as test output.

🤖 Generated with Claude Code

Silthus and others added 2 commits October 27, 2026 07:28
…used

Django mirrors the shared sandbox SES tenant state from the tenant-state
webhook and the daily reconciliation. The worker skips every sandbox
send while that tenant is paused, and checks again after the daily
allowance claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant