Skip to content

feat(workflows): cap sandbox sends per project and recipient each day - #604

Draft
Silthus wants to merge 2 commits into
workflows-sandbox-delivery/routefrom
workflows-sandbox-delivery/daily-caps
Draft

Silthus wants to merge 2 commits into
workflows-sandbox-delivery/routefrom
workflows-sandbox-delivery/daily-caps

Conversation

@Silthus

@Silthus Silthus commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Problem

The sandbox sender lets a team without a verified domain send workflow email to its own members through one shared PostHog SES identity.
Without a daily limit, one project could send unlimited mail through that identity, or one member could receive a flood of it.
This PR caps sandbox sends per project and per recipient address each day, so a send over either cap skips instead of going out.

Part of Silthus/personal#199 (sandbox delivery, slice 1 of 3 there). Stacked on #599.
Replaces #441.

Stack

Impact

Grow Workflows, activation rate: the caps are a condition for enabling the sandbox sender in production.
No production team sees this yet: WORKFLOWS_SANDBOX_SENDER_ENABLED stays off.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

  • Only sends through a sandbox integration claim the caps. Own-domain sends do not touch the new code.
  • The caps fail closed: a Valkey error, or an empty or invalid cap setting, skips every sandbox send.
  • With the switch on, the CDP API opens the SES Valkey pool (SES_RATE_LIMITER_VALKEY_*) for test sends. That pool is non-fatal and bounds its waits, so a Valkey outage skips sandbox test sends instead of stopping the CDP API. With the switch off, it opens nothing new.
  • New Lua scripts in RateLimiterService (claimAllOrNothing, returnClaim). Existing callers do not use them.

Production lines: 382 changed (git diff --numstat against #599, tests and docs excluded).

Changes

Right before delivery, every sandbox send, test sends included, claims two daily allowances at once:

Allowance Setting (default) Charged
Per project WORKFLOWS_SANDBOX_DAILY_TEAM_CAP (25) One token per recipient copy
Per recipient address, across the organization WORKFLOWS_SANDBOX_DAILY_RECIPIENT_CAP (5) One token per distinct address in the send
  • Both allowances are token buckets that refill over 24 hours. A send claims both or neither, in one Valkey script.
  • A reached allowance skips the send with a run log line and a workflows sandbox email blocked event with reason cap_reached.
  • The worker checks members again after the claim. When that check skips the send, it gives the claim back.
  • An SES throttle response gives the claim back before the delivery retry.
  • Returns are deduplicated. Each claim carries its own deadline, and Valkey ignores a return that arrives more than five minutes after the claim.
  • Recipient keys hash the lowercased address with SHA-256 under the organization's hash tag, so the keys hold no address.

How did you test this code?

Live run on the slice tip

  • Live run of the V1 tip 1c08de0b9b4 on a PostHog preview. Stubs: a local SES endpoint that records each call, and example.com seeded as a valid MX domain.

  • Channels created the sandbox sender. A test send to a verified member was accepted and reached SES with the sandbox tenant, its configuration set, the fixed From address and the footer.

  • A test send to an address outside the organization was skipped before SES.

  • Daily cap, on the V3 preview: five test sends to one member were accepted; the sixth and seventh were skipped with the per-address limit message, and SES got five calls.

  • Lane gate on HEAD: tsc passes. eslint, oxlint and prettier pass on the changed files. semgrep --config .semgrep/rules/security/ reports nothing on the lines this PR adds.

  • Red: 22 new tests fail on feat(workflows): route sandbox email through its fixed ses identity #599's code. For the review fixes, the unreachable-pool test on the old pool stopped the jest process with SIGTERM. Green: email.service.test.ts, rate-limiter.service.test.ts and sandbox-email-sender.test.ts pass on HEAD.

  • Test seams:

    • EmailService.executeSendEmail with a sandbox integration, against the test Postgres, a real Redis limiter, and a stubbed SES client: send or skip, run log, events, and the claim left in Redis.
    • RateLimiterService.claimAllOrNothing and returnClaim against real Redis: replay, a return after the claim's window, and an unreachable Valkey through the CDP API's optional pool.
  • Proofs owed (fleet proof gate):

    • Review: GPT-6.1 Sol, round 1 with 2 findings, both fixed in d68a209; the final read found nothing.
    • Live run: gap. Not run within the lane's time cap; the tests drive sends against real Postgres and Redis with the SES client stubbed. Owed before the sandbox switch goes on.
    • Prod shape: gap, accepted until read-only production access arrives. No sandbox integration exists in production.
    • CI: gap. Fork CI cannot run on Silthus/posthog.

Test rationale: the daily caps block drives real sends through executeSendEmail with a real Redis limiter, because the regressions that matter are a send over a cap going out, a repeated address charged twice, or a claim kept after the send skipped. The limiter tests cover the replay and late-return guards, which no send-level test can reach.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

docs/internal/workflows-email-sending-tiers.md documents the two allowances, their settings, and the fail-closed behavior.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (implementation). Design pass and review by Codex, GPT-6.1 Sol.

Ported from the reviewed source PR above onto the route PR as a wayfinder ticket lane. Skills: tdd, /writing-tests, pr-description, merge-risk, qa-swarm design pass.
No duplicate: this replaces the source fork PR named above, which stays open until the stack exists.
Public artifact: all test data is invented (example.com addresses).

Decisions

Question Answer Who Where
Slice order of the sandbox sender stack Option B: the Channels row moves into V1 so a team has a sender to test from; V1's worker PRs land first 👍 Accepted default map
Organization for the recipient allowance The team's organization from the team cache, the same source as the footer 🤖 Agent ticket
When to claim Right before delivery, then recheck members and return the claim if the recheck skips 🤖 Agent ticket
CDP API pool on Valkey errors Non-fatal with bounded waits, so an outage skips sandbox sends; the email worker keeps its fatal pool 🤖 Agent ticket
Claim return window Starts at the claim, carried in the grant 🤖 Agent ticket
Packed log line for blocked addresses Left as a follow-up to keep this PR under the size budget 🤖 Agent ticket

Involvement: no grilling rounds; design pass (1 lane, no findings); review (GPT-6.1 Sol, 1 round with 2 findings fixed; final read clean); proof captured as test output.

🤖 Generated with Claude Code

Silthus and others added 2 commits October 28, 2026 09:28
Sandbox sends claim a daily allowance per project and per recipient
address across the organization right before delivery. A reached
allowance skips the send. SES throttles give the claim back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CDP API's SES Valkey pool no longer stops the process on repeated
errors, and it bounds connection and command waits, so an outage skips
sandbox sends instead. A claim now carries its own return deadline, so
an old claim cannot refund into a refilled bucket.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant