Skip to content

feat(webhook): durable outbox with retry and dead-letter (opt-in) - #1707

Open
manuc66 wants to merge 2 commits into
split/webhook-outboxfrom
split/webhook-outbox-durable
Open

feat(webhook): durable outbox with retry and dead-letter (opt-in)#1707
manuc66 wants to merge 2 commits into
split/webhook-outboxfrom
split/webhook-outbox-durable

Conversation

@manuc66

@manuc66 manuc66 commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Problem

The base webhook PR delivers events best-effort: a single POST, and a briefly unreachable receiver (n8n down, 429, timeout) loses the event.

Solution

An opt-in durable outbox behind --webhook-durable-outbox / webhook_durable_outbox / WEBHOOK_DURABLE_OUTBOX (default: off, keeping the simple best-effort path):

  • Events are written to the outbox (atomic write, mode 0600) before delivery.
  • Retried at startup and after each scan with backoff on 408/429/5xx and redirects.
  • Permanent 4xx dead-lettered to <id>.failed.json instead of being lost.
  • Pending events survive restarts (same idempotency-key on retries).
  • The HTTP POST is shared with the best-effort path (sendEventOnce).

Stacked on the webhook PR (split/webhook-outbox): merge that one first, then this one.

Review checklist

Already fixed (verify):

  • Redirects are never followed: a 301/302 would downgrade the POST to an empty GET and acknowledge a lost event (maxRedirects: 0, 3xx → retry).

Outbox reliability:

  • Retried outbox entries reference a deleted local path when keepFiles: false — mark it unavailable (path: null).
  • Dead-letter files accumulate and a corrupt entry re-logs forever — add TTL cleanup and quarantine.
  • Outbox delivery order is not guaranteed across retries — sort by creation time and document.
  • No atomic claim on outbox entries — two processes sharing the dir can double-deliver.
  • Every scan re-attempts the whole backlog (O(N)) — cap attempts per flush or move the flush off the scan path.
  • The 10s HTTP timeout is not configurable — slow-but-working receivers get dead-lettered.
  • No SIGTERM/SIGINT handling — add graceful shutdown and .tmp cleanup.

Testing

  • test/webhook.test.ts (outbox suite): retry with the same idempotency key, 4xx dead-letter, 429/5xx retry, max-attempts cap, restart survival, mode 0600, redirect handling.
  • Full suite green (704 passing).

Follow-up to the base webhook PR (simple best-effort POST). With
--webhook-durable-outbox / webhook_durable_outbox / WEBHOOK_DURABLE_OUTBOX
(default: off), events are written to an outbox first and retried at
startup and after each scan (backoff on 408/429/5xx), with permanent 4xx
dead-lettered to <id>.failed.json instead of being lost. Entries are
written with mode 0600 and survive restarts. The HTTP POST is shared
with the best-effort path (sendEventOnce).
…ble branch

# Conflicts:
#	test/postProcessing.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant