Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,14 @@ STORAGE_SECRET_KEY=astroid-secret
# Queue (BullMQ) — shares Redis by default
QUEUE_PREFIX=astroid
QUEUE_CONCURRENCY=5
WEBHOOK_DELIVERY_MAX_ATTEMPTS=5
WEBHOOK_RETRY_DELAY_MS=2000
WEBHOOK_HTTP_TIMEOUT_MS=5000
WEBHOOK_FAILED_JOB_RETENTION_SECONDS=86400
WEBHOOK_DEAD_LETTER_RETENTION_SECONDS=604800
# Optional; required to accept signed POST /api/v1/webhooks/receive requests.
# Generate and store a unique secret (at least 32 characters) in a secret manager.
# WEBHOOK_INGRESS_SECRET=

# Rate limiting (@nestjs/throttler - per-tier steady-state limits per TTL window)
# Counters are stored in Redis so every API replica enforces the same budget.
Expand Down
35 changes: 35 additions & 0 deletions API_DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,41 @@ export function verifyAstroidWebhook({ secret, headers, rawBody }) {
}
```

### Delivery retries and failures

Outgoing deliveries are asynchronous and provide **at-least-once**, not
exactly-once, semantics. The event ID remains stable across retries; consumers
should use it as an idempotency key. By default, BullMQ makes five total
attempts with exponential backoff starting at 2 seconds and 20% jitter. Operators
can configure the attempt limit, base delay, HTTP timeout, and failed-job
retention with `WEBHOOK_DELIVERY_MAX_ATTEMPTS`, `WEBHOOK_RETRY_DELAY_MS`,
`WEBHOOK_HTTP_TIMEOUT_MS`, and `WEBHOOK_FAILED_JOB_RETENTION_SECONDS`.

Network errors, timeouts, HTTP 408, HTTP 429, and 5xx responses are retried.
Other 3xx/4xx responses are treated as permanent and stop immediately. Redirects
are not followed. Exhausted jobs are routed to the shared dead-letter queue and
recorded in the webhook delivery audit trail. Failed BullMQ jobs are retained
for 24 hours by default; the dead-letter queue retains failed records for seven
days by default. `WEBHOOK_DEAD_LETTER_RETENTION_SECONDS` configures that
retention. Operators can inspect and re-drive dead-letter jobs through the
existing queue administration tools.

### Inbound signature verification

`POST /{API_PREFIX}/webhooks/receive` requires both
`x-astroid-timestamp` and `x-astroid-signature`. The inbound signature is the
lowercase 64-character hex HMAC-SHA256 digest of UTF-8
`<timestamp>.` concatenated with the exact raw request body bytes. The request
timestamp must be within five minutes of server time. Verification uses a
constant-time comparison; missing, malformed, stale, or invalid signatures are
rejected before the ingress handler runs.

The route uses the dedicated `WEBHOOK_INGRESS_SECRET`, which must contain at
least 32 characters. It is separate from each outbound subscription's
`whsec_...` secret; when unset, inbound requests are rejected. Deployments
should provide it through a secret manager. Nest captures raw request bytes
before JSON parsing so whitespace and key order are preserved for verification.

## Health Probes (`/health`)

The liveness and readiness probes are served **outside** the API prefix, so
Expand Down
23 changes: 19 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,10 +135,25 @@ are rejected:

### Queues (BullMQ)

| Variable | Default | Description |
| ------------------- | --------- | ----------------------------- |
| `QUEUE_PREFIX` | `astroid` | Key prefix for BullMQ queues. |
| `QUEUE_CONCURRENCY` | `5` | Default worker concurrency. |
| Variable | Default | Description |
| ----------------------------------------- | ------- | ----------- |
| `QUEUE_PREFIX` | `astroid` | Key prefix for BullMQ queues. |
| `QUEUE_CONCURRENCY` | `5` | Default worker concurrency. |
| `WEBHOOK_DELIVERY_MAX_ATTEMPTS` | `5` | Maximum webhook delivery attempts, including the initial delivery; bounded to 25. |
| `WEBHOOK_RETRY_DELAY_MS` | `2000` | Initial exponential retry delay in milliseconds; bounded to 300000. |
| `WEBHOOK_HTTP_TIMEOUT_MS` | `5000` | Outbound request timeout in milliseconds; bounded to 120000. |
| `WEBHOOK_FAILED_JOB_RETENTION_SECONDS` | `86400` | How long BullMQ retains failed webhook jobs before cleanup; bounded to 30 days. |
| `WEBHOOK_DEAD_LETTER_RETENTION_SECONDS` | `604800` | How long the dead-letter queue retains webhook failures; bounded to 30 days. |

Webhook retries use BullMQ exponential backoff with 20% jitter. Permanent
redirect and 4xx responses stop retrying; HTTP 408, 429, 5xx, and network or
timeout failures are retried up to the configured attempt limit.

### Webhook ingress

| Variable | Default | Description |
| -------- | ------- | ----------- |
| `WEBHOOK_INGRESS_SECRET` | _(unset)_ | Dedicated secret for inbound callbacks. Must be at least 32 characters when set. Requests are rejected while unset. Never reuse an outbound subscriber secret. |

### Rate limiting

Expand Down
7 changes: 6 additions & 1 deletion src/app.module.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { MiddlewareConsumer, Module, NestModule } from '@nestjs/common';
import { MiddlewareConsumer, Module, NestModule, RequestMethod } from '@nestjs/common';
import { APP_FILTER, APP_GUARD, APP_INTERCEPTOR } from '@nestjs/core';
import { ConfigService } from '@nestjs/config';
import { ThrottlerModule } from '@nestjs/throttler';
Expand All @@ -16,6 +16,7 @@ import { REDIS_CLIENT } from './common/locks/locks.constants';
import { EncryptionModule } from './common/encryption/encryption.module';
import { RequestIdMiddleware } from './middleware/request-id.middleware';
import { StructuredRequestLoggingMiddleware } from './middleware/structured-request-logging.middleware';
import { RawBodyMiddleware } from './common/middleware/raw-body.middleware';
import { REQUEST_ID_HEADER } from './common/constants/headers';

import { JwtAuthGuard } from './common/guards/jwt-auth.guard';
Expand Down Expand Up @@ -166,5 +167,9 @@ export class AppModule implements NestModule {
configure(consumer: MiddlewareConsumer): void {
consumer.apply(RequestIdMiddleware, StructuredRequestLoggingMiddleware).forRoutes('*');
consumer.apply(RequestMetricsMiddleware).forRoutes('*');
consumer.apply(RawBodyMiddleware).forRoutes({
path: 'webhooks/receive',
method: RequestMethod.POST,
});
}
}
Loading
Loading