feat(webhook): scan event notifications with a durable outbox - #1706
Open
manuc66 wants to merge 12 commits into
Open
feat(webhook): scan event notifications with a durable outbox#1706manuc66 wants to merge 12 commits into
manuc66 wants to merge 12 commits into
Conversation
manuc66
force-pushed
the
split/webhook-outbox
branch
from
September 5, 2026 05:24
4b5ae0e to
e8abaf3
Compare
Every completed scan POSTs a JSON event (scan-completed, or scan-delivery-failed when a delivery target failed) to a configured URL: - enriched scan metadata (job info, duplex mode, color depth, instance id/uptime) and one descriptor per file (name, size, SHA-256 and where to fetch it: local, S3 bucket/key or Nextcloud WebDAV URL) - HMAC-SHA256 signing (custom header), bearer/basic auth - Idempotency-Key header and a durable outbox with atomic writes, retry with backoff (408/429/5xx), permanent-4xx dead-letter and a flush at startup and after each scan - redirects (3xx) are never followed and are retried instead of silently losing the event (maxRedirects: 0) Configured via --webhook-* CLI options, webhook_* config keys or WEBHOOK_* Docker env vars. Contract documented in protocol_doc/webhook/openapi.yaml. S3 file-location helpers are reused from the S3 delivery PR.
manuc66
force-pushed
the
split/webhook-outbox
branch
from
September 5, 2026 05:28
e8abaf3 to
8f00548
Compare
The webhook delivery tests (sendScanEvent, flushOutbox, dead-letter, retry, redirect handling, outbox restart survival) were missing from the PR branch. Restore them from the combined branch — they pass against the shipped webhook module.
Outbox files were created with the process umask (typically 0644), world-readable on shared hosts even though they embed scan metadata and auth tokens. atomicallyWrite now writes them with 0600.
The webhook file descriptor contentType was only present when the device reported it, leaving PDFs and locally generated images without a type. A small extension-based fallback fills it in.
An exception in sendScanEvent (disk full, unwritable outbox dir) used to propagate out of postProcessing and abort the scan metadata logging and the file cleanup, even though every delivery target had succeeded. The webhook notification is now wrapped so a notification failure never breaks the scan pipeline.
Consumers could only approximate the scan duration from event.time minus startedAt, which also covered uploads and retries. The emitted metadata now carries the actual endedAt and durationMs (already computed for the log output).
The outbox (atomic writes, backoff retry, dead-letter) was always on, turning a simple notification into a mini message-queue. It is now opt-in via --webhook-durable-outbox / webhook_durable_outbox / WEBHOOK_DURABLE_OUTBOX (default: off). Default behavior is best-effort: the event is POSTed once (same auth, idempotency-key and redirect handling) and failures are logged. With the flag, the existing outbox persists undelivered events and retries them at startup and after each scan. The HTTP POST itself was extracted into sendEventOnce shared by both paths.
The pages field was required in OpenAPI but always sent as an empty array. It now carries one descriptor per scanned page (pageNumber, format, width, height, resolution, sizeBytes) — useful for document analysis when the merged PDF is a single file and especially for ADF scans whose pages are not all the same size. The local path is not exposed.
Only fields an automated N8N consumer can act on are published, each justified by a use case (document analysis or scan context): - metadata.settings: inputSource, contentType, format, mode, resolution, isDuplex, duplex assembly context, pageCountingStrategy, paperSize (drop colorDepth/channels, sourceFormat, width/height, filePattern, paperDim, paperOrientation) - metadata.target: label + resourceURI only (drop transient destinationURI/compEventURI/agingStamp) - metadata.job: state + count only (drop jobs[].uri/uuid) - metadata.instance: id + startedAt (drop uptimeMs) - files[].path: only present when store is local (never leak the host path for S3/Nextcloud files) pages[] carries one descriptor per scanned page (pageNumber, format, dimensions, resolution, sizeBytes) without the local path, which is what makes the field useful for ADF scans of varying page sizes. The OpenAPI contract is updated to match.
The webhook PR is now the mergeable baseline: a single best-effort POST per event (auth, idempotency-key, no redirect following), nothing persisted, failures logged. The durable outbox (retry, dead-letter, restart survival) moved to a dedicated follow-up PR behind --webhook-durable-outbox; this PR keeps the simple path only.
These scan settings have consumer value: - sourceFormat tells the consumer the format requested to the device (compression may have happened before the delivered format). - colorDepth/channels give an explicit quality signal for OCR without mapping mode by hand. - paperDim gives the real physical size when paperSize is 'Max'. filePattern is dropped: it is a template to parse, not metadata; the filename and the fields that produce it (scanCount, startedAt) are already in the event.
manuc66
force-pushed
the
split/webhook-outbox
branch
from
September 5, 2026 06:50
9f3cb81 to
a493f10
Compare
Owner
Author
Future integration:
|
When the PDF merge fails (e.g. a scan produced no pages), the delivery array only contained the pdf failure, so a consumer could not tell a skipped target from a non-configured one. Configured paperless / nextcloud / s3 targets are now published with an explicit skipped status.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
There was no way to be notified when a scan completed, or to report the outcome of each delivery target to an automation (n8n/Zapier/...).
Solution
A webhook event posted after every completed scan (best-effort: a single POST, failures are logged):
scan-completed/scan-delivery-failedevent types, with the scan context a consumer can act on: settings (inputSource, format, sourceFormat, mode, colorDepth/channels, resolution, isDuplex, duplex assembly, page counting, paperSize/paperDim), target label + resourceURI, device, instance,endedAt/durationMs, delivery outcomes, and one descriptor per scanned page (pages[]: number, format, dimensions, resolution, size).name,size,sha256,contentType, plus where to fetch it: local path, S3bucket/keyor Nextcloud WebDAV URL). The localpathis only exposed when the file is stored locally.Idempotency-Keyheader.maxRedirects: 0).Configured via
--webhook-*CLI options,webhook_*config keys orWEBHOOK_*Docker env vars. Contract documented inprotocol_doc/webhook/openapi.yaml. Reuses the S3 file-location helpers (S3 PR).A follow-up PR (
#1707) adds the durable outbox (retry + dead-letter) behind an opt-in flag.Review checklist
Already fixed (verify):
maxRedirects: 0and3xxclassified as retry (not followed as an empty GET).logger.serializeErrorstripsconfig/requestfrom axios errors.files[].pathonly exposed when the file is stored locally.pages[]populated with per-page descriptors (no local path).settingscarriessourceFormat,colorDepth/channels,paperDim(quality/compression/physical-size context);filePatterndropped (a template to parse, not metadata).Contract / payload (open follow-ups):
files[](only the soon-deleted source images are listed).deliverywhen the PDF merge fails — configured paperless/nextcloud/s3 are now published with an explicit skipped status.contentTypefallback and MIME inference unified with S3 (contentTypeForExtension).scan-completedwithfiles: [].Observability / security / config:
singleScanCmdreports any delivery failure as "paperless/nextcloud" — make the message reflect the actual failed targets./healthzignores the webhook/outbox state — report degraded.logScanMetadata(SonarCloud gate), validate the payload against the OpenAPI contract.--webhook-urlis plain HTTP withauth: none.delivery[].errorcarries raw error strings — sanitize them.Merge after the processing-pipeline PR and the S3 delivery PR; the durable outbox PR (#1707) merges after this one.
Testing
test/webhook.test.ts: delivery, idempotency-key, signed payload, bearer/basic auth, content-type inference, pages,endedAt/durationMs, redirect non-following, best-effort persistence-free delivery.test/webhook-config.test.ts: auth inference and credential guard.698 passing), README help snapshots regenerated.