feat: S3 delivery target and webhook events with durable outbox - #1697
Open
manuc66 wants to merge 29 commits into
Open
feat: S3 delivery target and webhook events with durable outbox#1697manuc66 wants to merge 29 commits into
manuc66 wants to merge 29 commits into
Conversation
- S3 transport (SigV4-signed PUT via built-in crypto, no new dependency)
for images and PDFs: path-style or virtual-host addressing (MinIO,
Cloudflare R2, Wasabi), optional prefix and STS session token
- webhook events ('scan-completed' / 'scan-delivery-failed') carrying
the enriched scan metadata and per-file size/sha256, with an
Idempotency-Key header and optional HMAC-SHA256 signature
- durable outbox (~/.node-hp-scan-to/outbox) with atomic writes, retry
with backoff, 4xx-408/429 handling, dead-letter, and a startup flush
so pending events survive restarts
- richer per-scan metadata: device jobs (state/uri/uuid), real duplex
mode (incl. emulated duplex), color depth/channels, instance id and
uptime, per-page timing and device content-type
Verified with unit tests (SigV4 re-verified server-side, outbox retry/
dead-letter/restart, README help blocks); full suite: 627 passing (4
pre-existing Windows PathHelper failures unrelated).
…et-file - --webhook-auth (none|hmac|bearer|basic), inferred from the configured credentials when not set explicitly - --webhook-auth-header to override the HMAC signature header name (default x-webhook-signature, matches n8n's configurable HMAC auth) - --webhook-secret-file (parity with paperless/nextcloud) and --webhook-token / --webhook-username / --webhook-password for Authorization: Bearer / Basic - config-file equivalents (webhook_auth, webhook_auth_header, ...)
…istence - root/app.sh: new S3_* and WEBHOOK_* environment variables mapped to the corresponding CLI flags (secret-file variants take precedence) - README: document that the webhook outbox lives on the container layer (survives docker restart, lost on container re-creation) and how to persist it with a mounted volume + WEBHOOK_OUTBOX_DIR; add S3/webhook rows to the Docker env table and to the compose example
- files[] now carry store ('local' | 's3' | 'nextcloud') and a location
(S3 bucket/key or Nextcloud WebDAV URL) computed from the configured
targets, alongside the existing size/sha256
- keeps the local path, but the event is a metadata notification: the
receiver can fetch the object with its own credentials (no presigned
URLs in events) or use the local path while it exists
- export s3ObjectLocation() and nextcloudWebdavFileUrl() helpers with
unit tests; README clarifies the semantics of the local path when
--keep-files is disabled
…less/n8n - docker-compose.test.yml starts MinIO (bucket 'scans'), Nextcloud (WebDAV), Paperless-ngx and n8n with fixed test credentials - scripts/real-services-pump.ts pushes a real sample PDF through the actual upload/download modules (S3/WebDAV/multipart/webhook) - scripts/real-services-test.sh resolves the services, runs the pump and verifies each object on the real service (mc stat, WebDAV PROPFIND, paperless API, n8n executions); --scanner adds a full single-scan with all targets against a physical printer - the stub keeps proving self-consistency for CI; this harness proves real-world compatibility (SigV4 vs MinIO, WebDAV, paperless API, n8n receipt). n8n HMAC auth on the node stays a documented manual step.
…in-memory FIFO queue The listen and adf-autoscan loops now drop each completed scan onto a single background drain instead of awaiting uploads/PDF merge/POST-processing inline, so the loop keeps polling the printer while a scan is being processed. This is the execution-model seam: the in-memory array is exactly where a durable inbox will land next. single-scan keeps its synchronous contract (the process exits only once delivery completed, success or not) by routing through processScanProcessingJob.
…s PDF merge jspdf works synchronously in-process, so a large merge starves the event loop and freezes health checks and printer HTTP responses while it runs. The queue moved that work off the capture loop, but the CPU itself still blocks. Letting the loop breathe between pages (and before the final PDF serialization) keeps pending requests handled during multi-page merges.
- nextcloud.test.ts: convertToPdf writes the PDF next to its source page and jspdf embeds a timestamp, so the PDF test overwrote the tracked test/asset/nextcloud_sample.pdf on every run (different bytes each time). Generate it in a temp dir instead. - bmp/ppm conversion tests regenerate their input_*.raw and output_*.bmp fixtures in test/tmp/ on every run (the "too_small" error path even truncates too_small.bmp), dirtying tracked files. Untrack test/tmp/ and ignore it: it is a self-contained scratch directory.
The PDF generator already makes the output reproducible (createPdfFrom pins jsPDF's /ID via setFileId + setCreationDate from the scan date), so this tracked test/asset/nextcloud_sample.pdf simply drifted from the current generation and appeared "modified" on every test run. It is never read by any test, so remove the dead fixture. (The nextcloud PDF test already generates into a temp dir.)
jspdf works synchronously in-process, so merging a large scan on the main thread starves the event loop and freezes health checks and printer HTTP responses for the whole merge. createPdfFrom now hands the page metadata to a dedicated worker thread (src/pdfMergeWorker.ts) that reads the pages, builds the jsPDF document and writes the output; the main thread just waits for the result. This supersedes the previous between-pages setImmediate yields: the CPU is no longer on the event loop at all. The worker is loaded as .ts under tsx (dev, inheriting the loader via execArgv) and as the compiled .js in dist under plain node.
… on a subpage The processing-pipeline behavior (capture kept responsive, FIFO delivery queue, jspdf merge in a worker thread, single-scan still synchronous, durability not in this version and conditional on network destinations when it lands) is documented on a standalone GitHub Pages subpage under docs/, instead of growing the already long README.
Owner
Author
|
Openapi doc missing |
Resolve the postProcessing.test.ts overlap (kept the paperless/nextcloud delivery and S3/webhook event location suites), fix type errors surfaced by the merge in loggerOutput and scanJobHandlers flow tests, and give the bun-compiled logger test a generous compile timeout.
…h credentials - S3 object URLs kept the endpoint path but dropped the separator when the endpoint had a path prefix (e.g. https://gw.example/s3), uploading to the wrong key; the join is now explicit while the encoded key stays untouched. - Webhook auth inference now treats a lone --webhook-username or --webhook-password as basic auth so the existing assertion rejects the incomplete pair instead of silently sending events unauthenticated; blank signing secrets (option or file) are treated as not configured. - Add the post_processing_sample.jpg test asset referenced by postProcessing.test.ts, and drop the dead id return of enqueueEvent.
- Prefer optional chain for the blank webhook secret check. - Replace the empty setTimeout callback in the logger test entry with a top-level awaited promise (no-empty-function). - Drop an unnecessary type assertion in loggerOutput.test.ts.
A Windows user passing --s3-prefix "2026\\08" (or the equivalent config key) produced S3 object keys with literal backslashes (2026\\08/scan.pdf), invalid on S3. The prefix is now normalized to forward slashes before building the key.
An empty S3_SESSION_TOKEN (or --s3-session-token "") used to emit an x-amz-security-token: "" header and include it in the SigV4 signed headers, which AWS S3 rejects with InvalidSecurityToken. Blank tokens are now ignored for both signing and the request headers.
… event axios/follow-redirects downgrades a POST to an empty GET when following a 301/302. With the default redirect-following, a webhook endpoint that redirected (http->https, trailing slash, proxy) acknowledged an empty request as success and the outbox removed the event: silent data loss. deliverEvent now sets maxRedirects: 0 and classifies 3xx as retry (with permanent 4xx still dead-lettered).
Virtual-host addressing prefixes the bucket to the endpoint hostname, so an IP endpoint without --s3-force-path-style produced a bogus scans.192.168.1.100 host that failed DNS (ENOTFOUND). IP (IPv4 and IPv6) endpoints now always use path-style addressing.
Existing setups without WEBHOOK_URL need no extra volume. A mounted outbox is recommended only for new webhook usage so retries survive container re-creation.
Brings the FIFO processing queue, worker-thread PDF merge and staged post-processing pipeline (PR #1699) into the S3/webhook/outbox branch. # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two new delivery targets and a full scan metadata enrichment:
PUTvia built-incrypto, no new dependency) for images and PDFs. Supports path-style and virtual-host addressing (AWS S3, MinIO, Cloudflare R2, Wasabi...), optional prefix and STS session token. CLI:--s3-url/--s3-region/--s3-access-key-id/--s3-secret-access-key(-file)/--s3-bucket/--s3-prefix/--s3-force-path-style/--s3-session-token+ config-file equivalents.scan-completed/scan-delivery-failed) with:name/size/sha256/format/contentType,Idempotency-Keyheader + optional HMAC-SHA256 signature (X-Webhook-Signature),~/.node-hp-scan-to/outbox, overridable) with atomic writes, retry with backoff, correct 408/429 retry vs permanent 4xx dead-letter, and a flush on startup so pending events survive restarts.Testing
test/s3.test.ts: SigV4 signature is re-verified server-side by an independent recomputation; path-style + virtual-host, images + PDF, error handling.test/webhook.test.ts: delivery, signed payload, 4xx dead-letter, 429/5xx retry with the same idempotency key, max-attempts cap, outbox survival across a "restart".PathHelperissues unrelated to this change.typecheck/lintclean.Notes
keepFiles=false, retried events reference local paths that may have been cleaned up; sizes/hashes are embedded so the payload remains valid.