feat(s3): S3-compatible upload target for scans - #1705
Open
manuc66 wants to merge 3 commits into
Open
Conversation
manuc66
force-pushed
the
split/s3-delivery
branch
from
September 5, 2026 05:24
f1b39e3 to
71a8d12
Compare
Adds an S3 delivery target (AWS S3, MinIO, Cloudflare R2, Wasabi...): - SigV4 request signing with no new dependency (src/s3/s3.ts) - path-style and virtual-host addressing, bucket prefix, optional STS session token (src/s3/S3Config.ts) - CLI options (--s3-*), config file keys (s3_*), Docker env (S3_*) - upload of individual images or merged PDFs wired into postProcessing Also fixes, verified by tests: - Windows \ separators in --s3-prefix normalized to / - empty --s3-session-token treated as absent (no empty header) - IP endpoints force path-style addressing automatically - S3_FORCE_PATH_STYLE=false / =0 no longer enables the flag in Docker Ships test/s3.test.ts and README/CHANGELOG entries.
manuc66
force-pushed
the
split/s3-delivery
branch
from
September 5, 2026 05:28
71a8d12 to
c3ee272
Compare
A partial S3 configuration (--s3-url without bucket or credentials) was ignored silently: no warning, no error, uploads never happened, and the webhook still reported scan-completed without the s3 delivery. getS3Config now throws with the list of missing options as soon as any S3 option is provided but the configuration is incomplete (mirroring the webhook auth guard).
The independent signature re-computation in test/s3.test.ts only covered host/x-amz-content-sha256/x-amz-date. It now also folds in x-amz-security-token when present, and a dedicated case uploads with an STS session token to prove the signed-headers list matches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Scans could only be delivered to Paperless-ngx and Nextcloud. Users on AWS S3, MinIO, Cloudflare R2 or Wasabi had no way to store scans in object storage.
Solution
A new S3 delivery target:
node:crypto(no new dependency).postProcessingalongside Paperless/Nextcloud.--s3-*CLI options,s3_*config-file keys orS3_*Docker env vars.Included robustness fixes (each with a failing-test-first commit):
\separators in--s3-prefixnormalized to/(Windows).--s3-session-tokentreated as absent (nox-amz-security-token: "").scans.192.168.1.100would fail DNS).S3_FORCE_PATH_STYLE=false/=0no longer enables the flag in Docker.Review checklist
Already fixed in this PR (verify):
\separators in--s3-prefixnormalized to/(buildObjectKey).--s3-session-tokentreated as absent (no emptyx-amz-security-tokenheader, no signed-header entry).scans.192.168.1.100would fail DNS).S3_FORCE_PATH_STYLE=false/=0no longer enables the flag in Docker.Follow-ups to realize in this PR:
--s3-urlwithout bucket or key) is ignored silently —getS3Confignow throws with the list of missing options.--s3-region auto; the defaultus-east-1produces confusing SigV4 errors. Add a per-provider README example.test/s3.test.ts's independent signature re-computation to cover the STS path (x-amz-security-token).contentTypeForExtension) with the webhook descriptor (shared module once the webhook PR lands).Themes to reflect on:
keepFilesis global across targets; per-target granularity is a follow-up in the webhook PR.Testing
test/s3.test.ts: SigV4 signature re-verified server-side by an independent recomputation; path-style + virtual-host, images + PDF, error handling, prefix normalization, empty session token, IP endpoints.683 passing), README help snapshots regenerated.Merge order
Merge after the processing-pipeline PR (foundation). The webhook/outbox PR follows and depends on the S3 file-location helpers.