Skip to content

feat(file-upload): add S3-compatible BlobStore backend (#370) - #906

Draft
stepandra wants to merge 11 commits into
neohaskell:mainfrom
stepandra:s3/pr1-blobstore-s3
Draft

stepandra wants to merge 11 commits into
neohaskell:mainfrom
stepandra:s3/pr1-blobstore-s3

Conversation

@stepandra

@stepandra stepandra commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Apps running on several machines can keep uploaded files in an S3-compatible object store instead of one machine's disk. Configure Service.FileUpload.BlobStore.S3.createBlobStore with an existing bucket, endpoint, region and credentials; the existing upload, download and deduplication code keeps using the same BlobStore interface. Local storage remains the default, and existing files are not moved.

The backend uses Core HTTP and Crypto helpers from #905. It signs path-style requests with SigV4, requires HTTPS outside literal loopback, does not retry or follow redirects, treats only 404 as missing, and keeps authentication/transport failures as errors. Multipart uploads and bucket creation are outside this PR.

Review follow-up

Verification

Local macOS checks with the pinned toolchain:

  • S3/signing tests: 26 examples, 0 failures. The invalid-label test failed before the validation fix.
  • Full Core suite: 1109 examples, 0 failures, 3 pending.
  • Exact integration probes and combined-report validation: all 15 HTTP/S3 selectors pass; omitted Core evidence fails.
  • Doctests, lint, spec/criterion/signature checks, codemap, expectation guard, workflow checks and whitespace checks pass.

These checks use a local S3-protocol fixture, not a live S3 provider. The full PostgreSQL/Hurl suite was not rerun. Still a draft; no final maintainer approval is claimed.

Migration: none. Spec 017. Depends on #905. Closes #370.

Generalize the existing getSecure/getRaw GET path into one method-agnostic
primitive: raw Bytes body, non-throwing status codes, response headers, same
redirect/proxy/TLS/size defaults. getSecure and getRaw become Get
specializations. Spec: docs/changes/016-http-client-byte-level-methods.md.
…ent Http.Client helpers

Protocol code inside nhcore (the S3 signer in change 017, FileUpload's content
hash) needs the raw digest, a chainable HMAC and hex encoding, and today reaches
for crypton/Data.ByteArray and Bytes.unwrap directly. Per the neohaskell#748 review
("add it to the Bytes/Core module instead of unwrapping"), the primitives now
live in Crypto, which already owns that boundary. signWith is re-expressed on
hmacSha256 + toHex (one implementation; output unchanged, pinned by a test).

Known-answer tests: FIPS 180-4 SHA-256 vectors, RFC 4231 HMAC case 2, hex.
Spec: docs/changes/021-crypto-sha256-hmac-hex.md (touches core-primitives).

Also adds the one-line Haddock the doc-ratchet wants on Http.Client's
existing helpers and regenerates codemap: undocumented_exports 560 -> 545,
undocumented_doctest_modules 315 -> 314.

Gates: ./dev check 0; ./dev test Crypto 24/0; ./dev test Http.Client 37/0;
./dev doctest 0 failures; ./dev lint No hints; ./dev codemap-check OK;
./dev spec-check 021 OK; git diff --check 0.
Local-shaped facade: S3Config + createBlobStore, four BlobStore ops over
Http.Client.sendSecure/sendRaw with hand-written SigV4 (crypton/memory,
already dependencies). Pure SigV4 module tested against the AWS S3 API
reference vectors; facade tested against an in-process WAI fake S3 that
speaks only the S3 wire protocol. Spec: docs/changes/017-s3-blob-store.md.
…elpers

SigV4 now calls Crypto.sha256 / Crypto.hmacSha256 / Crypto.toHex (change 021)
instead of importing crypton and Data.ByteArray and unwrapping Bytes — the
neohaskell#748 review rule. Public API and the AWS test vectors are unchanged.
Adds the one-line Haddock on S3/SigV4 helpers plus two doctests
(isDnsBucketName, sha256Hex) so the doc-ratchet improves; codemap regenerated.

Gates: ./dev check 0; ./dev test Service.FileUpload.BlobStore.S3 23/0;
./dev doctest 0 failures; ./dev lint No hints; ./dev codemap-check OK;
./dev spec-check 017 OK; git diff --check 0.
@coderabbitai

coderabbitai Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@NickSeagull

Copy link
Copy Markdown
Member

Hey, thanks for this PR! Two things to reconcile:

  1. Bucket validation accepts a..b, -ab and ab- despite the stated DNS-style bucket constraint. Check label boundaries and adjacent dots, with a focused rejection example, so invalid names fail before the request.
  2. Change 017 calls the S3 fake tests unit / none, but FakeS3 starts a real HTTP listener. Reclassify the HTTP examples as integration / http:real and attest the exact selectors; keep pure signer/config cases at unit / none.

Thanks!

…tion

ClientRawSpec's sendRaw and getRaw status examples send real loopback HTTP
to an in-process Warp server, so they are integration / http:real with
exact selector attestations in test-surfaces.json. The scheme check,
methodName and the closed-port network error stay unit / none.

Amp-Thread-ID: https://ampcode.com/threads/T-01a11c53-f9bd-70df-81b8-4369dbfd3e1b
isDnsBucketName only checked length and characters, so a..b, -ab, ab-,
a.-b and a-.b were accepted. Every dot-separated label must now be
non-empty and start and end with a letter or digit; createBlobStore
rejects such names before any store exists to send a request.

Amp-Thread-ID: https://ampcode.com/threads/T-01a11c53-f9bd-70df-81b8-4369dbfd3e1b
…egration

S3Spec's store, retrieve, exists, delete, status-mapping and key examples
send real loopback HTTP to the FakeS3 Warp listener, so they are
integration / http:real with exact selector attestations. Signer and
configuration cases, including the new bucket-label checks, and the
closed-port unreachable case stay unit / none.

Amp-Thread-ID: https://ampcode.com/threads/T-01a11c53-f9bd-70df-81b8-4369dbfd3e1b
… so 021 selectors resolve

The C1/C2 locators named "abc" and "Jefe" in quotes, but the hspec
source escapes them, so the exact-title selector never matched. Rename the
two example titles and their locators; every assertion is unchanged.

Amp-Thread-ID: https://ampcode.com/threads/T-01a11c53-f9bd-70df-81b8-4369dbfd3e1b
Integration criteria may name nhcore-test-core locators (change 016), but
the --criteria-runtime --report gate in test-service read only the service
report, so every core locator failed report mode. test-core now tees its
direct output to nhcore-test-core.report under pipefail and uploads it
fail-closed; test-service needs test-core, downloads that artifact and
gates on the concatenation of both reports. workflow-check contract 6
freezes the wiring with self-test regressions.

Amp-Thread-ID: https://ampcode.com/threads/T-01a11c53-f9bd-70df-81b8-4369dbfd3e1b

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: S3-compatible backend for file uploads (BlobStore)

2 participants