feat(encryptor): :aes-gcm authenticated encryption; seal/unseal for raw bytes

[whilo]

Why

The :aes encryptor is unauthenticated AES-256-CBC (geheimnis v1, which its own README now marks deprecated and not safe for new use). A tampered blob is undetectable: it either decrypts to garbage that fressian then parses, or throws a padding error whose path is padding-oracle-shaped. For a content-addressed store like datahike that means a modified blob silently becomes a tree node.

Important

Blocked on replikativ/geheimnis#8. deps.edn points at geheimnis 0.2.36, which is that PR. CI will fail until it is merged and released to Clojars.

What

:aes-gcm (header byte 2)

AES-256-GCM via geheimnis v2. Per blob part: [salt 32B][nonce 12B][ct ‖ tag 16B].

• A fresh CSPRNG salt on every write derives a per-blob key (HKDF-SHA-256), so no key ever encrypts more than one message and GCM's one catastrophic failure mode — nonce reuse — cannot arise, whatever the store's write volume.
• The AAD binds each ciphertext to (layout version, store-key, :meta|:value). A blob cannot be relocated under another key, and a meta blob cannot be swapped into the value slot — both things an attacker with write access to the storage medium can otherwise do without touching a single ciphertext byte.
• Byte-identical on JVM and ClojureScript (geheimnis carries interop KATs), so a store written by one platform reads on the other.
• The key must be 256 bits — 32 raw bytes or a 64-char hex string; konserve.encryptor/generate-key makes one. konserve deliberately does not stretch passphrases: a guessable one would be brute-forced offline against the stored blobs, and geheimnis has no password KDF yet.
• On CLJS this requires the async API — Web Crypto has no synchronous cipher, so {:sync? true} is an explicit error rather than a hang. The JVM supports both tiers.

Overhead is ~12 µs per 1 KiB blob on the JVM (CSPRNG salt+nonce → HKDF → GCM), and 60 bytes on disk — less than the current 64-byte salt plus CBC padding.

PEncryptor

Encryption becomes its own protocol (-encrypt / -decrypt over whole byte arrays) rather than a PStoreSerializer decorator. An AEAD cipher must verify its tag over the complete ciphertext before any plaintext reaches the deserializer, so there was nothing to stream — and Web Crypto is Promise-only, so the sync decorator shape could not have survived regardless. Ops return a channel, or the byte array directly under {:sync? true}.

This is the breaking change for anyone with a custom encryptor: implement PEncryptor, not PStoreSerializer.

Binary values

bassoc/bget hand your bytes to storage untouched, bypassing the serializer path — so they were never encrypted, and a store configured with an encryptor wrote them in the clear, silently. That is not what a user of an encrypted store expects. They now require an explicit choice:

;; encrypted under the store's key, and bound to :thumb
(k/bassoc store :thumb (<? (k/seal store :thumb raw-bytes)) {:raw? true})
(k/bget store :thumb
        (fn [{is :input-stream}] (go (<? (k/unseal store :thumb (slurp-bytes is)))))
        {:raw? true})

;; or: I own this format and its confidentiality
(k/bassoc store :blob my-ciphertext {:raw? true})

New konserve.core/seal / unseal encrypt raw bytes under the store's own key and bind them to a konserve key, so a sealed binary blob is as tamper-evident and as un-relocatable as an EDN value. Encrypting binary by default needs chunked AEAD framing to keep bassoc streaming; :raw? keeps its meaning when that lands.

:aes — deprecated, still works

Existing blobs stay readable and writable, byte-for-byte, pinned by a golden vector generated from the pre-refactor code.

Bugs found along the way

• :lz4 + any encryptor was unreadable. Writes nested the encryptor outside the compressor (compress → encrypt) while reads nested them the other way (defaults.cljc:70 vs :150), so a read tried to LZ4-decompress the ciphertext. Nobody hit it because the default compressor is the null one. The protocol split makes the order symmetric by construction.
• The :aes salt is not from a CSPRNG. It is (edn-hash (uuid)), and hasch's random-uuid is Math.random on CLJS — so the per-blob IV and derived key are predictable in the browser today.
• :aes is not actually cross-platform. The JVM and JS salt encodings disagree (inc on one side, signed/unsigned reads on the other), so a blob written on one platform fails to decrypt on the other whenever a salt byte lands on 128 — roughly one blob in five. Preserved rather than fixed, since fixing it would break the stores it currently works for. :aes-gcm has no such problem.

Not addressed

-read-binary already materializes the whole payload on every backend (JVM allocates a ByteBuffer of the full value size and hands locked-cb a ByteArrayInputStream over it — there is a ;; TODO use FileInputStream to not load the file in memory sitting right there; Node reads a full Buffer; IndexedDB hands over the whole blob). Only -write-binary genuinely streams. So framed AEAD and that TODO are the same piece of work, and belong together in a follow-up.

Also worth recording: do not wrap GCM in javax.crypto.CipherInputStream when that follow-up happens. It has historically swallowed AEADBadTagException and returned EOF (JDK-8154523), converting a detected forgery into a silent truncation — strictly worse than no authentication, because the caller believes it verified.

Testing

• JVM: 79 tests, 1318 assertions, 0 failures — :aes-gcm compliance (sync + async), wrong-key rejection, byte-flip tamper rejection against a real filestore blob, a scan of every blob on disk confirming a marker string never lands there in the clear, seal/unseal round-trip and cross-key rejection, {:raw? true} pass-through, and the legacy :aes golden vector.
• Node: 47 tests, 319 assertions, 0 failures — :aes-gcm over Web Crypto.
• Browser build compiles clean; cljfmt and clj-kondo clean (no new warnings).