feat(migrate): robust, type-exact, verifiable, streaming export/import (rebased from @alekcz) - #922
Open
whilo wants to merge 6 commits into
Open
feat(migrate): robust, type-exact, verifiable, streaming export/import (rebased from @alekcz)#922whilo wants to merge 6 commits into
whilo wants to merge 6 commits into
Conversation
Rework datahike.migrate from a flat CBOR EAVT snapshot into a full-history, type-exact, verifiable, bounded-memory dump/restore that also targets external stores and diskless containers. Codec & correctness - #633: values encode by runtime CLASS via EDN tagged literals, so :db.type/double never round-trips back as Float; float-array/double-array/bytes/bigint/bigdec/ symbol/uuid/instant/tuple are class-exact. Closed EDN reader (no read-string/eval). - #377 full history; #262 schema-before-data ordering; #508/#531 attribute-refs translate-not-insert; #287 max-tx owned by load-entities. Scale & memory - External merge sort (bounded fan-in) on export; streaming tx-aligned import. Memory bounded by :sort-buffer/:chunk-size/:batch-size, not db size — validated: a 1.2 GB store exported to a 285 MB dump and re-imported under a 144 MB heap. - estimate-import-memory reports the -Xmx to set (id-remap map is O(entities)); import-db warns on preflight and echoes :recommended-heap. Targets - Filesystem (chunked dir or flat file) OR a konserve store (S3 / S3-compatible via konserve-s3, JDBC, Redis, mem) — no new hard dependency. manifest-last commit + per-chunk SHA-256 (object-store safe). - :sort? false no-scratch streaming export for hard read-only / diskless containers. Verification, safety, compatibility - Per-chunk SHA-256 + order-independent semantic digest; verify + finalize-import!; config/emptiness/format guards; closed reader; path validation; owner-only perms. - Legacy 2-arity still reads old CBOR dumps. Tests: 17 tests / 63 assertions, green on persistent-set + hitchhiker-tree and under spec instrumentation. Docs: doc/import-export-design.md, doc/backup.md, dev/migrate_scale.clj. CHANGELOG left to upstream.
Wanderung is no longer a factor and the target branch is incidental — the one real design fork is EDN-lines vs CBOR (both plug into the same codec seam). Design doc §13 and the walkthrough updated accordingly.
A `:db.type/store-ref` datom holds a hasch content id, which is both the value
in the datom and the key the bytes live under. Exporting the datom exported the
REFERENCE and not the referent: the dump restored perfectly and produced datoms
naming objects that were not in the target store. A backup that loses its blobs
is not a backup.
Blobs are now enumerated with `gc/reachable-store-refs` — the mark the GC
already computes across branches and through retained history — copied into the
dump under `store-refs/<content-id>`, and restored with `k/bassoc` BEFORE any
datom that names them. That ordering is the constraint the konserve-sync walker
already meets by shipping blobs ahead of the branch head, and the one this dump
meets by writing the manifest last: a reference must never exist without its
referent.
Naming each blob by its content id makes the FILE NAME the checksum, so
verification is recomputing `blob-id`, identical content deduplicates for free,
and no side table maps names to hashes.
WHAT CANNOT BE ASSUMED: datahike does not always have the bytes. A store-ref
says *what* an object is, never *where* it lives — bytes in a raw bucket a
browser PUT to with a presigned URL never transit this JVM, by design. So the
plan splits the live set into `:carried` and `:external`, the manifest records
both plus `:self-contained?`, and import REFUSES a dump with externals unless
the caller passes `:accept-external-blobs? true`. The restored database would
name objects the import did not place; that has to be a decision rather than a
later failing read. This keeps the division of labour the GC already states:
datahike owns the mark, the operator owns the copy.
Content is verified against its own id on the way in — writing unverified bytes
under a content-addressed key would leave something every later reader trusts.
Two constraints found while wiring it:
* `reachable-store-refs` walks index ADDRESSES, so it needs a flushed index
and raises on an unflushed in-memory db. Plenty of legitimate exports are of
such a db (`db-with`, a `:memory` store), and those cannot hold in-store
blobs anyway — so the walk is gated on the schema actually declaring a
store-ref attribute. That also makes the common case pay nothing.
* `k/bget`'s callback IS the scope in which the stream handle is valid, so on
an async store it must synchronously return a CHANNEL. Returning the bytes
directly makes `bget` hand back a byte array where the caller expects
something to take from. A broad `catch → nil` around that misreported every
blob as external and would have shipped a dump carrying none of them — the
catch is gone, and nil now means only "not in this store".
A flat single-file dump has nowhere to put blobs, so exporting a blob-bearing
database to one raises rather than dropping them.
`.cljc` and hasch-based throughout, so the format code stays portable.
Tests: 24 tests / 92 assertions in migrate-test (his 20 + 4), full clj-pss
866 tests / 7221 assertions / 0 failures. Verified end to end: before this the
"blob bytes restored" assertion was false while every datom restored fine.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
A version stamp is the right check for a STORE: `connector/version-check`
refuses one written by a newer datahike because the on-disk index is not forward
compatible. A dump is different — it is logical, so a v3 dump that happens to
use no v3-only feature IS readable by v2, and refusing it on the stamp would work
against datahike's commitment to backwards compatibility. Accepting one whose
features cannot be represented is worse: it drops data silently.
So the dump now carries both, for different jobs:
* `:datahike/meta` — `datahike.tools/meta-data`, the same shape the store
carries, so a dump and a store can be reasoned about with one vocabulary.
Provenance and diagnostics.
* `:requires` — the capability set needed to INTERPRET the dump. This is what
the accept/reject decision reads.
`check-capabilities!` names exactly what is missing: "this dump requires
:db.type/double-array" is actionable where "written by a newer version" is not,
and capabilities we do support in the same list pass without comment. A dump with
no `:requires` predates the declaration and is read as before.
The part that keeps this honest over time: value-type capabilities are DERIVED,
not hand-listed. `:requires` comes from the schema's declared types and the
supported set from `ds/builtin-value-types`, so a value type added in a future
version appears in a dump automatically and an older reader refuses it by
construction — rather than by someone remembering to update a table. That is what
makes DOWNGRADE well defined instead of aspirational.
Feature capabilities are derived the same way from facts already at hand:
history, carried blobs, external blobs, attribute-refs.
Tests: 26 tests / 104 assertions in migrate-test, full clj-pss 868 tests /
7233 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
`verify` checked chunk checksums, counts, a semantic digest and a sampled
structural diff — and said nothing about the blobs the dump declares. So a dump
whose `store-refs/` was short reported `:ok? true`: counts matched, the digest
matched, every tier passed, and the dump was unrestorable. That is precisely the
reassurance nobody should get from a backup tool.
The blob check now feeds `:ok?` rather than sitting beside it. Measured on a dump
with one blob deleted and the datoms untouched:
[ok? tier1 tier2 blobs] = [false true true false]
Counts and digest still match; :ok? is false because a referent is missing.
Because a blob's FILE NAME is its content hash, verification needs nothing from
the manifest beyond the id, and it catches a torn or tampered object — right
name, wrong bytes — which a count never could.
`:external` ids are counted, never checked: those bytes were never ours to carry.
They make a dump not self-contained, which the manifest states and which import
refuses without an explicit opt-in; placing them is the operator's half of the
contract.
A dump with no `:store-refs` section — no blobs, or predating blob carriage —
verifies as before.
Tests: 27 tests / 116 assertions in migrate-test, full clj-pss 869 tests /
7245 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
…anguage evidence
The codec choice was the one question the design left open, argued on size and
speed. Neither is the deciding factor: export/import is IO-bound, and the
property that matters is that the scenario a dump exists for is the one where
datahike is unavailable or not trusted. So the question is whether a foreign
reader can produce native values — and that is measurable rather than arguable.
Measured with clj-cbor 1.1.1, then those exact bytes read with Python cbor2:
bigdec -> Decimal('1.50') scale intact
instant -> datetime(2026,1,1, tz=utc) native
uuid -> UUID('...0001') native
bytes -> b'\x00\x01\x7f\xff' native
bigint -> 123456789012345678901234567890 exact
That is the argument for CBOR that compactness does not make. The EDN encoding
routes every non-trivial type through a `#datahike/*` tag: portable in principle,
Clojure-only in practice.
It also corrects a premise in the design note. §5.3 says EDN tags are needed
because "CBOR's float encoding" narrows a double. True of SHORTEST-FORM encoding
— which RFC 8949's deterministic profile prescribes — but it is an encoder
policy, not a property of CBOR. clj-cbor encodes by class, so `(double 2.0)`
stays f64 (`fb 4000000000000000`) even though 2.0 fits f32 exactly. #633 is
fixable with CBOR today.
ONE MEASURED GAP, narrow and now pinned: clj-cbor encodes zero, NaN and
±Infinity as f16 regardless of class, and f16 decodes to Float — so a `Double`
0.0 round-trips as a `Float`. #633 surviving for exactly three values. It does
NOT affect the dump as implemented: a full export/import of `:db.type/double`
0.0, 1.5 and 2.0 restores all three as `Double` (verified end to end). It is the
one thing a move to CBOR must address, with a width-preserving float policy.
Byte-level vectors rather than round-trips, deliberately: a round-trip proves a
library agrees with itself, which is precisely what it cannot tell you about
another language. The octets are checked against the tags IANA registers, so any
conformant decoder must read them — and the vectors become the contract a codec
swap has to satisfy, making the FORMAT the commitment and the LIBRARY an
implementation detail. That is what lets an unproven codec be adopted later on
evidence instead of faith.
Tests: 5 tests / 27 assertions in migrate-codec-test; full clj-pss 874 tests /
7272 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebases alekcz/datahike#5 onto current
mainso it can be reviewed and CI-checked here. All implementation and design work is by @alekcz (Alex Oloo) — his two commits are preserved with original authorship; anything we add lands as separate commits on top withCo-authored-bytrailers.It came out of a real ~98M-datom production migration that the current exporter could not survive.
Why rebase rather than review in place
His PR was opened against his fork's
mainate4e26c68(#905), which is 8 commits behind. That made the diff showquery.cljc+268/−83,lower.cljke+150/−1,execute.cljc,relation.cljcand four query test files — all of which is already-merged upstream work (#908–#917), carried along as artifact. Reviewing it linearly would have wasted real effort, and merging as-is risked reverting recent query-engine fixes.Against current
mainhis actual change is 12 files, +2880/−360, zero query-engine files.Verified on this branch, against current main
clj-pss: 862 tests, 7207 assertions, 0 failuresbb formatcleanWhat it does
Reworks
datahike.migratefrom the flat CBOR snapshot — marked "temporary solution, pending developments in Wanderung" in the source — into a full-history, type-exact, verifiable, bounded-memory dump & restore targeting local disk, any konserve store (S3 / S3-compatible / JDBC), or a diskless container.:db.type/doubleno longer round-trips asFloat; values encode by runtime class, every builtin type incl.float-array/double-array/tuplehistory/as-ofmatch at everytBounded memory at any scale (external merge sort out, streaming tx-aligned import in — validated 1.2 GB store → 285 MB dump → re-imported and verified under a 144 MB heap), tiered verification,
estimate-import-memory, and a:sort? falsezero-scratch mode.Read
doc/import-export-design.md§2 firstIts §2 grounds a prior spec against live source and invalidates several load-bearing parts of it. Two are worth calling out because they are the kind of thing a fresh implementation would get wrong:
transaction.cljc:524), and a retract matches an exact[e a v]and is a no-op if that value was already replaced (:426,upsert?at:1511). Retract-before-add — the intuitive choice — therefore silently drops a same-tx replacement's retraction.:migrationid-map is not durable (writing.cljc:56-58,227-281), so it lives only on the in-memory db value. Honest recovery is recreate-and-restart, andimport-dbrefuses a non-empty target. Export is resumable, because chunks are content-addressed.It also correctly drops two things that prior spec demanded — a
read-stringRCE fix (the importer usesclj-cbor; no live vuln) and a core change for forward refs (transact-entities-directlyalready allocates eids attransaction.cljc:1499-1500).Known gaps, to be addressed as separate commits here
store-ref.store-refis covered as a value type (rides as#uuid) but the referent bytes are not carried, so restoring a database that uses blobs yields datoms pointing at objects absent from the target store.gc/reachable-store-refsis the enumeration hook, andgc/record-store-refsalready solved the identical ordering constraint for the konserve-sync walker (blobs must arrive before the reference that names them — the same discipline as manifest-last).datahike.tools/meta-data(the mapconnector/version-checkalready enforces for stores) plus a:requirescapability set. A bare version is too blunt for a dump: a v3 dump using no v3-only features is readable by v2, and refusing it would work against datahike's backwards-compat stance. Capabilities make downgrade well-defined — fail precisely, never silently drop.store-refs/.Where this sits in the migration story
Four axes, and this is the one that was missing:
pg_dump/COPYNone substitutes for another for structural reasons: konserve-sync ships the index format so it cannot cross an incompatible version; a logical dump re-derives ids so it cannot give cheap incremental replication.