CBOR wire for the kabel peers, and a robust streaming export/import - #930
Open
whilo wants to merge 112 commits into
Open
CBOR wire for the kabel peers, and a robust streaming export/import#930whilo wants to merge 112 commits into
whilo wants to merge 112 commits into
Conversation
Rework datahike.migrate from a flat CBOR EAVT snapshot into a full-history, type-exact, verifiable, bounded-memory dump/restore that also targets external stores and diskless containers. Codec & correctness - #633: values encode by runtime CLASS via EDN tagged literals, so :db.type/double never round-trips back as Float; float-array/double-array/bytes/bigint/bigdec/ symbol/uuid/instant/tuple are class-exact. Closed EDN reader (no read-string/eval). - #377 full history; #262 schema-before-data ordering; #508/#531 attribute-refs translate-not-insert; #287 max-tx owned by load-entities. Scale & memory - External merge sort (bounded fan-in) on export; streaming tx-aligned import. Memory bounded by :sort-buffer/:chunk-size/:batch-size, not db size — validated: a 1.2 GB store exported to a 285 MB dump and re-imported under a 144 MB heap. - estimate-import-memory reports the -Xmx to set (id-remap map is O(entities)); import-db warns on preflight and echoes :recommended-heap. Targets - Filesystem (chunked dir or flat file) OR a konserve store (S3 / S3-compatible via konserve-s3, JDBC, Redis, mem) — no new hard dependency. manifest-last commit + per-chunk SHA-256 (object-store safe). - :sort? false no-scratch streaming export for hard read-only / diskless containers. Verification, safety, compatibility - Per-chunk SHA-256 + order-independent semantic digest; verify + finalize-import!; config/emptiness/format guards; closed reader; path validation; owner-only perms. - Legacy 2-arity still reads old CBOR dumps. Tests: 17 tests / 63 assertions, green on persistent-set + hitchhiker-tree and under spec instrumentation. Docs: doc/import-export-design.md, doc/backup.md, dev/migrate_scale.clj. CHANGELOG left to upstream.
Wanderung is no longer a factor and the target branch is incidental — the one real design fork is EDN-lines vs CBOR (both plug into the same codec seam). Design doc §13 and the walkthrough updated accordingly.
A `:db.type/store-ref` datom holds a hasch content id, which is both the value
in the datom and the key the bytes live under. Exporting the datom exported the
REFERENCE and not the referent: the dump restored perfectly and produced datoms
naming objects that were not in the target store. A backup that loses its blobs
is not a backup.
Blobs are now enumerated with `gc/reachable-store-refs` — the mark the GC
already computes across branches and through retained history — copied into the
dump under `store-refs/<content-id>`, and restored with `k/bassoc` BEFORE any
datom that names them. That ordering is the constraint the konserve-sync walker
already meets by shipping blobs ahead of the branch head, and the one this dump
meets by writing the manifest last: a reference must never exist without its
referent.
Naming each blob by its content id makes the FILE NAME the checksum, so
verification is recomputing `blob-id`, identical content deduplicates for free,
and no side table maps names to hashes.
WHAT CANNOT BE ASSUMED: datahike does not always have the bytes. A store-ref
says *what* an object is, never *where* it lives — bytes in a raw bucket a
browser PUT to with a presigned URL never transit this JVM, by design. So the
plan splits the live set into `:carried` and `:external`, the manifest records
both plus `:self-contained?`, and import REFUSES a dump with externals unless
the caller passes `:accept-external-blobs? true`. The restored database would
name objects the import did not place; that has to be a decision rather than a
later failing read. This keeps the division of labour the GC already states:
datahike owns the mark, the operator owns the copy.
Content is verified against its own id on the way in — writing unverified bytes
under a content-addressed key would leave something every later reader trusts.
Two constraints found while wiring it:
* `reachable-store-refs` walks index ADDRESSES, so it needs a flushed index
and raises on an unflushed in-memory db. Plenty of legitimate exports are of
such a db (`db-with`, a `:memory` store), and those cannot hold in-store
blobs anyway — so the walk is gated on the schema actually declaring a
store-ref attribute. That also makes the common case pay nothing.
* `k/bget`'s callback IS the scope in which the stream handle is valid, so on
an async store it must synchronously return a CHANNEL. Returning the bytes
directly makes `bget` hand back a byte array where the caller expects
something to take from. A broad `catch → nil` around that misreported every
blob as external and would have shipped a dump carrying none of them — the
catch is gone, and nil now means only "not in this store".
A flat single-file dump has nowhere to put blobs, so exporting a blob-bearing
database to one raises rather than dropping them.
`.cljc` and hasch-based throughout, so the format code stays portable.
Tests: 24 tests / 92 assertions in migrate-test (his 20 + 4), full clj-pss
866 tests / 7221 assertions / 0 failures. Verified end to end: before this the
"blob bytes restored" assertion was false while every datom restored fine.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
A version stamp is the right check for a STORE: `connector/version-check`
refuses one written by a newer datahike because the on-disk index is not forward
compatible. A dump is different — it is logical, so a v3 dump that happens to
use no v3-only feature IS readable by v2, and refusing it on the stamp would work
against datahike's commitment to backwards compatibility. Accepting one whose
features cannot be represented is worse: it drops data silently.
So the dump now carries both, for different jobs:
* `:datahike/meta` — `datahike.tools/meta-data`, the same shape the store
carries, so a dump and a store can be reasoned about with one vocabulary.
Provenance and diagnostics.
* `:requires` — the capability set needed to INTERPRET the dump. This is what
the accept/reject decision reads.
`check-capabilities!` names exactly what is missing: "this dump requires
:db.type/double-array" is actionable where "written by a newer version" is not,
and capabilities we do support in the same list pass without comment. A dump with
no `:requires` predates the declaration and is read as before.
The part that keeps this honest over time: value-type capabilities are DERIVED,
not hand-listed. `:requires` comes from the schema's declared types and the
supported set from `ds/builtin-value-types`, so a value type added in a future
version appears in a dump automatically and an older reader refuses it by
construction — rather than by someone remembering to update a table. That is what
makes DOWNGRADE well defined instead of aspirational.
Feature capabilities are derived the same way from facts already at hand:
history, carried blobs, external blobs, attribute-refs.
Tests: 26 tests / 104 assertions in migrate-test, full clj-pss 868 tests /
7233 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
`verify` checked chunk checksums, counts, a semantic digest and a sampled
structural diff — and said nothing about the blobs the dump declares. So a dump
whose `store-refs/` was short reported `:ok? true`: counts matched, the digest
matched, every tier passed, and the dump was unrestorable. That is precisely the
reassurance nobody should get from a backup tool.
The blob check now feeds `:ok?` rather than sitting beside it. Measured on a dump
with one blob deleted and the datoms untouched:
[ok? tier1 tier2 blobs] = [false true true false]
Counts and digest still match; :ok? is false because a referent is missing.
Because a blob's FILE NAME is its content hash, verification needs nothing from
the manifest beyond the id, and it catches a torn or tampered object — right
name, wrong bytes — which a count never could.
`:external` ids are counted, never checked: those bytes were never ours to carry.
They make a dump not self-contained, which the manifest states and which import
refuses without an explicit opt-in; placing them is the operator's half of the
contract.
A dump with no `:store-refs` section — no blobs, or predating blob carriage —
verifies as before.
Tests: 27 tests / 116 assertions in migrate-test, full clj-pss 869 tests /
7245 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
…anguage evidence
The codec choice was the one question the design left open, argued on size and
speed. Neither is the deciding factor: export/import is IO-bound, and the
property that matters is that the scenario a dump exists for is the one where
datahike is unavailable or not trusted. So the question is whether a foreign
reader can produce native values — and that is measurable rather than arguable.
Measured with clj-cbor 1.1.1, then those exact bytes read with Python cbor2:
bigdec -> Decimal('1.50') scale intact
instant -> datetime(2026,1,1, tz=utc) native
uuid -> UUID('...0001') native
bytes -> b'\x00\x01\x7f\xff' native
bigint -> 123456789012345678901234567890 exact
That is the argument for CBOR that compactness does not make. The EDN encoding
routes every non-trivial type through a `#datahike/*` tag: portable in principle,
Clojure-only in practice.
It also corrects a premise in the design note. §5.3 says EDN tags are needed
because "CBOR's float encoding" narrows a double. True of SHORTEST-FORM encoding
— which RFC 8949's deterministic profile prescribes — but it is an encoder
policy, not a property of CBOR. clj-cbor encodes by class, so `(double 2.0)`
stays f64 (`fb 4000000000000000`) even though 2.0 fits f32 exactly. #633 is
fixable with CBOR today.
ONE MEASURED GAP, narrow and now pinned: clj-cbor encodes zero, NaN and
±Infinity as f16 regardless of class, and f16 decodes to Float — so a `Double`
0.0 round-trips as a `Float`. #633 surviving for exactly three values. It does
NOT affect the dump as implemented: a full export/import of `:db.type/double`
0.0, 1.5 and 2.0 restores all three as `Double` (verified end to end). It is the
one thing a move to CBOR must address, with a width-preserving float policy.
Byte-level vectors rather than round-trips, deliberately: a round-trip proves a
library agrees with itself, which is precisely what it cannot tell you about
another language. The octets are checked against the tags IANA registers, so any
conformant decoder must read them — and the vectors become the contract a codec
swap has to satisfy, making the FORMAT the commitment and the LIBRARY an
implementation detail. That is what lets an unproven codec be adopted later on
evidence instead of faith.
Tests: 5 tests / 27 assertions in migrate-codec-test; full clj-pss 874 tests /
7272 assertions / 0 failures.
Co-authored-by: Alex Oloo <alekcz@gmail.com>
db->stored strips the live index roots, storage handle and schema caches from :db-before / :db-after before a tx-report crosses a connection. That call lived INSIDE the fressian write handler, which made a correct wire representation a property of one codec rather than of this namespace. It is also a hard blocker for any other codec. A TxReport is a defrecord, and a serializer that handles records natively -- as CBOR tag 27 does -- writes the record's raw fields with no opportunity for a write handler to intervene. Under such a codec the stripping would simply never happen and the live state would go out on the wire, with no error anywhere. tx-broadcast/tx-report->wire now does the projection at the two publish sites (tx-broadcast/publish-tx-report! and handlers/global-dispatch-handler), before the value reaches any middleware. This is codec-agnostic and changes nothing downstream: the client already expects a plain map here -- writer/ reconstruct-tx-report and reconstruct-stored-db both branch on stored-map vs live-DB. One behavioural subtlety the tests caught immediately. The fressian handler fired only on a genuine TxReport record, so it never met anything else; projecting at the publish site means we do, and a caller may legitimately pass a plain map (a test stub, a partially-built report). tx-report->wire therefore projects only values for which dbu/db? holds and passes everything else through unchanged, which preserves the previous behaviour exactly. Verified GREEN ON FRESSIAN before any codec change, which is the gate: tx-broadcast-test + writer-test 9 tests / 37 assertions kabel integration-test 6 tests / 41 assertions The fressian DB/TxReport write handlers are now unreachable but deliberately left in place, along with the read handlers, as a separate cleanup.
datahike.boring lives in src/, not src-kabel/, because the same encoding has to serve two consumers -- the kabel wire middleware and konserve's BoringSerializer (byte 3). Two definitions would drift, and for a persisted format that means a store written by one side is misread by the other. Wire content matches datahike.kabel.fressian-handlers value for value: a Datom carries [e a v tx added], a DB carries db->stored, a TxReport carries its map with both DBs projected. Only the framing differs, so a switch is provably content-preserving. Everything rides CBOR tag 27 with the same type-name strings the Fressian module uses, so no private tag number is claimed and a reader with no datahike handlers still decodes to a value that keeps the name and the fields and re-encodes to identical bytes. A Datom is POSITIONAL where a DB is not, and the reason is measured: over 512 datoms a field map costs 54.5 B each against 29.5 for a vector, 1.85x, because the five repeated keys are more than half the payload. Datoms dominate -- an index leaf holds hundreds. A stored DB is one value per message with a dozen genuinely-named fields, so it keeps its map. The cost of positional is that an unregistered reader cannot ask for :e by name, which is why only the one dominant type pays it. The ratio is asserted in the tests, not just recorded here. The vertical test builds a real 1000-entity database, encodes the DB and queries the decoded copy. 1000 rather than a handful on purpose: with a branching factor of 512 a small index is a single leaf, so the branch path would be dead code that looks covered. .gitignore's development-symlink entries are now anchored. Unanchored, `kabel` matched src-kabel/datahike/kabel/ as well, so this commit's boring_handlers.cljc was silently dropped by `git add`. The existing files there only survive because they predate the rule.
A straight replacement, not a dual-format rollout: the kabel wire is a live connection, not a persisted store, so there is no old data to keep readable -- only old peers, and both ends upgrade together. The switch is small because production code never chose a format. The only thing connector.cljc and writer.cljc used from the fressian module was the storage registry, which is neither fressian's nor boring's: it belongs to persistent-sorted-set, is keyed by the store-config :id, and is what a flushed root resolves its own storage by. Those two call sites now name it through datahike.boring and the comments no longer claim it is format-specific. Everything else was peer construction in tests. Evidence the codec is actually engaged, rather than something degrading quietly into a passing suite: swapping the datahike registry for an empty one makes the same 16 tests HANG on the first one, because a DB whose PSS roots cannot be framed never completes a sync. Restored, the suite is 16 tests / 82 assertions green. The dual-format helpers are gone from this namespace -- dead code that documents a rollout we are not doing is worse than no code. kabel.middleware.boring still ships the composition for consumers who cannot upgrade both ends at once. fressian_handlers.cljc is now unreferenced and is removed separately, so that half is trivially revertable on its own.
Unreferenced after the previous commit. Kept as its own commit so the deletion is revertable independently of the switch that made it dead. Also corrects two things in datahike.boring's docstring that the split exposed: it still spoke of "the Fressian module" in the present tense, and it quoted 56.1/31.1 bytes per datom from the exploratory measurement rather than 54.5/29.5 from the one the test actually asserts. A docstring number that does not match the test is worse than no number.
datahike.boring -> datahike.cbor, and datahike.kabel.boring-handlers -> datahike.kabel.cbor-handlers. These tag-27 shapes are precisely what a Rust, Python or JavaScript reader needs in order to read a datahike dump, so naming them after our Clojure implementation would be parochial for the one thing here that is meant to cross languages. org.replikativ/boring is still the library. Also renames test-boring/ -> test-cbor/ and tests-boring.edn -> tests-cbor.edn to match.
doc/distributed.md's server and client examples both required datahike.kabel.fressian-handlers, which was removed when the wire moved to CBOR. Anyone following the guide got a namespace-not-found on the first require -- the failure mode of a document nothing executes. Both examples now use datahike.kabel.cbor-handlers. The CHANGELOG entry states it as breaking for anyone who wired the middleware by hand, and points at kabel.middleware.dual for a deployment that has to roll peers one at a time rather than in a single flag day.
An unregistered reader now gets a clojure.lang.TaggedLiteral for a Datom rather than an UnknownRecord. The docstring said such a reader "cannot ask for :e by name", which understated it: UnknownRecord claimed IPersistentMap over a five-element vector, so keys/assoc/into threw raw JVM exceptions and (:e frame) answered nil silently. A TaggedLiteral never claims map-ness, so those now fail as ordinary "not a map" errors. A DB keeps its map payload and still degrades to an UnknownRecord with every field reachable by name -- the asymmetry the positional trade was made for. Tests updated to boring.data/frame-name and frame-payload, which read either fallback shape without branching.
boring 0.1.5 is on Clojars, and konserve 0.9.364 carries the byte-3 serializer (replikativ/konserve#160, merged). Both were :local/root, which resolves only on a machine with the sibling checkouts. persistent-sorted-set and kabel stay :local/root: their CBOR support is still in open PRs (replikativ/persistent-sorted-set#21, replikativ/kabel#15). This alias folds into :test once both land. Also adds boring to the :test alias, where it belongs and was missing. src-kabel is on the :test path and its connector requires datahike.cbor -- the wire format the kabel peers now speak -- so the whole :test suite fails to LOAD without it, not merely the CBOR tests. That is a pre-existing breakage on this branch, and this fixes the first link: :test now gets past boring and stops at persistent-sorted-set.cbor instead, which needs PSS#21. It cannot be fully green until that lands. The CBOR gate itself passes against the released artifacts: 9 tests, 35 assertions.
0.1.6 adds register-records and cuts decode allocation; no wire change.
Carries the memoised boring read registry (replikativ/konserve#161).
Every dependency this branch was pinning to a working copy is on Clojars now: persistent-sorted-set 0.4.137 -> 0.4.138 (#21, the CBOR node handlers) kabel 0.3.100 -> 0.3.102 (#15, the frame-14 CBOR wire) konserve-sync 0.1.35 -> 0.1.38 konserve 0.9.363 -> 0.9.365 boring 0.1.5 -> 0.1.6 So the :cbor alias loses its :override-deps block entirely and is now just the extra test path. That block was the reason `clojure -M:test` could not even LOAD this branch: src-kabel's connector requires datahike.cbor, which requires persistent-sorted-set's cbor module, which existed only in a working copy. The whole suite failed at require time, not merely the CBOR tests. Verified: datahike.kabel.cbor-handlers, datahike.cbor and datahike.migrate all load under plain `clojure -M:test`. The alias stays separate rather than folding into :test the way konserve-sync's did: tests.edn runs several suites that rebind *default-index*, and the CBOR tests are about the wire codec rather than the index, so running them once per index variant would be noise.
The dump is now a CBOR sequence (RFC 8742) -- one datom per top-level item, no delimiter -- encoded with boring's :archival profile. WHY EDN-LINES EXISTED. Only one reason: clj-cbor narrows zero, NaN and +-Infinity doubles to float16 and reads them back as java.lang.Float, silently changing a :db.type/double value's class (#633). Every #datahike/* tag it carried was a workaround for something EDN cannot express, not a design. boring removes the cause, so the workaround goes with it: #datahike/float -> CBOR f32, natively #datahike/bytes -> major type 2, no base64 wrapper #datahike/farray -> RFC 8746 typed array, natively #datahike/darray -> RFC 8746 typed array, natively #datahike/sysref -> the only one left (tag 27, boring's record form) That is not just tidier. base64 cost 33% before compression AND destroyed the byte-level redundancy zstd would find, and made every binary value opaque to a foreign reader -- which defeats the point of choosing a standard codec. It also deleted the namespace's JVM coupling: the EDN codec needed java.util.Base64, java.nio.ByteBuffer and Float/toString, which is why it could not be .cljc and datahike.migrate.cbor is. WHY :archival AND NOT :canonical. A dump wants reproducible bytes AND host type identity, which RFC 8949's deterministic profile pins together and which are actually separate. :canonical implements 4.2.2 shortest-form floats, so it narrows EVERY Double that fits -- #633 reintroduced, and strictly worse than clj-cbor, which mangles only three values. :archival sorts map keys the same way but keeps :float-policy :preserve-width. Added to boring for this (boring#3). KNOCK-ON. Framing needed no code: consecutive per-record byte arrays ARE a CBOR sequence, so the writer has no delimiter logic and decode-seq-from reads it back bounded by the largest item rather than the file. sort.clj run files became CBOR sequences and now carry RECORDS through the k-way merge instead of strings it re-parsed for every sort key -- simpler and strictly less work. store.clj chunks became konserve BINARIES via bassoc/bget rather than strings, so a chunk hashes identically whether it lands on disk or in S3 instead of being re-encoded by konserve's own serializer. The tier-2 fingerprint hashes the CBOR encoding rather than pr-str, which makes it a function of the values and portable. clj-cbor is gone from the main graph, including cli.clj and codegen/cli.clj. It stays TEST-SCOPE for one reason: legacy dumps were WRITTEN by clj-cbor, so the only honest way to test that boring reads them is to have clj-cbor produce the bytes. Two new tests do exactly that -- 19 value types decoded by both libraries and compared for value AND class, plus the one genuine difference (clj-cbor gives java.time.Instant for tag 1, boring gives java.util.Date) pinned as denoting the same moment. `instance-to-date` in the legacy importer already normalised that before the swap, so it is a no-op rather than a fix. ONE VECTOR MOVED THAT IS NOT A DEFECT: instants are tag 0 (RFC 3339 string) rather than tag 1 (epoch int). Both are registered and DATAHIKE-REQUIREMENTS 2 permits either. It costs ~18% compressed on a transaction-heavy dump, which is being taken up with boring rather than worked around here. migrate 87 tests / 477 assertions, codec vectors 15 / 90, CBOR suite 9 / 35 -- all 0 failures. cljfmt clean.
The alias existed because persistent-sorted-set's cbor module, kabel's frame-14 middleware and konserve's byte-3 serializer were all unreleased, so test-cbor/ needed :override-deps pointing at working copies. All of them are released, the overrides went in 7879624, and what was left was an alias holding one :extra-paths entry plus a second kaocha config invoked by hand. That mattered more than tidiness: CI runs `clojure -M:test` and never passed --config-file tests-cbor.edn, so the CBOR codec suite has not run in CI at any point on this branch. It runs now. Folding it in also GAINED coverage rather than merely relocating it. tests.edn runs three suites over `datahike.test.*` — clj-pss, clj-hht and specs — so these 9 tests become 27, and the ones that build real databases (`a-flushed-db-round-trips-and-is-still-queryable`, `a-db-without-a-registered-store-degrades-to-its-stored-map`) now exercise the hitchhiker-tree index as well as persistent-set. I argued the opposite earlier — that running them per index variant would be noise — which was wrong: the codec carries a DB, so the index it was built with is part of what is being encoded. The suite itself stays. It is codec-level and nothing else covers it: the kabel suite would catch "a peer cannot talk" but not "a Datom is encoded as a field map rather than positionally", which `positional-datoms-are-half-the-size-of-a- field-map` pins as a measurement rather than a commit-message claim. 129 tests / 672 assertions across the migrate, codec-vector and CBOR suites; kabel 16 / 82; cljfmt clean. All under plain `clojure -M:test`.
`transact-entities-directly` builds the entity/tx id mapping incrementally --
an id is allocated the first time it is seen, while datoms are being written.
That is a fold, and it blocks three things:
1. Bulk index build. Building a PSS tree from sorted input needs the FINAL
ids before sorting, because the sort order is over those ids.
2. Resumable import. `import-db` refuses a non-empty target with "import is
not resumable -- recreate and restart", because a partial import leaves
ids allocated and a re-run would allocate different ones.
3. Import into a populated database -- same root cause as (2).
`datahike.migrate.ids/build-mapping` computes the whole mapping in one pass
before any datom is written; `apply-mapping` is then a PURE function of one
record, which is what makes the second pass sortable and parallelisable.
This does NOT reduce memory. `estimate-import-memory` already calls the
O(entities) id map "the dominant, unavoidable term"; the pre-pass relocates
that cost rather than removing it. What it buys is that the map is complete
before writing starts.
Deliberately .cljc and IO-free -- the caller supplies a reducing function over
the records, so it works over a file, a konserve store or a channel. It is the
part of the import path with no platform coupling, so it is where the
portability work starts.
Three id positions matter and missing any one silently corrupts a restore: `e`
(except on a tx-entity datom, where e IS the transaction), `t` always, and `v`
when the attribute is a ref -- a ref value can point FORWARD to an entity whose
own datoms appear later, so it allocates on sight rather than on encounter.
Verified in a REPL against a real dump before being written down, and the
properties are asserted directly rather than inferred from a round-trip -- a
round-trip passes while the mapping is wrong in ways only a populated target
exposes, and that is exactly the case import-db refuses today:
- empty target maps to identity (so a bulk build can skip the rewrite)
- populated target allocates strictly above max-eid, no collisions, injective
- ref VALUES map with their targets (the failure a datom count would miss)
- the mapping is deterministic given (dump, target maxima) -- the property
resumability rests on
- a :db.type/long that happens to equal an entity id is NOT rewritten; the
guard is the schema, not the shape of the number
Uses a local `teardown` rather than `utils/teardown-db`, which derefs the
connection after releasing it and throws :connection-has-been-released.
`migrate_test` already carries the same workaround.
15 tests / 36 assertions for the pre-pass; 144 / 708 across migrate, ids,
codec vectors and CBOR. cljfmt clean.
…ator TWO CHANGES, both preparing the bulk-build path without committing to it. 1. `:translate` on import-db — (fn [[e a v t op]] -> record | nil). Rather than special facilities for attribute renames, value rewrites, unit conversions and redaction, one hook: they are the same operation, and it costs nothing in a streaming pipeline because it is per record. Returning nil DROPS the record. Three constraints, forced rather than chosen, and documented as such: - PURE and deterministic, because a resumable import will re-derive ids from a pre-pass over the same records and the two passes must agree. - 1 -> 0 or 1 -> 1 only; emitting several would break the manifest counts and the tx-alignment the batcher depends on. - It runs AFTER sysref resolution, so a user function sees a plain [e a v t op] with real ids and no internal type ever leaks into it. Verification stays honest instead of being switched off: dropped records are SUBTRACTED from the expected count, so a deliberate drop is not reported as corruption, while records that should have landed and did not are still caught. The report carries :translated? and :dropped. A verification that cries wolf on correct usage is one people disable, and then it is not there when it matters. The rename test makes a point worth keeping: a rename must rewrite TWO positions -- the attribute of a data datom AND the value of the schema datom `[e :db/ident :name]` that declares it. A special-purpose :rename option would have hidden that; the general hook makes it visible. 2. `external-sort` now takes a COMPARATOR over records, not a key function. A key function cannot express index order. An eavt key would be [e a v t] and `v` is heterogeneous, so `(compare [1 :a "x" 5] [1 :a 7 5])` throws ClassCastException -- verified, not assumed. datahike's own index comparators (`datom/index-type->cmp-quick`) exist for that reason, and a bulk index build will pass one of those rather than a key function. The export path keeps its precomputed key (`sort-key` IS Comparable, and precomputing beats recomputing per comparison) behind the `by-sort-key` comparator, so export ordering is unchanged and remains a format guarantee. 156 tests / 744 assertions across migrate, ids, codec vectors and CBOR; 0 failures. cljfmt clean.
`di/init-index` sorts its input in memory -- `arrays/asort` over the whole array -- so building an index costs O(n) heap. Fine when the database fits in memory, which is precisely the case a bulk restore is not. `di/init-index-sorted` takes datoms ALREADY in the index's order and streams them into persistent-sorted-set's `from-sorted-seq` (replikativ/persistent-sorted-set#22), which holds one partial node per level. This is the seam the import path needs: the caller produces the order with an external merge sort, and the index build never materialises it. Verified equal, not assumed: for eavt and aevt, an index built this way has the same count and the same datoms in the same order as one built by `init-index` over the same input. The stronger claim -- same TREE SHAPE, not just same contents -- is asserted upstream in persistent-sorted-set's own suite against `from-sorted-array`, because that is where the builder lives; here the question is only whether datahike drives it correctly. Two deliberate choices worth stating: `from-sorted-seq` is resolved at CALL time via requiring-resolve rather than referenced directly. It is not in the released persistent-sorted-set, and a direct reference makes this namespace -- and therefore all of datahike -- fail to COMPILE against the declared dependency. A clear error for the one caller who asked for the fast path is much better than that. JVM only. There is no cljs `from-sorted-seq` yet, so the cljs path keeps using `init-index`: correct, just not memory-bounded. The tests skip with a message when the streaming builder is absent, so the default suite stays green until persistent-sorted-set#22 releases; the `:bulk` alias points at the working copy meanwhile. The skip ASSERTS rather than merely printing, because kaocha reports a zero-assertion test as a failure -- a bare println skip turns an absent optional dependency into a red suite, which is the opposite of skipping. 165 tests / 762 assertions across migrate, ids, bulk, codec vectors and CBOR; 0 failures either way (18 assertions skipping, 33 with :bulk). cljfmt clean.
A round-trip test that compares query results is much weaker than it sounds: a database whose HISTORY is wrong answers every present-tense query correctly. So this compares the DB RECORD field by field across all six indexes and the schema-derived maps, and replays `as-of` at EVERY transaction rather than only at the end. The generator produces the shapes where history actually goes wrong, because a uniform random database exercises almost none of them: card-one overwrite, card-many add/retract, retract-then-REASSERT the same value, a ref to an entity retracted later, schema added mid-history, and a :db/noHistory attribute overwritten twice. Written BEFORE the bulk-build import on purpose. It is the oracle that path has to satisfy -- produce a DB record equal to the transact path's -- so the fast path is measured against something independent rather than against itself. WHAT IT FOUND, on the EXISTING importer: max-tx drifts by +1. The import ends via `transact-entities-directly`, which bumps max-tx once more, so the restored database skips a transaction id and numbers its next transaction differently from the one it replaced. Datom content is unaffected -- every index matches exactly. `import-db` now REPORTS this as :max-tx-drift and warns on stderr, rather than diverging silently. The test pins it at exactly +1 and asserts it does NOT compound across a second round trip; if it ever becomes +2, or starts accumulating, that is a different and much worse bug and this is what catches it. :op-count also diverges (55 vs 58 on the adversarial db, though it matches on a simple one). Left alone deliberately, with the reasoning recorded in a test rather than a commit message: persistent-set ignores the argument entirely (`_op-count` in every index op), and hitchhiker-tree -- which does use it, as the sequence number for its message buffers -- is deprecated. The test asserts the divergence so that if it ever disappears, someone rechecks whether excluding it from the comparison is still warranted. Everything else round-trips exactly: all six indexes including the temporal ones, schema, rschema, max-eid, system-entities, ident-ref-map, ref-ident-map, secondary-indices, `as-of` at every t, full history, and :db/noHistory. 141 tests / 639 assertions across migrate, ids, fidelity and bulk; 0 failures. cljfmt clean.
… datahike
A bulk-build import reconstructs six index trees from a dump that carries only
history, so it must decide which datoms are CURRENT without replaying
transactions. Get that wrong and the restored database answers every
present-tense query correctly while diverging under `as-of` -- the failure mode
an ordinary round-trip test cannot see. So the rule is settled by measurement
before anything is built on it.
THE RULE IS SIMPLER THAN EXPECTED. Fold history in transaction order keyed by
[e a v]: an assertion adds that exact datom, a retraction removes it. No schema,
no cardinality distinction, no special case for retractEntity or :db/noHistory.
That works because datahike writes an EXPLICIT retraction of the old value even
for a cardinality-one overwrite. Traced it: `{:db/id e :score 10}` over an
existing 1 produces BOTH [e :score 1 t false] and [e :score 10 t true] in the
same transaction, so there is never an implicit supersede to model.
Two consequences that a hand-written version tends to invert:
- Order WITHIN a transaction does not matter, because each [e a v] key is
touched at most once per tx, so the retract and the assert above commute. A
rule keyed by [e a] would NOT commute -- processing the assert first and the
retract second drops the new value -- and that is precisely the bug that
produces a database correct in the present and wrong in the past.
- Cardinality is irrelevant HERE. It governs whether the transactor emits the
retraction; by the time we read history that decision is already recorded.
VERIFIED AGAINST DATAHIKE ITSELF, not against my reading of it: `derive-current`
is compared to `(d/datoms db :eavt)` on databases built to contain card-one
overwrite (once and twice), card-many add and retract, retract-then-REASSERT of
the same value, retractEntity, a ref to an entity retracted later, a same-tx
overwrite alongside an unrelated retraction, schema added mid-history, and a
:db/noHistory attribute overwritten twice -- each in isolation so a failure names
the shape, then all of them interacting, then 12 randomised histories of 25
transactions each.
`split-current` returns {:current :history} from one pass, which is the shape the
bulk build wants: the temporal trees take everything, the current trees take the
subset, and the dump is read once.
Pure and .cljc -- no IO, no db -- so it is testable in isolation and portable
when the rest of the import path gets there.
12 tests / 81 assertions for the derivation; 153 / 720 across migrate, ids,
fidelity, history and bulk. 0 failures. cljfmt clean.
`derive-current` sorts its whole input and accumulates a set of every live datom. That is O(n) in memory, which is fine for a test and useless on a real history -- exactly the case a bulk build exists to serve. The docstrings did not say so; they do now. `current-from-eavt-sorted` is the version the import path will use. The trick is the sort order rather than the fold: `[e a v t]` puts every record for one datom ADJACENT and in transaction order, so deciding whether it is current means looking at the last record of that run and nothing else. State is one partial run, not a set. The sort costs nothing extra. datahike's own temporal eavt comparator IS `[e a v t]` order -- verified -- and the import needs that sort anyway to feed the temporal-eavt index. One sort serves both: the raw stream builds temporal-eavt, the folded stream builds eavt. MEASURED, not assumed: n=100k live-at-mid-stream 18.3 MB n=800k live-at-mid-stream 18.3 MB ratio 1.00x over an 8x input That measurement is now a test. The last unmeasured bounded-memory claim in this stack was false by a factor of n, and the test written to catch it did not -- twice -- so a claim of this kind does not go in without a number behind it. Correctness is checked against the set version AND against datahike directly, including 8 randomised histories, and the emitted records are asserted to be assertions carrying their transaction (so a bulk build can index them without a second lookup) with the LATEST value rather than the first. 24 tests / 129 assertions. cljfmt clean.
…id them
All three were verified against datahike before fixing, and all three were
silent -- no count, digest or query would have caught them.
1. TX METADATA WAS ORPHANED. `build-mapping` decided "is this the transaction
entity?" with `ds/meta-attr?`, a closed set of five idents. But
`flush-tx-meta` writes ARBITRARY user attributes onto the tx entity, so
`{:tx-meta {:author "alice"}}` produced:
[tx :db/txInstant ..] -> mapped through :tids -> 900000001
[tx :author "alice"] -> mapped through :eids -> 600000002
One entity split in two, its metadata stranded on an id nothing references.
The discriminator is structural: `e` names the transaction exactly when
`e` = `t`. No schema needed.
2. THE CURRENTNESS FOLD WAS ORDER-DEPENDENT, on a shape I claimed impossible.
The docstring asserted "each [e a v] key is touched at most once per
transaction, so retract and assert commute". False:
[[:db/retract 100 :tag :x] [:db/add 100 :tag :x]]
in ONE transaction produces both [100 :tag :x t false] AND [100 :tag :x t
true]. `tx-order` keyed on [t e a] -- no v, no op -- so `sort-by` (stable)
just preserved input order, and the same multiset gave #{} one way and
#{[100 :tag :x]} the other. datahike says the datom is PRESENT.
3. THE DUMP'S WITHIN-TRANSACTION ORDER WAS ARBITRARY. `sort-key` also omitted
`v` and `op`, so a card-one overwrite's retract/assert pair COMPARED EQUAL --
verified -- and `merge-runs` breaks ties by PriorityQueue order, which is not
stable. So "a deterministic total order ... a format guarantee" was false, and
it fed (2) directly. Both sorts now end with value then op, retraction before
assertion: deterministic, and matching how datahike emits an overwrite.
A ref VALUE naming a transaction is now resolved through :tids as well; it
was being allocated an entity id, i.e. a dangling pointer.
AND THE TESTS THAT DID NOT CATCH THEM:
`ref-values-map-with-their-targets` PASSED with `apply-mapping` stubbed to
identity -- it only compared the mapped records against each other, which the
unmapped ones already satisfy. It is the exact defect class its own docstring
claimed to guard. It now asserts the ids actually MOVED and that the ref value
matches what the mapping says; stubbed, it fails 3 assertions.
`mapping-is-deterministic` compared f(x) with f(x) on one in-memory vector.
It now re-derives from the dump twice AND asserts a different target maximum
produces a DIFFERENT mapping -- without which the equality proves nothing.
Also: `from-sorted-array` hung to OutOfMemoryError at :branching-factor 2. The
`avg >= 2` guard had been added only to the streaming builder; the arithmetic is
shared. (persistent-sorted-set, separate commit.)
177 tests / 870 assertions across migrate, ids, history, fidelity and codec
vectors. 0 failures. cljfmt clean.
The core of the bulk path. Verified against a real database: for every index
family, both the current and the temporal tree built from a dump equal the ones
datahike built by transacting -- eavt 411/411, aevt 411/411, avet 405/405.
THREE SORTS, NOT SIX. The obvious reading of six trees is six sorts. It is three,
because the temporal comparator is a REFINEMENT of the current one:
current eavt [e a v tx]
temporal eavt [e a v tx added]
Same prefix, so a temporally-sorted stream is already sorted for the current
index -- verified, not assumed. One sorted file feeds both trees of a family.
The `added` tie-break is also exactly what the currentness fold needs: the
temporal comparator orders retraction before assertion (-1) while the current one
calls them equal (0), so the current comparator could not drive the fold even
though it sorts the same records.
`external-sort-to-file` exists for this. `external-sort` returns a lazy seq
backed by run files that close on exhaustion, so it can be consumed once; the
build needs the same order twice, and re-sorting would double the most expensive
step.
TWO THINGS I HAD WRONG, both found by checking against a live database rather
than reasoning:
`:db/noHistory` attributes are in the CURRENT trees and absent from the temporal
ones. My first build fed the whole history to both and produced temporal trees
larger than the source's (415 vs 411). This is also why
`(d/datoms (d/history db) :eavt)` is the UNION of the two rather than the
temporal tree alone -- a distinction that reads like an implementation detail
until a restore silently starts keeping history for an attribute the user asked
to forget.
The `:avet` filter now runs BEFORE the builder rather than inside it, so nothing
sorts datoms it is about to discard.
Scope, stated in the namespace: JVM only (no cljs `from-sorted-seq` yet) and
empty targets only -- building indexes directly cannot honour the upsert
semantics `load-entities` applies against an existing database.
Still to come: DB record assembly and the `import-db` gate, both measured
against the fidelity harness.
digest.clj -> digest.cljc, so a dump written on the JVM verifies on node. SHA-256 was the easy half: goog.crypt.Sha256 is synchronous and incremental, the same shape as MessageDigest, ships in the Closure library datahike already depends on, and works in the browser as well as node. It is also the precedent hasch set with goog.crypt.Sha512. Web Crypto's SubtleCrypto was the obvious alternative and is unusable here — Promise-returning and one-shot, with no incremental digest object anywhere in the standard. The semantic digest was the real work. `xor64+sum64` is 64-bit arithmetic and ClojureScript has no 64-bit integer: numbers are doubles, exact only to 2^53, and bitwise operators truncate to 32. Each record contributes a full 64-bit hash, so a naive port would have agreed with the JVM on tiny inputs and diverged on every real one — silently, in the one number whose whole job is to say whether a dump matches a database. goog.math.Long gives exact xor and wrapping add, and rendering its two 32-bit halves as unsigned 8-digit hex reproduces the JVM's %016x exactly. The test pins literal constants rather than comparing a JVM run against a cljs run, so neither platform can drift by agreeing with the other, and the SHA-256 cases are the published FIPS 180-4 vectors. Verified it can fail: breaking the pinned xor and the single-record hash produces 3 failures on node, so the cljs branch really is computing these and really is correct. Also renames migrate/bulk.cljc to bulk.clj. It was .cljc by mistake — it requires migrate.sort, which is .clj, so a cljs compile would have failed on "No such namespace". Nothing in the cljs build reaches it, so the mistake was dormant rather than broken, but a .cljc extension is a promise the file could not keep. JVM: 198 tests, 873 assertions, 0 failures. node: 189 tests, 1273 assertions, 0 failures.
store.clj -> store.cljc, under konserve's async+sync. This is the medium a ClojureScript export or import goes through: the filesystem medium is JVM-only and stays that way (directories, POSIX permissions, path canonicalisation, .tmp renames have no browser counterpart), and on node a konserve store IS the filesystem when you want it to be. Bumps konserve to 0.9.366 for `konserve.binary/to-bytes` (replikativ/konserve#162), which is where the knowledge about `bget`'s four different handle shapes now lives instead of here. Three things had to change shape, and none is cosmetic: * `reduce` over chunks became `loop`. async+sync is a syntactic postwalk and the async branch is a core.async `go` block, whose state machine covers only code LEXICALLY inside it — not inside a nested `fn`. IO in a reducing function is invisible to it. The old code compiled, passed on the JVM (where `<?-` is rewritten to `do`), and would have deadlocked on node. * `clojure.core.async` must be `:refer-macros [go]` in this namespace. `go-try-` expands into `core.async/go`, and without the refer the CLOJURE macro is used, whose `go-impl` walks `&env` expecting symbol keys — cljs `&env` is the compiler map, with keyword keys. It dies at macroexpansion with a ClassCastException pointing at `go-try-` and explaining nothing. * `connect-store`'s take is hoisted out of the map literal it was returned in. The go transformer cannot rewrite a parked take inside a map literal: it walks the form attaching metadata, and a literal's keyword keys are not IObj. Also portable: `migrate.cbor` gains `concat-records` (chunk bytes end to end, which is all an RFC 8742 sequence is) and `decode-records-from`, since boring's cljs reader takes a whole buffer with no `decode-seq-from` counterpart. That is not a gap the store medium feels — a chunk arrives as one value already bounded by :chunk-size. `decode-records`, the streaming one, stays JVM-only for the filesystem medium, which pushes a multi-gigabyte flat dump through one handle. The new test exercises the medium in BOTH modes. That matters because every existing store test runs {:sync? true} — `export-db` is synchronous — so the async branch had never executed anywhere. Running {:sync? false} on the JVM catches the go-block mistakes without leaving Clojure; node then covers the backend handle shapes. It also pins how a corrupt chunk surfaces in each mode: sync throws, async delivers the exception as the channel value, which is the superv.async convention and is exactly how a caller using `<!` instead of `<?` would get a silent corrupt import. Existing callers are untouched: the old arities remain, defaulting to {:sync? true}. JVM: 264 tests, 1179 assertions, 0 failures. node: 191 tests, 1278 assertions, 0 failures.
A dump directory is a FORMAT, not an implementation detail: plain files anyone can read with `head` and `cbor2`, and byte-identical whoever wrote them. Routing Node through a konserve store instead would have produced konserve-framed blobs under the same names, so a Node dump could not be read by the JVM importer — which defeats the point of having an archival format. Declaring the filesystem medium JVM-only was the right call for the BROWSER, which has no directories, and wrong for Node, which has `fs`. All nineteen operations `migrate` performs have a Node counterpart, `chmod` and `rename` included, so this is a seam rather than a redesign. Paths are strings — the only spelling both runtimes share. Reading and writing are `sink` and `puller`, where a puller is `(fn [] -> bytes | nil)`: exactly the source shape `boring/decode-seq-from` takes (0.1.8, replikativ/boring#4), so a multi-gigabyte flat dump streams through one handle on either runtime and `migrate` has one spelling for it instead of two. `fs.readSync` is genuinely synchronous, which is why a FILE dump can be portable where an arbitrary async source cannot. Loading the namespace in a browser is fine; calling it is not. `js/require` is resolved lazily and guarded, so a browser build that pulls this in through `migrate` still links, and each function then fails with a named error pointing at the konserve-store medium instead of a ReferenceError out of Closure. Also adds incremental SHA-256 to migrate.digest. `write-chunk-stream!` hashes while records stream past and never re-reads what it wrote, so a portable filesystem writer needs an incremental digest, not just the one-shot. Both platforms have one natively (MessageDigest, goog.crypt.Sha256) with the same update/digest shape. The tests assert on BYTES and directory contents rather than on calls not throwing, since the two runtimes share no code here. They cover the chunked writer's actual sequence (mkdir, temp chunk, rename into place, record the size), the flat dump's manifest-line-then-payload layout, UTF-8 text, streaming in bounded pieces, skipping a header without re-reading, an empty file, `..` resolution — the escape a hostile manifest would use — and that hashing while writing equals hashing what was written. One bug found on Node: `(apply (.-join p) p parts)` passes the path module as join's first ARGUMENT rather than as its receiver, so Node raised ERR_INVALID_ARG_TYPE. Every fs test errored until it was `.apply`'d properly. JVM: 300 tests, 1269 assertions, 0 failures. node: 203 tests, 1308 assertions, 0 failures.
`export-to-sink` had three awaited callbacks to `import-source`'s one, and
none of them had run outside the JVM. Its async test passed `{:sync? false}`
explicitly — which is precisely the coverage `import-source` had while it was
broken on Node, since `default-sync?` is FALSE there and the default path was
the untested one.
Two tests, both passing no `:sync?` at all:
* a sink whose :open, :write and :close all return channels — asserting the
same properties the JVM does (dump order, :db/txInstant leading each
transaction, no transaction split across chunks), so a cljs-only
divergence in any of them shows up as a difference rather than a silence.
* the two seams composed: out through export-to-sink, back in through
import-source, reproducing the database — on Node, end to end, with no
dump between them.
Both live in the existing node test namespace rather than a third one; its
docstring now covers both directions. Note again that registering a Node test
takes TWO edits — the require list AND `-main`'s explicit enumeration — which
is already noted at the require site.
Node 252 tests / 1511 assertions / 0 failures (250 / 1504 before).
## pss off :local/root
`{:local/root "../persistent-sorted-set"}` meant nobody outside this machine
could build the branch, and the stale-separator advisory named a fix nobody
could consume. PR #23 is merged and released, so this pins {:mvn/version
"0.5.140"}. Suite against the published artifact: 3001 tests / 35024
assertions / 0 failures — identical to the :local/root numbers, so what was
released is what we developed against.
Two version floors were stale and are corrected. The CHANGELOG said
"correctness requires >= 0.4.137, which is what this PR pins", and BOTH halves
were wrong: it pinned :local/root, and the floor is now 0.5.140 because that
is the release carrying the diff-buf and B-tree correctness pass — including
the stale separator that could leave a datom present but unreachable by a
fully-specified lookup. It now points at pss's advisory and says the repair is
export + re-import. doc/write-amplification.md carried the same stale floor.
## boring into the main :deps
`datahike.cbor` is src/, so boring is a RUNTIME dependency — the dump codec
and the kabel wire format both go through it. It was declared only under
:test, so the tests ran against the pinned version while a CONSUMER resolved
whatever kabel and konserve-sync dragged in transitively: 0.1.6, ELEVEN
versions behind.
That is not a hypothetical. Measured: `export-db` on 0.1.6 throws
`boring: unknown profile — {:type :boring/unknown-profile, :profile
:archival}`. EXPORT WAS BROKEN FOR EVERY CONSUMER while CI stayed green on
3001 tests, because CI runs the alias. Verified after this change with no
alias at all: export-db and import-db round-trip, values intact.
The alias entry is REMOVED rather than left alongside. Two declarations is
precisely how the tested and the shipped versions came apart, and kabel's own
deps.edn names the hazard — "green CI on a version nothing else in the stack
uses". A comment at the new site records the symptom so nobody moves it back
while tidying.
0.1.17 is both what the alias already pinned and the current latest on
Clojars, so no version changes here — only where it is declared.
Note: CHANGELOG.md also carries a pre-existing paragraph about malli
registration that was already in the working tree; it is not part of this
change.
## :xform was applied TWICE on every export `export-record-seq` was extracted in b632521 to stop `export-db` and `export-to-sink` disagreeing about the record stream — and it took the transducer application with it while leaving the one at export-db's call site. So every dump was transformed twice. Measured: a source holding 1,2 with an incrementing `:xform` exported and re-imported as 3,4. Silent corruption, and the manifest, the semantic digest and `verify` ALL agreed with the wrong values, because each is derived from the doubly-transformed stream. It also made the two paths disagree about `:xform` — the sink applied it once — which is the exact drift the extraction was meant to prevent. Nothing caught it because every export-`:xform` test uses an IDEMPOTENT transducer: `(take 120)` in migrate_completeness_test, a `filter` in migrate_init_import_test. Those cannot distinguish one application from two. A non-idempotent case still needs adding. The rationale now lives with the function that owns the application, including why "one instance for the whole export" requires it to live in exactly one place. ## Two memory claims of mine were false `tx-aligned-chunks` said "partition-by streams, and this holds one chunk at a time". True only while `t` VARIES. A stream whose `t` never changes — a database built by one large `transact`, or by `load-entities` — is a single partition-by group, and `into` materializes all of it: 3M constant-t records OOM at -Xmx256m even when consumed lazily. The docstring now says the bound is the largest TRANSACTION, not `n`, and that `:max-pending` does NOT reach it — that backstop lives in the importer's batcher, downstream of the chunker, so a caller who reads `import-source` rule 3 and sets it is not protected. `records->chunk-src` is the one place that breaks `import-source`'s own rule that a descriptor be "small, data-free metadata … never the records themselves": it makes each descriptor a vector of records AND realizes the whole chunk list, so 3M records OOM at -Xmx256m before the importer reads one. The comment now says so plainly and says what to do instead — a source that can address its own storage must build cheap descriptors and a `:read` that fetches, which is the entire point of the two-key shape. Both were found by review agents and confirmed by measurement.
This also REPAIRS THE BUILD. `migrate.cljc` has called `dt/delivered!` in four
places since the A-cluster fixes, and nothing defined it — `datahike.migrate`
does not load at the previous four commits. The definition was written and
never committed. It lands here, with the call sites it serves.
## The failure mode
`go-try-` converts an escaping Exception into a channel VALUE, so `(<?- ch)`
rethrows it and the caller sees the failure. That covers the ordinary case, and
it is why the import path reads as though it were already safe. It does not
cover the two ways a channel closes EMPTY:
* a bare `go` — or a `go-try-` whose throwable is not an Exception, a JVM
`Error` or on ClojureScript any throw of a non-`js/Error` — closes its
channel instead of putting;
* `put!`/`>!` refuse nil, so an operation that legitimately produces nil and
forwards it throws inside the forwarding block, closing that channel too.
In both cases `<?-` yields `nil`. Not an error — a value. Every consumer then
does something plausible and wrong: `(reduce rf acc nil)` is `acc`, so a chunk
that FAILED TO READ becomes an EMPTY chunk, the import continues, reports
success, and is short by exactly that chunk.
The defect is not that an operation can fail. It is that failing became
indistinguishable from returning nothing.
`delivered!` makes the distinction explicit at the four sites that consume a
chunk or a batch report, and `ctx` is merged into the ex-data so each site names
itself — a bare "unexpected nil" is nearly as unhelpful as the silence.
## The writer hangs rather than errors
`writer.cljc` forwarded a completed op with `(go (>! callback (<! res)))`. `>!`
refuses nil, so a closed `res` throws inside that BARE go, which closes the go's
own channel silently and never delivers the callback. The caller's promise then
never resolves: a hang, not an error. `gc-storage` reaches this path on every
call. A closed `res` now forwards an `:async/no-result` ex-info instead.
## `:checksums :skip` was honoured on one medium only
Also in `store.cljc`, because it is the same shape of defect — a documented
option that silently does not apply. Measured, the same dump both ways:
FS bad sha256, :checksums :skip -> imported 20
STORE bad sha256, :checksums :skip -> REFUSED :import/checksum-failed
The store path verified chunk checksums unconditionally, so the option was
honoured in `assert-dump-manifest!` and then overridden here.
The fix shipped; its tests did not. `:on-error :collect` promises to survive a bad RECORD and name it. It was applied to every failure, and because the label was a FALLBACK — `(or (:error (ex-data ex)) :import/corrupt-datom)` — rather than a judgement, a store outage, a released connection or a full disk was recorded as a corrupt datom, collected, and the import continued past it. The report then said the data was bad when the storage was. Someone reading it goes to inspect a dump that is intact. These nine tests pin the distinction: a record-level failure is collected and named, an infrastructure failure ABORTS regardless of `:on-error`, and the `:error` key is the one the throwing layer chose rather than a default applied on the way out.
Every integrity signal a dump carries is derived from the WRITE PATH: the manifest's `:datom-count`, the semantic digest, the per-chunk SHA-256. So a dump that lost records agrees with itself perfectly, and checking it against itself proves only that it is internally consistent — which a truncated dump also is. Measured before this: a 205-datom database exported to a 120-datom dump whose manifest said 120, and which `verify` passed as `:ok? true`. `build-manifest` now records `:source-datom-count` — what the database HELD — alongside `:datom-count` — what was WRITTEN. One number cannot tell a complete dump from a short one; two can. `import-db` refuses an unexplained shortfall as `:import/incomplete-dump`, carrying `:missing`; `:allow-partial?` overrides. `:transformed?` is what makes the shortfall EXPLAINED. An `:xform` that filters a tenant out is a legitimately smaller dump; a shortfall with no xform is unaccounted-for loss. `migrate_import_hostile_test`'s rewrite helper sets it for that reason — every shrinking rewrite there, the zero-record dump most obviously, is otherwise refused before the test's actual subject is reached. Those dumps are intact and wrong, which is a different thing from truncated. `:source-datom-count` is nil under `:count-source? false`: one extra index scan is real cost on a large database, and "unknown" is an honest answer where "equal" would be a guess. The consumer side of this was committed earlier without the producer, so the refusal could not fire. It can now. `doc/backup.md` also documents `:verification`, because `:verified? nil` alone cannot distinguish "switched off" from "nothing to compare against" from "failed under :on-error :collect". The report now carries `:status` saying which.
The coverage gap the audit named last, and the one that hid the whole A-cluster. Everything the JVM suite proves about failure handling it proves under `default-sync? true`. On ClojureScript that flag is FALSE, so Node runs the async path exclusively — and the async path is where a closed channel becomes an empty chunk, since the sync path has no channels to close. The tests for the defect could not reach the runtime most exposed to it. Nine tests: a failing chunk read, a released store mid-import, a source that throws a non-`js/Error`, and the attribution split between record-level and infrastructure failures.
Online GC reclaims blobs from persistent-sorted-set's `markFreed` stream, which
pss documents as a HINT and not a reachability claim. Under diff-buf that hint
is sound only for a LINEAR history: a parent's slot names an anchor PLUS a diff,
so two versions can name the same anchor and neither owns it. Storing one of
them may FLUSH that child — write it out whole and free the anchor — which is
correct for the version being stored and wrong for the other.
Measured in pss against a backend that acts on the callback:
budget <= 4, non-linear history 25 read failures / 768 trials
linear histories clean, 432/432 cells
budget 0 clean, 864/864 cells
Refused at connect rather than silently disabled, so an operator learns their GC
is not running. Checked there because that is the first point where both values
are known: `:diff-buf-size` is create-time-fixed and has just been adopted from
the stored config, while `:online-gc` arrives at connect. The dangerous case is
exactly a reconnect — a database created long ago with a budget, later connected
with online GC switched on. `:allow-unsafe-config` overrides, consistent with
the create-time-fixed conflict beside it, and `online-gc!` then skips with a
warning rather than acting on the stream.
Offline GC (`d/gc-storage`) is unaffected: it derives reachability itself
instead of trusting the hint, and is the supported way to reclaim here.
`doc/gc.md` also corrects the stated reason for the multi-branch restriction.
It said freed nodes "may still be referenced by other branches through
structural sharing", which implies sharing is confined to branches. It is not —
nodes are shared between any two versions, including two versions of one branch.
`datahike.api.specification` has carried `[:=> …]` function schemas for every public operation for a long time and nothing ever checked them: `emit-api` used them only to derive `:arglists` and `:doc`, so they were never registered with malli and never validated a call. Schemas that are documentation-only drift, and these had. Registering them surfaced seven broken ones. The representative case is a `:function` declaring two branches that BOTH admit two arguments: malli dispatches on arity, so it took whichever came first and reported a type error against the other form — a form the implementation accepts. Non-overlapping arities now, which says the same thing and dispatches. `with` is excluded BY NAME rather than fixed, and `uninstrumentable` records why. Its three branches — two of them 2-arity — are the JAVA BINDING's shape: `param-type->java` maps `STransactions` to `List` and `SWithArgs` to `Object`, so those two branches emit two distinct Java overloads, and the `List` one marshals through `Util.normalizeCollections` while the other does not. malli rejects duplicate arities, and merging them into `[:or …]` satisfies malli and DELETES `with(Object, List)` from the generated Java — measured against the generated source — taking the collection normalisation with it. The user-facing binding wins over the check. It is the only such operation, and it is named rather than falling through a silent default. Separately in `codegen/pod.clj`: the variadic-marker test knew `[:* …]` and not `[:+ …]`, so an arity declaring `[:cat A B [:+ :any]]` generated a fixed three-argument fn instead of `[a b & args]`, and `(pod/datoms db :eavt 1 :age)` raised `ArityException: Wrong number of args (4)`. Both markers mean the tail is variadic; only the minimum differs, and that does not survive into `& args`.
Export/import (`datahike.migrate.*`) and `experimental.diff` are one feature with one index requirement: the bulk index build refuses anything but persistent-set, and diff-buf is a persistent-set feature. Running their tests under `:clj-pss`, `:clj-hht` and `:specs` alike was two-thirds waste. Measured before the split: 666 tests / 49.6s across the three suites, roughly 222 tests duplicated twice. They now run once, in a `:migrate` tier that CI takes as its own parallel job. A regex on `:ns-patterns` rather than `^:migrate` metadata on twenty namespaces — same effect, and it keeps the tiering in the file that describes tiers instead of scattering it across the tests it selects.
…er than error A fixed container name and a fixed port 3910 meant two runs on one machine collided — this branch's own test was not concurrency-safe, which cost two phantom errors during review. Both are UUID-suffixed now. It also ERRORED when the garage container was simply absent, so a developer without S3 running could not tell a missing prerequisite from a real failure. Setup failure now skips. The roughly fifteen other fixed `/tmp` paths in the suite are pre-existing and left alone, but note the consequence: the suite is still not safe to run concurrently with itself.
…ound combinations
## The retention rule, corrected
I previously stated the rule as "a binding that CROSSES a park is retained".
That is wrong, and acting on it would have fixed one of the three sites.
core.async only decomposes subforms that CONTAIN a park — but once a park
anywhere in the enclosing form forces decomposition, EVERY `let`/`loop` binding
in that region becomes a local inside the state machine's `try`, where Clojure's
locals-clearing does not apply, and one read from a second block goes into the
state array, which is never nulled. Park POSITION is irrelevant, and so is
whether the arm holding the binding is the one that runs.
Measured, 400k records, used heap sampled inside the block (baseline 15 MB):
bind -> park -> consume 63 MB
bind -> consume SYNCHRONOUSLY -> park 63 MB <- no park in between
bind -> if(parking arm NOT taken) -> use 63 MB <- untaken arm leaks too
inlined into the arm 15 MB
loop parking every iteration, seq unnamed 14 MB
Row three is why `export-db` needed both arms fixed: the store arm's park
decomposes the shared `if`, so the binding is promoted for the FILESYSTEM arm as
well. Under the "crosses a park" rule I would have fixed the store arm and left
`write-chunked!` holding the whole dump.
Three sites, all now building the stream at the point of use through a
`sorted-record-seq` helper that documents the rule:
* `export-db`, both arms;
* `export-to-sink`, in the loop init;
* `init.cljc`'s `sort-family!` call, which held the whole spool per family —
contradicting that function's own docstring, which claimed a seq held
across all three families would pin the dump. It pinned it one at a time.
`async+sync` compiles the same source to a plain `let` under `:sync? true`, so
none of this is visible on the JVM default. `default-sync?` is FALSE on
ClojureScript, so async is the only mode on Node.
The retention test asserts REACHABILITY with a `WeakReference` rather than
sampling used heap: no `-Xmx` tuning, no threshold, no large fixture, and it is
the actual property. It pins `{:sync? false}`, since it would pass against the
pre-fix code on the JVM default. Its first version watched `export-record-seq`,
which `external-sort` drains — so it passed whether or not the outer seq was
bound. Watching `sorted-record-seq` makes it bite; verified by reintroducing
the leak.
## `:xform` applied once, tested with a transducer that can tell
The previous commit fixed the double application and said a non-idempotent case
still needed adding. Two transducers, because they fail differently: an
incrementing one catches DOUBLE APPLICATION, and a stateful ordinal-stamping one
catches the stream being fed through more than once.
The stateful assertion was vacuous at first — it asserted the ordinals were
DISTINCT, and a monotone counter produces distinct ordinals no matter how many
times the stream is transformed (a deliberate double application gave 3,5,7 and
passed). It now asserts they are exactly 0,1,2. Verified against two separate
injections: double application on the export-db side fails two tests, on the
sink side three.
This matters more now than before: creating the stream at the point of use means
"one instance for the whole export" is no longer structural. It holds because
only one arm of a `store-target?` fork runs.
## Three refusals
`export-to-sink` + `:sort? false` — that mode walks `:eavt`, so `t` interleaves
arbitrarily and `partition-by` yields runs of about one record. Measured on a
database where entity order and transaction order disagree, `:chunk-size 40`:
`:sort? true` split nothing across 9 chunks, `:sort? false` put two transactions
in all nine. The seam's whole contract is that a sink can transact what it is
handed. The cost is named in the docstring — this was the only diskless route to
a non-dump target, and a sink that does not transact per chunk is refused along
with the rest. If that becomes a real need the shape is `:tx-aligned? false` on
the sink, not dropping the guarantee for everyone.
`import-source` + `:build-indexes? true` + `:on-error :collect` — the index-build
path carries none of the streaming path's `collect-apply!` / `record-fault?`
machinery, and `reduce-source-records` skips `validate-record!` entirely under
`:collect`. Measured, one source and one set of opts through both paths:
streaming gave `[:OK 10 [{:error :import/malformed-record …}]]`, the index build
gave `[:OK 11 []]` — a nil-valued datom written into the index and zero errors
claimed. Reachable before only through a corrupt dump; a public record source
would have made it routine.
`build-indexes-refusal` was passed `nil` as its manifest, so the clause testing
`(contains? manifest :schema)` was always false and that branch was DEAD —
refusing by naming a manifest a record source can never have. `{:schema (:schema
opts)}` would have been wrong the other way, since a literal map always contains
the key. `(select-keys opts [:schema])` is the one form that tests what it
means. Measured with the guard defeated: a source with no schema under `:eids
:allocate` produced a ref pointing at an entity that does not exist (`#{[3 101]}`
where the schema gives `#{[3 4]}`) — the silent dangling ref the clause exists
to prevent.
## Docstring
`import-source` had 43 duplicated lines — the async and memory sections appeared
twice verbatim. Removed, and the paragraph orphaned by the de-duplication now
sits under a heading that says what it is.
…usal The three user-facing things on this branch with no entry. `import-source` / `export-to-sink` is the headline one — it is what makes the claim "datahike migrates from anywhere" true rather than half true, and it had no mention at all. The entry states the record shape, both seam shapes, the transaction-alignment guarantee a sink gets and `export-db` does not, and the three refused combinations with their measurements. The other two are the go-block retention fix (ClojureScript only, since the JVM default is synchronous) and the online-GC/diff-buf refusal, which is a behaviour change an operator needs to know about: their GC stops running and says so.
whilo
force-pushed
the
feature/boring-wire
branch
from
August 11, 2026 19:23
ef8a7bd to
30f10dd
Compare
whilo
marked this pull request as ready for review
August 11, 2026 19:25
`bb format` failed on it. Whitespace only, but the old indentation was actively misleading: the two assertions were indented as though they sat inside `with-redefs`, when they are siblings following it. That is the correct place for them — they read atoms the export populated, so they must run after it returns, not while the redef is installed — and cljfmt's indentation now says so. Test unchanged: 2 tests, 4 assertions, 0 failures.
`datahike.migrate.datomic` gives both directions over the record seam that landed in #930 — `import-source` and `export-to-sink` — so a Datomic migration inherits the batcher, the id mapping, verification and error attribution rather than reimplementing them. (require '[datahike.migrate.datomic :as dtm]) (dtm/import-from-datomic! dh-conn datomic-conn) (dtm/export-to-datomic! @dh-conn datomic-conn) Peer API, so Datomic PRO. Cloud and Local speak the client API, which has no `d/log` and therefore no way to read the history this source is built on; that is a different namespace, not a different require. `src-datomic` beside `src-kabel` / `src-hitchhiker-tree` / `src-secondary`, added to `config.edn`'s `:src-dirs` so it SHIPS: datahike takes no Datomic dependency, and a user who has one gets migration by requiring the namespace. Tests on `test-datomic`, a path no other tier names — the namespace requires `datomic.api` and fails to LOAD without the peer jar, so being visible from `:integration` (selects by path) or `:migrate` (regex over `datahike.test.*`) would have broken those tiers rather than skipped. ## Identity, measured rather than assumed Datomic's ids do not fit datahike's. `emax`/`txmax` are 2 147 483 647; a Datomic user eid is ~1.76e13 and a tx entity id ~1.3e13. `{:eids :preserve}` is REFUSED rather than silently downgraded, because datahike does not range-check an incoming eid — it reallocates, so passing them through would look like it worked. `:eids` still takes a map or a function. datahike assigns its own `t` regardless: a stream stamped `tx0+1000, tx0+1001` lands as `536870913, 536870914`. Raw Datomic `t` cannot be passed through either — below `tx0` it is refused as `:import/malformed-record`. So the correspondence is recorded as DATA: each transaction carries `:datomic/t` and `:datomic/tx-eid` as datoms ON THE TRANSACTION ENTITY, with the source emitting their schema. Provenance is a query, not arithmetic: (d/q '[:find ?tx :in $ ?t :where [?tx :datomic/t ?t]] @conn 1004) This needs nothing new from datahike — `transact-entities-directly` already carries any attribute on a transaction entity. A `:tids` option beside `:eids` was considered and rejected: the lever exists, but it would still not reproduce Datomic's ids, only choose a different mapping, which these two attributes already do queryably. `{:provenance? false}` turns them off. ## Two mismatches the tests found DATOMIC REQUIRES SCHEMA IN AN EARLIER TRANSACTION than its first use; datahike accepts both in one, and `export-db` emits them together. Replaying verbatim raised `:db.error/not-an-entity Unable to resolve entity: :datomic/t`. A source transaction that installs schema now becomes two Datomic transactions. That fix then broke the round trip: the schema half carried no `:db/txInstant`, so Datomic stamped it `now`, advancing the basis past every historical instant after it — `:db.error/past-tx-instant` on a FRESH database, the case documented as safe. It now takes the source instant minus 1ms. The schema's install time is therefore approximate; its content and ordering are exact. ## Round trips, both directions Datomic -> datahike -> Datomic, compared against the ORIGINAL by reducing each log to per-transaction multisets of `[attribute value op]`: datom-for-datom identical, with exactly one extra transaction (the schema split) and one extra instant (its timestamp). Asserted precisely, so a second source of difference still fails. datahike -> Datomic -> datahike preserves current values and the history of an overwritten value. ## Memory Descriptors are `t` WINDOWS, not records, and `:read` fetches one window. Measured at 3000 transactions / 12012 datoms under -Xmx700m: 60 descriptors, heap 61 -> 61 -> 63 MB, 0 MB to build the source. `log-t-range` originally bound the log seq to a name and then walked it to the end — a full scan AND a retained head, holding the entire log. That is the same defect the export path carries a WeakReference test against, written the same day. `basis-t` answers in O(1); a test counts realized log entries and is red-checked against the old spelling. A `WeakReference` test on chunk results was DELETED rather than kept: it passed against the buggy and the fixed code alike, so it asserted nothing. What replaced it — that `:read` returns a realized, re-entrant collection — can fail. `:max-tx` is deliberately absent from `:source-meta`: it is a drift check meaningful only where `t` is preserved, and declaring it made every successful import warn `max-tx drifted by -3998` about working as designed. Verified: 14 tests / 44 assertions in the datomic tier; 2983 / 34927 / 0 in the main suite, unchanged from before these changes.
Native images have been failing on main since 2026-08-04 with
No matching field found: v for class datahike.datom.Datom
thrown from `transact` inside the bb-pod tests that run against the built
binary. The job is named "Building native images", but the image built fine —
it was the tests exercising it that died.
`transaction.cljc:946` bound
old ^Datom (if upsert? …)
Metadata on the VALUE form hints the `if` expression and never reaches the
local, so both `(.-v old)` sites below compiled to REFLECTIVE field access. The
hint has to sit on the symbol:
^Datom old (if upsert? …)
Reflection resolves on the JVM, so nothing in a 2983-test suite noticed —
`*warn-on-reflection*` had been printing two warnings for this file the whole
time and nothing checks them. A native image has no reflection metadata for the
field, so it throws.
Introduced by 08c2ad8, which added the no-op re-assertion handling: the
mis-hinted local is on the `upsert?` path, and the pod test that fails
transacts `{:db/id 3, :age 25}` over an existing value — exactly that path.
Native was green on 08-03 and red from 08-04.
Verified where it actually matters, since the JVM cannot see this class of
defect:
*warn-on-reflection* on datahike.api 2 warnings -> 0
bb ni-cli builds (6m20s, 141 MB)
bb test bb-pod against ./dthk 1 test, 21 assertions, 0 failures
(CI aborted at 10 assertions, 1 error)
full JVM suite 2983 / 34927 / 0, unchanged
Worth adding separately: nothing gates on reflection warnings, and this one sat
in the open for a week. A check that `datahike.api` loads with zero would have
caught it at the commit rather than in a macOS native job the next day.
Three problems, one of which made the other two invisible.
## They were unreachable
`backup.md`, `import-export-design.md` and `import-export-invariants.md` —
1393 lines — linked only to EACH OTHER. Nothing in `doc/README.md` pointed at
any of them, so the only way in was already knowing the filename. `backup.md`
now sits under Advanced Features beside the other subsystem docs.
## Two of them were development artefacts
`import-export-design.md` opens "design rationale … it is no longer a proposal",
and `import-export-invariants.md` calls itself "the testable contract of
datahike.migrate". A proposal with a status line, and a spec that duplicates
what the tests and docstrings now assert — a second place to drift. Both are
DELETED from the repository and kept only on disk under `.internal/`, which is
gitignored, alongside the twenty documents already there.
Note `git mv` would NOT have achieved that: it stages the destination
explicitly, and `.gitignore` governs only UNTRACKED files, so moving them into
an ignored directory leaves them tracked and published — the opposite of the
intent, and invisible unless you check `git ls-files`.
`backup.md` is the user-facing one and stays.
## The docs did not mention the feature the work is about
Zero occurrences of `import-source` or `export-to-sink` across all three:
they were written before the seam existed and describe `export-db`/`import-db`
as though the dump were the only way in or out. Someone reading the docs would
conclude the record seam does not exist.
`backup.md` gains a "Beyond dumps" section covering both entry points, the six
things a source owes (record shape, order, varying `t`, its own schema
datoms, re-entrant `:read`, descriptors that are metadata rather than records),
and what a sink gets — transaction-aligned chunks, `:close` on the failure path,
the `{:sort? false}` refusal.
`doc/migrate-datomic.md` is new: how to get the namespace on the classpath, why
the scope is Pro and the peer API, the id arithmetic that makes `:eids
:preserve` impossible, provenance as a query, both round trips and their one
structural difference, the limits worth knowing before starting (fresh target,
unique-identity upsert, stripped vocabulary, excision), merge semantics, the
measured memory figures, and how to run the tests.
Closes the documentation half of #134.
whilo
force-pushed
the
feature/boring-wire
branch
from
August 12, 2026 00:14
5454f64 to
3300cfc
Compare
`backup.md` was named and written as though the dump were the product and
everything else an extension of it — a "Beyond dumps" section bolted onto a
backup guide. That has the emphasis backwards. `datahike.migrate` reads and
writes a database as a stream of `[e a v t op]` records; a dump is that stream
with a manifest and checksums around it, and a foreign system is the same
stream with an adapter around it.
Three jobs, one code path:
* backup and restore, via `export-db` / `import-db`;
* changing storage backend, by exporting from one medium and importing to
another — the same dump, written to a filesystem path or a konserve store;
* moving between SYSTEMS, via `import-source` / `export-to-sink`, where the
other side is a live database rather than a file.
Documenting them apart implied they had different properties. They do not:
bounded memory, `:xform`, verification and history fidelity are properties of
the record stream, so they hold for all three. The file is `migration.md` now
and opens by saying which job you are here for.
Renamed rather than copied, so `git log --follow` keeps the history. Three
inbound links updated (`doc/README.md`, `doc/migrate-datomic.md`,
`CHANGELOG.md`); no reference to the old name remains. External deep links to
`doc/backup.md` will break — it has never been in a release, so the exposure is
limited to anyone who followed a link from this branch.
Per the repo's own definitions: Beta is "tested and functional, but API may receive changes"; Experimental is "API likely to change significantly". `datahike.migrate` earns Beta. The dump format and `export-db` / `import-db` are settled — type-exact, full history, deterministic, integrity that fails closed, bounded memory, one implementation on both runtimes. Not Stable, because the API may still change, and the RECORD SEAM is the newest surface and the likeliest to move: every adapter written against it tests whether its shape is right, and there is currently one. Both the doc and the CHANGELOG say that rather than implying the whole namespace is equally settled. `datahike.migrate.datomic` stays Experimental, and the docs now say why it is structured the way it is: it may move out of datahike into its own library. That is what the arm's-length arrangement buys — its own source path, no dependency datahike carries, reachable only when the caller supplies the peer jar, and nothing in datahike requiring it. The move would be cheap because it was built to be. Recorded as a **Status changes** entry, which is the convention this CHANGELOG documents for promotions, rather than by quietly editing the feature entry's label.
`doc/migrate-datomic.md` told readers to depend on `io.replikativ/datahike`. The group is `org.replikativ` — `config.edn` says `:org "replikativ"` and the root README uses it in the Clojars badge, the cljdoc links and the install snippet. The only occurrence in a place a reader would copy from, and the one snippet in that document a reader is most likely to copy, since it is the first thing the page asks them to do.
`migration.md` ended its data-protection section with "Datahike has no
history-excision primitive." That is false, and it is the kind of false that
matters: it tells someone facing a right-to-erasure request that the database
cannot help them.
There are four operations, and they remove from the temporal index rather than
appending a retraction:
[:db/purge e a v] one datom, current and historical
[:db.purge/entity e] every datom of an entity, across history
[:db.purge/attribute e a] every value an entity held for an attribute
[:db.history.purge/before t] all history older than t
Measured rather than read off the source: on a `:keep-history? true` database,
an entity whose email was asserted, changed, then purged leaves NO value of that
attribute in `d/history`, and NONE in a subsequent `{:history? true}` export.
history BEFORE purge: (changed@example.com pii@example.com)
history AFTER purge: ()
PII in :history? true EXPORT: ()
The section now says purge first, export second — the order matters, since a
dump taken before the purge still holds the data and is not retroactively
cleaned — and keeps the two remaining limits: purging rewrites the index but
does not itself reclaim the superseded nodes from the store (that is GC's job),
and a dump already written elsewhere is beyond datahike's reach.
The rest of the section stands: a `:history? true` export really does resurrect
retracted data, which is exactly why the purge/export order is the point.
…ort's
A pass over `migration.md`'s inherited claims, prompted by the erasure sentence
that turned out to be false.
Checked mechanically first: every option keyword and function name the document
names, cross-referenced against `src/` and `src-datomic/`. All of them exist —
the only apparent misses were `ex-info` and `reset-vals!` (clojure.core) and
`endpoint` / `konserve-s3` (prose). The vocabulary is accurate.
Then the documented defaults against the real ones. One gap:
export-db / export-to-sink :sort-buffer 1000000 (migrate.cljc:445, :626)
:build-indexes? import :sort-buffer 200000 (migrate.cljc:2126)
The options table lists 200k and annotates it `(:build-indexes?)`, so it is
correct for the table it is in — I nearly filed it as an error before checking
which path it described. But the EXPORT default appears nowhere in the document,
while the same section tells you to size `:sort-buffer` to your heap for exports.
A reader tuning an export would reasonably take 200k as the starting point and
be off by 5x on the knob that governs export peak memory.
Both are now stated, with the reason they differ: it is one option name for two
different sorts.
Everything else in the numeric defaults checked out: `:chunk-size` 100k,
`:batch-size` 100k, `:spool-chunk-size` 100k, `:dangling-sample` 10, and the
boolean/keyword defaults (`:verify?`, `:on-error`, `:checksums`, `:merge?`,
`:eids`, `:build-indexes?`, `:allow-partial?`, `:spool-codec`).
Four operations were excluded from malli registration by name, because
their `:args` could not describe what they accept without costing the
Java binding an overload. `transact`, `transact!`, `db-with` and `with`
all take EITHER a transaction vector or an arg-map `{:tx-data … :tx-meta
…}`, both documented, both 2-arity — and malli rejects a `:function`
whose branches share an arity (`:malli.core/duplicate-arities`).
The remaining way to say it, `[:or …]`, collapsed to `Object` in
`codegen/java`, which is how `with(Object, List)` and its
`Util.normalizeCollections` were deleted once before. So the binding won
and four operations went unchecked.
`expand-or-args` closes that: an `[:or …]` argument becomes one overload
per distinct Java type. Blindly, that is worse than the collapse — over
this specification exactly three operations expand non-additively:
q (Object) (Object,Object)
-> (Object) (List<?>,Object) (Map<?,?>,Object) (String,Object)
explain (Object,Object) -> (List<?>,Object) (Map<?,?>,Object)
query-stats (Object) (Object,Object)
-> (Object) (List<?>,Object) (Map<?,?>,Object)
Each LOSES its `Object` overload, so any caller holding a variable
declared `Object` stops compiling. So expansion is gated on being purely
ADDITIVE — every signature the collapsed rendering produced must survive
— and that is computed, not maintained as a list of names.
Measured against the generated source, HEAD vs now:
+ transact(Object, Object) + transactAsync(Object, Object)
+ dbWith(Object, Object) + release(Object, Object)
with(…) — the same three overloads, reordered only
No signature removed. `javac` clean, java-bindings-test green.
`with` is the one that settles it: expansion reproduces its exact three
overloads, `Util.normalizeCollections` intact, while compiling as malli.
It was the sole original justification for the exclusion list, so the
list is deleted rather than shortened — an operation that cannot be
registered is a schema bug or a codegen gap, and both are fixable. The
test now asserts all 48 are registered, and names the four individually
so a regression says which.
Also:
* `release` declared one arity; `connector/release` has two, so
instrumentation rejected a call the implementation accepts.
* TypeScript emitted `any | any` for `entity` — branches that differ in
malli need not differ in TS. Deduplicated, same as the Java side.
`codegen_or_args_test` pins both sides of the gate. Red-checked by
removing it: 15 failures, `q`/`explain`/`query-stats` losing `Object`.
The import needs the entity-id mapping to travel WITH the call, so this branch had widened `writer/load-entities` and `writing/load-entities` to three arguments. Both are published: `load-entities` is in the API specification and generates the Java, TypeScript, pod and CLI bindings. Widening it changed a public contract for the sake of an internal one. `load-entities` goes back to two arguments and `load-entities-migrating` carries the mapping, `^:no-doc` because it is a seam for the migrate path rather than an API. The dispatch is shared: `dispatch-load!` takes the argument VECTOR, so the two ops differ only in what they pass. `migrate_silent_failure_test` had been stubbing `dwriter/load-entities` to inject a failure — which the import no longer calls, so the test had silently disarmed itself while still passing. It stubs `load-entities-migrating` now; the migrate tier is 239/1212/0.
`datahike.migrate.datomic` fails to LOAD without the Datomic peer jar rather than skipping, so its tests must be unreachable from every tier that runs without the `:datomic` alias. Declaring the tier in tests.edn broke a plain `clojure -M:test` — kaocha scans `:test-paths` off disk regardless of what is on the classpath. The first fix was worse than the problem: `:kaocha.testable/skip true` beats `--focus`, so `bb test datomic` ran 0 of 8 tests and reported success. A separate `tests-datomic.edn` has neither failure mode.
Expanding an `[:or …]` argument into overloads gave `transact` a
`(Object, Object)` beside `(Object, List)`, so the arg-map form became
callable from Java for the first time. It did not become USABLE:
`convert-arg` decides to marshal from the Java TYPE, and only `List` and
`Map` qualify, so the new overload passed its argument through raw.
Measured from actual Java, before this commit:
Datahike.transact(conn, javaList) -> ok
Datahike.transact(conn, javaMap) -> throws
"Bad argument to transact, expected map, vector or sequence"
{:argument-type java.util.HashMap}
Datahike.transact(conn, Util.normalizeCollections(javaMap)) -> ok
It failed loudly and wrote nothing, so this was a wart rather than a
corruption — but the signature invited a call it then refused, and the
remedy was not discoverable from it.
Both branches of an `[:or …]` denote the SAME Clojure parameter, so both
must marshal. `expand-cat` marks `:normalize?` on an argument whose
`[:or …]` has any branch that is a Java collection, and `convert-arg`
honours the flag as well as the type. Four method bodies change; no
signature does. `release(Object, Object)` is untouched, correctly — its
second parameter is `:any`, not an `[:or …]`.
This also closes the same gap on `with(Object, Object)`, which predates
the expansion work: that overload has never marshalled.
`Util.normalizeCollections` is an O(1) identity on anything already a
Clojure collection, so no existing caller changes behaviour.
Pinned by behaviour rather than by asserting on generated source:
`DatahikeTest.transactAcceptsAJavaArgMap` builds a plain HashMap with
String keys — what a Java caller actually writes, and the case that
failed — and covers `dbWith` too so the second operation cannot pass by
coincidence. Red-checked by forcing the flag off: 3 failures, one per
tier, with the error quoted above.
JVM 3022/35075/0, Node 252/1511/0.
`update-max-tx` and `*import-batch-size*` are not new: they date from af6fe0c, are on origin/main, and are in tags back to v0.3.6-SNAPSHOT. This branch did not delete them — it moved them to `datahike.migrate.legacy` along with the single-file reader they serve. The two are not equal in standing, though, and the move treated them as if they were. CHANGELOG line 129 announces one of them by its fully-qualified name: **Imports are now batched** — `datahike.migrate/import-db` now imports flat-files in configurable batches (`datahike.migrate/*import-batch-size*`, default `10000`) instead of one transaction. ([#845]) So `*import-batch-size*` goes back to `datahike.migrate`. Its value is passed to `import-db-legacy` as an argument rather than read there, so it is read at the call site and therefore inside the caller's `binding` scope; `import-db` still detects a legacy dump and routes to it, so a binding written against that CHANGELOG entry behaves exactly as before. Two things ruled out, both recorded in the docstring so they are not retried: it cannot be defined in `legacy` and referred to from here, because `datahike.migrate` already requires that namespace and the reverse require is a cycle; and it cannot be aliased, because an alias is a NEW var — `(binding [datahike.migrate/*import-batch-size* 5] …)` would set something nothing reads, which is a silent no-op where today's behaviour is a loud compile error. `update-max-tx` stays in `legacy`, deprecated: never documented anywhere, and `max-tx` is maintained by `load-entities` now. CHANGELOG records the move. Line 129 needs no correction — with the var back, it is true as written. The test that covered this was decorative. It bound the var and then asserted only on the imported datoms, which are identical at any batch size, so it passed whether or not the binding was plumbed through. Two candidate observables do not work either: transaction entities, and the datoms' `:tx` — a legacy import PRESERVES the tx from the dump (that is what `update-max-tx` is for) and every datom in the fixture carries 536870913. The batch sizes handed to `transact` are the honest signal. Red-checked by bypassing the binding: `got [12]` against `[5 5 2]`. JVM 3022/35076/0, Node 252/1511/0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related bodies of work on this branch.
Both upstream blockers are resolved: this pins released artifacts
(
boring 0.1.17,persistent-sorted-set 0.5.140), with no:local/rootoverrides, so the branch builds anywhere.
1. The CBOR wire —
datahike.cborThe kabel peer wire moves from fressian to CBOR, via
boring. Datoms ride positionally
(
[e a v t added]), not as field maps — asserted, because that choice was madefrom a measurement, and a measurement that lives only in a commit message stops
being true.
Why it matters beyond speed: a datahike peer's frames become readable by any
CBOR library in any language. A
Datomthat reaches a reader with no datahikehandlers registered arrives as a tagged value carrying its type name and fields
rather than being lost — which is what makes a dump inspectable by something
that has never heard of datahike, and the reason for tag 27 over a private tag
number.
The fressian kabel handlers are removed rather than kept alongside; the
TxReportwire projection moved out of the fressian write handler first, sothe two changes are separable in review.
2. Export and import —
datahike.migrateThis is the lineage of alekcz#5, reworked from a flat CBOR snapshot
into a full-history, type-exact, verifiable dump and restore that survives at
scale, and then generalised past the dump format entirely.
The dump. A directory: an EDN
manifest.edn— human-readable because youread it before you know the codec, and written LAST as the commit marker — plus
numbered chunks that are RFC 8742 CBOR
sequences under boring's
:archivalprofile, so two exports of one database arebyte-identical. gzip by default. The target may be a filesystem path or a
konserve store, so a diskless container backs up to its configured store and a
browser — which has no directories — has a medium at all. Memory is bounded
throughout by
:chunk-size,:sort-buffer, and an external merge sort thatspills runs rather than holding the database.
Import from and export to anything.
import-sourceandexport-to-sinkmake the record stream itself public —
[e a v t op], the same tuples the dumpholds. A source is
{:chunks :read}whose:readmay park, so it can be anHTTP fetch or a Datomic log read; a sink is
{:open :write :close}, eachawaited. Chunks handed to a sink are transaction-aligned, because a sink that
transacts what it is handed must not commit a fragment of a transaction. The two
compose with no dump between them. This is what makes "datahike migrates from
anywhere" true rather than half true, and it is the seam a CSV or Datomic
adapter plugs into without further refactoring.
Building a database from sorted input (
{:build-indexes? true}) constructsall six index trees directly rather than replaying the dump datom by datom,
published in one commit, and is verified equal field-for-field to what the
streaming path produces.
Both runtimes. All of it runs on ClojureScript/Node as well as the JVM from
one implementation under
async+sync, including the external merge sort. Thatis load-bearing rather than a bonus:
default-sync?is false onClojureScript, so Node exercises the async compilation exclusively, and several
of the defects below were invisible on the JVM for exactly that reason.
Datomic Pro, in and out (
datahike.migrate.datomic,src-datomic, shipsbeside
src-kabel/src-secondary). The first adapter built on the seam, andthe proof it carries its weight: source and sink are ~370 lines because the
batcher, id mapping, verification and error attribution come from the seam
rather than being rewritten. Peer API, so Pro — Cloud and Local speak the client
API, which has no
d/logand so no history to read.Datomic's ids do not fit datahike's (
emax/txmaxare 2 147 483 647; a Datomiceid is ~1.76e13), so
{:eids :preserve}is REFUSED rather than silentlydowngraded — datahike does not range-check an incoming eid, it reallocates, so
passing them through would look like it worked. The correspondence is recorded
as data instead: each transaction carries
:datomic/tand:datomic/tx-eidasdatoms on the transaction entity, making provenance a query. That needs nothing
new from datahike — a
:tidsoption beside:eidswas considered and rejected,since it would still not reproduce Datomic's ids, only choose a different
mapping.
Round trips are measured in both directions:
Datomic → datahike → Datomicisdatom-for-datom identical to the original bar one extra transaction and its
timestamp (Datomic will not use an attribute in the transaction that installs
it, so a source transaction carrying both becomes two);
datahike → Datomic → datahikepreserves values and history. Memory is measured too — 3000transactions, 60 descriptors, heap 61 → 63 MB, 0 MB to build the source.
Integrity, and what it cost to get right
Most of the review effort went here, because the failure mode of a backup tool
is not an error — it is a confident success. Recurring shape: every integrity
signal a dump carried was derived from the write path, so a dump that lost
records agreed with itself perfectly.
short to 120 produced a manifest saying 120, a matching digest,
:verified? true, andverify→:ok? true.:statsnow carries:source-datom-count,an independent witness counted from the database, and
:transformed?, whichsays when a shortfall is explained.
go-try-converts a thrownException into a channel value; it does not cover a channel that closes, and
<?-then yieldsnil— not an error, a value.(reduce rf acc nil)isacc. Measured on 20 datoms in 5 chunks with one read failing: 16 restored,reported as success. The same bug in the writer's result forwarding made
d/gc-storagehang rather than fail.:on-error :collectwas appliedto every failure under a label that was a fallback rather than a judgement: an
IOExceptionfromcommit!killed the writer, after which the narrowing retryfiled all 74 remaining datoms as
:import/corrupt-datom— the first of them a:db/txInstantthe exporter itself wrote.:verified? nilcovered three unrelated outcomes, and the two importpaths disagreed about one of them. Verification is now decided in one place
and the report says which outcome happened.
:xformwas applied twice on every export, silently, with the manifest,digest and
verifyall agreeing with the doubly-transformed values. No testcaught it because every
:xformtest used an idempotent transducer.Memory
async+synccompiles one source form to a plainletunder{:sync? true}andto a core.async
goblock otherwise. In thegocompilation a named lazysequence is retained for the lifetime of the block — core.async only decomposes
subforms containing a park, but once decomposition happens every binding in that
region becomes a state-machine local that is never cleared. The record stream is
the whole database, so an export bounded to
:chunk-sizeby design held all ofit anyway. Measured at 400k records: 63 MB retained vs 15 MB with the stream
built at the point of use.
Two things that do not help, both easy to reintroduce: moving the park does
nothing (same decomposed region), and the arm that leaks need not be the arm
that parks — which is why the filesystem export path needed the same fix as the
store one. A
WeakReferencetest pins reachability so this cannot regressquietly.
Refusals
Where a combination cannot be honoured it is refused with a reason rather than
silently downgraded —
export-to-sinkwith{:sort? false},{:build-indexes? true}with{:on-error :collect}, online GC on a:diff-buf-sizedatabase,and the several narrow refusals guarding the index-build path.
Test evidence
Against released artifacts, no
:local/root. The migrate tests move to theirown tier and CI job: export/import and
experimental.diffsupport exactly oneindex, so running them once per index backend was two-thirds waste — 666 tests /
49.6s across three suites, about 222 duplicated twice.
Several of the fixes above are guarded by tests written to fail first, and two
of those tests had to be repaired before they would:
external sort drains — so it passed whether or not the leak was present;
:xformtest originally asserted the stateful transducer's ordinals weredistinct, which a monotone counter satisfies however many times the stream
is transformed. It asserts the exact ordinals now.
Both were caught by reintroducing the defect and confirming the test went red.
Also fixes the red native build on main
Native images have failed on main since 2026-08-04 with
No matching field found: v for class datahike.datom.Datom. The job is named "Building nativeimages", but the image builds fine — the bb-pod tests running against the binary
were dying.
transaction.cljcboundold ^Datom (if …). Metadata on the value form hintsthe
ifexpression and never reaches the local, so both(.-v old)sitescompiled to reflective field access: fine on the JVM, fatal in a native image
with no reflection metadata for the field. The hint belongs on the symbol.
Introduced by 08c2ad8 on the
upsert?path — the same path the failing podtest exercises. Verified by building the image and running the pod tests against
it (21 assertions, 0 failures; CI aborted at 10 with 1 error), because the JVM
suite was green through this bug for a week and proves nothing about it.
Worth doing separately: nothing gates on reflection warnings, and
*warn-on-reflection*had been reporting these two for the whole time.Deliberately out of scope
CSV and tabular importers (the seam is the prerequisite, not the feature),
Datomic Cloud/Local (client API — a different namespace, not a different
require),
:db.unique/identityresolution on merge, and secondary-indexbulk construction. Tracked separately.