feat(rdf): RDF / RDF-star via interned literals (federated named graphs) - #864
Open
whilo wants to merge 1 commit into
Open
feat(rdf): RDF / RDF-star via interned literals (federated named graphs)#864whilo wants to merge 1 commit into
whilo wants to merge 1 commit into
Conversation
whilo
added a commit
that referenced
this pull request
Jul 7, 2026
Member
Author
Reconciling with #881 — rework to interned-literal entities (option C)#881 removed the The bug. The rework (option C): a literal becomes an interned entity keyed by its canonical term hash — which is exactly Virtuoso's The irreversible part is the canonical term-id function (lang→lowercase, plain→ Named graphs: decided to ship federation (one store = one dataset; named graphs = federation via the reference layer) as a documented 1.0 scope limit, not a quad. A 4th indexed datom position is the alternative and was consciously deferred (it's a storage-format change, revisitable post-1.0). Full design, the interned-entity schema, and the faithfulness fixes (reified-triple object typing, |
Reworks #864 to be consistent with #881, which removed the open value-type registry. Evaluating the reconciliation against Virtuoso and the SPARQL 1.1 spec found the native :db.type/literal was not merely core weight but a CORRECTNESS BUG: rdfliteral-compare returned (compare numkey numkey) with no lexical/datatype tiebreak, so "1"^^xsd:integer, "01"^^xsd:integer and "1"^^xsd:long all compareTo 0 and the sorted-set AVET index collapsed three distinct RDF terms into one slot — lost datoms, wrong COUNT/DISTINCT — while equals keyed on lexical and disagreed. Root cause: the value type fused ORDERING and IDENTITY into one Comparable and then sacrificed identity for numeric order. RDF's index must key on TERM (lexical ∧ datatype ∧ lang), never merging distinct terms; value equality is a query-time op. So a literal is now an INTERNED ENTITY keyed by its canonical term hash — Virtuoso's RO_ID model — giving term dedup for free and making the collapse structurally impossible. Numeric range stays index-native via a :literal/num (:db.type/bigdec) shadow that d/index-range seeks, with no RDF code in datahike's core comparator. The irreversible part, the canonical term id, canonicalizes lang→lowercase and plain→xsd:string and preserves the lexical form verbatim; rdf_test pins all four rules (they are on-disk identity) and the no-collapse case the value type got wrong. RDF-star: annotate now ASSERTS the ground triple (real, typed, queryable datoms) AND reifies it as an annotatable statement addressable by a static term-string lookup-ref; quote-triple reifies only. The old design created only the reification, so the ground pattern returned nothing. rdf-term-compare stays as the query-side SPARQL ORDER BY. Named graphs are a federation of stores (db-id ≙ graph), documented as a deliberate 1.0 scope: it fits graph-scoped datasets but is not the single-dataset-many-graphs SPARQL model, and a quad (4th datom position) is a deferred storage-format decision. Drops datahike.value-types, RDFLiteral, and all custom-value-type codecs. No open value-type seam — RDF literals were the one arguable case for it, and interning serves them better and correctly. Cross-platform (JVM + cljs compile clean).
whilo
force-pushed
the
feat/rdf-custom-value-types
branch
from
July 15, 2026 00:32
e76b49f to
407327c
Compare
whilo
changed the title
whilo
added a commit
that referenced
this pull request
Jul 16, 2026
…by GC (#881) * feat(gc): :db.type/store-ref — datom values that name objects in the store Lets a datom value NAME AN OBJECT in the database's own konserve store — a blob, an out-of-line document — and makes the garbage collector keep that object alive. The rule: THE DATABASE IS THE ROOT SET. An object in the store lives iff a datom points at it; anything else is garbage by definition. Users write the object themselves (k/bassoc under a hasch content hash, or k/assoc for EDN — binary vs EDN is k/bget vs k/get, their business) and transact the key. No d/put-blob: it would be a strictly worse re-export of konserve's API (sync?/async, bget callbacks, streaming), and the store is right there. WHY GC HAD TO CHANGE. The mark walks the index TREES and collects node addresses; it never looks INSIDE a datom's value. An object named only by a value is therefore unreachable from the mark's point of view, and the sweep deletes it — so a blob could not be stored in the database's own store at all. datahike.value-types (extracted from #864, minus RDF) gains :reachable-keys: a value type declares which konserve keys its values keep alive, and the mark unions them in. This is not scope creep on #864 — it closes a hole #864 OPENS: a custom type today can hold a konserve key and GC will silently delete it. The registry is the only place that can state the contract. :db.type/store-ref is the seam's first client and its smallest possible one: the value is a UUID, so its predicate, ordering and codecs already exist. The whole type IS its GC contract. The mark slices AEVT for exactly the attributes whose valueType is key-bearing, reusing the same storage-bound, root-seeded index -mark already builds. A database that declares no such attribute pays NOTHING. Under :keep-history? the temporal index is scanned too — a retracted store-ref is still readable as-of, so the object must outlive the retraction. Fails closed: a historical record whose schema names an unregistered type refuses to collect rather than marking nothing. Two schema shapes are REJECTED, not documented, because both lose data silently: - a tuple holding a store-ref: the attribute's valueType is :db.type/tuple, so the registry is never consulted and the keys nested in the vector are invisible to the mark. - :db/noHistory + store-ref: retracted values are not retained in the temporal indices, so an as-of read could name an object that was collected. :attribute-refs? needs :db.type/store-ref as a system entity (id 52, APPENDED — a :db/valueType is a REF there, and without an entity to point at, the schema transaction is rejected). Safe for existing databases: system-entities and max-eid are stored per-database, so an old db keeps its own. Writing an object and referencing it in a LATER transaction leaves a window where nothing names it; hold gc-guard/with-unreferenced-writes across the two (#879). s/def :db.type/store-ref is NOT redundant with the registry's :pred — value-valid? validates through the CLOJURE.SPEC registry, and #864's :pred is dead code. describe-type rebuilds the legal-type enum (now sorted, so the error is deterministic) rather than printing the open predicate's source. * feat(gc): store-refs name objects — datahike owns the mark, you own the sweep Splits the blob story along the line that actually matters: WHICH IDS ARE STILL NAMED (hard — immutable, structurally shared, branched, historical) versus HOW TO DELETE THE BYTES (easy — and store-specific). Datahike does the first and refuses to guess at the second. :db.type/store-ref names an object. It deliberately does NOT say where the bytes live, and the same type covers both deployments: IN THIS KONSERVE STORE — gc-storage! spares it while referenced and reclaims it once nothing names it; delete-database erases it with the database; it works on every backend, so the local :file setup behaves like S3. But konserve frames a binary value as [header][meta][payload], so the object is NOT raw at that key: every byte must pass through k/bget / k/bassoc, i.e. through your process. Fine for a thumbnail, wrong for a 50 MB upload. ANYWHERE ELSE — the browser PUTs straight to s3://…/blobs/{uuid} with a presigned URL and a content-type, and the bytes never touch your JVM. That is the only sane shape at real sizes, and it is what the SaaS template's users asked for. Datahike cannot delete from there and does not pretend to. So `reachable-store-refs` exposes the MARK without the sweep: every store-ref the database still names, across all branches and through retained history, honouring remove-before exactly as index nodes do. External blob GC is then ~15 lines in the application: list the prefix, delete what is not in the set. Marking an external id is a harmless no-op for the local sweep, so the two mix freely. Retention falls out rather than being special-cased: a store-ref named only by history is still live, because an as-of read can still reach the datom naming it — and a retracted attachment you can no longer fetch is not history. Also states, in datahike.value-types, the rule the whole seam turns on: A VALUE TYPE IS SOMETHING THAT SORTS. Datahike's AVET order IS the value's compareTo. If your value has no meaningful total order it is not a value type — it is a blob (store-ref) or a document (transact it as datoms, or index it). Datoms are already sparse; you never had to declare your fields. So storing a document as an opaque value buys no flexibility you did not already have, and costs you the indices. Every database in this family agrees — XTDB indexes every field automatically and offers no opaque option at all; Datalevin's nippy escape hatch explicitly gives up range queries; pg-datahike stores jsonb opaquely and pays for it with a full scan and a parse per row. Tests cover both deployments, plus the two that keep the mark honest: an object written but never referenced IS collected (the root-set rule), and one named only by retained history is NOT. * feat(blob): content ids — datahike.blob/blob-id, and the git model for names The id must PIN THE CONTENT, and the reason is time travel, not tidiness. A store-ref is dereferenced when you read it — INCLUDING when you read it as-of an old transaction. If the id is a mutable pointer, that old read hands you the reference, you fetch it, and you get whatever is behind it NOW rather than what was there THEN. The database's central promise silently stops holding for that attribute, and nothing tells you. A content hash makes that impossible: the reference IS the content, so dereferencing an old reference necessarily yields the old bytes. Same reason index nodes are content-addressed and the branch head is the ONLY mutable cell in the store — a blob behind a mutable name is a SECOND mutable cell, one datahike can neither see nor protect. The requirement is WRITE-ONCE. Content addressing is how you guarantee it. A random uuid stays permitted (you cannot always hash the bytes — a streaming upload you never buffer, a third-party id), but then time travel holds only for as long as you never rewrite that object, and dedup and idempotent re-upload are gone. WHY A PATH IS NEVER THE ID, AND WHERE THE PATH GOES. Paths move; objects at a path get overwritten; a rename would break every historical reference. This is the git model: blobs are addressed by CONTENT, and TREES map names to hashes. So the name is its own datom: {:blob/id #uuid "…" ; :db.type/store-ref — identity :blob/path "tenant/acme/2026/invoice.pdf"} ; :db.type/string :db/index true — location You lose nothing: with :db/index the path is in AVET, so "tenant/acme/2026/" is a range scan — the sorting you wanted a string id for, without making the id mutable. A rename touches one datom, the object does not move, and every historical reference still resolves. A foreign id you do not control is just another string attribute. And the id is a UUID for a MECHANICAL reason, not a stylistic one: the index order is the value's compareTo (datahike.datom/compare-value ends in `(compare v1 v2)`), and (compare #uuid "…" "a-string") throws ClassCastException. An attribute holding both would have no well-defined AVET order. datahike.blob/blob-id is the one function: bytes -> hasch uuid. No put-blob — writing bytes is konserve's job if the object lives in the store, and your object store's job if it does not. What datahike offers is the id, because the id is where the semantics live. Verified that hasch hashes byte-array CONTENT, not identity — two distinct arrays with the same bytes give the same id, which is what the whole story rests on. * refactor(schema): store-ref is a plain builtin — drop the value-type registry `datahike.value-types` (extracted from #864) let a `:db/valueType` be registered at runtime. On reflection we do not want an OPEN extension point at this level: - Nobody in the family has one. Datomic's request sits unanswered; Datalevin's types are a closed byte-tagged enum (its only seam widens nippy's allowlist for UNORDERED blobs); XTDB infers, no registry. The seam would be datahike's alone. - It is a PERMANENT, uncontrollable commitment: the :db/valueType keyword lives in the stored schema, so a user type ends up in someone's database forever, and the on-disk B-tree order becomes third-party code (assert-registered! catches an ABSENT impl, not a CHANGED comparator — which makes data unfindable, not wrong). - It bought nothing even for its own first client. Adding :db.type/store-ref still needed an s/def (value-valid? goes through the clojure.spec registry, so the registry's :pred was dead code), a describe-type fix, and a system-schema entity for :attribute-refs?. The registry did not remove one core edit; it hid which ones you still owed. Two ways to define a :db.type/* is worse than one. - Everything with real demand — geo, vectors, full-text, JSON paths — needs a specialised INDEX STRUCTURE, i.e. the SECONDARY-INDEX seam, not a value type. A value type only helps when the value has a total order you want in the PRIMARY B-tree and cannot express as an existing type or a tuple. That set is basically {RDF literals}. You can always open a closed seam later; never the reverse. So store-ref is now a plain builtin, defined exactly like every other type: an s/def, membership in builtin-value-types, a system-schema entity, codecs where needed (a UUID's are already present). Its GC contract collapses from a per-type :reachable-keys fn to a def — `schema/key-bearing-value-types #{:db.type/store-ref}` — because the value IS the key. The mark reads the attribute's values directly; no registry lookup, no impl indirection. The four seam hooks that existed only to merge registry codecs (persistent_set, remote, http/client, connector) revert to main. If RDF's :db.type/literal belongs in core it becomes a builtin the same way, losing nothing it actually uses. No behaviour change for store-ref; the registry is simply gone. * changelog: fill in PR number (#881) * docs: link store-refs from doc index; drop stale registry references The store-refs doc was not reachable from doc/README.md. Also scrub comments in schema.cljc that described a rejected runtime value-type registry (:reachable-keys / "registered custom types") the PR never shipped. * docs: reframe in-store blob tradeoff as a proxy hop, not a heap limit The old wording implied a 50 MB payload was unworkable in-store. With a streaming handler the heap stays flat; the real cost is an extra network hop through the JVM and the loss of S3-native range/multipart/CDN. Say that instead, in both the doc and the reachable-store-refs docstring. * test(store-ref): port to cljc deftest-async; run on Node; make guard macro cljs-usable The store-ref tests were JVM-only. Rewrite them as portable `deftest-async` and add them to the Node suite, per the policy of growing cljs coverage rather than adding clj-only tests. The feature is portable and its headline case (a browser referencing an out-of-line object) is cljs, so the tests should run there. Also make `datahike.gc-guard/with-unreferenced-writes` available to ClojureScript by self-requiring the macro namespace (the `datahike.test.async` pattern). It was `#?(:clj (defmacro ...))` in a .cljc but never exposed to cljs; `put-blob!` now uses it on both platforms, so the doc's write-window example is portable too. Platform seams are isolated in helpers (create/connect/delete, byte construction, binary read-back). The raw-byte read-back assertion is JVM-only: konserve's Node sync binary read miscomputes the value length, an upstream quirk unrelated to the store-ref contract — the datahike-level guarantee (survives GC, round-trips cold) is asserted portably. Verified: JVM 9 tests/28 assertions; Node full suite 137 tests/1062 assertions, both green. * feat(store-ref): follow store-refs in sync too — move the konserve-sync walker into datahike The GC mark was taught to follow :db.type/store-ref values in this PR, but the konserve-sync replication walker had the identical blind spot: it walks the index TREES and never reads a datom's value, so a blob named only by a store-ref was never shipped — a subscriber ended up holding a live datom pointing at an object that never replicated. Fix it where the knowledge belongs. The walker reaches into datahike's stored-db record format, so it moves out of konserve-sync into `datahike.kabel.walker`, versioning in lockstep with the record format (konserve-sync's recent :fuse-index-roots? fix was that cross-repo skew biting). It now unions the new `datahike.gc/record-store-refs` — the per-record store-ref slice gc-storage! already runs — into the NODE portion of the walk, ahead of the mutable pointer cells, so a blob arrives before the head that makes its referencing datom live. `record-store-refs` needs only a store + a stored-db record (schema, attribute-refs? and the index all read from the record itself), so it is reusable and connection-free. No konserve-sync release is required: the walker imports no konserve-sync namespace (the walk-fn is a value passed to the already-public register-store!), and datahike depends on konserve-sync only via the kabel alias. konserve-sync's copy is deprecated in place (separate repo). Verified: gc/record-store-refs (JVM + Node via store_ref_test), datahike.kabel.walker (walker_test), and the kabel integration suite (6 tests/41 assertions) all green. * test(walker): port to cljc deftest-async; run on Node The datahike konserve-sync walker runs on cljs too — a browser subscriber walks the same way — so its store-ref-following test should. Convert walker_test to a portable deftest-async and add it to the Node suite; shadow resolves the walker (in src-kabel) via the :test alias classpath, no build config change needed. Verified: JVM 1 test/4 assertions; Node full suite 139 tests/1068 assertions. * fix(store-ref): reject :db/noHistory added to a live store-ref in a later tx key-bearing-misuse only inspected the transacted entity, so the guard against store-ref + :db/noHistory (silent data loss: retracted values aren't retained, yet an as-of read can still reach a datom naming a since-collected object) only fired for the all-in-one declaration. A two-step add — a partial entity map {:db/id attr :db/noHistory true} or a raw [:db/add attr :db/noHistory true], neither restating :db/valueType — slipped past it. Move the guard to update-schema, the universal chokepoint every schema datom flows through (entity-map-derived and raw ops alike), and check the RESULTING schema entry: whichever op completes the bad shape trips it, regardless of path or datom order. Benign changes (:db/doc, …) and :db/noHistory on a non-store-ref attr are unaffected. check-schema-update keeps its early entity-level check for a clean all-in-one error. Verified: JVM store-ref/schema/tuples/transact/entity-spec/migrate suites green; Node full suite 139 tests/1070 assertions. * fix(test): store-ref orphan sweep raced the collection cutoff; correct gc schema-fallback comment CI flake in unreferenced-blob-is-not-kept: gc-storage! returned an empty sweep, so the just-written orphan wasn't reclaimed. Diagnosis: konserve's sweep! spares any object whose last-write is NOT strictly before the cutoff (`(<= cutoff last-write)`), and the cutoff is min(safe-point, now) = now when nothing is in flight. The test bassoc'd the orphan and collected with no time gap, so on a fast box the write and the cutoff land in the same wall-clock millisecond and the orphan is spared. Ruled out an in-flight guard (0/80) and safe-point pinning (now-safepoint 0..-1ms); the orphan is the sole sweep candidate (swept-count uniformly 1), so a same-ms write is the only way the sweep comes back empty. Let a beat pass before collecting — a real collection never runs in the same instant as the write it reclaims. Also correct the gc schema-fallback comment: `(:schema record)` is non-nil only for old inline-schema databases, which predate :db.type/store-ref and declare no key-bearing attribute, so the fallback mirrors stored->db for parity — it does not (and cannot) guard store-refs as the old comment claimed. Verified: JVM store-ref 10 tests/32 assertions; Node full suite 139 tests/1070. * test: address Copilot review — explicit :refer-macros; drop stale registry comment - store_ref_test (cljs): refer with-unreferenced-writes via :refer-macros rather than :refer. The plain :refer already resolved (gc-guard self-requires its macros + cljs implicit macro loading, and the Node suite was green), but :refer-macros is the unambiguous, conventional form for a macro and removes any doubt. - schema_test: correct a stale comment claiming a datahike.value-types registry and an "open predicate" :db.type/value. Neither exists — :db.type/value is a fixed set (builtin-value-types); describe-type just materializes it sorted for a deterministic error message. (Same stale-registry family scrubbed from schema.cljc earlier; this test file was missed.) Verified: JVM schema+store-ref 19 tests/100 assertions; Node full suite 139/1070.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
RDF / semantic-web modelling on datahike reduces to one genuinely new primitive: a native literal value type. RDF's other node kinds already map onto datahike — an IRI is a
:db/uniqueentity (a ref, addressed by lookup-ref, the same primitive as cross-database references), a blank node is an eid, and a quoted triple is a reified statement — so the only shape datahike lacks is a literal(lexical, datatype, lang).Rather than special-case RDF in core, this PR adds a general seam for custom value types and makes RDF its first client. This is useful on its own: users can add a
:db/valueType(geo, money, tensors, …) without hacking datahike.What this adds
datahike.value-types— a registry seam. Register a:db/valueTypewith its predicate,Comparableordering, and fressian/transit/edn codecs:datahike.datom/compare-valueends in(compare v1 v2), andcomparedelegates toComparable— so aComparablevalue sorts in AVET with zero change to the comparator and zero cost to the scalar path. The value'scompareTois the on-disk order.:db/valueTypekeyword persists in the stored schema and travels to every peer for free; the impl does not.conn-from-dbcallsassert-registered!, so opening a store whose schema names an unregistered custom type fails at connect with a clear error, not deep in a later read.datahike.experimental.rdf— the first client::db.type/literal— a native RDF literal, sorted in exact SPARQL literal order (numerics compare numerically viabigdec; else lexical → datatype → lang). RDF 1.1 term equality.literal/literal-attrhelpers.[s p o]tuple, so a statement is addressable by lookup-ref, annotatable, and idempotent on re-assertion (RDF set semantics). Corresponds to RDF 1.2'srdf:reifiesreifier.rdf-term-compare— the full SPARQL ORDER BY order over mixed terms, for ordering query results.Core touches (all additive)
value_types.cljc(new) — the registry.schema.cljc—:db.type/valuespec consults the registry (builtin ∪ registered);describe-typekeeps the helpful builtin list.index/persistent_set.cljc— merge registered fressian handlers into the store serializer (the durability path).remote.cljc/http/client.clj— merge registered transit/edn handlers.connector.cljc— connect-time registration check.No comparator change, no rschema change, no change to existing stores.
Verification
Validated end-to-end on a file backend: durable write → release → reconnect reconstructs
RDFLiteralvalues from disk and preserves datatype-aware order (9 < 42, not"42" < "9"); AVET reverse-lookup by a full literal survives reload; transit + EDN round-trip; connect-time check passes for a registered type and throws for an unregistered one. Core suites (core/schema/tuples) pass (24 tests / 137 assertions, 0 fail / 0 error).clj -M:ffixclean.See
doc/rdf.mdanddatahike.experimental.rdf/demofor a runnable showcase.Scope / limits