HNSW index in Datahike

[matthiasautrata]

The Datahike README mentions that integration of Proximum as a secondary index is "coming soon". Could you elaborate on the planned mechanics for this integration?

Specifically, I am interested in how the Datalog engine will handle approximate or "likely" neighbors natively. In many semantic architectures, the most difficult "meaning gap" is the transition from binary, discrete links (which Datalog handles perfectly) to proximal relationships where a link is a function of semantic distance.

A few specific questions:

Query Interface: Do you envision proximity as a custom predicate (e.g., [(proximal ?e1 ?e2 ?threshold)]) or as a specialized index that filters results within a standard :where block?

Hyper-Relations: How would this index handle entities that participate in multi-dimensional relationships (e.g., a 4-way hyper-relation) where proximity might need to be evaluated across a composite vector?

Timeline: Is there an experimental branch or a rough timeline for when vector proximity can be used to augment the symbolic Datalog constraints?

The prospect of modeling and searching approximate knowledge directly within a persistent, temporal Datalog engine is extremely promising.

PS: I found cozodb interesting in their approach to HNSW integration.

[whilo]

Hey @matthiasautrata. Thanks for these excellent questions and reminding me to look into cozo. Regarding the "meaning gap" between discrete symbolic links and continuous semantic proximity my current approach is:

Query Interface

Both forms work, and they serve different purposes:

;; Predicate: filters existing bindings by proximity
[(proximal? ?e1 ?e2 0.3)]

;; Function: generates new bindings from vector search
[(nearest ?vec 10) [[?neighbor ?distance]]]

In Datahike's Datalog, the function form is more general because it integrates with the standard :where binding pattern. The predicate form is useful when you already have entity bindings and want to filter by semantic distance.

Hyper-Relations

We handle this through compound keys rather than composite vectors:

;; Different semantic spaces for same entity
["doc-1" :title]     → title embedding
["doc-1" :content 0] → content chunk 0
["doc-1" :metadata :author] → author embedding

For true multi-dimensional proximity (4-way hyper-relations), two approaches:

1. Weighted field combination: Independent indices per dimension, scores combined at query time with user-specified weights
2. ColBERT late interaction: Each entity has N token vectors, MaxSim scoring aggregates token-level matches

Composite vectors (concatenation/weighted average) lose the independence of semantic spaces - compound keys preserve it.

Timeline

Available now experimentally in feature/compound-keys (Feedback welcome!):

• Compound key multi-field indexing
• ColBERT MaxSim search
• Weighted field search
• Hybrid (field + token) search

Datahike schema integration: In progress. The rough design is:

{:db/ident :document/embedding
 :db/valueType :db.type/vector
 :db/index true
 :db.index/type :db.index/hnsw
 :db.index/hnsw {:M 16 :ef-construction 200}}

On transact, Datahike would write to both EAVT and the Proximum index via shared Konserve storage. You can already achieve the integration by writing the Datahike transactions into Proximum manually, I have demonstrated this for Lucene here.

The "Meaning Gap"

The deeper insight is that Datalog's discrete links and vector proximity are orthogonal dimensions. Symbolic constraints narrow the search space; vector search expands it with semantic neighbors. The integration challenge is keeping them composable rather than conflating them.

[matthiasautrata]

Super interesting. I am thinking about building a semantic data-catalog: physical, logical, ontology, concepts with hard relationships (discrete link as you call it), e.g. for (logical represented-by physical) and also proximity, e.g., (auto-loan is-close-to personal-loan) for consumer banking marketing purposes. Basically, relationship edges with weights where weight 1=discrete/asserted link). You see immediately how I end up with a graph that describes facts and neighborhoods. We'll likely author the structures in LinkML and then parse that into the graph. LinkML's meta-model lends itself to be represented in a EAV graph and LinkML's semantics lend themselves to be represented in datalog. Amusingly, one could view datalog as the ontology-virtual-machine (OVM) that formalizes LinkML semantics as well as a subset of OWL/SHACL. It would thus build the ideal bridge between those worlds.