Skip to content

fix(query): never answer from a secondary index that is still building - #914

Open
whilo wants to merge 2 commits into
mainfrom
fix/secondary-building-index
Open

fix(query): never answer from a secondary index that is still building#914
whilo wants to merge 2 commits into
mainfrom
fix/secondary-building-index

Conversation

@whilo

@whilo whilo commented Jul 28, 2026

Copy link
Copy Markdown
Member

A schema-declared secondary index is created with status :building and backfilled by the writer. A database built with d/db-with has no connection and therefore no writer, so the index stays :building indefinitely and accumulates only the datoms of transactions made after it was declared — while the aggregate path never looked at the status.

(def db (d/db-with (db/empty-db {:num/v {}
                                 :idx/a {:db.secondary/type :stratum
                                         :db.secondary/attrs [:num/v]}})
                   [{:db/id 1 :num/v 10} {:db/id 2 :num/v 30}]))

(d/q '[:find (min ?x) :where [?e :num/v ?x]] db)   ;; => [[nil]]   reference: [[10]]
;; …after one more transaction adding 50:
                                                   ;; => [[50]]    reference: [[10]]

A silent wrong answer on any column type, not a stale one, and independent of the open stratum questions.

The fix

Skip an index whose :db.secondary/status is :building or :disabled — the same check the transaction path already makes. The query then falls back to a path that reads the primary indices and is correct.

Why this gate and not another

:building separates the broken construction route from every working one exactly. Verified on all three:

route status delegate result
hand-assembled assoc :secondary-indices (what every existing test uses) nil runs correct
connection, declared then populated :ready runs correct
d/db-with, declared + populated :building declined now correct

Gating on (= :ready …) instead would have excluded the hand-assembled route, whose index is complete by construction and which the existing secondary-index tests depend on — including the one that asserts the delegate actually runs.

Nothing is lost but the unsound read: in the declined case the query is still answered by the columnar path that materialises from the primary indices.

Verification

  • Full suite 2554 tests / 21469 assertions / 0 failures
  • Focused secondary-index namespaces 75 tests / 537 assertions / 0 failures
  • New regression test covers the declaring transaction, a further transaction (where an unguarded read returned 50), and the hand-assembled route still using the delegate

Found while an agent was studying the stratum defect cluster; the remaining items there (dictionary-code aggregates, storage-time Double truncation, untransactable Boolean/UUID values) are separate and still open.

whilo added 2 commits July 28, 2026 13:51
A schema-declared secondary index is created with status :building and
backfilled by the WRITER. A database built with `d/db-with` has no connection and
therefore no writer, so the index stays :building indefinitely and accumulates
only the datoms of transactions made after it was declared — while the aggregate
path never looked at the status.

  (d/db-with (db/empty-db {:num/v {} :idx/a {:db.secondary/type :stratum …}})
             [{:db/id 1 :num/v 10} {:db/id 2 :num/v 30}])
  (min ?x)  =>  nil          (reference: 10)
  …after one more transaction adding 50:
  (min ?x)  =>  50           (reference: 10)

A silent wrong answer on any column type, not a stale one. Indices whose status
is :building or :disabled are now skipped, matching the check the transact path
already makes.

The gate is precisely targeted, verified on all three construction routes: a
hand-assembled index (status nil — what every existing test uses) and a :ready
one both still reach the columnar delegate, while :building declines to the path
that reads the primary indices and answers correctly. Nothing is lost but the
unsound read.

Found by an agent studying the stratum cluster; it is independent of the stratum
questions still open there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant