fix(query): never answer from a secondary index that is still building - #914
Open
whilo wants to merge 2 commits into
Open
fix(query): never answer from a secondary index that is still building#914whilo wants to merge 2 commits into
whilo wants to merge 2 commits into
Conversation
A schema-declared secondary index is created with status :building and
backfilled by the WRITER. A database built with `d/db-with` has no connection and
therefore no writer, so the index stays :building indefinitely and accumulates
only the datoms of transactions made after it was declared — while the aggregate
path never looked at the status.
(d/db-with (db/empty-db {:num/v {} :idx/a {:db.secondary/type :stratum …}})
[{:db/id 1 :num/v 10} {:db/id 2 :num/v 30}])
(min ?x) => nil (reference: 10)
…after one more transaction adding 50:
(min ?x) => 50 (reference: 10)
A silent wrong answer on any column type, not a stale one. Indices whose status
is :building or :disabled are now skipped, matching the check the transact path
already makes.
The gate is precisely targeted, verified on all three construction routes: a
hand-assembled index (status nil — what every existing test uses) and a :ready
one both still reach the columnar delegate, while :building declines to the path
that reads the primary indices and answers correctly. Nothing is lost but the
unsound read.
Found by an agent studying the stratum cluster; it is independent of the stratum
questions still open there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A schema-declared secondary index is created with status
:buildingand backfilled by the writer. A database built withd/db-withhas no connection and therefore no writer, so the index stays:buildingindefinitely and accumulates only the datoms of transactions made after it was declared — while the aggregate path never looked at the status.A silent wrong answer on any column type, not a stale one, and independent of the open stratum questions.
The fix
Skip an index whose
:db.secondary/statusis:buildingor:disabled— the same check the transaction path already makes. The query then falls back to a path that reads the primary indices and is correct.Why this gate and not another
:buildingseparates the broken construction route from every working one exactly. Verified on all three:assoc :secondary-indices(what every existing test uses)nil:readyd/db-with, declared + populated:buildingGating on
(= :ready …)instead would have excluded the hand-assembled route, whose index is complete by construction and which the existing secondary-index tests depend on — including the one that asserts the delegate actually runs.Nothing is lost but the unsound read: in the declined case the query is still answered by the columnar path that materialises from the primary indices.
Verification
50), and the hand-assembled route still using the delegateFound while an agent was studying the stratum defect cluster; the remaining items there (dictionary-code aggregates, storage-time Double truncation, untransactable Boolean/UUID values) are separate and still open.