fix(gc): a secondary index must not be silently collectable - #900
Open
whilo wants to merge 1 commit into
Open
Conversation
`datahike.gc` unions `mark-from-key-map` over every stored secondary
key-map and treats the result as the reachable set. Whatever that
returns is the only thing standing between an index and the sweep.
Nothing tested it. Not at any level, for any backend: stratum's
commit-chain walk, scriptum's and proximum's `#{}`, and the union in
gc.cljc were all uncovered. This adds the first coverage — a real
file-backed collection that must leave a scriptum index searchable
afterwards, and the primary data intact.
The reason it matters now is the konserve switch. `#{}` is the honest
answer for an index whose bytes live on the filesystem, and it becomes
"delete the whole index" for one whose bytes live in konserve — the same
value, opposite meanings, no error either way. So a key-map now DECLARES
where its storage lives via `:backing`, and the default method refuses
rather than guesses: an index that says `:konserve` and has no marking
implementation raises instead of reporting itself unreachable. A broken
GC run is a cheap failure; a silently deleted index is not.
scriptum's method keys off the same declaration, so wiring it to
scriptum.konserve cannot land without the marking landing too.
GC tests must be file-backed — `:memory` refuses to mark at all ("Index
needs to be properly flushed before marking"), so a GC test on it would
prove nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
datahike.gcunionsmark-from-key-mapover every stored secondary key-map and treats the result as the reachable set (gc.cljc:99-103). Whatever that returns is the only thing standing between an index and the sweep.Nothing tested it — at any level, for any backend. Not stratum's ~40-line commit-chain walk, not scriptum's or proximum's
#{}, not the union ingc.cljc. Greppingmark-from-key-mapacrosstest/returns nothing.This adds the first coverage: a real file-backed collection that must leave a scriptum index searchable afterwards and the primary data intact.
Why it matters now
#{}is the honest answer for an index whose bytes live on the filesystem — konserve holds nothing of it to mark. It becomes "delete the whole index" for one whose bytes live in konserve. Same value, opposite meanings, and no error in either direction.That transition is imminent (replikativ/scriptum#3 moves scriptum's segments into konserve), and the failure mode is silent and total.
So a key-map now declares where its storage lives:
and the default method refuses rather than guesses — an index declaring
:konservewith no marking implementation raises instead of reporting itself unreachable. A broken GC run is a cheap failure; a silently deleted index is not.scriptum's method keys off the same declaration, so wiring it to
scriptum.konservecannot land without the marking landing too.Notes
:memoryrefuses to mark at all ("Index needs to be properly flushed before marking"), so a GC test on it proves nothing about GC. Cost me one debug cycle; worth knowing.:backingbehaves exactly as before.Not in this PR
The actual konserve marking for scriptum — returning the branch manifest plus every blob it references — lands with the wiring, once scriptum exposes a writer over its konserve Directory. The guard here is what makes that ordering enforceable instead of merely intended.