Skip to content

fix(gc): a secondary index must not be silently collectable - #900

Open
whilo wants to merge 1 commit into
mainfrom
feat/scriptum-konserve-gc
Open

fix(gc): a secondary index must not be silently collectable#900
whilo wants to merge 1 commit into
mainfrom
feat/scriptum-konserve-gc

Conversation

@whilo

@whilo whilo commented Jul 25, 2026

Copy link
Copy Markdown
Member

The gap

datahike.gc unions mark-from-key-map over every stored secondary key-map and treats the result as the reachable set (gc.cljc:99-103). Whatever that returns is the only thing standing between an index and the sweep.

Nothing tested it — at any level, for any backend. Not stratum's ~40-line commit-chain walk, not scriptum's or proximum's #{}, not the union in gc.cljc. Grepping mark-from-key-map across test/ returns nothing.

This adds the first coverage: a real file-backed collection that must leave a scriptum index searchable afterwards and the primary data intact.

Why it matters now

#{} is the honest answer for an index whose bytes live on the filesystem — konserve holds nothing of it to mark. It becomes "delete the whole index" for one whose bytes live in konserve. Same value, opposite meanings, and no error in either direction.

That transition is imminent (replikativ/scriptum#3 moves scriptum's segments into konserve), and the failure mode is silent and total.

So a key-map now declares where its storage lives:

(sec/konserve-backed? key-map)   ; => (= :konserve (:backing key-map))

and the default method refuses rather than guesses — an index declaring :konserve with no marking implementation raises instead of reporting itself unreachable. A broken GC run is a cheap failure; a silently deleted index is not.

scriptum's method keys off the same declaration, so wiring it to scriptum.konserve cannot land without the marking landing too.

Notes

  • GC tests must be file-backed. :memory refuses to mark at all ("Index needs to be properly flushed before marking"), so a GC test on it proves nothing about GC. Cost me one debug cycle; worth knowing.
  • Backwards compatible: a key-map without :backing behaves exactly as before.
  • Full secondary suite green: 87 tests, 396 assertions.

Not in this PR

The actual konserve marking for scriptum — returning the branch manifest plus every blob it references — lands with the wiring, once scriptum exposes a writer over its konserve Directory. The guard here is what makes that ordering enforceable instead of merely intended.

`datahike.gc` unions `mark-from-key-map` over every stored secondary
key-map and treats the result as the reachable set. Whatever that
returns is the only thing standing between an index and the sweep.

Nothing tested it. Not at any level, for any backend: stratum's
commit-chain walk, scriptum's and proximum's `#{}`, and the union in
gc.cljc were all uncovered. This adds the first coverage — a real
file-backed collection that must leave a scriptum index searchable
afterwards, and the primary data intact.

The reason it matters now is the konserve switch. `#{}` is the honest
answer for an index whose bytes live on the filesystem, and it becomes
"delete the whole index" for one whose bytes live in konserve — the same
value, opposite meanings, no error either way. So a key-map now DECLARES
where its storage lives via `:backing`, and the default method refuses
rather than guesses: an index that says `:konserve` and has no marking
implementation raises instead of reporting itself unreachable. A broken
GC run is a cheap failure; a silently deleted index is not.

scriptum's method keys off the same declaration, so wiring it to
scriptum.konserve cannot land without the marking landing too.

GC tests must be file-backed — `:memory` refuses to mark at all ("Index
needs to be properly flushed before marking"), so a GC test on it would
prove nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant