feat(gc): the safe point belongs to the store - #159
Conversation
A persistent index on konserve writes every value the new state references and only THEN the pointer that makes them reachable, so a torn write leaves collectable orphans rather than a dangling pointer. That rule is also a blind spot for the collector: inside the window the fresh values are reachable from nothing while ALREADY being older than `now`, so a sweep cutting off at `now` deletes them and the pointer lands on holes. datahike has carried the guard for this privately. It does not belong there. One konserve store commonly carries several independent index structures — a datahike database, a geschichte repository, a scriptum fulltext index — each running its own values-then-pointer sequences, and ONE sweep covers all of them. A guard held inside one cannot protect the others; the safe point has to be shared, and it is shared by agreeing on a store-id. So `sweep!` now takes `:store-id` and derives its own cutoff as min(ts, safe-point) instead of trusting the caller's `ts`. Combining those two readings is easy to get wrong — the guard must be read AFTER the collection's start instant, or a sequence opening in between is missed — and that reasoning should live once, next to the sweep, not in every caller. Without `:store-id` behaviour is unchanged. Tests assert the property, not the bookkeeping: the same sweep over the same in-flight sequence eats its values unguarded and spares them guarded, and a sweep triggered by one writer spares another writer's in-flight values on the shared store. Cutoffs in those tests are derived (successor of the high-water mark) rather than observed. The write clock is pinned to wall time, so a cutoff read straight after a write ties with it about half the time and the sweep spares ties — the first version of this test was flaky for exactly that reason. Where the clock does have to move, the global high-water mark is restored afterwards: it is `max`-based and process-wide, so leaving it advanced makes every later test stamp ahead of wall time and silently breaks anything comparing a stamp to a cutoff.
The file was .cljc but pulled `<!!` unconditionally, which does not exist in cljs — the build failed at the ns form before running anything. Split along the real line rather than demoting the file to .clj. The guard's own logic is platform-independent and now runs on both: safe-point when idle, its retreat to the oldest open sequence, and that `cutoff` never advances past the collection's start. The tests that assert the CONSEQUENCE — a sweep actually eating or sparing values — drive a store with blocking takes and stay JVM-only, because `konserve.gc/sweep!` is async-only and expressing them as cljs async tests would obscure what they demonstrate without covering anything the guard does differently there. Verified on both: 88 tests / 1312 assertions on the JVM, 50 tests / 319 assertions under node.
`validate-store-config` requires a UUID `:id` on every store and then every backend drops it. Nothing on a connected store carried it — not even a DefaultStore, whose `:config` holds the backend's behaviour options (`:in-place?`, `:lock-blob?`, `:sync-blob?`) rather than the store's identity — and backends that bypass DefaultStore entirely, like LMDB, keep less still. So anything that had to NAME a store was handed the name separately and had to keep it in agreement by hand. `sweep!` took a `:store-id` argument for exactly this reason. Two components naming one physical store differently is invisible until a collection deletes something, which is a poor property for the mechanism whose whole job is to decide what may be deleted. `PStoreIdentity` makes the store answer for itself. The default implementation reads an id that `konserve.store` attaches on connect/create — the one place every backend passes through with the full config in hand, LMDB included — so no backend has to change. A backend that would rather hold its id in a real field can implement the protocol and become authoritative. `sweep!` now derives the id and consults the guard by default. The explicit `:store-id` remains for stores built through a backend constructor directly (`connect-fs-store` and friends), which never took an id to begin with; both paths are tested, and the derived-id test proves the emptiness came from the guard by collecting the same value once the sequence closes. JVM 90 tests / 1318 assertions, node 50 / 319.
Follow-up: the store now names itself (53da620)Investigating whether The id is validated and then thrown away. (let [s (ks/create-store {:backend :file :path "..." :id id} {:sync? true})]
(:id s) ;=> nil
(get-in s [:config :id]) ;=> nil
(keys (:config s))) ;=> (:in-place? :lock-blob? :sync-blob?)A Which means anything that had to name a store was handed the name separately and had to keep it in agreement by hand. Two components naming one physical store differently is invisible until a collection deletes something — a poor property for the mechanism whose entire job is deciding what may be deleted.
|
The store now reports the config it was connected with, via `PStoreConfig`/`-store-config`, with `store-id` a convenience over it. The config is more useful than the id alone and it is what the caller supplied anyway; deriving identity from it beats attaching a lone field. It is NOT merged into `:config`. A DefaultStore already has one of those, holding the backend's behaviour options (`:in-place?`, `:lock-blob?`, `:sync-blob?`), and overwriting it would break every backend that reads it. The attached config gets its own namespaced key, which also lands in a record's extension map without disturbing declared fields — LMDBStore included. Credentials are stripped on the way in. Store configs carry secrets: `:access-key` and `:secret` for S3, `:password` and `:jdbcUrl` for JDBC. Those are fine in a config a caller passes once and forgets; they are not fine sitting on a long-lived object that any pr-str, log line or ex-info payload can carry off — and the store did not retain them before, so keeping them would have been a new leak rather than an existing one. Identity survives, which is what makes the attached config worth having. `credential-keys` is the extension point for a backend with secret-bearing keys of its own. The test asserts the stripping the way it actually matters: no secret survives into a printable form. JVM 91 tests / 1330 assertions, node 50 / 319.
The race
A persistent index on konserve writes every value the new state references and only then the pointer that makes them reachable. That ordering is what makes a torn write leave collectable orphans rather than a dangling pointer.
It is also a blind spot for the collector. Inside that window the fresh values are reachable from nothing while already being older than
now— so a sweep cutting off atnowdeletes them, and the pointer then lands on holes. A timestamp cutoff cannot save you here, because the objects genuinely predate the collection.Why it belongs in konserve
datahike has carried a private guard for this. It shouldn't.
One konserve store commonly carries several independent index structures — a datahike database, a geschichte repository, a scriptum fulltext index — each running its own values-then-pointer sequences, and one sweep covers all of them. A guard held inside one library cannot protect the others. The safe point is a property of the store, and it is shared by agreeing on a
store-id.What changed
konserve.gc-guard—writing!/done!/in-flight?/safe-point/cutoff/with-unreferenced-writes, keyed bystore-id, stamped from the existing monotonic write clock (konserve.utils/now).konserve.gc/sweep!takes:store-idand derives its own cutoff asmin(ts, safe-point)rather than trusting the caller'sts. Combining those two readings is easy to get wrong — the guard must be read after the collection's start instant, or a sequence opening in between is missed — so that reasoning lives once, next to the sweep. Without:store-idbehaviour is unchanged, so this is backwards compatible.Scope is in-process, matching konserve's concurrency contract (single writer per runtime, or coordinate above konserve). Documented on the namespace.
Tests
The tests assert the property rather than the bookkeeping: the same sweep over the same in-flight sequence eats its values unguarded and spares them guarded, and a sweep triggered by one writer spares another writer's in-flight values on a shared store.
Two things worth flagging for review, both learned the hard way here:
max-based and process-wide, so leaving it advanced makes every later test stamp ahead of wall time; unrestored, it brokemonotonic-clock-testandasync-gc-testin the same run.Full suite green three consecutive runs: 87 tests, 1310 assertions.
Follow-ups (not in this PR)
datahike should drop its local copy and call this one — in the same release, so the two cannot disagree about the clock. scriptum and proximum need the guard around their own values-then-pointer sequences.