Skip to content

feat(gc): the safe point belongs to the store - #159

Open
whilo wants to merge 4 commits into
mainfrom
feat/gc-guard
Open

feat(gc): the safe point belongs to the store#159
whilo wants to merge 4 commits into
mainfrom
feat/gc-guard

Conversation

@whilo

@whilo whilo commented Jul 25, 2026

Copy link
Copy Markdown
Member

The race

A persistent index on konserve writes every value the new state references and only then the pointer that makes them reachable. That ordering is what makes a torn write leave collectable orphans rather than a dangling pointer.

It is also a blind spot for the collector. Inside that window the fresh values are reachable from nothing while already being older than now — so a sweep cutting off at now deletes them, and the pointer then lands on holes. A timestamp cutoff cannot save you here, because the objects genuinely predate the collection.

Why it belongs in konserve

datahike has carried a private guard for this. It shouldn't.

One konserve store commonly carries several independent index structures — a datahike database, a geschichte repository, a scriptum fulltext index — each running its own values-then-pointer sequences, and one sweep covers all of them. A guard held inside one library cannot protect the others. The safe point is a property of the store, and it is shared by agreeing on a store-id.

What changed

  • konserve.gc-guardwriting! / done! / in-flight? / safe-point / cutoff / with-unreferenced-writes, keyed by store-id, stamped from the existing monotonic write clock (konserve.utils/now).
  • konserve.gc/sweep! takes :store-id and derives its own cutoff as min(ts, safe-point) rather than trusting the caller's ts. Combining those two readings is easy to get wrong — the guard must be read after the collection's start instant, or a sequence opening in between is missed — so that reasoning lives once, next to the sweep. Without :store-id behaviour is unchanged, so this is backwards compatible.

Scope is in-process, matching konserve's concurrency contract (single writer per runtime, or coordinate above konserve). Documented on the namespace.

Tests

The tests assert the property rather than the bookkeeping: the same sweep over the same in-flight sequence eats its values unguarded and spares them guarded, and a sweep triggered by one writer spares another writer's in-flight values on a shared store.

Two things worth flagging for review, both learned the hard way here:

  • Cutoffs are derived (successor of the high-water mark), not observed. The write clock is pinned to wall time, so a cutoff read straight after a write ties with it about half the time and the sweep spares ties — the first version of this test was flaky for exactly that reason.
  • Where the clock does have to move, the global high-water mark is restored afterwards. It is max-based and process-wide, so leaving it advanced makes every later test stamp ahead of wall time; unrestored, it broke monotonic-clock-test and async-gc-test in the same run.

Full suite green three consecutive runs: 87 tests, 1310 assertions.

Follow-ups (not in this PR)

datahike should drop its local copy and call this one — in the same release, so the two cannot disagree about the clock. scriptum and proximum need the guard around their own values-then-pointer sequences.

A persistent index on konserve writes every value the new state
references and only THEN the pointer that makes them reachable, so a
torn write leaves collectable orphans rather than a dangling pointer.
That rule is also a blind spot for the collector: inside the window the
fresh values are reachable from nothing while ALREADY being older than
`now`, so a sweep cutting off at `now` deletes them and the pointer
lands on holes.

datahike has carried the guard for this privately. It does not belong
there. One konserve store commonly carries several independent index
structures — a datahike database, a geschichte repository, a scriptum
fulltext index — each running its own values-then-pointer sequences,
and ONE sweep covers all of them. A guard held inside one cannot
protect the others; the safe point has to be shared, and it is shared
by agreeing on a store-id.

So `sweep!` now takes `:store-id` and derives its own cutoff as
min(ts, safe-point) instead of trusting the caller's `ts`. Combining
those two readings is easy to get wrong — the guard must be read AFTER
the collection's start instant, or a sequence opening in between is
missed — and that reasoning should live once, next to the sweep, not
in every caller. Without `:store-id` behaviour is unchanged.

Tests assert the property, not the bookkeeping: the same sweep over the
same in-flight sequence eats its values unguarded and spares them
guarded, and a sweep triggered by one writer spares another writer's
in-flight values on the shared store.

Cutoffs in those tests are derived (successor of the high-water mark)
rather than observed. The write clock is pinned to wall time, so a
cutoff read straight after a write ties with it about half the time and
the sweep spares ties — the first version of this test was flaky for
exactly that reason. Where the clock does have to move, the global
high-water mark is restored afterwards: it is `max`-based and
process-wide, so leaving it advanced makes every later test stamp ahead
of wall time and silently breaks anything comparing a stamp to a cutoff.
whilo added 2 commits July 24, 2026 18:06
The file was .cljc but pulled `<!!` unconditionally, which does not
exist in cljs — the build failed at the ns form before running anything.

Split along the real line rather than demoting the file to .clj. The
guard's own logic is platform-independent and now runs on both:
safe-point when idle, its retreat to the oldest open sequence, and that
`cutoff` never advances past the collection's start. The tests that
assert the CONSEQUENCE — a sweep actually eating or sparing values —
drive a store with blocking takes and stay JVM-only, because
`konserve.gc/sweep!` is async-only and expressing them as cljs async
tests would obscure what they demonstrate without covering anything the
guard does differently there.

Verified on both: 88 tests / 1312 assertions on the JVM, 50 tests / 319
assertions under node.
`validate-store-config` requires a UUID `:id` on every store and then
every backend drops it. Nothing on a connected store carried it — not
even a DefaultStore, whose `:config` holds the backend's behaviour
options (`:in-place?`, `:lock-blob?`, `:sync-blob?`) rather than the
store's identity — and backends that bypass DefaultStore entirely, like
LMDB, keep less still.

So anything that had to NAME a store was handed the name separately and
had to keep it in agreement by hand. `sweep!` took a `:store-id`
argument for exactly this reason. Two components naming one physical
store differently is invisible until a collection deletes something,
which is a poor property for the mechanism whose whole job is to decide
what may be deleted.

`PStoreIdentity` makes the store answer for itself. The default
implementation reads an id that `konserve.store` attaches on
connect/create — the one place every backend passes through with the
full config in hand, LMDB included — so no backend has to change. A
backend that would rather hold its id in a real field can implement the
protocol and become authoritative.

`sweep!` now derives the id and consults the guard by default. The
explicit `:store-id` remains for stores built through a backend
constructor directly (`connect-fs-store` and friends), which never took
an id to begin with; both paths are tested, and the derived-id test
proves the emptiness came from the guard by collecting the same value
once the sequence closes.

JVM 90 tests / 1318 assertions, node 50 / 319.
@whilo

whilo commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Follow-up: the store now names itself (53da620)

Investigating whether sweep!'s :store-id argument could come from the store's config instead turned up something better and slightly alarming.

The id is validated and then thrown away. validate-store-config requires a UUID :id on every store, and no backend retains it. Verified empirically:

(let [s (ks/create-store {:backend :file :path "..." :id id} {:sync? true})]
  (:id s)               ;=> nil
  (get-in s [:config :id])  ;=> nil
  (keys (:config s)))   ;=> (:in-place? :lock-blob? :sync-blob?)

A DefaultStore's :config holds the backend's behaviour options, not the store's identity — the UUID appears nowhere on the record at all. And this isn't a backend gap: konserve-lmdb has its own LMDBStore that explicitly "Bypasses DefaultStore", so there was no shared field to read even in principle.

Which means anything that had to name a store was handed the name separately and had to keep it in agreement by hand. Two components naming one physical store differently is invisible until a collection deletes something — a poor property for the mechanism whose entire job is deciding what may be deleted.

PStoreIdentity

Added to konserve.protocols, following the PMultiKeySupport idiom (protocol + Object default):

(p/-store-id store)   ; or konserve.store/store-id

The default implementation reads an id that konserve.store/connect-store/create-store attach — the one place every backend passes through with the full config in hand, LMDB included, since dispatch happens there. So no backend has to change. A backend that would rather hold its id in a real field can implement the protocol and become authoritative.

Effect on sweep!

It now derives the id and consults the guard by default. :store-id remains for stores built through a backend constructor directly (connect-fs-store and friends), which never took an id to begin with.

Both paths are tested. The derived-id test proves the emptiness came from the guard rather than from a sweep that never ran, by collecting the same value once the sequence closes.

JVM 90 tests / 1318 assertions, node 50 / 319 — both green.

The store now reports the config it was connected with, via
`PStoreConfig`/`-store-config`, with `store-id` a convenience over it.
The config is more useful than the id alone and it is what the caller
supplied anyway; deriving identity from it beats attaching a lone field.

It is NOT merged into `:config`. A DefaultStore already has one of
those, holding the backend's behaviour options (`:in-place?`,
`:lock-blob?`, `:sync-blob?`), and overwriting it would break every
backend that reads it. The attached config gets its own namespaced key,
which also lands in a record's extension map without disturbing declared
fields — LMDBStore included.

Credentials are stripped on the way in. Store configs carry secrets:
`:access-key` and `:secret` for S3, `:password` and `:jdbcUrl` for JDBC.
Those are fine in a config a caller passes once and forgets; they are
not fine sitting on a long-lived object that any pr-str, log line or
ex-info payload can carry off — and the store did not retain them
before, so keeping them would have been a new leak rather than an
existing one. Identity survives, which is what makes the attached config
worth having. `credential-keys` is the extension point for a backend
with secret-bearing keys of its own.

The test asserts the stripping the way it actually matters: no secret
survives into a printable form.

JVM 91 tests / 1330 assertions, node 50 / 319.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant