Skip to content

Explicit GC roots: liveness that does not expire - #898

Open
whilo wants to merge 1 commit into
mainfrom
feat/gc-roots
Open

Explicit GC roots: liveness that does not expire#898
whilo wants to merge 1 commit into
mainfrom
feat/gc-roots

Conversation

@whilo

@whilo whilo commented Jul 21, 2026

Copy link
Copy Markdown
Member

The problem

datahike's liveness rule is reachability and recency: reachable-in-branch
follows ancestry only (when in-range? parents). That is correct when a branch
head is the only pointer — but a consumer can hold pointers datahike cannot see,
and then age becomes the only criterion left.

Geschichte is the case in hand. It stores Git refs as ordinary datoms, and
a commit's tree as a datahike commit (:geschichte.commit/snapshot, resolved
through commit-as-db). datahike sees no pointer, so every Geschichte commit
looks like unreferenced old history.

Measured: 3 commits plus a non-epoch cutoff reclaimed 75 keys, after which

repo/tree     -> throws "Geschichte commit names a missing Datahike checkpoint"
repo/status   -> throws
repo/tree-at  -> throws
repo/read     -> still works

Silent, partial destruction — surfacing later, in a different operation than the
one that caused it.

What a root is

git's rule: an object is live iff reachable from a ref, regardless of age;
dates only govern a grace period for things already known to be unreachable.
A root gives a consumer that rule back.

(gc-roots store)              ;; the current root set
(gc-root! store commit-id)    ;; declare; idempotent
(gc-unroot! store commit-id)  ;; drop; idempotent

Roots are walked with the time gate disabled — root and full ancestry stay
live regardless of remove-before. Branches are unchanged.

With no roots present, behaviour is bit-for-bit what it was.

Design notes

Persisted in the store, not passed to gc-storage!. A generic sweeper — one
walking every store in a fleet without knowing what any of them contains — stays
correct only if a store can declare its own liveness. An argument would push
that knowledge into every caller.

gc-root! refuses an unresolvable commit-id. A root naming a missing commit
retains nothing, and the symptom would otherwise appear much later as a store
that has already lost the history the root was meant to protect.

Both mutators use k/update, so two concurrent declarations cannot silently
drop one.

What this does to remove-before

Once roots exist, remove-before stops meaning "delete history older than X"
and starts meaning what git's reflog expiry means: unreferenced things older
than X may go. That is the point — it is what makes a cutoff safe for a consumer
whose refs datahike cannot see.

Follow-up: :db.type/commit-ref

The better long-term shape is to derive roots from datoms, the way
record-store-refs already derives blob retention from :db.type/store-ref.
Then there is no second set to keep in sync, a retracted ref un-roots
automatically, and fork refs root their own commits without anyone reasoning
about branch coverage. (Geschichte's schema comment already anticipates this.)

The time-gated? walk added here is what both designs need, so this is a
step toward that rather than an alternative to it.

Testing

2410 tests, 20362 assertions, 0 failures. cljfmt clean.

The new tests assert the property with a control: a rooted commit survives a
full-cutoff GC, and after gc-unroot! the same GC collects it — so the root is
demonstrably what protected it, not luck. Plus the unresolvable-commit rejection.

datahike's liveness rule is reachability AND recency: `reachable-in-branch`
follows ancestry only `(when in-range? parents)`. That is right when a branch
head is the only pointer, but a consumer can hold pointers datahike cannot see.

Geschichte is the case in hand. It stores Git refs as ordinary DATOMS and a
commit's tree as a datahike commit (`:geschichte.commit/snapshot`, resolved via
`commit-as-db`). datahike sees no pointer, so it falls back on its only other
criterion — age — and every Geschichte commit looks like unreferenced old
history. Measured: 3 commits plus a non-epoch cutoff reclaimed 75 keys, after
which `repo/tree`/`status`/`tree-at` all throw "names a missing Datahike
checkpoint" while `repo/read` still works. Silent, partial destruction.

A root restores git's rule for such a consumer: referenced means live,
regardless of age.

  (gc-roots store)              ;; the current root set
  (gc-root! store commit-id)    ;; declare; idempotent
  (gc-unroot! store commit-id)  ;; drop; idempotent

Roots are PERSISTED in the store, not passed to `gc-storage!`. A generic
sweeper — one walking every store in a fleet without knowing what any contains —
stays correct only if a store can declare its own liveness. Passing roots as an
argument would push that knowledge into every caller.

`gc-root!` refuses a commit-id that does not resolve: a root naming a missing
commit retains nothing, and the symptom would otherwise appear much later as a
store that has already lost the history the root was meant to protect. Both
mutators go through `k/update`, so two concurrent declarations cannot silently
drop one.

With no roots present, behaviour is unchanged.

WHAT THIS DOES TO remove-before

Once roots exist, `remove-before` stops meaning "delete history older than X"
and starts meaning what git's reflog expiry means: UNREFERENCED things older
than X may go. That is the point — it is what makes a cutoff safe for a consumer
whose refs datahike cannot see.

FOLLOW-UP

`:db.type/commit-ref` is the better long-term shape: derive roots from datoms
the way `record-store-refs` already derives blob retention from
`:db.type/store-ref`, so there is no second set to keep in sync and a retracted
ref un-roots automatically. The `time-gated?` walk added here is what both
designs need. See .internal/gc-roots.md.

Tests assert the property WITH A CONTROL — a rooted commit survives a
full-cutoff GC, and after `gc-unroot!` the same GC collects it, so the root is
demonstrably what protected it — plus the unresolvable-commit rejection.

Suite: 2410 tests, 20362 assertions, 0 failures. cljfmt clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant