Explicit GC roots: liveness that does not expire - #898
Open
whilo wants to merge 1 commit into
Open
Conversation
datahike's liveness rule is reachability AND recency: `reachable-in-branch` follows ancestry only `(when in-range? parents)`. That is right when a branch head is the only pointer, but a consumer can hold pointers datahike cannot see. Geschichte is the case in hand. It stores Git refs as ordinary DATOMS and a commit's tree as a datahike commit (`:geschichte.commit/snapshot`, resolved via `commit-as-db`). datahike sees no pointer, so it falls back on its only other criterion — age — and every Geschichte commit looks like unreferenced old history. Measured: 3 commits plus a non-epoch cutoff reclaimed 75 keys, after which `repo/tree`/`status`/`tree-at` all throw "names a missing Datahike checkpoint" while `repo/read` still works. Silent, partial destruction. A root restores git's rule for such a consumer: referenced means live, regardless of age. (gc-roots store) ;; the current root set (gc-root! store commit-id) ;; declare; idempotent (gc-unroot! store commit-id) ;; drop; idempotent Roots are PERSISTED in the store, not passed to `gc-storage!`. A generic sweeper — one walking every store in a fleet without knowing what any contains — stays correct only if a store can declare its own liveness. Passing roots as an argument would push that knowledge into every caller. `gc-root!` refuses a commit-id that does not resolve: a root naming a missing commit retains nothing, and the symptom would otherwise appear much later as a store that has already lost the history the root was meant to protect. Both mutators go through `k/update`, so two concurrent declarations cannot silently drop one. With no roots present, behaviour is unchanged. WHAT THIS DOES TO remove-before Once roots exist, `remove-before` stops meaning "delete history older than X" and starts meaning what git's reflog expiry means: UNREFERENCED things older than X may go. That is the point — it is what makes a cutoff safe for a consumer whose refs datahike cannot see. FOLLOW-UP `:db.type/commit-ref` is the better long-term shape: derive roots from datoms the way `record-store-refs` already derives blob retention from `:db.type/store-ref`, so there is no second set to keep in sync and a retracted ref un-roots automatically. The `time-gated?` walk added here is what both designs need. See .internal/gc-roots.md. Tests assert the property WITH A CONTROL — a rooted commit survives a full-cutoff GC, and after `gc-unroot!` the same GC collects it, so the root is demonstrably what protected it — plus the unresolvable-commit rejection. Suite: 2410 tests, 20362 assertions, 0 failures. cljfmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
datahike's liveness rule is reachability and recency:
reachable-in-branchfollows ancestry only
(when in-range? parents). That is correct when a branchhead is the only pointer — but a consumer can hold pointers datahike cannot see,
and then age becomes the only criterion left.
Geschichte is the case in hand. It stores Git refs as ordinary datoms, and
a commit's tree as a datahike commit (
:geschichte.commit/snapshot, resolvedthrough
commit-as-db). datahike sees no pointer, so every Geschichte commitlooks like unreferenced old history.
Measured: 3 commits plus a non-epoch cutoff reclaimed 75 keys, after which
Silent, partial destruction — surfacing later, in a different operation than the
one that caused it.
What a root is
git's rule: an object is live iff reachable from a ref, regardless of age;
dates only govern a grace period for things already known to be unreachable.
A root gives a consumer that rule back.
Roots are walked with the time gate disabled — root and full ancestry stay
live regardless of
remove-before. Branches are unchanged.With no roots present, behaviour is bit-for-bit what it was.
Design notes
Persisted in the store, not passed to
gc-storage!. A generic sweeper — onewalking every store in a fleet without knowing what any of them contains — stays
correct only if a store can declare its own liveness. An argument would push
that knowledge into every caller.
gc-root!refuses an unresolvable commit-id. A root naming a missing commitretains nothing, and the symptom would otherwise appear much later as a store
that has already lost the history the root was meant to protect.
Both mutators use
k/update, so two concurrent declarations cannot silentlydrop one.
What this does to
remove-beforeOnce roots exist,
remove-beforestops meaning "delete history older than X"and starts meaning what git's reflog expiry means: unreferenced things older
than X may go. That is the point — it is what makes a cutoff safe for a consumer
whose refs datahike cannot see.
Follow-up:
:db.type/commit-refThe better long-term shape is to derive roots from datoms, the way
record-store-refsalready derives blob retention from:db.type/store-ref.Then there is no second set to keep in sync, a retracted ref un-roots
automatically, and fork refs root their own commits without anyone reasoning
about branch coverage. (Geschichte's schema comment already anticipates this.)
The
time-gated?walk added here is what both designs need, so this is astep toward that rather than an alternative to it.
Testing
2410 tests, 20362 assertions, 0 failures.cljfmtclean.The new tests assert the property with a control: a rooted commit survives a
full-cutoff GC, and after
gc-unroot!the same GC collects it — so the root isdemonstrably what protected it, not luck. Plus the unresolvable-commit rejection.