feat(konserve): back an index with konserve, branch by manifest

[whilo]

Konserve becomes the source of truth for a scriptum index; the local directory becomes a derived cache that can be deleted at any time. This is proximum's dual-storage model (proximum.vectors) applied to Lucene — and Lucene fits it better than vectors do, because segment files are write-once, so a cached file is valid forever and never needs invalidating.

The substantive change: where a branch lives

The path-based design encodes a branch as branches/<name>/ and enumerates branches with newDirectoryStream — which makes the filesystem the branch registry, the role konserve is supposed to hold. Here:

[:scriptum :manifest <branch>]  ->  {lucene-filename -> content-address}
[:scriptum :blob <address>]     ->  the bytes

Three properties follow, each with a test:

1. Forking copies a manifest. No bytes move.
2. Shared segments are one blob and one inode. The address is a content hash, and each branch view is hard links into a content-addressed pool — so a segment shared by two branches costs disk and page cache once, not twice.
3. Merging is branch-local. A merge writes new blobs under new addresses and leaves the old ones for whoever still references them.

What this makes unnecessary

• (3) is why BranchAwareMergePolicy exists
• (2) is what BranchedDirectory's base/overlay composition was for
• reachability from the live manifests replaces BranchDeletionPolicy's ref-counting

All three can go. Not in this PR — removing them is a breaking API change that deserves its own review, and I would rather land the replacement with tests first and delete against a green baseline.

Concurrency: no lock of our own

Lucene's write.lock lives in the per-branch view directory, which is scriptum's contract. Tested: a second writer on one branch fails loudly with LockObtainFailedException, writers on different branches proceed in parallel.

Two regressions pinned

Both were live in earlier drafts of this work and are worth knowing about:

• A cache keyed only by filename let Lucene see another branch's files and continue its index — branch B's durable manifest ended up containing branch A's segments. listAll is manifest-driven now; the local directory is never authoritative.
• Caching the manifest at construction left openIfChanged permanently blind, which is exactly what a remote reader polling a shared store depends on. listAll re-reads it, so a reader polls a small mutable pointer and never re-reads immutable segment data.

GC

sync is wrapped in konserve's gc-guard — it is precisely a values-then-pointer sequence, and a collection landing between the blobs and the manifest would sweep what the manifest is about to reference.

gc! blocks on the sweep rather than returning its channel; returning it unconsumed let a caller observe the store before the sweep had run (that cost me a debugging cycle). Collection is eventual: stamps are millisecond-granular and the sweep spares ties, and an explicit cutoff can only ever hold a collection back, never hurry it, since the sweep clamps to min(cutoff, safe-point).

Dependency

Needs konserve carrying konserve.gc-guard (replikativ/konserve#159, unreleased) — hence the :local alias. Full suite green with it: 48 tests, 305 assertions.

Known limitation

Whole-file materialization: a large merged segment must download completely before any read. Fine locally, and the S3 pain point to address next by porting proximum's chunking.

[whilo]

CI build-and-test is red for a known reason, not a defect in this change.

scriptum.konserve requires konserve.gc-guard, which does not exist in released konserve 0.9.350 (verified: zero matching entries in the published jar). CI runs -M:test without the :local alias, so the namespace cannot resolve and the suite fails at load.

With konserve carrying the guard (replikativ/konserve#159), the full suite is green locally:

clojure -M:local:test
Ran 48 tests containing 305 assertions.
0 failures, 0 errors.

I deliberately did not paper over this with a :git/sha dependency on the konserve branch — that would go green now and then need unwinding before release, and releases here go to Maven.

Merge order: konserve#159 → konserve release → bump the org.replikativ/konserve version in deps.edn here → CI goes green → merge this.