Conversation
konserve-gcs sat under "Unofficial Backends", pointing at The-Literal-Company/konserve-gcs, while the repository it is developed in is replikativ/konserve-gcs. Moved to the supported list with the right URL, and added to the reference-implementations sentence alongside konserve-jdbc, which was also missing from it. The unsupported-backend error listed the backends too -- in FOUR copies, one per `:default` multimethod -- and all four had already drifted: neither :jdbc nor :gcs appeared in any of them, while both are listed in the README as available. An error message that names an incomplete set sends someone looking for a backend that exists. One `known-backends` string now, four call sites. Found because an audit of "the backends the README lists" missed konserve-gcs entirely: it was in the README, just in a section that read as unmaintained. 123 tests / 1398 assertions green.
… frame
The boring serializer used `write-indexed!` on the JVM, on the grounds that
`encode-indexed` "builds the whole array and then WALKS it to derive the index
-- two passes and two copies". That has not been true since boring consolidated
its index builders: `encode-indexed` IS `write-indexed!` into a
ByteArrayOutputStream and captures nodes from the writer the same way. There
was no second pass left to avoid, and this store stages into a
ByteArrayOutputStream regardless, so nothing was being streamed either.
What the buffered entry point can do that the streaming one cannot is decline
an index frame that describes nothing. An indexed write seals a frame whenever
it opens a stringref namespace, even with no container worth a node, because
"namespace with no pointer table" has to keep meaning one thing for
`boring.nav`. On a large value that is noise; on the small values a KV store is
mostly made of it is the whole file.
Through this serializer:
{:a 1} 50 -> 7 bytes
{} 44 -> 1
{:id 1 :name "alice"} 65 -> 22
20 konserve-shaped values 3195 -> 2514, 21% smaller
Large values are untouched -- they have containers, so the frame was always
earning its bytes. The +0.8% / +0.3% / +4.4% figures in `indexing?` re-measure
identically.
Also bumps boring to 0.1.16 and fixes one test for boring's source/root split:
`(nav/source bs)` returns a source now, and a source is deliberately not a
cursor. The small-value assertion that was failing needed no change -- it
asserted the plain encoding, which is exactly what this restores.
Picks up the reader-side fixes made since 0.1.16: the stringref binary search (a repeated key is written as a reference, and the probe is built as a literal, so every present key was missed by the search and rescued by a full rescan), the degenerate index frame dropped in both writers, and typed failures from registered handlers. Suite green against it: 123 tests, 1398 assertions.
…a map
WHAT WENT WRONG. `(multi-assoc store kvs (uniform-meta (keys kvs) {:immutable?
true}) opts)` silently dropped the annotation whenever the keys were
2-element vectors. `uniform-meta` has to DERIVE the keys from its argument,
and it decides "seq of [k v] pairs" vs "seq of keys" by asking whether the
first element is sequential with count 2 -- which a key like `[:a 1]` also
is. So it returned `{:a ..}`, `(get meta [:a 1])` was nil, and every value
was written without the metadata. No error. Found while chasing why
konserve-lmdb could not exploit `:immutable?`.
THE HEURISTIC CANNOT BE FIXED, and this commit does not try. `[[:a 1]]` is
both a valid one-key seq and a valid one-pair seq; the two are the same
data. Narrowing the test to `MapEntry` was implemented and reverted -- it
fixes `(keys m)` and breaks the hand-built pair-vector instead, which only
moves the failure onto a different caller, and the compliance suite has one
(`:ord-a`/`:ord-b`, four failures).
SO: REMOVE THE NEED TO GUESS, AND MAKE THE GUESS NON-SILENT.
1. `{:meta-all m}` in opts annotates the whole batch. It needs no keys at
all, so it cannot get them wrong, and it is the majority case -- bulk
content-addressed writes. Passing both `meta` and `:meta-all` throws.
2. `multi-assoc` refuses a per-key `meta` whose keys are DISJOINT from
`kvs`. That is always a mistake and it is the exact signature of a
misread shape, or of a typo in a hand-built map. Partial overlap is
allowed and must stay allowed: annotating a SUBSET is the documented
mixed-batch case, immutable nodes plus the mutable pointer that makes
them reachable.
HOOKS SEE ONE SHAPE. `:meta-all` is expanded into the per-key map at the
write-hook boundary and never escapes this namespace, so konserve-sync keeps
its `(get m k)` and needs no change. Teaching every consumer a second
spelling would recreate the dropped-annotation bug one layer out, in any
consumer not updated. Expansion happens only when a hook is registered, so
the write path still avoids building an N-entry map to say one thing.
Verified against konserve-lmdb, which is multi-key capable where the memory
store is not: disjoint raises :konserve/meta-keys-disjoint, `:meta-all`
round-trips to `get-meta`, subset annotation still allowed.
123 tests, 1399 assertions, 0 failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
silently dropped the annotation whenever the keys were 2-element vectors —
[:a 1],["user" 7],[:db :root].uniform-metahas to derive the keys from its argument, and it separates "seq of[k v]pairs" from "seq of keys" by asking whether the first element is sequential with count 2 — which such a key also is. So it returned{:a …},(get meta [:a 1])was nil, and every value was written without the metadata. No error.Found while investigating why konserve-lmdb could not exploit
:immutable?to skip its metadata read.The heuristic cannot be fixed, and this PR does not try
[[:a 1]]is both a valid one-key seq and a valid one-pair seq — the same data. Narrowing the test toMapEntrywas implemented and reverted: it fixes(keys m)and breaks the hand-built pair-vector instead, which only moves the failure onto a different caller. The compliance suite has one (4 failures).So the need to guess is removed, and the guess is made non-silent.
Changes
{:meta-all m}in opts annotates the whole batch. It carries no keys, so it cannot misread them, and it is the majority case — bulk content-addressed writes. Passing bothmetaand:meta-allthrows.multi-assocrefuses a per-keymetadisjoint fromkvs(:konserve/meta-keys-disjoint). That is always a mistake and is the exact signature of a misread shape, or a typo in a hand-built map. Partial overlap stays legal — annotating a subset is the documented mixed-batch case (immutable nodes plus the mutable pointer that makes them reachable).Hooks see one shape.
:meta-allis expanded into the per-key map at the write-hook boundary and never escapeskonserve.core, so konserve-sync needs no change — it keeps its(get m k). Teaching every consumer a second spelling would recreate the dropped-annotation bug one layer out. Expansion happens only when a hook is registered, so the write path still avoids building an N-entry map to say one thing.uniform-metais unchanged, with its docstring now stating that its shape test is a guess that cannot be made sound, and pointing at:meta-all.Why it matters
konserve-lmdb uses
:meta-all {:immutable? true}to skip the metadata read-modify-write that konserve's contract otherwise requires — a content-addressed value is written once, so there is no prior metadata worth merging::meta-all {:immutable? true}That is datahike's node-storage shape.
Verification
123 tests, 1399 assertions, 0 failures. Behaviour verified end to end against konserve-lmdb, which is multi-key capable where the memory store is not (
Store does not support multi-key operations, so the guard is unreachable there):metaraises:konserve/meta-keys-disjoint:meta-allround-trips toget-metaScope note
This branch also carries three pre-existing local commits that were not on
origin/main:ada3aa6(boring 0.1.17),57edebd(serializers: encode-indexed),1c5c9e8(README backends). Happy to rebase them out if you would rather they land separately.