Skip to content

Whole-batch metadata (:meta-all), and refuse a meta map that names no key - #168

Open
whilo wants to merge 4 commits into
mainfrom
meta-all
Open

Whole-batch metadata (:meta-all), and refuse a meta map that names no key#168
whilo wants to merge 4 commits into
mainfrom
meta-all

Conversation

@whilo

@whilo whilo commented Aug 12, 2026

Copy link
Copy Markdown
Member

The bug

(k/multi-assoc store kvs (k/uniform-meta (keys kvs) {:immutable? true}) opts)

silently dropped the annotation whenever the keys were 2-element vectors[:a 1], ["user" 7], [:db :root].

uniform-meta has to derive the keys from its argument, and it separates "seq of [k v] pairs" from "seq of keys" by asking whether the first element is sequential with count 2 — which such a key also is. So it returned {:a …}, (get meta [:a 1]) was nil, and every value was written without the metadata. No error.

Found while investigating why konserve-lmdb could not exploit :immutable? to skip its metadata read.

The heuristic cannot be fixed, and this PR does not try

[[:a 1]] is both a valid one-key seq and a valid one-pair seq — the same data. Narrowing the test to MapEntry was implemented and reverted: it fixes (keys m) and breaks the hand-built pair-vector instead, which only moves the failure onto a different caller. The compliance suite has one (4 failures).

So the need to guess is removed, and the guess is made non-silent.

Changes

{:meta-all m} in opts annotates the whole batch. It carries no keys, so it cannot misread them, and it is the majority case — bulk content-addressed writes. Passing both meta and :meta-all throws.

multi-assoc refuses a per-key meta disjoint from kvs (:konserve/meta-keys-disjoint). That is always a mistake and is the exact signature of a misread shape, or a typo in a hand-built map. Partial overlap stays legal — annotating a subset is the documented mixed-batch case (immutable nodes plus the mutable pointer that makes them reachable).

Hooks see one shape. :meta-all is expanded into the per-key map at the write-hook boundary and never escapes konserve.core, so konserve-sync needs no change — it keeps its (get m k). Teaching every consumer a second spelling would recreate the dropped-annotation bug one layer out. Expansion happens only when a hook is registered, so the write path still avoids building an N-entry map to say one thing.

uniform-meta is unchanged, with its docstring now stating that its shape test is a guess that cannot be made sound, and pointing at :meta-all.

Why it matters

konserve-lmdb uses :meta-all {:immutable? true} to skip the metadata read-modify-write that konserve's contract otherwise requires — a content-addressed value is written once, so there is no prior metadata worth merging:

µs/key batch put
ordinary batch 52.6 91.8K ops/s
:meta-all {:immutable? true} 27.0 163K ops/s (1.78x)

That is datahike's node-storage shape.

Verification

123 tests, 1399 assertions, 0 failures. Behaviour verified end to end against konserve-lmdb, which is multi-key capable where the memory store is not (Store does not support multi-key operations, so the guard is unreachable there):

  • disjoint meta raises :konserve/meta-keys-disjoint
  • :meta-all round-trips to get-meta
  • subset annotation still allowed

Scope note

This branch also carries three pre-existing local commits that were not on origin/main: ada3aa6 (boring 0.1.17), 57edebd (serializers: encode-indexed), 1c5c9e8 (README backends). Happy to rebase them out if you would rather they land separately.

whilo added 4 commits August 5, 2026 04:07
konserve-gcs sat under "Unofficial Backends", pointing at
The-Literal-Company/konserve-gcs, while the repository it is developed in is
replikativ/konserve-gcs. Moved to the supported list with the right URL, and
added to the reference-implementations sentence alongside konserve-jdbc, which
was also missing from it.

The unsupported-backend error listed the backends too -- in FOUR copies, one
per `:default` multimethod -- and all four had already drifted: neither :jdbc
nor :gcs appeared in any of them, while both are listed in the README as
available. An error message that names an incomplete set sends someone looking
for a backend that exists. One `known-backends` string now, four call sites.

Found because an audit of "the backends the README lists" missed konserve-gcs
entirely: it was in the README, just in a section that read as unmaintained.

123 tests / 1398 assertions green.
… frame

The boring serializer used `write-indexed!` on the JVM, on the grounds that
`encode-indexed` "builds the whole array and then WALKS it to derive the index
-- two passes and two copies". That has not been true since boring consolidated
its index builders: `encode-indexed` IS `write-indexed!` into a
ByteArrayOutputStream and captures nodes from the writer the same way. There
was no second pass left to avoid, and this store stages into a
ByteArrayOutputStream regardless, so nothing was being streamed either.

What the buffered entry point can do that the streaming one cannot is decline
an index frame that describes nothing. An indexed write seals a frame whenever
it opens a stringref namespace, even with no container worth a node, because
"namespace with no pointer table" has to keep meaning one thing for
`boring.nav`. On a large value that is noise; on the small values a KV store is
mostly made of it is the whole file.

Through this serializer:

  {:a 1}                    50 -> 7 bytes
  {}                        44 -> 1
  {:id 1 :name "alice"}     65 -> 22
  20 konserve-shaped values 3195 -> 2514, 21% smaller

Large values are untouched -- they have containers, so the frame was always
earning its bytes. The +0.8% / +0.3% / +4.4% figures in `indexing?` re-measure
identically.

Also bumps boring to 0.1.16 and fixes one test for boring's source/root split:
`(nav/source bs)` returns a source now, and a source is deliberately not a
cursor. The small-value assertion that was failing needed no change -- it
asserted the plain encoding, which is exactly what this restores.
Picks up the reader-side fixes made since 0.1.16: the stringref binary search
(a repeated key is written as a reference, and the probe is built as a literal,
so every present key was missed by the search and rescued by a full rescan),
the degenerate index frame dropped in both writers, and typed failures from
registered handlers.

Suite green against it: 123 tests, 1398 assertions.
…a map

WHAT WENT WRONG. `(multi-assoc store kvs (uniform-meta (keys kvs) {:immutable?
true}) opts)` silently dropped the annotation whenever the keys were
2-element vectors. `uniform-meta` has to DERIVE the keys from its argument,
and it decides "seq of [k v] pairs" vs "seq of keys" by asking whether the
first element is sequential with count 2 -- which a key like `[:a 1]` also
is. So it returned `{:a ..}`, `(get meta [:a 1])` was nil, and every value
was written without the metadata. No error. Found while chasing why
konserve-lmdb could not exploit `:immutable?`.

THE HEURISTIC CANNOT BE FIXED, and this commit does not try. `[[:a 1]]` is
both a valid one-key seq and a valid one-pair seq; the two are the same
data. Narrowing the test to `MapEntry` was implemented and reverted -- it
fixes `(keys m)` and breaks the hand-built pair-vector instead, which only
moves the failure onto a different caller, and the compliance suite has one
(`:ord-a`/`:ord-b`, four failures).

SO: REMOVE THE NEED TO GUESS, AND MAKE THE GUESS NON-SILENT.

1. `{:meta-all m}` in opts annotates the whole batch. It needs no keys at
   all, so it cannot get them wrong, and it is the majority case -- bulk
   content-addressed writes. Passing both `meta` and `:meta-all` throws.

2. `multi-assoc` refuses a per-key `meta` whose keys are DISJOINT from
   `kvs`. That is always a mistake and it is the exact signature of a
   misread shape, or of a typo in a hand-built map. Partial overlap is
   allowed and must stay allowed: annotating a SUBSET is the documented
   mixed-batch case, immutable nodes plus the mutable pointer that makes
   them reachable.

HOOKS SEE ONE SHAPE. `:meta-all` is expanded into the per-key map at the
write-hook boundary and never escapes this namespace, so konserve-sync keeps
its `(get m k)` and needs no change. Teaching every consumer a second
spelling would recreate the dropped-annotation bug one layer out, in any
consumer not updated. Expansion happens only when a hook is registered, so
the write path still avoids building an N-entry map to say one thing.

Verified against konserve-lmdb, which is multi-key capable where the memory
store is not: disjoint raises :konserve/meta-keys-disjoint, `:meta-all`
round-trips to `get-meta`, subset annotation still allowed.

123 tests, 1399 assertions, 0 failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant