[josaum]

The bug

build_embedding_array in src/kb.rs fed entry.is_none() into Arrow's
validity bitmap. That bitmap is true = VALID, so the mapping was
inverted: every successfully embedded row was stored behind a null mask, and
every unembedded row was marked valid.

This is pre-existing and independent of #12, which touched embed.rs,
config.rs, node_rows.rs and export.rs but never src/kb.rs. #12 did not
introduce it; it is simply what you hit next once the encoder width became
configurable and kb build actually produced vectors.

Why it never surfaced

Every observable signal said the collection was embedded:

• kb build reported the full count — embedded: 67 of 67 rows
• lexical_only: false
• the schema carried fixed_size_list<item: float>[768] with the right width
• embedding_model was stamped correctly on every row

...while kb query returned "cosine": null on every hit. Anyone reading the
build output would conclude vector search was working. row_vector reads
through the null mask, finds nothing at row 0, and the
if row_vector(&batch, 0).is_some() guard then skips embedding the query
entirely — so the encoder was never even contacted at query time.

The result is a silent, permanent fall back to lexical ranking on a collection
that embedded cleanly. That invisibility is the main argument for the
regression test added here. Confirmed by reading the stored IPC stream
directly:

rows 67 | embedding null count: 67
schema: embedding: fixed_size_list<item: float>[768]
embedding_model: {'google/embeddinggemma-2'}

This is write-side only, so a binary with the fix reads an old store fine, but
re-running kb build on an unfixed binary reproduces the null vectors.

The fix

1. Map entry.is_some() into the validity bitmap.
2. Pad the values buffer with dim zeros for null slots. A
   FixedSizeListArray indexes its values by position, so a partially embedded
   collection previously shifted every row after the first gap.

Before / after

Same collection, same query — "how are silver models tested" — against a
local google/embeddinggemma-2 encoder at 768-d.

Before (vectors present but masked null, so lexical only):

#	score	cosine	hit
1	1.5076	null	dbt/STYLE_GUIDE.md
2	1.3723	null	docs/ADF_DEPLOY_DESIGN.md > Known gaps
3	1.2276	null	dbt/STYLE_GUIDE.md > Layers
4	1.2276	null	dbt/STYLE_GUIDE.md > Project configuration

After:

#	score	cosine	hit
1	11.2937	0.6711	lean/README.md — EdpMedallionProofs
2	11.1264	0.6599	dbt/STYLE_GUIDE.md > Project configuration
3	11.0708	0.6375	dbt/STYLE_GUIDE.md
4	11.0640	0.7084	dbt/STYLE_GUIDE.md > **Silver layer**

Lexical matched on "models"/"style" and returned three slices of the same
style guide, none of them about testing. The vector arm surfaces the Silver
layer section — the highest cosine of the four — and the Lean medallion
proofs, neither of which lexical ranked at all.

Verification

Regression test kb::embedding_array_marks_embedded_rows_valid covers both
halves: the validity of embedded vs unembedded rows, and that a null slot does
not shift the row after it. It fails on the previous implementation.

Full suite green (555 lib tests + all integration targets), cargo fmt --check and cargo clippy --all-targets clean.

Post-fix store:

rows 67 | embedding nulls: 0
row0 dim: 768   row0 norm: 1.0027

Related but not fixed here, worth its own issue: kb build stamps
embedding_model into the collection meta even when embedded == 0, which
makes a lexical-only collection look embedded.

🤖 Generated with Claude Code