Add compact() to reclaim soft-deleted nodes

[danielebarbaro]

Closes #27.

Soft deleted HNSW nodes are never removed, so the graph and hnsw.bin grow with every deleteDocument() and updateDocument().

Changes

• VectorDatabase::compact(): int rebuilds the HNSW graph from active documents using their stored vectors (no re-embedding) and returns the number of reclaimed nodes. Persistent databases are saved under the folder lock. No deletions means no rebuild and no write.
• VectorDatabase::deletedCount(): int, to compact above a threshold.
• HNSW\Index::compact(), HNSW\Index::deletedCount(), and an optional explicit node ID on HNSW\Index::insert(). VectorDatabase now passes its own node ID instead of relying on both counters staying in lockstep.

Notes

• Active nodes keep their node IDs. Renumbering would mean renaming docs/{n}.bin files, which cannot be done atomically with the index files. The serializer already stores node IDs explicitly, so there is no format change.
• BM25 is not rebuilt: removeDocument() already removes documents fully.
• The rebuild costs about as much as a fresh build, so it belongs in a maintenance job.

Not included

• Reusing deleted slots on insert. It would conflict with tombstone recovery in open(), and compact() covers the need.
• save() writes meta.json before hnsw.bin, so a crash in between can make deleted nodes searchable again. The window predates this PR and is tracked in save() is not atomic across meta.json, hnsw.bin and bm25.bin #28.

[github-actions]

Benchmark Comparison

🔴 Significant regressions detected (>5%) — 34 metrics compared

📊 marks 11 throughput metrics that moved within the run to run variance of a shared runner. Only a drop beyond 50% counts as a regression there. Memory and disk figures repeat exactly between runs, so they gate at 5%.

xs

Metric	Baseline	Current	Delta	Status
insert (ops/s)	19.20 ops/s	17.83 ops/s	-7.1%	📊
insert (memory delta)	48.00 MB	48.00 MB	+0.0%	🟢
vector_search (QPS)	224.27 queries/s	222.19 queries/s	-0.9%	📊
vector_search (memory delta)	0.00 MB	0.00 MB	N/A	🟢
text_search (QPS)	429.49 queries/s	426.02 queries/s	-0.8%	📊
text_search (memory delta)	2.00 MB	2.00 MB	+0.0%	🟢
hybrid_search (QPS)	128.25 queries/s	122.00 queries/s	-4.9%	📊
hybrid_search (memory delta)	0.00 MB	0.00 MB	N/A	🟢
update (ops/s)	16.62 ops/s	15.92 ops/s	-4.2%	📊
update (memory delta)	8.00 MB	8.00 MB	+0.0%	🟢
delete (ops/s)	728,397.90 ops/s	815,468.46 ops/s	+12.0%	🟢
delete (memory delta)	0.00 MB	0.00 MB	N/A	🟢
save (MB/s)	0.02 MB/s	0.02 MB/s	+0.0%	🟢
save (disk size)	13.12 MB	13.12 MB	+0.0%	🟢
save (memory delta)	56.00 MB	56.00 MB	+0.0%	🟢
open (MB/s)	79.34 MB/s	79.66 MB/s	+0.4%	🟢
open (memory delta)	26.00 MB	26.00 MB	+0.0%	🟢

small

Metric	Baseline	Current	Delta	Status
insert (ops/s)	18.84 ops/s	17.86 ops/s	-5.2%	📊
insert (memory delta)	0.00 MB	0.00 MB	N/A	🟢
vector_search (QPS)	220.38 queries/s	214.20 queries/s	-2.8%	📊
vector_search (memory delta)	0.00 MB	0.00 MB	N/A	🟢
text_search (QPS)	426.40 queries/s	423.20 queries/s	-0.8%	📊
text_search (memory delta)	0.00 MB	0.00 MB	N/A	🟢
hybrid_search (QPS)	127.96 queries/s	121.47 queries/s	-5.1%	📊
hybrid_search (memory delta)	0.00 MB	0.00 MB	N/A	🟢
update (ops/s)	16.31 ops/s	15.82 ops/s	-3.0%	📊
update (memory delta)	0.00 MB	0.00 MB	N/A	🟢
delete (ops/s)	657,145.50 ops/s	677,885.22 ops/s	+3.2%	🟢
delete (memory delta)	0.00 MB	0.00 MB	N/A	🟢
save (MB/s)	0.02 MB/s	0.01 MB/s	-50.0%	📊
save (disk size)	13.12 MB	13.12 MB	+0.0%	🟢
save (memory delta)	2.00 MB	0.00 MB	-100.0%	🟢
open (MB/s)	65.14 MB/s	68.34 MB/s	+4.9%	🟢
open (memory delta)	2.00 MB	4.00 MB	+100.0%	🔴

[danielebarbaro]

The flagged row is a 2 MB shift between phases, not extra memory. In the small scenario save (memory delta) dropped from 2 MB to 0 MB while open (memory delta) rose from 2 MB to 4 MB, so the total is unchanged. PHP's allocator grows in 2 MB chunks, which is why a single chunk moving from one phase to the next reads as +100%.

#29 and #30 show exactly the same pattern while touching unrelated code, and neither changes the open() path, so the baseline most likely predates a change already on main.