v7: the fixed MAX_OPS header reserve is a per-file floor of ~2048 rows, so small indexes sync to files 67-87x larger than write()

[RyanCodrai]

Every v7 file reserves two header slots at full Geo::hdr_len (turbovec/src/io_v7.rs:288-296), dominated by MAX_OPS * (5 + op_size) with MAX_OPS = 1024 and op_size = 1 + row_bytes + 4 + id_bytes. write_full materializes both slots explicitly (turbovec/src/io_v7.rs:718-721: w.write_all(&vec![0u8; geo.hdr_len() - h.len()]) then w.write_all(&vec![0u8; geo.hdr_len()])), so the reserve is real allocated blocks — du confirms the files are not sparse.

The floor is content-independent and equals roughly 2048 rows' worth of codes:

   dim  bw       n         v6()         sync()   ratio    2*hdr_len   v7-2*hdr
    64   4       0          146          96675   662x         96520        155
   768   4       0          146         839395  5749x        839240        155
  1536   4       0          146        1649635 11299x       1649480        155
  1536   4      32        24850        1674339    67x       1649480      24859
 16384   4       1       262294       17314275    66x      17314120        155
   768   4    2048       794770        1634019   2.1x        839240     794779
   768   4   20000      7760146        8599395   1.1x        839240    7760155

v7 - 2*hdr_len equals the v6 payload to within ~155 B at every geometry, so the formula fully explains the gap. At MAX_DIM = 16384 the reserve is ~17.3 MB per file. A realistic per-tenant / per-collection pattern — 100 indexes of 5 rows at dim 768 — costs 83.9 MB of v7 files where write() would produce ~1.2 MB.

This is a disclosed design cost, not a doc violation: the MAX_OPS comment (io_v7.rs:86-96) says the raise from 64 to 1024 costs "reserved header space only … an idle cap is free per sync", and that is accurate per sync. What was not quantified at the time (#476 estimated "~150KB reserved slack" from a small-dim test) is that the reserve scales with dim, so at production embedding widths it is megabytes per file and dominates any index below ~2048 rows.

Repro

TurboQuantIndex::new(dim, bw), add N rows, sync(p) and write(q), compare metadata().len(). N=0 shows the floor directly.

Fix direction (no change to the crash protocol)

MAX_OPS is already a per-file u32 in the superblock (io_v7.rs:393); load only rejects it because it demands exact equality with the build constant (io_v7.rs:924). Carry the file's value in Geo, derive hdr_len from it, accept file_max_ops <= MAX_OPS at load, and have write_full pick a cap scaled to the index (e.g. clamp(n/32, 32, MAX_OPS) or a byte budget). Nothing about the A/B commit protocol depends on the cap being a global constant: offsets are stable within a file because the value is frozen in the superblock, and the only writer that can change it is write_full, which is already temp-file + atomic rename. When carried > file_max_ops, the existing plan_incremental → None → write_full escape (lib.rs:1618) both re-caps and compacts. The tradeoff self-corrects — a small file gets a small reserve and an early full-rewrite cliff, and full-rewriting a small file is cheap, which is the same argument as #477. Writing a non-1024 cap makes new files unreadable to current binaries, so bump V7_VERSION as the constant's own doc instructs; v7 is unreleased (latest tag py-v0.8.0 predates #476), so the cost is zero today.

Relationships

Distinct from #477 (bytes one sync writes, not the file's floor) and from #480 (a loaded index retaining heap equal to the file — that finding compounds this one, since the reserve is part of the retained buffer).

---

Found independently by two overnight bug-hunt agents (wave 1) against main @ c8d7ec0; reproduced and adversarially validated by a third before filing.

[RyanCodrai]

Closing as intentional. The fixed MAX_OPS = 1024 reserve is a deliberate design choice, not an oversight.

A per-file cap would have to be re-chosen on a full rewrite, since the reserve sits at fixed offsets at the head of the file and can't grow in place. That means a growing index would pay a full rewrite each time it steps up to a larger cap — trading a predictable, content-independent layout for a variable one, plus a V7_VERSION bump.

The cost is real but narrow: it only bites on the many-small-indexes-at-large-dim pattern (100 indexes of 5 rows at dim 768 = 84 MB vs ~1.2 MB), which isn't the target workload. write() remains the right call for small, static indexes; sync() is for indexes that actually accumulate incremental changes.

Requested in #531.