Context
From the 2026-07-27 concurrency-fix series (patch 0010, "vacuum: persist row moves through CDC"). Vacuum on persisted tables is now correct — row moves flow through the CDC stream via the VacuumPersistence sink, so the on-disk indexes follow the moves and the event-id stream stays gapless (tests/persistence/vacuum.rs proves reload works, including secondary indexes) — but it is consistency-only.
The gap
EmptyDataVacuum::defragment frees pages in memory (stats.pages_freed), and the moved rows' bytes are persisted at their new links. Nothing ever shrinks or truncates the on-disk .wt.data file:
- Pages emptied by vacuum remain in the data file with stale bytes; the indexes simply stop pointing at them.
SpaceData::save_batch_data (src/persistence/space/data.rs) only ever writes pages; there is no free-page bookkeeping on disk beyond SpaceInfoPage.empty_links_list.
- After many vacuum cycles the file is dominated by dead pages.
So "vacuum" on a persisted table today = in-memory compaction + disk consistency, not disk space reclamation.
Suggested direction
Options, roughly in order of ambition:
- Persist the freed-page set (extend
SpaceInfoPage / TOC) so freed pages are reused by later save_batch_data page creation instead of appending — stops growth without shrinking.
- Physical compaction: rewrite the tail of the data file into freed pages and truncate — needs link rewrites, i.e. the same CDC move machinery vacuum already uses, so
VacuumPersistence::apply_move is the natural hook.
- Offline compaction on load (cheapest to reason about; no concurrency).
Verification hints
tests/persistence/vacuum.rs::test_vacuum_on_persisted_table_survives_reload is the harness to extend: assert on the data file size (or page count) after vacuum + reload. Full suite runs take ~4s warm; 30-50 run loops are cheap for flake checks.
Context
From the 2026-07-27 concurrency-fix series (patch 0010, "vacuum: persist row moves through CDC"). Vacuum on persisted tables is now correct — row moves flow through the CDC stream via the
VacuumPersistencesink, so the on-disk indexes follow the moves and the event-id stream stays gapless (tests/persistence/vacuum.rsproves reload works, including secondary indexes) — but it is consistency-only.The gap
EmptyDataVacuum::defragmentfrees pages in memory (stats.pages_freed), and the moved rows' bytes are persisted at their new links. Nothing ever shrinks or truncates the on-disk.wt.datafile:SpaceData::save_batch_data(src/persistence/space/data.rs) only ever writes pages; there is no free-page bookkeeping on disk beyondSpaceInfoPage.empty_links_list.So "vacuum" on a persisted table today = in-memory compaction + disk consistency, not disk space reclamation.
Suggested direction
Options, roughly in order of ambition:
SpaceInfoPage/ TOC) so freed pages are reused by latersave_batch_datapage creation instead of appending — stops growth without shrinking.VacuumPersistence::apply_moveis the natural hook.Verification hints
tests/persistence/vacuum.rs::test_vacuum_on_persisted_table_survives_reloadis the harness to extend: assert on the data file size (or page count) after vacuum + reload. Full suite runs take ~4s warm; 30-50 run loops are cheap for flake checks.