Context
The wait-for-ops race fix (patch 0014 of the pending fix series, src/persistence/task.rs) closed two problems:
PersistenceTask::wait_for_ops could return while a popped operation was still in flight — between Queue::pop's len.fetch_sub and the engine loop's in_progress store-back, all three wait triggers held at once. The pop now sets the busy flag under the queue mutex before decrementing len, so the empty+idle state is never observable with an op in flight. (Observed as rare flakes: test_insert_primary_duplicate 1/30 under full-suite load reading data = 1000 instead of 100 from a reused pk after a stale pk_gen_state reload; test_space_insert_many_sync with pk_gen state 4917 vs 5000.)- The engine task's
AbortHandle was stored but never used, so a dropped table's engine task kept running — and writing — indefinitely. PersistenceTask now aborts it in Drop.
Residual hazard
Abort-on-drop narrows the problem but cannot fully eliminate it: tokio::task::AbortHandle::abort cancels at the task's next await point. If that point is inside PersistenceEngine::apply_batch_operation, the batch is torn — part of it is on disk, part isn't.
This is the ceiling for what a Drop impl can do:
Drop::drop is synchronous. It cannot .await the engine's JoinHandle, and block_on inside it panics/deadlocks when the drop happens on the runtime driving the engine task (which is where tables get dropped).- The two behaviors reachable from
Drop are both imperfect: abort stops promptly but may tear a batch; a graceful shutdown flag checked at batch boundaries never tears a batch but lets the old engine keep writing for up to one batch after the table is gone — re-opening a bounded version of the stale-writer overlap the fix exists to close.
Abort was chosen because every reopen path that matters calls wait_for_ops first; with the fixed triggers the engine is then parked at the idle queue pop, where cancellation is clean. The torn-write case only arises for callers that drop without waiting — and those were already discarding queued operations.
Note the torn-batch state is identical to the process being killed mid-write: power loss and kill -9 produce it too, and no Drop impl helps with those.
Proposed work
- Explicit graceful close —
async fn close(self) on PersistenceTask, plumbed through codegen to the table API: signal shutdown, let an in-flight batch finish, await the engine task's exit. Drop's abort stays as the backstop for callers that never call it. Gives well-behaved callers zero overlap and zero torn writes. The vacuum-sink path is the realistic beneficiary. This is patch-0015-shaped: an API addition, separate from the race fix.
- Crash-atomic batch application (separate roadmap item) — a write-ahead journal or shadow-page scheme in the space format so a half-applied batch is detected and discarded on load. This is the only complete fix, because it also covers real crashes, not just drops. It changes the on-disk format, so it needs its own compatibility story and should not ride in a bug-fix series.
Context
The wait-for-ops race fix (patch 0014 of the pending fix series,
src/persistence/task.rs) closed two problems:PersistenceTask::wait_for_opscould return while a popped operation was still in flight — betweenQueue::pop'slen.fetch_suband the engine loop'sin_progressstore-back, all three wait triggers held at once. The pop now sets the busy flag under the queue mutex before decrementinglen, so the empty+idle state is never observable with an op in flight. (Observed as rare flakes:test_insert_primary_duplicate1/30 under full-suite load readingdata = 1000instead of100from a reused pk after a stalepk_gen_statereload;test_space_insert_many_syncwith pk_gen state 4917 vs 5000.)AbortHandlewas stored but never used, so a dropped table's engine task kept running — and writing — indefinitely.PersistenceTasknow aborts it inDrop.Residual hazard
Abort-on-drop narrows the problem but cannot fully eliminate it:
tokio::task::AbortHandle::abortcancels at the task's next await point. If that point is insidePersistenceEngine::apply_batch_operation, the batch is torn — part of it is on disk, part isn't.This is the ceiling for what a
Dropimpl can do:Drop::dropis synchronous. It cannot.awaitthe engine'sJoinHandle, andblock_oninside it panics/deadlocks when the drop happens on the runtime driving the engine task (which is where tables get dropped).Dropare both imperfect: abort stops promptly but may tear a batch; a graceful shutdown flag checked at batch boundaries never tears a batch but lets the old engine keep writing for up to one batch after the table is gone — re-opening a bounded version of the stale-writer overlap the fix exists to close.Abort was chosen because every reopen path that matters calls
wait_for_opsfirst; with the fixed triggers the engine is then parked at the idle queue pop, where cancellation is clean. The torn-write case only arises for callers that drop without waiting — and those were already discarding queued operations.Note the torn-batch state is identical to the process being killed mid-write: power loss and
kill -9produce it too, and noDropimpl helps with those.Proposed work
async fn close(self)onPersistenceTask, plumbed through codegen to the table API: signal shutdown, let an in-flight batch finish, await the engine task's exit.Drop's abort stays as the backstop for callers that never call it. Gives well-behaved callers zero overlap and zero torn writes. The vacuum-sink path is the realistic beneficiary. This is patch-0015-shaped: an API addition, separate from the race fix.