Observed on main @ 103b1d1 (debug build, single-node dev stack) during independent Stage-3 qualification reruns.
Sequence: a fresh stack (--metadata-create), a handful of coordination writes (workspace create, three publishes), then SIGKILL of the serving owner ~1 second after the last write - i.e. inside the metadata flush window. Every subsequent reopen attempt (--metadata-reopen) fails permanently with:
nokv: metadata store failed: metadata store is corrupt: node corrupt at FileBlobStore::Manifest::duplicate slot
The corruption is durable: no retry ever succeeds and the store is lost. Killing the owner after a few seconds of quiescence never reproduces it (all prior Stage-2/Stage-3 SIGKILL drills passed with the kill issued post-quiescence); the mid-write timing is the trigger.
Impact: crash-consistency - a kill -9 (or power loss) at the wrong moment costs the whole metadata store. The write-path claim that a crash between operations is recoverable does not currently extend to a crash inside the manifest write itself.
Evidence: the corrupt metadata directory (blobs.dat / journal.wal / manifest.log) is preserved from the failing run and can be shared for offline analysis. The failure signature matches the historical holt duplicate-slot class (manifest slots written non-atomically), reproduced here through NoKV's own store rather than an external holt consumer.
Suggested direction: manifest slot writes need the same torn-write protection as the journal (write-ahead + atomic slot activation), or reopen needs a repair path that discards a torn trailing manifest entry instead of failing closed on the whole store.
Observed on main @ 103b1d1 (debug build, single-node dev stack) during independent Stage-3 qualification reruns.
Sequence: a fresh stack (
--metadata-create), a handful of coordination writes (workspace create, three publishes), then SIGKILL of the serving owner ~1 second after the last write - i.e. inside the metadata flush window. Every subsequent reopen attempt (--metadata-reopen) fails permanently with:The corruption is durable: no retry ever succeeds and the store is lost. Killing the owner after a few seconds of quiescence never reproduces it (all prior Stage-2/Stage-3 SIGKILL drills passed with the kill issued post-quiescence); the mid-write timing is the trigger.
Impact: crash-consistency - a kill -9 (or power loss) at the wrong moment costs the whole metadata store. The write-path claim that a crash between operations is recoverable does not currently extend to a crash inside the manifest write itself.
Evidence: the corrupt metadata directory (blobs.dat / journal.wal / manifest.log) is preserved from the failing run and can be shared for offline analysis. The failure signature matches the historical holt duplicate-slot class (manifest slots written non-atomically), reproduced here through NoKV's own store rather than an external holt consumer.
Suggested direction: manifest slot writes need the same torn-write protection as the journal (write-ahead + atomic slot activation), or reopen needs a repair path that discards a torn trailing manifest entry instead of failing closed on the whole store.