Skip to content

Lock-free insert races locked mutations: ghost-visible rows, delete panic, stall under same-key churn #169

Description

@pathscale

Context

Found by the upsert starvation test added in review round 3 (tests/worktable/upsert.rs). insert() (src/table/mod.rs) takes no row lock and publishes the primary-key entry before unghosting the row data, while update/delete/upsert assume row-lock-serialized access. Three observable consequences under same-key churn (one key, concurrent insert/delete/update):

  1. Ghost-visible rows: a reader resolving the pk between insert's index publish and its unghost gets PagesError(Ghosted) for a row that is about to exist. upsert now treats a row-absent PagesError from its update arm as a transient and retries (fixed at the upsert level in patch 03 rev 3), but every other consumer of update() can still surface it.
  2. Delete panic: with raw-insert churn (insert + delete loop racing 4 upserters on one key), the churn task panicked with Option::unwrap() on None inside generated/library delete code (~1/5 runs at 5000 flips). Locked delete racing the lockless insert hits an invariant that assumes lock-serialized state.
  3. Stall: even with churn through locked operations only (upsert+delete), a churn round occasionally exceeded a 60s timeout at >=500 flips (~1/30 runs at 500, ~3/20 at 5000). Root cause not yet isolated; plausibly the same publish-before-unghost window interacting with lock waits.

Repro

tests/worktable/upsert.rs::upsert_completes_under_extreme_same_key_churn (currently #[ignore]d) hits mode 3 directly; switch its churn task to raw table.insert(...) + table.delete(...) to hit modes 1 and 2. The enabled smoke tier (100 flips, 4x200 upserts) is 40/40 stable and guards the upsert-level retry behavior.

Suggested direction

Make insert's publish order safe: unghost data before (or atomically with) publishing the pk entry, or give insert the same row-lock discipline as update/delete (lock the pk before index publish — the lock map supports locking keys that have no row yet, which upsert's future single-lock design in #164 also wants). Audit delete for the unwrap in mode 2 regardless.

Un-ignore the extreme churn test when fixed. Related: #164 (upsert linearization), #165 (lock map retention).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions