Skip to content

Non-CDC index mutations silently burn CDC event ids #167

Description

@pathscale

Context

From the 2026-07-27 concurrency work (root-cause analysis behind patches 0006/0010). indexset's CDC maps assign event ids with fetch_add inside every mutation, including the non-CDC convenience methods: insert()/remove() call put_cdc/remove_cdc internally and then discard the events (see indexset src/concurrent/map.rs, e.g. insertput_cdc(..).0; try_insert.ok().map(|_| ())).

Consequence: any non-CDC mutation on an index that also feeds a persistence stream consumes ids without queueing events → a permanent gap in the event-id sequence. Since the strict-deferral fix (patch 0006), BatchOperation::validate never force-applies a gapped stream, so such a gap stalls persistence forever, loudly (tracing::error after 8 attempts). That is deliberate: a loud stall instead of silent positional corruption of the on-disk index. Vacuum was the one reachable case on persisted tables and was fixed in patch 0010 (VacuumPersistence sink).

What remains

  • In-memory tables still use non-CDC mutations (e.g. the non-CDC reinsert in src/table/mod.rs, vacuum's sink-less path in src/table/vacuum/vacuum.rs). Harmless today — no persistence consumer exists there — but it is a trap: attaching any future consumer (replication, S3 sync, CDC subscribers) to a path that mixes in even one non-CDC mutation produces an unfixable stall by design.
  • Nothing enforces the invariant "an index with a persistence consumer is only mutated through *_cdc".

Suggested direction

One or more of:

  1. Documentation: state the invariant on LockMap-adjacent index types and in the persistence module docs.
  2. Debug guard: wrap indexes destined for persistence in a newtype that only exposes *_cdc methods (compile-time enforcement, the strongest option).
  3. Runtime tripwire: a debug_assert!/counter comparing ids consumed vs events queued per index (catches regressions in tests immediately, given the suite already runs everything through wait_for_ops with timeouts).

Verification hints

The stall symptom is deterministic: wait_for_ops hangs, tracing::error names the last applied and next available ids. tests/persistence/vacuum.rs shows the pattern for turning a stall into a 30s-timeout failure. Reverting patch 0010's one-line .with_persistence(..) wiring reproduces a genuine gap stall on demand.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions