Skip to content

Fix RemoveKeys leaving stale keys on the calling peer - #32

Merged
thrawn01 merged 1 commit into
mainfrom
thrawn01/eng-579-batch-remove-leaves-stale-keys-on-the-caller
Oct 29, 2026
Merged

thrawn01 merged 1 commit into
mainfrom
thrawn01/eng-579-batch-remove-leaves-stale-keys-on-the-caller

Conversation

@thrawn01

@thrawn01 thrawn01 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

After RemoveKeys, the peer that made the call kept its own cached copies of keys owned by other peers, so later reads there returned the removed value. Removing the same keys one at a time with Remove was correct. Reported in #27.

// daemon 0 reads a-e, then removes them
group.RemoveKeys(ctx, "a", "b", "c", "d", "e")
group.Get(ctx, "b", sink) // before: stale value from hot cache, getter not called; after: reloads

The calling peer now clears every removed key locally, even when an owner request fails. Errors from peers are still returned. This differs on purpose from Remove, which returns early and keeps the local copy when the owner fails.

Implementation

  • group.go: RemoveKeys calls LocalRemove for every key instead of only the keys this peer owns. LocalRemove clears both the hot and main cache.
  • remove_keys_test.go: TestRemoveKeysClearsCallerCopies reads keys a-e through daemon 0 on a 3-node cluster, calls RemoveKeys, reads again, and asserts the getter ran twice per key. It failed on main for keys b-e (getter ran once) and passes now. The existing TestRemoveKeys uses key1-key3, which daemon 0 owns, so it never covered this.

Verified with go test -race ./..., all packages pass. A fresh sub-agent reviewed the commit by reading it and found no defect; it could not run tests. Not done: a test for the owner-failure case.

Known limit: the caller clears its copy before owners delete theirs, so a concurrent Get on the caller can reload the old value from an owner in that window. Remove has the same window for non-owner peers, and the ticket asks for the clear to happen even when the owner fails.

Fixes ENG-579

@thrawn01
thrawn01 merged commit 45be68b into main Oct 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant