Skip to content

fix(runtime): preserve exact callback ownership - #10887

Draft
ReubenBond wants to merge 18 commits into
dotnet:mainfrom
ReubenBond:rb-exact-callback-ownership
Draft

fix(runtime): preserve exact callback ownership#10887
ReubenBond wants to merge 18 commits into
dotnet:mainfrom
ReubenBond:rb-exact-callback-ownership

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Aug 28, 2026

Copy link
Copy Markdown
Member

Depends on #10062

The visible diff includes the base PR until #10062 merges.

This follow-up makes callback ownership exact:

  • Removes callbacks conditionally using both the correlation ID and expected CallbackData instance.
  • Moves callback-owned cleanup through ICallbackDataTarget so cancellation, timeout, silo failure, and shutdown unregister only their own callback.
  • Preserves replacement callbacks when stale lifecycle completion races with correlation ID reuse, providing ABA safety.
  • Adds focused lifecycle coverage for response races, cancellation, timeout, shutdown, duplicate registration, and exact striped removal.
Microsoft Reviewers: Open in CodeFlow

Copilot AI lite review requested due to automatic review settings August 28, 2026 03:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.csTryRemove(id, expected) uses ReferenceEquals(value, expected), which will always return false…
What changed in this PR

This PR refines Orleans runtime callback lifecycle handling to make callback ownership exact (ABA-safe) when callbacks are removed due to cancellation, timeouts, shutdown, or silo failure. It builds on the striped callback tracking work from #10062 by ensuring that cleanup paths unregister only the specific CallbackData instance they own.

Changes:

  • Introduces ICallbackDataTarget and routes callback-owned cleanup through it so unregister operations are instance-exact.
  • Updates InsideRuntimeClient and OutsideRuntimeClient callback registries to enforce exact/unambiguous unregister semantics and detect duplicate registrations.
  • Adds focused unit tests for striped dictionary behavior and callback lifecycle races (response vs timeout/cancellation/shutdown, replacement safety, duplicate registration).
File Description
test/​Orleans.Runtime.Tests/​StripedCallbackDictionaryTests.cs Adds coverage for stripe distribution, snapshot enumeration, concurrency, and exact-removal semantics.
test/​Orleans.Runtime.Tests/​CallbackDataTests.cs Adds targeted lifecycle race/ABA tests and updates test harness to use ICallbackDataTarget.
src/​Orleans.Runtime/​Core/​InsideRuntimeClient.cs Switches callback tracking to StripedCallbackDictionary<CallbackData> and implements instance-exact unregister via ICallbackDataTarget.
src/​Orleans.Core/​Runtime/​SharedCallbackData.cs Removes the unregister delegate from shared state (unregister now flows through a target).
src/​Orleans.Core/​Runtime/​OutsideRuntimeClient.cs Implements ICallbackDataTarget and performs exact unregister via key/value pair removal.
src/​Orleans.Core/​Runtime/​CallbackData.cs Adds ICallbackDataTarget and updates callback cleanup paths to call target.Unregister(this).
src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.cs Adds the striped dictionary implementation, including an “expected value/instance” removal overload and snapshot iteration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Orleans.Core/Messaging/StripedCallbackDictionary.cs Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Code coverage

78.54% line coverage - 99,514 / 126,708 lines

Coverage details

Copilot AI review requested due to automatic review settings August 28, 2026 09:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (1)
Severity Finding
Medium severity src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.csTryRemove(id, expected) uses ReferenceEquals(value, expected), which will always return false… View resolved comment
Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.Core/Messaging/StripedCallbackDictionary.cs:153

  • CountWhere executes the predicate while holding the stripe lock. That means an expensive predicate (or one which calls back into this dictionary) can stall other operations and can also deadlock via re-entrancy. Consider snapshotting the stripe values (as ForEach does) and evaluating the predicate outside the lock.
            lock (stripe.Lock)
            {
                foreach (var value in stripe.Dictionary.Values)
                {
                    if (predicate(value))

Copilot AI review requested due to automatic review settings August 28, 2026 14:06
@ReubenBond
ReubenBond force-pushed the rb-exact-callback-ownership branch from 50e8327 to 0eedd2d Compare August 28, 2026 14:06
ReubenBond and others added 11 commits August 28, 2026 07:08
Distribute callbacks across striped dictionaries using correlation-id
bits so concurrent request registration and completion contend on
independent locks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Close callback registration before shutdown sweeps and eagerly capture silo services so late responses cannot resolve disposed providers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Port striped callback storage onto the current nullable-safe runtime clients and clear pooled snapshots only when their entries contain references.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.csTryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the…

Comment thread src/Orleans.Core/Messaging/StripedCallbackDictionary.cs Outdated
Copilot AI review requested due to automatic review settings August 28, 2026 14:16
@ReubenBond
ReubenBond force-pushed the rb-exact-callback-ownership branch from 0eedd2d to 92004f0 Compare August 28, 2026 14:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

Review tier: Lite
Findings: 1 Medium severity

Pre-existing issues (1)
Severity Finding
Medium severity src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.csTryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the… View comment
Suppressed comments (1)

src/Orleans.Core/Messaging/StripedCallbackDictionary.cs:105

  • TryRemove(id, expected) currently uses ReferenceEquals(value, expected). For value-type TValue (or structs), this will always be false (boxing), so exact removal can never succeed. If this dictionary is intended to support non-reference TValue, switch to value equality for value types while keeping reference identity for reference types (so CallbackData exact ownership semantics remain unchanged).
            if (!stripe.Dictionary.TryGetValue(id, out var value) || !ReferenceEquals(value, expected))
            {
                return false;
            }

Copilot AI review requested due to automatic review settings August 28, 2026 14:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (1)
Severity Finding
Medium severity src/​Orleans.Core/​Messaging/​StripedCallbackDictionary.csTryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the… View resolved comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants