fix(runtime): preserve exact callback ownership - #10887
Conversation
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs — TryRemove(id, expected) uses ReferenceEquals(value, expected), which will always return false… |
What changed in this PR
This PR refines Orleans runtime callback lifecycle handling to make callback ownership exact (ABA-safe) when callbacks are removed due to cancellation, timeouts, shutdown, or silo failure. It builds on the striped callback tracking work from #10062 by ensuring that cleanup paths unregister only the specific CallbackData instance they own.
Changes:
- Introduces
ICallbackDataTargetand routes callback-owned cleanup through it so unregister operations are instance-exact. - Updates
InsideRuntimeClientandOutsideRuntimeClientcallback registries to enforce exact/unambiguous unregister semantics and detect duplicate registrations. - Adds focused unit tests for striped dictionary behavior and callback lifecycle races (response vs timeout/cancellation/shutdown, replacement safety, duplicate registration).
| File | Description |
|---|---|
| test/Orleans.Runtime.Tests/StripedCallbackDictionaryTests.cs | Adds coverage for stripe distribution, snapshot enumeration, concurrency, and exact-removal semantics. |
| test/Orleans.Runtime.Tests/CallbackDataTests.cs | Adds targeted lifecycle race/ABA tests and updates test harness to use ICallbackDataTarget. |
| src/Orleans.Runtime/Core/InsideRuntimeClient.cs | Switches callback tracking to StripedCallbackDictionary<CallbackData> and implements instance-exact unregister via ICallbackDataTarget. |
| src/Orleans.Core/Runtime/SharedCallbackData.cs | Removes the unregister delegate from shared state (unregister now flows through a target). |
| src/Orleans.Core/Runtime/OutsideRuntimeClient.cs | Implements ICallbackDataTarget and performs exact unregister via key/value pair removal. |
| src/Orleans.Core/Runtime/CallbackData.cs | Adds ICallbackDataTarget and updates callback cleanup paths to call target.Unregister(this). |
| src/Orleans.Core/Messaging/StripedCallbackDictionary.cs | Adds the striped dictionary implementation, including an “expected value/instance” removal overload and snapshot iteration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage78.54% line coverage - 99,514 / 126,708 lines Coverage details
|
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs — TryRemove(id, expected) uses ReferenceEquals(value, expected), which will always return false… View resolved comment |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs:153
- CountWhere executes the predicate while holding the stripe lock. That means an expensive predicate (or one which calls back into this dictionary) can stall other operations and can also deadlock via re-entrancy. Consider snapshotting the stripe values (as ForEach does) and evaluating the predicate outside the lock.
lock (stripe.Lock)
{
foreach (var value in stripe.Dictionary.Values)
{
if (predicate(value))
50e8327 to
0eedd2d
Compare
Distribute callbacks across striped dictionaries using correlation-id bits so concurrent request registration and completion contend on independent locks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Close callback registration before shutdown sweeps and eagerly capture silo services so late responses cannot resolve disposed providers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Port striped callback storage onto the current nullable-safe runtime clients and clear pooled snapshots only when their entries contain references. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs — TryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the… |
0eedd2d to
92004f0
Compare
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs — TryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the… View comment |
Suppressed comments (1)
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs:105
TryRemove(id, expected)currently usesReferenceEquals(value, expected). For value-typeTValue(or structs), this will always be false (boxing), so exact removal can never succeed. If this dictionary is intended to support non-reference TValue, switch to value equality for value types while keeping reference identity for reference types (so CallbackData exact ownership semantics remain unchanged).
if (!stripe.Dictionary.TryGetValue(id, out var value) || !ReferenceEquals(value, expected))
{
return false;
}
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
src/Orleans.Core/Messaging/StripedCallbackDictionary.cs — TryRemove(id, expected) currently uses ReferenceEquals(value, expected), which makes the… View resolved comment |

Depends on #10062
The visible diff includes the base PR until #10062 merges.
This follow-up makes callback ownership exact:
Microsoft Reviewers: Open in CodeFlow