feat(runtime): add efficient runtime state dissemination

[ReubenBond]

Orleans currently relies on all-to-all publication for several kinds of silo runtime state, including deployment load statistics, which scales poorly as clusters grow.

This adds an internal topic-based dissemination subsystem using deterministic fixed-tree broadcast with digest-based anti-entropy repair. It integrates deployment load statistics, membership snapshots, and cluster manifests while retaining their existing correctness backstops. Dissemination remains opt-in globally and per topic so existing behavior is preserved by default.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

Adds an internal, opt-in runtime dissemination subsystem to reduce all-to-all communication for high-rate silo state (notably deployment load statistics) while preserving existing correctness backstops via digest-based anti-entropy repair and legacy fallbacks.

Changes:

• Introduces a topic-based dissemination protocol/service with deterministic fixed-tree broadcast and periodic anti-entropy repair.
• Integrates dissemination (with legacy fallback) for deployment load statistics and membership updates; improves manifest convergence via hash summaries + fetch-by-hash caching.
• Adds internal tests plus supporting public options/API surface updates and design documentation.

Show a summary per file

File	Description
test/Orleans.Runtime.Internal.Tests/Orleans.Runtime.Internal.Tests.csproj	Adds conditional test dependency for net10.0 model/spec testing.
test/Orleans.Runtime.Internal.Tests/Dissemination/DisseminationProtocolTests.cs	Adds extensive unit/property tests for dissemination routing, batching, anti-entropy, and topic behaviors.
src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs	Publishes runtime statistics via dissemination when enabled, otherwise falls back to direct publication.
src/Orleans.Runtime/MembershipService/MembershipGossiper.cs	Attempts membership updates via dissemination before legacy partner gossip.
src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs	Adds manifest hash caching and peer-assisted filling to reduce direct manifest fetches.
src/Orleans.Runtime/Hosting/DefaultSiloServices.cs	Registers dissemination transport/service/system-target and topic implementations + validators/formatters.
src/Orleans.Runtime/GrainTypeManager/ClusterManifestSystemTarget.cs	Adds hash-oriented manifest APIs for peer fill/fetch-by-hash.
src/Orleans.Runtime/Dissemination/OrleansDisseminationTransport.cs	Implements dissemination transport over Orleans system targets and membership snapshots.
src/Orleans.Runtime/Dissemination/MembershipDisseminationTopic.cs	Implements membership topic with snapshot + diff payloads and bounded history.
src/Orleans.Runtime/Dissemination/ManifestHashCalculator.cs	Canonical content hashing for GrainManifest to enable CAS-style reuse.
src/Orleans.Runtime/Dissemination/IDisseminationTransport.cs	Defines transport contract and membership scope snapshot model.
src/Orleans.Runtime/Dissemination/IDisseminationTopic.cs	Defines topic contract (digests, value materialization/apply, fallback, membership scope).
src/Orleans.Runtime/Dissemination/DisseminationTopicNames.cs	Centralizes internal dissemination topic name constants.
src/Orleans.Runtime/Dissemination/DisseminationSystemTarget.cs	Exposes dissemination ingress/anti-entropy via a system target, wired into lifecycle.
src/Orleans.Runtime/Dissemination/DisseminationService.cs	Serializes protocol execution and runs periodic anti-entropy loop.
src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs	Implements fixed-tree routing, batching/coalescing, validation, backoff, and anti-entropy.
src/Orleans.Runtime/Dissemination/DisseminationInstruments.cs	Adds metrics for gossip, values, bytes, repair, fallbacks, and drops.
src/Orleans.Runtime/Dissemination/DisseminationEvents.cs	Adds DiagnosticListener events for value apply and payload drops.
src/Orleans.Runtime/Dissemination/DisseminationApplyResult.cs	Defines apply outcomes used by topics/protocol.
src/Orleans.Runtime/Dissemination/DeploymentLoadStatisticsDisseminationTopic.cs	Implements deployment-load topic (latest-wins per silo) with fallback refresh.
src/Orleans.Runtime/Configuration/Options/DisseminationOptionsValidator.cs	Adds validators for dissemination global/topic options and per-topic integration validators.
src/Orleans.Runtime/Configuration/Options/DeploymentLoadPublisherOptions.cs	Adds per-topic dissemination options to deployment load publisher options.
src/Orleans.Core/SystemTargetInterfaces/IDisseminationSystemTarget.cs	Adds dissemination wire DTOs and system-target contract for gossip/anti-entropy.
src/Orleans.Core/Runtime/Constants.cs	Adds a system target type identifier/name for dissemination.
src/Orleans.Core/Manifest/IClusterManifestSystemTarget.cs	Adds manifest hash summary + fetch-by-hash APIs and supporting DTOs.
src/Orleans.Core/Configuration/Options/DisseminationOptions.cs	Adds public global/overlay/topic dissemination options.
src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs	Adds per-topic dissemination options to membership options.
src/api/Orleans.Runtime/Orleans.Runtime.cs	Updates generated public API surface for Orleans.Runtime option changes.
src/api/Orleans.Core/Orleans.Core.cs	Updates generated public API surface for new dissemination options + membership option additions.
efficient-broadcast.md	Documents the implemented approach and design rationale for this branch.
dissemination.md	Adds broader design doc/background for the dissemination subsystem.
Directory.Packages.props	Adds central package version for Microsoft.Accordant.

Copilot's findings

Suppressed comments (2)

src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs:963

• Removing from _failureBackoffUntil while iterating it will throw InvalidOperationException. Iterate over a snapshot (e.g., _failureBackoffUntil.ToArray()) before removing.

            foreach (var (peer, until) in _failureBackoffUntil)
            {
                if (until <= now || !IsCurrentParticipant(peer))
                {
                    _failureBackoffUntil.Remove(peer);

src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs:974

• Removing from _pendingGossip while iterating over _pendingGossip.Keys will throw InvalidOperationException. Iterate over a snapshot (e.g., Keys.ToArray()) before removing.

            foreach (var peer in _pendingGossip.Keys)
            {
                if (!IsCurrentParticipant(peer))
                {
                    _pendingGossip.Remove(peer);

• Files reviewed: 32/32 changed files
• Comments generated: 4

[Copilot]

Copilot's findings

Suppressed comments (2)

src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs:554

• DisseminationOptions.MaxConcurrentSends is currently not honored: SendGossipBatches sends to peers strictly sequentially, so the effective concurrency is always 1 regardless of configuration. This can significantly slow down dissemination and makes the public option misleading.

Consider enforcing MaxConcurrentSends by limiting the number of in-flight SendGossipBatch tasks with a SemaphoreSlim (or similar).

    private async Task SendGossipBatches(List<(SiloAddress Peer, ImmutableArray<PendingTopicValues> ValuesByTopic)> batches, CancellationToken cancellationToken)
    {
        foreach (var queued in batches)
        {
            await SendGossipBatch(queued.Peer, queued.ValuesByTopic, cancellationToken);
        }
    }

src/Orleans.Runtime/MembershipService/MembershipGossiper.cs:26

• TryGossipViaDissemination takes a gossipPartners parameter which is never used. Keeping this unused parameter is confusing (it suggests partner-scoped dissemination) and adds noise for future maintenance.

Consider removing the parameter and updating the call site accordingly.

        if (await TryGossipViaDissemination(gossipPartners, snapshot))
        {
            return;
        }

• Files reviewed: 32/32 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

Suppressed comments (2)

test/Orleans.Runtime.Internal.Tests/Dissemination/DisseminationProtocolTests.cs:1044

• WaitUntil uses DateTime.UtcNow for its timeout tracking. That can make the test flaky on CI if the system clock changes (e.g., NTP adjustments) and it makes the timeout behavior less predictable. Using a monotonic timer (Stopwatch) avoids clock-skew issues while keeping the same behavior.
  src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:154
• The 'members' parameter is not used in this method body, which makes the call site look like the membership set influences dissemination when it currently does not. Renaming the parameter to '_' clarifies that it's intentionally unused (or remove it entirely if not needed).

        private async Task<bool> TryPublishStatisticsViaDissemination(SiloRuntimeStatistics myStats, IReadOnlyCollection<SiloAddress> members)

• Files reviewed: 32/32 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

Suppressed comments (2)

src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs:37

• _manifestCache is an unbounded ConcurrentDictionary which only ever grows (entries are added in multiple code paths but never removed). In long-running clusters with frequent rolling deployments (and therefore many distinct GrainManifest hashes), this can lead to unbounded memory growth. Consider adding a simple bound/eviction policy (eg, size cap with LRU/clock, or periodic pruning to manifests referenced by the current ClusterManifest plus the local manifest).

        private readonly ConcurrentDictionary<ManifestHash, GrainManifest> _manifestCache = new();

src/Orleans.Runtime/Dissemination/ManifestHashCalculator.cs:55

• ManifestHashCalculator.AppendString allocates new single-byte arrays on every call ("new byte[] { 0 }" / "new byte[] { 0xff }"). Since hashing a manifest can involve many entries/properties, this creates avoidable GC pressure and can become noticeable when manifests are hashed frequently (e.g., building hash summaries). Consider using a stackalloc span or static readonly byte[1] buffers for the separators instead.

        hash.AppendData(length);
        hash.AppendData(new byte[] { 0 });
        hash.AppendData(bytes);
        hash.AppendData(new byte[] { 0xff });

• Files reviewed: 32/33 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

Suppressed comments (3)

src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:120

• members is materialized with ToArray() even when dissemination succeeds, which adds an avoidable allocation/enumeration cost on the fast path. Since the direct all-to-all publish is only used as a fallback, defer building the members array until dissemination declines/fails.

                var members = _siloStatusOracle.GetApproximateSiloStatuses(true).Keys.ToArray();
                if (!await TryPublishStatisticsViaDissemination(myStats))
                {
                    await PublishStatisticsDirectly(myStats, members);
                }

src/Orleans.Runtime/Dissemination/DisseminationService.cs:58

• StartAsync always starts the anti-entropy background loop, even when DisseminationOptions.Enabled is false. Since dissemination is opt-in and disabled by default, this still creates a long-lived background task and periodic wakeups on every silo. Consider short-circuiting StartAsync when dissemination is disabled (or otherwise gating the loop) to keep the default configuration cost near-zero.

    internal Task StartAsync(CancellationToken cancellationToken)
    {
        if (_antiEntropyTask is { IsCompleted: false })
        {
            return Task.CompletedTask;

src/Orleans.Runtime/Dissemination/ManifestHashCalculator.cs:54

• AppendString allocates new 1-byte arrays for delimiters on every call (new byte[] { 0 } / new byte[] { 0xff }). Since this runs per entry/property during manifest hashing, it can generate a lot of tiny allocations. Use stackalloc spans (or cached static buffers) for the delimiter bytes to avoid per-call heap allocations.

        var length = Encoding.UTF8.GetBytes(bytes.Length.ToString(CultureInfo.InvariantCulture));
        hash.AppendData(length);
        hash.AppendData(new byte[] { 0 });
        hash.AppendData(bytes);
        hash.AppendData(new byte[] { 0xff });

• Files reviewed: 32/33 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

Suppressed comments (2)

src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:184

• Self-skip uses == for SiloAddress, which is reference equality (SiloAddress does not overload operator==). If the oracle returns an equivalent-but-distinct SiloAddress instance, this will fail to skip and will make a redundant system-target call to self (and re-notify listeners). Use Equals instead.

                // No need to make a grain call to ourselves.
                if (siloAddress == _siloDetails.SiloAddress)
                {
                    continue;
                }

src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs:284

• This uses peer == _localSiloAddress, but SiloAddress does not overload operator==, so this is reference equality. That can fail to skip the local silo and cause unnecessary self-requests. Use Equals for value equality.

                if (peer == _localSiloAddress)
                {
                    continue;
                }

• Files reviewed: 32/33 changed files
• Comments generated: 0 new

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs:426

• When the hash-based fetch succeeds and the payload hash is validated, the result is returned but not added to _manifestCache. If publishing the updated cluster manifest fails (e.g., due to a concurrent publish), this misses an opportunity to reuse the fetched manifest and can lead to repeated remote fetches.

                var manifest = await remoteManifestProvider.GetSiloManifestByHash(hash).AsTask().WaitAsync(_shutdownCts.Token);
                if (manifest is not null && ManifestHashCalculator.ComputeHash(manifest) == hash)
                {
                    return manifest;
                }

• Files reviewed: 33/34 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 33/34 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Copilot review overview

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	src/​Orleans.Runtime/​Placement/​DeploymentLoadPublisher.cs — UpdateRuntimeStatisticsInternal holds _statisticsUpdateLock while calling…

Suppressed comments (1)

src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:325

• OnSiloStatusChange holds _statisticsUpdateLock while invoking subscriber callbacks via NotifyAllStatisticsChangeEventsSubscribers. This can deadlock if a subscriber calls back into the publisher and tries to acquire _statisticsUpdateLock (lock-order inversion), and it can unnecessarily block stats updates/removals while callbacks run. Prefer updating _periodicStats under the lock, then notifying subscribers after releasing it.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Review tier: Lite
Findings: None

Issues resolved since last review (1)

Severity	Finding
	src/​Orleans.Runtime/​Placement/​DeploymentLoadPublisher.cs — UpdateRuntimeStatisticsInternal holds _statisticsUpdateLock while calling… View resolved comment

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs:111

• GetUnconfirmedPeers(...) relies on _confirmedPeerTopics, but ReceiveGossip only confirms the immediate batch.Sender. Since forwarded gossip batches set Sender to the forwarder (not the originator/root), most silos will never be confirmed for a topic, so callers like DeploymentLoadPublisher.PublishStatistics will keep falling back to direct all-to-all publication even when dissemination is healthy. Consider also confirming per-value originators (e.g., item.Root) for the topic(s) present in the batch so topic confirmation converges cluster-wide.

            foreach (var item in values)
            {
                cancellationToken.ThrowIfCancellationRequested();
                await ApplyReceivedValue(topicName, topic, item, batch.Sender, forward: true, cancellationToken);
            }

[ReubenBond]

Fresh CI on e8357231c9 failed StaticRebalancingTests.Should_Move_Activations_From_Silo1_And_Silo3_To_Silo2_And_Silo4 on macOS for both net8.0 and net10.0.

Run: https://github.com/dotnet/orleans/actions/runs/33249205979

Because the latest change moves DeploymentLoadPublisher notifications outside _statisticsUpdateLock, this may indicate an ordering/timing regression in activation-rebalancing statistics delivery rather than an unrelated single-matrix flake. The failed jobs have been retried and the branch is being investigated against the focused activation-rebalancing test.

[ReubenBond]

Follow-up investigation confirms this is an unrelated macOS arm64 readiness/timing flake rather than a regression in the publisher notification change.

• Attempt 1 failed the static rebalancing test on both TFMs with zero migrations after ~9.3s.
• Attempt 2 passed the static test on net10.0, while net8.0 instead failed the dynamic test with zero migrations after its fixed 15s window.
• Local stress runs passed static 22/22 and dynamic 6/6 across net8.0/net10.0.

The changing failing test and zero-migration signature indicate rebalancer startup/readiness missed fixed wall-clock deadlines under loaded CI. This is tracked in #10922. The failed job is being retried again; no branch code change is warranted.