Flaky test: SMS stream restart retains duplicate publisher

[ReubenBond]

Description

UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts intermittently reports two publishers after restarting all silos, when one publisher is expected.

Failure

• Workflow run: https://github.com/dotnet/orleans/actions/runs/31529937783
• Job: 93907285539 (Azure Storage provider tests, net8.0)

After SendItem - PublisherCount for stream MemoryStreamProvider/StreamReliabilityNamespace/... = 2; expected 1

The failure occurred alongside the independently tracked AQ_StreamRel_SiloJoins timeout (#10452) on a PR which only changes a rebalancer test assertion. The corresponding behavior is unrelated to that PR.

Expected behavior

After all silos restart and the producer is restored, the pub/sub registry should converge deterministically to one publisher before the test checks its count.

[ReubenBond]

Another occurrence on PR #10493:

• Run/job: https://github.com/dotnet/orleans/actions/runs/31549169073/job/93967879974
• Environment: Azure Storage provider, net8.0
• Test: UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts
• Failure: publisher count was 2, expected 1, after SendItem following all-silo restart.

The Azure Storage net10.0 sibling job passed on the same head.

[ReubenBond]

Recurrence on current-main-based #10457 head \1c8940c8c7a0a6a4efa0f4c65e57c6438db4d1e8:\n\n- Run/job: https://github.com/dotnet/orleans/actions/runs/31560515935/job/94001664189\n- Azure Storage, net8.0\n- \SMS_StreamRel_AllSilosRestart_PubSubCounts\ observed publisher count \2, expected \1, after \SendItem.\n\n#10457 changes transaction lock timeout behavior and TOC tests only. The sibling net10 stream assembly passed. Open #10532 is the applicable fix; this recurrence is independent of #9556/#10457.

[ReubenBond]

A sequential 10-run workflow_dispatch audit on main reproduced this in 1/10 workflow runs and 1/20 Azure Storage matrix jobs. There were no canceled attempts.

• net8.0 job: https://github.com/dotnet/orleans/actions/runs/31554717699/job/93984651195
• Exact test: UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts
• Failure: After SendItem - PublisherCount for stream MemoryStreamProvider/StreamReliabilityNamespace/... = 2; expected 1
• Stack: StreamTestUtils.CheckPubSubCount line 88 -> CheckPubSubCounts line 59 -> StreamReliabilityTests.Test_AllSilosRestart_PubSubCounts line 747 -> SMS_StreamRel_AllSilosRestart_PubSubCounts line 269.

[ReubenBond]

Additional recurrence on PR #10587 after rebasing onto current main:

• Workflow run: 31824115099
• Job: Azure Storage provider tests (net10.0, AzureStorage), 94844056528
• Head: ed790dfc5a575ba95b99069e6c4fdd6c97a7e072
• Test: UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts
• Failure: after SendItem, publisher count was 2, expected 1, at StreamTestUtils.CheckPubSubCount line 88 / StreamReliabilityTests.Test_AllSilosRestart_PubSubCounts line 747.
• Suite result: 1 failed, 278 passed, 35 skipped.

PR #10587 changes membership status publication, placement candidate selection, lifecycle stop tracking, version-selector caching, and related Core tests. It does not modify streaming or Azure provider code. The failed job is being rerun.

[ReubenBond]

Rerun update: workflow attempt 2 did not reproduce SMS_StreamRel_AllSilosRestart_PubSubCounts. The Azure net10 job failed instead in the separate reminder-suite flake tracked by #10499 (Rem_Azure_UpdateReminder_DoesNotRestartLocalReminder, missing expected diagnostic update event). This supports the stream failure in attempt 1 being intermittent.

[ReubenBond]

Recurrence on final integration PR #10599 after review fix, head 49ea8ebc01750dc0f1d88aa1b3380452a13842a9: Azure Storage net10 job https://github.com/dotnet/orleans/actions/runs/31843071044/job/94903878381 failed SMS_StreamRel_AllSilosRestart_PubSubCounts. The PR changes placement compatibility caching and a membership-service test double only; Azure net8 remains in progress on the same head.

[ReubenBond]

Recurrence on PR #10569 after rebasing onto current main, head \98d1b1972cead6bf6e06a11fe5b75d530b0682c0: Azure Storage net10 job https://github.com/dotnet/orleans/actions/runs/32329032710/job/96305981800 failed \SMS_StreamRel_AllSilosRestart_PubSubCounts. After \SendItem, publisher count was \2, expected \1. PR #10569 changes activation scheduler ownership and ExecutionContext handling; it does not modify streaming. The failed job is being rerun.

[ReubenBond]

PR #10532 is rebased onto current \main\ at \�7a9e3e1c. Its post-rebase CI completed with all 72 required checks passing, including Azure Storage on net8.0 and net10.0. The targeted restart regression, fresh-membership validation, and rolling RPC compatibility tests also pass on both runtime target frameworks. The PR is ready for maintainer approval.