test(reminders): add lifecycle conformance scenarios

src/Orleans.Reminders.TestKit/README.md

@@ -12,6 +12,8 @@ cardinality setup and cleanup, bounded retries, diagnostics, and failure message
 | --- | --- |
 | `ReminderTableTestRunner` | Direct conformance facts for every documented table guarantee. |
 | `ReminderServiceTestRunner` | Cluster-level registration, replacement, lookup, enumeration, and removal conformance. |
+| `ReminderServiceLifecycleTestRunner` | Deterministic startup, ownership, exact-due, reconciliation, churn, and cleanup-isolation conformance. |
+| `IReminderServiceLifecycleHarness` | Adapter contract for one cluster, one reminder clock, diagnostics, and explicit topology barriers. |
 | `ReminderTableModelBasedTestRunner` | Generated sequential conformance against the same full contract. |
 | `IdealizedReminderTable` | Deterministic, strongly consistent reference implementation and fault-injection oracle. |
 | `ReminderTableTestFixture` | In-process cluster fixture which deploys and resolves a provider. |
@@ -173,6 +175,54 @@ public sealed class MyReminderServiceTests
 }
 ```
 
+### Run lifecycle and churn conformance
+
+`ReminderServiceLifecycleTestRunner` adds the shared service-level contract. The same eight scenarios run for every
+service provider: startup readiness, single registration ownership, in-place schedule update, removal quiescence,
+exact-due recovery, stale-owner registration reconciliation, one-silo join/leave transfer, and cleanup isolation.
+
+Use `ReminderTestClock` as the sole time driver and `ReminderDiagnosticObserver` as the lifecycle/tick source. The
+`ReminderServiceLifecycleHarness` adapter for `InProcessTestCluster` supplies explicit membership and reminder-range
+reconciliation barriers:
+
+```csharp
+var clock = builder.AddReminderTestClock();
+var cluster = builder.Build();
+await cluster.DeployAsync();
+
+var options = cluster.Silos[0].ServiceProvider
+    .GetRequiredService<IOptions<ReminderOptions>>().Value;
+var harness = new ReminderServiceLifecycleHarness(
+    cluster,
+    clock,
+    clock.DiagnosticObserver,
+    options.ReminderLoadingWindow);
+
+public sealed class MyLifecycleTests : ReminderServiceLifecycleTestRunner
+{
+    public MyLifecycleTests(IReminderServiceLifecycleHarness harness)
+        : base(harness, "MyProvider", seed: 42)
+    {
+    }
+
+    [Fact]
+    public override Task ReminderService_OneSiloJoinLeaveTransfersOwnership()
+        => base.ReminderService_OneSiloJoinLeaveTransfersOwnership();
+}
+```
+
+Do not replace harness barriers with delays, retry loops, longer timeouts, or provider-specific skips. Scenario cleanup
+uses its own bounded token, removes only deterministic scenario rows using their current ETags, advances one explicit
+refresh for owner quiescence, and verifies their absence; it never clears unrelated provider rows or replaces the
+original scenario failure. Ownership assertions count local reminder instance identities, including duplicate
+instances on one silo, rather than counting distinct silo addresses.
+
+The built-in in-memory, Azure Table, Cosmos DB, ADO.NET SQL Server, PostgreSQL, MySQL, Redis, DynamoDB, and Firestore
+providers all expose these same inherited facts. Their adapters contain only backend precondition/setup and provider
+registration. External-service availability can skip fixture construction, but no provider disables individual
+lifecycle guarantees. Add a documented capability boundary here before omitting a future provider which cannot host
+the Orleans reminder service or participate in in-process silo churn.
+
 ## Deterministic oracle and cluster testing
 
 `IdealizedReminderTable` supplies a strongly consistent reference implementation for TestKit self-tests and
@@ -200,7 +250,9 @@ The oracle exposes:
 - `FreezeReads` for stale-read convergence scenarios; and
 - lifecycle cancellation and invariant checks.
 
-The TestKit cluster integration suite uses these controls to cover exact-due delivery, exact-due storage recovery,
+`ReminderTestClock` creates its lifecycle observer before the cluster is built, allowing startup conformance to await
+one `ReminderServiceStarted` event for every silo before liveness and range-reconciliation barriers. The TestKit
+cluster integration suite uses these controls to cover exact-due delivery, exact-due storage recovery,
 due times beyond the platform timer limit, stale-refresh suppression after unregister, and single-owner delivery in a
 multi-silo cluster.
 

src/Orleans.Reminders/Orleans.Reminders.csproj

@@ -21,9 +21,9 @@
     <InternalsVisibleTo Include="Orleans.Azure.Tests" />
     <InternalsVisibleTo Include="Orleans.Core.Tests" />
     <InternalsVisibleTo Include="Orleans.Reminders.Tests" />
+    <InternalsVisibleTo Include="Orleans.Testing.Reminders" />
     <InternalsVisibleTo Include="Orleans.Runtime" />
     <InternalsVisibleTo Include="Orleans.Runtime.Internal.Tests" />
     <InternalsVisibleTo Include="TestInternalGrains" />
   </ItemGroup>
 </Project>
-

src/Orleans.Runtime/Orleans.Runtime.csproj

@@ -40,6 +40,7 @@
     <InternalsVisibleTo Include="Orleans.Streaming" />
     <InternalsVisibleTo Include="Orleans.Streaming.Tests" />
     <InternalsVisibleTo Include="Orleans.TestingHost" />
+    <InternalsVisibleTo Include="Orleans.Testing.Reminders" />
     <InternalsVisibleTo Include="TestInternalGrains" />
   </ItemGroup>
 </Project>

test/Extensions/Orleans.AWS.Tests/Orleans.AWS.Tests.csproj

@@ -19,6 +19,7 @@
     <ProjectReference Include="$(SourceRoot)src\Orleans.Persistence.TestKit\Orleans.Persistence.TestKit.csproj" PrivateAssets="all" />
     <ProjectReference Include="$(SourceRoot)src\Orleans.Reminders.TestKit\Orleans.Reminders.TestKit.csproj" PrivateAssets="all" />
     <ProjectReference Include="$(SourceRoot)test\Orleans.Runtime.Internal.Tests\Orleans.Runtime.Internal.Tests.csproj" />
+    <ProjectReference Include="$(SourceRoot)test\Orleans.Reminders.Tests\Orleans.Reminders.Tests.csproj" />
     <ProjectReference Include="$(SourceRoot)test\Orleans.Streaming.Tests\Orleans.Streaming.Tests.csproj" />
   </ItemGroup>
 </Project>

test/Extensions/Orleans.AWS.Tests/Reminder/DynamoDBRemindersTableTests.cs

@@ -2,14 +2,75 @@
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
 using Orleans.Configuration;
+using Orleans.Hosting;
 using Orleans.Reminders.DynamoDB;
+using Orleans.Testing.Reminders;
+using Orleans.TestingHost;
 using TestExtensions;
 using UnitTests;
 using UnitTests.RemindersTest;
+using UnitTests.TimerTests;
 using Xunit;
 
 namespace AWSUtils.Tests.RemindersTest
 {
+    public sealed class DynamoDBReminderServiceLifecycleFixture : BaseInProcessTestClusterFixture
+    {
+        private ReminderTestClock? _clock;
+
+        public ReminderTestClock Clock
+        {
+            get
+            {
+                EnsurePreconditionsMet();
+                return _clock ?? throw new InvalidOperationException("The reminder clock has not been configured.");
+            }
+        }
+
+        protected override void CheckPreconditionsOrThrow()
+        {
+            if (!AWSTestConstants.IsDynamoDbAvailable)
+            {
+                throw Xunit.Sdk.SkipException.ForSkip("Unable to connect to AWS DynamoDB simulator");
+            }
+        }
+
+        protected override void ConfigureTestCluster(InProcessTestClusterBuilder builder)
+        {
+            _clock = builder.AddReminderTestClock();
+            builder.ConfigureSilo((_, siloBuilder) =>
+                siloBuilder.UseDynamoDBReminderService(options =>
+                    options.ParseConnectionString($"Service={AWSTestConstants.DynamoDbService}")));
+        }
+
+        public override async ValueTask DisposeAsync()
+        {
+            try
+            {
+                await base.DisposeAsync();
+            }
+            finally
+            {
+                _clock?.Dispose();
+            }
+        }
+    }
+
+    [TestCategory("Reminders"), TestCategory("AWS"), TestCategory("DynamoDb")]
+    [Collection(TestEnvironmentFixture.DefaultCollection)]
+    [TestSuite("Functional")]
+    [TestProvider("DynamoDB")]
+    [TestArea("Reminders")]
+    public sealed class DynamoDBReminderServiceLifecycleTests
+        : ReminderServiceLifecycleTestsBase, IClassFixture<DynamoDBReminderServiceLifecycleFixture>
+    {
+        public DynamoDBReminderServiceLifecycleTests(DynamoDBReminderServiceLifecycleFixture fixture)
+            : base(fixture.Clock, fixture.HostedCluster, "DynamoDB")
+        {
+            fixture.EnsurePreconditionsMet();
+        }
+    }
+
     /// <summary>
     /// Tests DynamoDB implementation of the Orleans reminders table for storing and retrieving grain reminders.
     /// </summary>

test/Extensions/Orleans.AdoNet.Tests/Reminders/ReminderServiceLifecycleTests_AdoNet.cs

@@ -0,0 +1,110 @@
+using Orleans.Hosting;
+using Orleans.Tests.SqlUtils;
+using Orleans.Testing.Reminders;
+using Orleans.TestingHost;
+using TestExtensions;
+using UnitTests.TimerTests;
+
+namespace Tester.AdoNet.Reminders;
+
+public abstract class AdoNetReminderServiceLifecycleFixture : BaseInProcessTestClusterFixture
+{
+    private string _connectionString = null!;
+    private ReminderTestClock? _clock;
+
+    protected abstract string Invariant { get; }
+
+    protected abstract string DatabaseName { get; }
+
+    public ReminderTestClock Clock
+    {
+        get
+        {
+            EnsurePreconditionsMet();
+            return _clock ?? throw new InvalidOperationException("The reminder clock has not been configured.");
+        }
+    }
+
+    protected override void CheckPreconditionsOrThrow()
+        => UnitTests.General.RelationalStorageForTesting.CheckPreconditionsOrThrow(Invariant);
+
+    public override async ValueTask InitializeAsync()
+    {
+        if (!PreconditionsMet)
+        {
+            return;
+        }
+
+        var relationalStorage = await UnitTests.General.RelationalStorageForTesting.SetupInstance(
+            Invariant,
+            DatabaseName,
+            cancellationToken: TestContext.Current.CancellationToken);
+        _connectionString = relationalStorage.CurrentConnectionString;
+        await base.InitializeAsync();
+    }
+
+    protected override void ConfigureTestCluster(InProcessTestClusterBuilder builder)
+    {
+        _clock = builder.AddReminderTestClock();
+        builder.ConfigureSilo((_, siloBuilder) =>
+            siloBuilder.UseAdoNetReminderService(options =>
+            {
+                options.ConnectionString = _connectionString;
+                options.Invariant = Invariant;
+            }));
+    }
+
+    public override async ValueTask DisposeAsync()
+    {
+        try
+        {
+            await base.DisposeAsync();
+        }
+        finally
+        {
+            _clock?.Dispose();
+        }
+    }
+}
+
+public sealed class PostgreSqlReminderServiceLifecycleFixture : AdoNetReminderServiceLifecycleFixture
+{
+    protected override string Invariant => AdoNetInvariants.InvariantNamePostgreSql;
+
+    protected override string DatabaseName => "OrleansTest_PostgreSql_ReminderLifecycle";
+}
+
+public sealed class MySqlReminderServiceLifecycleFixture : AdoNetReminderServiceLifecycleFixture
+{
+    protected override string Invariant => AdoNetInvariants.InvariantNameMySql;
+
+    protected override string DatabaseName => "OrleansTest_MySql_ReminderLifecycle";
+}
+
+[TestSuite("Functional")]
+[TestProvider("PostgreSql")]
+[TestArea("Reminders")]
+[TestCategory("Functional"), TestCategory("Reminders"), TestCategory("AdoNet"), TestCategory("PostgreSql")]
+public sealed class PostgreSqlReminderServiceLifecycleTests
+    : ReminderServiceLifecycleTestsBase, IClassFixture<PostgreSqlReminderServiceLifecycleFixture>
+{
+    public PostgreSqlReminderServiceLifecycleTests(PostgreSqlReminderServiceLifecycleFixture fixture)
+        : base(fixture.Clock, fixture.HostedCluster, "AdoNet.PostgreSql")
+    {
+        fixture.EnsurePreconditionsMet();
+    }
+}
+
+[TestSuite("Functional")]
+[TestProvider("MySql")]
+[TestArea("Reminders")]
+[TestCategory("Functional"), TestCategory("Reminders"), TestCategory("AdoNet"), TestCategory("MySql")]
+public sealed class MySqlReminderServiceLifecycleTests
+    : ReminderServiceLifecycleTestsBase, IClassFixture<MySqlReminderServiceLifecycleFixture>
+{
+    public MySqlReminderServiceLifecycleTests(MySqlReminderServiceLifecycleFixture fixture)
+        : base(fixture.Clock, fixture.HostedCluster, "AdoNet.MySql")
+    {
+        fixture.EnsurePreconditionsMet();
+    }
+}

test/Extensions/Orleans.AdoNet.Tests/Reminders/ReminderTests_AdoNet_SqlServer.cs

@@ -17,6 +17,20 @@
 
 namespace Tester.AdoNet.Reminders
 {
+    [TestSuite("Functional")]
+    [TestProvider("SqlServer")]
+    [TestArea("Reminders")]
+    [TestCategory("Reminders"), TestCategory("AdoNet"), TestCategory("SqlServer")]
+    public sealed class ReminderServiceLifecycleTests_AdoNet_SqlServer
+        : ReminderServiceLifecycleTestsBase, IClassFixture<ReminderTests_AdoNet_SqlServer.Fixture>
+    {
+        public ReminderServiceLifecycleTests_AdoNet_SqlServer(ReminderTests_AdoNet_SqlServer.Fixture fixture)
+            : base(fixture.ReminderClock, fixture.HostedCluster, "AdoNet.SqlServer")
+        {
+            fixture.EnsurePreconditionsMet();
+        }
+    }
+
     /// <summary>
     /// Integration tests for Orleans reminders functionality using SQL Server as the reminder service backend.
     /// </summary>
@@ -101,7 +115,7 @@ public async ValueTask InitializeAsync()
             await ClearReminderTableAsync(TestContext.Current.CancellationToken)
                 .WaitAsync(TestConstants.InitTimeout, TestContext.Current.CancellationToken);
         }
-        
+
         // Basic tests
 
         [Fact]

test/Extensions/Orleans.Azure.Tests/Reminder/ReminderTests_AzureTable.cs

@@ -14,6 +14,20 @@
 
 namespace Tester.AzureUtils.TimerTests
 {
+    [TestSuite("Functional")]
+    [TestProvider("AzureStorage")]
+    [TestArea("Reminders")]
+    [TestCategory("Reminders"), TestCategory("AzureStorage")]
+    public sealed class ReminderServiceLifecycleTests_AzureTable
+        : ReminderServiceLifecycleTestsBase, IClassFixture<ReminderTests_AzureTable.Fixture>
+    {
+        public ReminderServiceLifecycleTests_AzureTable(ReminderTests_AzureTable.Fixture fixture)
+            : base(fixture.ReminderClock, fixture.HostedCluster, "AzureStorage")
+        {
+            fixture.EnsurePreconditionsMet();
+        }
+    }
+
     /// <summary>
     /// Tests for Azure Table Storage-based reminder service, including basic operations, failover, and multi-grain scenarios.
     /// </summary>

test/Extensions/Orleans.Cosmos.Tests/ReminderTests_Cosmos.cs

@@ -11,6 +11,20 @@
 
 namespace Tester.Cosmos.Reminders;
 
+[TestSuite("Functional")]
+[TestProvider("Cosmos")]
+[TestArea("Reminders")]
+[TestCategory("Reminders"), TestCategory("Cosmos")]
+public sealed class ReminderServiceLifecycleTests_Cosmos
+    : ReminderServiceLifecycleTestsBase, IClassFixture<ReminderTests_Cosmos.Fixture>
+{
+    public ReminderServiceLifecycleTests_Cosmos(ReminderTests_Cosmos.Fixture fixture)
+        : base(fixture.ReminderClock, fixture.HostedCluster, "Cosmos")
+    {
+        fixture.EnsurePreconditionsMet();
+    }
+}
+
 /// <summary>
 /// Tests for Orleans reminders functionality using Azure Cosmos DB as the reminder service backing store.
 /// </summary>

test/Extensions/Orleans.Redis.Tests/Orleans.Redis.Tests.csproj

@@ -22,6 +22,7 @@
     <ProjectReference Include="$(SourceRoot)src\Orleans.Persistence.TestKit\Orleans.Persistence.TestKit.csproj" PrivateAssets="all" />
     <ProjectReference Include="$(SourceRoot)src\Orleans.Reminders.TestKit\Orleans.Reminders.TestKit.csproj" PrivateAssets="all" />
     <ProjectReference Include="$(SourceRoot)test\Orleans.Runtime.Internal.Tests\Orleans.Runtime.Internal.Tests.csproj" />
+    <ProjectReference Include="$(SourceRoot)test\Orleans.Reminders.Tests\Orleans.Reminders.Tests.csproj" />
     <ProjectReference Include="$(SourceRoot)test\Orleans.Streaming.Tests\Orleans.Streaming.Tests.csproj" />
     <ProjectReference Include="$(SourceRoot)src\Redis\Orleans.Journaling.Redis\Orleans.Journaling.Redis.csproj" />
   </ItemGroup>