feat(reminders): add advanced reminder service

[KSemenenko]

Adaptive Reminder Service - Proposal #9586

Microsoft Reviewers: Open in CodeFlow

Problem

Classic Orleans reminders load reminder definitions into silo memory during startup and partition them across the cluster. Total registration count therefore contributes directly to startup work and cluster memory pressure. Large deployments need bounded recovery pages, near-term delivery loading, durable far-future schedules, and paged administration.

Solution

This PR adds the opt-in Microsoft.Orleans.AdvancedReminders* package family, built on Orleans.DurableJobs, while preserving the existing Orleans.Reminders* API and runtime contract.

Advanced Reminders provides:

• interval, absolute UTC, one-shot, and timezone-aware cron schedules;
• grain, registry, and activation-attribute registration APIs;
• Azure Storage, Cosmos DB, DynamoDB, Redis, and ADO.NET definition providers;
• paged management, filtering, iteration, priority, missed-occurrence, repair, and cleanup controls;
• stable schedule identity and at-least-once delivery through Durable Jobs.

Scalability guarantees

This design resolves #947 through explicit runtime boundaries:

• Silo startup does not materialize the reminder table. Reconciliation traverses 4,096 hash ranges, advances through 256 ranges per cycle, and requests provider continuation pages capped at 256 rows.
• Every provider implements native bounded recovery paging: stable keysets for in-memory and ADO.NET, Azure Table and Cosmos continuation tokens, DynamoDB Limit/LastEvaluatedKey, and Redis immutable lexicographic key continuations.
• Each registration persists only its next occurrence. Recurring and cron schedules are expanded one occurrence at a time.
• DurableJobsOptions.ShardLoadLookaheadPeriod controls the near-term delivery horizon. Far-future Advanced Reminder definitions persist without creating Durable Jobs or resident writable shards; recovery creates the job after the occurrence enters the horizon.
• Advanced Reminders enforces a 32-minute structural lookahead floor, and operators size the horizon above measured full-scan duration and recovery margin.
• Each shard holds at most 10,000 live jobs by default, dequeues at most 1,024 ready jobs per pass, and applies configurable operation-count, encoded-size, and pending-operation limits.
• ShardStripeCount distributes concentrated due-time windows across independent journals, while MaxConcurrentJobsPerSilo bounds callback execution.
• Administrative queries use continuation tokens and bounded pages. The Dashboard avoids a full-table count for each page while preserving classic First/Previous/Next/Last navigation.
• Persisted job handles remain authoritative during execution and retries. Recovery groups membership checks by shard, rotates occurrence identity before replacement, and ignores superseded jobs.

These guarantees make startup and active-delivery memory depend on bounded pages and the configured near-term horizon rather than total reminder count.

Operational behavior

The reminder table is the source of truth. The dispatcher re-reads the definition and validates schedule identity before delivery and again after the callback. Updates, unregister operations, repairs, retries, and rolling recovery preserve the current registration generation.

Provider storage is service-scoped, concurrency-controlled, and validated. Cleanup remains policy-driven through DeleteReminderWhenGrainTypeIsUnavailable, MaximumDeliveryAttempts, and explicit paged management operations.

Fixes #947

[KSemenenko]

@benjaminpetit

[Copilot]

Pull request overview

This PR advances the “Adaptive Reminder Service” proposal by extending Orleans reminders with cron-based scheduling and adaptive metadata (next-due, last-fire, priority/action), adding management/paging APIs and iterators, and updating multiple storage providers plus tests to round-trip the new fields.

Changes:

• Add cron scheduling support and adaptive reminder metadata (CronExpression, NextDueUtc, LastFireUtc, Priority, Action) across reminder contracts and storage providers.
• Introduce reminder management paging/filtering APIs plus iterator/extension helpers and new validation/registration conveniences (including [RegisterReminder] activation registration).
• Update timer infrastructure and expand test coverage across NonSilo + provider-specific reminder table tests, plus add SQL migrations.

Reviewed changes

Copilot reviewed 86 out of 86 changed files in this pull request and generated 22 comments.

Show a summary per file

File	Description
test/TesterInternal/RemindersTest/ReminderTableTestsBase.cs	Adds round-trip tests for cron + adaptive fields in reminder table providers.
test/NonSilo.Tests/Reminders/SiloBuilderReminderExtensionsTests.cs	Verifies DI registrations for legacy vs adaptive reminder services and options wiring.
test/NonSilo.Tests/Reminders/ReminderStressTests.cs	Adds high-load paging/iterator stress coverage for reminder management APIs.
test/NonSilo.Tests/Reminders/ReminderRegistryValidationTests.cs	Adds validation tests for new registry overloads (cron/absolute/priority/action).
test/NonSilo.Tests/Reminders/ReminderRegistrationExtensionsTests.cs	Tests typed cron builder/expression extension overloads for registry/service/grain.
test/NonSilo.Tests/Reminders/ReminderOptionsValidatorTests.cs	Adds validation coverage for new adaptive reminder options.
test/NonSilo.Tests/Reminders/ReminderManagementGrainExtensionsTests.cs	Tests paging enumeration extension helpers and iterator factory.
test/NonSilo.Tests/Reminders/ReminderIteratorTests.cs	Tests ReminderIterator paging behavior and large-streaming scenario.
test/NonSilo.Tests/Reminders/RegisterReminderAttributeTests.cs	Tests [RegisterReminder] attribute ctor behavior and validation.
test/NonSilo.Tests/Reminders/RegisterReminderActivationConfiguratorProviderTests.cs	Tests activation-time reminder registration behavior using lifecycle hooks.
test/NonSilo.Tests/NonSilo.Tests.csproj	Adds project reference to Orleans.Reminders for new reminder APIs/tests.
test/Grains/TestGrainInterfaces/IReminderTestGrain2.cs	Extends test grain interface to exercise new reminder options/cron/raw upserts.
test/Extensions/TesterAzureUtils/AzureRemindersTableTests.cs	Adds Azure reminder table round-trip tests for cron + adaptive fields.
test/Extensions/TesterAdoNet/StorageTests/DbExtensionsInt32ConversionTests.cs	Adds coverage for widened GetInt32 conversions and overflow behavior.
test/Extensions/TesterAdoNet/Reminders/SqlServerRemindersTableTests.cs	Adds SQL Server reminder table round-trip tests for new fields.
test/Extensions/TesterAdoNet/Reminders/PostgreSqlRemindersTableTests.cs	Adds PostgreSQL reminder table round-trip tests for new fields.
test/Extensions/TesterAdoNet/Reminders/MySqlRemindersTableTests.cs	Adds MySQL reminder table round-trip tests for new fields.
test/Extensions/Tester.Redis/Reminders/RedisReminderTableTests.cs	Adds Redis reminder table round-trip tests for new fields.
test/Extensions/Tester.Redis/Reminders/RedisReminderTableSerializationTests.cs	Adds Redis serialization/parsing tests for enums + temporal formatting.
test/Extensions/AWSUtils.Tests/Reminder/DynamoDBRemindersTableTests.cs	Adds DynamoDB reminder table round-trip tests for new fields.
test/Extensions/AWSUtils.Tests/Reminder/DynamoDBReminderTableEnumParsingTests.cs	Adds DynamoDB enum parsing tests for missing/invalid/valid values.
src/Redis/Orleans.Reminders.Redis/Storage/RedisReminderTable.cs	Extends Redis reminder payload format to include cron/temporal/enums with robust parsing.
src/Orleans.Runtime/Timers/AsyncTimerFactory.cs	Adds TimeProvider support to async timers via factory injection.
src/Orleans.Runtime/Timers/AsyncTimer.cs	Switches timer time sources/delays to TimeProvider for testability/determinism.
src/Orleans.Reminders/Timers/ReminderQueryFilter.cs	Adds server-side filter DTO + status mask enum for management paging queries.
src/Orleans.Reminders/Timers/ReminderManagementPage.cs	Introduces page DTO for reminder management results.
src/Orleans.Reminders/Timers/ReminderManagementGrainExtensions.cs	Adds IAsyncEnumerable helpers for paging through management APIs.
src/Orleans.Reminders/Timers/ReminderIterator.cs	Implements iterator facade over reminder management paging APIs.
src/Orleans.Reminders/Timers/IReminderRegistry.cs	Adds new registry overloads (absolute due, cron, priority/action).
src/Orleans.Reminders/Timers/IReminderManagementGrain.cs	Adds management grain interface with paging/filtering and administrative operations.
src/Orleans.Reminders/Timers/IReminderIterator.cs	Adds iterator interface for streaming pages without full materialization.
src/Orleans.Reminders/Timers/IRemindable.cs	Adds schedule kind + priority/action + cron surface area to reminder contracts.
src/Orleans.Reminders/SystemTargetInterfaces/IReminderTable.cs	Extends ReminderEntry + ReminderData to carry cron/temporal/enums.
src/Orleans.Reminders/SystemTargetInterfaces/IReminderService.cs	Adds new reminder service overloads (absolute due, cron, priority/action).
src/Orleans.Reminders/ReminderService/ReminderRegistry.cs	Adds validation + new overloads for cron/absolute due and priority/action.
src/Orleans.Reminders/ReminderService/RegisterReminderActivationConfiguratorProvider.cs	Adds activation-time [RegisterReminder] support via lifecycle observer.
src/Orleans.Reminders/ReminderService/LocalReminderService.cs	Adds TimeProvider usage, cron scheduling support, adaptive fields, and legacy suppression behavior.
src/Orleans.Reminders/ReminderService/AdaptiveReminderServiceRegistrationMarker.cs	Adds DI marker for suppressing legacy reminder service when adaptive is enabled.
src/Orleans.Reminders/ReminderCronRegistrationExtensions.cs	Adds typed-cron overloads for registry/service cron registration.
src/Orleans.Reminders/RegisterReminderAttribute.cs	Introduces [RegisterReminder] attribute with validation and priority/action support.
src/Orleans.Reminders/Orleans.Reminders.csproj	Enables unsafe blocks to support cron parsing implementation.
src/Orleans.Reminders/Options/ReminderOptions.cs	Adds adaptive options (look-ahead, poll interval, bucket size, priority toggle, legacy toggle) + validation.
src/Orleans.Reminders/Hosting/SiloBuilderReminderExtensions.cs	Adds overloads to configure ReminderOptions and wires activation configurator provider.
src/Orleans.Reminders/Hosting/SiloBuilderAdaptiveReminderExtensions.cs	Adds AddAdaptiveReminderService() DI wiring + idempotency behavior.
src/Orleans.Reminders/GrainReminderExtensions.cs	Adds reminder registration overloads for absolute due and priority/action.
src/Orleans.Reminders/GrainReminderCronExtensions.cs	Adds grain-level cron reminder registration extension methods.
src/Orleans.Reminders/Cron/ReminderCronExpression.cs	Adds typed cron expression wrapper with parsing/format detection and UTC guards.
src/Orleans.Reminders/Cron/ReminderCronBuilder.cs	Adds helper builder for common cron schedules and validation.
src/Orleans.Reminders/Cron/Internal/TimeZoneHelper.cs	Adds internal DST ambiguity helpers for cron scheduling.
src/Orleans.Reminders/Cron/Internal/ReminderCronParser.cs	Adds internal cron parsing helper (5/6 field detection) for runtime use.
src/Orleans.Reminders/Cron/Internal/LanguageFeatures.cs	Adds polyfills for compiler/runtime attributes for older target frameworks.
src/Orleans.Reminders/Cron/Internal/DateTimeHelper.cs	Adds internal DateTimeOffset rounding helpers for cron logic.
src/Orleans.Reminders/Cron/Internal/CronFormatException.cs	Adds custom exception for cron format failures.
src/Orleans.Reminders/Cron/Internal/CronFormat.cs	Adds internal enum controlling cron parse format options.
src/Orleans.Reminders/Cron/Internal/CronField.cs	Adds cron field metadata (ranges/names) used by parser.
src/Orleans.Reminders/Cron/Internal/CronExpressionFlag.cs	Adds internal flags to represent cron expression modifiers.
src/Orleans.Reminders/Cron/Internal/CalendarHelper.cs	Adds internal calendar helpers for cron evaluation.
src/Orleans.Reminders/Constants/ReminderOptionsDefaults.cs	Adds defaults for look-ahead, poll interval, and bucket size.
src/Azure/Shared/Storage/AzureTableDataManager.cs	Adds optional update mode to table upsert, enabling replace semantics when needed.
src/Azure/Orleans.Reminders.Cosmos/Models/ReminderEntity.cs	Adds cron/temporal/enums fields to Cosmos reminder entity model.
src/Azure/Orleans.Reminders.Cosmos/CosmosReminderTable.cs	Reads/writes cron/temporal/enums fields for Cosmos reminder storage.
src/Azure/Orleans.Reminders.AzureStorage/Storage/RemindersTableManager.cs	Adds columns for cron/temporal/enums and switches upsert to Replace mode.
src/Azure/Orleans.Reminders.AzureStorage/Storage/AzureBasedReminderTable.cs	Adds parsing/serialization for new fields, with tighter exception handling.
src/AdoNet/Shared/Storage/RelationalOrleansQueries.cs	Reads new reminder columns and extends upsert to persist new fields/enums.
src/AdoNet/Shared/Storage/DbStoredQueries.cs	Adds new column parameter bindings and simplifies version read using GetInt32.
src/AdoNet/Shared/Storage/DbExtensions.cs	Expands GetInt32 conversion support with invariant conversion and nullable variant.
src/AdoNet/Orleans.Reminders.AdoNet/SQLServer-Reminders.sql	Adds new columns and index + updates stored queries for SQL Server provider.
src/AdoNet/Orleans.Reminders.AdoNet/ReminderService/AdoNetReminderTable.cs	Passes new reminder fields through to relational upsert.
src/AdoNet/Orleans.Reminders.AdoNet/PostgreSQL-Reminders.sql	Adds new columns/index and updates upsert function/queries for PostgreSQL provider.
src/AdoNet/Orleans.Reminders.AdoNet/Oracle-Reminders.sql	Adds new columns/index and updates function/queries for Oracle provider.
src/AdoNet/Orleans.Reminders.AdoNet/MySQL-Reminders.sql	Adds new columns/index and updates stored queries for MySQL provider.
src/AdoNet/Orleans.Reminders.AdoNet/Migrations/SQLServer-Reminders-10.0.0.sql	Adds migration script for pre-10.0.0 SQL Server reminder tables.
src/AdoNet/Orleans.Reminders.AdoNet/Migrations/PostgreSQL-Reminders-10.0.0.sql	Adds migration script for pre-10.0.0 PostgreSQL reminder tables.
src/AdoNet/Orleans.Reminders.AdoNet/Migrations/Oracle-Reminders-10.0.0.sql	Adds migration script for pre-10.0.0 Oracle reminder tables.
src/AdoNet/Orleans.Reminders.AdoNet/Migrations/MySQL-Reminders-10.0.0.sql	Adds migration script for pre-10.0.0 MySQL reminder tables.
src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs	Adds cron/temporal/enums read/write behavior with robust parsing defaults.

Comments suppressed due to low confidence (1)

test/Grains/TestInternalGrains/ReminderTestGrain2.cs:287

• The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.
  The format string ignores this supplied value.

[rkargMsft]

Do we want to be changing the existing Reminder interface as opposed to creating a new one that can exist side by side (with perhaps a separate migration process to move existing reminder instances to the new interface)?

Also, I had assumed that new reminders would be built on top of the new DurableJob construct. Was that an expectation or have I misremembered?

[KSemenenko]

@ReubenBond I don't know much about DurableJob but I would love to help here, becase I really want reminders with cron so bad =)

[KSemenenko]

@ReubenBond @rkargMsft shoud I move this into separate project? or just add separete exteions like UseAdaptiveReminders and then add extra extions for registeter for remainders?

[KSemenenko]

@pentp comments are resolved

Also I gives me an idea to add one more propery TimeZone - in case I want to run remainder each 9 am each morneing in ET time.

I found this is usfull. what do you think?

[KSemenenko]

@copilot review

[Copilot]

Pull request overview

Copilot reviewed 138 out of 138 changed files in this pull request and generated 15 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[egil]

Does this include a way to register a fake TimeProvider in the services collection of a InProcessTestClusterBuilder and the be able to effectively test the reminder and related logic?

[KSemenenko]

@egil Yes, with the current shape this can be done via the normal host configuration hooks on InProcessTestClusterBuilder by replacing the DI registration for TimeProvider.

Both client and silo default registrations use TryAddSingleton<TimeProvider>(TimeProvider.System), so tests can override that with something like:

var fakeTimeProvider = new FakeTimeProvider();
var builder = new InProcessTestClusterBuilder();
builder.ConfigureHost(hostBuilder =>
{
    hostBuilder.Services.Replace(ServiceDescriptor.Singleton<TimeProvider>(fakeTimeProvider));
});

That flows through to the silo runtime, and the reminder/timer path uses the DI-provided TimeProvider, so reminder logic can be tested against a fake provider that way.

I also added a small integration test in this branch to lock that in: InProcessTestClusterBuilderTimeProviderTests.ConfigureHost_CanReplaceTimeProvider_ForClientAndSiloServices.

[egil]

Awesome to see TimeProvider supported. I would suggest adding a few more test cases demonstrating controlling time during testing.

[KSemenenko]

@egil done!

[Copilot]

Pull request overview

Copilot reviewed 147 out of 148 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 160 out of 161 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 160 out of 161 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/Dashboard/Orleans.Dashboard/ServiceCollectionExtensions.cs:333

• The fallback response type is swapped: this classic reminders endpoint returns an AdvancedReminderResponse when reminders are not configured, which will break the dashboard client contract (it expects ReminderResponse).
  src/Dashboard/Orleans.Dashboard/ServiceCollectionExtensions.cs:351
• The fallback response type is swapped: this advanced reminders endpoint returns a ReminderResponse when advanced reminders are not configured, which will break the dashboard client contract (it expects AdvancedReminderResponse).
  src/Azure/Orleans.AdvancedReminders.AzureStorage/AzureStorageReminderSiloBuilderReminderExtensions.cs:74
• The ISiloBuilder overload taking a connection string only configures TableServiceClient, but the advanced reminders Azure Storage provider also requires BlobServiceClient for the durable-jobs backend. This overload will leave BlobServiceClient null and can fail at runtime/validation.

[KSemenenko]

Reason: In two test-cluster implementations, the timeout check was inverted - the cluster stopped stabilizing after about 100 ms instead of waiting up to 60 seconds. Due to incomplete membership convergence, two silos could deliver the same reminder tick at the same time.

is this looks right? @ReubenBond

[KSemenenko]

@ReubenBond shoud this src/Orleans.AdvancedReminders/ReminderService/ReminderManagementGrain.cs be called AdvancedReminderManagementGrain ?

[ReubenBond]

Current CI has a deterministic PR-specific dashboard regression across every BVT platform. MapOrleansDashboard_NoPrefix_MapsExactRootStaticAndApiEndpointInventory, MapOrleansDashboard_CustomPrefix_MapsExactNestedInventoryWithoutRootEndpoints, and MapOrleansDashboard_ReturnedGroupRequireAuthorization_AppliesNamedPolicyToEveryEndpoint all fail because the new /AdvancedReminders endpoint is present but missing from the expected endpoint inventory/order. Update the dashboard routing expectations (and authorization coverage) for the new endpoint before retrying. Failed run: https://github.com/dotnet/orleans/actions/runs/33171066904