feat(reminders): harden and migrate DynamoDB reminder reads

[ReubenBond]

Fixes #10909

Problem

DynamoDB reminder point reads use strongly consistent base-table reads, but grain and hash-range reads depend on eventually consistent GSIs. A GSI can return stale candidates or omit completed writes. Scanning the shared table on silo startup, topology changes, or periodic refresh would make capacity and latency scale with the full table multiplied by silo count, so that approach is explicitly rejected.

Cost-bounded Legacy hardening

Legacy mode treats GSI rows as discovery candidates and strongly point-validates each one. Before removing a locally known reminder omitted by discovery, the runtime strongly point-confirms absence. Completed mutations reconcile through strong point reads and a bounded three-hop owner notification; per-identity generations prevent out-of-order reads from restoring older schedules. Notifications which cannot identify a stable owner are logged and fall back to normal GSI convergence.

Legacy startup, topology changes, and periodic refresh never perform full-table scans. Their cost is bounded to existing GSI range queries, strong point reads for returned candidates or known omissions, and at most four owner-notification messages/point reads per mutation.

This is the strongest safe no-key-schema-change path, but it cannot supply complete strong set reads. A GSI omission reveals no identity to point-read, so cold startup, newly acquired ranges, arbitrary grain/range reads, and exhausted topology notifications can miss completed writes until GSI propagation. Retries cannot repair that information gap without a scan, and the implementation does not disguise it with either mechanism.

V2 solution

The complete guarantee uses a versioned V2 table whose base key is sharded across 32 ServiceId-scoped hash buckets. Fixed-width unsigned hash prefixes and bounded, delimiter-safe identity components support exact point, grain-prefix, normal range, and wrap-around queries using strongly consistent base-table reads. Long identities use collision-checked SHA-256 key components to remain within DynamoDB limits.

Migration protocol

Migration is explicit and resumable. Migrate retains V1 reads, transactionally dual-writes V1/V2, acquires a renewable generation-fenced lease, checkpoints strongly consistent V1 migration pages, reconciles deletes and concurrent updates, and persists Ready only after bounded-memory bidirectional verification. V2 checks fresh compatibility markers for every nonterminal cluster member, repeats fenced verification, confirms stable membership, then publishes Cutover. Rollback preserves V2 reads until verification succeeds and publishes RolledBack. V2Only performs an irreversible final verification and fence before operators retire V1; legacy data is never deleted automatically.

Rationale

DynamoDB cannot strongly read GSIs and key schemas are immutable. Cost-bounded Legacy hardening safely rejects stale known rows and accelerates successful mutations, but only a separate sharded base table can make unknown identities discoverable by strongly consistent key queries without production full-table scans.

Operational rollout

Existing deployments receive scan-free Legacy hardening without a schema change. For the complete guarantee, deploy all silos with Migrate, wait for Ready, then deploy V2. Keep transactional dual writes for the rollback window and move every silo to V2Only before removing V1. A pre-protocol binary must never be started after cutover. The provider documentation covers IAM, migration scan capacity, interruption recovery, observability, residual Legacy guarantees, rollback, and retirement.

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Schema.cs — CreateDualWriteFence uses a condition expression which fails when the migration state item…
	test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBReminderMigrationTests.cs — CreateClient always constructs BasicAWSCredentials for non-local DynamoDB. If access/secret are…

What changed in this PR

Adds a new DynamoDB reminders V2 table schema and an explicit, resumable migration protocol to enable strongly consistent point, grain-prefix, and hash-range reads without relying on eventually-consistent GSIs. This fits into the Orleans AWS reminders provider by introducing a sharded base-key design plus operator-controlled migration modes, backed by functional tests and updated documentation.

Changes:

• Introduces a sharded V2 DynamoDB reminders schema (32 buckets) with delimiter-safe, bounded key encoding enabling strongly consistent queries on the base table.
• Implements a fenced, lease-based migration workflow (backfill, verify, cutover, rollback, retire) with cluster compatibility markers and resumable checkpoints.
• Adds new configuration surface (TableMode, V2TableName, MigrationPageSize), wiring in builder/config parsing, plus extensive test coverage and documentation updates.

File	Description
test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBRemindersTableTests.cs	Adds schema and legacy key-encoding verification tests plus stronger isolation/consistency assertions.
test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBReminderMigrationTests.cs	Adds end-to-end functional tests for migration states, fencing, contention, rollback/retirement, and range semantics.
src/​AWS/​Shared/​Storage/​DynamoDBStorage.cs	Adds cancellation-aware query paging and a bounded strongly-consistent scan-page helper used by migration.
src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Schema.cs	Implements V2 table schema init, key encoding, strongly consistent V2 reads, and dual-write/V2-only write paths.
src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Migration.cs	Implements lease-fenced backfill/reconcile/verify logic, compatibility markers, and mode-driven cutover/rollback/retire state machine.
src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.cs	Wires V2 init/migration startup, read-mode refresh, and write routing into the reminder table implementation.
src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDbReminderStorageOptionsExtensions.cs	Extends connection-string parsing to include V2 table and migration-mode settings.
src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDbReminderStorageOptions.cs	Adds public configuration surface for migration mode, V2 table name, and page size.
src/​AWS/​Orleans.Reminders.DynamoDB/​README.md	Documents the high-level strongly consistent migration rollout and operator workflow.
src/​AWS/​Orleans.Reminders.DynamoDB/​DynamoDBRemindersProviderBuilder.cs	Adds configuration binding for the new migration settings on the silo builder.
src/​api/​AWS/​Orleans.Reminders.DynamoDB/​Orleans.Reminders.DynamoDB.cs	Updates public API surface to include the new enum and options properties.
docs/​site/​src/​content/​docs/​grains/​reminders/​dynamodb.md	Adds detailed operational documentation for V2 migration, rollback, retirement, and prerequisites.

Suppressed comments (1)

src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:149

• StartAsync always calls InitializeTable for the legacy (V1) table using the configured CreateIfNotExists/UpdateIfExists flags. In TableMode=V2Only (and potentially other post-cutover modes), automatically recreating an accidentally-deleted V1 table is dangerous: the subsequent reconciliation step can treat all existing V2 rows as orphans and delete them. Consider failing closed if the legacy table is missing once migration has reached Cutover, and avoid creating V1 in V2Only mode (matching the docs).

            await this.storage.InitializeTable(this.options.TableName,
                new List<KeySchemaElement>
                {
                    new KeySchemaElement { AttributeName = REMINDER_ID_PROPERTY_NAME, KeyType = KeyType.HASH },
                    new KeySchemaElement { AttributeName = GRAIN_HASH_PROPERTY_NAME, KeyType = KeyType.RANGE }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBReminderMigrationTests.cs — CreateClient always constructs BasicAWSCredentials for non-local DynamoDB. If access/secret are… View resolved comment
	src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Schema.cs — CreateDualWriteFence uses a condition expression which fails when the migration state item… View resolved comment

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:120

• In the early-retired startup path, the compatibility heartbeat is started but InitializeMigration is not wrapped in the same try/catch used later. If InitializeMigration throws (for example, when configured with TableMode=Rollback against a Retired service), the heartbeat task will be left running and the CTS undisposed. Wrap InitializeMigration in a try/catch and stop the heartbeat on failure.

                {
                    await StartCompatibilityHeartbeat();
                    await InitializeMigration(cancellationToken);
                    return;
                }

src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:11

• using System.Linq; appears to be unused in this file, which will trigger CS8019 and can fail the build when warnings are treated as errors. Remove the unused using directive.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBRemindersTableTests.cs — These new DynamoDB integration tests don't skip when the DynamoDB simulator is unavailable. Unlike…

Suppressed comments (3)

test/Extensions/Orleans.AWS.Tests/Reminder/DynamoDBRemindersTableTests.cs:222

• This new DynamoDB integration test doesn't skip when the DynamoDB simulator is unavailable, so it can fail in environments where other DynamoDB tests are correctly skipped. Add an AWSTestConstants.IsDynamoDbAvailable guard (as done elsewhere in this file).

        public async Task DynamoDBReminderTable_UpsertRow_EncodesV1PrimaryKeyExactly()
        {
            const string ServiceId = "phase-1-key-service";
            const string ReminderName = "foo/bar\\#b_a_z?";
            var tableName = $"OrleansReminders-{Guid.NewGuid():N}";

test/Extensions/Orleans.AWS.Tests/Reminder/DynamoDBRemindersTableTests.cs:289

• This new DynamoDB integration test doesn't skip when the DynamoDB simulator is unavailable. Add the same availability guard used by the other DynamoDB tests in this class to avoid hard failures when DynamoDB isn't configured.

        public async Task DynamoDBReminderTable_PointRead_IsImmediatelyConsistent_AfterWriteAndDelete()
        {
            const string ServiceId = "phase-2-consistency-service";
            const string ReminderName = "immediate-point-read";
            var tableName = $"OrleansReminders-{Guid.NewGuid():N}";

test/Extensions/Orleans.AWS.Tests/Reminder/DynamoDBRemindersTableTests.cs:330

• This new DynamoDB integration test doesn't skip when the DynamoDB simulator is unavailable, which can cause failures in environments where other DynamoDB tests are skipped. Add an availability check at the start of the test.

        public async Task DynamoDBReminderTable_TwoServicesSharingTable_AreFullyIsolated()
        {
            const string ServiceA = "phase-2-service-a";
            const string ServiceB = "phase-2-service-b";
            const string ReminderName = "shared-reminder";

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (1)

Severity	Finding
	test/​Extensions/​Orleans.AWS.Tests/​Reminder/​DynamoDBRemindersTableTests.cs — These new DynamoDB integration tests don't skip when the DynamoDB simulator is unavailable. Unlike… View resolved comment

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:532

• StopAsync only cancels the compatibility-heartbeat loop. If shutdown occurs while migration lease renewal is active (e.g., during StartAsync/migration), the leaseRenewalTask can continue running until process exit, potentially renewing the lease longer than intended and leaking a background task. Consider stopping lease renewal as part of StopAsync (and avoid the early-return being based solely on heartbeatCancellation).

        public async Task StopAsync(CancellationToken cancellationToken = default)
        {
            if (heartbeatCancellation is null)
            {
                return;

src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:291

• ReadRows(IReadOnlyList<(uint Begin, uint End)> ranges, ...) calls RefreshReadMode() once, but in the legacy/eventual path it then calls ReadRows(range.Begin, range.End) for each range, which calls RefreshReadMode() again. This adds extra strongly-consistent metadata reads (one per subrange) to the V2 state item. Consider factoring out the legacy range-read logic into a helper which assumes the read mode is already refreshed, so the ranges overload can reuse it without repeated state reads.

            var legacyResult = new List<ReminderEntry>();
            foreach (var range in ranges)
            {
                legacyResult.AddRange((await ReadRows(range.Begin, range.End)).Reminders);
            }

src/Orleans.Reminders/SystemTargetInterfaces/IReminderTable.cs:66

• The new ReadRows(ranges, ...) default interface implementation can return null! when an older provider returns null, but the XML docs/return type still imply a non-null ReminderTableData. Since this is a public API surface, it would help to document that null can be returned for compatibility so new callers don’t assume it’s impossible.

        /// <summary>
        /// Reads all rows in the provided ownership ranges using one provider-selected consistency operation.
        /// </summary>
        /// <param name="ranges">The owned ranges, each represented as an exclusive lower and inclusive upper bound.</param>
        /// <param name="requireStrongConsistency">Whether discovery must not depend on an eventually consistent index.</param>
        /// <returns>The reminder entries in the provided ranges.</returns>

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	src/​Orleans.Reminders/​ReminderService/​LocalReminderService.cs — ReadAndReconcileRange accepts IRingRange but immediately casts it to ISingleRange to access…
	src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Migration.cs — CopyLegacyRecord declares etag but never uses it. With TreatWarningsAsErrors, this will fail…

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	src/​AWS/​Orleans.Reminders.DynamoDB/​Reminders/​DynamoDBReminderTable.Migration.cs — CopyLegacyRecord declares etag but never uses it. With TreatWarningsAsErrors, this will fail… View resolved comment
	src/​Orleans.Reminders/​ReminderService/​LocalReminderService.cs — ReadAndReconcileRange accepts IRingRange but immediately casts it to ISingleRange to access… View resolved comment