Orleans.Reminders.TestKit.Tests intermittently fails under host load because its test-only retry budgets can expire between invoking an already-completed operation and checking the remaining stopwatch budget.
Observed failures:
UniformMutationPolicy_DetectsPersistentContentionWithLastException on net10.0: the expected contention exception was replaced by TimeoutException: The retry timeout elapsed while starting the operation.FaultyReminderTableTests.CorrectImplementation_PassesTheSameGeneratedSequences on net8.0: an idealized synchronous upsert timed out on its first attempt with a 75 ms budget and the same TimeoutException.
Both pass when rerun alone (including 20 consecutive isolated runs of the first test), but have failed during full TestKit runs on an otherwise unchanged branch. The second failure reproduced while validating #10896 and was 134/135 for the full net8 suite.
The likely race is in ReminderTableRetryPolicy.ExecuteUntilAsync: after operation() returns, it recomputes remaining; if host scheduling consumed the small test budget, it overwrites the last operation exception/observation with a synthetic startup timeout without first observing an already-completed task.
The test timing should remain fast while deterministically observing synchronously completed/faulted attempts before applying the hard bound to incomplete operations.
Orleans.Reminders.TestKit.Testsintermittently fails under host load because its test-only retry budgets can expire between invoking an already-completed operation and checking the remaining stopwatch budget.Observed failures:
UniformMutationPolicy_DetectsPersistentContentionWithLastExceptiononnet10.0: the expected contention exception was replaced byTimeoutException: The retry timeout elapsed while starting the operation.FaultyReminderTableTests.CorrectImplementation_PassesTheSameGeneratedSequencesonnet8.0: an idealized synchronous upsert timed out on its first attempt with a 75 ms budget and the sameTimeoutException.Both pass when rerun alone (including 20 consecutive isolated runs of the first test), but have failed during full TestKit runs on an otherwise unchanged branch. The second failure reproduced while validating #10896 and was 134/135 for the full net8 suite.
The likely race is in
ReminderTableRetryPolicy.ExecuteUntilAsync: afteroperation()returns, it recomputesremaining; if host scheduling consumed the small test budget, it overwrites the last operation exception/observation with a synthetic startup timeout without first observing an already-completed task.The test timing should remain fast while deterministically observing synchronously completed/faulted attempts before applying the hard bound to incomplete operations.