Description
The Ubuntu/x86 race UT in PR #27758 failed in pkg/common/morpc.TestAutoCreateRetryableFactoryIsBoundedAndReleasesGeneration:
https://github.com/matrixorigin/matrixone/actions/runs/33165201407/job/98844086158
The test expects every concurrent caller to return ErrBackendCreateTimeout (backend connection closed), but one caller can return the independently valid ErrBackendCreateQueueTimeout (morpc backend create queue wait timed out).
expected error in chain: backend connection closed
actual error chain: internal error: morpc backend create queue wait timed out
Root cause
The test starts 16 callers and configures both independent timing budgets to the same short 50 ms value:
- backend-create queue admission timeout: 50 ms
- admitted factory execution timeout: 50 ms
The test intends to prove the factory retry budget and generation release contract. Under -race and scheduler/load variation, however, a caller may exhaust the queue-admission budget before its factory work is admitted. The implementation then correctly returns ErrBackendCreateQueueTimeout, while the test only accepts ErrBackendCreateTimeout.
This is a test isolation problem rather than a #27758 ANALYZE-stats regression. #27758 does not change pkg/common/morpc. The test was added by #27559 and passed there under the same ut_parallel=8, demonstrating that the failure is timing-dependent.
Required fix
- Isolate the factory-budget test from queue-admission timing by making the queue budget non-competing with the tested factory budget.
- Keep the assertion for
ErrBackendCreateTimeout; do not weaken it to accept arbitrary errors.
- Preserve the bounded completion and expired-generation release assertions.
- Preserve independent coverage for queue admission timeout and its exact typed error.
- Do not add sleeps, retries, skipped tests, or scheduler-dependent timing as synchronization.
Acceptance criteria
- Exact focused test passes repeatedly under
-race in one process.
- Existing queue-admission timeout test continues to assert
ErrBackendCreateQueueTimeout.
pkg/common/morpc owning-package race test passes.
- No production behavior or timeout defaults change.
Description
The Ubuntu/x86 race UT in PR #27758 failed in
pkg/common/morpc.TestAutoCreateRetryableFactoryIsBoundedAndReleasesGeneration:https://github.com/matrixorigin/matrixone/actions/runs/33165201407/job/98844086158
The test expects every concurrent caller to return
ErrBackendCreateTimeout(backend connection closed), but one caller can return the independently validErrBackendCreateQueueTimeout(morpc backend create queue wait timed out).Root cause
The test starts 16 callers and configures both independent timing budgets to the same short 50 ms value:
The test intends to prove the factory retry budget and generation release contract. Under
-raceand scheduler/load variation, however, a caller may exhaust the queue-admission budget before its factory work is admitted. The implementation then correctly returnsErrBackendCreateQueueTimeout, while the test only acceptsErrBackendCreateTimeout.This is a test isolation problem rather than a #27758 ANALYZE-stats regression. #27758 does not change
pkg/common/morpc. The test was added by #27559 and passed there under the sameut_parallel=8, demonstrating that the failure is timing-dependent.Required fix
ErrBackendCreateTimeout; do not weaken it to accept arbitrary errors.Acceptance criteria
-racein one process.ErrBackendCreateQueueTimeout.pkg/common/morpcowning-package race test passes.