Skip to content

[Flaky UT]: MORPC factory-budget test can observe queue-admission timeout under race #27811

Description

@XuPeng-SH

Description

The Ubuntu/x86 race UT in PR #27758 failed in pkg/common/morpc.TestAutoCreateRetryableFactoryIsBoundedAndReleasesGeneration:

https://github.com/matrixorigin/matrixone/actions/runs/33165201407/job/98844086158

The test expects every concurrent caller to return ErrBackendCreateTimeout (backend connection closed), but one caller can return the independently valid ErrBackendCreateQueueTimeout (morpc backend create queue wait timed out).

expected error in chain: backend connection closed
actual error chain:       internal error: morpc backend create queue wait timed out

Root cause

The test starts 16 callers and configures both independent timing budgets to the same short 50 ms value:

  • backend-create queue admission timeout: 50 ms
  • admitted factory execution timeout: 50 ms

The test intends to prove the factory retry budget and generation release contract. Under -race and scheduler/load variation, however, a caller may exhaust the queue-admission budget before its factory work is admitted. The implementation then correctly returns ErrBackendCreateQueueTimeout, while the test only accepts ErrBackendCreateTimeout.

This is a test isolation problem rather than a #27758 ANALYZE-stats regression. #27758 does not change pkg/common/morpc. The test was added by #27559 and passed there under the same ut_parallel=8, demonstrating that the failure is timing-dependent.

Required fix

  • Isolate the factory-budget test from queue-admission timing by making the queue budget non-competing with the tested factory budget.
  • Keep the assertion for ErrBackendCreateTimeout; do not weaken it to accept arbitrary errors.
  • Preserve the bounded completion and expired-generation release assertions.
  • Preserve independent coverage for queue admission timeout and its exact typed error.
  • Do not add sleeps, retries, skipped tests, or scheduler-dependent timing as synchronization.

Acceptance criteria

  • Exact focused test passes repeatedly under -race in one process.
  • Existing queue-admission timeout test continues to assert ErrBackendCreateQueueTimeout.
  • pkg/common/morpc owning-package race test passes.
  • No production behavior or timeout defaults change.

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingneeds-triageNeeds evaluation before prioritization. Not yet decided whether to proceed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions