[Bug]: TestAllocateIDByKeyWaiterRetriesAfterRefillFailure can hang before refill mock admission

[gouhongshen]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

c26f32368e71c6a8cfbefbc662e5bf21ddf86e54

Other Environment Information

• Hardware parameters: GitHub-hosted Ubuntu x86 runner
• OS type: Ubuntu
• Others: MatrixOne ALL CI, Matrixone UT Coverage / UT Coverage on Ubuntu/x86, workflow run 33192757034 attempt 1, failed job 98922233855.

Actual Behavior

The all-package coverage run timed out pkg/logservice.TestAllocateIDByKeyWaiterRetriesAfterRefillFailure after 10 minutes. The failure summary contains exactly one failed package/test event:

--- FAIL: TestAllocateIDByKeyWaiterRetriesAfterRefillFailure (600.00s)
panic: test timed out after 10m0s
.../pkg/logservice/hakeeper_client_test.go:2117

Line 2117 waits unconditionally for firstRefillStarted. The test gives the leader only a 20 ms context, starts it in a goroutine, and expects its package-global sendCNAllocateIDFunc mock to close that channel. AllocateIDByKeyWithBatch and allocateID both check ctx.Err() before invoking the mock. If CI scheduling delays the leader until the 20 ms context has expired, the leader returns context.DeadlineExceeded into its buffered leaderDone channel without closing firstRefillStarted; the main test then waits until the package timeout.

This is independent of PR #27813: its diff only changes pkg/lockservice/service_forward_test.go, while the failure is in a separate pkg/logservice test binary during the all-package coverage command.

Expected Behavior

The test must use a bounded and observable admission phase. A delayed leader must produce a short, diagnostic test failure rather than an unbounded wait and a 10-minute package timeout. The intended production behavior from #27119 remains unchanged.

Steps to Reproduce

1. Run MatrixOne ALL CI coverage UT on the commit above, using the same all-package `go test -json -short -v -tags matrixone_test -p 6` coverage invocation.
2. Under scheduling pressure, let the leader goroutine in TestAllocateIDByKeyWaiterRetriesAfterRefillFailure start after its 20 ms context deadline.
3. The early context validation returns before sendCNAllocateIDFunc is called.
4. Observe the test block at hakeeper_client_test.go:2117 waiting for firstRefillStarted until the 10-minute package timeout.

The supplied CI run is the confirmed reproduction. A local isolated run can pass because it does not reproduce the coverage job scheduler pressure.

Additional information

• Failing job: https://github.com/matrixorigin/matrixone/actions/runs/33192757034/job/98922233855
• Workflow run: https://github.com/matrixorigin/matrixone/actions/runs/33192757034
• The run was attempt 1 and reported no build failures; the sole failed package was github.com/matrixorigin/matrixone/pkg/logservice after 601.106 seconds.
• TestAllocateIDByKeyWaiterRetriesAfterRefillFailure was added by merged PR fix(logservice): prevent connection ID refill convoys #27119 for production issue [Bug]: burst connections fail when HAKeeper connection-ID batch refill times out #27114. Those track the connection ID refill convoy, not this test-fixture liveness failure.
• Searches for the exact test name found no existing issue. A broader HAKeeper/refill search found closed [Bug]: burst connections fail when HAKeeper connection-ID batch refill times out #27114, which is not a duplicate because its production fix is already merged and it does not describe this CI timeout.
• A focused repair should preserve the refill-failure takeover assertions while replacing the unbounded firstRefillStarted receive and scheduler-sensitive 20 ms leader-start budget with deterministic phase control.

[gouhongshen]

PR #27819 is trying to fix this issue.

[gouhongshen]

PR #27819 has merged and is awaiting manual validation. The issue remains open.