Skip to content

[Mocha] Cancel message handlers and release interrupted messages on shutdown - #10504

Draft
tobias-tengler wants to merge 2 commits into
mocha/shutdown-minimalfrom
mocha/shutdown-handler-cancellation
Draft

tobias-tengler wants to merge 2 commits into
mocha/shutdown-minimalfrom
mocha/shutdown-handler-cancellation

Conversation

@tobias-tengler

@tobias-tengler tobias-tengler commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

When a receive endpoint stopped, in-flight handlers were never cancelled because ReceiveContext discarded the transport's cancellation token, so they kept running after shutdown began. Stopping also left PostgreSQL messages behind. A handler that finished after the stop began could not delete its message, because the delete used the already cancelled token, and messages that were leased but not finished stayed locked until the 5 minute lease timeout. Both were then redelivered and processed a second time. Finally, a transport or endpoint that failed to stop kept the remaining ones from stopping.

This PR passes the endpoint's cancellation token to handlers, including batch handlers, and keeps shutdown cancellation from triggering retries, fault handling or error logs. RabbitMQ and Azure Service Bus settle messages with CancellationToken.None, so an interrupted message is still nacked or abandoned. Batches are collected and processed per receive endpoint, so stopping one endpoint only cancels its own batches, and MaxConcurrentBatches now applies per endpoint.

The PostgreSQL endpoint deletes or releases a handled message with CancellationToken.None. When polling is cancelled, it releases the messages of the current batch that it still holds, so they are redelivered right away instead of after the lease timeout.

MessagingRuntime.StopAsync and MessagingTransport.StopAsync keep stopping the remaining transports and endpoints when one fails, then rethrow the first failure. A transport that failed to stop stays started, so disposing the runtime tries it again before the consumers are disposed.

@tobias-tengler
tobias-tengler added this pull request to stack #10505 October 7, 2026 20:00
@tobias-tengler
tobias-tengler marked this pull request as draft October 7, 2026 20:00
@tobias-tengler
tobias-tengler force-pushed the mocha/shutdown-handler-cancellation branch 2 times, most recently from 171c463 to 2fa5296 Compare October 7, 2026 20:06
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Patch coverage

88.7% of changed lines covered (197/222)

File Covered Changed Patch %
…/Middlewares/Receive/RabbitMQAcknowledgementMiddleware.cs 1 2 50.0% 🔴
src/Mocha/src/Mocha/Transport/MessagingTransport.cs 16 23 69.6% 🔴
…/src/Mocha.Transport.Postgres/PostgresReceiveEndpoint.cs 32 37 86.5% 🟡
…/Mocha/src/Mocha/Consumers/Implementations/BatchConsumer.cs 82 93 88.2% 🟡
src/Mocha/src/Mocha/Consumers/Batching/BatchCollector.cs 15 16 93.8% 🟡
…/Receive/AzureServiceBusAcknowledgementMiddleware.cs 2 2 100.0% 🟢
…/src/Mocha.Transport.InMemory/InMemoryReceiveEndpoint.cs 3 3 100.0% 🟢
…/Mocha.Transport.Postgres/Connection/PostgresMessageStore.cs 23 23 100.0% 🟢
…/Connection/RabbitMQConsumerManager.cs 1 1 100.0% 🟢
src/Mocha/src/Mocha/Endpoints/ReceiveEndpoint.cs 2 2 100.0% 🟢
…/Mocha/src/Mocha/Middlewares/Consume/Retry/RetryExecutor.cs 2 2 100.0% 🟢
…/src/Mocha/Middlewares/Receive/ReceiveDeadLetterMiddleware.cs 1 1 100.0% 🟢
…/src/Mocha/Middlewares/Receive/ReceiveFaultMiddleware.cs 1 1 100.0% 🟢
…/Receive/Redelivery/ReceiveRedeliveryMiddleware.cs 1 1 100.0% 🟢
src/Mocha/src/Mocha/Middlewares/ReceiveContext.cs 1 1 100.0% 🟢
src/Mocha/src/Mocha/Runtime/MessagingRuntime.cs 13 13 100.0% 🟢
src/Mocha/src/Mocha/ThrowHelper.cs 1 1 100.0% 🟢
Uncovered changed lines (JSON)
{
  "sha": "6b7eef39cad5cfca0d54862dd3eaffea551b0125",
  "files": [
    { "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Middlewares/Receive/RabbitMQAcknowledgementMiddleware.cs", "ranges": [[36, 36]] },
    { "path": "src/Mocha/src/Mocha/Transport/MessagingTransport.cs", "ranges": [[251, 254], [272, 274]] },
    { "path": "src/Mocha/src/Mocha.Transport.Postgres/PostgresReceiveEndpoint.cs", "ranges": [[222, 222], [246, 249]] },
    { "path": "src/Mocha/src/Mocha/Consumers/Implementations/BatchConsumer.cs", "ranges": [[91, 93], [266, 267], [270, 274], [276, 276]] },
    { "path": "src/Mocha/src/Mocha/Consumers/Batching/BatchCollector.cs", "ranges": [[85, 85]] }
  ]
}

Project coverage: 58.6% (314284/536564 lines)

@tobias-tengler
tobias-tengler force-pushed the mocha/shutdown-handler-cancellation branch from 2fa5296 to b84e810 Compare October 8, 2026 07:05
@tobias-tengler
tobias-tengler force-pushed the mocha/shutdown-handler-cancellation branch 3 times, most recently from f616eee to 57f2418 Compare October 8, 2026 08:39
@tobias-tengler tobias-tengler changed the title [Mocha] Cancel message handlers on shutdown [Mocha] Cancel message handlers and release interrupted messages on shutdown Oct 8, 2026
@tobias-tengler
tobias-tengler force-pushed the mocha/shutdown-handler-cancellation branch from f16380f to 7a16145 Compare October 8, 2026 13:29
… failures

Delete or release handled Postgres messages with a token that stop does not cancel, and release the rest of a batch when polling is cancelled, so in-flight messages are not redelivered after the lock timeout.

Keep stopping the remaining transports and endpoints when one fails, then rethrow the first failure.
@tobias-tengler
tobias-tengler force-pushed the mocha/shutdown-handler-cancellation branch from 7a16145 to 85ffdaf Compare October 8, 2026 13:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant