Skip to content

Improve Redis connection resilience for BullMQ job queue - #7775

Open
m-altaifi wants to merge 2 commits into
Chocobozzz:release/v8.3.xfrom
m-altaifi:fix/job-queue-redis-resilience
Open

Improve Redis connection resilience for BullMQ job queue#7775
m-altaifi wants to merge 2 commits into
Chocobozzz:release/v8.3.xfrom
m-altaifi:fix/job-queue-redis-resilience

Conversation

@m-altaifi

Copy link
Copy Markdown

Summary

This PR improves Redis connection resilience for BullMQ workers and queues, addressing issues where idle TCP connections get dropped (causing read ETIMEDOUT) and workers subsequently fail background lock renewal or stop processing jobs (e.g. #7774).

Changes

  1. TCP Keepalive: Configured keepAlive: 30000 in Redis.getRedisClientOptions() to send periodic TCP keep-alive probes across Redis connections.
  2. BullMQ Connection Options: Passed maxRetriesPerRequest: null and enableReadyCheck: false to BullMQ Worker, Queue, QueueEvents, and FlowProducer connections as recommended by BullMQ to allow graceful reconnection and prevent worker command rejection during transient socket blips.

Closes #7774

@Chocobozzz Chocobozzz left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you target release/v8.3.x branch?

Comment thread server/core/lib/job-queue/job-queue.ts Outdated

this.flowProducer = new FlowProducer({
connection: Redis.getRedisClientOptions('FlowProducer'),
connection: Redis.getRedisClientOptions('FlowProducer', { maxRetriesPerRequest: null, enableReadyCheck: false }),

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wrap { maxRetriesPerRequest: null, enableReadyCheck: false } in an object to not repeat it across the file, and add a small comment explaining why we chose these options

@m-altaifi
m-altaifi changed the base branch from develop to release/v8.3.x September 1, 2026 09:09
Address review: hoist { maxRetriesPerRequest, enableReadyCheck } into a
single constant instead of repeating it, and document why these options
and the TCP keepalive are set.
@m-altaifi
m-altaifi force-pushed the fix/job-queue-redis-resilience branch from 329c7a6 to 947dd97 Compare September 1, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No jobs seem to be running after some time running the instance

2 participants