Add pg_notify support to reduce polling latency#121
Draft
stephenbinns wants to merge 1 commit into
Draft
Conversation
Idle workers currently only notice new jobs on their next poll (wake_interval, 5s by default). Add a migration (schema v7) that triggers pg_notify() on insert, and a Listener class that lets WorkerGroup wake idle workers immediately on notification instead of waiting out the full interval. This is a supplement to polling, not a replacement - workers keep polling exactly as before, so a missed notification just costs the next poll's worth of latency rather than losing the job. Opt-in via `listen: true` (WorkerGroup) / `--listen` (bin/que), since it requires running the new migration and reserving one extra long-held connection per worker group for the LISTEN, for pooled adapters. Also fixes WorkerGroup#stop to broadcast the wakeup condvar so a large wake_interval doesn't force a hard Thread#raise kill on graceful shutdown when listen mode is enabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
||
| ### Reducing polling latency with pg_notify | ||
|
|
||
| By default, idle workers only notice new jobs on their next poll (`--wake-interval`/`wake_interval`, 5 seconds by default). Running `Que.migrate!` up to the latest schema version adds a trigger that calls `pg_notify` whenever a job becomes available, and passing `--listen` to `bin/que` (or `listen: true` to `Que::WorkerGroup.start`) makes idle workers wake up on that notification instead of waiting out the full interval. Polling itself is unaffected and still runs as normal - notifications are purely a way to shorten the wait, so a missed one just means the job is picked up on the next poll as before. |
There was a problem hiding this comment.
so a missed one just means the job is picked up on the next poll as before.
What are the instances that miss might happen (and how) ?
|
|
||
| By default, idle workers only notice new jobs on their next poll (`--wake-interval`/`wake_interval`, 5 seconds by default). Running `Que.migrate!` up to the latest schema version adds a trigger that calls `pg_notify` whenever a job becomes available, and passing `--listen` to `bin/que` (or `listen: true` to `Que::WorkerGroup.start`) makes idle workers wake up on that notification instead of waiting out the full interval. Polling itself is unaffected and still runs as normal - notifications are purely a way to shorten the wait, so a missed one just means the job is picked up on the next poll as before. | ||
|
|
||
| This holds one dedicated connection open for the lifetime of each worker group to listen on, in addition to the connections workers use to run queries. If you're using a pooled adapter (ActiveRecord, Sequel, Pond), make sure your pool has capacity for one extra connection per worker group when enabling `--listen`. |
There was a problem hiding this comment.
An extra connection per worker group could be costly (in some instances - large number of groups and limited connections) - Wondering if it'd help having some guidance on which queues this is useful for vs which not?
For eg: if the queue is not heavily busy - then this might be more useful and if the queue is high frequency (busy) - then probably not worth it?
There was a problem hiding this comment.
- need a dedicated, long-lived session connection in both Postgres & YB, combined with connection manager in YB means we hold that connection (it will need to be a sticky connection on the node)
Contributor
Author
There was a problem hiding this comment.
Yeah its not ideal is it, I do wonder if it needs to be session or transaction based. My gut feeling is that because its a long running connection it might not even matter. I wonder if we have any findings from the use of goodjob elsewhere
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Idle workers currently only notice new jobs on their next poll (wake_interval, 5s by default). Add a migration (schema v7) that triggers pg_notify() on insert, and a Listener class that lets WorkerGroup wake idle workers immediately on notification instead of waiting out the full interval. This is a supplement to polling, not a replacement - workers keep polling exactly as before, so a missed notification just costs the next poll's worth of latency rather than losing the job.
Opt-in via
listen: true(WorkerGroup) /--listen(bin/que), since it requires running the new migration and reserving one extra long-held connection per worker group for the LISTEN, for pooled adapters.Also fixes WorkerGroup#stop to broadcast the wakeup condvar so a large wake_interval doesn't force a hard Thread#raise kill on graceful shutdown when listen mode is enabled.