fix(rmw_wait): trigger the per-node graph guard conditions on graph change (rosbag2 late joiners) - #43
Merged
Conversation
This was referenced Jul 30, 2026
…e 200 ms poll Two defects, each with a scenario test that fails before the fix: - A latched (TRANSIENT_LOCAL) publisher in an idle process never replayed to late joiners. The top-of-wait registry check scavenged its context from the first subscription/service/client in the wait set, so a wait set holding only guard conditions -- the shape a publish-only node's executor produces -- got a null context and skipped the check entirely. The wait set now stores its context at rmw_create_wait_set. - The 200 ms poll bound that made replay work at all broke the rmw_wait timeout contract: every wait returned RMW_RET_TIMEOUT at 200 ms regardless of the caller's deadline (a 600 ms wait returned at 200 ms, and an infinite wait, which must never time out, returned TIMEOUT), and every process woke 5x/s while idle. The poll is gone. Each context binds a doorbell socket at rmw_init, registered as ENTRY_DOORBELL before the first generation snapshot so no mutation can fall into the gap between the snapshot and the wiring. Every registry mutation sends one octet to every registered doorbell strictly AFTER bumping the generation, and rmw_wait drains its doorbell strictly BEFORE reading the generation. That ordering pair makes a lost wakeup impossible: a mutation either lands in the generation the waiter is about to read, or leaves a datagram queued on a level-triggered fd. A doorbell-only wake re-checks the registry and re-blocks for the caller's remaining time, so RMW_RET_TIMEOUT surfaces only at the caller's own deadline and an infinite wait blocks until a real event. No new threads, no registry layout change. Best-effort edge cases, each reproduced before being handled: - AF_UNIX datagrams stay charged to the sender until the receiver consumes them, so one participant that never drains could exhaust the ring socket's budget and silently mute wakeups to healthy peers. The ring fd is recreated on EAGAIN and the send retried once; on a fresh fd, EAGAIN can only mean the destination's own queue is full, i.e. a wakeup is already pending there. - The slot type is re-validated inside the seqlock window, so a slot recycled to a data endpoint mid-scan cannot receive the wake octet. - The ring socket is thread-local and closed at thread exit, so ringing takes no lock and leaks no fd. Cleanup needs no new machinery: the doorbell is a PID-owned slot holding a socket path, so graceful shutdown removes it like any endpoint and the existing stale-PID reaper reclaims it after a crash (slot teardown unlinks the file). Known limit: a build that predates this change bumps the generation but never rings, so a fleet running mixed builds can miss wakeups during the upgrade window -- upgrade together, as with the shm payload flag. The dead ctx->graph_guard_condition trigger is deliberately left untouched: wiring the per-node graph guard conditions rcl actually waits on is a separate defect with its own fix. Full suite green on Jazzy (128 tests). Replay reaches a late joiner ~20 ms after it joins; a 600 ms wait now blocks the full 600 ms.
- Wait mechanism summary and wait-sequence steps 2-4: the wait set carries its context, the doorbell fd is armed with the entity fds, and step 4 blocks with no internal poll interval and honors the caller's deadline. - New subsection: the doorbell -- why an AF_UNIX datagram socket over the alternatives, the ring-after-bump / drain-before-read ordering pair that makes lost wakeups impossible, the best-effort edge cases, crash cleanup via the existing reaper, and the two accepted limits. - Registry slot state: ENTRY_DOORBELL, and the generation bump is now followed by the ring. - Operational requirements: /tmp/ros2_uds holds live sockets and must be exempted from tmp cleaners. - Limitations: notification requires a thread inside rmw_wait.
benaliabderrahmane
marked this pull request as ready for review
August 3, 2026 14:19
devel is now the integration branch (contributor PRs land there before main), so it needs the same CI coverage.
…hange rmw_node_get_graph_guard_condition returns the per-node guard condition created in rmw_create_node, and that is the object rclcpp's GraphListener waits on. The only trigger in rmw_wait fired ctx->graph_guard_condition instead -- a context-level field that is declared and never assigned anywhere, so the branch was dead code and the object rcl waits on was never triggered. The loop above it had an empty body and a comment claiming the node triggers its own guard condition, which nothing does. Consequence: Node::wait_for_graph_change() never woke, so everything built on graph events was blind to endpoints appearing after it started -- wait_for_service, wait_for_publisher-style helpers, and rosbag2's topic discovery, which re-reads the topic list only when a graph event fires. A recorder started before its publishers subscribed to nothing and recorded zero messages, while plain subscribers on the same topics worked, because normal pub/sub declares interest through the registry up front and never needs a graph event. That asymmetry is why this went unnoticed: everything works except the watch-the-graph machinery. Fix: the context keeps a mutex-guarded list of every node's graph guard condition. rmw_create_node appends as its last step, so no earlier failure path has to undo it; rmw_destroy_node removes the entry under the same mutex before destroying the guard condition, so the trigger loop can never fire a freed object. When rmw_wait observes a registry generation change it triggers every guard condition in the list. The dead context-level field, now orphaned, is removed. A wait set holding only guard conditions -- GraphListener's exact shape -- is woken by the registry doorbell, re-runs the generation check, triggers its own guard condition and reports it ready, so this works with no data traffic on the domain at all. Fixes #40. Full suite green on Jazzy (129 tests). The new test blocks on the node's graph guard condition alone and asserts the wait wakes with it ready when a subscription appears: it fails against the parent commit (times out at 3 s, never triggered) and passes here, firing ~1 ms after the graph change.
- Which guard condition the graph change wakes: the old wiring-gap text is replaced by the per-context list of per-node guard conditions that is now actually triggered, and how a guard-condition-only wait set makes progress. - The graph guard condition: the per-node object rcl waits on is the object rmw_wait triggers; the known-seam paragraph is gone with the seam. - Wait sequence step 2 triggers every node's graph guard condition.
The test passed even with the doorbell wake path deleted, so it was not testing what it claimed. The fixture's own rmw_create_node leaves an unconsumed registry generation edge behind, so the waiter's first rmw_wait ran the generation check, triggered the graph guard condition, and found it already ready at the top-of-wait check -- returning before it ever blocked. That proved only that a pending edge triggers the guard condition, not that a graph change arriving while a wait is blocked wakes it, which is the property rosbag2's discovery loop actually depends on. The test now consumes pending edges until a wait genuinely blocks and times out, then asserts the blocked wait woke with the guard condition ready well before its 3 s timeout AND that it really had been blocked across the 200 ms before the subscription was created. Verified three ways: fails on the parent commit (nothing triggers the guard condition), fails with register_fd(doorbell_fd) commented out (the wait now blocks to its full 3 s timeout instead of returning early), passes here.
benaliabderrahmane
force-pushed
the
fix/registry-doorbell-wakeup
branch
from
August 3, 2026 14:24
c4d662b to
2f250ad
Compare
benaliabderrahmane
force-pushed
the
fix/graph-guard-condition-events
branch
from
August 3, 2026 14:24
5ac0d6a to
81eb719
Compare
benaliabderrahmane
changed the base branch from
fix/registry-doorbell-wakeup
to
devel
August 3, 2026 14:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40.
Problem
rmw_node_get_graph_guard_condition()returns the per-node guard condition created inrmw_create_node, and that is the object rclcpp'sGraphListeneractually waits on. The only trigger inrmw_waitfiredctx->graph_guard_conditioninstead — a context-level field that is declared and never assigned anywhere, so the branch was dead code and the object rcl waits on was never triggered. The loop just above it had an empty body and a comment claiming the node triggers its own guard condition, which nothing does.So
Node::wait_for_graph_change()never woke, for anyone. Everything built on graph events was blind to endpoints that appeared after it started:wait_for_graph_change→check_and_clear). With no event, the recorder never subscribes —ros2 bag infoshows 0 messages. Start the publishers first and the same command records fine.wait_for_service()could hang even though the service was up.wait_for_publisher-style helpers andon_graph_changecallbacks never fired.Plain pub/sub was unaffected, because a subscription declares its interest through the registry up front and never needs a graph event. That asymmetry is why this went unnoticed for so long: everything works except the watch-the-graph machinery.
Fix
The context keeps a mutex-guarded list of every node's graph guard condition:
rmw_create_nodeappends as its last step, so no earlier failure path has to undo it.rmw_destroy_noderemoves the entry under the same mutex before destroying the guard condition, so the trigger loop can never fire a freed object.rmw_waitobserves a registry generation change, it triggers every guard condition in the list.UdsContext::graph_guard_conditionfield is removed.A wait set holding only guard conditions —
GraphListener's exact shape — is woken by the registry doorbell from #42, re-runs the generation check, triggers its own guard condition and reports it ready. So graph events work with no data traffic on the domain at all.End-to-end:
ros2 bag recordstarted before the publisherReal rosbag2, recorder up 4 s before
ros2 topic pubappears, ROS 2 Jazzy:develrcl_wait unexpectedly timed outin theGraphListenerthreadTests
NodeGraphGuardConditionTriggersOnGraphChangeblocks a thread on the node's graph guard condition alone — rosbag2's discovery shape — then creates a subscription from another thread and asserts the wait wakes with the guard condition ready.The third commit hardens that test. As first written it passed even with the doorbell wake path deleted: the fixture's own
rmw_create_nodeleaves an unconsumed registry generation edge, so the waiter's firstrmw_waittriggered the guard condition and found it already ready at the top-of-wait check, returning before it ever blocked. That proved only that a pending edge triggers the guard condition — not the property rosbag2 depends on. It now drains pending edges until a wait genuinely blocks and times out, then asserts the blocked wait both woke early and had really been blocked across the 200 ms before the subscription appeared.Verified three ways:
register_fd(doorbell_fd)commented outFull suite green on Jazzy — 129 tests. Compiles clean against Kilted headers locally; CI green on Jazzy, Kilted and Rolling.
Note
Because this PR's base is #42's branch rather than the default branch, GitHub will not auto-close #40 on merge. Once #42 lands on
devel, retarget this PR's base todevel(which also gives it PR-triggered CI); close #40 when both have reachedmain.