Skip to content

fix(rmw_wait): trigger the per-node graph guard conditions on graph change (rosbag2 late joiners) - #43

Merged
benaliabderrahmane merged 6 commits into
develfrom
fix/graph-guard-condition-events
Aug 3, 2026
Merged

fix(rmw_wait): trigger the per-node graph guard conditions on graph change (rosbag2 late joiners)#43
benaliabderrahmane merged 6 commits into
develfrom
fix/graph-guard-condition-events

Conversation

@benaliabderrahmane

@benaliabderrahmane benaliabderrahmane commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Fixes #40.

Stacked on #42 (base is fix/registry-doorbell-wakeup, not devel). Review #42 first; this PR's own diff is its last three commits.

Problem

rmw_node_get_graph_guard_condition() returns the per-node guard condition created in rmw_create_node, and that is the object rclcpp's GraphListener actually waits on. The only trigger in rmw_wait fired ctx->graph_guard_condition instead — a context-level field that is declared and never assigned anywhere, so the branch was dead code and the object rcl waits on was never triggered. The loop just above it had an empty body and a comment claiming the node triggers its own guard condition, which nothing does.

So Node::wait_for_graph_change() never woke, for anyone. Everything built on graph events was blind to endpoints that appeared after it started:

  • rosbag2 records nothing when the recorder starts before the publishers. Its topic discovery re-reads the topic list only when a graph event fires (wait_for_graph_changecheck_and_clear). With no event, the recorder never subscribes — ros2 bag info shows 0 messages. Start the publishers first and the same command records fine.
  • wait_for_service() could hang even though the service was up.
  • wait_for_publisher-style helpers and on_graph_change callbacks never fired.

Plain pub/sub was unaffected, because a subscription declares its interest through the registry up front and never needs a graph event. That asymmetry is why this went unnoticed for so long: everything works except the watch-the-graph machinery.

Fix

The context keeps a mutex-guarded list of every node's graph guard condition:

  • rmw_create_node appends as its last step, so no earlier failure path has to undo it.
  • rmw_destroy_node removes the entry under the same mutex before destroying the guard condition, so the trigger loop can never fire a freed object.
  • When rmw_wait observes a registry generation change, it triggers every guard condition in the list.
  • The now-orphaned UdsContext::graph_guard_condition field is removed.

A wait set holding only guard conditions — GraphListener's exact shape — is woken by the registry doorbell from #42, re-runs the generation check, triggers its own guard condition and reports it ready. So graph events work with no data traffic on the domain at all.

End-to-end: ros2 bag record started before the publisher

Real rosbag2, recorder up 4 s before ros2 topic pub appears, ROS 2 Jazzy:

build result
devel recorder aborts after 209 ms — rcl_wait unexpectedly timed out in the GraphListener thread
#42 alone survives, but never subscribes → 0 messages recorded
#42 + this PR subscribes → 91 messages recorded over 9 s

Tests

NodeGraphGuardConditionTriggersOnGraphChange blocks a thread on the node's graph guard condition alone — rosbag2's discovery shape — then creates a subscription from another thread and asserts the wait wakes with the guard condition ready.

The third commit hardens that test. As first written it passed even with the doorbell wake path deleted: the fixture's own rmw_create_node leaves an unconsumed registry generation edge, so the waiter's first rmw_wait triggered the guard condition and found it already ready at the top-of-wait check, returning before it ever blocked. That proved only that a pending edge triggers the guard condition — not the property rosbag2 depends on. It now drains pending edges until a wait genuinely blocks and times out, then asserts the blocked wait both woke early and had really been blocked across the 200 ms before the subscription appeared.

Verified three ways:

variant result
parent commit (#42's tip) fails — 3 s timeout, guard condition never triggered
this PR with register_fd(doorbell_fd) commented out fails — blocks the full 3 s (proves the test really blocks)
this PR as-is passes — wakes ~200 ms in, i.e. on the graph change

Full suite green on Jazzy — 129 tests. Compiles clean against Kilted headers locally; CI green on Jazzy, Kilted and Rolling.

Note

Because this PR's base is #42's branch rather than the default branch, GitHub will not auto-close #40 on merge. Once #42 lands on devel, retarget this PR's base to devel (which also gives it PR-triggered CI); close #40 when both have reached main.

…e 200 ms poll

Two defects, each with a scenario test that fails before the fix:

- A latched (TRANSIENT_LOCAL) publisher in an idle process never replayed to
  late joiners. The top-of-wait registry check scavenged its context from the
  first subscription/service/client in the wait set, so a wait set holding only
  guard conditions -- the shape a publish-only node's executor produces -- got a
  null context and skipped the check entirely. The wait set now stores its
  context at rmw_create_wait_set.

- The 200 ms poll bound that made replay work at all broke the rmw_wait timeout
  contract: every wait returned RMW_RET_TIMEOUT at 200 ms regardless of the
  caller's deadline (a 600 ms wait returned at 200 ms, and an infinite wait,
  which must never time out, returned TIMEOUT), and every process woke 5x/s
  while idle.

The poll is gone. Each context binds a doorbell socket at rmw_init, registered
as ENTRY_DOORBELL before the first generation snapshot so no mutation can fall
into the gap between the snapshot and the wiring. Every registry mutation sends
one octet to every registered doorbell strictly AFTER bumping the generation,
and rmw_wait drains its doorbell strictly BEFORE reading the generation. That
ordering pair makes a lost wakeup impossible: a mutation either lands in the
generation the waiter is about to read, or leaves a datagram queued on a
level-triggered fd. A doorbell-only wake re-checks the registry and re-blocks
for the caller's remaining time, so RMW_RET_TIMEOUT surfaces only at the
caller's own deadline and an infinite wait blocks until a real event. No new
threads, no registry layout change.

Best-effort edge cases, each reproduced before being handled:
- AF_UNIX datagrams stay charged to the sender until the receiver consumes
  them, so one participant that never drains could exhaust the ring socket's
  budget and silently mute wakeups to healthy peers. The ring fd is recreated
  on EAGAIN and the send retried once; on a fresh fd, EAGAIN can only mean the
  destination's own queue is full, i.e. a wakeup is already pending there.
- The slot type is re-validated inside the seqlock window, so a slot recycled
  to a data endpoint mid-scan cannot receive the wake octet.
- The ring socket is thread-local and closed at thread exit, so ringing takes
  no lock and leaks no fd.

Cleanup needs no new machinery: the doorbell is a PID-owned slot holding a
socket path, so graceful shutdown removes it like any endpoint and the existing
stale-PID reaper reclaims it after a crash (slot teardown unlinks the file).

Known limit: a build that predates this change bumps the generation but never
rings, so a fleet running mixed builds can miss wakeups during the upgrade
window -- upgrade together, as with the shm payload flag.

The dead ctx->graph_guard_condition trigger is deliberately left untouched:
wiring the per-node graph guard conditions rcl actually waits on is a separate
defect with its own fix.

Full suite green on Jazzy (128 tests). Replay reaches a late joiner ~20 ms
after it joins; a 600 ms wait now blocks the full 600 ms.
- Wait mechanism summary and wait-sequence steps 2-4: the wait set carries its
  context, the doorbell fd is armed with the entity fds, and step 4 blocks with
  no internal poll interval and honors the caller's deadline.
- New subsection: the doorbell -- why an AF_UNIX datagram socket over the
  alternatives, the ring-after-bump / drain-before-read ordering pair that makes
  lost wakeups impossible, the best-effort edge cases, crash cleanup via the
  existing reaper, and the two accepted limits.
- Registry slot state: ENTRY_DOORBELL, and the generation bump is now followed
  by the ring.
- Operational requirements: /tmp/ros2_uds holds live sockets and must be
  exempted from tmp cleaners.
- Limitations: notification requires a thread inside rmw_wait.
@benaliabderrahmane
benaliabderrahmane marked this pull request as ready for review August 3, 2026 14:19
benaliabderrahmane and others added 4 commits August 3, 2026 16:21
devel is now the integration branch (contributor PRs land there before main),
so it needs the same CI coverage.
…hange

rmw_node_get_graph_guard_condition returns the per-node guard condition created
in rmw_create_node, and that is the object rclcpp's GraphListener waits on. The
only trigger in rmw_wait fired ctx->graph_guard_condition instead -- a
context-level field that is declared and never assigned anywhere, so the branch
was dead code and the object rcl waits on was never triggered. The loop above it
had an empty body and a comment claiming the node triggers its own guard
condition, which nothing does.

Consequence: Node::wait_for_graph_change() never woke, so everything built on
graph events was blind to endpoints appearing after it started -- wait_for_service,
wait_for_publisher-style helpers, and rosbag2's topic discovery, which re-reads
the topic list only when a graph event fires. A recorder started before its
publishers subscribed to nothing and recorded zero messages, while plain
subscribers on the same topics worked, because normal pub/sub declares interest
through the registry up front and never needs a graph event. That asymmetry is
why this went unnoticed: everything works except the watch-the-graph machinery.

Fix: the context keeps a mutex-guarded list of every node's graph guard
condition. rmw_create_node appends as its last step, so no earlier failure path
has to undo it; rmw_destroy_node removes the entry under the same mutex before
destroying the guard condition, so the trigger loop can never fire a freed
object. When rmw_wait observes a registry generation change it triggers every
guard condition in the list. The dead context-level field, now orphaned, is
removed.

A wait set holding only guard conditions -- GraphListener's exact shape -- is
woken by the registry doorbell, re-runs the generation check, triggers its own
guard condition and reports it ready, so this works with no data traffic on the
domain at all.

Fixes #40.

Full suite green on Jazzy (129 tests). The new test blocks on the node's graph
guard condition alone and asserts the wait wakes with it ready when a
subscription appears: it fails against the parent commit (times out at 3 s,
never triggered) and passes here, firing ~1 ms after the graph change.
- Which guard condition the graph change wakes: the old wiring-gap text is
  replaced by the per-context list of per-node guard conditions that is now
  actually triggered, and how a guard-condition-only wait set makes progress.
- The graph guard condition: the per-node object rcl waits on is the object
  rmw_wait triggers; the known-seam paragraph is gone with the seam.
- Wait sequence step 2 triggers every node's graph guard condition.
The test passed even with the doorbell wake path deleted, so it was not
testing what it claimed. The fixture's own rmw_create_node leaves an unconsumed
registry generation edge behind, so the waiter's first rmw_wait ran the
generation check, triggered the graph guard condition, and found it already
ready at the top-of-wait check -- returning before it ever blocked. That proved
only that a pending edge triggers the guard condition, not that a graph change
arriving while a wait is blocked wakes it, which is the property rosbag2's
discovery loop actually depends on.

The test now consumes pending edges until a wait genuinely blocks and times
out, then asserts the blocked wait woke with the guard condition ready well
before its 3 s timeout AND that it really had been blocked across the 200 ms
before the subscription was created.

Verified three ways: fails on the parent commit (nothing triggers the guard
condition), fails with register_fd(doorbell_fd) commented out (the wait now
blocks to its full 3 s timeout instead of returning early), passes here.
@benaliabderrahmane
benaliabderrahmane force-pushed the fix/registry-doorbell-wakeup branch from c4d662b to 2f250ad Compare August 3, 2026 14:24
@benaliabderrahmane
benaliabderrahmane force-pushed the fix/graph-guard-condition-events branch from 5ac0d6a to 81eb719 Compare August 3, 2026 14:24
@benaliabderrahmane
benaliabderrahmane changed the base branch from fix/registry-doorbell-wakeup to devel August 3, 2026 14:44
@benaliabderrahmane
benaliabderrahmane merged commit bccc45b into devel Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant