fix(rmw_wait): event-driven registry wakeup via a doorbell; remove the 200 ms poll

[benaliabderrahmane]

Fixes #41.

Problem

Two defects, one root cause: a registry change in one process cannot wake another process that is asleep in rmw_wait.

1. A latched publisher in an idle process never serves late joiners. The top-of-wait registry check scavenged its UdsContext from the first subscription, service, or client in the wait set. A publish-only node's executor has none of those, so its wait set holds only guard conditions, ctx came out nullptr, and the whole generation check — including TRANSIENT_LOCAL replay — was skipped no matter how long the process waited.

2. The 200 ms poll that made replay work broke the rmw_wait timeout contract. The clamp was applied ahead of the infinite-wait branch, so it rewrote every caller's deadline: a 600 ms wait returned RMW_RET_TIMEOUT after 200 ms (measured), and an infinite wait — which must never time out — returned TIMEOUT too. rclcpp's GraphListener treats that as fatal and aborts the process. Every process also woke 5×/s while idle. This is why the replay fix was reverted from main in 75d75cd.

That abort is not theoretical. On devel, ros2 bag record dies 209 ms after starting:

[ERROR] [rclcpp]: caught std::exception exception in GraphListener thread: rcl_wait unexpectedly timed out
terminate called after throwing an instance of 'std::runtime_error'
  what():  rcl_wait unexpectedly timed out

Fix

The wait set stores its context at rmw_create_wait_set, so the check runs for every wait, and the poll is removed entirely.

Each context binds one doorbell socket at rmw_init, registered as ENTRY_DOORBELL before the first generation snapshot so no mutation can fall into the gap between the snapshot and the wiring. Every registry mutation sends one octet to every registered doorbell, strictly after bumping the generation counter; rmw_wait drains its doorbell strictly before reading the generation.

That ordering pair is the whole correctness argument: a mutation either lands in the generation value the waiter is about to read, or it leaves a datagram queued on a level-triggered fd that makes the next epoll_wait return immediately. The datagram queues whether or not the target is currently blocked, so there is no check-then-block race to lose. A doorbell-only wake re-checks the registry and re-blocks for the caller's remaining time — so RMW_RET_TIMEOUT now surfaces only at the caller's own deadline, and an infinite wait blocks until a real event.

No new threads. No registry layout change. No daemon.

Why an AF_UNIX datagram socket

It reuses the one primitive this transport is already made of. The alternatives were considered and rejected: signals (process-wide hygiene), passing eventfds across processes (needs a rendezvous this design deliberately lacks), inotify (does not cover shm writes), io_uring futex (container seccomp). The doorbell is also a PID-owned registry slot holding a socket path, so cleanup needs no new machinery — graceful shutdown removes it like any endpoint, and the existing stale-PID reaper reclaims it after a crash, since slot teardown already unlinks the socket file.

Best-effort edge cases

Each was reproduced before being handled:

• A slow peer must not mute the others. AF_UNIX datagrams stay charged to the sender's buffer until the receiver consumes them, so one participant that never drains could exhaust the ring socket's budget and silently break wakeups to healthy peers. On EAGAIN the ring fd is closed, recreated, and the send retried once; on a fresh fd, EAGAIN can only mean the destination's own queue is full — a wakeup is already pending there, so the drop is safe.
• A recycled slot must not receive the octet. The slot type is re-validated inside the seqlock window, so a slot that was a doorbell when the scan started but has since been reused by a data endpoint cannot be sent to.
• No fd leak, no lock. The ring socket is thread-local and closed at thread exit.

Cost

ring_doorbells is one sendto per participant process per registry mutation, on a lock-free path (the registry uses CAS plus a seqlock, so nothing is held across the ring). The registry holds 32768 slots, so the one extra slot per process is noise. Steady-state publish and take are untouched: the only per-wait addition is one recv drain on an empty socket.

Note the shape of that cost: the ring is O(processes) per mutation and each ring makes every process re-read the generation, so bringing up a large system is O(entities × processes) wakeups. There is no coalescing or debounce — a deliberate omission, since the alternative is a timer or a batching layer, and the registry API mutates one entity at a time so there is no natural batch point. Worth revisiting if startup cost is ever measured to matter at scale.

Scope: what this PR does not fix

This is deliberately only half the graph story. The dead ctx->graph_guard_condition trigger is left untouched here — wiring the per-node graph guard conditions rcl actually waits on is a separate defect (#40) fixed in the stacked PR #43.

Concretely, with only this PR merged, ros2 bag record started before its publishers no longer crashes, but still records nothing — it never subscribes. That is an improvement over devel (which aborts) and not yet a fix for #40.

Tests

Scenario-level, public rmw API only. Each fails on devel and passes here:

test	on devel	here
TransientLocalLateJoinerWhilePublisherProcessIdle	late joiner never receives the retained message	passes, ~20 ms after joining
WaitBlocksForFullCallerTimeout	returns TIMEOUT at 200 ms for a 600 ms wait	blocks the full 600 ms
LatchedTopicSurvivesAnUnresponsiveParticipant	wedged participant starves a healthy idle publisher	passes

Full suite green on Jazzy — 128 tests. Compiles clean against Kilted headers locally; CI green on Jazzy, Kilted and Rolling.

Notes

• Supersedes Fix rcl_wait timeout for infinite wait #38. With the clamp gone at the root, the wait-set-scoped narrowing is no longer needed.

• The third commit adds devel to the CI triggers — without it a PR based on devel gets no checks at all.

• Mixed old/new builds miss rings during a fleet upgrade, so upgrade together (same as the shm payload flag). A process whose threads never enter rmw_wait still cannot replay; that gap pre-dates this change and is unchanged.

• Known limit — a queue filled by another thread while this thread is blocked. rmw_wait checks entity queues before blocking (step 3), but a doorbell-only wake re-runs only the registry check and re-blocks; it does not re-scan the queues. So if a second thread drains the same subscription's socket into its queue while this thread sits in epoll_wait, the socket is no longer readable and this wait is not woken by it — the message is reported on the next event rather than immediately. It is delayed, never lost. Reaching it needs the same subscription to be in a blocked wait set while another thread takes from it, i.e. a reentrant callback group; rclcpp's default mutually-exclusive groups exclude a busy entity from the next wait set. devel's 200 ms poll masked this by re-checking 5×/s. The minimal fix is to re-run step 3's queue scan on a doorbell-only wake before re-blocking; it is left out here because it is a behavior change that deserves its own test rather than a rider on this PR. Raised by an adversarial review pass of this PR and confirmed against the code by hand.

• Removing the poll also removes an accidental self-healing property: the 200 ms retry used to paper over any missed wakeup. Wakeups are now event-driven and correct by the ordering argument above, but a doorbell socket file destroyed by an outside actor (a /tmp cleaner) is no longer recovered from. DESIGN.md gains an operational note requiring /tmp/ros2_uds to be exempted from tmp sweepers; data sockets have always had this same exposure.