Fix possible race condition with reentrant callback groups in EventsCBGExecutor scheduler

[skyegalaxy]

Description

Since #3178, although the setting of handle->in_queue = true was protected under ready_callback_groups_mutex, the read is not. This makes the following races possible with more than one worker thread:

if handle->in_queue is loaded stale false while another thread enqueues it and marks handle->in_queue = true, a duplicate handle can be pushed. Removing the callback group or node from a running executor later on can lead to a possible use-after-free if checking for work on the remaining duplicate (dangling) callback handle.

In a more rare case, if the check for handle->in_queue is loaded stale true by the last worker thread to touch it, while another thread sets handle->in_queue = false, a reentrant CBG could have ready work but not be enqueued and stay permanently starved during the lifetime of the executor.

Fixed by ensuring that handle->in_queue is checked under ready_callback_groups_mutex.

Also adds a check in remove_callback_group so that it is only removed from ready_callback_groups if it was found in the queue. previously if it wasn't ready when being removed, std::find could return ready_callback_groups.end() and unconditionally erase that from ready_callback_groups, which is undefined behavior.

Is this user-facing behavior change?

fixes the possibility of duplicate reentrant groups being pushed to the queue, or a push being skipped indefinitely if the last pending worker thread loads a stale in_queue = true

Did you use Generative AI?

This came up as a possible race while running a large benchmark sweep, partially instrumented by Claude Opus 5

Additional Information

[skyegalaxy]

Pulls: #3234
Gist: https://gist.githubusercontent.com/skyegalaxy/62e9d8ec1a15e2aceae600eb16291340/raw/aaf957dcc7b37d287d026ec83e9e27a9d0d31cd5/ros2.repos
BUILD args: --packages-above-and-dependencies rclcpp
TEST args: --packages-above rclcpp
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/20170

• Linux
• Linux-aarch64
• Linux-rhel
• Windows