Repository navigation
RDKCOM-5679: RDKBDEV-3480 Rbus hangs in Timeout Cleanup Path - #457
Open
mohamedakif1 wants to merge 4 commits into
Open
mohamedakif1 wants to merge 4 commits into
mohamedakif1 wants to merge 4 commits into
Conversation
Technical details : It was identified that during the issue scenario, the rbuscli call was entering a hung state and remained blocked in the wait queue. As a result, the parent process was unable to receive a response from its dependent process and remained stuck, leading to the observed service hang.In such scenarios, the DUT may reach an unexpected state, and recovery would require a Factory Reset. To address this: Identified and fixed the gap in the main RBUS code that could result in the rbuscli call hanging indefinitely. Implemented timeout and retry mechanisms wherever rbuscli calls are invoked. Upon timeout, the rbuscli call is retried for the configured number of attempts. If the call continues to fail even after all retry attempts are exhausted, the script exits gracefully instead of remaining in a hung state.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
A critical connection-lifetime race remains, and the promised caller-side timeout/retry handling is not included.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
This PR synchronizes RBUS request timeout and error cleanup to prevent semaphore lifetime races.
Changes:
- Tracks mutex ownership across request cleanup paths.
- Synchronizes pending-request removal with the connection mutex.
- Updates semaphore cleanup handling in
rtConnection.c.
| File | Description |
|---|---|
src/rtmessage/rtConnection.c |
Synchronizes timeout and error cleanup of pending requests and semaphores. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1162
to
+1163
| if (!mutex_held) | ||
| pthread_mutex_lock(&con->mutex); |
Comment on lines
+1158
to
+1163
| /* Timeout/error paths release con->mutex before waiting/reading. Re-acquire | ||
| * before touching pending_requests_list and before destroying the reply | ||
| * semaphore so the reader thread cannot Post a freed semaphore (hang in | ||
| * futex_wait_queue_me). Early send-failure goto still holds the mutex. */ | ||
| if (!mutex_held) | ||
| pthread_mutex_lock(&con->mutex); |
pradeeptakdas
changed the title
Reason for change: Fix pending request lifecycle and connection teardown safety - Move pending requests to heap allocation with centralized cleanup to prevent leaks and use-after-free issues. - Ensure proper reply ownership, semaphore cleanup, and error handling across success, timeout, and connection failure paths. - Replace teardown delay with coordinated request draining and reject new requests during destruction. - Clear recycled list-node data to prevent stale pointers. - Update thread pool task cleanup to remain compatible with list-node changes. Test Procedure: Ensure no memory leak from rtMessage layer Risks: Low Priority: P3 Signed-off-by: Chris Buchter <Chris_Buchter@cable.comcast.com>
This reverts commit eff1a08. This commit did not follow a PR.
|
@mohamedakif1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Technical details :
It was identified that during the issue scenario, the rbuscli call was entering a hung state and remained blocked in the wait queue. As a result, the parent process was unable to receive a response from its dependent process and remained stuck, leading to the observed service hang.In such scenarios, the DUT may reach an unexpected state, and recovery would require a Factory Reset.
To address this:
Identified and fixed the gap in the main RBUS code that could result in the rbuscli call hanging indefinitely. Implemented timeout and retry mechanisms wherever rbuscli calls are invoked. Upon timeout, the rbuscli call is retried for the configured number of attempts. If the call continues to fail even after all retry attempts are exhausted, the script exits gracefully instead of remaining in a hung state.