fix(fault_manager): keep a black box for faults confirmed mid post-roll - #561
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a fault_manager edge case where a fault confirming during another fault’s post-roll window was previously dropped (no rosbag “black box”). It implements “burst” semantics: multiple correlated fault codes can reference the same in-flight recording, with shared-bag lifetime management and corrected storage accounting so quotas don’t evict shared bags incorrectly.
Changes:
- Attach confirmations arriving during an active post-roll to the in-flight recording (cap 32 attached codes), and persist one
rosbag_filesrow per fault code pointing at the samefile_path. - Make rosbag deletion and storage-total calculations aware of shared bags (unlink only when last reference is removed; count shared bag once).
- Add integration/unit tests covering shared-bag attachment, cleanup behavior, and storage-total accounting.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ros2_medkit_fault_manager/src/rosbag_capture.cpp | Implements attach-to-active-recording and writes metadata rows for attached fault codes on bag finalization. |
| src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/rosbag_capture.hpp | Documents burst/shared recording behavior; adds attachment tracking and cap. |
| src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp | Prevents deleting shared bag paths; counts total storage per unique file_path. |
| src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp | Adds helper to detect shared file_path references. |
| src/ros2_medkit_fault_manager/src/fault_storage.cpp | Mirrors shared-bag semantics in in-memory backend (delete safety + deduped totals). |
| src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp | Updates API docs to clarify shared-bag deletion and accounting semantics. |
| src/ros2_medkit_fault_manager/test/test_rosbag_capture.cpp | Adds integration tests for post-roll attachment and shared-bag cleanup. |
| src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp | Adds tests for shared-bag refcounted deletion and deduped storage totals. |
| src/ros2_medkit_fault_manager/CMakeLists.txt | Adds std_msgs dependency for new test publisher helper. |
Comments suppressed due to low confidence (1)
src/ros2_medkit_fault_manager/src/rosbag_capture.cpp:997
post_fault_timer_callback()persists rosbag rows for every code inattached_fault_codes_unconditionally. If an attached fault is cleared (andauto_cleanupruns) before the post-roll timer fires, the earlierdelete_rosbag_file()call is a no-op (no row yet), but the callback will still store a new row for that cleared fault — potentially keeping a shared bag alive indefinitely.
// One recording, one row per fault it covers: the correlated faults that
// confirmed inside this window each need their own lookup key to serve it.
for (const auto & code : attached) {
info.fault_code = code;
storage_->store_rosbag_file(info);
A confirmation arriving while the previous post-roll was running was skipped, so exactly the correlated faults of a burst lost their bag. They now attach to that recording, and storage keeps a shared bag until its last referencing fault is gone.
mfaferek93
force-pushed
the
fix/558-shared-post-roll-bag
branch
from
July 26, 2026 20:41
ec8e1ff to
ccaeff0
Compare
bburda
reviewed
Jul 27, 2026
Collaborator
There was a problem hiding this comment.
Additional findings outside the diff:
- Could you update the docs?
docs/tutorials/snapshots.rstanddocs/config/fault-manager.rststill say rosbag "records one fault at a time", as do the package README andconfig/fault_manager.yaml. Theauto_cleanuprow says bags are deleted when faults are cleared, theentitymode row says only the faulting node's topics are written, and "Bags are namedfault_{code}_{timestamp}/" now uses another fault's code. It would also help to say that a shared bag counts once againstmax_total_storage_mband that eviction removes a whole burst. - Could you change
test_05_multiple_faults_separate_bagsto wait for the first bag instead of sleeping? It asserts the two faults get different bags, and it passes only because its 1.0 s sleep is longer than the 0.5 s post-roll. Under load the second fault attaches and the assert fails. - Would you add a launch test for this? Reporting two faults with no sleep between them in
test_rosbag_integration.test.py, and asserting they return the samefile_path, would cover the fix. That test fails on main. - Could you also put a shared recording id in the
x-medkitblock?GET /{entity}/bulk-data/rosbagsnow lists one descriptor per fault for a shared bag, each with the full size, and a client cannot tell they are one file.
bburda
reviewed
Jul 27, 2026
Drop faults cleared mid post-roll from the recording state so no leftover row pins the shared bag, widen the entity filter with an attached fault's topics, batch a burst's rows into one SQLite transaction, and never throw out of the shutdown finalize path.
Faults of one burst share a recording, and each fault's descriptor reports the full bag size. Expose the bag directory basename in the x-medkit block so clients can group descriptors serving the same bytes.
Update the rosbag docs for the shared post-roll bag: the attach window and its 32-fault cap, last-reference auto-cleanup, entity filter widening for attached faults, and once-per-bag storage accounting.
A rollback resurrects the old rows, so the bags they point at must still exist; collect replaced paths and remove them only once the transaction is durable. Also label the attach log when the primary cleared mid post-roll.
Compare two descriptors sharing one path against a different bag instead of the same pure call twice.
Collaborator
Author
|
The outside-diff items:
|
bburda
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #559.
A fault that confirmed while another fault's post-roll was running was dropped
("Already recording post-fault data, skipping confirmation"), so the correlated
faults - the ones a burst is actually about - ended up with no black box at all.
With the shipped
duration_after_sec: 1.0that is anything confirming within asecond of the first fault.
A burst is one root cause in one window, so it gets one recording with several
lookup keys:
attach_to_active_recording()registers the additional faultagainst the in-flight bag (capped at 32 codes) and one metadata row per code is
written when the bag closes. The
rosbag_filesschema already allowed severalcodes to name the same
file_path.Deferring the flush instead does not work: the ring buffer is not filled during a
post-roll (messages go straight to the writer), so a queued confirmation would
flush an empty buffer.
Shared bags need refcounted cleanup, so
delete_rosbag_file/store_rosbag_fileunlink a bag directory only when no other fault still references the path, and
get_total_rosbag_storage_bytescounts a shared bag once instead of once perfault - otherwise the quota evicts bags that fit. Both storage backends.
The recording state (
current_fault_code_,current_bag_path_,recording_post_fault_) is now published underpost_fault_timer_mutex_, whichalso removes an unlocked read of
current_bag_path_.Tests:
FaultConfirmedDuringPostRollGetsTheSameBag,ClearingOneFaultOfABurstKeepsTheSharedBag,SharedRosbagSurvivesUntilTheLastFaultIsDeleted,SharedRosbagCountsOnceTowardsStorageTotal. Full fault_manager suite green;clang-format clean.
Known edge: with
duration_after_sec: 0there is no post-roll, and a burst therestill loses bags because the first flush drains the ring buffer. Different
mechanism, left alone.