feat: count managed loop iterations, retry failed health discovery#3400
feat: count managed loop iterations, retry failed health discovery#3400chet wants to merge 1 commit into
Conversation
|
@coderabbitai full_review, thanks! |
|
(=^･ω･^=) ✅ Action performedFull review finished. |
Summary by CodeRabbit
WalkthroughChangesManaged loop observability
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ServiceDiscoveryIteration
participant run_discovery_loop
participant record_iteration
participant MetricsRegistry
participant TokioTimer
run_discovery_loop->>ServiceDiscoveryIteration: run_once()
ServiceDiscoveryIteration-->>run_discovery_loop: Discovery result and stats
run_discovery_loop->>record_iteration: Record manager and outcome
record_iteration->>MetricsRegistry: Increment managed-loop counter
run_discovery_loop->>TokioTimer: Sleep interval or backoff delay
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
🌿 Preview your docs: https://nvidia-preview-pull-request-3400.docs.buildwithfern.com/infra-controller |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
crates/utils/src/background_loop.rs (1)
117-189: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider using
check_casesinstead of a hand-rolled table loop.Each row here carries three expected fields (
expect_manager,expect_outcome,expect_warn_error) — exactly the scenario the style guide calls out forcarbide_test_support::check_casesrather than a bespokeCase/forloop, especially sincecarbide-test-supportis already wired in as a dev-dependency for this crate.Based on coding guidelines: "Use
check_cases/check_valuesdirectly when a macro would obscure a table with several inputs or several expected fields per row in table-driven tests."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/utils/src/background_loop.rs` around lines 117 - 189, The test uses a bespoke Case table and loop despite matching the intended check_cases pattern. Refactor record_iteration_counts_every_outcome_and_logs_only_failures to use carbide_test_support::check_cases, preserving each scenario’s manager, result, expected labels, and log assertions within the case checks.Source: Coding guidelines
crates/health/src/discovery/mod.rs (1)
47-70: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider threading a
CancellationTokenthroughrun_discovery_loop.The loop now runs forever with no way to stop gracefully other than aborting the task, unlike the sibling background loops in this same PR (
machine_update_manager,machine_validation,republisher), which all accept acancel_tokenper the established pattern for client-less background jobs. Worth aligningrun_discovery_loop/ServiceDiscoveryIterationwiring inlib.rswith that pattern for consistency and graceful shutdown.As per coding guidelines: "For background tasks that have no clients, but instead run forever at some interval until the container is terminated, ... prefer accepting an explicit cancellation token from the toplevel initialization method."
♻️ Suggested direction
pub(crate) async fn run_discovery_loop( interval: Duration, backoff_config: BackoffConfig, mut iteration: impl DiscoveryIteration, + cancel_token: CancellationToken, ) { let mut backoff = ExponentialBackoff::new(&backoff_config); - loop { + while !cancel_token.is_cancelled() { let result = iteration.run_once().await; background_loop::record_iteration(BackgroundLoopManager::HealthDiscovery, &result); let delay = match result { Ok(()) => { backoff.reset(); interval } Err(_) => backoff.next_delay(), }; - tokio::time::sleep(delay).await; + cancel_token.run_until_cancelled(tokio::time::sleep(delay)).await; } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/discovery/mod.rs` around lines 47 - 70, Thread a CancellationToken through run_discovery_loop and the ServiceDiscoveryIteration wiring in lib.rs, following the established pattern used by machine_update_manager, machine_validation, and republisher. Update the loop to select between cancellation and each iteration/sleep so shutdown exits gracefully while preserving backoff behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/api-core/src/machine_update_manager/mod.rs`:
- Around line 196-201: Update the tracing::warn! call in the
AcquireLockError::WorkAlreadyLocked branch to pass the error as a structured
field (for example, error = %e) rather than interpolating {e} into the message,
while preserving the existing descriptive message.
---
Nitpick comments:
In `@crates/health/src/discovery/mod.rs`:
- Around line 47-70: Thread a CancellationToken through run_discovery_loop and
the ServiceDiscoveryIteration wiring in lib.rs, following the established
pattern used by machine_update_manager, machine_validation, and republisher.
Update the loop to select between cancellation and each iteration/sleep so
shutdown exits gracefully while preserving backoff behavior.
In `@crates/utils/src/background_loop.rs`:
- Around line 117-189: The test uses a bespoke Case table and loop despite
matching the intended check_cases pattern. Refactor
record_iteration_counts_every_outcome_and_logs_only_failures to use
carbide_test_support::check_cases, preserving each scenario’s manager, result,
expected labels, and log assertions within the case checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e9524a02-cee4-4cf4-b1f5-eb63a12a2f59
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/background_loop.rscrates/utils/src/lib.rsdocs/observability/core_metrics.md
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/health/src/discovery/mod.rs (1)
47-70: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
run_discovery_loophas no shutdown mechanism.
run_discovery_loopruns forever with noCancellationToken, andrun_service(crates/health/src/lib.rs) spawns it without one either. Every other client‑less background loop touched in this same PR stack —MachineUpdateManager::start,MachineValidationManager::start, andManagedHostStateRepublisher::start— accepts an explicitcancel_token: CancellationTokenper the documented convention for loops without client handles.Without a token, this loop can only be stopped by aborting the task or killing the process; there is no graceful in‑process shutdown path, and the pattern is now inconsistent with its siblings introduced in this same feature.
As per coding guidelines, "For background tasks that have no clients, but instead run forever at some interval until the container is terminated, prefer accepting an explicit cancellation token from the toplevel
initialize_and_start_controllersmethod, and stop your work when that token is cancelled." As per path instructions, "joined/cancellable background tasks" is an explicit review focus forcrates/**/*.rs.♻️ Sketch of a cancellable loop
pub(crate) async fn run_discovery_loop( interval: Duration, backoff_config: BackoffConfig, mut iteration: impl DiscoveryIteration, + cancel_token: tokio_util::sync::CancellationToken, ) { let mut backoff = ExponentialBackoff::new(&backoff_config); - loop { + while !cancel_token.is_cancelled() { let result = iteration.run_once().await; background_loop::record_iteration(BackgroundLoopManager::HealthDiscovery, &result); let delay = match result { Ok(()) => { backoff.reset(); interval } Err(_) => backoff.next_delay(), }; - tokio::time::sleep(delay).await; + if cancel_token.run_until_cancelled(tokio::time::sleep(delay)).await.is_none() { + break; + } } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/discovery/mod.rs` around lines 47 - 70, Make run_discovery_loop accept a CancellationToken and stop cleanly when it is cancelled, including while waiting between iterations by selecting cancellation against tokio::time::sleep. Update run_service and its caller chain to receive and pass the shared token when spawning the discovery loop, following the patterns used by MachineUpdateManager::start, MachineValidationManager::start, and ManagedHostStateRepublisher::start.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/health/src/discovery/mod.rs`:
- Around line 47-70: Make run_discovery_loop accept a CancellationToken and stop
cleanly when it is cancelled, including while waiting between iterations by
selecting cancellation against tokio::time::sleep. Update run_service and its
caller chain to receive and pass the shared token when spawning the discovery
loop, following the patterns used by MachineUpdateManager::start,
MachineValidationManager::start, and ManagedHostStateRepublisher::start.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 672f0202-f04c-4d6b-9fbf-edae67013a3f
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/background_loop.rscrates/utils/src/lib.rsdocs/observability/core_metrics.md
chet
force-pushed
the
gh-issue-3175-heartbeat
branch
from
6063d36 to
d9e2b8c
Compare
|
@coderabbitai full_review, thanks! |
|
(=^･ω･^=) Action performedFull review triggered. |
🔍 Container Scan Summary
Per-CVE detail lives in the per-service |
|
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
crates/health/src/discovery/mod.rs (1)
52-70: 🩺 Stability & Availability | 🔵 TrivialOperational note: in-process retry supersedes the crash-restart signal.
This loop now survives iteration failures and retries with capped backoff rather than terminating the discovery task. The consequence is that a persistently failing discovery (e.g., Carbide API unreachable) will no longer surface as a pod crash-loop; the sole liveness/failure signal becomes
carbide_background_loop_iterations_total{manager="health_discovery"}— its heartbeat (series stops moving) and itsoutcome="error"rate. Recommend ensuring operator alerting is wired to that series (stale-heartbeat + error-rate) so the loss of the previous restart-based signal does not create a blind spot.The retry/backoff mechanics themselves are sound.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/discovery/mod.rs` around lines 52 - 70, Ensure operational monitoring and alerting covers the retrying health discovery loop in run_discovery_loop: wire alerts to carbide_background_loop_iterations_total for manager="health_discovery", detecting both stale or stopped heartbeats and elevated outcome="error" rates to replace the former crash-loop signal.crates/api-core/src/mqtt_state_change_hook/republisher.rs (1)
173-178: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a targeted test for the lock-acquisition error branch.
crates/api-core/src/mqtt_state_change_hook/republisher.rs:173-178now returns an error for non-WorkAlreadyLockedfailures, but this path still lacks a focused test. A small case that forcestry_acquire_lockto fail with a different error would protect this heartbeat regression path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/api-core/src/mqtt_state_change_hook/republisher.rs` around lines 173 - 178, The non-WorkAlreadyLocked error path in the republish lock acquisition needs coverage. Add a focused test near the existing republisher lock tests that makes try_acquire_lock return a distinct failure, invokes the relevant republish function, and asserts the returned error preserves that failure and includes the managed host state republish lock context.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/api-core/src/machine_update_manager/mod.rs`:
- Around line 196-208: Add a focused test for the non-WorkAlreadyLocked branch
in run_single_iteration(), configuring try_acquire_lock() to return another
AcquireLockError and asserting the method returns
Err(CarbideError::internal(...)) with the expected message rather than Ok(()).
---
Nitpick comments:
In `@crates/api-core/src/mqtt_state_change_hook/republisher.rs`:
- Around line 173-178: The non-WorkAlreadyLocked error path in the republish
lock acquisition needs coverage. Add a focused test near the existing
republisher lock tests that makes try_acquire_lock return a distinct failure,
invokes the relevant republish function, and asserts the returned error
preserves that failure and includes the managed host state republish lock
context.
In `@crates/health/src/discovery/mod.rs`:
- Around line 52-70: Ensure operational monitoring and alerting covers the
retrying health discovery loop in run_discovery_loop: wire alerts to
carbide_background_loop_iterations_total for manager="health_discovery",
detecting both stale or stopped heartbeats and elevated outcome="error" rates to
replace the former crash-loop signal.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9d6dc6c8-a4b1-4b3d-a4b9-45600cd70e79
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/background_loop.rscrates/utils/src/lib.rsdocs/observability/core_metrics.md
chet
force-pushed
the
gh-issue-3175-heartbeat
branch
from
d9e2b8c to
aaf3c24
Compare
chet
changed the title
|
@coderabbitai full_review, thanks! |
|
(=^･ω･^=) ✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/observability/core_metrics.md`:
- Line 24: Update the metrics documentation entry for
carbide_background_loop_iterations_total to match the exported
carbide_managed_loop_iterations_total metric, and update the generated source or
associated test if that row is generated; otherwise remove the stale row. Ensure
the published metrics list reflects only the service’s actual metric name.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ac83d8fe-5372-4af0-8645-1a0ee4f6ddb1
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/api-core/src/tests/machine_update_manager.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/lib.rscrates/utils/src/managed_loop.rsdocs/observability/core_metrics.md
✅ Files skipped from review due to trivial changes (1)
- crates/health/src/main.rs
🚧 Files skipped from review as they are similar to previous changes (6)
- crates/api-core/src/dpa/handler.rs
- crates/api-core/src/machine_update_manager/mod.rs
- crates/utils/Cargo.toml
- crates/api-core/src/mqtt_state_change_hook/republisher.rs
- crates/health/src/discovery/mod.rs
- crates/health/src/lib.rs
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/api-core/src/mqtt_state_change_hook/republisher.rs (1)
171-174: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider a dedicated test for the infrastructure lock-failure path, mirroring
machine_update_manager.This branch now flips the sweep's outcome to
error(viamanaged_loop::record_iteration) instead of silently swallowing it — the same class of change thatmachine_update_manager/mod.rsmade, which was backed bytest_run_single_iteration_propagates_lock_infrastructure_failure(shutting down the work-lock manager to force a non-WorkAlreadyLockederror). No equivalent test is visible here forrun_sweep.As per path instructions for
crates/**/*.rs, findings about behavior and missing tests are preferred over style-only comments.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/api-core/src/mqtt_state_change_hook/republisher.rs` around lines 171 - 174, Add a dedicated run_sweep test covering infrastructure lock acquisition failure, mirroring machine_update_manager’s test_run_single_iteration_propagates_lock_infrastructure_failure. Shut down the work-lock manager to produce a non-WorkAlreadyLocked error, invoke run_sweep, and assert the sweep outcome is recorded as error via managed_loop::record_iteration rather than swallowed.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/api-core/src/mqtt_state_change_hook/republisher.rs`:
- Around line 171-174: Add a dedicated run_sweep test covering infrastructure
lock acquisition failure, mirroring machine_update_manager’s
test_run_single_iteration_propagates_lock_infrastructure_failure. Shut down the
work-lock manager to produce a non-WorkAlreadyLocked error, invoke run_sweep,
and assert the sweep outcome is recorded as error via
managed_loop::record_iteration rather than swallowed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 53ab5511-7551-41cf-8d56-3d03718d0938
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/api-core/src/tests/machine_update_manager.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/lib.rscrates/utils/src/managed_loop.rsdocs/observability/core_metrics.md
Every long-lived manager loop now reports the same heartbeat -- one `carbide_managed_loop_iterations_total{manager, outcome}` count per pass -- so "is this manager alive, and how often does it fail" is one PromQL query across the machine update manager, machine validation, the managed-host-state republisher, and health discovery, instead of four different guesses. And health discovery now survives what used to kill it: a single failed iteration terminated the whole task; it warns, backs off, and retries instead.
- `carbide_managed_loop_iterations_total{manager, outcome}` -- the shared event lives in `carbide-utils` (`managed_loop::record_iteration`), logs nothing on ok (a per-iteration info would be spam) and owns each site's historical failure warn at its historical level. The measured boot metrics collector is deliberately absent: its latency histogram's per-outcome `_count` already is its iteration count, and the metric description points there. Lock contention stays an `ok` pass (another instance is doing the work), but a lock acquisition failing for infrastructure reasons -- database down or overloaded -- now propagates and counts as the `error` it is.
- Health discovery retries with the crate's own `ExponentialBackoff` (1s doubling to a 30s cap, jitter, reset on success) instead of propagating the first error out of the loop. Until now the task died, `run_service` returned, and the pod restart was the de-facto retry -- so expect restart counts to drop and the failure signal to live in this counter and its warn instead. The paused-clock test scripts fail/fail/ok/fail/ok and pins the widening gaps, the reset after success, and the exact counter deltas.
- The health binary's framework-emitted metrics actually export now: `run_service` installs the meter provider and hands the framework registry to the existing `/metrics` merge (`expose_framework_registry` had no caller until now), and registers the log-events metric that `main.rs`'s comment promised. This also lights up the health egress counters (`carbide_health_otlp_export_failures_total`, `carbide_health_report_submissions_total`).
- Drive-by: the DPA MQTT client stats `println!` is now a structured `tracing::debug!`.
Tests added!
This supports NVIDIA#3175
Signed-off-by: Chet Nichols III <chetn@nvidia.com>
chet
force-pushed
the
gh-issue-3175-heartbeat
branch
from
aaf3c24 to
99bace2
Compare
|
@coderabbitai full_review, thanks! |
|
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
crates/utils/src/managed_loop.rs (1)
114-186: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRoute this table through
check_casesinstead of a hand-rolled loop.This test's
Casestruct carries six fields per row — exactly the "several expected fields per row" scenario the style guide calls out forcheck_cases/check_values(as opposed to a manualforloop with ad-hocassert!/assert_eq!calls). Usingcheck_casesdirectly would keep per-case failure reporting consistent with the rest of the codebase's table-driven tests.As per path instructions, "Use
check_cases/check_valuesdirectly when a macro would obscure a table with several inputs or several expected fields per row in table-driven tests."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/utils/src/managed_loop.rs` around lines 114 - 186, Refactor record_iteration_counts_every_outcome_and_logs_only_failures to pass its Case table through the repository’s check_cases helper instead of manually iterating and asserting. Preserve all existing log, metric, and per-scenario expectations while expressing the six expected fields through the established check_cases pattern.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/discovery/mod.rs`:
- Around line 52-70: Update run_discovery_loop to accept a CancellationToken and
use it to stop the loop during shutdown. Make each iteration and its delay
cancellable, including selecting cancellation alongside iteration.run_once() and
tokio::time::sleep(delay), and thread the token from the caller that spawns this
managed background loop.
---
Nitpick comments:
In `@crates/utils/src/managed_loop.rs`:
- Around line 114-186: Refactor
record_iteration_counts_every_outcome_and_logs_only_failures to pass its Case
table through the repository’s check_cases helper instead of manually iterating
and asserting. Preserve all existing log, metric, and per-scenario expectations
while expressing the six expected fields through the established check_cases
pattern.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8dd9bedf-886c-4674-bb00-5ae91e45279a
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
crates/api-core/src/dpa/handler.rscrates/api-core/src/machine_update_manager/mod.rscrates/api-core/src/machine_validation/mod.rscrates/api-core/src/mqtt_state_change_hook/republisher.rscrates/api-core/src/tests/machine_update_manager.rscrates/health/src/discovery/mod.rscrates/health/src/lib.rscrates/health/src/main.rscrates/utils/Cargo.tomlcrates/utils/src/lib.rscrates/utils/src/managed_loop.rsdocs/observability/core_metrics.md
1 participant
Every long-lived manager loop now reports the same heartbeat --
carbide_managed_loop_iterations_total{manager, outcome}-- and health discovery survives failures instead of dying on the first one.ok(another instance holds the work); a lock layer that is down counts as theerrorit is._countalready counts its iterations, and the metric description says where to look.run_serviceinstalls the meter provider and merges the framework registry into/metrics(expose_framework_registryfinally has a caller), which also lights up the health egress counters and log-event counts.println!is now a structuredtracing::debug!.Tests added!
This supports #3175