Claude/redis broker memory cleanup pbx3fc#121
Merged
Conversation
The monitor treated a broker (Redis) outage as invisible and its recovery as ~one worker-crash alert per worker type. When the broker restarts (e.g. after an OOM-kill), the persisted monitor:worker_state:* keys reload showing every worker "alive" while no live heartbeats have re-registered, so _resolve_worker_states emits a crashed transition for each -- burying the real event (the broker went down) under a flood of derived symptoms. And while the broker is fully down, collect_snapshot throws on the heartbeat scan and the poll loop just logs and skips, so the outage itself never alerts. Add broker-availability monitoring to the alert engine, mirroring the existing startup-grace pattern: - The poll loop PINGs the broker each tick (poller.probe_broker). A sustained failure (debounced by monitor_broker_down_grace_sec, default 15s) fires one broker_down critical alert; recovery fires one broker_recovered info alert. Firing is gated on in-process state, not a Redis cooldown key, since Redis is unreachable exactly when it matters; Slack/email dispatch needs no broker. - Across the outage and a post-recovery grace window (monitor_broker_recovery_grace_sec, default 30s) worker-down alerts are suppressed. heartbeat_lost rules are skipped without arming their cooldown and with the breach streak preserved, so a worker that is genuinely still down once the window elapses alerts as normal. - When the broker is down the poll loop short-circuits with a minimal broker_up=False snapshot instead of throwing on the keyspace scan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011r7mxToRqeamrfFa5hQp2L
The poll loop already emits a broker_up=false snapshot while the broker is unreachable, but the UI rendered it as empty panels -- every worker card blanking to "No workers reporting yet", which misreads as the whole fleet vanishing rather than the dashboard's data source being down. Add a full-width critical banner (themed with the existing --bad tokens) that shows above all sections when broker_up is false, latching the outage start time client-side to display "since HH:MM:SS (Nm ago)". While down the panels are frozen at their last-known values instead of being wiped, so the banner is the single clear signal and the last good state stays visible for context. The banner clears on the first broker_up snapshot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011r7mxToRqeamrfFa5hQp2L
The broker_down/broker_recovered alerts are already archived to monitor_events (as type=alert with payload.rule), so the history events endpoint returns them -- they just showed as isolated dots. Pair them into intervals and draw a shaded red band spanning each outage behind the event dots, so a glance at the Events timeline shows when the broker was down and for how long (hover for the exact window + duration). computeBrokerOutages handles the two clipped cases: an outage that began before the window (a recovered with no in-window down clamps its start to the window edge) and one still open at the window's end (a down with no recovery runs to `until`, drawn with an open right edge and flagged ongoing). A legend row appears only when the window actually contains an outage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011r7mxToRqeamrfFa5hQp2L
Three new alert families for the monitor, extending the broker-outage work: Postgres availability (engine-level, like the broker). The poller already tracked reachability but only logged it; now a sustained outage fires one postgres_down (critical) alert and recovery fires postgres_recovered (info), debounced by monitor_postgres_down_grace_sec. Simpler than the broker case -- a PG outage doesn't zero out worker heartbeats, so there's no derived flood to suppress, and the alert path (Redis cooldowns, Slack/email) doesn't depend on PG. Driven off the postgres.error the snapshot already carries. This covers an otherwise-silent failure: while PG is down the janitor can't purge or reap abandoned queries. Redis memory + eviction (rule-based) -- the blind spot behind the original OOM-kill incident. _collect_redis_info now surfaces maxmemory, a per-tick evicted_keys delta, policy, and fragmentation. Two new rule kinds: redis_memory fires at a percent-of-cap threshold (85% warn / 95% crit) as a lead indicator, and redis_eviction fires on any eviction delta -- with volatile-ttl that means the cap was reached and blobs are being shed. Both are no-ops on an uncapped broker. Wedged-worker detection (rule-based). New stuck_pending rule kind flags a consumer holding stream messages it hasn't acked past idle_ms, using the per-consumer pending/idle_ms the poller already collects. Catches a worker that still heartbeats (so heartbeat_lost misses it) but stopped processing; scans the whole fleet when no stream is named. UI: the live infra panel shows used/cap, a colorized mem %, and eviction activity; the history Redis-memory chart gets a dashed maxmemory reference line; new redis:maxmemory_bytes / redis:evicted_keys metrics are persisted for trending. 16 alert-engine tests (up from 11) cover the new paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011r7mxToRqeamrfFa5hQp2L
Codecov Report❌ Patch coverage is
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.