Claude/worker logs memory redis a7vsv8#131
Merged
Merged
Conversation
Three fixes reported together: - merge_message worker logs were missing from the terminal. b508bd7 demoted the worker's operational lines (obtained lock / drained N / finished task) from INFO to DEBUG to stop them flooding the query response log list; that also silenced them on the console (which runs at INFO). Route these plumbing lines through a dedicated `shepherd.merge_message` console logger that propagates to the console handler but carries no query_log_handler, restoring terminal visibility without re-polluting the stored query logs. Also fix the module-level pool-sizing / poll-loop lines that were logging to the unconfigured root logger. - Monitor memory percentage. The per-replica worker table now shows RSS as a percentage of the container's memory cap (a new cgroup memory-limit read in the heartbeat, mirroring cpu_count; None when uncapped), and the Redis memory row inlines the % of maxmemory alongside used/cap. - Redis eviction no longer spams inaccurate worker-down alerts. Key eviction does not disconnect workers, but under volatile-ttl it can evict their short-TTL heartbeat keys, so a live worker momentarily reads as zero-alive and the monitor fires bogus worker-down/crash alerts. Add a Redis-eviction grace window (mirroring the broker-recovery grace) that suppresses worker-down alerts while eviction is active and briefly after, so genuine losses still fire once it clears. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3Nn3SxzDnNwh2djT3orGs
Root-cause follow-up to the eviction-grace alert suppression: exempt the heartbeat keys from eviction entirely instead of only papering over the false alerts they cause. The broker runs a volatile-ttl maxmemory policy, which evicts the keys nearest to expiry first. Worker heartbeat keys carried a 15s TTL, making them the very first thing shed under memory pressure -- so a live worker would vanish from the keyspace and read as crashed, even though key eviction never actually disconnects a worker. This mirrors the redis.conf design already in place for the task streams, which are exempt precisely because they carry no TTL. Write heartbeats as persistent keys (no TTL) and judge liveness from the payload's last_seen timestamp instead of key existence. A shared is_heartbeat_fresh() is now the single definition every "is this worker alive?" reader uses -- the monitor poller (already last_seen-based), the task reclaimer, and the janitor -- so they can't drift. Because a persistent key can't expire itself when a worker crashes, the poller reaps heartbeats left unrefreshed past HEARTBEAT_REAP_SEC (2 min) so dead consumers don't pile up or linger as phantom stale rows; clean shutdown still deletes the key immediately. Reclaim keeps its ~15s freshness cutoff (not the 2 min reap) and reads last_seen directly, so crash-recovery latency is unchanged and never depends on the monitor being up to reap. Tests: persistent key has no TTL; is_heartbeat_fresh by last_seen; reclaim skips a fresh owner but claims from a stale-but-present one; poller reaps dead keys. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3Nn3SxzDnNwh2djT3orGs
Back out the dedicated console_logger sidecar added for the merge_message worker's operational lines. The plumbing lines return to their prior state (logger.debug on the per-task query logger; module-level logging.info/error), leaving the log-level decisions to be made directly per line instead. The memory-percentage and Redis-eviction changes from the same original commit are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3Nn3SxzDnNwh2djT3orGs
…recovery)
Replace the two independent redis_memory YAML threshold rules with an automatic
engine state machine, mirroring the broker/postgres health alerts. This fixes
three things about the old behavior:
- Tiering without duplicates: usage is classified into one level (ok / warning
>=85% / critical >=95%), and only the current level alerts. Crossing the
critical line no longer also sends the redundant warning alert.
- Backoff: while usage stays elevated it re-notifies at most once per
monitor_redis_memory_cooldown_sec (a full day by default) instead of every
10 minutes.
- Recovery: when usage falls back below the warning threshold it emits a
redis_memory_recovered ("it's better") alert, like broker/postgres recovery.
A grace window debounces level changes so a single-tick spike or dip doesn't
alert or clear. Thresholds, grace, and cooldown are tunable via the new
monitor_redis_memory_* settings. Drops the now-unused redis_memory Rule type /
YAML rules; redis_eviction is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3Nn3SxzDnNwh2djT3orGs
Codecov Report❌ Patch coverage is
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.