Claude/old unacked tasks monitor m04zo0#123
Merged
Merged
Conversation
A message that fails _build_task_context (malformed/legacy payload, bad otel JSON, missing query_id) was being dropped from the poll loop without being acked. Because get_task reads with the '>' cursor (never re-delivers PEL entries) and reclaim_orphaned skips messages owned by a live/self consumer, such a message stayed pending under the running worker forever -- invisible to reclaim and to the janitor (which only touches dead consumers). This surfaced as the monitor's stuck_pending alert firing on very old unacked tasks attributed to workers that are up and healthy. Make an un-buildable message terminal instead of leaking it: ack + delete it so it leaves the PEL and stream, and route it to finish_query with an ERROR status when the parent query is still identifiable (mirroring handle_task_failure). Applied at both build sites in get_tasks (the reclaim path and the normal '>' delivery path); the latter previously wasn't even wrapped in try/except, so a poison payload also crashed and restarted the generator under the same consumer name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F4MdXm6dPBjPkgEUBGqZr8
reclaim_orphaned and the monitor's janitor/poller invoked XPENDING via
broker_client.execute_command("XPENDING", ...). redis-py registers a response
callback on the XPENDING command name that only parses the *summary* form
(XPENDING key group); on the *extended* form (IDLE/start/end/count) it raises
IndexError. In reclaim_orphaned that exception was swallowed by the surrounding
try/except at DEBUG level, so reclaim silently returned [] for every stream --
orphaned pending messages left by crashed/restarted workers were never
XCLAIMed and accumulated forever under dead consumers, invisible to the live
workers (which only read new '>' messages) but visible to the monitor's XINFO
scan, tripping stuck_pending on healthy workers.
The same mistake affected the summary-form callers, which received the parsed
dict but checked for a list/tuple:
- janitor._trim_stream read pending as 0 and computed the trim MINID from
last-delivered-id, which trims away *unacked* pending messages (their ids are
<= last-delivered-id) -- deleting live work from the stream and leaving
dangling PEL entries. This is very likely how the orphans were created.
- janitor.reclaim_dead_consumers (the admin cleanup) hit the same extended-form
crash and found no messages to drop.
- poller._collect_streams always reported per-stream pending as 0 on the
dashboard.
Switch all sites to redis-py's typed helpers (xpending_range / xpending) and
parse the dict results. Verified against a real Redis: reclaim now claims a
present orphan and clears a dangling PEL entry (message already deleted by the
old trim). Adds tests/unit/test_reclaim.py, which reproduces the swallowed-error
regression via fakeredis (it shares redis-py's XPENDING callback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4MdXm6dPBjPkgEUBGqZr8
Codecov Report❌ Patch coverage is
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.