Claude/worker diagnosis g2v5cw#129
Merged
Merged
Conversation
A single oversized sort_results_score response was OOM-killed (an uncatchable SIGKILL) mid-load. Because the worker died before acking, the message stayed pending in the Redis stream and was re-delivered to every restart via reclaim, turning one bad task into a ~2-hour CrashLoopBackOff for the whole queue. Two layered fixes: 1. Delivery-count circuit breaker (always on). reclaim_orphaned now surfaces each candidate's Redis Streams times_delivered via an optional out-param, and get_tasks dead-letters (ack + route to finish_query with ERROR) any reclaimed message delivered >= max_task_deliveries times without completing, instead of retrying it forever. Bounds any poison pill -- OOM, segfault, hard crash -- to N attempts. Configurable via max_task_deliveries (default 5; 0 disables). 2. Pre-flight response-size guard (opt-in). enforce_response_size_limit checks a response's stored (compressed) blob size with a cheap Redis STRLEN before get_message, and raises ResponseTooLargeError -- which the existing run_task_lifecycle turns into a clean finish_query ERROR -- before the memory-expanding decompress/parse/sort ever runs. Turns an uncatchable OOM SIGKILL into a catchable failure with no crash. Wired into sort_results_score; other workers can adopt the same one-liner. Configurable via max_response_size (default 0 / disabled). _discard_unprocessable_task is refactored onto a shared _terminate_task helper; behavior is unchanged. Adds unit tests for the delivery-count population, the get_tasks dead-letter path, the size guard, and the worker raising before load. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018oXSgLFH28b4KTU7m6wnno
…essed size Follow-up on the OOM: the 320MB figure is the *uncompressed* response. The worker dies during the load (before "Returning sorted results"), because decode_message decompresses to the full JSON bytes and orjson parses that into a Python object tree ~5-6x larger for TRAPI-shaped data (measured 5.7x); the tree and the decompressed bytes coexist during the parse, so peak is roughly uncompressed x 6-7 plus the process baseline -- ~2.5GB for a 320MB response, over a 2Gi limit. - sort_results_score now sorts in place (list.sort, not sorted()) instead of allocating a second copy of the results list and of every result's analyses list -- wasted peak on exactly the payloads big enough to matter. The results list is rebound at the end so a response that arrived without a results field still gets one (behavior preserved), without a copy. - The size guard now gates on the response's *uncompressed* size (the number the /response endpoint returns), read cheaply from the zstd frame header via a GETRANGE of a few bytes -- no fetch, no decompress -- instead of the compressed STRLEN. This is the size that actually drives peak memory, so the configured limit is intuitive: for a 2Gi pod, cap around 200-250M. Adds get_response_size; keeps get_blob_size for the compressed length. The dominant cost (parse expansion + the decompressed bytes held alongside the tree during orjson.loads) is intrinsic to an all-at-once parse and can't be removed without streaming decode; the in-place sort removes the duplicates that are ours to remove. Tests updated/added for the uncompressed-size read, the in-place sort, and the missing-results-key path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018oXSgLFH28b4KTU7m6wnno
Apply the in-place, no-duplicate approach from sort_results_score to the other workers that transform the large TRAPI response so peak memory tracks one copy of the payload, not two: - filter_results_top_n: del results[n:] in place instead of binding a results[:n] slice. The slice allocated a second list and, worse, kept the dropped results (and everything they reference) alive until the function returned -- i.e. through the save. Deleting the tail frees them before the re-encode. - filter_analyses_top_n: del result["analyses"][n:] in place, same reasoning, per result. - filter_kgraph_orphans (shared): delete orphan nodes/edges/auxiliary_graphs from the existing dicts in place instead of rebuilding each with a comprehension, which held the full original dict and the filtered copy simultaneously for the largest structure in the response. Snapshot the keys to delete first to avoid mutating during iteration; the empty auxiliary_graphs key is still created when absent to preserve behavior. - merge_message: check the original query's presence with a cheap EXISTS (new db.message_exists) instead of loading, decompressing and parsing the whole query blob just to discard it. example_score and filter_kgraph_orphans' worker already mutated in place; aragorn_score/sipr deepcopies are algorithmic and left untouched. Adds tests for message_exists and the filter workers' missing-results path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018oXSgLFH28b4KTU7m6wnno
Codecov Report❌ Patch coverage is
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.