Self-heal workers wedged off the broker

[maximusunc]

Two changes so a single worker that loses its broker connection recovers without manual intervention, while a genuine fleet-wide broker outage does not turn into a crash-loop storm:

• Set explicit TCP keepalive options (TCP_KEEPIDLE/INTVL/CNT) on both redis connection pools. The pools previously passed socket_keepalive_options={}, so a half-open connection (broker endpoint moved, stale conntrack) was only detected via the 7s per-command socket_timeout and reconnects kept hitting the same dead endpoint. Active probing reaps a dead socket in ~60s on Linux so redis-py rebuilds it against a live endpoint. Guarded with hasattr so it is a no-op on platforms without these options (e.g. macOS dev boxes).

• Add a broker-liveness self-exit. get_task now records success/failure on a BrokerHealth tracker, and the shared get_tasks poll loop exits non-zero once the worker has gone longer than broker_unhealthy_exit_sec (default 300s) without a successful read, so Kubernetes reschedules it with a fresh network setup. The window is deliberately generous and the health clock starts fresh on boot, so a real outage recycles the fleet slowly (one restart per window per pod) rather than crash-looping; set to 0 to disable.

Covered by unit tests for the keepalive options, BrokerHealth accounting, get_task success/failure recording, and the self-exit threshold behavior.

Claude-Session: https://claude.ai/code/session_014xLtVzkJYp4Z2SHLgGbTVP

[codecov]

Codecov Report

❌ Patch coverage is 89.47368% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.40%. Comparing base (9594469) to head (bcd0f81).
⚠️ Report is 38 commits behind head on main.

Files with missing lines	Patch %	Lines
shepherd_utils/broker.py	88.46%	0 Missing and 3 partials ⚠️
shepherd_utils/shared.py	90.90%	1 Missing ⚠️

Files with missing lines	Coverage Δ
shepherd_utils/config.py	89.47% <100.00%> (-10.53%)	⬇️
shepherd_utils/shared.py	73.36% <90.90%> (+0.35%)	⬆️
shepherd_utils/broker.py	78.15% <88.46%> (+5.42%)	⬆️

... and 6 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2de994...bcd0f81. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.