Probes run sequentially, per datastack and across datastacks. Each runs under a 15s wall-clock timeout implemented by abandoning a daemon thread (_call_with_timeout in src/echo/probes/base.py). Two concerns for a long-running --poll loop:
- Cycle time is unbounded. With about 10 datastacks and low hundreds of probes each, a slow cycle can exceed the probe interval, and one hung endpoint stalls that datastack's whole cycle.
- Threads and connections leak. A timed-out probe is abandoned, not cancelled. The daemon thread stays blocked on a socket read until the OS gives up, so stuck endpoints accumulate threads and connections over days of polling.
Approach (to investigate)
- Set a transport-level connect and read timeout on the caveclient or requests session so the underlying HTTP call actually aborts. The wall-clock timeout then becomes a backstop. Check whether
caveclient exposes a session timeout.
- Bound probe concurrency with a small worker pool to cap cycle time.
- Confirm abandoned work is reclaimed, or cap the number of in-flight abandoned probes.
Acceptance criteria
Probes run sequentially, per datastack and across datastacks. Each runs under a 15s wall-clock timeout implemented by abandoning a daemon thread (
_call_with_timeoutinsrc/echo/probes/base.py). Two concerns for a long-running--pollloop:Approach (to investigate)
caveclientexposes a session timeout.Acceptance criteria