Skip to content

Audit concurrency and threading; fix the probe thread leak #22

Description

@bdpedigo

Probes run sequentially, per datastack and across datastacks. Each runs under a 15s wall-clock timeout implemented by abandoning a daemon thread (_call_with_timeout in src/echo/probes/base.py). Two concerns for a long-running --poll loop:

  1. Cycle time is unbounded. With about 10 datastacks and low hundreds of probes each, a slow cycle can exceed the probe interval, and one hung endpoint stalls that datastack's whole cycle.
  2. Threads and connections leak. A timed-out probe is abandoned, not cancelled. The daemon thread stays blocked on a socket read until the OS gives up, so stuck endpoints accumulate threads and connections over days of polling.

Approach (to investigate)

  • Set a transport-level connect and read timeout on the caveclient or requests session so the underlying HTTP call actually aborts. The wall-clock timeout then becomes a backstop. Check whether caveclient exposes a session timeout.
  • Bound probe concurrency with a small worker pool to cap cycle time.
  • Confirm abandoned work is reclaimed, or cap the number of in-flight abandoned probes.

Acceptance criteria

  • Connect and read timeouts applied to probe HTTP calls.
  • Bounded concurrency caps the worst-case cycle time.
  • No unbounded growth of threads or connections under a sustained poll with a permanently stuck endpoint (validated).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions