Skip to content

Reconnect the AWS IoT WebSocket after a dropped connection - #172

Open
OdynBrouwer wants to merge 1 commit into
cdpuk:mainfrom
OdynBrouwer:fix/aws-iot-ws-reconnect
Open

OdynBrouwer wants to merge 1 commit into
cdpuk:mainfrom
OdynBrouwer:fix/aws-iot-ws-reconnect

Conversation

@OdynBrouwer

Copy link
Copy Markdown

Fixes #154. Likely also fixes #138 ("heartbeat failed, had to reload").

The reconnect scheduler was gated on _running, which is exactly the flag a drop leaves wrong: the listen loop called _schedule_reconnect() while _running was still True, the backoff elapsed, and the attempt re-entered connect() only to be turned away by its own "already running" guard. Nothing reconnected, _notify_disconnected() never fired, and the coordinator stayed on the 5-minute WebSocket polling interval until the entry was reloaded. Gizwits had the mirror image of the same bug: no guard at all in its scheduler, so it also reconnected after an intentional disconnect().

Scheduling is now gated on intent rather than on state. BaseWebSocketClient gains _should_run ("we want a live connection", set by connect(), cleared by disconnect()) alongside _running ("the socket is live"), and owns _handle_disconnect() plus _schedule_reconnect() for both clients - the two had drifted into a right and a wrong copy of the same logic, which is how this survived.

_handle_disconnect() clears _running, notifies, and schedules the retry as its own task: its caller is the listen loop, and the retry path cancels the listen and heartbeat tasks, which a task cannot do to itself. It is a no-op once disconnect() has been called, which is what keeps "stop" meaning stop.

Two more instances of the same mistake fall out of that:

  • A failed initial connect() never retried at all: its failure path called _schedule_reconnect() while _running was still False, so the guard returned immediately and the client stayed offline for good.
  • A failed heartbeat broke out of the heartbeat loop without notifying or reconnecting, leaving the client on a socket that never delivered again - the "heartbeat failed, reload to recover" report in Heartbeat failed for device #138.

_schedule_reconnect() is a loop rather than connect() calling itself, since the chain now actually runs: a recursive version would add a stack frame per attempt, and a device that stays offline overnight would reach Python's frame limit before it comes back.

Tests: the regression test drives a real ConnectionClosed through the listen loop without mocking the scheduler, and asserts the attempt sees _running False - the old test mocked _schedule_reconnect, which is precisely why the bug stayed hidden. Added coverage for the retry after a failed first attempt, for a retry that stops when disconnect() lands during the backoff, and for the disconnect callback firing once per outage rather than once per attempt.

Verified

Beyond the unit tests, I drove this against a real websockets server over loopback - a clean 1000 close and an abrupt transport abort - which is the library behaviour the listen loop's assumptions rest on:

connections seen after the drop
current main 1, then WARNING ... WebSocket already running
this branch 2, in both drop flavours

Full suite (290 tests) and the complete pre-commit gate were run locally on Linux.

No live check was possible on my side: the only device I have access to is a SmartSpa one, and that backend has no WebSocket at all.

@OdynBrouwer
OdynBrouwer marked this pull request as draft September 16, 2026 15:51
@OdynBrouwer
OdynBrouwer marked this pull request as ready for review September 16, 2026 16:21
The reconnect scheduler was gated on `_running`, which is exactly the flag a
drop leaves wrong: the listen loop called `_schedule_reconnect()` while
`_running` was still True, the backoff elapsed, and the attempt re-entered
`connect()` only to be turned away by its own "already running" guard. Nothing
reconnected, `_notify_disconnected()` never fired, and the coordinator stayed on
the 5-minute WebSocket polling interval until the entry was reloaded. Gizwits
had the mirror image of the same bug: no guard at all in its scheduler, so it
also reconnected after an intentional `disconnect()`.

Scheduling is now gated on intent rather than on state. `BaseWebSocketClient`
gains `_should_run` ("we want a live connection", set by `connect()`, cleared by
`disconnect()`) alongside `_running` ("the socket is live"), and owns
`connect()`, `_handle_disconnect()` and `_schedule_reconnect()` for both clients
- the two had drifted into a right and a wrong copy of the same logic, which is
how this survived.

`_handle_disconnect()` clears `_running`, notifies, and schedules the retry as
its own task: its caller is the listen loop, and the retry path cancels the
listen and heartbeat tasks, which a task cannot do to itself. It is a no-op once
`disconnect()` has been called, which is what keeps "stop" meaning stop.

Two more instances of the same mistake fall out of that:

- A failed *initial* connect() never retried at all: its failure path called
  `_schedule_reconnect()` while `_running` was still False, so the guard
  returned immediately and the client stayed offline for good.
- A failed heartbeat broke out of the heartbeat loop without notifying or
  reconnecting, leaving the client on a socket that never delivered again -
  the "heartbeat failed, reload to recover" report in cdpuk#138.

The retry chain is flat, which is the fix for cdpuk#155: `_connect_once()` reports a
failed attempt by leaving `_running` False instead of scheduling its own retry,
and the loop in `_schedule_reconnect()` owns every subsequent attempt, so each
one returns to the same frame. A chain of connect() -> _schedule_reconnect() ->
connect() adds two frames per attempt and - with backoff capped at 60s - reaches
Python's recursion limit after roughly eight hours of sustained downtime, i.e.
exactly when a device is most likely to be offline, and the resulting
RecursionError reads like a crash rather than a symptom.

Tests: the regression test drives a real ConnectionClosed through the listen
loop without mocking the scheduler, and asserts the attempt sees `_running`
False - the old test mocked `_schedule_reconnect`, which is precisely why the
bug stayed hidden. Added coverage for the retry after a failed first attempt,
for a retry that stops when `disconnect()` lands during the backoff, and for the
disconnect callback firing once per outage rather than once per attempt. Each
client also gets a stack-depth test that runs 40 failed attempts and asserts
every retry after the first sits at the same frame depth, a shape a recursive
chain cannot hold.
@OdynBrouwer
OdynBrouwer force-pushed the fix/aws-iot-ws-reconnect branch from 487d188 to ed83f2e Compare September 16, 2026 16:36
@OdynBrouwer

Copy link
Copy Markdown
Author

Also fixes #155 - the retry chain is flat now, so a device that stays offline for hours cannot hit Python's recursion limit.

_connect_once() reports a failed attempt by leaving _running False instead of scheduling its own retry, and connect() hands over to the _schedule_reconnect() loop, which returns to its own frame each time. Previously it was connect() -> _schedule_reconnect() -> connect(), adding two frames per attempt.

Each client now has a stack-depth test that runs 40 failed attempts and asserts every retry after the first sits at the same frame depth - a shape a recursive chain cannot hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS IoT WebSocket never reconnects after a dropped connection Heartbeat failed for device

1 participant