Reconnect the AWS IoT WebSocket after a dropped connection - #172
Open
OdynBrouwer wants to merge 1 commit into
Open
OdynBrouwer wants to merge 1 commit into
OdynBrouwer wants to merge 1 commit into
Conversation
OdynBrouwer
marked this pull request as draft
September 16, 2026 15:51
OdynBrouwer
marked this pull request as ready for review
September 16, 2026 16:21
The reconnect scheduler was gated on `_running`, which is exactly the flag a
drop leaves wrong: the listen loop called `_schedule_reconnect()` while
`_running` was still True, the backoff elapsed, and the attempt re-entered
`connect()` only to be turned away by its own "already running" guard. Nothing
reconnected, `_notify_disconnected()` never fired, and the coordinator stayed on
the 5-minute WebSocket polling interval until the entry was reloaded. Gizwits
had the mirror image of the same bug: no guard at all in its scheduler, so it
also reconnected after an intentional `disconnect()`.
Scheduling is now gated on intent rather than on state. `BaseWebSocketClient`
gains `_should_run` ("we want a live connection", set by `connect()`, cleared by
`disconnect()`) alongside `_running` ("the socket is live"), and owns
`connect()`, `_handle_disconnect()` and `_schedule_reconnect()` for both clients
- the two had drifted into a right and a wrong copy of the same logic, which is
how this survived.
`_handle_disconnect()` clears `_running`, notifies, and schedules the retry as
its own task: its caller is the listen loop, and the retry path cancels the
listen and heartbeat tasks, which a task cannot do to itself. It is a no-op once
`disconnect()` has been called, which is what keeps "stop" meaning stop.
Two more instances of the same mistake fall out of that:
- A failed *initial* connect() never retried at all: its failure path called
`_schedule_reconnect()` while `_running` was still False, so the guard
returned immediately and the client stayed offline for good.
- A failed heartbeat broke out of the heartbeat loop without notifying or
reconnecting, leaving the client on a socket that never delivered again -
the "heartbeat failed, reload to recover" report in cdpuk#138.
The retry chain is flat, which is the fix for cdpuk#155: `_connect_once()` reports a
failed attempt by leaving `_running` False instead of scheduling its own retry,
and the loop in `_schedule_reconnect()` owns every subsequent attempt, so each
one returns to the same frame. A chain of connect() -> _schedule_reconnect() ->
connect() adds two frames per attempt and - with backoff capped at 60s - reaches
Python's recursion limit after roughly eight hours of sustained downtime, i.e.
exactly when a device is most likely to be offline, and the resulting
RecursionError reads like a crash rather than a symptom.
Tests: the regression test drives a real ConnectionClosed through the listen
loop without mocking the scheduler, and asserts the attempt sees `_running`
False - the old test mocked `_schedule_reconnect`, which is precisely why the
bug stayed hidden. Added coverage for the retry after a failed first attempt,
for a retry that stops when `disconnect()` lands during the backoff, and for the
disconnect callback firing once per outage rather than once per attempt. Each
client also gets a stack-depth test that runs 40 failed attempts and asserts
every retry after the first sits at the same frame depth, a shape a recursive
chain cannot hold.
OdynBrouwer
force-pushed
the
fix/aws-iot-ws-reconnect
branch
from
September 16, 2026 16:36
487d188 to
ed83f2e
Compare
Author
|
Also fixes #155 - the retry chain is flat now, so a device that stays offline for hours cannot hit Python's recursion limit.
Each client now has a stack-depth test that runs 40 failed attempts and asserts every retry after the first sits at the same frame depth - a shape a recursive chain cannot hold. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #154. Likely also fixes #138 ("heartbeat failed, had to reload").
The reconnect scheduler was gated on
_running, which is exactly the flag a drop leaves wrong: the listen loop called_schedule_reconnect()while_runningwas still True, the backoff elapsed, and the attempt re-enteredconnect()only to be turned away by its own "already running" guard. Nothing reconnected,_notify_disconnected()never fired, and the coordinator stayed on the 5-minute WebSocket polling interval until the entry was reloaded. Gizwits had the mirror image of the same bug: no guard at all in its scheduler, so it also reconnected after an intentionaldisconnect().Scheduling is now gated on intent rather than on state.
BaseWebSocketClientgains_should_run("we want a live connection", set byconnect(), cleared bydisconnect()) alongside_running("the socket is live"), and owns_handle_disconnect()plus_schedule_reconnect()for both clients - the two had drifted into a right and a wrong copy of the same logic, which is how this survived._handle_disconnect()clears_running, notifies, and schedules the retry as its own task: its caller is the listen loop, and the retry path cancels the listen and heartbeat tasks, which a task cannot do to itself. It is a no-op oncedisconnect()has been called, which is what keeps "stop" meaning stop.Two more instances of the same mistake fall out of that:
_schedule_reconnect()while_runningwas still False, so the guard returned immediately and the client stayed offline for good._schedule_reconnect()is a loop rather thanconnect()calling itself, since the chain now actually runs: a recursive version would add a stack frame per attempt, and a device that stays offline overnight would reach Python's frame limit before it comes back.Tests: the regression test drives a real ConnectionClosed through the listen loop without mocking the scheduler, and asserts the attempt sees
_runningFalse - the old test mocked_schedule_reconnect, which is precisely why the bug stayed hidden. Added coverage for the retry after a failed first attempt, for a retry that stops whendisconnect()lands during the backoff, and for the disconnect callback firing once per outage rather than once per attempt.Verified
Beyond the unit tests, I drove this against a real
websocketsserver over loopback - a clean1000close and an abrupt transport abort - which is the library behaviour the listen loop's assumptions rest on:mainWARNING ... WebSocket already runningFull suite (290 tests) and the complete
pre-commitgate were run locally on Linux.No live check was possible on my side: the only device I have access to is a SmartSpa one, and that backend has no WebSocket at all.