v1.1.x (tmodbus) causes "Async TCP connection closed by remote host" on USR-N540 gateways

[HAPV2]

Description: Since upgrading to v1.1.1 (the switch to tmodbus), my USR-N540 gateway is actively dropping the TCP connections after a few minutes of polling. Rolling back to the older pymodbus version immediately restores 100% stability.

Hardware Setup:

3x Sunsynk Inverters connected to a single USR-N540 gateway.
Polling over Modbus TCP.
Troubleshooting Steps Taken: In an attempt to stop the TCP drops, we aggressively throttled the new driver to mimic the old driver's rhythm:

Set READ_MESSAGE_SPACING: 0.25 (and tested 1.0)
Set READ_SENSORS_BATCH_SIZE: 10
Set READ_EVERY: 15 (to ensure the spaced-out loop had plenty of time to finish and wasn't overlapping).
Despite the loop running perfectly within the 15-second schedule without overlapping, the USR-N540 actively kills the TCP connection after ~2.5 minutes:

text
[13:11:11] INFO Async TCP connection closed: 10.10.55.241:34
[13:11:11] INFO Async TCP connection closed by remote host.
[13:11:11] INFO Modbus TCP protocol connection established.
[13:11:11] INFO Async TCP connection established: 10.10.55.241:34
[13:11:14] ERROR ExceptionGroup: poll_need_to_read: Exception: OSError reading 3 registers from 109: timeout reading register 109

Conclusion: It seems the new tmodbus async socket implementation is handling TCP connections, framing, or keep-alives in a way that causes the USR-N540 firmware to panic and send a TCP RST/FIN. The old pymodbus driver handles this perfectly. For now, I have rolled back, but wanted to flag this hardware compatibility issue with the new custom engine!

[kellerza]

I can't see your config, but you can try pymodbus-tcp://... on edge

[kellerza]

Release 1.2.0 is out which also includes tmodbus improvements

[HAPV2]

Hi @kellerza, thank you for the super fast release! I updated to 1.2.0 this morning and tested both options, but unfortunately, my USR-N540 gateway is still completely unusable on v1.2.0.

Here are the results of my testing today:

Test 1: Using the pymodbus-tcp:// fallback I set my ports to pymodbus-tcp://IP_ADDRESS:32. The add-on successfully connects and finishes the discovery phase (publishing discovery info to MQTT). However, the moment discovery finishes, the add-on completely deadlocks. It sits frozen indefinitely and never attempts a single read (I left it running for 9 minutes with absolutely zero logs or outputs after discovery). Note: I increased READ_EVERY to 30 to ensure the asyncio loop wasn't overlapping tasks, but it still froze immediately.

Test 2: Using standard tcp:// (tmodbus 0.6.0) I switched back to standard tcp:// to test the new synchronization logic. It successfully reads data, but almost exactly 2 minutes into polling, it still drops the connection and times out, exactly like it did on v1.1.1:

[08:47:27] ERROR Read register 59 x1: ModbusTimeoutError: read_holding_registers(59, 1): Response timeout after 3 seconds for transaction with ID 0xe6 [attempt 1/3]

My Configuration (Sanitized): For your reference, here is the configuration I am using. For now, I have rolled back to the older version of the add-on which uses the old pymodbus engine and is running 100% stable with this exact config.

INVERTERS:
  - SERIAL_NR: 'REDACTED_SERIAL_1'
    HA_PREFIX: INV1
    MODBUS_ID: 1
    PORT: pymodbus-tcp://IP_ADDRESS:32
  - SERIAL_NR: 'REDACTED_SERIAL_2'
    HA_PREFIX: INV2
    MODBUS_ID: 2
    PORT: pymodbus-tcp://IP_ADDRESS:34
  - SERIAL_NR: 'REDACTED_SERIAL_3'
    HA_PREFIX: INV3
    MODBUS_ID: 3
    PORT: pymodbus-tcp://IP_ADDRESS:29
SENSOR_DEFINITIONS: single-phase
SENSORS:
  - energy_management
  - power_flow_card
  - pv1_power
  - pv1_voltage
  - pv2_power
  - pv2_voltage
  - battery_temperature
  - radiator_temperature
  - fault
SENSORS_FIRST_INVERTER:
  - load_limit
  - settings
  - battery_max_charge_current
  - battery_max_discharge_current
MANUFACTURER: Sunsynk
READ_ALLOW_GAP: 2
READ_SENSORS_BATCH_SIZE: 20
READ_MESSAGE_SPACING: 0.25
READ_ATTEMPTS: 3
TIMEOUT: 3
STALE_INVERTER_AFTER_SECONDS: 120
STALE_INVERTER_SKIP_SECONDS: 600
SCHEDULES:
  - KEY: W
    READ_EVERY: 15
    REPORT_EVERY: 60
    CHANGE_ANY: false
    CHANGE_BY: 80
    CHANGE_PERCENT: 0
NUMBER_ENTITY_MODE: auto
PROG_TIME_INTERVAL: '15'
MQTT_HOST: core-mosquitto
MQTT_PORT: 1883
MQTT_USERNAME: MyMqqt
MQTT_PASSWORD: REDACTED_PASSWORD

[kellerza]

IF you only run a single inverter, do you still get a deadlock with pymodbus-tcp:// ?

[HAPV2]

Hi @kellerza, great question!

I just tested it with only a single inverter configured (INV1 using pymodbus-tcp://10.10.55.241:32 and READ_EVERY: 5).

The result: It still deadlocks. Exactly like before, it successfully connects and finishes the MQTT discovery phase, but immediately after discovery finishes, it freezes silently. About 5 minutes later, the backlog finally times out and throws ModbusTimeoutError.

Here are the logs from the single-inverter test:

[09:50:50] INFO MQTT: Payload >20000: 20848 (MQTTExplorer will truncate the message)
[09:50:51] INFO MQTT: Payload >20000: 20848 (MQTTExplorer will truncate the message)
[09:55:24] ERROR Read register 268 x12: ModbusTimeoutError: read_holding_registers(268, 12): Modbus Error: [Input/Output] No response received after 0 retries, continue with next request [attempt 1/3]
[09:56:33] ERROR Read register 150 x1: ModbusTimeoutError: read_holding_registers(150, 1): Modbus Error: [Input/Output] No response received after 0 retries, continue with next request [attempt 1/3]

[kellerza]

Can you enable debug for this?
And you are sure there are no mqtt updates

[HAPV2]

Hi @kellerza,

I re-ran the tests as requested using the pymodbus-tcp:// prefix with DEBUG: 1. Here are the results:

Test 1 (All 3 Inverters using pymodbus-tcp://): As reported earlier, it connects and publishes the MQTT discovery payloads, but then completely deadlocks. It sits frozen indefinitely and never attempts a single read.

Test 2 (Single Inverter using pymodbus-tcp:// with DEBUG=1): When I strip the config down to just 1 inverter, the deadlock doesn't happen! It successfully survives the discovery phase and immediately begins polling data perfectly. (I have attached the debug logs for this successful single-inverter run).

Conclusion: It appears the new pymodbus-tcp:// wrapper works perfectly for a single inverter. However, when tasked with initializing all 3 inverters at once, the heavy synchronous load during setup/discovery seems to be blocking the event loop, causing it to deadlock before the first read cycle can even begin.

Let me know if you'd like me to test any further tweaks to the wrapper!

sunsynk_debug.txt

[HAPV2]

[08:15:36] ERROR Task exception was never retrieved
future: <Task finished name='Task-206222' coro=<AInverter.publish_sensors() done, defined at /.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py:252> exception=ConnectionError('MQTT: Connection timeout (30s)')>
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py", line 261, in publish_sensors
await state.publish(value)
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_sensor.py", line 92, in publish
await self.entity.send_state(MQTT, val, retain=self.retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/entities.py", line 80, in send_state
await client.publish(self.state_topic, tostr(payload), retain=retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 176, in publish
await self.wait_connected()
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 135, in wait_connected
raise ConnectionError(msg)
ConnectionError: MQTT: Connection timeout (30s)
[08:15:39] INFO MQTT: Home Assistant online. Publish discovery info for ['inv1', 'inv2', 'inv3']
[08:15:39] INFO MQTT: Payload >20000: 20858 (MQTTExplorer will truncate the message)
[08:15:40] ERROR Task exception was never retrieved
future: <Task finished name='Task-206237' coro=<AInverter.publish_sensors() done, defined at /.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py:252> exception=ConnectionError('MQTT: Connection timeout (30s)')>
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py", line 261, in publish_sensors
await state.publish(value)
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_sensor.py", line 92, in publish
await self.entity.send_state(MQTT, val, retain=self.retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/entities.py", line 80, in send_state
await client.publish(self.state_topic, tostr(payload), retain=retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 176, in publish
await self.wait_connected()
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 135, in wait_connected
raise ConnectionError(msg)
ConnectionError: MQTT: Connection timeout (30s)
[08:15:55] ERROR Task exception was never retrieved
future: <Task finished name='Task-206236' coro=<AInverter.publish_sensors() done, defined at /.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py:252> exception=ConnectionError('MQTT: Connection timeout (30s)')>
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_inverter.py", line 261, in publish_sensors
await state.publish(value)
File "/.venv/lib/python3.13/site-packages/ha_addon_sunsynk_multi/a_sensor.py", line 92, in publish
await self.entity.send_state(MQTT, val, retain=self.retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/entities.py", line 80, in send_state
await client.publish(self.state_topic, tostr(payload), retain=retain)
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 176, in publish
await self.wait_connected()
File "/.venv/lib/python3.13/site-packages/mqtt_entity/client.py", line 135, in wait_connected
raise ConnectionError(msg)
ConnectionError: MQTT: Connection timeout (30s)

[kellerza]

Thanks for the extra DEBUG run and the latest stack traces — they helped narrow this down.

What the logs show

The new errors are not Modbus or a frozen event loop. Poll reads were running, but MQTT state publishes were failing:

ConnectionError: MQTT: Connection timeout (30s)
  publish_sensors → MQTT.publish → wait_connected()

Those show up as Task exception was never retrieved because publishes run in a background task and the error is not logged on the main path. So it can look like a silent freeze even though Modbus polling is active.

Root cause (multi-inverter)

During startup we were calling MQTT.connect() once per inverter. Each call disconnects and reconnects the MQTT client (loop_stop → connect_async → loop_start). With 3 inverters that churns the connection while discovery and early poll publishes are in flight — which matches:

• 3× pymodbus-tcp:// → discovery completes, then no reliable MQTT updates
• 1× inverter → only one connect, works fine (your DEBUG=1 test)

This is unrelated to the pymodbus-tcp wrapper itself.

Fix

Edge now connects MQTT once, after all inverters have finished Modbus init and registered their discovery entities. Please retry on the next edge build with your 3× pymodbus-tcp:// config.

Separate issue: USR-N540 + tcp:// (tmodbus)

The ~2 minute disconnect on standard tcp:// is still the original USR-N540 / tmodbus compatibility problem. pymodbus-tcp:// remains the right workaround for that hardware until we have a tmodbus-side fix.

Let us know how the next edge build goes.

[HAPV2]

Hi @kellerza,

I can confirm that the edge build completely fixed the discovery/MQTT deadlock! All 3 inverters successfully connect and publish discovery payloads without freezing. Thank you!

However, my sensors still stop updating entirely exactly at the 1m40s mark. I checked the debug logs, and it seems to be because the 3 pymodbus-tcp threads are perfectly synchronized and firing send packets at the USR-N540 gateway at the exact same millisecond.

Since my 3 inverters are on 3 separate TCP ports on the USR (ports 32, 34, and 29), the add-on opens 3 sockets and polls them in parallel. The poor USR's buffer gets overwhelmed and drops the connections:

[21:01:43] ERROR No response received after 0 retries, continue with next request

send: 0x0 0xf1 0x0 0x0 0x0 0x6 0x1 0x3 0x0 0xd2 0x0 0x2
send: 0x0 0xa1 0x0 0x0 0x0 0x6 0x2 0x3 0x0 0xf3 0x0 0x1
send: 0x0 0xa1 0x0 0x0 0x0 0x6 0x3 0x3 0x0 0xf3 0x0 0x1
recv: 0x0 0xa1 0x0 0x0 0x0 0x5 0x2 0x3 0x2 0x0 0x1 extra data:
recv: 0x0 0xa1 0x0 0x0 0x0 0x5 0x3 0x3 0x2 0x0 0x1 extra data:
[21:01:43] ERROR Read register 210 x2: ModbusTimeoutError: read_holding_registers(210, 2): Modbus Error: [Input/Output] No response received after 0 retries, continue with next request [attempt 1/3]

Notice the three back-to-back send commands for ID 1, 2, and 3 before the USR chokes.

Is there any way to stagger the polling loops across multiple connections so they don't fire at the exact same instant?

Baie dankie
Gert

[kellerza]

Hi Gert, try the latest edge, yesterday's did not include any changes.

I'm not sure it would be possible to spread these out if they are seperate ports.

Can you check if they can be reached on the same port? https://kellerza.github.io/sunsynk/guide/fault-finding#parallel-inverters

[HAPV2]

Hi @kellerza,

Thanks for the guidance! I ran the latest edge (commit 0fdfaed).

To clarify my hardware setup: I am using a USR-N540 (a 4-port serial to ethernet gateway). Currently, I have 3 separate RS485 cables going directly from each inverter into 3 separate physical ports on the USR. They are not daisy-chained on a single bus.

Because they are on 3 separate physical RS485 ports, I have to map them to the 3 separate TCP ports in the config (32, 34, 29). If I try to point the config to just one TCP port, it successfully reads the first inverter but immediately crashes with "No response" for the others, since they aren't physically wired to that port.

So, sticking with the 3 separate TCP ports, the add-on successfully connects and gets past the discovery phase (yay!). However, exactly at the 1m40s mark, the USR drops the connection. Looking at the debug logs, I noticed that all three polling tasks seem to fire their send packets at the exact same millisecond:

[21:01:43] ERROR No response received after 0 retries, continue with next request

send: 0x0 0xf1 0x0 0x0 0x0 0x6 0x1 0x3 0x0 0xd2 0x0 0x2
send: 0x0 0xa1 0x0 0x0 0x0 0x6 0x2 0x3 0x0 0xf3 0x0 0x1
send: 0x0 0xa1 0x0 0x0 0x0 0x6 0x3 0x3 0x0 0xf3 0x0 0x1
recv: 0x0 0xa1 0x0 0x0 0x0 0x5 0x2 0x3 0x2 0x0 0x1 extra data:
recv: 0x0 0xa1 0x0 0x0 0x0 0x5 0x3 0x3 0x2 0x0 0x1 extra data:
[21:01:43] ERROR Read register 210 x2: ModbusTimeoutError: read_holding_registers(210, 2): Modbus Error: [Input/Output] No response received after 0 retries, continue with next request [attempt 1/3]

Could the add-on polling 3 separate TCP sockets in parallel be overwhelming the USR-N540's buffer? I'm not sure if there's any way to manage that from the add-on side when dealing with separate ports. Let me know what you think—if it's not feasible to handle in software, I can look into physically rewiring them into a single daisy-chain!

Baie dankie
Gert