Hi,
I ran into an issue during a full planet import against a remote OpenSearch cluster and wanted to share it.
After about 15-16 hours of import at a steady ~1820 docs/second, the process died on a single ConnectionClosedException: Connection closed by peer. No 429s before it, no other warnings, throughput was stable up to this point.
Just one dropped TCP connection and the whole import exits with the full 101M documents having to be redone from scratch.
Trimmed logs:
2026-08-15 13:48:42,832 [main] INFO ImportThread - Imported 98000000 documents [1807.97/second]
... (steady ~1820 docs/sec for the previous ~15h, no warnings) ...
2026-08-15 14:10:24,011 [main] INFO ImportThread - Imported 101300000 documents [1825.04/second]
2026-08-15 14:10:30,597 [main] ERROR ImportThread - Import error.
java.lang.RuntimeException: Error inserting new documents
at de.komoot.photon.opensearch.Importer.add(Importer.java:99)
...
Caused by: java.lang.RuntimeException: Bulk request failed
at de.komoot.photon.opensearch.BulkRetryHelper.sendWithRetry(BulkRetryHelper.java:61)
at de.komoot.photon.opensearch.Importer.submitLoop(Importer.java:219)
Caused by: org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:358)
...
Looking at the BulkRetryHelper.sendWithRetry#L60 (see here), the rety loop catches IOException but only retreties when it is a ResponseException with status 429.
A ConnectionClosedException is an IOException (see here) but not a ResponseException - the connection was closed before the response arrived - so it is being thrown instead of retried. The same holds true for other transport level errors that I would argue are transient and retryable with backoff. For a batch import that runs many hours it feels a bit harsh to lose everything on a single dropped connection when the retry helper already has the backoff logic in place.
Would you be open to a small PR that also retries on transport-level IOExceptions (not just 429 responses)? Same backoff, same max attempts, no new configuration surface.
Hi,
I ran into an issue during a full planet import against a remote OpenSearch cluster and wanted to share it.
After about 15-16 hours of import at a steady ~1820 docs/second, the process died on a single
ConnectionClosedException: Connection closed by peer. No 429s before it, no other warnings, throughput was stable up to this point.Just one dropped TCP connection and the whole import exits with the full 101M documents having to be redone from scratch.
Trimmed logs:
Looking at the
BulkRetryHelper.sendWithRetry#L60(see here), the rety loop catchesIOExceptionbut only retreties when it is aResponseExceptionwith status 429.A
ConnectionClosedExceptionis anIOException(see here) but not aResponseException- the connection was closed before the response arrived - so it is being thrown instead of retried. The same holds true for other transport level errors that I would argue are transient and retryable with backoff. For a batch import that runs many hours it feels a bit harsh to lose everything on a single dropped connection when the retry helper already has the backoff logic in place.Would you be open to a small PR that also retries on transport-level
IOExceptions(not just 429 responses)? Same backoff, same max attempts, no new configuration surface.