Conversation
Uploads fail now and then with
AssetUpload, POST, http://…/api/assets
Post "http://…/api/assets": EOF
with --on-errors=stop (the default) that aborts the whole run.
Immich's Node.js server closes a keep-alive connection that has been
idle for 5 seconds. The client kept idle connections for 90 seconds, so
after any pause of about 5 seconds between requests (a video being
extracted and hashed, a slow directory) the next request could go out on
a connection the server was closing at that moment. Go's transport
retries such requests only when it can replay the body, which it can't
for a streamed multipart upload, so the error surfaces and the asset is
not uploaded. It typically hits several uploads at once, as the workers
resume together on connections that went idle together.
Set IdleConnTimeout to 2 seconds, well under the server's 5. In a
harness with a server idle timeout of 50 ms and requests paced to hit
the window, 729 of 2000 POSTs failed with the 90 s setting and 0 of
2000 with the client timeout below the server's.
Refs simulot#956
The comment on IdleConnTimeout said the "EOF" failure is not retried for uploads, which suggests other requests are. Go's transport retries a request that hit the server's idle-close only when it is idempotent (GET/HEAD/OPTIONS/TRACE), or carries an Idempotency-Key header and a replayable body; this client sets neither, so none of its POST/PUT requests are retried, uploads being merely the least retryable.
Say that the request is not retried, without the details of Go's retry rules.
It restated the PR's rationale, and sitting between two struct fields it split gofmt's alignment run, dragging an unrelated whitespace change on MaxIdleConns into the diff. The 5-second Node.js keep-alive background lives in the PR description and the commit history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix the occasional upload failure
Post "http://…/api/assets": EOF, which under the default--on-errors=stopaborts the whole run. Refs #956.Cause
Immich's Node.js server drops keep-alive connections after 5 idle seconds; immich-go reused idle connections for up to 90. A request sent on a connection the server is closing at that moment fails with
EOF, and Go's transport does not retry a streamed multipart upload.Fix
IdleConnTimeout90 s → 2 s. In a test harness paced to hit the closing window, that takes the failure rate from about a third of requests to zero. No unit test: it would hinge on timing.Notes
Based on
mainrather thandevelop, because it was tested against Immich v3, whose support is onmainonly.