Skip to content

fix: prevent intermittent connection resets in internal service client (#4971) - #5698

Open
krxshes wants to merge 3 commits into
bentoml:mainfrom
krxshes:fix/4971-aiohttp-keepalive-mismatch
Open

fix: prevent intermittent connection resets in internal service client (#4971) #5698
krxshes wants to merge 3 commits into
bentoml:mainfrom
krxshes:fix/4971-aiohttp-keepalive-mismatch

Conversation

@krxshes

@krxshes krxshes commented Aug 22, 2026

Copy link
Copy Markdown

What does this PR address?

Fixes intermittent httpx.RemoteProtocolError/aiohttp connection resets when calling a dependent service via bentoml.depends() (e.g. ServiceAServiceB), surfaced to callers as unhandled 500s.

Two related issues in SessionManager/_call (_bentoml_impl/client/proxy2.py):

  1. The tcp-scheme branch of _make_client() left connector=None, so aiohttp fell back to its default keepalive_timeout (15s), which exceeds uvicorn's default timeout_keep_alive (5s). A pooled connection could be closed server-side while the client still considered it reusable, causing writes to fail mid-request.
  2. _call() had no retry on connection-level failure. Added a retry-once path that rebuilds the request payload (rather than reusing the original, since it can be a one-shot async iterator/file stream) before retrying.

Fix:

  • Explicitly set keepalive_timeout=4.0 on the TCPConnector used for tcp-scheme sessions, keeping it safely under the server's 5s default.
  • Wrap the outbound request in a targeted retry that catches aiohttp.ClientConnectionError/aiohttp.ServerDisconnectedError and retries once with a freshly rebuilt payload.

Testing:

  • Added a regression test in tests/unit/_internal/client/test_session_manager.py asserting the connector's keepalive_timeout.
  • Verified with a local load-test reproduction (two dependent @bentoml.service instances, bursts of 200 requests timed against the keep-alive boundary): before this fix, this reliably produced a cluster of 500s; after, zero failures across repeated runs.

Fixes #4971

Before submitting:

Two related issues in SessionManager/_call (_bentoml_impl/client/proxy2.py):

1. The tcp-scheme branch of _make_client() left connector=None, so aiohttp fell back to its default keepalive_timeout (15s), which exceeds uvicorn's default timeout_keep_alive (5s). A pooled connection could be closed server-side while the client still considered it reusable, causing writes to fail mid-request.
2. _call() had no retry on connection-level failure. Added a retry-once path that rebuilds the request payload (not reuses the original, since it can be a one-shot async iterator/file stream) before retrying.

Verified with a local load-test reproduction: before this fix, a burst of requests timed at the keep-alive boundary reliably produced a cluster of 500s; after, zero failures across repeated runs.

Fixes bentoml#4971
@krxshes
krxshes requested a review from a team as a code owner August 22, 2026 09:20
@krxshes
krxshes requested review from bojiang and removed request for a team August 22, 2026 09:20
@krxshes

krxshes commented Aug 27, 2026

Copy link
Copy Markdown
Author

Hi @bojiang, just following up to see if you have time for a quick review on this fix. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Intermittent httpx.RemoteProtocolError: Server disconnected without sending a response.

1 participant