Describe the Bug
single_generate_async has no retry, unlike its sync sibling single_generate, in all three
concrete LLM clients: OpenAILLM (evoagentx/models/openai_model.py), LiteLLMModel
(evoagentx/models/litellm_model.py), and OpenRouterLLM (evoagentx/models/openrouter_model.py).
Each single_generate carries @retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(5)),
but the matching single_generate_async right below it has no decorator and just re-raises any
exception as a RuntimeError on the first failure. AliyunLLM, SiliconFlowLLM, and NovitaLLM
subclass OpenAILLM without overriding either method, so they inherit the same gap.
This matters most for batch_generate_async (evoagentx/models/base_model.py, asyncio.gather
over per-message tasks), the path concurrent multi-agent workflows go through, since it has zero
resilience against the transient rate-limit or timeout errors the sync path was deliberately
hardened against.
Operating System
Ubuntu 22.04 (also reproduces on any OS, this is a pure logic bug, not platform-specific)
Python Version
3.11.9
Steps to Reproduce
- Install the package at current
main (fd6b9a63).
- Run this against the real, unmodified
OpenAILLM class, with the underlying client mocked
to raise once and then succeed:
from unittest.mock import MagicMock
from evoagentx.models.openai_model import OpenAILLM
from evoagentx.models.model_configs import OpenAILLMConfig
llm = OpenAILLM(config=OpenAILLMConfig(model="gpt-4o-mini", openai_key="sk-test"))
calls = {"n": 0}
def flaky(*a, **kw):
calls["n"] += 1
if calls["n"] == 1:
raise RuntimeError("transient error")
resp = MagicMock()
resp.choices = [MagicMock(message=MagicMock(content="ok", tool_calls=None))]
return resp
fake_client = MagicMock()
fake_client.chat.completions.create = flaky
llm._client = fake_client
llm.ensure_client = lambda: fake_client
print(llm.single_generate(messages=[{"role": "user", "content": "hi"}]))
# succeeds after 2 calls, tenacity retried the transient error
- Do the same for
single_generate_async (async flaky, llm._async_client /
llm.ensure_async_client) and await it.
Logs or Screenshots
[sync] single_generate succeeded after 2 calls -> 'ok'
[async] single_generate_async raised after 1 call(s): Error during single_generate_async of OpenAILLM: transient error
The sync call retries and recovers. The async call gives up on the first transient failure.
Additional Context
Checked git log -G on single_generate_async in openai_model.py: it has never had a retry
decorator since it was introduced, so this is a standing gap rather than a regression. Confirmed
the same shape in litellm_model.py:165 and openrouter_model.py:334.
Happy to send a PR if useful. The cleanest fix looked like extracting a shared
_call_with_retry_async helper (or a tenacity AsyncRetrying wrapper) in base_model.py rather
than duplicating the decorator three times, but that is a call for whoever owns the direction of
that file.
Describe the Bug
single_generate_asynchas no retry, unlike its sync siblingsingle_generate, in all threeconcrete LLM clients:
OpenAILLM(evoagentx/models/openai_model.py),LiteLLMModel(
evoagentx/models/litellm_model.py), andOpenRouterLLM(evoagentx/models/openrouter_model.py).Each
single_generatecarries@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(5)),but the matching
single_generate_asyncright below it has no decorator and just re-raises anyexception as a
RuntimeErroron the first failure.AliyunLLM,SiliconFlowLLM, andNovitaLLMsubclass
OpenAILLMwithout overriding either method, so they inherit the same gap.This matters most for
batch_generate_async(evoagentx/models/base_model.py,asyncio.gatherover per-message tasks), the path concurrent multi-agent workflows go through, since it has zero
resilience against the transient rate-limit or timeout errors the sync path was deliberately
hardened against.
Operating System
Ubuntu 22.04 (also reproduces on any OS, this is a pure logic bug, not platform-specific)
Python Version
3.11.9
Steps to Reproduce
main(fd6b9a63).OpenAILLMclass, with the underlying client mockedto raise once and then succeed:
single_generate_async(asyncflaky,llm._async_client/llm.ensure_async_client) and await it.Logs or Screenshots
The sync call retries and recovers. The async call gives up on the first transient failure.
Additional Context
Checked
git log -Gonsingle_generate_asyncinopenai_model.py: it has never had a retrydecorator since it was introduced, so this is a standing gap rather than a regression. Confirmed
the same shape in
litellm_model.py:165andopenrouter_model.py:334.Happy to send a PR if useful. The cleanest fix looked like extracting a shared
_call_with_retry_asynchelper (or a tenacityAsyncRetryingwrapper) inbase_model.pyratherthan duplicating the decorator three times, but that is a call for whoever owns the direction of
that file.