Skip to content

[Bug] single_generate_async has no retry, unlike single_generate, in OpenAILLM/LiteLLMModel/OpenRouterLLM #275

Description

@AmirF194

Describe the Bug

single_generate_async has no retry, unlike its sync sibling single_generate, in all three
concrete LLM clients: OpenAILLM (evoagentx/models/openai_model.py), LiteLLMModel
(evoagentx/models/litellm_model.py), and OpenRouterLLM (evoagentx/models/openrouter_model.py).
Each single_generate carries @retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(5)),
but the matching single_generate_async right below it has no decorator and just re-raises any
exception as a RuntimeError on the first failure. AliyunLLM, SiliconFlowLLM, and NovitaLLM
subclass OpenAILLM without overriding either method, so they inherit the same gap.

This matters most for batch_generate_async (evoagentx/models/base_model.py, asyncio.gather
over per-message tasks), the path concurrent multi-agent workflows go through, since it has zero
resilience against the transient rate-limit or timeout errors the sync path was deliberately
hardened against.

Operating System

Ubuntu 22.04 (also reproduces on any OS, this is a pure logic bug, not platform-specific)

Python Version

3.11.9

Steps to Reproduce

  1. Install the package at current main (fd6b9a63).
  2. Run this against the real, unmodified OpenAILLM class, with the underlying client mocked
    to raise once and then succeed:
from unittest.mock import MagicMock
from evoagentx.models.openai_model import OpenAILLM
from evoagentx.models.model_configs import OpenAILLMConfig

llm = OpenAILLM(config=OpenAILLMConfig(model="gpt-4o-mini", openai_key="sk-test"))

calls = {"n": 0}
def flaky(*a, **kw):
    calls["n"] += 1
    if calls["n"] == 1:
        raise RuntimeError("transient error")
    resp = MagicMock()
    resp.choices = [MagicMock(message=MagicMock(content="ok", tool_calls=None))]
    return resp

fake_client = MagicMock()
fake_client.chat.completions.create = flaky
llm._client = fake_client
llm.ensure_client = lambda: fake_client
print(llm.single_generate(messages=[{"role": "user", "content": "hi"}]))
# succeeds after 2 calls, tenacity retried the transient error
  1. Do the same for single_generate_async (async flaky, llm._async_client /
    llm.ensure_async_client) and await it.

Logs or Screenshots

[sync]  single_generate succeeded after 2 calls -> 'ok'
[async] single_generate_async raised after 1 call(s): Error during single_generate_async of OpenAILLM: transient error

The sync call retries and recovers. The async call gives up on the first transient failure.

Additional Context

Checked git log -G on single_generate_async in openai_model.py: it has never had a retry
decorator since it was introduced, so this is a standing gap rather than a regression. Confirmed
the same shape in litellm_model.py:165 and openrouter_model.py:334.

Happy to send a PR if useful. The cleanest fix looked like extracting a shared
_call_with_retry_async helper (or a tenacity AsyncRetrying wrapper) in base_model.py rather
than duplicating the decorator three times, but that is a call for whoever owns the direction of
that file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions