Skip to content

[Bug]: GoalPipeline ignores max_retries for invalid structured outputs #2497

Description

@helloJamest

Prerequisites

  • I have searched the existing issues and discussions, and this is not a duplicate.
  • This is a bug, not a usage question.

Background / Description

GoalPipeline.__init__ documents max_retries as the maximum number of retries for the executor and verifier to generate valid structured outputs, and stores it as self.max_retries. However, neither structured-output retry loop reads that value.

When either agent repeatedly completes without structured_output, GoalPipeline.reply_stream() invokes it indefinitely. The verifier has another unbounded path when its stream ends without a completed final message: if final_msg is None: continue.

This is separate from max_iters: max_iters limits executor-verifier goal attempts after an explicit fail verdict, while malformed or missing structured output is retried inside a single goal attempt.

Expected behavior

max_retries should bound malformed/missing structured-output retries independently for the executor and verifier. Following the existing AgentScope retry convention, max_retries=1 should allow the initial call plus one retry. Once exhausted, the pipeline should raise a clear error instead of silently appearing to finish.

HITL and external-execution events should continue to park the pipeline without consuming the malformed-output retry budget.

Actual behavior

With max_retries=1, both executor and verifier can be called more than two times and continue indefinitely if they never produce structured output.

Error Messages

No exception is raised by GoalPipeline. A guarded reproducer stopped each loop after the fifth call:
executor: guard stopped executor; calls=5; configured_retries=1
verifier: guard stopped verifier; calls=5; configured_retries=1

Steps to Reproduce

The existing StubAgent and _no_output helpers in tests/pipeline_goal_test.py can reproduce the behavior:

executor = StubAgent("executor", [[_no_output("executor")]])
verifier = StubAgent("verifier", [[_verdict("pass")]])
pipe = GoalPipeline(executor, verifier, max_retries=1)

await self._run(pipe, self.query)  # never returns

The same occurs on the verifier path:

executor = StubAgent("executor", [[_report()]])
verifier = StubAgent("verifier", [[_no_output("verifier")]])
pipe = GoalPipeline(executor, verifier, max_retries=1)

await self._run(pipe, self.query)  # never returns

Proposed fix

  • Enforce max_retries + 1 total attempts separately in the executor and verifier structured-output loops.
  • Treat a verifier stream with no completed final message as an invalid attempt too.
  • Preserve existing HITL and max_iters behavior.
  • Add regression tests for executor exhaustion, verifier exhaustion, and success on the last allowed attempt.

I would be happy to prepare a focused PR for this.

Environment

  • AgentScope Version: main at 10eaaac269101e25ad9f808353c45e305c6d7231
  • Python Version: 3.11
  • OS: Windows 11

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions