Skip to content

Crashes if the LLM response does not include an Answer: #6

Description

@ghiculescu

(This might only be an Anthropic issue, I haven't tested with other LLMs.)

class MyAgent < Regent::Agent
  tool(:list_staff, "List all employees")
  tool(:get_user, "Get a user's data")
  tool(:update_user, "Change a user's data")
  def list_staff; []; end
  def get_user(id); {}; end
  def update_user(id, data); {}; end
end
ctx = "You are a helpful payroll and HR reporting assistant for Bob's Burgers. Today is #{Date.today}. Please provide clear, concise responses with relevant data when available."
MyAgent.new(ctx, model: "claude-sonnet-4-20250514").run("What is the capital of Japan?") # "The capital of Japan is Tokyo."
MyAgent.new(ctx, model: "claude-sonnet-4-20250514").run("What can you do") # 💥 

Most of the time, the g'day request will raise with:

undefined method 'include?' for nil (NoMethodError)

        content.include?(SEQUENCES[:answer])
               ^^^^^^^^^

Here: https://github.com/alchaplinsky/regent/blob/d3eb8ae6fa69455c527031759d59d2e57e453c89/lib/regent/engine/react.rb#L37

The first response from Claude looks like this:

{"id":"msg_01PHUU4bqErhtXPzBJ6cs3JH","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[{"type":"text","text":"G'day! Welcome to Bob's Burgers HR and payroll assistance. I'm here to help you with any employee information, payroll questions, or HR reporting needs you might have. \n\nWhat can I help you with today? I can look up staff information, check employee details, or assist with other HR-related queries."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":305,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":75,"service_tier":"standard"}}

And so regent sends a follow up request, which gets a response like this, which crashes:

{"id":"msg_01FXQmGEwXZBBV6HxkYVDiwp","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":377,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":3,"service_tier":"standard"}}

Logging this issue so I don't forget about it. I am going to do some trial and error but I feel like the gem needs to either tighten up the prompt, or handle the case where answer_present? and action_present? are both false.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions