Multi-Artifact Chat Responses: Framework-agnostic multiple messages in chat UIs

[ericmjl]

Overview

Enable AgentBot/ToolBot to return multiple separate chat messages across different UI frameworks (Marimo, Panel, Streamlit), allowing for rich responses with plots, dataframes, and text as separate messages.

Problem

Currently, chat responses are limited to a single message that can only contain one type of content. Users want to see:

1. "Thinking..." message
2. "Executing code..." message
3. Plot result message
4. DataFrame result message
5. Final summary message

Architecture Decision

Use framework-agnostic design with streaming/callback pattern where AgentBot yields messages during its ReAct loop instead of returning a single response at the end.

Framework-agnostic design: Start with Marimo as reference implementation, then create adapters for Panel and Streamlit.

Implementation Approach

1. Create Framework-Agnostic Message System

Create llamabot/components/multi_message.py:

• ChatMessage dataclass for consistent message structure
• MessageRenderer abstract base class for framework-specific rendering
• Framework-specific renderers: MarimoRenderer, PanelRenderer, StreamlitRenderer

2. Create Framework-Agnostic Chat Interface

Create llamabot/components/chat_interface.py:

• MultiMessageChat class with unified API across frameworks
• Methods: add_text(), add_plot(), add_dataframe(), add_code()
• Consistent interface regardless of UI framework

3. Update AgentBot with Streaming Support

Modify llamabot/bot/agentbot.py:

• Add on_message_callback parameter to __init__
• During ReAct loop, call callback with intermediate messages:
  • After thought phase: callback with thought text
  • After tool execution: callback with tool results
  • At final response: callback with final answer
• Callback signature: callback(message: ChatMessage) -> None

4. Framework-Specific Implementations

Marimo Implementation

def chat_turn(messages, config):
    """Generator that yields multiple messages."""
    user_message = messages[-1].content
    
    # Create chat interface
    chat = MultiMessageChat(MarimoRenderer(mo))
    
    def message_callback(msg: ChatMessage):
        yield chat.add_message(msg)
    
    # Use AgentBot with callback
    agent = lmb.AgentBot(
        tools=[write_and_execute_code(globals())],
        on_message_callback=message_callback,
        model_name="gpt-4.1",
    )
    
    try:
        final_result = agent(user_message, describe_dataframes_in_globals(globals()))
        yield chat.add_text(final_result.content)
    except Exception as e:
        yield chat.add_text(f"Error: {str(e)}", role="system")

Panel Implementation

import panel as pn

class PanelChatInterface:
    def __init__(self):
        self.chat = MultiMessageChat(PanelRenderer())
        self.chat_area = pn.Column()
        self.input_area = pn.Row()
    
    def add_message(self, message: ChatMessage):
        rendered = self.chat.add_message(message)
        self.chat_area.append(rendered)
        return rendered
    
    def send_message(self, content: str):
        # Add user message
        self.add_message(ChatMessage(role="user", content=content, message_type="text"))
        
        # Process with AgentBot
        agent = lmb.AgentBot(
            tools=[write_and_execute_code(globals())],
            on_message_callback=self.add_message,
        )
        
        try:
            result = agent(content, describe_dataframes_in_globals(globals()))
            self.add_message(ChatMessage(role="assistant", content=result.content, message_type="text"))
        except Exception as e:
            self.add_message(ChatMessage(role="system", content=f"Error: {str(e)}", message_type="error"))

Streamlit Implementation

import streamlit as st

class StreamlitChatInterface:
    def __init__(self):
        self.chat = MultiMessageChat(StreamlitRenderer())
    
    def add_message(self, message: ChatMessage):
        with st.chat_message(message.role):
            if message.message_type == "text":
                st.markdown(message.content)
            elif message.message_type == "plot":
                st.pyplot(message.content)
            elif message.message_type == "dataframe":
                st.dataframe(message.content)
            elif message.message_type == "code":
                st.code(message.content["code"], language="python")
                if message.content["result"]:
                    self.chat.renderer.render_artifact(message.content["result"])
    
    def send_message(self, content: str):
        # Add user message
        self.add_message(ChatMessage(role="user", content=content, message_type="text"))
        
        # Process with AgentBot
        agent = lmb.AgentBot(
            tools=[write_and_execute_code(globals())],
            on_message_callback=self.add_message,
        )
        
        try:
            result = agent(content, describe_dataframes_in_globals(globals()))
            self.add_message(ChatMessage(role="assistant", content=result.content, message_type="text"))
        except Exception as e:
            self.add_message(ChatMessage(role="system", content=f"Error: {str(e)}", message_type="error"))

Files to Create/Modify

Core Framework-Agnostic Components

• Create: llamabot/components/multi_message.py (ChatMessage, MessageRenderer, framework renderers)
• Create: llamabot/components/chat_interface.py (MultiMessageChat class)
• Create: tests/components/test_multi_message.py
• Create: tests/components/test_chat_interface.py

Framework-Specific Implementations

• Create: llamabot/components/marimo_chat.py (Marimo-specific chat interface)
• Create: llamabot/components/panel_chat.py (Panel-specific chat interface)
• Create: llamabot/components/streamlit_chat.py (Streamlit-specific chat interface)

AgentBot Updates

• Modify: llamabot/bot/agentbot.py (add callback support)
• Create: tests/bot/test_agentbot_streaming.py

Example Notebooks

• Modify: notebooks/toolbot_chatdata.py (use new framework-agnostic interface)
• Create: examples/panel_multi_message_chat.py
• Create: examples/streamlit_multi_message_chat.py

Key Design Decisions

1. Framework-agnostic core: ChatMessage and MessageRenderer abstract the differences
2. Pluggable renderers: Each framework has its own renderer implementation
3. Unified interface: MultiMessageChat provides consistent API across frameworks
4. Callback pattern: AgentBot calls back with ChatMessage objects
5. Progressive enhancement: Start with Marimo, add Panel/Streamlit later

Benefits

• Cross-framework compatibility: Same code works with Marimo, Panel, Streamlit
• Consistent API: Unified interface regardless of UI framework
• Extensible: Easy to add new frameworks (Gradio, Dash, etc.)
• Type safety: ChatMessage provides clear message structure
• Testable: Framework-agnostic components are easy to test

Implementation Priority

1. Phase 1: Core framework-agnostic components + Marimo implementation
2. Phase 2: Panel implementation
3. Phase 3: Streamlit implementation
4. Phase 4: Additional frameworks (Gradio, Dash, etc.)

Example Usage

# Marimo
def chat_turn(messages, config):
    chat = MultiMessageChat(MarimoRenderer(mo))
    agent = lmb.AgentBot(on_message_callback=chat.add_message)
    result = agent(messages[-1].content)
    yield chat.add_text(result.content)

# Panel
chat_interface = PanelChatInterface()
agent = lmb.AgentBot(on_message_callback=chat_interface.add_message)
result = agent(user_input)

# Streamlit
chat_interface = StreamlitChatInterface()
agent = lmb.AgentBot(on_message_callback=chat_interface.add_message)
result = agent(user_input)

This design provides a clean, framework-agnostic way to handle multi-message chat responses across different UI frameworks!

Status

Ready for implementation when needed. All architectural decisions made and implementation plan detailed.

Related

• Current ToolBot pattern in notebooks/toolbot_chatdata.py
• Marimo chat UI documentation
• AgentBot ReAct loop implementation