TwilioMultiCallMemory

A voice agent that remembers callers across disconnected calls, built on Twilio ConversationRelay and Conversation Memory. If a caller phones the shop, says "my Civic needs new brake pads," and hangs up, then dials back an hour later, the agent should open with something like "Welcome back — last time we were discussing brake pads on your Civic."

The persona is Owlbert's Auto Repair, a fictional car repair shop. There is no database or data storage in the app itself. The only knowledge the agent has about the caller is whatever Twilio Conversation Memory hydrates.

How it remembers

On every call, the app:

1. Creates a Twilio Conversation via POST https://conversations.twilio.com/v2/Conversations, wiring the caller's phone number as the CUSTOMER participant and the Twilio number as the AI_AGENT participant. Returns a conv_conversation_... ID.
2. Creates or resolves a Profile for the caller's phone number via POST /v1/Stores/{store}/Profiles/Lookup (falling back to POST /v1/Stores/{store}/Profiles). Returns a mem_profile_... ID.
3. Recalls prior summaries via POST /v1/Stores/{store}/Profiles/{profileId}/Recall and prepends them to the OpenAI system prompt.
4. Runs the live conversation over the ConversationRelay WebSocket.
5. On call end, asks the model to summarize the transcript in 4–8 sentences and writes it via POST /v1/Stores/{store}/Profiles/{profileId}/ConversationSummaries. That summary is what surfaces on the next call in step 3.

Prerequisites

• .NET 10 SDK
• A Twilio account migrated to the new Twilio Console (Conversation Memory only shows up there)
• A voice-capable Twilio phone number
• An OpenAI API key
• ngrok (or any HTTPS tunnel)
• curl in your shell

Setup

1. Clone and configure

git clone <this-repo-url>
cd TwilioMultiCallMemory
cp .env.example .env

2. Create a Twilio API Key

In Console → Account → API keys & tokens → Create API key. Choose Main as the key type (Standard keys don't work with the Orchestrator endpoints). Copy the SID (SK...) and Secret into .env as TWILIO_API_KEY and TWILIO_API_SECRET.

3. Create a Memory Store (Console)

Navigate to Products & services → Conversation Memory → Overview → Get started. Twilio creates a store for you. Open Settings and copy the mem_store_... ID into .env as TWILIO_MEMORY_STORE_ID.

4. Create a Conversation Configuration (curl)

The Console wizard doesn't expose voice captureRules cleanly, so this one is done via API:

export TWILIO_API_KEY=SK...
export TWILIO_API_SECRET=...
export MEMORY_STORE_ID=mem_store_...
export TWILIO_NUMBER=+15551234567

curl -X POST "https://conversations.twilio.com/v2/ControlPlane/Configurations" \
  -u "$TWILIO_API_KEY:$TWILIO_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "displayName": "owlbert-voice",
    "description": "Voice traffic for the Owlbert repair shop agent",
    "conversationGroupingType": "GROUP_BY_PARTICIPANT_ADDRESSES_AND_CHANNEL_TYPE",
    "memoryStoreId": "'"$MEMORY_STORE_ID"'",
    "channelSettings": {
      "VOICE": {
        "statusTimeouts": { "inactive": 5, "closed": 30 },
        "captureRules": [
          { "from": "'"$TWILIO_NUMBER"'", "to": "*" },
          { "from": "*", "to": "'"$TWILIO_NUMBER"'" }
        ]
      }
    }
  }'

Copy the returned id into .env as TWILIO_CONFIGURATION_ID. Also put your Twilio phone number into .env as TWILIO_PHONE_NUMBER in E.164 format.

5. Fill in the rest of .env

Your final .env should have:

OPENAI_API_KEY=sk-...
TWILIO_API_KEY=SK...
TWILIO_API_SECRET=...
TWILIO_MEMORY_STORE_ID=mem_store_...
TWILIO_CONFIGURATION_ID=<id from step 4>
TWILIO_PHONE_NUMBER=+15551234567

6. Build and run

dotnet restore
dotnet build
dotnet run

Listens on http://localhost:5189.

7. Expose it and point the number at it

ngrok http 5189

In the Console, open your phone number → Voice Configuration → A call comes in → Webhook → https://<your-ngrok-host>/voice, method HTTP POST. Save.

Test it end-to-end

1. Dial your Twilio number. You should hear: "Thanks for calling Owlbert's Auto Repair. How can I help you today?" Say something specific: "Hi, this is John Smith. My 2018 Honda Civic needs new brake pads." Have a short chat, then hang up.

2. In the app terminal, watch for:

   Call summary (N chars): The caller, John Smith, mentioned his 2018 Honda Civic needs new brake pads...
   Saved conversation summary for profile mem_profile_... (conv conv_conversation_..., N chars)
   

   That's the write path confirming.

3. Dial again from the same number. You should see:

   Call connected from +1...
   Conversation conv_conversation_..., profile mem_profile_..., memory N chars
   

   The agent should open with recognition — name, vehicle, and brake pads all mentioned unprompted.

Project layout

File	Role
Program.cs	Minimal API host. POST /voice returns ConversationRelay TwiML; /ws accepts the WebSocket.
ConversationRelayHandler.cs	WebSocket lifecycle. On setup calls memory.StartCallAsync; on socket close, summarizes and calls memory.FinishCallAsync.
OpenAiService.cs	Streams gpt-4o-mini chat completions with the memory context prepended. SummarizeConversationAsync runs at call end.
ConversationMemoryService.cs	The Twilio Conversation Memory integration — Create Conversation, Lookup/Create Profile, Recall, ConversationSummaries. Handles 409 (previous call still open) by reusing the existing conversation.
.env.example	Template for local secrets.

Troubleshooting

• 401 Authenticate on Create Conversation — your API Key isn't a Main key. Regenerate as Main in Console → Account → API keys & tokens.
• 409 Address mapping already exists — expected when calling back before the previous conversation auto-closes. The app handles this by reusing the existing conversation; look for Reusing existing conversation ... in the log.
• Second call doesn't remember — check the terminal after your first call for Saved conversation summary for profile .... If that line never printed, the write failed and Recall on the next call will return nothing.
• The requested operation requires an element of type 'Object' — response parsing hit an unexpected shape. The service logs the raw response (Profile lookup raw response: ...) so you can see what came back; extend the parser if needed.
• Caller phone is empty in the setup frame — some carriers withhold caller ID. Memory is keyed on that phone number, so if it's blank there's nothing to look up.

Next steps

• On-demand recall. Currently we only recall once at call start. For long calls, a search_memory(query) tool could pull more context mid-conversation.
• Cross-channel continuity. Add an SMS number to the same Conversation Configuration; the same profile accumulates memory across voice and text.
• Explicit conversation close. Right now we rely on Twilio's statusTimeouts.closed (30s) to close conversations. For deterministic behavior, add an explicit close call at the end of FinishCallAsync.

References

• Conversation Memory overview
• Using Conversation Memory APIs (quickstart)
• Conversation Orchestrator quickstart
• Active Creation guide
• Create Profile Conversation Summary
• ConversationRelay TwiML reference