Repository navigation
Document that a Moddy LLM proxy must end streamed responses - #1127
Conversation
|
|
||
| * Your proxy can read the user identifier from the request body for OpenAI and Anthropic without configuration. For Google Gemini and Mistral, you will need to configure your proxy to read the `x-moderne-user-id` header. In LiteLLM, for example, add `header_name: x-moderne-user-id` with `litellm_user_role: customer` under `general_settings.user_header_mappings`. | ||
| * Routing by country is a rule you add to your proxy that reads `x-moderne-user-country`. No proxy applies one by default. | ||
| * Moddy asks your LLM for every answer as a stream. Your proxy will need to pass the stream through and end its response when the provider's stream ends. |
There was a problem hiding this comment.
I feel like I'm a bit confused by this addition. What does "pass through" mean? I feel like there are a variety of options of what it could mean.
I also feel like it sort of hides the "end its response when the provider's stream ends" part - which I think is the more important part?
There was a problem hiding this comment.
Well if you are confused, pretty good chance it won't land with customers. I'll try to make it more clear.
There was a problem hiding this comment.
I feel like I'm a bit confused by this addition. What does "pass through" mean? I feel like there are a variety of options of what it could mean.
After thinking about it more, I'm not specifically calling out who ends the stream and where to end it. I've updated the wording to very pointedly state that the provider should end the stream in Connector.
I also feel like it sort of hides the "end its response when the provider's stream ends" part - which I think is the more important part?
I changed it to lead with the ask.
I left the last sentence as the explanation to answer the "why" portion of the ask.
Appreciate the call out since I couldn't see the forest for the trees on the first pass.
There was a problem hiding this comment.
Thanks! It sounds much better to me now.
Problem
The Moddy page tells admins to point the provider's
URIat their own LLM proxy, but not that Moddy asks for every answer as a stream. A proxy that keeps its response open after the provider's stream ends makes every Moddy chat fail with an error.Objectives
Changes
curlcheck proposed in the issue is left out as more than the page needs. Closes Moddy: document how an LLM proxy must handle streamed responses #1126.Verification
yarn buildandyarn lint:markdownpass, with no warnings beyond those already on main.