Repository navigation
Extract provider specific logic from llm crate into a codec trait - #32
Merged
Merged
Conversation
Provider-specific request options currently have nowhere to live: any new knob forces a typed field on LlmRequest or RequestOptions that every provider then sees. Add Extensions, a TypeId-keyed side channel carried on every request, so codecs can pick up the options meant for them. For now nothing inserts anything; provider_order and session_id keep their typed fields. Wiring happens over the next commits.
The codec functions are hardwired into the API, so a provider cannot change the wire format without forking the whole API struct. Split the two concerns: LlmApiCodec owns request serialization and chunk decoding, ChatCompletionsApi owns transport, SSE framing, and cross-frame state. DefaultChatCompletionsCodec implements the trait over the existing wire code, so the default path is unchanged. Providers attach their own codec via with_codec; wire structs stay private to each codec, the contract is just a body string in and CodecChunks out.
OpenRouter's provider routing block, sticky session_id, and usage cost parsing lived inside the generic chat completions codec, so every OpenAI-compatible provider was sent OpenRouter-only fields and cost had nowhere provider-neutral to live. BaseRequest and BaseResponse stay in llm as the canonical wire shapes; OpenRouterCodec flattens them and adds its own fields, then fills Usage cost fields from the flattened usage object. The provider block and session_id now only appear on OpenRouter requests, read from request extensions (openrouter::Options, SessionId) instead of typed fields. zai and local providers use the default codec and their bodies lose the stray provider/session_id keys; OpenRouter's own body is unchanged.
provider_order and session/prompt-cache keys were typed fields on the canonical request types, so every provider saw OpenRouter concepts. The canonical types now describe only what every chat API shares; anything provider-specific rides in Extensions, keyed by types defined next to their consumers: - llm keeps SessionId, the one cross-provider concept (agent sets it, codecs translate: OpenRouter to session_id + prompt_cache_key, default codec to prompt_cache_key only) - providers keeps OpenRouterOptions (routing order), built by alan from settings and passed at bind time - Model merges model-level and per-request extensions, so both the bound options and the agent's session id reach the codec - Agent::set_provider_order becomes set_extensions, agnostic of provider ModelOptions and CompletionInput carry extensions instead of provider_order; wire behavior for OpenRouter is unchanged and zai keeps sending only prompt_cache_key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.