Skip to content

inference-router: substituted model names leak into the client's session state and fail to restore #1597

Closed as not planned

Description

@huang195

What happened

When the inference-router sends a request to a server that refuses the agent's model, it resends the request with the server's substitute (chooseModel: the server's main for a request with tools, helper without). The server then answers with model: naming the substitute it served — and at least Claude Code persists that name as the session's model.

On the conversation's next restart, the client cannot restore it:

⏺ Session model rits/zai-org/glm-5-3 could not be restored (not a model this
  version of Claude Code recognizes) — using opus instead.

Live trace, 2026-10-29 (laptop install, agent routed to a GLM server behind LiteLLM):

  • Session 1439e16a: requests carried requestedModel: claude-opus-5-5, sent model rits/zai-org/glm-5-3 (router substitution on glm); every response reported model: rits/zai-org/glm-5-3.
  • The conversation continued into a new session id (ca3c9830) at 22:01; Claude Code failed to restore rits/zai-org/glm-5-3 and silently fell back to opus.

Impact

  • Every restart of a continued conversation whose server substitutes models shows the restore error and falls back to the default model. Substitution is the default behavior for any server that does not serve the agent's own model names, so this hits every routed agent on such a server.
  • The substitute name only exists on the wire because the router rewrote the request; the server answered honestly with what it served. The leak is ours.
  • Compounds the already-documented continued-in behavior: the new session id is judged a continuation and not routed, so the conversation lands on the terminal's own base URL and on the default model.

Fix options (decision needed, not presuming one)

  1. Rewrite the response's model field back to the name the client asked for. This is a WritesResponseBody, which is the SSE streaming predicate — declaring it would force the buffered relay path for all traffic. Probably too expensive for a cosmetic field.
  2. Accept and document — the client's fallback already works (it uses opus); the cost is the error banner and the model reset. Doc note in docs/agents/claude-code.md goes with this issue.
  3. Something cheaper than 1 that still stops the name reaching client session state — none identified yet.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Activity

  1. huang195 commented on Oct 30, 2026

    @huang195
    MemberAuthor

    Doc note: #1598

  2. huang195 commented on Oct 30, 2026

    @huang195
    MemberAuthor

    Closing without action. Fix options assessed: rewriting the response's model field needs WritesResponseBody, which forces the buffered relay path for all streaming traffic; a streaming-frame rewrite would need a new framework capability (frames are read-only today). Neither is worth it for this. The client's opus fallback works and is visible, so leaving this as accepted behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions