fix(closes OPEN-11901): key the LangChain provider map on ls_provider - #220
Merged
gustavocidornelas merged 1 commit intoJul 30, 2026
Conversation
LANGCHAIN_TO_OPENLAYER_PROVIDER_MAP was keyed on LangChain *Python* `_llm_type`
strings (openai-chat, chat-ollama, vertexai, azure-openai) while the lookup reads
LangChain *JS* `metadata.ls_provider`. Most lookups missed, so steps recorded
provider "Unknown", the step name degraded to the raw run name, and cost
estimation resolved to $0 (the backend prices on an exact lowercased
(provider, model) match with no aliasing).
Normalize before lookup — case-fold and strip separators — so one table covers
all three shapes `ls_provider` takes in the wild: vendor literals
("google_vertexai", "amazon_bedrock"), the PascalCase class name that
@langchain/core's BaseChatModel.getLsParams() emits by default for any model not
overriding it ("VertexAI", "MistralAI"), and the legacy Python `_llm_type` values,
which now work as free aliases so nothing that works today regresses.
Add Ollama, Bedrock, Watsonx, Mistral, Groq, xAI and Cerebras, each verified
against the live cost table. Fix Azure: the canonical value was "Azure OpenAI",
which lowercases to "azure openai" and matched nothing, so Azure steps were
silently $0 as well; only "azure" exists upstream, and it matches the Python SDK.
The step name keeps the fuller "Azure OpenAI Chat Completion" label.
Warn once per unrecognized provider instead of silently writing "Unknown" — the
silence is what hid this through two review cycles — and drop the heuristic
'meta' branch, since Llama is served by many vendors and "meta" is not a slug.
Regression tests constrain the tables rather than just correcting them: every
canonical provider must declare a cost slug, must lowercase to it, must resolve a
real step name, and every map key must be normalizer-reachable.
Verified end to end against a live pipeline: a Vertex run prices at $0.00155 and
an Azure run at $0.0075, both matching hand computation from the cost table, while
an unmapped control reproduces "Unknown" and the degraded step name at $0.
Known gap, deliberately not fixed here: upstream splits Bedrock pricing across
"bedrock" and "bedrock_converse" by model family, so Claude-on-Bedrock still
prices at $0. That fallback belongs in the backend's _resolve_price, not in the
SDK, which should not encode pricing topology.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGNjH6vRD8HBw1cVkqv2yj
gustavocidornelas
merged commit Jul 30, 2026
efd984d
into
vini/open-11695-ts-langchain-gemini-provider-cost
4 checks passed
gustavocidornelas
deleted the
vini/open-11901-ts-langchain-callback-provider-map-keys-dont-match-langchain
branch
July 30, 2026 12:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes OPEN-11901.
The bug
The provider map was keyed on LangChain Python
_llm_typestrings (openai-chat,chat-ollama,vertexai,azure-openai) while the lookup reads LangChain JSmetadata.ls_provider— a different vocabulary. Most lookups missed, so steps recordedprovider = "Unknown", the step name degraded to the raw run name, and cost resolved to $0.Why normalization, not just re-keying
@langchain/core'sBaseChatModel.getLsParams()defaults togetName().replace('Chat',''), so any model that doesn't override it emits a PascalCase class name (ChatDeepSeek→"DeepSeek"). Literal keys can never match those. Normalizing (case-fold + strip separators) covers vendor literals, PascalCase defaults, and the legacy Python keys — which survive as aliases, so nothing that works today regresses.Added, each verified against the live cost table:
ollama,amazon_bedrock/bedrock,watsonx,mistral,groq,xai,cerebras. Values were verified by grepping the installed packages — note@langchain/awsemitsamazon_bedrock, notbedrock(that's legacy@langchain/community), and no package emitshuggingface.Azure was also silently $0
The canonical value
"Azure OpenAI"lowercases toazure openai, which matches nothing; onlyazureexists upstream. NowAzure, matching what the Python SDK already emits. The step name keeps the"Azure OpenAI Chat Completion"label — only the cost-key field changed.Verified end to end
Five synthetic runs published to a real pipeline, then read back for the cost the backend computed:
google_vertexaiGooglegemini-2.5-flashgoogle_genaiGooglegemini-2.5-flashazureAzuregpt-4oollamaOllamallama2Unknowngemini-2.5-flashBoth non-zero figures match hand computation from the cost table, so the price resolved rather than defaulting. The control row reproduces both original symptoms, confirming the map gates pricing.
Regression tests
OPEN-11314/11316 reviewed this handler and missed the bug, so the tests constrain the tables rather than checking today's values: every canonical provider must declare a cost slug and lowercase to it, every slug must be one verified present in the cost table, every key must be normalizer-reachable, and every mapped provider must resolve a real step name. Plus ground-truth vocabulary, normalization, legacy-key non-regression, and warn-once-on-miss.
Reviewer notes
PROVIDER_TO_STEP_NAME['Google']is now"Google Chat Completion". Vertex and Gemini get accurate, now distinct names via anls_provider-keyed override (a Gemini Developer API call was previously mislabelled "Vertex AI"), but a heuristic-inferred Google step reads differently than before.'meta'branch: Llama is served by Groq, Bedrock and others, so the model name doesn't identify a vendor, andmetaisn't a slug.integrations/index.tsdoesexport *, so this is a small deliberate public-API addition.Known gaps (server-side; the SDK shouldn't encode pricing topology)
bedrockandbedrock_converseby model family, so one canonicalBedrockcan't satisfy both.gemini-3-pro-previewis still $0 underGoogle. Narrower than it looks: the feed suggestsgooglecovers only 24/50gemini-*ids, but the backend's table accumulates via an upsert-never-prune cron, and published rows confirmgemini-2.0-flashprices fine. Switching to the higher-coveragegeminislug isn't safe — 6 of 35 shared models differ in amount (gemini-flash-latestreturns $0.00525 underGooglevs $0.00155 undergemini), because the two slugs come from LiteLLM and OpenRouter respectively.Both stem from one root cause:
providerdoes double duty as display label and exact cost key, and those goals conflict (Azure OpenAI= good label, broken key;fireworks_ai= good key, ugly label). That's also whyfireworks/together/novitaare left unmapped. See also OPEN-11933 — OTel-ingested Azure traces are priced $0 for every customer today, same defect class on the ingestion path.Test plan
yarn jest tests/langchain-callback.test.ts— 50 passed;npx tsc --noEmitclean;yarn lintclean (prettier, types, attw, publint)tests/api-resources/*(need the mock server) and one case intests/openai-tracer.test.ts🤖 Generated with Claude Code
https://claude.ai/code/session_01JGNjH6vRD8HBw1cVkqv2yj