Skip to content

fix(closes OPEN-11901): key the LangChain provider map on ls_provider - #220

Merged
gustavocidornelas merged 1 commit into
vini/open-11695-ts-langchain-gemini-provider-costfrom
vini/open-11901-ts-langchain-callback-provider-map-keys-dont-match-langchain
Jul 30, 2026
Merged

fix(closes OPEN-11901): key the LangChain provider map on ls_provider#220
gustavocidornelas merged 1 commit into
vini/open-11695-ts-langchain-gemini-provider-costfrom
vini/open-11901-ts-langchain-callback-provider-map-keys-dont-match-langchain

Conversation

@viniciusdsmello

@viniciusdsmello viniciusdsmello commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Closes OPEN-11901.

Stacked on OPEN-11695. Base is vini/open-11695-ts-langchain-gemini-provider-cost; GitHub retargets to main once that merges.

The bug

The provider map was keyed on LangChain Python _llm_type strings (openai-chat, chat-ollama, vertexai, azure-openai) while the lookup reads LangChain JS metadata.ls_provider — a different vocabulary. Most lookups missed, so steps recorded provider = "Unknown", the step name degraded to the raw run name, and cost resolved to $0.

Why normalization, not just re-keying

@langchain/core's BaseChatModel.getLsParams() defaults to getName().replace('Chat',''), so any model that doesn't override it emits a PascalCase class name (ChatDeepSeek"DeepSeek"). Literal keys can never match those. Normalizing (case-fold + strip separators) covers vendor literals, PascalCase defaults, and the legacy Python keys — which survive as aliases, so nothing that works today regresses.

Added, each verified against the live cost table: ollama, amazon_bedrock/bedrock, watsonx, mistral, groq, xai, cerebras. Values were verified by grepping the installed packages — note @langchain/aws emits amazon_bedrock, not bedrock (that's legacy @langchain/community), and no package emits huggingface.

Azure was also silently $0

The canonical value "Azure OpenAI" lowercases to azure openai, which matches nothing; only azure exists upstream. Now Azure, matching what the Python SDK already emits. The step name keeps the "Azure OpenAI Chat Completion" label — only the cost-key field changed.

Verified end to end

Five synthetic runs published to a real pipeline, then read back for the cost the backend computed:

ls_provider provider model Cost
google_vertexai Google gemini-2.5-flash $0.00155
google_genai Google gemini-2.5-flash $0.00155
azure Azure gpt-4o $0.0075
ollama Ollama llama2 $0.0 (genuine — local models are free)
unmapped Unknown gemini-2.5-flash $0.0 + step name degraded

Both non-zero figures match hand computation from the cost table, so the price resolved rather than defaulting. The control row reproduces both original symptoms, confirming the map gates pricing.

Regression tests

OPEN-11314/11316 reviewed this handler and missed the bug, so the tests constrain the tables rather than checking today's values: every canonical provider must declare a cost slug and lowercase to it, every slug must be one verified present in the cost table, every key must be normalizer-reachable, and every mapped provider must resolve a real step name. Plus ground-truth vocabulary, normalization, legacy-key non-regression, and warn-once-on-miss.

Reviewer notes

  • A step name changed: PROVIDER_TO_STEP_NAME['Google'] is now "Google Chat Completion". Vertex and Gemini get accurate, now distinct names via an ls_provider-keyed override (a Gemini Developer API call was previously mislabelled "Vertex AI"), but a heuristic-inferred Google step reads differently than before.
  • Unmapped providers now warn once each — the silence is what let this survive two review cycles.
  • Dropped the heuristic 'meta' branch: Llama is served by Groq, Bedrock and others, so the model name doesn't identify a vendor, and meta isn't a slug.
  • Two constants are now exported so the invariants can be asserted; integrations/index.ts does export *, so this is a small deliberate public-API addition.

Known gaps (server-side; the SDK shouldn't encode pricing topology)

  • Claude-on-Bedrock is still $0 — OPEN-11932. Upstream splits Bedrock across bedrock and bedrock_converse by model family, so one canonical Bedrock can't satisfy both.
  • gemini-3-pro-preview is still $0 under Google. Narrower than it looks: the feed suggests google covers only 24/50 gemini-* ids, but the backend's table accumulates via an upsert-never-prune cron, and published rows confirm gemini-2.0-flash prices fine. Switching to the higher-coverage gemini slug isn't safe — 6 of 35 shared models differ in amount (gemini-flash-latest returns $0.00525 under Google vs $0.00155 under gemini), because the two slugs come from LiteLLM and OpenRouter respectively.

Both stem from one root cause: provider does double duty as display label and exact cost key, and those goals conflict (Azure OpenAI = good label, broken key; fireworks_ai = good key, ugly label). That's also why fireworks/together/novita are left unmapped. See also OPEN-11933 — OTel-ingested Azure traces are priced $0 for every customer today, same defect class on the ingestion path.

Test plan

  • yarn jest tests/langchain-callback.test.ts — 50 passed; npx tsc --noEmit clean; yarn lint clean (prettier, types, attw, publint)
  • Live end-to-end verification per the table above
  • Pre-existing failures confirmed identical on the base commit: tests/api-resources/* (need the mock server) and one case in tests/openai-tracer.test.ts

🤖 Generated with Claude Code

https://claude.ai/code/session_01JGNjH6vRD8HBw1cVkqv2yj

LANGCHAIN_TO_OPENLAYER_PROVIDER_MAP was keyed on LangChain *Python* `_llm_type`
strings (openai-chat, chat-ollama, vertexai, azure-openai) while the lookup reads
LangChain *JS* `metadata.ls_provider`. Most lookups missed, so steps recorded
provider "Unknown", the step name degraded to the raw run name, and cost
estimation resolved to $0 (the backend prices on an exact lowercased
(provider, model) match with no aliasing).

Normalize before lookup — case-fold and strip separators — so one table covers
all three shapes `ls_provider` takes in the wild: vendor literals
("google_vertexai", "amazon_bedrock"), the PascalCase class name that
@langchain/core's BaseChatModel.getLsParams() emits by default for any model not
overriding it ("VertexAI", "MistralAI"), and the legacy Python `_llm_type` values,
which now work as free aliases so nothing that works today regresses.

Add Ollama, Bedrock, Watsonx, Mistral, Groq, xAI and Cerebras, each verified
against the live cost table. Fix Azure: the canonical value was "Azure OpenAI",
which lowercases to "azure openai" and matched nothing, so Azure steps were
silently $0 as well; only "azure" exists upstream, and it matches the Python SDK.
The step name keeps the fuller "Azure OpenAI Chat Completion" label.

Warn once per unrecognized provider instead of silently writing "Unknown" — the
silence is what hid this through two review cycles — and drop the heuristic
'meta' branch, since Llama is served by many vendors and "meta" is not a slug.

Regression tests constrain the tables rather than just correcting them: every
canonical provider must declare a cost slug, must lowercase to it, must resolve a
real step name, and every map key must be normalizer-reachable.

Verified end to end against a live pipeline: a Vertex run prices at $0.00155 and
an Azure run at $0.0075, both matching hand computation from the cost table, while
an unmapped control reproduces "Unknown" and the degraded step name at $0.

Known gap, deliberately not fixed here: upstream splits Bedrock pricing across
"bedrock" and "bedrock_converse" by model family, so Claude-on-Bedrock still
prices at $0. That fallback belongs in the backend's _resolve_price, not in the
SDK, which should not encode pricing topology.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGNjH6vRD8HBw1cVkqv2yj
@viniciusdsmello viniciusdsmello self-assigned this Jul 28, 2026
@gustavocidornelas
gustavocidornelas merged commit efd984d into vini/open-11695-ts-langchain-gemini-provider-cost Jul 30, 2026
4 checks passed
@gustavocidornelas
gustavocidornelas deleted the vini/open-11901-ts-langchain-callback-provider-map-keys-dont-match-langchain branch July 30, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants