[Bug]: Additional LLM provider prices dropped at load time (missing from PROVIDERS_MAPPING)

[Anuj7411]

What component(s) are affected?

• Opik Server

Describe the problem

Following the provider-registration fixes already merged for azure (#7262), xai (#7433), deepseek (#7434), perplexity (#7432), fireworks_ai (#7554) and moonshot (#7604), several more high-volume providers are still dropped at load time. CostService.buildModelPrice returns null for any entry whose litellm_provider is not in PROVIDERS_MAPPING, so every model tagged with an unregistered provider gets no cost.

Scanning the bundled model_prices_and_context_window.json for litellm_provider values that carry non-zero-cost entries but are absent from PROVIDERS_MAPPING, the remaining gaps are (approx. non-zero-priced model counts):

• openrouter (~92, with ~32 carrying cache_read rates)
• novita (~85)
• deepinfra (~67)
• together_ai (~36)
• nebius (~30)
• databricks (~28)
• snowflake (~21, ~9 with cache_read)
• sambanova (~17)
• cerebras (~7)

The OpenAI-compatible inference providers (novita, deepinfra, nebius, snowflake, sambanova, cerebras, databricks) are the same shape as the ones already registered: map <provider> -> <provider> and route through textGenerationWithCacheCostOpenAI when the provider publishes cache rates, else textGenerationCost.

Aggregators (openrouter, together_ai) need more care. openrouter namespaces models under the origin vendor (e.g. moonshotai/*, anthropic/*) so it should reuse the existing provider-prefix fallback in findModelPrice rather than a naive openrouter -> openrouter mapping; together_ai currently only has parameter-size bucket entries (together-ai-4.1b-8b, ...) in the JSON, so it needs bucket-lookup logic, not just a mapping entry.

Additional info

Happy to open PRs for the clean single-vendor providers (they mirror the merged fireworks/moonshot PRs one-to-one) and to discuss the aggregator handling separately. Filing as a tracking issue first so the team can steer scope.

[github-actions]

I have enough information from what was already retrieved to write the response. The key findings are:

1. PROVIDERS_MAPPING in CostService.java is the gatekeeper — any litellm_provider not in it causes models to be silently dropped at parse time
2. The pattern for simple providers is well-established (map X -> X, add cache calculator when needed)
3. openrouter and together_ai need special handling

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing workaround. For the affected providers, CostService.buildModelPrice silently returns null for any JSON entry whose litellm_provider is not in PROVIDERS_MAPPING, so all ~286 priced models across the nine listed providers produce zero cost. The only mitigation today is to pass cost data via span metadata, which getCostFromMetadata will pick up as a fallback.

Code Investigation

The entire fix lives in one file:
apps/opik-backend/src/main/java/com/comet/opik/domain/cost/CostService.java

There are two maps that must be updated together:

1. PROVIDERS_MAPPING (line ~28) — maps litellm_provider value → canonical provider key used as the storage key in modelProviderPrices. Any litellm_provider absent here causes buildModelPrice to return null and the model is silently skipped.

private static final Map<String, String> PROVIDERS_MAPPING = Map.ofEntries(
    Map.entry("openai", "openai"),
    Map.entry("fireworks_ai", "fireworks_ai"),
    Map.entry("moonshot", "moonshot"),
    Map.entry("moonshotai", "moonshot"),
    // ... 15 more entries — novita, deepinfra, nebius, databricks,
    //     snowflake, sambanova, cerebras are NOT here
);

2. PROVIDERS_CACHE_COST_CALCULATOR (line ~58) — maps canonical provider → cost calculator BiFunction. Providers that publish cache_read_input_token_cost in the JSON need an entry here using SpanCostCalculator::textGenerationWithCacheCostOpenAI; otherwise they fall through to textGenerationCost.

private static final Map<String, BiFunction<ModelPrice, Map<String, Integer>, BigDecimal>> PROVIDERS_CACHE_COST_CALCULATOR = Map
    .ofEntries(
        Map.entry("openai", SpanCostCalculator::textGenerationWithCacheCostOpenAI),
        Map.entry("xai",    SpanCostCalculator::textGenerationWithCacheCostOpenAI),
        // ... deepinfra, snowflake, openrouter (cache-capable) are NOT here
    );

The referenced PRs for prior providers (#7433 xai, #7434 deepseek, #7432 perplexity, #7554 fireworks_ai, #7604 moonshot) all followed the same two-line pattern — one entry in each map.

Test coverage exists in apps/opik-backend/src/test/java/com/comet/opik/domain/cost/CostServiceTest.java and follows a consistent parameterised pattern that should be extended for each new provider.

Next Steps

The issue correctly splits work into two buckets:

Group A — Simple OpenAI-compatible providers (recommended as straightforward PRs):

Add to PROVIDERS_MAPPING (identity mapping):

Map.entry("novita",      "novita"),
Map.entry("deepinfra",   "deepinfra"),
Map.entry("nebius",      "nebius"),
Map.entry("databricks",  "databricks"),
Map.entry("snowflake",   "snowflake"),
Map.entry("sambanova",   "sambanova"),
Map.entry("cerebras",    "cerebras"),

For providers with cache rates (deepinfra, snowflake per the issue), also add to PROVIDERS_CACHE_COST_CALCULATOR:

Map.entry("deepinfra", SpanCostCalculator::textGenerationWithCacheCostOpenAI),
Map.entry("snowflake", SpanCostCalculator::textGenerationWithCacheCostOpenAI),

These exactly mirror the fireworks_ai/moonshot PRs and each warrants a focused unit test in CostServiceTest.java verifying that a representative model from the JSON resolves to a non-zero cost.

Group B — Aggregators (needs design discussion):

• together_ai: The JSON models are named together-ai-4.1b-8b etc. (parameter-bucket entries, not real model names). Naive identity mapping would index them by bucket name, but callers pass real model names like togethercomputer/llama-2-70b. A bucket-lookup function needs to be added — this is non-trivial and should be a separate issue/PR.
• openrouter: Models in the JSON are namespaced under the origin vendor (e.g. anthropic/claude-3-5-sonnet). The existing parseModelName() strip logic and provider-prefix fallback in findModelPrice may already handle lookups when the caller passes provider=openrouter + model=anthropic/claude-3-5-sonnet, but PROVIDERS_MAPPING still needs Map.entry("openrouter", "openrouter") plus cache calculator entries for the ~32 cache-capable models. The interaction with the existing vendor-prefix fallback needs careful testing.

For Group A, feel free to open a pull request — the changes are mechanical and well-precedented in the codebase. Group B benefits from a design discussion before coding.

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik