How Cortex turns tokens into money: what it ships, how to see what is in effect, where to get real numbers, how to override them, and the record it publishes for each request.
Cost is core, not a plugin. pricing: and cost_ledger: are top-level config
sections, siblings of session: and mtls:. settle.Settle decides what a request cost;
the durable ledger, the aggregator and /v1/usage are all in core/. A plugin only
feeds the machinery — inference-parser calls Settle because it is the component that
parses token counts, and litellm-budget-track reads the result for caps and drift.
Neither owns the cost model. One owner of rates means tool-prune,
litellm-budget-track, /v1/usage and agentop cannot disagree about what a request cost.
| Concern | Package | Documented |
|---|---|---|
| Rates, discounts, resolution | core/cost/pricing |
this page |
| Settling one request's cost | core/cost/settle |
this page (Cost records) |
| The published record | core/cost/event |
this page (Cost records) |
Durable per-minute ledger (cost_ledger:) |
core/cost/ledger |
laptop-service.md for the config; framework-architecture.md for why it is not hot-reloadable |
Windowed aggregation, /v1/usage |
core/cost/usage |
this page (Reading usage over HTTP) |
| Daily caps, drift check | core/plugins/litellm_budgettrack |
litellm-budget-track-plugin.md |
Terminology. Rate — money per token for one tier of one model at one endpoint. Tier — which kind of token: uncached input, cache write, cache read, output. Provenance — where a rate came from, which travels with every figure Cortex reports.
| Your setup | What to do |
|---|---|
Talking straight to api.anthropic.com or api.openai.com |
Nothing. Bundled rates are vendor list, which is what you are billed. |
| A gateway matching a shipped discount rule (see below) | Nothing. The discount is applied for you. |
| Any other gateway that bills below list | Set one multiplier — Discounts. |
| Using a model on Gemini, Bedrock or Vertex | Add rates — nothing ships for the resellers. Getting the numbers. |
| A model that reports priceable but unpriced anyway | A handful are, deliberately — see What is bundled. Add rates for it. |
| A gateway billing in something other than dollars | Set its unit — Billing units. Rates still have to be configured. |
Cortex warns once at startup if you have pinned nothing at all:
WARN pricing: every endpoint will price from the bundled table, which ships VENDOR LIST rates
effect=a gateway that bills below list is OVERSTATED ...
It is a warning, not an error — list pricing is correct for anyone talking to the vendor directly, and only silently wrong otherwise.
Generated, not hand-written: core/cost/pricing/bundled.go carries
// Code generated by pricing/internal/gen. DO NOT EDIT. It is derived from
BerriAI/litellm's
model_prices_and_context_window.json, pinned to one commit that agentop pricing --json
reports as upstreamCommit.
A local install does not stay on that commit. It downloads the same file at startup and
then hourly. An unchanged file answers 304 with no body, so checking costs nothing. The
last good download is kept in ~/.cortex/price-list.json, so a restart, or a laptop with no
network, prices from it. A download that fails, or is not a usable price map, changes
nothing. agentop pricing then reports rates from litellm's price list, downloaded <time>,
and --json carries it as listFetchedAt. bundled: false turns the download off with the
rest of the shipped table. Sidecars in a cluster make no such call and keep the compiled-in
table.
172 entries: 167 exact model ids plus 5 family globs. The exact rows split 28 Anthropic to 139 OpenAI. The globs are Anthropic-only and matter most there, because they are what catches a model released after the pinned commit:
| Model pattern | input | cache-write | cache-read | output |
|---|---|---|---|---|
*claude-*opus-* |
5 | 6.25 | 0.5 | 25 |
*claude-*sonnet-* |
2 | 2.5 | 0.2 | 10 |
*claude-*haiku-* |
1 | 1.25 | 0.1 | 5 |
*claude-*fable-* |
10 | 12.5 | 0.25 | 50 |
*claude-*mythos-* |
10 | 12.5 | 0.25 | 50 |
USD per million tokens, at host scope * (any endpoint). Run agentop pricing for the
full list including all 167 exact pins — that command is the source of truth, and this
table is deliberately only the stable part.
Some entries also carry a long-context tier, and the breakpoint depends on the vendor. Four Sonnet-4 entries price above 200,000 prompt tokens: input, cache-write and cache-read double, and output rises by half (3 -> 6 input, 15 -> 22.5 output). Eleven OpenAI entries price above 272,000 on the same shape (gpt-5.5: 5 -> 10 input, 30 -> 45 output), which a long session reaches easily — those models accept 922k to 1.05M tokens. Both apply strictly above the breakpoint, so a prompt of exactly 200,000 or 272,000 pays list, and a 250,000-token prompt pays list on an OpenAI model while paying the premium on a Sonnet-4 one.
Those two are the only breakpoints Cortex models. LiteLLM's _above_512k variants, its
_above_1hr cache-write premium, and its per-service-tier suffixes (_flex,
_priority, _batches, _decisions) are out of scope and are ignored rather than
flattened into the tier they resemble.
The generator keeps litellm_provider == "anthropic" and == "openai" and nothing else,
because only the first-party endpoints' rates are vendor list — the vertex_ai/,
bedrock/ and azure/ mirrors carry their own, and mixing more than one provider for a
model under one * host scope would make the table depend on map iteration order. The
filter reads that provider field rather than the key prefix: every model appears once per
reseller at different rates, and the field is the row's own statement of whose price it
is, where prefix-matching would be guessing at a naming convention.
So there are no bundled rates for Gemini, Bedrock or Vertex models. Traffic to them reports as priceable but unpriced until you add rates. This is by design: Cortex reports unpriced rather than under-priced.
Thirteen first-party models are deliberately left unpriced too, and the reason is a
bound rather than an omission. A bundled rate ships to everyone, so it has to be one the
cost pipeline will still accept after the largest multiplier configuration allows (10x)
— past that it would refuse a legitimate bill on a machine whose owner never chose the
rate. Ten rows exceed it on their list rate (o1-pro at $600/Mtok, and the
gpt-5{,.2,.4,.5}-pro tiers, plus their dated keys); three more exceed it only on their
above-272k output rate, at $112.50/Mtok (daybreak-red-latest, gpt-5.6-cyber,
gpt-daybreak-red-latest), and those are dropped whole rather than shipped at a list
rate that would under-price exactly the long requests the tier exists for. All thirteen
report as priceable but unpriced and take a pricing: entry of your own. Nothing about
this is vendor-specific: it is written on the rate, so the first Anthropic model past the
bound goes the same way.
A further 77 first-party rows carry no positive per-token rate, so there is nothing for
this table to hold. Most are image, video and audio models billed in other units, which a
pricing: entry cannot express either; the other five are OpenAI's moderation models,
which are free.
One hand-kept addition sits beside the generated table: OpenCode Zen's free models, priced
at zero on opencode.ai only. That covers every Zen model whose id ends in -free, and
big-pickle, the one free model without the suffix. A call to one of them is a priced
zero, so it counts toward coverage and does not ask you for a rate that does not exist.
It ships because config cannot say zero: a tier given a rate of 0 is left unset, and an
entry whose every rate is 0 is a startup error. bundled: false drops these rows along
with the rest of the shipped table. A -free model on any other endpoint stays unpriced,
because the suffix is Zen's naming and nobody else's.
A configured entry outranks a shipped one whatever its pattern, so a "*" entry for
opencode.ai, or one under hosts: ["*"], replaces these zeros with its own rate. To
price Zen's paid models, name them.
A zero still needs token counts to apply to. A response that reports none, such as an
OpenAI-dialect stream sent without stream_options.include_usage, is reported unpriced
and named in unpricedBy, as for any model. No rate fixes that, since there is nothing to
apply it to. A free call that is priced counts toward coverage, but its cost cells in
the events and sessions panes are blank, since there is no amount to show.
Bundled rates are vendor list, and most internal LiteLLM gateways bill below it. One discount ships, scoped to a host pattern:
$ agentop pricing
Gateway discounts
*.res.ibm.com 0.76 bundled
Scoped rather than global on purpose: a global default would understate anyone talking to
api.anthropic.com by 24%, and understating hides spend. Your own multiplier outranks
this — see Discounts.
No config file can answer this, because the effective figures come from your pricing:
section plus the table compiled into the binary plus any shipped discount. Four views:
agentop pricing # every row unscaled, plus a "Gateway discounts" section
agentop pricing --host <gateway> # what THAT endpoint is charged, discount applied
agentop pricing --json # machine-readable: rows, multipliers, upstreamCommit
curl localhost:47602/pricing/table # the same, over HTTP (add ?host=<gateway>)/pricing/table is served on the diagnostic listener, beside /config and
/reload/status.
Those localhost ports are the laptop install's. In Kubernetes the same endpoints are
on the sidecar's :9093 (stats, /pricing/table) and :9094 (session API, /v1/usage),
reachable via kubectl port-forward.
--host is the useful form — it resolves all three sources the way a request would:
$ agentop pricing --host ete-litellm.ai-models.vpc-int.res.ibm.com
Rates in effect for ete-litellm.ai-models.vpc-int.res.ibm.com
model input cache-wr cache-rd output provenance
$/Mtok $/Mtok $/Mtok $/Mtok
*claude-*opus-* 3.8 4.75 0.38 19 bundled
*claude-*sonnet-* 1.52 1.9 0.152 7.6 bundled
multiplier 0.76 applied, from the bundled rules — rates above are vendor list scaled by it
Note the rates shown are post-discount (opus: 5 × 0.76 = 3.8), and the footer names where the factor came from.
Read the provenance column to tell what Cortex knows from what it assumes:
| Provenance | Meaning |
|---|---|
configured |
From your pricing: section. Outranks everything below. |
discovered |
Reserved; no producer today (see Refreshing). |
bundled |
Compiled in — vendor list, possibly scaled by a shipped discount, or the zero shipped for Zen's free models. Expect it to be wrong for a gateway you have not pinned. |
authoritative |
A settled per-request cost from the gateway's X-Litellm-Response-Cost header. Not a rate. |
A request is priced only if every tier that carried tokens had a rate. Otherwise it is reported unpriced — never under-priced — and named, not merely counted:
agentop cost # "! N of M priceable requests unpriced"
curl localhost:47601/v1/usage | jq .unpricedBy
# {"api.us-east.bob.ibm.com premium-ide": 6, "... openai/gpt-oss-20b": 1}unpricedBy keys are <endpoint> <model>, which is exactly the pair you need for a
pricing: entry.
The Bob endpoint above is the worked example for Billing units: it bills in
Bobcoins rather than dollars, so closing its gap means giving that endpoint a unit: as well as
rates. Pricing it without one would record Bobcoins as dollars — the figure would look right and be
neither. The built-in local config (cortex --local --write-config, which agentop setup runs on a
fresh install, whether the installer or make dev-install started it) ships that entry, so a new
install prices Bob already. A ~/.cortex/config.yaml written before it is never rewritten, so
setup's config step — and agentop service install, run by hand — adds the entry to one that leaves
Bob unpriced, keeping the previous file as config.yaml.before-agentop-pricing. It only ever adds:
whatever already prices Bob's host, however it is written, is left alone, and a running proxy picks
the entry up without a restart. A config only ever run by hand as cortex --local, never through
setup or the service, still needs the block below added by hand.
The TUI annotates the same thing from the other direction: agentop observe's cost total
carries [bundled] or [configured] when a total is wholly one provenance, and names the
dominant source when it is mixed. A wholly authoritative total is left unannotated,
since that is the baseline a reader already assumes. Note this annotation is in the TUI
only — agentop cost does not print it.
GET /v1/usage on the session API (:9094 in Kubernetes, :47601 for the laptop
install) is the aggregate behind agentop cost. Query parameters:
| Parameter | Values | Notes |
|---|---|---|
window |
today, month, 7d, or a duration (1h, 6h) |
today, month and 7d are served from the durable ledger. A duration is served from the in-memory ring. |
group |
none, model, endpoint, session, agent, currency, status, plugin, host (method aliases model) |
See the caveat below. |
resolution |
a duration | Bucket size on a ring-served window. A ledger-backed window is answered as one bucket spanning the whole window and does not read this at all; bucketSeconds reports the span actually served. |
session |
a session id | Combining it with a symbolic window (today, month, 7d) is rejected with 400. |
agent |
an agent label, as group=agent reports it (claude-code, unknown) — versionless; a versioned label such as claude-code/2.1.284 is read as its agent |
Narrows the window to that agent's traffic, every version together. Longer than 96 bytes is rejected with 400. |
Response envelope: window, bucketSeconds, group, buckets[], totals and priced are
always present. Every other field is omitempty and appears only when it applies, so a clean
response is shorter than the struct — among them degraded, ungroupedCostMicros and
daysOutsideRetention, which is where dropped ledger rows, unattributable spend and a window
reaching past retention are disclosed. agent is present only when an agent= filter was
applied, which is how a client tells a server that ignored the parameter. Do not code against a
closed field list.
pricedBy, unpricedBy and incompleteBy are absent on a ledger-backed window even when
pricing gaps exist — a per-minute row cannot say which requests could not be priced, and
emitting one map without the other would read as "no gaps here". Read totals.pricedRequests
against totals.priceableRequests for that instead. pricedBy is keyed by provenance —
bundled, discovered, configured, authoritative, plus unlabelled for a producer that
settled a figure without naming its level. Nothing produces discovered today; it is a defined
level kept so that anything which later learns rates from a gateway has a slot in the precedence
order. unpricedBy is keyed by <endpoint> <model>.
The response reports the group it SERVED, not the one you asked for — and the
difference is silent. On a ledger-backed window (today, month, 7d) only
endpoint, agent, model and currency are actually grouped; host, session, status
and plugin fall back to group: "none" with no error and HTTP 200. With agent=, a window
served from the ring — a duration, or any window without a ledger — falls back to none for every
grouping. Always read the group field back:
curl -s 'localhost:47601/v1/usage?window=today&group=endpoint' | jq .group
# "endpoint" <- honoured
curl -s 'localhost:47601/v1/usage?window=today&group=host' | jq .group
# "none" <- silently degradedTwo aggregates answer the same question and may disagree slightly, which is not a bug
and is not spend: the ring prices a request nothing else priced from the rate table,
while the ledger reports it unpriced instead. agentop cost --help documents the window
semantics in more depth than this page, including why the two must never be subtracted
from each other.
https://platform.claude.com/docs/en/about-claude/pricing (also linked as https://claude.com/pricing). There is no rate-card API; the page is the source.
Its columns map onto Cortex's tiers:
| Anthropic column | Cortex key |
|---|---|
| Base input tokens | input_cost_per_million |
| 5m cache writes | cache_write_cost_per_million |
| Cache hits and refreshes | cache_read_cost_per_million |
| Output tokens | output_cost_per_million |
Use the 5-minute cache-write column. Cortex does not model the 1-hour write premium.
Sanity-check a row you typed by hand — for most Claude models the published ratios are cache-write = 1.25x input and cache-read = 0.1x input. A row that does not sit near those ratios is usually a transcription slip. (Fable 5.1 and Mythos 5.1 read at 0.025x, and some newer Opus models at 0.05x, so those are legitimately different.)
https://developers.openai.com/api/docs/pricing. The flagship text table splits short and long context:
| OpenAI column | Cortex key |
|---|---|
| Short context input | input_cost_per_million |
| Short context cache writes | cache_write_cost_per_million |
| Short context cached input | cache_read_cost_per_million |
| Short context output | output_cost_per_million |
| Long context * | an above: block — see Long-context tiers |
Two routes, neither of which Cortex does for you:
The public price map — the same file the bundled table is generated from:
curl -sS https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json \
| jq '.["gpt-5"] | {input_cost_per_token, output_cost_per_token, cache_read_input_token_cost}'Those are per token, so either use the _per_token keys in config or multiply by 10^6.
Your own gateway's /model/info — what your deployment actually has configured,
including operator overrides and alias resolution:
curl -sS -H "Authorization: Bearer $LITELLM_KEY" https://<gateway>/model/info \
| jq '.data[] | {model_name, input: .model_info.input_cost_per_token,
output: .model_info.output_cost_per_token}'Requires a virtual key. Be aware some gateways return zeros here while still billing — they apply pricing in middleware rather than in model config. Zeros mean "this endpoint will not tell you", not "free".
pricing:
bundled: true # default. false = price ONLY what you configure,
# which also drops the shipped discounts
endpoints:
- hosts: ["gw.internal", "gw-alt.internal"] # host globs, port stripped.
# "*" or omitted = any endpoint
models:
"*claude-opus-*": # model glob, matched case-insensitively
input_cost_per_million: 3.80
cache_write_cost_per_million: 4.75
cache_read_cost_per_million: 0.38
output_cost_per_million: 19.00
above: # optional long-context override
- prompt_tokens: 200000
input_cost_per_million: 7.60hosts is a list because gateways commonly share a rate card — two replicas, or a
service name and its external alias, bill identically, and repeating the models block per
host invites the copies to drift. Each host becomes its own table row.
Rates are in US dollars unless an endpoint says otherwise. This is the entry the built-in local config ships for IBM Bob, a flat rate on every model and tier:
pricing:
endpoints:
- hosts: ["api.us-east.bob.ibm.com"]
unit: Bobcoins # absent means USD
models:
"*":
input_cost_per_million: 2.00
output_cost_per_million: 2.00
cache_read_cost_per_million: 2.00 # deliberate: this gateway gives no cache discount
cache_write_cost_per_million: 2.00unit sits on the endpoint, because that is where a gateway's billing is decided, and it
covers every request to that endpoint: only rates in its unit can price its traffic, so a model no
block in that unit names is left unpriced there rather than priced from the dollar table. A figure
the gateway reports itself is labelled with the same unit.
A unit needs a models block — a block with only a multiplier has no rates for it to apply to
— and it needs hosts: a block with none, or with "*" among them, covers every endpoint, so it
would put every endpoint no other block names in that unit and leave the dollar table pricing
nothing there. Two blocks naming the same host must also agree on its unit. All three fail
startup otherwise.
Figures in different units are never added, on a window served from the ledger or from the
in-memory ring. Where such a window holds more than one, agentop cost withholds the combined total, names the units it found, and
points you at --by currency:
COST — today
2 units 1057 requests 298M tokens
! this window holds USD and credits, which cannot be added — no combined figure is shown
use --by currency for a figure per unit; tokens and requests above are unit-free
A caveat printed under a wrong number leaves the wrong number on screen, so the figure is withheld rather than annotated. Tokens and requests are still reported: they carry no unit and stay comparable across gateways.
--by currency is then the axis that can show figures, because each row names its own unit:
CURRENCY REQUESTS TOKENS COST
USD 1000 0 $146.36
credits 57 0 0.08 credits
On any other axis a mixed window withholds the cost cells instead — one agent calling two gateways
is a single row whose figure would be the cross-unit sum, and nothing in a per-axis breakdown can
separate it. A window in one non-USD unit is not mixed and prints its total normally, labelled
in that unit rather than behind a $.
agentop observe follows the same rules on every money figure it draws: a unit is shown beside the
figure it belongs to, a span holding several units prints each one's figure side by side and never
their sum, and a row that itself spans units reads (mixed). It knows each row's unit because
/v1/usage reports the units of every series for the model, endpoint and agent breakdowns
(seriesCurrencies) and /v1/sessions reports each session's (currencies).
--json carries a currencies array whenever the producer computes one, so a script can make the
same refusal. One entry is not a caveat — it is the answer to "what unit is this total in".
unit names a currency, so it is letters, digits, - and _ only, at most 16 bytes, and an
unusable value fails startup naming the endpoint. It is compared case-insensitively, and stored
as you typed it except for USD: every spelling of the default is stored as USD, because that is
the one unit Cortex has a canonical name for and five separate consumers test a figure against it
to decide whether it may be labelled $. Any other unit keeps your spelling, so
agentop pricing --host <host> shows back what you typed, beside each row's provenance. The default
agentop pricing table has no unit column and does not show it — the rates endpoint sends it either
way.
A multiplier never crosses units — scaling credits by 0.76 leaves credits.
Nothing converts between units. There are no exchange rates here: a unit partitions figures, it is never an operand.
Both forms are accepted per tier: *_cost_per_million or *_cost_per_token. Setting
both for one tier fails startup, naming the tier — they differ by 10^6, and
silently picking a winner would misprice by that factor with nothing in the readout to say
which was honoured.
unpricedBy said api.example.com gpt-5-mini. Read the vendor page, then:
pricing:
endpoints:
- hosts: ["api.example.com"]
models:
"gpt-5-mini":
input_cost_per_million: 0.25
cache_read_cost_per_million: 0.025
output_cost_per_million: 2.00Confirm with agentop pricing --host api.example.com — the row should read configured.
Then re-check unpricedBy is empty for that pair.
Within one provenance level, an exact model key beats a glob, and a longer glob beats a shorter one.
The trap: the family globs will happily price a model released after the pinned LiteLLM
commit — at the family's older rate. If a vendor launches a cheaper model in an existing
family, the glob overstates it, and nothing looks wrong. For example, a new Opus
priced at 4 / 5 / 0.2 / 20 would be matched by *claude-*opus-* and billed at
5 / 6.25 / 0.5 / 25.
On a local install the downloaded list closes this for any model LiteLLM has listed, which
for claude-opus-5-5 was a week before its first request reached the proxy. What remains is
the gap between a launch and LiteLLM listing it, and every install that is not local. For
those, two ways out: refresh the bundled table, or pin that one model:
pricing:
endpoints:
- hosts: ["*"]
models:
"claude-opus-5-5": # exact key beats the family glob
input_cost_per_million: 4.00
cache_write_cost_per_million: 5.00
cache_read_cost_per_million: 0.20
output_cost_per_million: 20.00Compare agentop pricing against the vendor page after a model launch; that is the only
check that catches this.
above: lists breakpoints by prompt size. A request past the breakpoint is charged the
rates in that block; tiers you omit keep their base value.
"claude-sonnet-4-5":
input_cost_per_million: 3.00
output_cost_per_million: 15.00
above:
- prompt_tokens: 200000
input_cost_per_million: 6.00
output_cost_per_million: 22.50agentop pricing renders these as an indented above 200,000 tok: line beneath the model
— or above 272,000 tok: on an OpenAI one, since the breakpoint is the vendor's. Its
footnote names every breakpoint in the table it just printed.
Most gateways bill a uniform fraction of list, and for those the whole answer is one scalar:
pricing:
endpoints:
- hosts: ["my-gateway.example.com"]
multiplier: 0.76 # a FRACTION of list, so 0.76 is a 24% discountOne number rather than twelve, and it tracks upstream repricing: the gateway's price
is derived from list, so refreshing the bundled table moves both together. A copied rate
card freezes today's numbers and goes stale silently. The factor is capped at 10, because
multiplier: 76 for 0.76 would inflate every figure a hundredfold and reads as
plausible in a config file.
A catch-all you configure beats a specific rule Cortex ships. These are not equivalent:
pricing:
endpoints:
- hosts: ["*"] # applies to EVERY endpoint, including ones with a
multiplier: 0.9 # shipped discount — 0.9 replaces their 0.76Deliberate: a configured factor means an operator checked their bill, and a rule compiled
into a binary should never silently win over that. But a catch-all written for one gateway
quietly reprices the rest. Scope hosts unless you mean every endpoint, and check with
agentop pricing --host <gateway>.
To drop a shipped discount without scaling, set multiplier: 1.0 explicitly. That is
a configured rule of 1.0, which outranks the shipped factor and leaves rates at list.
Omitting multiplier does not do this — it leaves the shipped rule in force.
You do not have to be told it. A LiteLLM gateway reports what it charged, so the factor is one division — send a non-streamed request and compare the header to list:
# --proto '=https' so a mistyped http:// URL fails instead of putting $KEY on the
# wire in cleartext.
curl -sS -D- -o/dev/null --proto '=https' -X POST "$GATEWAY/v1/messages" \
-H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
-d '{"model":"claude-sonnet-4-5","max_tokens":1,
"messages":[{"role":"user","content":"hi"}]}' \
| grep -i x-litellm-response-costDivide the reported cost by what the same token counts would cost at the list rates
agentop pricing shows. Repeat for a couple of models; if the factor is uniform, it is your
multiplier.
It must be non-streamed. A streamed response reports 0 in that header by design, which is why this cannot be learned from live agent traffic — agents stream.
If you pin real per-model rates for a host that also carries a shipped multiplier, the multiplier drops itself:
A multiplier weaker in provenance than the rates it would scale is dropped.
Your pinned figures are already post-discount, so scaling them again would understate spend by the factor — and understating hides cost. A configured multiplier with configured rates still both apply: you asked for both.
Per (endpoint, model, tier), first hit wins:
authoritative— a settled per-request cost from the gateway (X-Litellm-Response-Costor its-Originalvariant). A figure, not a rate.configured— yourpricing:entry. An override is an override.discovered— reserved; no producer ships today.bundled— the compiled-in table.- none — the request is reported unpriced.
Within a level: specificity decides — endpoint axis before model axis, exact before glob, longer glob before shorter.
Before any of that, the endpoint's billing unit is settled by the same order on
the endpoint axis alone, and only rows in that unit are candidates above. So the most specific
block covering a host decides its unit, and a block that omits unit decides USD: a
*.bob.ibm.com block in credits beside an api.us-east.bob.ibm.com block with no unit makes
that one host USD, and leaves the credits block's models unpriced there. Two patterns that rank
equally on the endpoint axis, such as *.bob.ibm.com and api.*.ibm.com, are decided by the
pattern that sorts first — never by a model name.
| Symptom | Cause | Fix |
|---|---|---|
Requests in unpricedBy |
No rate for that (endpoint, model) |
Add a models entry; the key names the pair |
| Totals look ~1.3x high | Gateway bills below list, nothing pinned | Set multiplier |
| Totals look low after pinning rates | Expected — a bundled multiplier will not scale configured rates | Nothing; or set your own multiplier if you meant one |
| A new model prices at its family's old rate | Family glob matched it: LiteLLM has not listed it yet, or this is not a local install | Pin the exact model id, or refresh the bundled table |
| Startup fails naming a tier | Both _per_million and _per_token set for it |
Remove one |
| Startup fails on a multiplier | Factor above 10 | You probably meant a fraction: 0.76, not 76 |
| A rate change had no effect | Another entry outranks it | agentop pricing --host <gateway> and read provenance |
# find the latest commit touching the price map
curl -sS 'https://api.github.com/repos/BerriAI/litellm/commits?path=model_prices_and_context_window.json&per_page=1' \
| jq -r '.[0].sha'
make pricing-table COMMIT=<sha> # add NO_PROXY_FOR_GEN=1 if your proxy blocks githubThis rewrites core/cost/pricing/bundled.go and the test snapshot, and reruns
TestBundled. It does not touch bundled_multipliers.go — the discounts are measured
against live gateways rather than generated, and are kept in a separate file precisely so
the generator cannot clobber them.
A local install does not need this: it downloads the list itself (see The bundled rate table). Refreshing still matters for the sidecar images, and for a local install's first start with no network.
Asking the gateway for its own rates is a different matter. Via LiteLLM's GET /model/info it was
designed, prototyped and dropped: it needed a virtual key minted and mounted, an outbound
dependency and a refresh loop, to save transcribing three numbers. discovered remains in
the resolution order with no producer so any later mechanism has a defined precedence.
Every priced request publishes one record on its response session event, under the key
cost. One producer, one record, one number.
The key names the concern, not the producer. It used to be the producing plugin's name
(litellm-budget-track), which made moving costing to the component that actually knows the
token counts a breaking wire change — for live consumers and for every event already in a
session store. The framework set that precedent itself: pipeline/context.go publishes
body-mutation from the core, "because a switch of plugin names in a future refactor
shouldn't break operators' dashboards". The legacy key is still written and still read, and
comes out a release after the rename ships.
| field | meaning |
|---|---|
cost_usd |
what the whole exchange cost — prompt and completion together, unlike the two halves below. source says whether the gateway reported it (gateway-header) or it was modelled from token counts (usage-fallback); provenance says how much to trust the rates behind a modelled figure |
settled |
a figure exists — including a deliberate zero, which is the gateway saying the call was free. Without it, "free" and "nobody priced this" were indistinguishable, and a cache hit got re-priced from the rate table |
prompt_usd |
the modelled cost of the prompt alone, tier-weighted. A breakdown, not a component of a sum: it is the table's figure even when cost_usd is the gateway's |
output_usd |
the modelled cost of the generated tokens alone, at the output rate, resolved at the request's prompt size so a long-context premium reaches the completion too. The other half of the same breakdown, so a per-row UI can show what a response row cost rather than what the exchange cost. Priced from the table, never cost_usd − prompt_usd: that difference mixes a possibly-authoritative total with a modelled half, so it reports a gateway-vs-table delta as the completion's cost and can go negative. (modelled − prompt_usd would be sound; mixing sources is the problem, not subtraction.) prompt_usd + output_usd equals the modelled total to the micro |
avoided[] |
cost that was not incurred, attributed per component, with tokensAvoided, usd, the tier it came out of, and two honesty flags: estimated (derived from a byte ratio, not a tokenizer) and projected (measured but never applied — that money was spent) |
daily_total_usd, daily_max_usd |
added by litellm-budget-track when it is in the pipeline; the budget's business, not the cost owner's |
Nothing in avoided is spend. No consumer may add it to a cost, a budget or a usage
total; a test in core/cost/usage asserts the aggregator's totals are unchanged by its
presence. It is a container rather than a few flat fields because more counterfactuals are
coming — compaction, redaction, "what a cheaper model would have cost" — and as siblings of
cost_usd the record would become half-real and half-hypothetical, which is how someone
eventually sums two fields that must never be summed.
plugin-catalog.md— the plugins that consume these rates:inference-parser,litellm-budget-track,tool-prune.litellm-budget-track-plugin.md— daily caps, and the drift check that compares the rate table against what the gateway charged.superpowers/specs/2026-09-09-pricing-consolidation-design.md— why rates live in one place, kept for the reasoning rather than as current docs.