LiteLLM: metadata.previous_models leaks one member's credentials into another member's SpendLogs row #45

The mechanism

Router.previous_models is an instance attribute, not per-request state:

# router.py:579
self.previous_models: List = ...

log_retry appends to it and then attaches the shared list to the current request's metadata:

# router.py:6429-6434
if len(self.previous_models) > 3:
    self.previous_models.pop(0)
self.previous_models.append(previous_model)
kwargs[_metadata_var]["previous_models"] = self.previous_models

So every request served by that pod after a fallback carries the last ≤4 fallback events from whichever members happened to trigger them — and each entry is built with previous_model[k] = v over the failed attempt's kwargs, which includes the header dicts.

Measured

Community primary, 2026-09-09, one scan of a 20-minute window (7,105 rows):

rows with metadata.previous_models carrying `bearer sk-`   11
model_groups affected: qwen3.6, deepseek-v4-flash-fallback   (i.e. our own fallback chain)

A reviewer's independent 2-minute window found 4 rows, 4/4 carrying a token.

The affected groups are exactly the ones with a configured fallback, which is consistent with the mechanism rather than coincidental.

Why this is worse than #43

#43 is "the row contains the requester's own credential". That is bad, but the blast radius is the operator with DB access.

This is cross-member: member A's row can contain member B's live API key. And GET /spend/logs/ui/{request_id} returns proxy_server_request in full to the owner of the row (spend_management_endpoints.py:2109-2196, ownership checked on the row at :3560) — the ownership check protects the row, not the foreign credential inside it. /spend/logs/ui is in spend_tracking_routes (_types.py:581-591).

Exploitability is NOT confirmed and should be before this is prioritised: it depends on whether a community member key is permitted that route in our configuration. I have not tested it, because doing so needs a real member key and per feedback_e2e_test_pollutes_real_account that is not something to do casually. Confirm with a purpose-made key, not a member's.

Not closed by nan-devops#290

That PR scrubs the credential from secret_fields.raw_headers and proxy_server_request["headers"] in the pre-call hook, in place. Two reviewers argue it probably closes this one as a side effect — the scrub runs before the router attempts anything, log_retry copies by reference, so the entry should point at the already-cleaned dict. That is an argument, not a measurement. #290 states it as open for exactly that reason.

Verification after #290 deploys (this is the task):

1. Force a fallback on community (the deepseek-v4-flash → deepseek-v4-flash-fallback chain, or glm5.3 → qwen3.6) with a purpose-made key.
2. Query, NULL-safe, projecting nothing but counts:

   SELECT count(*) FILTER (WHERE "proxy_server_request"->'metadata'->'previous_models' IS NOT NULL) AS with_pm,
          count(*) FILTER (WHERE ("proxy_server_request"->'metadata'->'previous_models')::text ILIKE '%bearer sk-%') AS leaking
   FROM "LiteLLM_SpendLogs" WHERE "startTime" > now() - interval '15 minutes';

3. If leaking > 0, the scrub does not reach it and this needs its own fix — most likely scrubbing metadata.previous_models in the post-call path, or asking upstream why a Router instance attribute is attached to per-request metadata at all.

Regardless of the above

The shared-list design is worth reporting upstream on its own merits: previous_models mixes requests from different callers into one another's log rows even when there is no credential in them (prompts, model choices, timing). That is a tenancy leak in a multi-tenant proxy.

Evidence trail

• nan-devops#290 and its review threads
• LiteLLM community: member API keys are stored in cleartext in SpendLogs (secret_fields.raw_headers) #43 (the same column, single-member scope)
• litellm-community: the SpendLogs pruner is not running — retention window grows monotonically #44 (why the rows persist longer than configured)