litellm-community: one unknown API key is retrying ~900 times/hour and getting 401 every time #21

Activity

1. sre-helmcode

   assigned

   barckcode

   on Aug 9, 2026
2. sre-helmcode commented on Sep 12, 2026

   

   sre-helmcode

   on Sep 12, 2026

   ContributorAuthor

   More actions

   Still happening, and now attributable. Measured 2026-09-12.

   Volume

   115 responses of 401 against litellm-community in an 8-minute window
   (2026-09-12 10:57:21Z → 11:05:27Z), from Traefik access logs. That is ~860/hour,
   consistent with what the hook logs (~160 per 20 minutes across the 8 pods).

   Origin — answerable without the Cloudflare dashboard

   Traefik logs carry request_Cf-Connecting-Ip and request_User-Agent, so option 1 of this
   issue needs no Cloudflare access:

     45  150.136.217.31        AsyncOpenAI/Python 2.24.0    /v1/chat/completions
     44  51.178.136.79         undici                       /v1/embeddings
     16  78.30.14.225          python-httpx/0.28.1
      5  217.130.105.198
      4  2a01:4f9:c014:2fe9::1
   

   Three sources carry 91% of it. The user agents are real SDKs, the paths are coherent, and
   the rate is steady — this is automated clients retrying dead credentials, not a scanner
   probing for valid keys. Two of the three are hosting providers (Oracle Cloud, OVH), so they
   are agents running somewhere, not someone at a laptop.

   The keys are not ours and never were

     hashes observed in the 401s   -> 0 rows in LiteLLM_VerificationToken, not even soft-deleted
     key suffixes (sk-...2PYA, sk-...6jSA) -> 0 real keys end with them
   

   92fb6c51db26 from the original table is still there a month later; the top offender
   then (684d383f6140) is gone.

   What I could NOT establish

   Whether these are ex-members whose keys were hard-deleted (the 2026-08-20 mass delete
   removed 113 keys, leaving no rows). Nothing in the DB links a fabricated hash back to a
   person, and the key-alias join against members is not sound enough to answer it — I tried:
   67 subscribed members appear to have no key when joined on lower(key_alias) = lower(discord_user), but 21 of the first 60 of those consumed 199,529 metered rows in the
   last 30 days, so they do have working keys and the join simply misses aliases that differ
   from the Discord handle. Recording that here so nobody re-derives the same wrong number:
   that join does not answer this question.

   The log-noise argument is weaker than this issue claimed

   Measured over 20 minutes across all 8 pods:

     ERROR lines      2058   of which auth-related   353  (17%)
     WARNING lines     843   of which our 401 line   161  (19%)
   

   It is real but it does not dominate. The actual top ERROR source is unrelated: 794 lines
   (39%) are LiteLLM Redis Caching: async set() - Got exception from REDIS, the same Valkey
   timeout family as #51. If the goal is a readable ERROR stream, #51 buys more than this does.

   (An earlier draft of this comment said auth was 0% of ERROR. That was a broken grep, not a
   measurement.)

   Recommendation

   Option 2, bounded: rate-limit repeated auth failures per source IP at the edge. It is
   auth failure, so it happens before the hook and before any per-key limit — it has to live in
   Cloudflare or Traefik. Three IPs at ~860/h is well inside what a WAF rule handles, and it
   needs no per-member diagnosis, which is the part that cannot be done from our side.

   Option 1 is closed as far as it can go: we know the origins, and they map to nothing we
   hold. Option 3 (lowering the log level) treats a symptom that is 17% of the problem.

   Not doing either unilaterally — an edge rule is a production change on the request path for
   every caller, and it should be a deliberate call, not a side effect of triage.
3. sre-helmcode commented on Sep 12, 2026

   

   sre-helmcode

   on Sep 12, 2026

   ContributorAuthor

   More actions

   Closing: investigated, attributed, and not worth the change. Owner decision 2026-09-12.

   The measurement above answers the question this issue asked, and the answer is that it does
   not justify touching the edge:

   volume	~860/h, against a proxy serving ~200 rps — 0.1% of traffic
   cost per attempt	one cache miss plus one DB lookup
   member impact	none — the keys never existed, so nobody legitimate is waiting behind them
   log cost	17% of ERROR, not the dominant source this issue assumed

   Acting would mean a rate-limit rule on the request path of every caller, to save 0.1% of
   traffic and 17% of one log stream. That is the wrong trade.

   What the investigation did settle, and is worth keeping:

   • The origins are identifiable from Traefik alone (Cf-Connecting-Ip, User-Agent) — no
     Cloudflare dashboard needed. Three hosting-provider IPs carry 91%, with real SDK user
     agents. Automated clients retrying dead credentials, not a scanner.
   • The lower(key_alias) = lower(discord_user) join does NOT answer "which members have no
     key" — 21 of 60 apparent misses had consumed 199,529 metered rows in 30 days. Recorded so
     nobody re-derives it.
   • The real top ERROR source is rate-limit hook: 1s Valkey socket_timeout strands concurrency slots for up to 660s #51 (Valkey timeouts, 39%), which unlike this one does
     cost members: a stranded concurrency slot burns 1 of their 5 for up to 11 minutes.

   Reopen if the volume changes shape — a jump in distinct source IPs would mean something
   different from three stuck agents, and that is worth a WAF rule.
4. sre-helmcode

   closed this as not plannedon Sep 12, 2026