monitoring: alert when a live community model alias disappears (ProxyModelNotFoundError) #115

Context

nan-devops#617 removed ProxyModelNotFoundError from the numerator of litellm-community-model-400-rate, because members calling a deliberately retired name (mimo-v2.5 hard-cut, #610) and made-up names (gpt-4o-mini, x, test) were firing it.

Accepted gap

A live alias deleted or renamed by mistake answers with the same class, and no alert sees it any more. Models live in the LiteLLM DB, one bad /model/delete away.

Proposal

A dedicated rule: rate of ProxyModelNotFoundError per requested_model and on (requested_model) that name had litellm_deployment_success_responses_total in the previous day.

Trade-off to decide: it also rings for up to a day after every deliberate retirement, unless retirements come with a silence. That is why it did not ride in on the noise fix.

🤖 Generated with Claude Code