Skip to content

docs/alerts nits from #639 and #640 reviews #124

Open

Description

@sre-helmcode

Non-blocking follow-ups from the reviews of nan-devops#639 (Novita Gatus probe) and #640 (live-alias-gone alert):

  1. #639 Gatus rationale is wrong about who gets 1010. It says "no UA / Python-urllib → 1010". Re-measured: only Python-urllib gets Cloudflare 1010. No UA, curl, python-requests and Go all get INVALID_API_KEY. Fix the values comment, the test docstring and the PR text.
  2. #639 runbook, enterprise primary. deepseek-upstream-outage.md points at the enterprise Cluster name litellm-enterprise-pg-1, while the primary pod is litellm-enterprise-pg-1-1. Say so explicitly.
  3. #639 runbook, kubeconfig. litellm-community-model-400-rate.md has a new PRIMARY=$(kubectl get clusters…) block without --kubeconfig. Use $KC, the absolute path.
  4. #639 runbook, probe traffic. Note that ~2,880 invalid-key probe requests a day go to Novita from our egress. If that is ever flagged, the probe itself goes red.
  5. #640 blind spot. The rule only sees names still on a team's model list. Removing a name from nan-eu or nan-eu-premium produces a 401 Exception with no requested_model label, and the rule is blind to it. Document the mechanism in the runbook, or add a rule for it.
  6. #640 requested_model="None" bucket. Large recurring bursts come from real clients there (311 in 30 min on 09-29 from 10+ keys). Find what produces None, then either cover it or soften runbook step 3.
  7. #640 KUBERNETES.md wording. It says the rule "pages". It posts to Slack like every other alert.
  8. #640 finding on litellm-community-model-400-rate comments. Junk model names now arrive as exception_class="Exception" (a team-list 401 before routing), not as ProxyModelNotFoundError. Fix that rule's comment and runbook.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions