Repoint nan-system-dev platform-api at litellm-test and rotate the prod LiteLLM master key #127

Context

nan-system-dev platform-api talks to production community LiteLLM. Its platform-api-secrets (Vault-managed) carry:

• litellm-url: litellm-community.litellm-community.svc.cluster.local:4000 (prod);
• litellm-master-key: the prod master key (same as nan-system);
• litellm-db-url: litellm-community-pg-rw.litellm-community.svc.cluster.local:5432 (prod DB, prod role).

Onboarding a user in dev therefore mints prod LiteLLM keys and writes the prod LiteLLM DB.

helmcode/nan-devops#650 (ingress NetworkPolicies for the LiteLLM namespaces, GHSA-4xpc) deliberately does not allow nan-system-dev on the prod :4000 and :5432 policies. It is allowed only on litellm-test. Once #650 is rolled out, dev loses prod LiteLLM access. That is intended.

Tasks

1. Repoint dev at litellm-test. In Vault, for nan-system-dev/platform-api-secrets:

   • litellm-url -> http://litellm-test.litellm-test.svc.cluster.local:4000;
   • litellm-master-key -> the litellm-test master key;
   • litellm-db-url -> litellm-test-pg-rw.litellm-test.svc.cluster.local:5432, with a litellm-test role;
   • the litellm-team-* / litellm-premium-team-id values -> teams that exist in litellm-test.

   Then let VSO sync, and restart dev platform-api.

2. Rotate the prod community master key, since dev has held it:

   • update nan-system/platform-api-secrets and every other consumer of the prod master key;
   • verify that platform-api key lifecycle, the discord bot's metrics job (nan-eu001, which holds LITELLM_ADMIN_KEY) and gatus keep working;
   • confirm that the old key returns 401.

3. Verify dev's LiteLLM features against litellm-test: onboarding (key creation in the right team), key revoke, usage and spend reads, model listing, and any feature that reads the LiteLLM DB.

Check that no dev-created keys are left in prod LiteLLM. Keys created by dev onboarding carry dev members' user ids or aliases: list them and decide whether to delete.

🤖 Generated with Claude Code