Repository navigation
litellm-test usage-hook ConfigMap is at 98.1% of the annotation cap and will break the next hook edit #42
Description
What breaks, and when
k8s/system/litellm-test-usage-hook-cm.yaml is applied client-side by hand from a runbook, which writes the whole object into the kubectl.kubernetes.io/last-applied-configuration annotation. Annotations are capped at 262144 bytes.
Measured 2026-09-08 after the nan-devops#278 deploy, counting the way Kubernetes counts (sum of key+value bytes, not the escaped JSON):
| ConfigMap | data | annotations | % of cap | headroom |
|---|---|---|---|---|
litellm-test-usage-hook |
248,706 B | 257,270 B | 98.1% | 4,874 B |
litellm-community-usage-hook |
260,182 B | 121 B | 0.0% | 262,023 B |
litellm-enterprise-usage-hook |
188,428 B | 195,572 B | 74.6% | 66,572 B |
Community is the one that already hit the wall and was fixed (server-side apply + annotation removal, nan-devops#284). Test is next, and it has ~4.8 KB of room.
ConfigMap "litellm-community-usage-hook" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes
And the next edit is not optional. The generator asserts that the community and test bodies are byte-identical outside the gated-set markers, so any change to scripts/litellm-usage-hook/usage_hook.py lands in both. The community copy is now safe; the test copy is not, and it is the one that will fail.
For scale: the change that broke community added ~4 KB. Test has ~4.8 KB.
Why it is worse than community's was
Community's was applied by ArgoCD, so the failure surfaced as a stuck app with a real error message that someone would eventually notice. This one is applied by a human from a runbook, so it surfaces as a command that errors in someone's terminal and may simply not be retried — and litellm-test is where changes are supposed to be validated before prod. A silently-stale test proxy is worse than no test proxy.
The test ConfigMap also carries no argocd.argoproj.io/tracking-id: nothing owns it.
Fix
Two steps, same as nan-devops#284 adapted to a hand-applied object:
kubectl -n litellm-test apply --server-side -f k8s/system/litellm-test-usage-hook-cm.yaml
kubectl -n litellm-test annotate cm litellm-test-usage-hook \
kubectl.kubernetes.io/last-applied-configuration-The second command is the one that actually recovers the budget: server-side apply stops writing the annotation but does not delete the one already there. Verified on this cluster: 26 objects migrated CSA→SSA still carry a stale copy; the 44 born server-side have none. On community those two steps took the annotations from 260,793 B to 121 B.
Then update the k8s/system/README.md entry so the runbook says --server-side.
Worth considering instead: give the test ConfigMap its own ArgoCD app, as was done for the community and enterprise hooks. The reason those apps exist is written in their configs — a hand-applied hook sat 85 days stale in the cluster while git was correct. The test copy is in exactly that position today.
Enterprise: the number in the code is now stale
kubernetes/apps/prod/nan-community/litellm-community/litellm-community-usage-hook/config.yaml records "the sibling enterprise hook is at 166787 B (64% of the cap)". That was true when written; it is now 195,572 B (74.6%), because enterprise is also hand-applied from the same runbook and every manual apply regenerates the annotation at the current data size. Its headroom shrinks with each apply, not just with each code change. The comment should be corrected to the current figure, or better, to say that the number moves and to give the command for checking it:
kubectl get cm -n <ns> <cm> -o json | python3 -c "import json,sys; a=json.load(sys.stdin)['metadata'].get('annotations') or {}; print(sum(len(k.encode())+len(v.encode()) for k,v in a.items()))"Found while deploying nan-devops#278/#284.
Activity
[-]litellm-test usage-hook ConfigMap is at 97.6% of the annotation cap and will break the next hook edit[/-][+]litellm-test usage-hook ConfigMap is at 98.1% of the annotation cap and will break the next hook edit[/+]sre-helmcode commentedon Sep 8, 2026
Correction: it was not "will break". It was already broken, and it is now fixed.
Review caught that the framing was wrong.
litellm-testwas running a hook 8,126 B behind git and could not be updated at all:nan-devops#278 regenerated the test ConfigMap in git, it was committed, and the manual apply step was not merely skipped — it was already impossible. The test proxy has been silently stale since, which is the exact failure mode this issue describes, one environment earlier than predicted.
Fixed
Steps applied, the same two as nan-devops#284 plus the restart:
The second command is what recovers the budget: server-side apply stops writing the annotation but does not remove the existing one.
What remains open
The durable fix, and the reason to keep this issue open:
k8s/system/README.mdstill tells a human to run plainkubectl apply -f, which will fail again the next time the annotation is regenerated. The runbook needs--server-side.Measuring it correctly
Note for whoever picks this up: measure the way Kubernetes does — the sum of key+value bytes, not the escaped JSON representation, which inflates the figure: