Repository navigation
usage-hook: successful gated calls occasionally arrive with no user_api_key_user_id and go undebited #20
Description
What
nan_usage_untracked_total{cause="no_member_id"} keeps moving on litellm-community: 41 in 7 days, and still ~16 per 20h after PR helmcode/nan-devops#113. Every increment is a successful call to a capped, resold model that nobody was debited for.
The log line is "<outcome> event missing user_api_key_user_id; skipping" (usage_hook.py, _track).
Why it is odd
The pre-call gate already refuses any key it cannot resolve to a member (no_discord_id_on_key -> 402). So these requests passed the gate, which means the member WAS resolved before the call, and then the post-call logging event arrived without user_api_key_user_id in its metadata. It is a metadata-propagation gap inside LiteLLM, not an orphan key.
Consequence: the member consumed a per-token model we pay a third party for, their monthly counter did not move, and their remaining allowance is overstated by exactly that much.
What is known
- Only ever
deepseek-v4-flash. Never glm5.2, never mimo-v2.5. That asymmetry is the best lead. deepseek-v4-flashis the one gated group with multiple deployments/providers, andlitellm_deployment_successful_fallbacks_totalshows ~1,138 fallbacks withexception_status=404in 20h. Fallbacks rebuilding metadata is the first hypothesis to test, though the volumes do not match (1,138 vs ~16), so it would be a specific sub-case.- Synthetic probe keys are already exempted (
_is_synthetic_key), so this is not the Gatus monitor. - Volume is small: roughly 2 to 16 a day against ~110k requests/day to that model.
Suggested next step
Reproduce against litellm-test by forcing a fallback on a gated group and inspecting the kwargs metadata the failure/success hook receives. If the id is genuinely absent on the fallback path, the fix is to carry the pre-call-resolved id forward (the hook already mints a request id in _mint_request_id; the member id could ride the same way) rather than re-reading it from metadata post-call.
Related
- helmcode/nan-devops#113 removed the other untracked cause:
platform_api_unreachablewent from 30 in 7 days to 0 in 20h.
Activity
sre-helmcode commentedon Sep 11, 2026
Closing: both leaks this issue named are silent, and the debiting they were bypassing is
demonstrably working. Measured 2026-09-11.
The counter stopped, per model
nan_usage_untracked_total{cause="no_member_id"}, last instant the value ROSE (40d rangequery against VictoriaMetrics, 1h step):
no_member_id_on_abort, the cause #224 split out of this one, has never incremented at all.Which is not, by itself, evidence
Under multiprocess collection a counter that never increments is ABSENT from
/metricsrather than zero, so "no series" is exactly what a switched-off instrument looks like. Three
controls:
The metric family is live and moving. Same metric, other causes, same 30d window:
failure_no_usageat 156,277 on deepseek-v4-flash and 144,446 on qwen3.8-flash;platform_api_unreachablepresent on six model groups. The exporter is on and thisseries is being scraped.
The code that would increment it still exists —
cause="no_member_id"is stillemitted from
_trackinusage_hook.py. The cause was not deleted or renamed out fromunder the query.
The debiting is actually happening, which is the thing the counter existed to warn
about.
member_token_usage_log, last 24h:Both of the leaking model groups are being metered right now, per member, seconds ago.
What closed each one
The issue guessed right about the first: "Fallbacks rebuilding metadata is the first
hypothesis to test". nan-devops#155 (2026-08-17 09:08, "recognise synthetic probes through
the router fallback path") is the change adjacent to deepseek-v4-flash going quiet on
2026-08-18 — the gap is consistent with the rollout, since the hook mounts via
subPathandonly takes effect on pod replacement.
The second was NOT the fallback path and was not visible when this issue was written.
nan-devops#225 (2026-09-01 11:43) found that LiteLLM 1.83.14 does not propagate
user_api_key_user_idinto the logging event for the DashScope provider, which servesqwen3.8-flash, while the OpenAI-compatible path used by the other gated models does. It
stamps the gate-resolved id into a non-
user_api_key_*key that survives the strip.qwen3.8-flash last incremented at 11:00 that day, 43 minutes before that merge.
That also corrects this issue: it was never "only ever deepseek-v4-flash". Over the last
30 days the split is qwen3.8-flash 2,016 against deepseek-v4-flash 58 — the model that was
not yet gated when this was filed turned out to carry ~97% of the leak.
Residual, deliberately not chased
Absence over 10 and 24 days is not proof of impossibility, so the detector stays: the
nan-usage-hook-untrackedrule still fires onrate(...) > 0, andno_member_idremains adistinct cause so a return is attributable rather than buried in an aggregate.
Not closed by this and tracked separately: the size of
failure_no_usage(156k on one modelin 30 days) is a different cause with a different meaning and is not a debiting leak.