Repository navigation
router_settings: model_group_alias and default_fallbacks can serve paid inference unmetered, and no invariant reads them #77
Description
Found by the QA reviewer of helmcode/nan-devops#358, verified against the LiteLLM build running in prod. Neither is a regression from that PR: both are open on main today. They are filed here rather than fixed there because #358 touches router_settings.fallbacks and these live on two other surfaces.
1. model_group_alias overrides a real model group of the same name
Router._common_checks_available_deployment resolves _get_model_from_alias(model=model) (router.py:10465) before consulting self.model_names. So an alias wins over a real group with the same name:
model_group_alias: {qwen3.6: "glm5.3"}serves glm5.3 deployments to anyone asking for qwen3.6.
Why that is a metering hole, not just a routing surprise: qwen3.6 is not gated, so async_pre_call_hook returns at if model not in GATED_MODELS: return data before it extracts any identity or calls /api/internal/usage/check. The member is admitted with no cap and no entitlement check, then served a premium paid group. And async_function_with_fallbacks takes model_group = kwargs.get("model"), the un-aliased name, while _update_kwargs_before_fallbacks stamps metadata.model_group with the original, so the meter still sees qwen3.6.
Unlike the fallback path, this needs no failover at all. It is the normal path.
model_group_alias is a literal sibling of fallbacks: in the same router_settings block and is already populated in both tenants (glm5.2: "glm5.3"). Every invariant added in #358 reads only router_settings.fallbacks.
Suggested guard: reject any alias whose KEY is in _ZERO_COST_LOCAL, and require that the TARGET of an alias whose key is gated be canonicalised or free, with the same rule the existing _unsafe loop uses.
2. default_fallbacks materialises the "*" entry that #358 refuses
Router.__init__ (594-599):
if default_fallbacks is not None or litellm.default_fallbacks is not None:
_fallbacks = default_fallbacks or litellm.default_fallbacks
if self.fallbacks is not None:
self.fallbacks.append({"*": _fallbacks})So default_fallbacks: ["glm5.3"] produces exactly the "*" catch-all that #358's _bad_shape invariant rejects, by a supported and documented key the suite cannot see because it only reads fallbacks.
It is worse than a hand-written "*". _has_default_fallbacks() / _get_first_default_fallback() are consulted inside _common_checks_available_deployment (10519-10530): when the requested group has zero healthy deployments, the router reassigns model = fallback_model there, during deployment selection. That path never goes through run_async_fallback, so there is no model_group re-stamp and the swap is invisible to the meter. Concretely: a vLLM outage leaves qwen3.6 with no healthy deployments and a paid default gets served, metered as qwen3.6.
context_window_fallbacks and content_policy_fallbacks are the same class. Both were probed and pass green today. They resolve through _get_fallback_model_group_from_fallbacks ahead of fallbacks for their two error classes (router.py:6221-6278), and context_window_fallbacks does not even go through validate_fallbacks.
Neither is configured in either tenant right now.
Suggested guard: refuse default_fallbacks, context_window_fallbacks and content_policy_fallbacks in router_settings of both values files by shape, or run them through the same invariants as fallbacks.
Why this is its own issue
The invariants in #358 have now been through eight review rounds and nine distinct routes, every one of them the same root cause: the test suite resolved or assumed something the router does not. Two reviewers independently observed that the block is outgrowing its home (281 lines under a single heading, checks stacked far from the rationale that justifies them).
The right shape for this is probably not more invariants bolted onto one test file, but a single router_settings linter that parses both values files and models the router's resolution once, tested against the installed wheel. That is a design decision worth making deliberately rather than in the tail of a three-line config fix.
Related
- helmcode/nan-devops#358
- usage hook: partial paid tokens go unmetered when a stream is rescued mid-answer by a free rung #76 (the metering gap the same review surfaced)
Activity
sre-helmcode commentedon Sep 15, 2026
Two additions from the final QA round on #358.
A third route on the same surface: an UNGATED primary pointed at a paid group
Found by QA, verified green in both tenants, and pre-existing rather than a regression:
_unsafe(the "canonicalised or free" invariant) skips any primary not in that tenant's gated set, and_free_rung_keyedonly covers_ZERO_COST_LOCAL = {"qwen3.6"}. Butwhisperandgemma4are live ungated primaries in the enterprisefallbackslist today, and both run on our own fleet: free-local exactly like qwen3.6, just not named in that set.Changing
gemma4: ["qwen3.6"]togemma4: ["glm5.3"]is a one-word edit that admits a member with no pre-call gate at all, because an ungated model returns atif model not in GATED_MODELSbefore any cap or entitlement check, and then serves them a SOTA fail-closed group on failover. It is the hazard_free_rung_keyed's own comment describes, one hardcoded name short of covering it.QA verified that
whisper: ["whisper", "glm5.3"]IS caught, so the gap is specifically the ungated-primary premise, not the self-retry exemption. The-fallback-suffixed ungated primaries are fine, canonicalisation covers them.Likely shape of the fix, for whoever picks this up:
_ZERO_COST_LOCALshould be derived from what actually runs on our fleet rather than being a hardcoded single name, and the ungated-primary case needs its own rule instead of being skipped.Line citations in this issue have drifted
The mechanics are right but the numbers came from a different wheel than the one serving prod (v1.83.14). Corrected by QA:
_get_model_from_aliasis called at router.py:9367, not 10465default_fallbacksappend is at router.py:554-559, not 594-599The quoted
__init__snippet is verbatim-correct, so only the numbers need refreshing. Same drift bit #358 twice, so worth re-checking every pointer against the running image before acting on this issue.sre-helmcode commentedon Sep 15, 2026
Two more from the final UX round on #358, both comment-level and deferred here rather than fixed there:
A false claim still stands unmarked, eight lines above its own retraction. In
tests/usage-hook/test_litellm_community_usage_hook.py, theEXEMPTED BY NAME, NOT BY SHAPEparagraph still says enterprise'swhisper: ["whisper"]"is not a no-op: it lets the group retry across its OWN deployments after num_retries is spent". That reading was measured false and retracted inlitellm-enterprise/values.yaml, but a reader who stops at this paragraph takes away the old one. Pre-existing, and the retraction is close by, but the point of retracting in place is that it should not be possible to read the claim without the correction.One sentence overstates a hazard. "Add a second name here believing the old sentence, and you create a real dead end that the exemption would wave through" is stronger than what the suite actually allows. UX simulated it against the real fallbacks:
whisper: ["whisper", "whisper-openai"](unkeyed target) IS caught as a dead end, andwhisper: ["whisper", "glm5.3-fallback"]is genuinely safe. What the exemption really skips is the terminal-rung check on the whisper group itself. That is the accurate hazard and it should be stated as such.Both are in the same block the issue already proposes to replace with a proper
router_settingslinter, so they are worth folding into that work rather than patching in isolation.sre-helmcode commentedon Sep 15, 2026
Closed by helmcode/nan-devops#361, which replaced the invariants with a router_settings linter that models LiteLLM's resolution once. Both routes reported here are covered and mutation-tested: model_group_alias (shadowing the terminal rung, and sending an un-gated name to a gated group) and default_fallbacks with context_window_/content_policy_fallbacks. The third route from the comment thread, an ungated primary pointed at a gated group, is covered too. Remaining comment polish: #78.