usage hook: partial paid tokens go unmetered when a stream is rescued mid-answer by a free rung #76

Follow-up required by the Security review of helmcode/nan-devops#358 (approved on the condition that this issue exists before merge).

The gap

PR #358 gives every fallback group a terminal rung on qwen3.6, so a stream that dies after a failover gets rescued instead of returning 502. That is a clear win for the member, but it moves one accounting case from "metered" to "metered nowhere".

Verified in scripts/litellm-usage-hook/usage_hook.py:

• Before #358. The stream dies. async_post_call_streaming_iterator_hook never sets finished, so its finally calls _schedule_stream_abort_track(model=_model_name_request(request_data)). That name is the REQUESTED group, which is gated, so the OpenRouter tokens already streamed are posted against the member's cap. The member gets a 502 and is charged for it.
• After #358. The rescue happens inside LiteLLM's stream wrapper, below the proxy hook. The hook sees one continuous stream ending cleanly, finished becomes True, and the abort meter never fires. async_log_success_event then resolves the model through _model_name_event, gets qwen3.6, which is not in GATED_MODELS, and returns before /track. The partial paid tokens are metered nowhere.

Measured volume on community, 24h to 2026-09-15: 67 requests out of roughly 426,000, i.e. 0.016%, and only the partial tokens before the cut. Enterprise was never separately counted, so its number is unknown rather than small.

Why it was not fixed in #358

The obvious one-liner ("if the served group is un-gated but the requested group is gated, meter against the requested group") would also change long-standing deliberate behaviour: a primary that falls straight to qwen3.6 without burning a single paid token is not metered today, on purpose, because that traffic costs GPU time we already rent. Applying the one-liner would start charging those members, on a volume far larger than 67/day. It would trade a 0.016% visibility gap for over-charging at scale.

Closing this properly needs information the hook does not read today: something that distinguishes "fell to free AFTER burning paid tokens" from "went to free immediately". Candidates are the router's failover trail (previous_models) or the cost on the standard_logging_object.

The part that is a product decision, not an accounting one

Raised by the UX reviewer and worth deciding explicitly rather than inheriting:

Fixing this takes away from the member something #358 gives them. Once the hook can tell the two cases apart, those partial tokens start being charged against the gated cap again, and the member sees consumption for a stream whose model was swapped mid-answer without telling them (the model field changes mid-SSE and nothing else does). Charging for that may well be right, but it should be a deliberate call, not a side effect of the metering fix.

Two sub-questions worth answering at the same time:

1. Should the caller be told a fallback served the request at all? A response header (x-litellm-fallback-served-by or similar) is cheap and turns "the member cannot know" into "the member can know". Today the only signal is the mid-stream model change, which practically no OpenAI-compatible client reads.
2. LiteLLM's continuation prompt is hardcoded English ("You are a helpful assistant... Your response should be in continuation of this text:"). On a Spanish-speaking community that can flip the language of an answer mid-sentence.

Also worth folding in (non-blocking findings from the same review)

• The new invariant in tests/usage-hook/test_litellm_community_usage_hook.py only checks one hop, so a cycle of depth greater than 1 (A: [B] plus B: [A, qwen3.6]) would pass. No current config does this and there is no metering consequence, but walking the chain would close the whole class.
• _SELF_RETRY_OK exempts whisper as a target in any position, not just in its own entry. glm5.3: ["whisper"] would skip that check, though the existing "canonicalised or free" invariant still catches it. Anchoring the exemption to the (primary, target) pair would make it explicit.

Related

• helmcode/nan-devops#358 (the change that exposed this)
• The Novita server overload 429s that pushed the traffic onto the fallback groups in the first place, currently open with the provider.