fix(llm): no-cloud extraction — redirect ADK agents off Gemini + keep JSON mode on vLLM/Qwen (RES-944)

[jhkchan]

RES-944 has two root causes that together kept extraction on cloud Gemini and producing 0 facts on the no-cloud path. This PR fixes both. All changes are gated behind default-off flags / opt-in params → zero behaviour change for existing cloud installs.

Part 1 — config could not redirect the ADK agents (primary ticket symptom)

Even with LLM_FAST_MODEL=openai/vllm-qwen (and Settings() confirming it loaded), extraction still emitted google.genai 400s. LLMProvider.resolve_model resolves an agent's model as Assignment → DEFAULT_AGENT_MODELS → llm_fast_model, and DEFAULT_AGENT_MODELS hardcodes gemini-*, so the env var was shadowed and never took effect.

Fix — llm_self_hosted_only (default False):

• Bypasses the hardcoded Gemini default map; every un-assigned agent uses llm_fast_model. Explicit per-agent Assignments still win. Shared resolution extracted into _base_model_string so resolve_model and get_model_string can't drift.
• Routes un-assigned agents through llm_default_endpoint_id for the self-hosted endpoint's base_url + credential.
• Fails closed: resolve_model raises ConfigurationError if an agent still resolves to a cloud model; assert_self_hosted_only() (startup assertion) reports every offender at once — a no-cloud misconfig can never silently leak to Gemini.

Part 2 — JSON mode was dropped once on vLLM (0 facts)

The fact/entity extractors request structured output via response_mime_type="application/json". ADK's native Gemini path honours it, but ADK's LiteLlm wrapper builds response_format only from response_schema and ignores the mime type (verified in adk==2.2.0 _get_completion_inputs). So once an extractor reaches a self-hosted OpenAI-compatible endpoint the request carries no JSON directive → prose → recovery parser finds no fact array → 0 facts.

Fix — JsonAwareLiteLlm: lazily-defined LiteLlm subclass that bakes response_format={"type":"json_object"} into _additional_args (ADK merges those over the computed args, so it wins even when the computed response_format is None). Immutable after construction → concurrency-safe. Threaded via force_json_object through resolve_model_object → resolve_model; the two extractors opt in. No-op for native Gemini. vLLM honours json_object via guided decoding; litellm maps it back to response_mime_type for Gemini.

Tests

• tests/llm/test_self_hosted_only.py (15): _is_cloud_model matrix (incl. openai/gemini shim), default-map-shadows-env (the bug), cloud-mode bypass, explicit-assignment-wins, resolve + assert fail-closed on cloud, pass when all self-hosted, no-op when flag off.
• tests/llm/test_json_aware_litellm.py (7): subclass/lazy-singleton, response_format baked, explicit format wins, vLLM forces json on both gemini-flag paths, plain LiteLlm unforced, native-Gemini no-op.
• 22 passed. Full tests/llm suite otherwise green (the one test_embedding_runtime::test_cache_returns_within_ttl failure is a pre-existing flaky real-clock TTL test — reproduced identically on main with this PR's changes stashed; the test_provider_failover_seam errors are a pre-existing CREDENTIAL_MASTER_KEY-not-set env prerequisite, all pass once the key is exported).

Operational note (not code)

Making live Qwen extraction produce facts>0 also requires: (1) setting llm_self_hosted_only=true + llm_default_endpoint_id (or per-agent Assignments) pointing at vllm.votee.dev/v1; (2) the F2 token cap (RES-945, #336) so the prompt fits Qwen's 64k window. End-to-end validation must run on the POC box against a rebuilt stack, never against prod.

Part of epic RES-943 (RLP full-corpus scale + no-cloud gaps).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PMMM6KQXmzAEA42UxpyMUm

[jhkchan]

Independent adversarial review — the core fix is confirmed correct; one P1 egress hole found and now fixed in this PR.

Verified against the installed ADK (google/adk/models/lite_llm.py): __init__ stores **kwargs in _additional_args and does not pop response_format; generate_content_async does completion_args.update(self._additional_args) (line 2349), so the baked response_format={"type":"json_object"} overrides the computed None. The JSON-mode fix is a real fix, not a no-op. Also confirmed: _base_model_string is behavior-identical when the flag is off, force_json_object is threaded on all three wrap paths, and the per-call guard catches the headline "operator left LLM_FAST_MODEL at gemini" misconfig.

P1 (fixed in f9…/latest commit): the fail-closed guard was Gemini-only. With llm_self_hosted_only=true but a missing/typo'd/unloaded llm_default_endpoint_id, an openai/… model reached litellm with api_base=None → litellm's openai/ provider defaults to https://api.openai.com, silently shipping on-prem document text to OpenAI cloud — a direct violation of the no-cloud invariant. Now the guard also raises ConfigurationError when a resolved openai/ model has no base_url; a loaded self-hosted endpoint passes. +2 tests.

Remaining nits (follow-up, non-blocking):

• [P2] _is_cloud_model uses substring "gemini"/"vertex", so a legit self-hosted openai/gemini-distill-qwen would false-trip. Tighten to provider-prefix + host allowlist.
• [P2] assert_self_hosted_only() isn't wired into startup yet — either call it in the ingest bootstrap or soften the docstring.
• [P2] Add an end-to-end test that create_fact_extractor() yields a JsonAwareLiteLlm (guards against a future edit dropping force_json_object=True at the factory).