Skip to content

feat(inference): support extra generation model settings - #1394

Open
Teingi wants to merge 1 commit into
oceanbase:masterfrom
Teingi:feat/generation-model-settings
Open

feat(inference): support extra generation model settings#1394
Teingi wants to merge 1 commit into
oceanbase:masterfrom
Teingi:feat/generation-model-settings

Conversation

@Teingi

@Teingi Teingi commented Aug 30, 2026

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Closes #1393.

Related to #1379, which covers the broader workload-specific endpoint, header, embedding, and reranking configuration.

Rationale for this change

Some OpenAI-compatible generation endpoints require provider-specific request fields that cannot be expressed in a Pydantic AI model identifier. For example, a compatible Qwen deployment may require chat_template_kwargs.enable_thinking=false.

What changes are included in this PR?

  • Add JSON-compatible generation_model_settings to InferenceConfig and Server environment loading.
  • Forward the settings to extraction, Experience and Skill generation, Handoff generation, optional LLM reranking, and generation readiness.
  • Preserve PowerContext-owned bounds by overriding readiness max_tokens to 1 and reranking temperature to 0.
  • Reject settings without a generation model and keep credentials/static headers at the provider boundary.
  • Document extra_body.chat_template_kwargs.enable_thinking in the environment example and English/Chinese inference documentation.
  • Add focused configuration and request-propagation regression coverage.

Are there any user-facing changes?

Server operators can now set provider-specific generation parameters with one JSON environment variable:

export POWERCONTEXT_SERVER_INFERENCE_GENERATION_MODEL_SETTINGS='{"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}}'

The default remains an empty object, so existing deployments keep their current behavior. Only settings supported by the selected Pydantic AI model/provider have an effect. This setting does not replace provider credential configuration.

How was this change tested?

  • make check
  • make test (953 passed, 9 skipped)
  • make docs-test
  • Focused tests for nested environment parsing, validation, generator settings, and readiness propagation

AI usage statement

OpenAI Codex with GPT-5 was used for implementation, tests, documentation, and review.


provider_model = await resources.enter_async_context(infer_model(settings.generation_model))
model = provider_model if instrumentation is None else InstrumentedModel(provider_model, instrumentation)
model_settings = cast(ModelSettings, dict(settings.generation_model_settings)) or None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we either document the precedence between PowerContext-owned settings and extra_body, or enforce an explicit conflict policy? merge_model_settings only handles top-level keys.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, max_tokens=1 can conflict with extra_body.max_output_tokens=100, leaving the effective value dependent on the provider/SDK merge order.

@PsiACE PsiACE left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totally lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(inference): support extra generation request parameters

2 participants