Feature description
Allow Server operators to configure additional Pydantic AI model settings for the generation workload. This should support provider-specific request-body extensions, for example disabling thinking on a compatible OpenAI-style endpoint:
export POWERCONTEXT_SERVER_INFERENCE_GENERATION_MODEL_SETTINGS='{"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}}'
Problem and proposed solution
PowerContext currently configures only the generation model identifier, timeout, and request limit. Some OpenAI-compatible deployments require additional request parameters that are not part of the portable model identifier. Without a model-settings passthrough, operators cannot use those provider capabilities.
Add an optional JSON object to InferenceConfig and the corresponding Server environment variable. Pass it to every structured generation pipeline and to the generation readiness probe. An empty object must preserve current behavior. Credentials and static authorization headers remain provider configuration and must not be placed in model settings.
Scope
- Support JSON-compatible generation model settings.
- Preserve nested objects such as
extra_body.chat_template_kwargs.
- Apply the same settings to normal generation and readiness probing.
- Document the environment variable in English and Chinese.
This is a focused model-settings slice related to #1379; that issue also covers workload-specific endpoints, headers, embeddings, and reranking.
Alternatives considered
Adding an enable_thinking field directly would couple PowerContext to one provider convention and would not cover other model-specific parameters. A generic, bounded JSON settings object keeps the provider boundary in Pydantic AI.
Feature description
Allow Server operators to configure additional Pydantic AI model settings for the generation workload. This should support provider-specific request-body extensions, for example disabling thinking on a compatible OpenAI-style endpoint:
Problem and proposed solution
PowerContext currently configures only the generation model identifier, timeout, and request limit. Some OpenAI-compatible deployments require additional request parameters that are not part of the portable model identifier. Without a model-settings passthrough, operators cannot use those provider capabilities.
Add an optional JSON object to
InferenceConfigand the corresponding Server environment variable. Pass it to every structured generation pipeline and to the generation readiness probe. An empty object must preserve current behavior. Credentials and static authorization headers remain provider configuration and must not be placed in model settings.Scope
extra_body.chat_template_kwargs.This is a focused model-settings slice related to #1379; that issue also covers workload-specific endpoints, headers, embeddings, and reranking.
Alternatives considered
Adding an
enable_thinkingfield directly would couple PowerContext to one provider convention and would not cover other model-specific parameters. A generic, bounded JSON settings object keeps the provider boundary in Pydantic AI.