I4.0 - Chatbot module

This module adds chatbots to I4.0.

Service configuration parameters

What is modified_by_user?

The modified_by_user parameter allows the user to change this parameter when true (only for one inference). When false, the parameter in the model configuration (or service configuration of not found) will be forced and cannot be changed by the user.

What is default?

It's the default value of the parameter. If the parameter is in the model configuration, it will be overwritten for that model only.

Parameters

temperature

Controls the randomness of generated text by adjusting the probability distribution over possible next tokens.

Lower values will produce more predictable results, higher values will have more creativity.

top_p

Determines which tokens to consider when generating a response, based on cumulative probability.

top_k

Specifies how many of the most likely tokens should be considered when generating a response.

top_n_sigma

Selects tokens whose pre-softmax logits exceed a threshold measured in standard deviations.

min_p

Adjusts the sampling threshold based on the model's confidence.

typical_p

Controls the randomness of the output by adjusting the probability distribution of token selection.

seed

Controls the randomness of the model's output.

Setting this to null or -1 will randomize this parameter for each inference.

presence_penalty

Discourages the model from repeating words or phrases it has already used.

frequency_penalty

Discourages the model from repeating words or phrases by lowering the score of tokens that have already been used.

repeat_penalty

Discourages the model from repeating words or phrases by applying a penalty to tokens that have already appeared.

penalty_last_n

Controls the repetition penalty for the last N tokens generated by the model.

tools

Allows the model to execute tools. Tools can't be specified in the configuration, only disabled/enabled.

tool_choice

Specified which tools the model can utilize during function calling. This parameter can't be changed in the configuration, only enabled/disabled.

tool_start_token

Default tool call start token for the model. When this token is generated by the model, the tool calling will begin.

tool_end_token

Default tool call end token for the model. When this token is generated by the model, the tool calling will stop.

max_length

Maximum generation length.

test_inference_conversation

Conversation that will be tested for the model.

test_inference_configuration

Configuration for the test inference.

allow_assistant_prefill

Allows the usage of the assistant_prefill parameter. This parameter can sometimes crash the program with some models.

channel_start_token

Defines the start token for the channel. Example: <|channel> for Gemma 4.

channel_end_token

Defines the end token for the channel. Example: <channel|> for Gemma 4.

channel_name_end_token

Defines the token that will separate the channel name from the channel content.

Example:

<|channel>thought
thinking process...<channel|>

In this example, the token would be \n and the channel name will be thought.

channel_default

Default channel name.

Recommended to set to the response channel.

channel_null_name

Only used when channel_name_end_token is null, meaning the model only has two channels (usually thinking and response). This will be the name of the thinking channel.

channel_change_token

Token that will be sent to the client when the channel changes.

Example: \n yields "\n" when any channel is changed. {"A": "\n"} yields "\n" when the channel "A" is changed to any other channel. {"A": {"to": "B", "token": "\n"}} yields "\n" when the channel "A" is changed to channel "B". null or an invalid dictionary will ignore this parameter.

replace_roles

Dictionary of roles that must be replaced. For example, {"person": "user"} will set the role of all the messages with the role "person" to "user".

Model parameters

TODO