Skip to content

Optimizes Gemini tool schemas and token usage - #67

Open
noahbclarkson wants to merge 1 commit into
flachesis:mainfrom
noahbclarkson:optimise-function-calling
Open

Optimizes Gemini tool schemas and token usage#67
noahbclarkson wants to merge 1 commit into
flachesis:mainfrom
noahbclarkson:optimise-function-calling

Conversation

@noahbclarkson

Copy link
Copy Markdown
Contributor

Improves compatibility with Gemini by converting unsupported 'const' fields to single-value 'enum' arrays. This is highly important for getting correct responses from the model. The Gemini API does not support the const keyword, which was being sent in the current implementation. This meant that Gemini could not "see" that certain values were const, causing fixed values to be enforced at the prompt level. This occurs most especially for enums with #[serde(tag = "type")]. By switching all const values to an enum with a single field, Gemini can now see that a certain value is fixed as if it were const without explicitly using the const keyword.

Additionally, after some testing I noticed that I needed to inline all subschemas to ensure visibility to the model, as Gemini does not resolve external references. In other words, Gemini seems to "miss" $def and $ref keys and "guess their content". To mitigate the resulting increase in token usage, large duplicate descriptions are replaced with short internal references after their first occurrence.

This first issue is highly necessary to ensure consistent usage as the API docs explicitly state that const is not supported. However, the second issue is more of a "I tested this and it doesn't work well with the current setup". The API docs explicitly state that $def and $ref keys are supported, however, the model performs visibly worse in tests when inline subschemas is not enabled. This causes a new issue. Namely, that large schemars descriptions get duplicated across the schema. Again, in testing, stripping all but one schema description resulted in significantly degraded performance. To get around this, this PR introduces a system that deduplicates descriptions by hashing their content and using that hash to point repeated descriptions back to the main/first one. This worked the best in my tests (similar performance than not deduplicating at all) likely because the model can see what to refer to during the process of generating that section by referring to the alternative/referenced description, while reducing token usage in my tests by ~30% (I use schemars descriptions as the majority of my prompting so this number is probably closer to 10-15% for most users).

Please see what you think about issue 2 as it may be more advisable for a library like this to implement a better way to edit the schema dynamically at runtime rather than "enforcing" a custom parser, while I think issue 1 should remain the default.

Improves compatibility with Gemini by converting unsupported 'const' fields to single-value 'enum' arrays.

Inlines all subschemas to ensure visibility to the model, as Gemini does not resolve external references. To mitigate the resulting increase in token usage, large duplicate descriptions are replaced with short internal references after their first occurrence.
@flachesis
flachesis requested a review from npatsakula February 2, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant