Optimizes Gemini tool schemas and token usage - #67
Open
noahbclarkson wants to merge 1 commit into
Open
Conversation
Improves compatibility with Gemini by converting unsupported 'const' fields to single-value 'enum' arrays. Inlines all subschemas to ensure visibility to the model, as Gemini does not resolve external references. To mitigate the resulting increase in token usage, large duplicate descriptions are replaced with short internal references after their first occurrence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves compatibility with Gemini by converting unsupported 'const' fields to single-value 'enum' arrays. This is highly important for getting correct responses from the model. The Gemini API does not support the
constkeyword, which was being sent in the current implementation. This meant that Gemini could not "see" that certain values wereconst, causing fixed values to be enforced at the prompt level. This occurs most especially for enums with#[serde(tag = "type")]. By switching allconstvalues to an enum with a single field, Gemini can now see that a certain value is fixed as if it wereconstwithout explicitly using theconstkeyword.Additionally, after some testing I noticed that I needed to inline all subschemas to ensure visibility to the model, as Gemini does not resolve external references. In other words, Gemini seems to "miss"
$defand$refkeys and "guess their content". To mitigate the resulting increase in token usage, large duplicate descriptions are replaced with short internal references after their first occurrence.This first issue is highly necessary to ensure consistent usage as the API docs explicitly state that
constis not supported. However, the second issue is more of a "I tested this and it doesn't work well with the current setup". The API docs explicitly state that$defand$refkeys are supported, however, the model performs visibly worse in tests when inline subschemas is not enabled. This causes a new issue. Namely, that large schemars descriptions get duplicated across the schema. Again, in testing, stripping all but one schema description resulted in significantly degraded performance. To get around this, this PR introduces a system that deduplicates descriptions by hashing their content and using that hash to point repeated descriptions back to the main/first one. This worked the best in my tests (similar performance than not deduplicating at all) likely because the model can see what to refer to during the process of generating that section by referring to the alternative/referenced description, while reducing token usage in my tests by ~30% (I use schemars descriptions as the majority of my prompting so this number is probably closer to 10-15% for most users).Please see what you think about issue 2 as it may be more advisable for a library like this to implement a better way to edit the schema dynamically at runtime rather than "enforcing" a custom parser, while I think issue 1 should remain the default.