Repository navigation
feat(budget): add pure delivery pricing helpers [1/3] - #168
hasitpbhatt wants to merge 9 commits into
Conversation
![orcacode-review[bot]](https://avatars.githubusercontent.com/in/4254738?s=60&v=4){kind=small}
There was a problem hiding this comment.
🐳 OrcaCode Review
Found 1 issue in this PR: 🟠 1 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 121 calls · 2.5M tokens · 90% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
8f429e6 to
898bd48
Compare
There was a problem hiding this comment.
🐳 OrcaCode Review
Found 3 issues in this PR: 🟠 3 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 147 calls · 3.6M tokens · 91% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
🐳 OrcaCode Review
Found 2 issues in this PR: 🟠 2 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 120 calls · 2.8M tokens · 90% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
⚠️ Outdated review — the PR head changed from 9cb899d to 4d7295d during the review; these findings describe a diff the PR no longer has.
These findings describe9cb899d46c20. Re-run the review to check the PR as it stands now.
The gate evaluated to blocked, but it is not enforced here: branch protection will not see it.
🐳 OrcaCode Review
Found 1 issue in this PR: 🟠 1 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 150 calls · 4.2M tokens · 92% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
⚠️ Outdated review — the PR head changed from 4d7295d to 1a9ee7a during the review; these findings describe a diff the PR no longer has.
These findings describe4d7295d56b8c. Re-run the review to check the PR as it stands now.
The gate evaluated to blocked, but it is not enforced here: branch protection will not see it.
🐳 OrcaCode Review
Found 1 issue in this PR: 🟠 1 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 157 calls · 4.2M tokens · 91% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
🐳 OrcaCode Review
Found 1 issue in this PR: 🟠 1 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 155 calls · 4.2M tokens · 91% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
Characters-to-tokens estimation and usage detection for requests whose upstream never reported a measurement. No database or route state, so the pricing rules can be tested in isolation before settlement uses them.
`countable_usage` accepted any countable key that was merely present, so a
usage frame of `{"prompt_tokens": 0, "completion_tokens": 0}` — what LiteLLM
emits when the upstream reported nothing — counted as measured. Settling on it
records a zero cost for a delivered completion, which is exactly the bypass the
guard exists to prevent.
Only a countable key holding a positive number is now a measurement. A zero on
one side still counts when the other side is positive, so a prompt-only request
is unaffected. `bool` is rejected explicitly because `True` is an `int`.
A genuinely empty completion still prices at zero through
`blocking_delivery_chars` and `estimate_usage`, so this cannot overcharge.
text_chars only counted dict parts ({"text": "..."}), so an Anthropic-style
prompt of bare strings (`["hello"]`) priced at 0 characters. blocking_delivery_chars
counts the same shape on a completion, so a budgeted key's prompt was
under-priced whenever the wire used plain-string parts — the precondition is
/v1/messages forwarding Anthropic-style content, reachable by real clients.
Count plain string parts in text_chars the way blocking_delivery_chars does,
keeping the dict branch, so a prompt and a completion of the same content
price identical characters.
Verified: text_chars(["hello","world"]) == 10; a mixed list prices only its
textual parts.
Regression for the bare-string pricing fix: a list of plain strings (`["hello", "world"]`) and a mixed list with bare-string parts now have expected character counts. Mirrors blocking_delivery_chars, which already accepted str parts, so prompt and completion price the same characters.
…t on delivery, count tool_use parts - message_chars: prompt side counts tool_calls as well as content, matching the delivery path, so tool-call-only assistant turns no longer price at zero - countable_usage(delivered=True): a delivered completion needs a positive completion-side count; a prompt-only frame no longer settles it at zero - _part_texts: Anthropic tool_use parts count name and string input, shared by text_chars and blocking_delivery_chars so the two cannot drift
… with a test text_chars intentionally counts whitespace; blocking_delivery_chars skips it only for the delivered-or-not decision. Streamed deltas are priced through text_chars, so dropping whitespace there would undercount real output.
…vered opaque parts
- tool_use: count json.dumps(input or {}), the same serialization the
Anthropic and Gemini translators bill, instead of dropping object input
- delivery: count message.refusal, which reaches the client as output
- delivery: a non-text delivered part (image, audio) counts as one character
so it floors to one token and never settles at zero; the prompt-side
text_chars still counts it exactly
A non-text prompt part (image, tool_result) was billed but counted zero, so an image-only prompt floored to zero tokens. Such a part now counts one character, so it floors to one token. Both prompt and delivery paths share _part_texts.
…racter A tool_result's billable text is its content (string or text parts). It was priced as a single opaque character, undercharging multi-turn tool prompts by the length of every tool output. Its text is now counted; nested non-text parts still floor to one character.
83babc8 to
6af963f
Compare
🐳 OrcaCode ReviewThis pull request was not reviewed. A maintainer of this repository can comment No comments were posted for this commit. Details
OrcaCode Review — Route Smarter. Ship Safer. Spend Less. |
|
@orcacode review |
|
@xizhuomengcontin , can you take a look at #168 , #169 , #170 at your convenience? |
|
@orcacode review |
🐳 OrcaCode ReviewThis pull request was not reviewed. A maintainer of this repository can comment No comments were posted for this commit. Details
OrcaCode Review — Route Smarter. Ship Safer. Spend Less. |
|
@orcacode review |
There was a problem hiding this comment.
🐳 OrcaCode Review
✅ No findings — nothing to flag in this PR. Great work!
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 137 calls · 3.5M tokens · 91% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
|
Abandoning these stacked diffs, doesn't feel worth the effort. |
3 participants
Orca-Code-Review — push 2
✅ no blocking findings
Pure pricing helpers for budgeted requests, with no database, FastAPI or route
state. A budgeted key is charged for what the upstream delivered: when the
provider reports token usage the charge is that measurement, and when it does
not the charge is estimated from the characters that actually reached the client.
What lands here
app/budget_pricing.py:text_chars,message_chars: prompt character counts (content, plus tool calls formessage_chars)countable_usage(usage, *, delivered=False): whether a usage frame is a real measurementchars_to_tokens,estimate_usage: pricing from characterstool_call_text,blocking_delivery_chars: delivered-content countingNo request path changes. This is the base of the stack and is safe to merge
first and alone. Stack: #168 helpers → #169 models → #170 enforce.
Changes since the first push
boolusage fields are not a measurement; only a positive number counts.countable_usage(..., delivered=True)requires a positive completion-side count, so a prompt-only usage frame cannot settle a delivered completion at zero.message_chars), matching the delivery path.tool_useinput is counted asjson.dumps(input or {}), the same serialization the Anthropic and Gemini translators bill. Object input is no longer dropped.message.refusal, which reaches the client as output.text_charscounts whitespace on purpose: streamed deltas are priced through it, and dropping whitespace-only deltas would undercount real output. Documented and pinned by a test.Review threads
All review threads are resolved. Each fix has a test; see
tests/unit/test_budget_pricing.py.Review state
OrcaCode Review (local,
origin/main..budget/helpers) at1a9ee7a: passed, no findings.Verification
tests/unit/test_budget_pricing.py: 46 tests, passing.budget/helpers: 863 passing. Full suite onbudget/enforce(includes this branch): 884 passing.libstdc++; CI should not need the local workaround.