fix(usage): authoritative agent cost + cache tokens; honest budget labels - #142
Conversation
…bels
Harness reported ~$2.92 / 486k tokens for a Claude-account session that
actually cost ~$61 with ~69M tokens, and showed a misleading "0% used"
headline. Two root causes, both fixed:
1. Cost/token undercount. The ACP gateway recorded only the thin `usage_update`
(context-window tokens, no cache; a cache-excluded `cost.amount`). The SDK
`result` message — which carries the authoritative `total_cost_usd` and full
per-category token usage incl. cache — was on the wire (emitRawSDKMessages)
but unsubscribed. Now: subscribe to `{type:"result"}`, parse total_cost_usd
+ usage/modelUsage (snake top-level, camel per-model), record per-turn deltas
(cumulative-within-session, reset on a fresh ACP session) tagged
`authoritative`. agentUsage:record upserts — the authoritative row upgrades
the thin one for the same turnKey (non-regressing if the result never fires,
no double-count either way). New cache/input/output token columns surfaced
in getMyAgentUsage and the dialog.
2. Misleading "0% used". The daily/weekly bars only ever tracked Harness's own
OpenRouter budget ($0 for users on their own agent account). Relabelled as
"Harness budget" with a note; the sidebar badge is now a gauge ICON (color by
the real signal, opens the dialog) instead of "0% used" text. The Claude
account rate-limit % (the 66%/30% seen in Claude Code) is surfaced
best-effort from the captured `_meta._claude/rateLimit` snapshot via a
defensive parser (shown when it parses, omitted otherwise).
Tests: +2 Convex (authoritative upsert both orders), +12 FastAPI (result
parsing + cumulative→delta). Full suites: FastAPI 313, Convex 180, web 225;
biome clean, tsc 21/21 baseline.
Critical correctness fix from adversarial review (verified vs SDK docs): 1. [HIGH] The SDK result message fires once per ACP prompt turn (each turn is its own query() call), so its total_cost_usd + token usage are PER-TURN totals, NOT cumulative across the session. The previous delta-subtraction undercounted every turn after the first — clamping bursty cache-read tokens to 0 and undercounting cost. Record the result's cost+tokens directly per turn; drop the last_result_cost/last_result_tokens trackers and their reset. 2. [MED] A hollow result (cost 0 + empty usage, e.g. an error/aborted turn) no longer records an all-zero authoritative row that would clobber the good thin usage_update row (and the falsiness bug `0.0 or baseline` is gone). It returns None and the thin row stands. A subscription turn (cost 0 but real tokens) still records. 3. [LOW] Clamp the best-effort account % to [0,100] so an odd upstream value can't render an absurd label. 4. [LOW] latestAccountUsage now picks the credential whose turn is globally freshest (sort by lastTurnAt), matching its doc — not the highest-spend one. 5. [LOW] Fix the stale rail comment (the gauge always renders now). Tests rewritten for per-turn semantics (+ hollow-result + subscription cases). FastAPI 314, Convex 12, web 225; biome clean, tsc 21/21.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| // "Work" tokens = input+output (what the user recognizes); cache read/write | ||
| // is shown separately because it dominates raw counts but is cheap. Falls | ||
| // back to the legacy total when the per-category fields are absent. | ||
| const work = (r.inputTokens ?? 0) + (r.outputTokens ?? 0) || r.totalTokens; |
There was a problem hiding this comment.
Token display undercount for mixed thin + authoritative credentials
The || fallback was designed for the all-thin case (sum = 0 → fall back to totalTokens). But getMyAgentUsage accumulates r.inputTokens ?? 0 across all ledger rows — thin rows (which have no inputTokens) silently contribute 0 rather than undefined. So for any credential that has even one authoritative turn, the partial per-category sum is > 0, the || never fires, and every thin turn's tokens are silently dropped from the display.
Concrete example with mixed turns for one credential:
- Turn A (thin, result message never arrived):
usedTokens = 1 000, noinputTokens→ adds0to the per-category accumulator - Turn B (authoritative):
inputTokens = 300,outputTokens = 100→ sum = 400
Aggregated r: totalTokens = 1 500, inputTokens = 300, outputTokens = 100.
work = (300 + 100) || 1 500 evaluates to 400 — Turn A's 1 000 tokens are invisible.
The root cause is in getMyAgentUsage (packages/convex-backend/convex/agentUsage.ts): the accumulator treats "field absent" and "field = 0" identically by coercing with ?? 0. The fix should distinguish "no per-category data" from "per-category data that happens to sum to zero". One approach: track whether all rows in the scan contributed per-category data, and return inputTokens: undefined (or fall back to totalTokens in the accumulation) when the coverage is incomplete — so the frontend || fires correctly for mixed credentials too.
Harness/apps/web/src/components/usage-display.tsx
Lines 260 to 264 in 33fdc61
1 participant
Harness reported ~$2.92 / 486k tokens for a Claude-account agent session that actually cost ~$61 / ~69M tokens, and showed a misleading "0% used" headline. Two root causes, both fixed.
1. Cost/token undercount (correctness)
The ACP gateway recorded only the thin
usage_update(context-window tokens, no cache; a cache-excludedcost.amount). The SDKresultmessage — which carries the authoritativetotal_cost_usdand full per-category token usage incl. cache — was on the wire (emitRawSDKMessages) but unsubscribed.{type:"result"}/{type:"system",subtype:"result"}.query()call), so its cost + token usage are per-turn totals — recorded directly (no cross-turn deltas). (An adversarial review caught an initial cumulative-vs-per-turn mistake; fixed and re-verified against the SDK docs.)agentUsage:recordupserts: the authoritative row patches the thin one for the sameturnKey— non-regressing if the result never fires, no double-count either way. A hollow result (cost 0 + empty usage, e.g. an aborted turn) is skipped so it can't clobber the thin row; a subscription turn (cost 0, real tokens) still records.inputTokens/outputTokens/cacheReadTokens/cacheCreationTokenscolumns surfaced ingetMyAgentUsage; the dialog shows work tokens (input+output) with cache called out separately.2. Misleading "0% used" (honesty + design)
The daily/weekly bars only ever tracked Harness's own OpenRouter budget ($0 for users on their own agent account).
_meta._claude/rateLimitsnapshot via a defensive, clamped parser — shown when it parses, omitted otherwise. (The blob shape is upstream-defined; this should be confirmed against live staging data.)Process
Investigation workflow → root cause; implemented; adversarial review workflow (4 dims × skeptic) caught a high correctness bug (per-turn vs cumulative) + 5 lower issues — all fixed; a second workflow verified the fixes clean.
Validation
biome clean · tsc 21/21 (baseline) · FastAPI 314 · Convex 180 · web 225. New tests: Convex authoritative-upsert (both orders); FastAPI result parsing, per-turn (no baseline), hollow-result, subscription cases.