Repository navigation
feat: report Hermes model usage per invocation - #460
Open
mnajafian-nv wants to merge 165 commits into
Open
mnajafian-nv wants to merge 165 commits into
mnajafian-nv wants to merge 165 commits into
Conversation
The Hermes adapter returned no usage, so every Hermes run read unknown tokens. Hermes keeps token counters per session, and the adapter keeps one session per runtime, so each turn's result holds session totals. Each invocation now reports the change since the previous invocation's totals: input_tokens with cache reads and cache writes included, cached_input_tokens, output_tokens and total_tokens. The baseline is 0 for a new Hermes agent, is cleared while an invocation runs, and is set again only from a returned result. After an invocation that raised, was cancelled or timed out, or returned no counters, the next invocation reports no counters instead of a carried-over or whole-session total. A counter that went down is unknown for that invocation. This follows the rule the Codex adapter already uses for thread totals. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Pi adapter returned no usage although Pi reports token counts for every call it bills. Pi reports usage per call, so the adapter now sums what Pi bills during the invocation, the way Pi's own usage totals count it: assistant messages, tool results that report usage, finished compaction summaries and usage entries. The result holds input_tokens with cache reads and cache writes included, cached_input_tokens, output_tokens and total_tokens. A call without a complete usage report, or a compaction that was aborted or finished without usage, leaves the invocation without usage, so a partial sum is never reported as the invocation's total. Pi's cost figures come from its model catalog's rates, so they are not reported as cost_usd. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Pi keeps no usage for a provider call it stopped, so an aborted assistant message reports zero tokens although its input may already have been billed. Adding that zero made a cancelled invocation report a complete-looking total that left out the aborted call. An aborted assistant message now leaves the invocation without usage, the same rule as an aborted compaction. The comments and README now say "records" instead of "bills", because the adapter sums what Pi recorded, and the README lists every case in which the result has no usage. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A Hermes counter above the 64-bit range would make AgentUsage raise and turn a finished invocation into an error, so such a counter is now ignored like any other invalid value, matching the Codex adapter. Hermes records auxiliary calls such as context-compression summaries outside its session counters, so the README now says the usage covers the main model's calls. A test now checks that the invocation after a cancelled one reports no usage, which is the rule that keeps a cancelled turn's tokens out of the next result. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Hermes counts a provider response without usage as an API call but adds no tokens to its session counters, so the adapter reported such an invocation as a measured 0, and a mixed invocation as a complete subtotal. Count those responses through the context compressor's note_usage_less_response hook, which both Hermes runtimes call, and report no usage for an invocation that had one. A response that reported zero tokens still reports 0. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Pi fills a model call's usage with zeros and overwrites it only when the provider reports usage, so a response without usage was summed as a measured 0. Treat a model call whose counts are all zero as unreported. Pi's compaction summary retry also returns only the final attempt's usage, and failed attempts emit no usage, so a retried compaction undercounted. Leave the invocation's usage unknown in both cases. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
When a compaction cuts inside a turn, Pi summarizes the earlier history and the turn prefix in two model calls and reports only their combined usage on compaction_end. A summary call without a usage report is zero-filled, so the combined usage can look complete while one call is missing. Pi exposes the compaction plan, including isSplitTurn, only to extensions through session_before_compact. Register a hidden adapter extension that forwards that event to the running prompt, and leave the prompt's usage unknown unless the plan says the compaction is not split. State in the README and code comments that an all-zero provider report is discarded together with an unreported call. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Bring in the Claude adapter's per-invocation cost and cache token usage, plus the other main changes since this branch was cut. No conflicts. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A provider response can report usage without one of its counts, such as prompt_tokens with no completion_tokens. Hermes' normalize_usage and Pi's provider parsers fill the missing count with 0, so both adapters reported it as a measured 0 and derived total_tokens from it. The usage-less checks only caught a response with no usage at all. Both adapters now read the provider's raw usage before the SDK normalizes it and keep only the counts it reported. Hermes wraps turn_response_check.record_response_usage for the agent it owns and counts responses whose usage lacks an input or output count. Pi wraps the agent's stream function, reads each call's raw stream events through onProviderStreamEvent for every supported wire protocol, and keys the result by the call's final usage object, which covers assistant messages and compaction summaries. An omitted input count leaves out input_tokens, cached_input_tokens and total_tokens; an omitted output count leaves out output_tokens and total_tokens. A reported zero stays 0, which also lets Pi keep a provider report of all zeros that it used to discard. Tests cover a prompt-only report, a completion-only report, reported zeros and a partial response inside a multi-call invocation, through Hermes' real usage accounting and the real Pi SDK against a local streaming endpoint. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Anthropic Messages reports input_tokens without cache reads and writes, and both Hermes and Pi add a missing cache field back as 0. A response with input_tokens but no cache fields therefore still produced a measured input count. Input is now reported only when input_tokens and both cache fields are present, the rule the Claude adapter uses. When a provider sends no cache details, both SDKs count 0 cached tokens. cached_input_tokens is now left out in that case. For the protocols whose input count includes the cache, input_tokens and total_tokens stay known. The Hermes adapter replaced turn_response_check.record_response_usage on every start and never restored it, and nothing showed that Hermes' turn loop still calls that name. The wrapper is now installed once per process and dispatches to observers registered per agent, and stop() removes the agent's observer. Start fails with a lifecycle error unless record_response_usage is still Hermes' turn_usage function and check_api_response still calls it by that name. The mock endpoint takes a usage object per scenario, and the Hermes e2e test sends a partial report through Hermes' real turn loop. For Pi, only a message_delta reports the final Anthropic output count, because message_start carries a placeholder. Google output counts as reported when thoughtsTokenCount is present without candidatesTokenCount. The protocol comes from the model that served the stream event, and a protocol outside Pi's built-in ones reports no counts. PiTokenUsage now holds input with the cache included, so the cache reads can be unknown while input stays known. The READMEs now say which Hermes paths the check covers. Hermes' native Gemini and Bedrock transports, and its Anthropic transport when a stream omits usage, fill missing counts before the adapter sees them, and its Codex app-server runtime records usage elsewhere. They also say that a Pi cache warm-up during an invocation leaves the result without usage: the adapter receives the warm-up's raw events but cannot tie them to the warm-up's usage entry. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Hermes' check_api_response returns on a reply that ends with finish_reason length or content_filter before it calls record_response_usage, so the provider bills that call but Hermes' session counters never include it. A turn that hit max_tokens and continued therefore reported measured input, output and total tokens that left out the truncated call. The adapter's process-wide wrapper now also covers the two handlers check_api_response calls in those cases, recover_from_truncation and handle_content_policy_refusal, with the same once-per-process install and start-time check. When either handler returns before usage is recorded, the response counts like a response without usage, and the invocation reports no usage. A truncation that falls through to record_response_usage is still counted. The mock endpoint takes a finish reason per scenario, and the Hermes e2e test sends a length reply and a content_filter reply through Hermes' real turn loop. A terminal content_filter result already had no token counters, so that case reported no usage before this change; the fix matters when a configured fallback answers after the refusal. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Bedrock Converse reports input without the cache, like Anthropic Messages, but its API documents cacheReadInputTokens and cacheWriteInputTokens as optional. The adapter therefore takes inputTokens alone as the input count, and the README now says why. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Deep Agents adapter summed usage_metadata from the AI messages that reached graph state through the updates stream. Three cases came out wrong: - Deep Agents adds SummarizationMiddleware to every agent and subagent. Its summary call returns only text, so the summary model's message and usage never enter graph state, and an invocation that summarized under-reported input, output and total tokens. - When the invocation raised, including GraphRecursionError, the adapter returned no messages, so usage was None even though model calls had finished. - A count was summed over the calls that reported it and returned as known. ChatOpenAI also turns an omitted Chat Completions count into 0 and derives a missing total, so a partial provider report came out as measured zeros. The adapter now records usage from LangChain's chat-model callbacks, which fire for every chat-model call that runs in the adapter process: the main agent, declarative subagents and summarization. Agent Protocol subagents run remotely and are not included. A new recorder is created for each invocation, so earlier turns are never counted again. It outlives an exception, so a failed invocation still reports the fields that every counted call reported. Each token count and the cost are known only when every counted call in the invocation reported them, and an explicit zero stays a known zero. A request the provider rejects with a 4xx status (other than 408) contributes no usage. A 5xx response, timeout, cancellation or unfinished call leaves the invocation's counts and cost unknown, because the provider may have done work without reporting usage. OpenAI SDK retries happen inside one LangChain call, so a 5xx the SDK retries to success is not visible to the recorder, and only the final response's reported usage counts. When a response carries OpenAI-shaped raw usage, the raw counts decide what was reported: an omitted count stays unknown, the provider's total is used when present, and a derived total counts only when input and output were both reported. The graph tests run the real Deep Agents graph and ChatOpenAI against a scripted Chat Completions endpoint for summarization, parallel task subagents, 400, 408, 503 and timeout failures, the recursion limit, partial reports and explicit zeros. Separate ChatOpenAI.with_retry tests cover 429 then success and 503 then success. The existing usage tests now produce their messages through a LangChain chat-model call so the callbacks fire. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Hermes' check_api_response hands a response that fails validation to retry_invalid_response and returns before record_response_usage. The provider can still bill that call, and a successful retry then produced a measured invocation total that left it out. The adapter's process-wide wrapper now covers retry_invalid_response next to the truncation and refusal handlers, with the same start-time check. Unless the handler falls through to usage recording, the response counts like a response without usage and the invocation reports no usage. The start check now also rejects a handler name that the installed Hermes does not define. The Hermes e2e test answers with the router timeout sentinel that Hermes' chat-completions transport rejects in validate_response, with usage that bills the prompt and no completion tokens, followed by a successful retry through Hermes' real turn loop. It fails without the wrapper. The content_filter e2e case is replaced by a unit test of the refusal handler's dispatch with a fallback reply: a terminal refusal result carries no counters, so the e2e case passed without the wrapper, and Fabric's Hermes config enables no fallback. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Hermes README still said the adapter wraps only record_response_usage and listed only truncated and refused replies. The adapter also observes retry_invalid_response, and a streamed call that Hermes retries inside its API call, such as an empty stream, never reaches these handlers, so its billed usage is not observed. The README now names all three handlers and states that limit. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A RunResult says which agent and adapter ran, but not which Fabric version, models or MCP servers were behind it, so a rerun weeks later cannot tell what changed. Add an optional, versioned what_ran object. Each fact holds a value, or a state and a reason, plus how Fabric learned it: its source, collector and method. Core fills the facts it holds before it sends the request: its own version, the adapter ID, the requested models, the configured MCP servers with secret-free endpoints, and the environment provider. Core reads no adapter report yet, so adapter-reported facts read unsupported with reason key_not_declared. On the OpenShell host path, the host relabels the task core's facts as task_core, records its own version as host_core_version and sets the environment provider to openshell. A task result without a readable what_ran keeps what_ran absent. The Rust and Python readers apply the same structural checks, tested against one shared vector file. A malformed what_ran reads as absent and never fails the enclosing result. Core gains a direct dependency on sha2 0.10, already in the lockfile, for the string length rule. Rust code that builds RunResult with a struct literal must add the what_ran field. Python's RunResult.to_mapping() now always includes it. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Explain the fact envelope, the four no-value states, the endpoint rule, the OpenShell attribution and the structural read, and keep the integration skill's RunResult table in step with the guide. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Regenerate the Rust library reference with Rust 1.94.0 for the new what_ran module and the RunResult field, and the Python types reference for RunResult.what_ran. Only the pages this change affects are included. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A plan with many model roles or MCP servers, or very long names, could make what_ran arbitrarily large, and every RunResult carries it. Keep it bounded without ever failing a run. Core keeps the first 16 model roles and 32 MCP servers in sorted key order, and drops entries whose key is longer than 128 characters. While the compact JSON of what_ran is larger than 128 KiB, it drops MCP servers from the end of the sorted order, then model roles. If plan identifiers alone still exceed the budget, what_ran becomes a minimal object with only the schema name, the version and what_ran_truncated. A marker next to each map counts the dropped entries and names the last rule applied. The OpenShell host applies the same bounds again after it attributes the task core's facts, so they hold for a task core of any version. The six what_ran sections become optional in the Rust type so the minimal object can be represented. Every section is still required unless what_ran_truncated is present, and no section may be null. The Rust reader, the generated schema and the Python reader check this against the shared vectors. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Explain the entry caps, the key length rule, the byte budget, the truncation markers and the minimal object, and tell readers to check what_ran_truncated before reading a section. Keep the integration skill's RunResult notes in step. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Regenerate the Rust library reference pages of the what_ran module for TruncatedEntries and the optional sections, and update the Python types reference for RunResult.what_ran. Only the pages this change affects are included. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Hermes adapter's response observer skipped a falsy usage object, so an empty usage dict never reached the omitted-count check. Hermes' record_response_usage also treats an empty usage object as missing and calls note_usage_less_response for it, so the invocation's usage was already unknown through the compressor hook. The observer now checks for None explicitly and counts an empty usage object as a report with input, cached input and output left out, so the result no longer depends on that hook alone. The new test runs an empty usage object through Hermes' real record_response_usage and expects unknown usage, followed by a counted invocation. The Pi README said a tool result without a complete usage report leaves the result without usage. A tool result with no usage field adds nothing; only a usage object with a missing or invalid count makes the invocation's usage unknown. The README now says both. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The adapter rebuilt its baseline only from a turn's result, so after a failed, cancelled or counterless invocation the next invocation had no usage. It now reads the agent's session counters right before and after each turn, so every invocation reports its own change, including a failure result. Hermes returns on a truncated, invalid or refused response before adding its usage to the counters. With the 512-token default, truncation is common, and it made the whole invocation unknown. The adapter now adds the counts such a response reported, normalized with Hermes' own normalize_usage for the provider and API mode captured before the handler can switch to a fallback route. A response that falls through to record_response_usage is counted there only. The handler wrapper also passed the response positionally, but Hermes calls retry_invalid_response with a keyword-only response, so every invalid response raised TypeError and failed the turn. The wrapper now forwards the call unchanged and reads the response from the handler's signature. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Under rate limits Pi's auto-retry is on by default, but the rejected request's error message carries Pi's zero-filled usage with no reported field, so one 429 made the whole invocation unknown. A call that ends in an error with no successful HTTP response and no raw stream event never reached generation, so it now counts as 0 tokens. A stream that failed after it started stays unknown. The first threshold compaction of a single-prompt run cuts inside the turn, and Pi reports only the combined usage of its two summary calls. Both calls already pass the adapter's stream wrapper, so the adapter now sums their own records when the calls it saw are the ones the preparation needs and add up to the combined usage. A tool usage report missing one count made every count unknown; each count now stays known on its own. Records are also keyed by the assistant message, because an extension's message_end handler can replace the message's usage object while Pi keeps the message. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Run a local host invocation with two model roles and three MCP servers (npx with arguments, streamable-http and sse) and no environment section, and assert the exact what_ran a user gets, including the origin of every unsupported adapter-reported fact and that no configured secret appears. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Resolve a plan from each of the 16 bundled descriptors with a model and, where the adapter accepts MCP, three servers, and check that what_ran passes the schema, carries no truncation marker, and reads every adapter-reported fact as unsupported/key_not_declared. Also show that 16 roles and 32 servers with 128-character keys fit the byte budget, and that stdio placeholders read executable_unparsed. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
AgentUsage holds counts up to 2^64 - 1. Each session counter was checked against that bound, but the input sum, the total and the tokens added from skipped responses were not, so an out-of-range provider count made invoke raise a ContractValidationError instead of returning a result. A count that does not fit is now unknown, like any other count the adapter cannot trust, and a raw report above the bound counts as omitted. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The rule that a call with no successful HTTP response and no raw stream event was rejected relies on Pi's built-in protocols calling onResponse and onProviderStreamEvent. A custom protocol may call neither, so any of its failed calls looked rejected and counted as 0 tokens even after the provider generated output. Such calls now keep only the counts their raw events reported. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Use the system Bash path so macOS tests exercise Bash 3.2 even when PATH contains a newer Bash. This keeps the empty-array regression meaningful in CI. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Hermes main-agent counters do not cover all auxiliary work, and a final successful response can hide earlier failed attempts. Calls completed before a task failure still need to remain visible.
Collect supported main-agent and summary requests, plus auxiliary reports exposed by Hermes hooks. Preserve available usage while identifying missing reports, unfinished work and unsupported collection paths.
Dependencies: #442 supplies shared accounting; #458 supplies adapter-reported execution facts. Both branches are included. The diff against
mainincludes unmerged prerequisites.Details
Attach an observed HTTP client to the supported OpenAI-compatible agent path. Record requests, retries and bounded stream usage without changing which bytes the SDK reads. Replace earlier cumulative snapshots rather than adding them twice. A parse failure leaves a gap even if later chunks are readable. No HTTP status is assumed free.
Public middleware and auxiliary hooks retain other available reports. Shared accounting in #442 normalizes the snapshot at return. Late responses keep their initiating invocation and cannot add usage to another one. A timed-out version lookup remains unknown for that runtime without delaying later invocations.
Explicitly disable automatic background review and model-based title upgrades. Task-delegation scheduling is unchanged. OpenAI-compatible main-agent and summary paths are counted; native Anthropic main calls, delegated children, opaque compression and lossy auxiliary reports retain gaps. Unsupported source versions preserve execution with unverified accounting.
Validation
The TypeScript adapter checks remain blocked by the shared dependency audit tracked in #484.
Regression tests cover literal omissions, stream parsing, hidden retries, auxiliary reports, duplicate observations and source changes. Ownership cases cover concurrent runtimes and late usage; stream tests check that observation preserves the bytes and lazy reads seen by the SDK.
Where should the reviewer start?
Start with request observation and stream parsing in
adapters/python/hermes/src/nemo_fabric_adapters/hermes/usage.py. Check settings inconfiguration.py, then streaming, retry, summary and auxiliary cases intests/adapters/test_hermes_accounting.py.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Relates to: none
I confirm this contribution is my own work, or I have the right to submit it under this project's license.
I searched existing issues and open pull requests, and this does not duplicate existing work.