[mnajafian-nv]

Overview

Harnesses expose token counts with different cache conventions and different coverage. Some fill missing fields with zero or omit failed attempts. Adding those counters without shared rules can make incomplete runs appear cheaper.

Add a shared model-usage contract to NVIDIA NeMo Fabric and connect Deep Agents as its first collector. Results distinguish complete totals from available observations and explain which counts remain unknown. Users do not need recording or Relay to read this accounting.

Dependency: Merge after #476, which supplies the Relay startup compatibility changes. The diff against main includes unmerged prerequisites.

Details

Python and TypeScript helpers preserve reported fields, model identity and gaps. Core validates and normalizes that evidence into RunResult.model_usage. Input includes cache tokens; output includes reasoning tokens. Omitted values remain unknown, while a reported zero remains zero. Complete counts, observations and provider-reported cost remain separate. Per-model entries retain identities established by the source. An exhaustive empty model list cannot accompany positive token counts. Catalog pricing is added separately in #478.

Deep Agents combines model callbacks with passive HTTP-attempt observations. It covers adapter-created ChatOpenAI clients, including summarization and in-process subagents using the observed clients. Custom model overrides, unsupported clients, actual remote dispatch and unfinished work leave the relevant coverage unverified. Graph-message subtotals remain labeled diagnostics in output.usage.

Each model call stays with the invocation that started it. Accounting freezes when that invocation returns. Later reports cannot move into another invocation. Observations are not guaranteed lower bounds. A non-model failure preserves settled usage. Empty source coverage does not establish a measured zero. If validly shaped evidence cannot be normalized, retain the execution result and raw evidence with an accounting diagnostic; schema-invalid extensions still fail validation.

Consumers with older strict result decoders must upgrade before reading the new model_usage field.

Validation

The TypeScript adapter checks remain blocked by the shared dependency audit tracked in #484.

Shared fixtures check normalization across Rust, Python and TypeScript. Regression tests cover independent dimensions, reported zero, omissions, invalid evidence and schema-copy parity. Deep Agents tests cover retries, summaries, parallel in-process subagents, concurrent runtimes and late callbacks.

Where should the reviewer start?

Start with crates/fabric-core/src/model_usage.rs and usage_normalization.rs for complete versus observed values. Then read Deep Agents usage.py and the shared Python, Rust and TypeScript contract vectors.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to: none

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.

• I searched existing issues and open pull requests, and this does not duplicate existing work.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: 630820f
• Latest commit, not reviewed: a1d5527

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The change adds invocation-scoped model usage accounting across adapters, shared contracts, Fabric Core, and SDK results. It preserves unknown counts, records gaps, freezes evidence at invocation return, and exposes normalized usage through RunResult.model_usage. Relay readiness and OpenShell launcher argument handling also change.

Changes

Usage accounting

Layer / File(s)	Summary
Evidence contracts and reducers adapter-contract/python/..., adapter-contract/typescript/..., schemas/extensions/usage-accounting.schema.json, tests/fixtures/usage_accounting/*	Adds typed usage evidence and Python and TypeScript reducers. The reducers separate observed from complete totals, retain reported zeros, and represent missing or unverified evidence with gaps.
Deep Agents collection adapters/python/deepagents/src/nemo_fabric_adapters/deepagents/{adapter.py,usage.py}, adapters/python/deepagents/deepagents.fabric-adapter.json	Records model callbacks and HTTP attempts within an invocation, including cache, remote activity, retries, streaming, and failures. Adapter results use accounted usage; graph-message usage remains a separate diagnostic.
Core and SDK usage results crates/fabric-core/src/{usage_accounting.rs,usage_normalization.rs,model_usage.rs,runtime.rs}, sdk/python/nemo-fabric-runtime/src/nemo_fabric/{types.py,accounting.py}	Validates and normalizes adapter evidence, adds optional RunResult.model_usage, and provides summarize_usage against an explicit invocation inventory.
Schemas, documentation, and validation schemas/*, adapter-contract/typescript/*, docs/*, skills/*, tests/*	Adds usage schemas, API references, adapter guidance, shared conformance fixtures, and tests for accounting, normalization, runtime behavior, and SDK summaries.

Runtime operations

Layer / File(s)	Summary
Relay readiness adapters/python/common/src/nemo_fabric_adapters/common/relay_gateway.py, tests/adapters/test_adapters_common_relay_gateway.py, adapters/python/{claude,codex,common}/README.md	Changes the default readiness budget from 10 to 30 seconds. Each health request and sleep is bounded by the remaining deadline.
OpenShell argument handling examples/openshell/run-demo.sh, tests/scripts/test_openshell_launcher.py	Guards empty environment-argument arrays under Bash 3.2 with set -u and tests supplied and default argument combinations.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: Low · up to 63082

Some consumers may lose top-level token counts, and valid SDK results can raise errors when optional metadata is accessed. These are bounded compatibility risks requiring owner awareness.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 21.03% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 233 functions across 39 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "feat: report complete and observed model usage" follows Conventional Commits (type feat, lowercase, imperative, no trailing period, 46 characters). It matches the main change: shared invoca…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description covers the overview, implementation details, validation, reviewer starting points, and the related-issues and checklist items. It explains the shared model-usage contract, the Deep Age…

Full details: Title check

Explanation

The title "feat: report complete and observed model usage" follows Conventional Commits (type feat, lowercase, imperative, no trailing period, 46 characters). It matches the main change: shared invocation accounting that separates complete token counts from observations, surfaced as RunResult.model_usage.

Full details: Description check

Explanation

The description covers the overview, implementation details, validation, reviewer starting points, and the related-issues and checklist items. It explains the shared model-usage contract, the Deep Agents collector, the Core normalization into RunResult.model_usage, and the cache and reasoning conventions. It also states the compatibility impact for strict older decoders and the dependency on #476. The Related Issues line reads 'Relates to: none', which does not follow the template's 'Closes / Fixes / Resolves / Relates to #' format. The Validation section names test coverage, but it does not clearly state which checks passed. The objectives text reports the validation results, so the description is mostly complete.

✨ Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-442.docs.buildwithfern.com/nemo/fabric