Repository navigation
feat: report adapter and harness versions in run results - #458
Open
mnajafian-nv wants to merge 64 commits into
Open
mnajafian-nv wants to merge 64 commits into
mnajafian-nv wants to merge 64 commits into
Conversation
A RunResult says which agent and adapter ran, but not which Fabric version, models or MCP servers were behind it, so a rerun weeks later cannot tell what changed. Add an optional, versioned what_ran object. Each fact holds a value, or a state and a reason, plus how Fabric learned it: its source, collector and method. Core fills the facts it holds before it sends the request: its own version, the adapter ID, the requested models, the configured MCP servers with secret-free endpoints, and the environment provider. Core reads no adapter report yet, so adapter-reported facts read unsupported with reason key_not_declared. On the OpenShell host path, the host relabels the task core's facts as task_core, records its own version as host_core_version and sets the environment provider to openshell. A task result without a readable what_ran keeps what_ran absent. The Rust and Python readers apply the same structural checks, tested against one shared vector file. A malformed what_ran reads as absent and never fails the enclosing result. Core gains a direct dependency on sha2 0.10, already in the lockfile, for the string length rule. Rust code that builds RunResult with a struct literal must add the what_ran field. Python's RunResult.to_mapping() now always includes it. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Explain the fact envelope, the four no-value states, the endpoint rule, the OpenShell attribution and the structural read, and keep the integration skill's RunResult table in step with the guide. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Regenerate the Rust library reference with Rust 1.94.0 for the new what_ran module and the RunResult field, and the Python types reference for RunResult.what_ran. Only the pages this change affects are included. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A plan with many model roles or MCP servers, or very long names, could make what_ran arbitrarily large, and every RunResult carries it. Keep it bounded without ever failing a run. Core keeps the first 16 model roles and 32 MCP servers in sorted key order, and drops entries whose key is longer than 128 characters. While the compact JSON of what_ran is larger than 128 KiB, it drops MCP servers from the end of the sorted order, then model roles. If plan identifiers alone still exceed the budget, what_ran becomes a minimal object with only the schema name, the version and what_ran_truncated. A marker next to each map counts the dropped entries and names the last rule applied. The OpenShell host applies the same bounds again after it attributes the task core's facts, so they hold for a task core of any version. The six what_ran sections become optional in the Rust type so the minimal object can be represented. Every section is still required unless what_ran_truncated is present, and no section may be null. The Rust reader, the generated schema and the Python reader check this against the shared vectors. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Explain the entry caps, the key length rule, the byte budget, the truncation markers and the minimal object, and tell readers to check what_ran_truncated before reading a section. Keep the integration skill's RunResult notes in step. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Regenerate the Rust library reference pages of the what_ran module for TruncatedEntries and the optional sections, and update the Python types reference for RunResult.what_ran. Only the pages this change affects are included. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Run a local host invocation with two model roles and three MCP servers (npx with arguments, streamable-http and sse) and no environment section, and assert the exact what_ran a user gets, including the origin of every unsupported adapter-reported fact and that no configured secret appears. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Resolve a plan from each of the 16 bundled descriptors with a model and, where the adapter accepts MCP, three servers, and check that what_ran passes the schema, carries no truncation marker, and reads every adapter-reported fact as unsupported/key_not_declared. Also show that 16 roles and 32 servers with 128-character keys fit the byte budget, and that stdio placeholders read executable_unparsed. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Loop over whatever the catalog lists without pinning its size, so adding or removing a bundled adapter doesn't break the test. Name the size test after what it builds: ASCII keys and model names at their maximum length. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Adapters had no way to report their own facts, so the adapter package version, the harness name and version, the served model and the MCP runtime facts always read unsupported with reason key_not_declared. Core now reads nemo_fabric.versions.v1 and nemo_fabric.mcp.v1 from a result's extensions after the descriptor's extension check, when the descriptor's run_result schema names the key under properties. Only adapter-owned facts with source reported and collector adapter are kept; a malformed or forged fact reads invalid_report, an omitted one fact_omitted, an omitted key key_omitted, and a key over 64 KiB too_large. A reported server the plan does not configure becomes a harness entry. tool_names keeps 64 sorted names and counts the rest in a new tool_names_truncated marker, added to the schema and both readers. Ingestion never changes the run's status or error. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
A what_ran could not say which harness build ran. Six adapters now declare nemo_fabric.versions.v1 and report their package version and the harness name and version from a source that names the build that ran: - Claude Code: claude_code_version from the CLI's init message. - Codex: the thread's cli_version from one bounded thread/read request. - Hermes: get_version_info(), a blocking call that runs git, on a daemon thread with a 5 second limit. - OpenClaw: the --version check, or an attached gateway's /startupz version, both already read at startup. - Deep Agents and mini-swe-agent: package metadata, accepted only when the distribution installed the module the adapter loaded. A shared helper builds the report, applies core's string length rule, and runs blocking calls with at most one outstanding probe per runtime; stopping a runtime never waits for it. A source that times out, fails or answers nothing reads a state with a reason, never a substitute value. The adapter catalog is regenerated for the six changed descriptors. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
what_ran.models.<role>.served stayed unsupported for Claude Code even though each assistant message names the model that answered. The Claude Code adapter now reports, for the role it runs (default, or the sole configured role), the one distinct model named by this invocation's main-thread assistant messages. Messages with a parent_tool_use_id belong to subagents and are skipped, and so are Claude Code's <synthetic> API-error messages. Two distinct models read not_attributed, none reads empty_response, a failed stream reads collection_failed, and every invocation starts from its own messages. Tests feed saved Claude Code CLI streams through the SDK's parser. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The SDK guide and the integration reference said every adapter-reported fact reads key_not_declared. They now name the six adapters that report harness versions, the Claude Code served-model rule, the states a report key can produce and the tool_names cap. The adapter tutorial and the build-adapter skill show adapter authors how to declare and fill nemo_fabric.versions.v1. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Regenerated with just docs for the new report-key constants and the tool_names_truncated field. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
NOOA, NOOA Bench, OpenHands and Remote Agent left adapter.package_version at key_not_declared, so a run with them could not say which adapter build ran. Each now declares nemo_fabric.versions.v1 and attaches its installed package version to every result. These adapters have no harness version or served-model source yet, so those facts read unsupported with reason no_source. The catalog is regenerated. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Cline, Droid, Kilo Code, OpenCode, Pi and Qwen Code left adapter.package_version at key_not_declared. A shared withPackageVersion wrapper in the TypeScript common package reads the adapter's own package.json synchronously at start and adds nemo_fabric.versions.v1 to every result, with the harness and served model facts unsupported/no_source until their sources land. Each CLI wraps its runtime, each descriptor declares the key, and the catalog is regenerated. A catalog test checks that all bundled descriptors declare the key. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The SDK guide and the integration reference named only the six adapters with a harness version source. The TypeScript common README documents withPackageVersion. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Claude and Hermes adapters attach nemo_fabric.versions.v1 to every result, and core rejects a result extension that the descriptor's run_result schema does not declare. The Harbor SWE-bench copies of both descriptors lacked the block, so runs through them failed. A new test finds every checked-in copy of a bundled descriptor and checks that it declares the key. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
display_version drops the commit and the dirty marker of a source checkout, so two different Hermes builds could report the same version. Report derived_version, which Hermes's own --version output uses. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The regenerated references also picked up changes that come from the base branch, such as new FabricError variants, the telemetry field type and page positions. Restore those pages so the branch carries only the what_ran pages for the new report keys. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
BlockingProbe marked a probe as running before starting its thread. If the start raised, nothing cleared the mark, and every later probe in the runtime timed out at once. Clear the mark and re-raise. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Each adapter knows its harness name as a constant, so no collection technique stands behind it. The names reused the version's method, such as harness_init_message or package_metadata, which claimed a source the name never came from. Report them with adapter_report, as the design requires for an adapter fact with no technique behind it. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The Cline SDK's result names a model, but nothing yet shows that it is the model that served the invocation, so the design reads Cline's served model as unavailable with reason not_attributed rather than unsupported. packageVersionReport and withPackageVersion take an optional served-model fact for one role, and the Cline entry point passes it for the role Cline runs. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The integrate skill said key_not_declared applies whenever an adapter does not report a fact. Core uses it only when the descriptor does not declare the report key; a declared key that the result omits reads key_omitted, and a declared key that omits the fact reads fact_omitted. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
The probe mapped any ImportError to no_source, including one raised inside get_version_info(), and labeled the fact harness_api. Only a failed import of the version call now reads no_source, through the shared no_source fact with method adapter_report. An ImportError inside the call reads collection_failed with reason no_result like any other error. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
…r byte The copy check read every file on disk, skipped build, dist and target folders, and looked only for the versions key. It now lists tracked descriptors with git ls-files, walking the tree only outside a git checkout, matches copies by adapter ID, runner module or runner script, and requires each to equal its source byte for byte, as the CLI's packaged-descriptor test already does for its own copies. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
…pers The common package README described withPackageVersion's callback but not packageVersionReport's third parameter, the ServedModelFact type, or missingFact, which the Cline adapter uses to build its fact. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
mnajafian-nv
changed the title
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
This was referenced Oct 22, 2026
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
mnajafian-nv
marked this pull request as ready for review
October 23, 2026 12:37
Use the system Bash path so macOS tests exercise Bash 3.2 even when PATH contains a newer Bash. This keeps the empty-array regression meaningful in CI. Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
mnajafian-nv
changed the title
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The core execution-facts contract knows the resolved configuration, but cannot identify installed harness versions on its own. Without adapter reports, callers cannot use those results to investigate version changes between runs.
Add adapter-reported versions to
RunResult.what_ran, plus served-model evidence where supported. Failed lookups leave a missing fact without changing the run's execution outcome.Dependencies: #443 supplies the core execution-facts contract. The diff against
mainincludes unmerged prerequisites.Details
Validate result extensions before reading declared report keys. Accept facts only when labeled
reportedand collected byadapter. Reports cannot overwrite core facts, configured model identities or configured Model Context Protocol (MCP) endpoints. Plan servers take priority over harness-added servers under entry and byte limits.Read harness versions from six sources: Claude initialization messages, Codex thread data, Hermes
derived_version, OpenClaw's startup check, and Deep Agents and mini-SWE-agent package metadata associated with loaded modules. Other harness versions remain unsupported. Package metadata identifies the installed package, not proof of unmodified code. Hermes uses one bounded startup lookup; a timeout leaves the fact missing for that runtime.Claude served-model facts use main-thread assistant messages, excluding child and synthetic API-error messages. Ambiguous models remain unattributed. Cline reports
not_attributedbecause its aggregate model information cannot identify a configured role. Other adapters retain missing served-model facts.The TypeScript wrapper snapshots reports per result. Descriptor declarations, packaged copies and catalog entries stay aligned. Shared vectors cover ordering and string bounds across Rust, Python and TypeScript.
Validation
Regression tests cover declared report keys, forged fact origins, core-fact protection, plan-server priority, truncation and byte limits. Shared fixtures check role ordering across Rust, Python and TypeScript. Descriptor-copy tests check that packaged declarations match their canonical sources.
Where should the reviewer start?
Start with
ingest_adapter_reportsincrates/fabric-core/src/what_ran.rs. Then review the version-source collectors, shared provenance helpers and descriptor-copy test intests/python/test_adapter_catalog.py.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Relates to: none
I confirm this contribution is my own work, or I have the right to submit it under this project's license.
I searched existing issues and open pull requests, and this does not duplicate existing work.