[cmunley1]

What does this PR do?

Updates nemo_fabric_agent from NeMo Fabric 0.2.0 to 0.5.0 and adds an opt-in NeMo Relay path.

• Fabric 0.5.0. Adds the nooa, openhands, openclaw, and relay extras and presets for OpenHands and OpenClaw. The OpenClaw preset needs the CLI release Fabric 0.5.0 documents (npm install --global openclaw@2026.9.4). Like Fabric's Harbor integration, the agent expects harness CLIs to be preinstalled. The 0.5.0 TypeScript adapters (Cline, Droid, Kilo, OpenCode, Pi, Qwen Code) also run through this agent when their npm packages are installed and discovered with fabric_config.discovery.local_paths.
• relay: true. Records each run with NeMo Relay and builds the response from Relay's ATIF trajectory, so adapters with Relay support return the same item shape. fabric_result.relay_atif holds the trajectories.
  • Adapters without Relay support run without it, with a warning.
  • Embedded subagent trajectories (Deep Agents) are inlined at their call.
  • When a trajectory has fewer tool outputs than the adapter's own output, the adapter's output is kept.
  • Claude and Codex record through a Relay 0.10 gateway that accepts a plain-HTTP model URL only on loopback, so their Relay runs need an HTTPS or loopback model URL. feat(adapters): forward trusted HTTP model hosts to Relay NVIDIA/NeMo-Fabric#472 forwards insecure_http_model_hosts to Relay 0.11 for remote HTTP servers.
• send_model_base_url: false for adapters that read their endpoint elsewhere (Factory Droid).
• Observability from adapter output.
  • Claude thinking blocks and Codex reasoning events become reasoning items.
  • The adapter's own usage fills fields Fabric's normalized usage lacks (Codex reasoning tokens).
• Dependencies.
  • litellm<1.103, because later releases import mcp 2 types and nemo-gym pins mcp<2. Also pinned in the portable runtime script.
  • The OpenHands packages Fabric validates (1.50.0), with pillow==12.3.0 overriding hermes-agent's exact pin.
  • A server-local uv.toml that mirrors Gym's resolver policy but keeps cryptography, pathspec, and wcwidth, which OpenHands needs.

Validation

• pytest responses_api_agents/nemo_fabric_agent/tests: 51 passed, including planning for every preset, Relay config, ATIF parsing (Relay's structured extra.tool_result and empty results) and fallback, and the base URL opt-out.
• pre-commit on the changed files: passed.

Rollouts

Every Fabric 0.5.0 harness ran through Gym with a self-hosted DeepSeek-V4.1-Flash policy on three environments. Each cell is passes over attempts. Most harnesses also needed harness-specific settings, such as permission modes, model fields, or the model server's reasoning_field.

Harness	Math (5)	SWE-bench Verified (3)	SWE Atlas Q&A (3)
Claude	4/5	3/3	2/3
Codex	4/5	3/3	1/3
Deep Agents	3/5	3/3	1/3
Hermes	4/5	3/3	2/3
mini-SWE-agent	3/5	3/3	1/3
OpenHands	3/5	2/3	0/3 (see below)
OpenClaw	3/5	2/3	0/3 (see below)
Cline	3/5	3/3	1/3
Kilo	4/5	3/3 with fixes below	2/3 with fixes below
OpenCode	3/5	3/3	1/3
Pi	3/5	2/3	0/3 (each answer misses one rubric)
Qwen Code	3/5	3/3	2/3
Droid	3/5	3/3	not run
NOOA	2/5 with #5059	3/3 with #5059	1/3 with #5059

The remaining 0/3 cells come from limits and settings, not the model:

• OpenClaw: the adapter's 600 s read timeout ended a run after 96 model calls, and a slow gateway exit hit Fabric's 10 s stop deadline after a finished SWE-bench run. The preset now raises the first and shortens the gateway shutdown wait. OpenClaw also caps each completion at 8192 tokens unless fabric_config.models.default.max_tokens is set, and inline reasoning counts against it. The Alpine image's BusyBox setpriv lacks --pdeathsig, which the gateway needs.
• OpenHands: one rollout stopped at its default 500 iterations, one failed before its first model call (the adapter drops the exception), and one was still working when stopped.
• Kilo: Node's 300 s HTTP timeout cuts its single long session.prompt request, and its bash tool waits for an approval no one gives. A NODE_OPTIONS preload that disables undici's timeouts plus fabric_config.tools.enabled fixed both. The proper fix for the first belongs in Fabric's Kilo adapter.

Relay produced complete trajectories for Claude and Codex on SWE-bench (every tool call has its result). Known Fabric 0.5.0 and Relay 0.10 gaps, kept behind the opt-in flag:

• mini-SWE-agent passes a dict to nemo_relay.tools.call_end, so its trajectory has no tool results, and on math more of its runs failed with Relay.
• Deep Agents records the run as one step, so the agent keeps its native output.
• Hermes Relay needs a newer hermes-agent revision than the one pinned here.
• No adapter's trajectory carries reasoning.

Compatibility and benchmark impact

relay defaults to false, so existing runs are unchanged apart from the Fabric version. The pinned benchmark configs that install nemo-fabric[deepagents]==0.2.0 (deepsearchqa, widesearch) are left as is.

Follow-ups outside this PR:

• Gym's chat schema rejects the cache_control NOOA sends: fix(chat): drop cache_control hints from chat request messages #5059.
• anyswe drops the agent's edits when a harness fails, and a relay URL loses the rollout path that model-call capture needs: fix(anyswe_agent): keep rollout path on relay URLs, grade failed runs #5060.
• Capture records no tokens for OpenClaw's streamed calls, which ask for no usage chunk: fix(capture): record usage for streamed chat without a usage chunk #5061.
• The TypeScript adapters' native output carries no tool calls.
• Fabric reports no usage for the Hermes, mini-SWE-agent, Pi and NOOA adapters, and only the last turn's usage for Kilo. Model-call capture has the full counts. feat: add shared model usage accounting NVIDIA/NeMo-Fabric#442 adds a shared usage contract with per-model request counts, which can also replace turns_used.
• ng_perf stays empty because this agent emits no agent invocation records.

Checklist

• I have read the contributing guidelines.
• The change is focused.
• Tests added and pass locally.
• Documentation updated.
• Pre-commit checks pass locally.
• New source files include the required Apache-2.0 SPDX header (no new source files).
• All commits have DCO sign-off.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/ok to test 61e2e62

[cmunley1]

/review

[svcnvidia-nemo-ci]

HOLD — two BLOCKERs in Relay trajectory fidelity and three RISKs in configuration, proxy cleanup, and OpenClaw provisioning.

Completed the broad review of all nine changed files, including affected configuration composition, verifier handoff, installation paths, and existing regression tests. Relay is opt-in, so the trajectory and proxy findings are confined to Relay-enabled runs; the configuration and OpenClaw findings affect their respective setup paths.

Static inspection only: tests, dependency resolution, and model rollouts were not executed. Third-party Fabric/Relay implementations are not mounted, so their internal compatibility claims were not independently verified. The findings below are grounded in the mounted source and fixtures.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[cmunley1]

/ok to test 4d51f4a

[cmunley1]

/review

[svcnvidia-nemo-ci]

SHIP — all five prior findings are resolved in the inspected source; no eligible findings remain in this follow-up.

Verified fixes:

• ATIF normalization preserves structured Relay results, including false, zero, and null. Fallback selection now compares both nonempty and total tool outputs, retaining completed empty native results.
• Proxy cleanup cancels and awaits registered handlers and closes both streams. Post-proxy configuration construction now sits inside the cleanup boundary.
• The base YAML declares both options, allowing composed overrides without weakening unknown-field rejection.
• OpenClaw startup invokes the existing provisioning helper only for that adapter.

Accounted for all ten changed files and compared the fixes with the previous reviewed tree. Discovery remained limited to the prior failures and subsequent changes under the existing impact filter. Relevant regression assertions and directly affected configuration, verifier-handoff, and setup paths were inspected.

Static inspection only: tests, installations, dependency resolution, and rollouts were not executed. Third-party Fabric, Relay, and OpenClaw implementations are not mounted, so runtime compatibility was not independently verified.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[cmunley1]

/ok to test 1c5aed4

[cmunley1]

/review

[cmunley1]

/ok to test d6558fa

[cmunley1]

/review

[svcnvidia-nemo-ci]

SHIP — no eligible findings remain in this follow-up.

Compared the current head with the previously reviewed baseline and accounted for all ten changed files. The only subsequent source change adds OpenClaw’s read and shutdown timeout settings. Traced their handoff through configuration construction and invocation cleanup: adapter settings and the invocation timeout remain separate. No introduced high- or critical-impact regression was established under the existing impact filter.

The earlier repairs remain present: structured Relay results, completed-empty-result fallback, proxy cleanup, declared YAML options, and adapter-specific OpenClaw provisioning. Relevant regression assertions were inspected.

Static inspection only: tests, installations, dependency resolution, and rollouts were not executed. Third-party Fabric/Relay implementations are not mounted, so adapter-internal timeout and shutdown behavior could not be independently verified. Two delegated detailed reports were unavailable; their timeout and lifecycle topics were independently inspected before completion.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[cmunley1]

/ok to test bb178da

[cmunley1]

/review

[svcnvidia-nemo-ci]

SHIP — no eligible findings remain in this follow-up.

Compared all ten changed files with the previously reviewed tree. Discovery covered subsequent startup/lifecycle and accounting changes, plus directly affected configuration, verifier-handoff, and profiling contracts. No introduced high- or critical-impact regression was established under the existing impact filter.

Verified that the earlier structured Relay-result repair, completed-empty-result fallback, and declared YAML options remain intact. Relevant regression assertions were inspected. The former proxy-owned connection leak no longer applies because the proxy and its invocation boundary have been removed.

Static inspection only: tests, installations, dependency resolution, and rollouts were not executed. Third-party Fabric and Relay implementations are not mounted, so their internal runtime compatibility was not independently verified.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[cmunley1]

/ok to test ddc5b5c

[cmunley1]

/review

[svcnvidia-nemo-ci]

SHIP — no eligible findings remain in this follow-up.

Compared all ten changed files with the previously reviewed tree. Discovery focused on the subsequent UTF-8 artifact-handling change and its directly affected response and verifier-handoff paths. No introduced high- or critical-impact regression was established under the existing impact filter.

The reader now skips undecodable artifacts while retaining existing JSON/I/O handling and continuing to readable files. The new regression assertion covers an incomplete multibyte character, truncated JSON, and a subsequent readable trajectory. When no trajectories survive, the selection branch retains Fabric’s native output.

Earlier repairs remain intact: structured Relay results, the completed-empty-result regression assertions, and the declared YAML options.

Static inspection only: tests, installations, dependency resolution, and rollouts were not executed. Third-party Fabric and Relay implementations are not mounted. One delegated detailed report was unavailable; its eight ancillary-file comparisons were independently completed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[cmunley1]

/ok to test ec272b4