[Enhancement]: Save and inspect each NeMo Fabric request in one journal #422

Affected area

Rust core or runtime; Python SDK and binding; Experimentation CLI; Schemas; Documentation and examples

Problem or opportunity

SDK and CLI users need to inspect what happened during an NVIDIA NeMo Fabric request, what it consumed and which measurements remain unknown. Saving only a returned result loses evidence when startup or execution fails before returning one. Separate lifecycle, result and resource reports make that evidence harder to find and connect.

Proposed enhancement

Add opt-in recording and inspection for local Python and process adapter hosts. Enable runtime.record.enabled with an artifact root, or use the CLI flags --record --artifacts path/to/artifacts. Recording stays off by default and requires no benchmark runner or NeMo Relay. Configuration failures before recording starts do not create a journal.

Give each recorded request one reporting journal:

<artifact root>/.fabric/requests/<recording_id>/run.jsonl

Each line is one JSON record. Retain request identity, append order, UTC timestamps and elapsed seconds. Add runtime and invocation IDs when they become known. Records contain lifecycle events, bounded runtime stderr and optional resource samples. Diagnostic timestamps record capture time, and diagnostics can include background work.

Append one final record containing the full result or an explicit error when no result was returned. For a one-shot request, finalization includes owned runtime shutdown. An invocation on a persistent runtime ends before any later shutdown. Preserve the earlier invocation evidence and the final operation outcome. A journal without a final record does not establish completion or zero consumption.

The journal can contain sensitive output and diagnostics. Task files, native harness artifacts, Relay traces and internal protocol files can remain separate. Recording does not replace them. Keep existing resource-artifact behavior when recording is disabled.

Inspecting a recorded request

Run python -m nemo_fabric.reporting path/to/run.jsonl for a concise summary. Show the harness, requested and served model evidence, execution outcome, timestamps, scoped timing, available CPU and memory measurements, token counts and cost. Label partial amounts and unavailable totals beside their values.

--details adds execution history, measurement scope, sampling quality, pricing evidence, provenance and artifact references. --json prints the structured summary without creating another file. All views use the same saved evidence.

Reuse resource collection from #473, usage accounting from #442, pricing from #478 and execution facts from #443/#458. Use structured observations; do not infer token totals from diagnostic text or add another sampler. Missing values remain unknown. Keep provider-reported cost separate from estimates. Execution success does not establish task correctness.

Public contract and compatibility impact

• Add optional recording configuration, a versioned journal schema, a run_log artifact reference and SDK inspection support. Rust callers that construct RuntimeConfig exhaustively must supply its new optional field.
• Keep returned SDK results and CLI JSON on stdout. For a recorded one-shot failure, expose the journal path only after its first record was saved. Preserve the original execution error.
• Create journals without replacing existing files or symlinks. Recording uses synchronous, best-effort I/O. Write failures preserve execution outcomes and never retry model work. Report final-sync failures through returned diagnostics or stderr.
• An outer timeout describes what the caller observed; it does not prove inner or remote work stopped.
• Bound diagnostics and samples and report discarded observations. Keep the full final result outside these retention limits. Readers reject unsupported record sizes explicitly without changing saved data.
• Keep legacy saved-result and invocation-record readers. A final result is authoritative; earlier events and traces must not replace its unknown measurements.

Alternatives considered

• Saving returned results misses requests that never return one.
• Separate reporting files require more lookup and correlation.
• A conventional JSON document needs rewriting or recovery if interrupted. JSONL preserves each complete appended record independently.
• Logs and Relay traces help diagnose execution but do not replace normalized usage or lifecycle evidence.

Acceptance criteria

• Recording remains opt-in and validates supported execution locations and the artifact root.
• Each recorded request has one reporting journal, including enabled resource samples and final reductions. Resource collection without recording remains compatible.
• Requests that return no result retain available request and failure evidence without inventing execution IDs or measured zeros.
• Inspection uses the same two-column summary, UTC timestamps and durations in seconds. Reported zero, partial amounts and unavailable totals remain distinct.
• Details explain sources, resource scope, gaps and pricing without repeating summary rows. JSON preserves exact quantities and structured evidence.
• Complete records before a truncated trailing line remain readable. Reject malformed complete records, mismatched identities and duplicate final summaries.
• Warm and parallel requests have separate journals. Producers cannot append after closure. Diagnostics retain runtime scope; late resource samples stay with their original invocation or are discarded.
• Recording failures preserve the original execution outcome. Documentation explains that journals can contain sensitive data.
• Schema, lifecycle, reader, resource-routing and failure regressions pass. Public examples and PR descriptions match the delivered behavior.

Additional context

The recording changes are split across four PRs:

• feat: save run events and results in one journal #423 adds the journal and inspector.
• feat: save runtime diagnostics in run journals #424 adds bounded runtime diagnostics and failure context.
• feat: save resource observations in run journals #426 routes resource observations into the journal.
• feat: show execution facts in saved run reports #481 shows saved execution facts in the inspector.

Task comparison in #480 consumes this evidence and has separate completion criteria. Close this issue after the recording scope merges and its acceptance criteria are met.

Duplicate check

• I searched the existing issues and did not find a duplicate.