Make traces replayable and useful for step-by-step debugging

README.md

@@ -138,21 +138,42 @@ Tools:
 - `finish`: rejected until a test or build succeeds after the latest patch.
 
 DeepSeek thinking-mode `reasoning_content` is retained in memory and passed back
-after tool calls, as required by the API. JSONL traces redact the reasoning text
-by default while retaining its length and hash. Each event includes a schema
-version, run ID, sequence, and elapsed time; model context growth and tool-call
-latency are recorded as explicit events. Use `--trace-reasoning` only if you
-intentionally want to store raw reasoning.
+after tool calls, as required by the API. The default `--trace-level summary`
+records compact context metrics. `--trace-level debug` additionally records the
+complete sanitized provider payload and reasoning text for every model turn.
+Summary traces replace reasoning with its length and hash. Both levels redact
+common API keys, authorization values, tokens, passwords, and secrets. Debug
+traces contain sensitive model context and must be handled accordingly.
+
+Capture a replayable debug trace during an evaluation:
+
+```bash
+uv run yada eval \
+  --case benchmarks/swebench_verified/pytest-10051 \
+  --agent yada \
+  --yes \
+  --trace-level debug
+```
 
 Inspect a completed or interrupted run without manually scanning JSONL:
 
 ```bash
-uv run yada-trace .yada/runs/20260801T120000.000000Z.jsonl
+uv run yada-trace \
+  .yada/runs/fix-parser-edge-case__2026-08-02_12-26-26.123456Z.jsonl
+uv run yada-trace \
+  eval-results/pytest-dev__pytest-10051__2026-08-02_12-26-26.123456Z.artifacts/yada-trace.jsonl \
+  --step 8
+uv run yada-trace eval-results/<task>__<UTC-time>.artifacts/yada-trace.jsonl \
+  --verbose
 ```
 
 The report correlates model requests, tool-call IDs, errors, reminders, and the
-final verification state into a compact timeline. The source JSONL remains the
-durable, streaming-friendly record.
+final verification state into a compact timeline. `--step` and `--verbose`
+expand sanitized model messages, tool arguments, patches, stdout, and stderr.
+The source JSONL remains the durable, streaming-friendly record. Debug traces can
+contain source code and test output even after secret redaction, so handle them as
+sensitive artifacts. See [docs/tracing.md](docs/tracing.md) for the event
+reference, field-presence semantics, lifecycle, and `jq` recipes.
 
 ## Safety model
 
@@ -208,7 +229,8 @@ tests/
 ├── evals/
 ├── models/
 ├── tools/
-└── traces/
+├── traces/
+└── utils/
 ```
 
 `Planner` owns conversation policy and validates the next action without I/O.
@@ -230,8 +252,8 @@ and DeepSeek's official [thinking-mode](https://api-docs.deepseek.com/guides/thi
 and [tool-call](https://api-docs.deepseek.com/guides/tool_calls) contracts. The
 implementation is original and intentionally smaller than those systems.
 
-See [docs/architecture.md](docs/architecture.md) for the detailed contracts and planned
-ablations.
+See [docs/architecture.md](docs/architecture.md) for the detailed contracts and
+planned ablations, and [docs/tracing.md](docs/tracing.md) for the trace schema.
 
 ## Current non-goals
 

README.zh-CN.md

@@ -97,18 +97,33 @@ DeepSeek 工具调用
 - `finish`：最新修改后没有成功测试或构建时直接拒绝。
 
 DeepSeek 思考模式要求工具轮次继续回传 `reasoning_content`。Yada 会在内存中
-保留并正确回传，但 JSONL 轨迹默认只记录长度和 Hash；每个事件还带有
-`schema_version`、`run_id`、严格递增序号与累计耗时，并记录上下文增长、
-模型错误和工具耗时。只有显式使用 `--trace-reasoning` 才会落盘完整推理。
+保留并正确回传。默认的 `--trace-level summary` 只记录紧凑的上下文指标；
+`--trace-level debug` 还会保存每轮模型请求的完整脱敏 provider payload。
+JSONL 默认只保留 reasoning 的长度和 Hash，并脱敏常见 API key、
+Authorization、token、password 和 secret。只有显式使用
+`--trace-reasoning` 才会落盘完整推理。
+
+在评测中生成可还原的 debug trace：
+
+```bash
+uv run yada eval \
+  --case benchmarks/swebench_verified/pytest-10051 \
+  --agent yada \
+  --yes \
+  --trace-level debug
+```
 
 无需手工翻阅 JSONL，可以直接生成关联后的诊断时间线：
 
 ```bash
 uv run yada-trace .yada/runs/20260801T120000.000000Z.jsonl
+uv run yada-trace eval-results/<run>.artifacts/yada-trace.jsonl --step 8
+uv run yada-trace eval-results/<run>.artifacts/yada-trace.jsonl --verbose
 ```
 
-报告会汇总模型轮次、工具调用 ID、失败、协议提醒和最终验证状态；JSONL 仍然是
-可流式写入、崩溃后可恢复检查的原始记录。
+报告会汇总模型轮次、工具调用 ID、失败、协议提醒和最终验证状态。
+`--step` 和 `--verbose` 会展开脱敏后的模型消息、工具参数、Patch、stdout 和
+stderr。Debug trace 脱敏后仍可能包含源码和测试输出，应当作敏感 artifact 处理。
 
 ## 安全边界
 

docs/architecture.md

@@ -89,15 +89,25 @@ verification gate; only `test` and `build` do.
 ## Trace diagnostics
 
 JSONL is the crash-safe source of truth, but it is not the debugging interface.
-Each event includes `schema_version`, `run_id`, `sequence`, UTC time, and elapsed
-milliseconds. Model requests capture message count and serialized context size;
-responses capture provider metadata and latency; tool calls and results share a
-`tool_call_id`; `plan_decision` records the side-effect-free policy outcome.
-`model_error` leaves an explicit failed request in interrupted runs.
-
-`yada-trace PATH` validates this stream and renders a bounded timeline with run
-outcome, tool failures, protocol reminders, and per-step durations. It never
-echoes full prompts, arguments, reasoning, or command output into the summary.
+Schema v2 has two capture levels. `summary` records context size and event timing;
+`debug` also stores the sanitized provider payload built by the same client method
+used for the HTTP request. Responses, planner decisions, tool calls, and tool
+results remain correlated by step, request ID, and tool-call ID. `run_start`
+records Yada version/commit, workspace base commit, case ID when available, and
+the model configuration.
+
+`yada-trace PATH` renders the compact summary. `--step N` expands one request →
+response → tools slice, while `--verbose` expands the full timeline. Reasoning
+is length/hash-redacted in summary traces and automatically retained in debug
+traces. Common secret keys and bearer/API-key-like text are redacted in both
+modes. A debug trace can still contain reasoning, source code, and test output
+and must be handled as a sensitive artifact.
+The complete event and field reference lives in [tracing.md](tracing.md).
+
+The MVP stores a full sanitized request snapshot per turn. This deliberately
+favors deterministic inspection over delta complexity; content-addressed prompts
+or message deltas can replace it later if measured trace size justifies the added
+reader and compatibility cost.
 
 ## Security boundary
 

docs/evaluation.md

@@ -107,6 +107,9 @@ benchmark uses uv.
 ## Agent adapters
 
 The native adapter constructs Yada in process and records a normal JSONL trace.
+Use `--trace-level debug` to capture sanitized provider payloads for replayable
+step inspection, then use `yada-trace PATH --step N` or `--verbose` to expand it.
+The default `summary` level keeps request storage compact.
 It respects step and output-token budgets. A deployment-level container or
 process supervisor should enforce the hard wall-time limit.
 

docs/tracing.md

@@ -0,0 +1,153 @@
+# Yada trace reference
+
+Yada records one append-only JSON object per line. JSONL is the durable source of
+truth: a completed write remains inspectable if the agent later crashes, and the
+file can be streamed with ordinary tools. `yada-trace` is the terminal view over
+that source data.
+
+## Record envelope
+
+Schema v2 records have these top-level fields:
+
+| Field | Meaning |
+| --- | --- |
+| `schema_version` | Integer format version. Current value: `2`. |
+| `run_id` | Correlates every record produced by one agent run. |
+| `sequence` | One-based event order within the run. |
+| `timestamp` | UTC ISO 8601 wall-clock timestamp. |
+| `elapsed_ms` | Milliseconds since the `TraceWriter` was created. |
+| `event` | Event type described below. |
+| `data` | Event-specific object. |
+
+Readers must ignore unknown `data` fields. Adding an optional field does not
+require a schema-version increase; changing or removing an existing field does.
+`TraceWriter` accepts extension event names, but the ten names below are Yada's
+core event vocabulary.
+
+## Lifecycle and correlation
+
+```text
+run_start
+  step 1..N
+    model_request ──→ assistant ──→ plan_decision
+          │                              ├─→ protocol_reminder
+          └─→ model_error                ├─→ protocol_violation
+                                         └─→ tool_call ──→ tool_result
+run_end
+```
+
+- `step` groups one model request, its response, planning decision, and tools.
+- `request_id` joins `model_request` to `assistant` or `model_error`.
+- `tool_call_id` joins each `tool_call` to its `tool_result` and to the next
+  request's `role=tool` message.
+- Multiple tool calls can occur in one step and retain model-provided order.
+- A transport exception or process interruption can leave a trace without
+  `run_end`. Readers should report this as interrupted, not successful.
+
+## Core events
+
+| Event | Cardinality and meaning | Important `data` fields |
+| --- | --- | --- |
+| `run_start` | Once, before the loop. | `model`, `task`, `workspace`, `max_steps`, `trace_level`, `model_config`, `provenance` |
+| `model_request` | Once per attempted model turn. | `step`, `request_id`, `context`; debug traces also contain `payload` and `capture` |
+| `assistant` | Once after a successful model request. | `step`, `request_id`, `duration_ms`, `message`, `usage`, response metadata, `finish_reason` |
+| `model_error` | Instead of `assistant` when a request raises. | `step`, `request_id`, `duration_ms`, `error_type`, `error` |
+| `plan_decision` | Once after each `assistant`. | `step`, `action`, ordered `tools`, `rejection_error` |
+| `protocol_reminder` | When the assistant returns no tool call. | `step`, `text` |
+| `protocol_violation` | When a tool batch is rejected before side effects. | `step`, `error`, `call_count` |
+| `tool_call` | Before every attempted tool execution. | `step`, `tool_call_id`, `tool`, `arguments`; invalid calls use `raw_arguments`, rejected calls use `rejected` |
+| `tool_result` | After every `tool_call`, including rejected calls. | `step`, `tool_call_id`, `tool`, `duration_ms`, `result` |
+| `run_end` | Once after a graceful terminal outcome. | `finished`, `steps`, `summary`, accumulated `usage`, `final_state` |
+
+`tool_result.data.result.ok` means that the Yada tool protocol completed. For
+`run_command`, a process can still have a non-zero `exit_code`; check both fields
+when looking for failures.
+
+## Assistant message fields
+
+An assistant response is a compound message:
+
+- `content` is optional user-facing narration.
+- `reasoning_content` is DeepSeek thinking state.
+- `tool_calls` contains requested actions.
+
+A tool-calling response commonly has non-empty `reasoning_content`, one or more
+`tool_calls`, and `content=""`. That is a valid model response: the tool call is
+the action for the turn. Yada appends the whole assistant message to the
+conversation so DeepSeek reasoning is available on subsequent tool-call turns.
+
+The DeepSeek adapter normalizes an omitted `role` to `"assistant"` and an omitted
+`content` to `""`. Use `finish_reason` and `tool_calls` to interpret an empty
+content value: on a tool-calling turn, the tool call is the model's action.
+
+## Capture levels and sensitive data
+
+`--trace-level summary` records event timing, response data, tool activity, and
+compact request-context metrics. `--trace-level debug` additionally records the
+sanitized provider request payload built by the same client method used for the
+HTTP request.
+
+Summary traces replace reasoning with its character count and SHA-256 digest.
+Debug traces automatically retain reasoning because it is essential for
+understanding intermediate tool-calling turns where `content` may be empty.
+API-key-like fields and bearer, token, password, credential, and secret text are
+redacted in both capture levels. Redaction does not make a trace public: debug
+traces can contain reasoning, prompts, source code, patches, paths, and test
+output.
+
+## Inspection recipes
+
+Default paths include a sanitized task name and a readable UTC timestamp. A
+direct run resembles
+`.yada/runs/fix-parser-boundary-issue__2026-08-02_12-26-26.123456Z.jsonl`.
+An evaluation stores its trace under a directory such as
+`eval-results/pytest-dev__pytest-10051__2026-08-02_12-26-26.123456Z.artifacts/`.
+Explicit `--trace`, `--output`, and `--artifact-dir` values are never renamed.
+
+Render a run summary or expand one step:
+
+```bash
+uv run yada-trace TRACE.jsonl
+uv run yada-trace TRACE.jsonl --step 12
+uv run yada-trace TRACE.jsonl --verbose
+```
+
+List event counts:
+
+```bash
+jq -r '.event' TRACE.jsonl | sort | uniq -c
+```
+
+Show the exact sanitized request payload captured for step 12:
+
+```bash
+jq 'select(.event == "model_request" and .data.step == 12) | .data.payload' \
+  TRACE.jsonl
+```
+
+Show the response channels for every model turn:
+
+```bash
+jq 'select(.event == "assistant") |
+  {step: .data.step,
+   finish_reason: .data.finish_reason,
+   content: .data.message.content,
+   reasoning_content: .data.message.reasoning_content,
+   tool_calls: .data.message.tool_calls}' TRACE.jsonl
+```
+
+Find rejected tools and commands with non-zero exit codes:
+
+```bash
+jq 'select(.event == "tool_result" and
+  (.data.result.ok == false or ((.data.result.exit_code // 0) != 0))) |
+  {step: .data.step, tool: .data.tool, result: .data.result}' TRACE.jsonl
+```
+
+## Compatibility
+
+The reader accepts legacy records and rejects schema versions newer than it
+understands. Event order and correlation identifiers are the replay contract;
+timestamps are diagnostic and should not be used to reconstruct missing events.
+Yada currently records enough data for deterministic inspection, not automatic
+re-execution of model or shell side effects.