fix(run-limits): count max_turns as model API calls on every harness and enforce it on claude-code

.claude/notes/agents.md

@@ -61,14 +61,12 @@ intentionally brief and out of scope; trimming for DISPLAY belongs in the render
 
 - **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same
   thing on every backend, so a divergence is either fixed or documented — never silent.
-  **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool
-  calls, read live off the shared `EventCollector.visible_turn_count`, the same list
-  `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both,
-  so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit
-  (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT
-  the same budget across harnesses. OpenCode and Pi each keep a native unit too, because
-  their CLIs stream a real multi-step loop per `communicate()`
-  (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same
+  **`run_limits.max_turns` counts main-thread model API calls on every harness**, per
+  iteration (each retry and dialog exchange starts at zero). Codex and Antigravity run one
+  SDK turn per `communicate()`, so each counts calls from its own stream (Codex
+  `thread/tokenUsage/updated`, Antigravity MODEL steps). claude-code keeps the CLI's
+  `--max-turns` plus a backstop that counts main-thread `message_id`s. OpenCode and Pi
+  stream one `step_start` / `turn_start` per call. The cap is enforced on the same
   loop boundary as the cooperative early stop and finalizes cleanly as
   `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in
   `_drain()`, so the background-work poll loop honors it too.

docs/DIALOG_MODE.md

@@ -121,13 +121,15 @@ After each exchange the driver evaluates the stop conditions **in this order**,
 2. **`stop_on_criteria_pass`** (`criteria_passed`) — every success criterion passes. Requires
    per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default
    `end_of_dialog` is rejected at load time, since there would be nothing to check against.
-3. **`max_turns`** (`max_turns`) — the hard cap on exchanges. The agent exhausting its *own* inner
-   `max_turns` mid-exchange ends the dialog with the same reason.
+3. **`max_turns`** (`max_turns`) — the hard cap on exchanges.
 4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
    dialog ends and the task is **still scored** — unlike
    [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject
    agent only and aborts.
-5. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
+5. **`run_limits.max_turns`** (`agent_max_turns`): the agent used up its own model-call cap inside
+   one exchange. That cap restarts on every exchange, so this reason means one exchange ran long, not
+   that the dialog ran out of exchanges.
+6. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
    another message; the sentinel token in *that fresh utterance* ends the dialog. This is the
    workhorse in practice — the simulator decides, in character, that it got what it wanted — but it
    is evaluated **last**, so a turn that trips `max_turns` or the budget never gets the chance to

docs/TASK_DEFINITION_GUIDE.md

@@ -240,7 +240,7 @@ valid and an empty block is legal — every field defaults to "no limit".
 ```yaml
 run_limits:
   # Structural caps
-  max_turns: 20                       # hard cap on agent inner-loop turns per iteration
+  max_turns: 20                       # hard cap on model API calls per iteration
   expected_turns: 8                   # SOFT efficiency budget (visible turns) — never aborts
   task_timeout: 300                   # wall-clock cap for the full run envelope, seconds
   turn_timeout: 300                   # per-communicate() timeout, seconds
@@ -255,8 +255,8 @@ run_limits:
 
 | Field | Default | Constraint | Description |
 |-------|---------|------------|-------------|
-| `max_turns` | *unset* | `> 0` | Hard cap on agent inner-loop turns per iteration. Unset uses the SDK default. |
-| `expected_turns` | *unset* | `>= 1` | **Soft** target for cumulative visible turns. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
+| `max_turns` | *unset* | `> 0` | Hard cap on main-thread model API calls per iteration, Claude Code's turn, counted the same on every harness. The tools the last allowed call asks for still run; the turn ends when the next call begins. Each retry and each dialog exchange starts a fresh count. Unset uses the SDK default. See [HARNESS_PARITY.md](agents/HARNESS_PARITY.md). |
+| `expected_turns` | *unset* | `>= 1` | **Soft** target for visible turns (tool calls plus the final reply) summed over the whole task, a different unit from `max_turns`. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
 | `task_timeout` | *unset* | `>= 30` | Max seconds for the full run envelope, including agent work, grading, and post-run work. |
 | `turn_timeout` | *unset* | `>= 10` | Max seconds for the agent's single `communicate()` iteration. |
 | `max_input_tokens` | *unset* | `>= 1` | Max cumulative input (prompt) tokens. |
@@ -327,8 +327,8 @@ that did: a budgeted task that failed counts as over budget, while tasks with no
 `expected_turns` budget are excluded entirely (success or fail).
 
 The count compared against the budget is **visible turns** — one per tool call
-plus one for the agent's final reply — *not* the SDK's `total_turns` (which
-counts assistant messages and can bundle several tool calls into one).
+plus one for the agent's final reply. It is *not* `total_turns`, which counts
+model API calls (the `max_turns` unit), and one call can batch several tool calls.
 
 Set it to the number of turns a competent agent should need for the task. Pick
 budgets consistently across a suite — the headline % is only comparable when
@@ -1721,7 +1721,7 @@ The simulator runs as a tools-disabled Claude Code agent on its own resolved `Ap
 **Semantics:**
 
 - The task's `initial_prompt` is the user's *opening* message; the simulator picks up from turn 2.
-- `max_turns` is the intra-dialog cap (the worst-case agent call budget per trial). Use `n_trials` for variance sampling.
+- `max_turns` caps exchanges. Each exchange also gets a fresh `run_limits.max_turns` of model API calls, so the worst case per trial is the product of the two. Use `n_trials` for variance sampling.
 - The `reference` solution, if present, is hidden from the simulator (same security posture as for the coding agent).
 - When `n_trials > 1`, each trial becomes its own `ResolvedTask` with its own zero-padded replicate directory (`runs/<ts>/<variant_id>/<task_id>/<NN>/`) and its own `task.json` — the same fan-out mechanism as experiment `repeats`, which `n_trials` takes precedence over when simulation is enabled. Trial-level metadata appears under `simulation.replicate_index` / `simulation.n_trials` on the `EvaluationResult`.
 

docs/agents/ANTIGRAVITY.md

@@ -190,8 +190,9 @@ as every other agent.
 5. **`allowed_tools` / `disallowed_tools` are not read.** The harness runs with its
    full builtin tool set, so an Antigravity run has tools (web search, subagents,
    URL fetch) that the same task file denies on Claude Code and Codex.
-6. **`max_turns` counts visible turns.** One `communicate()` is a single SDK turn here,
-   so the cap counts resolved tool calls instead, enforced on the step loop. See
+6. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
+   here, so the harness counts model API calls itself (a MODEL step at a new
+   `step_index` opens one) and enforces the cap on the step loop. See
    [Run-Limit Parity](HARNESS_PARITY.md).
 7. **Shell commands over ~10s are moved to the background.** The localharness has a
    10-second maximum synchronous wait; past it the command becomes a background task

docs/agents/CODEX.md

@@ -217,7 +217,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
-| **`max_turns`** | Native SDK turn cap (assistant messages) | Visible-turn cap (tool calls), enforced on the notification pump |
+| **`max_turns`** | Model API calls: the CLI's `--max-turns`, plus a harness backstop | Model API calls, counted per `thread/tokenUsage/updated` and enforced on the notification pump |
 | **Early stop** | Supported (cooperative `should_stop`, polled between messages) | Supported — polled after each streamed notification; the in-flight turn is interrupted best-effort |
 
 Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).

docs/agents/HARNESS_PARITY.md

@@ -12,7 +12,7 @@ This page is the contract for what each run limit means per harness, plus the sh
 
 | Limit | claude-code | codex | antigravity | opencode | pi | delegate |
 |---|---|---|---|---|---|---|
-| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) | native step cap (the CLI's own agent-loop steps) | native turn cap (the CLI's own `turn_start` agent-loop steps) | message-event cap (forwarded `message`-type SDK events, NOT tool calls or backend round-trips — the host exposes no round-trip boundary) |
+| `run_limits.max_turns` (main-thread model API calls on every harness, see below) | native CLI cap, plus a harness backstop when call N+1 begins | a call runs from its first item to its `thread/tokenUsage/updated`, and one that ran tools opens the next call there | a call runs from its first new MODEL step to the step carrying its usage | one `step_start` step | one `turn_start` turn | a call opens when the previous call's tools have all returned |
 | `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline checked both between reads and while blocked inside one (`asyncio.wait_for`); force-kills the host subprocess and drops the handle so the next turn respawns |
 | `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
 | `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) | cooperative `should_stop` (event granularity — Pi streams incrementally) | cooperative `should_stop`, polled per forwarded SDK event; the host is abandoned (no interrupt command exists) and a fresh one spawns for the next turn |
@@ -531,59 +531,57 @@ needed to drive it.
   when a generation begins. Nothing in the timing accounting reads it — the
   head and tail are measured from the first and last `AssistantMessage`
   instead, which is uniform across all five — so this is recorded rather than
-  fixed. It is NOT a `max_turns` hazard: `EventCollector.visible_turn_count` is
-  `len(self._commands)`, derived from `ToolEndEvent`, and `_turn_starts` feeds
+  fixed. It is NOT a `max_turns` hazard: no cap reads it, and `_turn_starts` feeds
   only `assistant_turn_count` on the no-`AgentEndEvent` fallback path. The real
   cost of normalizing it is that the event drives the live renderers, so moving
   it changes the turn boundaries users watch during a run.
 
 All three are deliberately deferred; see `c/time-bugs-audit.md` for the
 measurements.
 
-## `max_turns` counts visible turns on Codex and Antigravity
-
-A "visible turn" is one entry in the run's timeline: one resolved tool call. It is
-the unit `result_metrics.visible_turn_count` reports and the unit that lands in
-`TurnRecord.commands`. Both backends count it live off the shared
-`EventCollector.visible_turn_count`, so one `max_turns` value means one thing on
-both.
-
-They need their own counter because a native one would be meaningless: Codex and
-Antigravity each deliver exactly **one SDK turn per `communicate()` call**, so an
-SDK-level cap would clamp at 1 no matter what the task asked for.
-
-The cap is enforced on the same loop boundary as the cooperative early stop: the
-step or notification that reaches the cap is processed whole, and the next one is
-never pulled. The in-flight turn is then cancelled server-side (best effort) so
-the cap actually stops spend. A run cut this way finalizes cleanly as
-`max_turns_exhausted` — it is not a crash, and it is not retried.
-
-**claude-code keeps its native SDK cap.** That is a real, honored cap, so it is
-left alone rather than reimplemented in a different unit. Its unit is the SDK's own
-agent-loop turn, which absorbs an arbitrary number of *parallel* tool calls, so the
-same number bounds very different amounts of work: under a prompt that encourages
-batching, a cap of N here permits many more than N tool calls, where it buys exactly
-N on the other two.
-
-**OpenCode also keeps a native unit — its stream's own steps.** Unlike Codex and
-Antigravity, `opencode run` executes a real multi-step agent loop per invocation
-and streams it (`step_start` / `step_finish`), so the natural agent-loop unit
-exists and is honored: `max_turns: N` allows N complete steps and cuts the run
-when step N+1 begins, with the completed steps' tokens intact. A step is one
-assistant generation and may carry several tool calls — so, as with claude-code,
-the same number is a looser tool-call budget than on the visible-turn backends.
-
-**Pi keeps a native unit too — its `turn_start` agent-loop steps.** Like OpenCode,
-`pi -p --mode json` runs a real multi-step agent loop per invocation and streams it
-(`turn_start` / `turn_end`), so `max_turns: N` allows N complete turns and cuts the
-run when turn N+1 begins, with the completed turns' tokens intact. Pi streams
-incrementally, so the cut genuinely stops spend mid-run. A Pi turn is one assistant
-generation and may carry several tool calls — the same looser budget as claude-code
-and OpenCode.
-
-**So holding `max_turns` constant across harnesses does not hold the budget
-constant.** If you are A/B-ing across backends and the cap is close to binding, that
-is the number to distrust.
+## `max_turns` counts model API calls on every harness
+
+One turn is one main-thread model API call, the unit Claude Code's `--max-turns`
+counts. `max_turns: N` lets the agent make N calls and still runs the tools the Nth
+call asked for. The turn ends when call N+1 begins, so a reply that finishes within
+N calls completes normally. Sub-agent calls do not count. `TurnRecord.num_turns`
+reports the same count, and a turn the cap ended reads N+1, as the Claude Code CLI
+reports it.
+
+Each harness finds the call boundary in its own stream (the table above):
+
+- **claude-code** applies the cap in the CLI, and the harness reads the CLI's
+  `error_max_turns` stop. The CLI does not apply it on every route, so the harness
+  also counts distinct main-thread `message_id`s and ends the turn itself when call
+  N+1 begins.
+- **Codex** sends `thread/tokenUsage/updated` once per call, after that call's
+  tools finish. An item starts as its tool runs, so waiting for the next call's
+  first item would let one tool of call N+1 act. A call that ran tools therefore
+  opens the next call at its `tokenUsage`, since the results always go back to the
+  model, and the cap fires before call N+1 can run anything.
+- **Antigravity** attaches `usage_metadata` to one step per call, and the next call
+  opens with a MODEL step at a new `step_index`.
+- **OpenCode** and **Pi** stream one `step_start` or `turn_start` per call.
+- **Delegate**'s SDK has no round-trip marker, and a tool-only reply streams only
+  its tool call, with no text before it. So the next call opens when every tool the
+  previous call announced has returned, since those results go back to the model,
+  and the cap fires before call N+1 can run anything. A reply that announces
+  several tools before their results counts once. If a tool never returns, the
+  next call opens at the model's next text or new tool call instead.
+
+Every harness enforces the cap on the same loop boundary as the
+cooperative early stop, then kills or cancels the in-flight turn so the cap stops
+spend. A run cut this way finalizes cleanly as `max_turns_exhausted`. It is not a
+crash, and it is not retried.
+
+One call can carry several parallel tool calls, so `max_turns` bounds model calls,
+not tool calls. A model that batches does more work per turn, on every harness
+alike.
+
+The count is per iteration: each retry and each dialog exchange starts at zero. A
+dialog whose agent hits the cap inside an exchange ends with `stop_reason:
+agent_max_turns`. That is distinct from `max_turns`, the simulator's cap on
+exchanges.
 
 ### What a capped run looks like
 
@@ -595,12 +593,12 @@ The signals a capped run leaves behind, on every backend:
   `MAX_TURNS_EXHAUSTED` (reporting category `failed`, icon `M`). Never `ERROR`,
   and never retried.
 - `max_turns_exhausted: true` on the task record.
-- On Codex and Antigravity, the count of *resolved* tool calls the model itself
-  issued equals the cap. Two things can add a further *recorded* command, and
-  neither means the cap leaked:
-    - A tool call already in flight when the cap fires is force-closed and recorded
-      with `result_status: unknown` rather than dropped, so the trajectory shows what
-      was interrupted.
+- `num_turns` is the cap plus one.
+- The calls under the cap are recorded whole. Two things can add a further
+  *recorded* command, and neither means the cap leaked:
+    - A tool call from call N+1 that the harness saw before it stopped is
+      force-closed and recorded with `result_status: unknown` rather than dropped,
+      so the trajectory shows what was interrupted.
     - On Codex, a sub-agent's inner tool calls are recovered from its rollout after
       the pump stops, so the child's work and its tokens still reach the record. The
       cap bounds what the model was allowed to do, not what the record may explain.