fix(run-limits): count max_turns as model API calls on every harness and enforce it on claude-code

[bai-uipath]

Fixes #138. Fixes #193.

Why

run_limits.max_turns meant a different thing on each harness: agent-loop API calls on claude-code, opencode and pi, resolved tool calls on codex and antigravity, and streamed text chunks on delegate. The same number in the same task file capped a different amount of work depending on --type, and codex reported num_turns: 1 for every turn.

claude-code was also the only harness that left the cap entirely to its CLI. On some routes the CLI does not apply --max-turns (221 turns against a cap of 75 in #193), and nothing in the harness bounded the turn.

What

• One turn is one main-thread model API call on every harness, the unit Claude Code's --max-turns counts. max_turns: N allows N calls: the tools the last allowed call asks for still run, and the turn ends when call N+1 begins. Codex, Antigravity and Delegate move to this unit; claude-code, OpenCode and Pi already used it.
• num_turns reports the same count on every harness, and reads max_turns + 1 when the cap ended the turn, as the Claude Code CLI reports it.
• claude-code no longer trusts the CLI to stop. The harness counts main-thread API calls itself and ends the turn when call N+1 begins, which a CLI that honors the cap never makes. The turn finalizes as max_turns_exhausted with num_turns: N+1, the same record a working CLI leaves.

This changes behavior: a codex or antigravity task with max_turns: N now allows N model calls however many tool calls each one batches, and a delegate task is no longer cut after N text chunks.

Not in scope: mid-turn token and USD budgets, and the evalboard expected_turns comparison. That work stays on bai/claude-code-runaway-turns.

Testing

max_turns on every harness

tasks/run_limits/max_turns_cap.yaml with -D run_limits.max_turns=2, one live run per harness on the tempdir driver. Every run stopped at the cap: max_turns_exhausted: true, num_turns: 3, two tools executed, and the criteria were still graded.

Harness	Model	Tools executed	Files written	Final status
claude-code	claude-sonnet-4-6 (Bedrock)	2	step-01	SUCCESS
codex	gpt-5.6-terra (Azure)	2	step-01, step-02	SUCCESS
antigravity	gemini-3.5-flash	2, plus call 3's streamed tool recorded as unknown (it never ran)	step-01, step-02	SUCCESS
opencode	us.anthropic.claude-sonnet-4-6 (Bedrock)	2	step-01	SUCCESS
pi	us.anthropic.claude-sonnet-4-6 (Bedrock)	2	step-01, step-02	SUCCESS
delegate	virtuoso-1-5 (alpha)	2	step-01	SUCCESS

The first delegate run executed a third tool: a tool-only reply streams no text event, so that call went uncounted until its tool had run. That is fixed here, and the re-run above stopped after two. Delegate also no longer stops counting when a tool never returns; a re-run with that change stopped the same way.

The fixture asks for one tool call per reply, so it checks where the cap stops rather than how batched calls count; the unit tests cover batching.

claude-code backstop (#193)

The runaway has not reproduced on a live route, so these runs simulate it: the CLI is launched without --max-turns while the harness keeps max_turns: 2. Same fixture, claude-sonnet-4-6 on Bedrock.

Run	CLI given the cap	Model calls	Tools executed	Files written	Fixture score
this PR	yes	3, CLI stopped with error_max_turns	2	step-01	1.0
this PR	no	3, harness ended the turn	2, plus call 3's streamed tool recorded as unknown (it never ran)	step-01	1.0
this PR, no turn_timeout	no	3, harness ended the turn and killed the CLI	2, plus call 3's streamed tool recorded as unknown (it never ran)	step-01	1.0
before this PR	no	25	24	step-01 to step-12	0.5

When the CLI honors the cap, it stops first and the backstop never fires. Without the backstop, the uncapped CLI ran the prompt to completion and reported subtype: success, the shape in #193; the fixture's "the cap bound the run" check failed.

The turn_timeout row covers a task with no turn timeout. The harness used to keep a handle on the CLI process only under a timeout, and closing the SDK's stream does not end the CLI, so in that case the turn ended while the CLI kept running. Across two runs it wrote step-02 after the cap and outlived the coder-eval process. The harness now keeps the handle whenever max_turns is set.

Dialog mode

A three-exchange simulated-user dialog under max_turns: 3 (scratch fixture). Exchanges 1 and 2 each need two calls, and exchange 3 asks for a long sequential chain.

Harness	Exchange 1	Exchange 2	Exchange 3
claude-code, codex, opencode, pi, delegate	2 calls	2 calls	cap hit, num_turns: 4, 3 tools, dialog ended with stop reason max_turns
antigravity	2 calls	2 calls	2 calls, then a 4th exchange of 1 call; the simulated user went off script, so the chain was never requested

The count restarts on every exchange and does not replay earlier exchanges. The antigravity user went off script because antigravity's agent_output includes the user's own message, so the simulator sees its message echoed back. That is the same on main and is not changed here.

🤖 Generated with Claude Code

[github-actions]

Claude finished @bai-uipath's task in 1m 18s —— View job

---

Code Review in Progress

Todo List:

• Read code review guidelines from .github/code_review.md
• Read CLAUDE.md for project conventions
• Review the full diff (git diff origin/main...HEAD)
• Read complete changed files for context
• Perform cross-file consistency checks
• Check "what's missing" from the implementation
• Provide comprehensive review feedback

[uipreliga]

Review: coder_eval_uipath — pr:194 (16 files) axis:1,2,3,4,5,6,7,8

Scope: pr:194 (16 files) axis:1,2,3,4,5,6,7,8 · branch bai/claude-code-turn-cap-backstop · 8a6f1fd · 2026-09-23T03:27Z · workflow variant

Change class: complex — changes max_turns semantics (counts model API calls) and turn-ending control flow in four agent adapters, incl. killing the CLI on a backstop stop

This PR changes max_turns to count model API calls on every harness, and its security, type safety and test base are healthy (9.1/10); but the counters do not yet agree across harnesses: Antigravity counts sub-agent calls, Codex can stop the last allowed call's own tool, the cap-vs-early-stop tie gives a different result per harness, and the Claude CLI keeps running after a cooperative stop, so the same agent output can get a different final_status depending on the harness, and these defects must be fixed before merge.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.5 / 10	0	0	3	0	DelegateAgent._handle_event complexity rises from C(16) to D(24) with an inline call-counting state machine
2. Type Safety	9.9 / 10	0	0	0	1	New per-turn step-dedup set uses Any for the step index. This keeps an untyped SDK value in agent state.
3. Test Health	9.4 / 10	0	0	1	1	The guards that stop non-model items/steps from counting as API calls are never exercised: removing either guard fails no test, and either one can change final_status
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.5 / 10	0	0	1	0	Redefined TurnRecord.num_turns contract is applied differently by each adapter (pattern drift across agents)
6. Error Handling & Resilience	8.4 / 10	0	1	1	1	Cooperative-stop break leaves the Claude CLI running: the PR adds a kill to the max_turns break but not to the sibling should_stop break
7. API Surface & Maintainability	9.5 / 10	0	0	1	0	Docs, docstrings and notes still describe the old max_turns / total_turns unit and an 'SDK default' that some harnesses lack
8. Evaluation Harness Quality	7.5 / 10	0	1	3	0	Antigravity max_turns counts sub-agent MODEL steps as main-thread API calls, so a sub-agent run is capped early and num_turns is inflated

Overall Score: 9.1 / 10 · Weakest Axis: Evaluation Harness Quality at 7.5 / 10
Totals: 🔴 0 · 🟠 2 · 🟡 10 · 🔵 3 across 8 axes.

Blockers

1. [Axis 6] Cooperative-stop break leaves the Claude CLI running: the PR adds a kill to the max_turns break but not to the sibling should_stop break (src/coder_eval/agents/claude_code_agent.py:1147) — The PR's own diagnosis (commit 8a6f1fd, and the comment at line 1231: "Closing the SDK's query() stream does not end the CLI: it never closes the generator it wraps.") also applies to the cooperative-stop break a few lines below the new backstop: if should_stop is not None and should_stop(): state.stopped_early_hit = True ... break (lines 1147-1150). That break does not call _kill_transport. The transport is also built only on if timeout is not None or max_turns is not None: (line 1234), so a stop_early task with no timeout and no cap has no process handle to kill. The result: an armed early stop ends the turn in the harness, but the CLI keeps running and spending tokens after the stop. This is the same leak the commit describes as having 'outlived the coder-eval process', and it recurs on every stopped-early row. This matches the Axis 6 High anchor ('missing cleanup on a rare-but-reachable path'). The path existed before this PR, but the PR fixed only one of the two identical break paths in _pump_messages. Fix: build the transport when should_stop is not None too (pass that into _build_claude_query), and call self._kill_transport(self._active_transport) before the cooperative-stop break, the same way the backstop does. Add a test that asserts the kill on the stop-early path, like test_claude_agent_max_turns_backstop_ends_a_turn_the_cli_did_not_cap.
2. [Axis 8] Antigravity max_turns counts sub-agent MODEL steps as main-thread API calls, so a sub-agent run is capped early and num_turns is inflated (src/coder_eval/agents/antigravity_agent.py:794) — The new counter reads step_key = (getattr(step, "trajectory_id", "") or "", step.step_index) and if ssource == _SOURCE_MODEL and step_key not in self._seen_steps and not self._in_api_call: (lines 793-797), and it also counts if step.usage_metadata is not None: if not self._in_api_call: self.api_calls += 1 (lines 818-821). No part of this filters by trajectory. Sub-agent trajectories stream on the same receive_steps() iterator. .claude/notes/agents.md (tool-id fallback section) and test_id_less_tool_calls_in_different_trajectories_do_not_collide both rely on that: sub-agent steps arrive with their own trajectory_id. So every MODEL step and usage_metadata from a sub-agent (start_subagent -> Task) adds to api_calls. A sub-agent step's usage can also close the main call early, because it sets _in_api_call = False. The result is that max_turns_reached() (return self.max_turns is not None and self.api_calls > self.max_turns, line 765) can fire while the main thread is still under the cap. The turn then finalizes as MAX_TURNS_EXHAUSTED, and num_turns (num_turns=self.api_calls) over-reports. This breaks the contract that this PR writes in docs/agents/HARNESS_PARITY.md:547, 'Sub-agent calls do not count'. Claude Code excludes them by parent_tool_use_id, and Codex never sees child threads, so on Antigravity the same task and trajectory get a different final_status. Fix: record the main trajectory id (the first one seen, or the one the conversation owns) and count only steps from that trajectory. Add a test that interleaves sub-agent MODEL steps that carry usage and asserts they do not move api_calls.

Non-blocking, but please consider before merge

1. [Axis 1] DelegateAgent._handle_event complexity rises from C(16) to D(24) with an inline call-counting state machine (src/coder_eval/agents/delegate_agent.py:727) — The PR puts the API-call counting logic straight into the event dispatcher. On origin/main, radon cc gives DelegateAgent._handle_event C(16). At HEAD it gives D(24). The added branches are if state.api_calls == 0 and (event_type in _TEXT_EVENT_TYPES or event_type == "tool_call"): / elif state.results_incomplete and (event_type in _TEXT_EVENT_TYPES or (event_type == "tool_call" and _tool_id(msg) not in state.open_tools)): (lines 727-734), plus state.results_incomplete = bool(state.open_tools) / if not state.open_tools: state.api_calls += 1 in the tool_result arm (lines 755-756). Move this into one _TurnState method, e.g. state.observe_call_boundary(event_type, msg), that returns whether open tools must be force-closed. _handle_event then stays a pure type dispatcher near its previous complexity, and the counting rule can be tested on its own.
2. [Axis 1] Codex and Antigravity each copy the same open/close API-call counter, with different field visibility (src/coder_eval/agents/antigravity_agent.py:819) — This PR adds the same 'begin call on first model output, close on usage, count a usage-only call' state machine in two adapters. antigravity_agent.py:819-821 has if not self._in_api_call: self.api_calls += 1 / self._in_api_call = False. codex_agent.py:684-686 has if not self.in_api_call: self.api_calls += 1 / self.in_api_call = False. Codex wraps the open step in _open_api_call() (line 522), but Antigravity inlines it (lines 794-796). The flag is private in one adapter (_in_api_call) and public in the other (in_api_call). The inline copy also moves _AntigravityTurnState.process_step from C(13) to C(18). Add a small shared counter in agents/, e.g. ApiCallCounter.open() / .close_on_usage() / .count, and use it in both adapters. The rule then exists in one place, both adapters use the same names, and the branches leave process_step.
3. [Axis 1] Claude Code max_turns_hit flag duplicates a condition finalize already derives from num_turns (src/coder_eval/agents/claude_code_agent.py:642) — The backstop sets two values together: state.max_turns_hit = True and state.num_turns = len(state.main_turn_ids) (lines 1140-1141). It sets them only when len(state.main_turn_ids) > state.max_turns, and then it breaks, so no later ResultMessage can overwrite num_turns. finalize already has or (self.max_turns is not None and self.num_turns is not None and self.num_turns > self.max_turns) (line 641), which is always true in that case. The added or self.max_turns_hit (line 642) and the field (line 226) are therefore never the deciding condition, and nothing else reads them. Delete the flag and its assignment, and keep the num_turns > max_turns check as the single source of truth. This follows 'delete before you guard'.
4. [Axis 3] The guards that stop non-model items/steps from counting as API calls are never exercised: removing either guard fails no test, and either one can change final_status (src/coder_eval/agents/codex_agent.py:562) — The new counting guards if not self.in_api_call and root_type not in _NON_MODEL_ITEM_TYPES: (codex_agent.py:562, set at :91 _NON_MODEL_ITEM_TYPES = frozenset({"userMessage", "hookPrompt", "contextCompaction"})) and if ssource == _SOURCE_MODEL and step_key not in self._seen_steps and not self._in_api_call: (antigravity_agent.py:794) have no direct test. No test in tests/ contains a userMessage, hookPrompt or contextCompaction item, and no max_turns test sends a non-MODEL Antigravity step. I checked by mutation: with both guards removed, tests/test_codex_agent.py, test_antigravity_agent.py, test_early_stop.py and test_agent_golden_master.py still pass (551 passed). Failure scenario: the last allowed call is a final reply (no tools, so no next call is pre-opened), and a contextCompaction item (Codex) or a SYSTEM step at a new step_index (Antigravity) follows it before turn/completed. Without the guard this opens a phantom call, api_calls becomes max_turns+1, and the cap fires. A run that finished inside its cap then records MAX_TURNS_EXHAUSTED and num_turns=cap+1. Add one test per harness based on the existing test_a_final_reply_on_the_last_allowed_call_completes: append a trailing hookPrompt/contextCompaction item (Codex) or a _step(..., source="SYSTEM", step_index=N) (Antigravity), then assert max_turns_exhausted is False and num_turns == max_turns.
5. [Axis 5] Redefined TurnRecord.num_turns contract is applied differently by each adapter (pattern drift across agents) (src/coder_eval/models/results.py:398) — The PR changes the field description to "Main-thread model API calls in this communicate() call, the unit max_turns caps; max_turns + 1 when the cap ended the turn. None when the agent crashed before reporting it" (results.py:398-400). The adapters touched in the same PR do not agree on this contract. antigravity_agent.py:1025 sends num_turns=self.api_calls, so it can be 0 and it is reported on a crash. codex_agent.py:767 sends num_turns=max(self.api_calls, 1), floored at 1 and reported on a crash. delegate_agent.py:922 sends num_turns=None if crashed else max(state.api_calls, 1), which is None on a crash as the description says. A zero-call or crashed turn therefore gives 0, 1 or None depending on the harness, and run_record.py:89 (total_turns = sum((t.num_turns or 0) ...)) sums those values across harnesses. Choose one rule for the floor and one rule for crashes, and apply it in every adapter. A shared cross-agent contract test (a parametrized case per adapter, like the CE036 ContractCase pattern) would stop this drift. Ripple: result_metrics.py:149 ("the SDK's num_turns which counts assistant messages") and run_record.py:182 ("Distinct from total_turns (SDK num_turns)") still describe the old meaning, and Codex total_turns changes from 1 per iteration to N. That is a change in the cross-repo run.json meaning, so state it in the PR.
6. [Axis 6] Delegate counts an unmatched tool_result (late after force-close, or duplicate) as a new API call, which can set MAX_TURNS_EXHAUSTED on a run under the cap (src/coder_eval/agents/delegate_agent.py:755) — _handle_event does self._handle_tool_result(msg, state, emit); state.results_incomplete = bool(state.open_tools); if not state.open_tools: state.api_calls += 1 (lines 754-756). It does this whether or not the result matched an open call. _handle_tool_result handles an unmatched result on purpose (line 791 telemetry = state.open_tools.pop(tool_id, None) then it creates a synthesized row, 'id mismatch or unknown shape'). Here is how this goes wrong. A tool that the new branch at lines 729-735 force-closed (the model spoke before it returned) sends its result late. At that time open_tools is empty, so api_calls goes up by one for a model call that never began. The same happens for a re-emitted duplicate result. Each stray result overcounts by one, and if max_turns is not None and state.api_calls > max_turns (line 673) can then abandon the host and finalize MAX_TURNS_EXHAUSTED on a run that was under its cap. Fix: make _handle_tool_result return whether it matched an open call, and add to api_calls only when a MATCHED result closes the last open tool. Add a test for a late result after the force-close.
7. [Axis 7] Docs, docstrings and notes still describe the old max_turns / total_turns unit and an 'SDK default' that some harnesses lack (docs/agents/HARNESS_PARITY.md:542) — HARNESS_PARITY.md:542 now reads "## max_turns counts model API calls on every harness". The pages that link to it still describe the removed unit. docs/agents/CODEX.md:220 says "| max_turns | Native SDK turn cap (assistant messages) | Visible-turn cap (tool calls), enforced on the notification pump |". docs/agents/ANTIGRAVITY.md:193-194 says "6. max_turns counts visible turns. One communicate() is a single SDK turn here, so the cap counts resolved tool calls instead". .claude/notes/agents.md:63-65 says the cap is "read live off the shared EventCollector.visible_turn_count", and this PR deletes that property. A user who reads the Codex or Antigravity page sets a tool-call budget, but the field now caps model calls, and one call can carry many parallel tool calls. Update the three passages in this PR. Make them link to the parity section and do not repeat the semantics.
8. [Axis 8] Codex cap can fire while the last allowed call's own tool is still running, so that tool is interrupted and recorded as unknown (src/coder_eval/agents/codex_agent.py:689) — on_token_usage_updated opens call N+1 as soon as the usage event arrives: if self.call_ran_tools: self._open_api_call() (lines 689-690). max_turns_reached() (line 520) is then true on the pump's next check, and the pump runs _interrupt_active_turn() and breaks. The docstring at line 517 assumes 'Codex closes a call with its tokenUsage event only after that call's tools finish'. The codebase already handles the opposite order: golden scenario d_cross_flush_is_error streams item/started cmd_d -> thread/tokenUsage/updated -> item/completed cmd_d, and on_item_started notes that 'is_error is patched at item/completed even after the message is flushed'. With max_turns: N, when call N's usage arrives before its tool's item/completed, the cap interrupts call N's own tool. That tool is then force-closed with result_status: unknown, which breaks the documented guarantee that the tools the Nth call asked for still run (TASK_DEFINITION_GUIDE.md:258, HARNESS_PARITY.md:545). File-based criteria and command_executed criteria can then score the capped run differently from the same trajectory on claude-code. Fix: when self.open_tools is not empty at tokenUsage, wait to open the next call until the last open tool completes. Or make max_turns_reached() also require not self.open_tools. Add a capped variant of scenario d.
9. [Axis 8] num_turns / run.json total_turns changes unit for codex, antigravity and delegate, and REPORT_SCHEMA.md has no note of it (src/coder_eval/models/results.py:398) — The field description now reads "Main-thread model API calls in this communicate() call, the unit max_turns caps; max_turns + 1 when the cap ended the turn." Codex num_turns changes from a constant 1 to max(self.api_calls, 1). Antigravity changes from _assistant_turns to api_calls, and delegate changes from message_events to api_calls. run_record.py:89 sums this value into run.json's total_turns, which the evalboard (evalboard/lib/runs.ts:896) and the separate coder-eval-uipath nightly dashboard read. No wire format changes, so the DockerRunner image needs no lockstep rebuild. But every codex, antigravity and delegate row shows a step change in total_turns at this commit, and claude-code backstop rows report the harness count (state.num_turns = len(state.main_turn_ids)), not the CLI's. This PR also reverted the evalboard fix (38c3f4b) that stopped comparing total_turns against expected_turns. Fix: state the semantic change and the date it takes effect in docs/REPORT_SCHEMA.md (next to total_turns at line 86 and num_turns at line 209), and in the PR body, and tell the eval-runner owners. Also update the stale comment at run_record.py:182 ('SDK num_turns').
10. [Axis 8] Cap vs cooperative-stop tie resolves differently per harness: claude-code backstop and delegate report MAX_TURNS_EXHAUSTED, codex and antigravity report STOPPED_EARLY (src/coder_eval/agents/claude_code_agent.py:1139) — The new backstop if state.max_turns is not None and len(state.main_turn_ids) > state.max_turns: state.max_turns_hit = True ... break (lines 1139-1146) runs BEFORE if should_stop is not None and should_stop():. On a tie, stopped_early_hit is therefore never set and finalize reports MAX_TURNS_EXHAUSTED. Delegate has the same order (delegate_agent.py:673 checks the cap before should_stop). Codex (codex_agent.py pump: 'After the cooperative stop, so an armed early-stop wins a tie') and Antigravity (same comment, antigravity_agent.py step loop) use the opposite order. HARNESS_PARITY.md:572 now says 'Every harness enforces the cap on the same loop boundary as the cooperative early stop', which implies they behave the same. When an armed stop_early gate and the cap fire on the same event, the final_status and the early-stop attribution depend on the harness. Fix: move the backstop after the should_stop check in claude-code (and in delegate), or state the tie rule in HARNESS_PARITY.md. Add a tie test for claude-code like test_cooperative_stop_outranks_the_cap.

Nits

1. [Axis 2] New per-turn step-dedup set uses Any for the step index. This keeps an untyped SDK value in agent state. (src/coder_eval/agents/antigravity_agent.py:728) — Line 728 declares self._seen_steps: set[tuple[str, Any]] = set(), and line 793 fills it with step_key = (getattr(step, "trajectory_id", "") or "", step.step_index). Coerce the index with int(step.step_index) and declare the set as set[tuple[str, int]]. As an alternative, reuse the f"{trajectory_id}:{step.step_index}" string key that _handle_tool_call already builds at lines 832-833, and declare set[str]. The same class then has one typed key scheme, not two. This is a cold-path private helper, and process_step(step: Any) was already untyped before this PR, so the severity is Low.
2. [Axis 3] The new 'num_turns is the cap plus one' contract for every backend is not asserted for OpenCode/Pi, and the deleted cross-agent test was not replaced (docs/agents/HARNESS_PARITY.md:591) — The PR adds - num_turns is the cap plus one. (line 591) under "What a capped run looks like ... on every backend". It also deletes tests/test_visible_turn_cap.py, which was the one test that tied the cap definition to more than one agent. The PR adds num_turns == cap + 1 checks for Claude Code, Codex, Antigravity and Delegate. tests/test_opencode_agent.py:1376 and tests/test_pi_agent.py:457 check only max_turns_exhausted is True, never num_turns. The deletion is otherwise safe: the three visible_turn_count tests are moot because the property is gone, and supports_cooperative_stop for Codex/Antigravity is still covered through the gating tests in tests/test_early_stop.py:787-791. Add assert record.num_turns == max_turns + 1 to the OpenCode and Pi cap tests, or add one parametrized test across all harnesses, so the documented contract is checked everywhere it is claimed.
3. [Axis 6] Codex reports num_turns one too high when a turn ends (crash, interrupt or normal end) right after a tool call's usage event (src/coder_eval/agents/codex_agent.py:689) — on_token_usage_updated counts the next call early: if self.call_ran_tools: self._open_api_call() (lines 689-690). When the turn then crashes, is interrupted or is cancelled before that call streams anything, finalize still reports num_turns=max(self.api_calls, 1) (line 767). This includes a call that never began. The partial record then overstates API calls, while Claude Code and Delegate report None for a crashed turn. The TurnRecord.num_turns description says the value is None when the agent crashed before reporting it. Fix: on a crashed finalize, subtract the pending call when in_api_call is set and no item has arrived for it, or report None on crash as the other adapters do.

What's Missing

Parallel paths:

• 🟠 The PR adds a transport kill to the Claude Code max_turns backstop break, but not to the cooperative-stop (should_stop) break in the same _pump_messages loop. The transport is also built only when timeout or max_turns is set, so a stop_early-only turn has no process handle to kill. (trigger: src/coder_eval/agents/claude_code_agent.py) (restates: Axis 6: Cooperative-stop break leaves the Claude CLI running)
• 🟠 Claude Code excludes sub-agent calls by parent_tool_use_id, and Codex never sees child threads. The new Antigravity counter has no main-trajectory filter, so sub-agent MODEL and usage steps count toward max_turns. HARNESS_PARITY.md says 'Sub-agent calls do not count'. (trigger: src/coder_eval/agents/antigravity_agent.py) (restates: Axis 8: Antigravity max_turns counts sub-agent MODEL steps as main-thread API calls)
• 🟡 Codex and Antigravity check should_stop before the cap, so an early stop wins a tie. Claude Code (the new backstop) and Delegate check the cap first. A tie on the same event therefore gives MAX_TURNS_EXHAUSTED on two harnesses and STOPPED_EARLY on the other two, and HARNESS_PARITY.md states no tie rule. (trigger: src/coder_eval/agents/claude_code_agent.py) (restates: Axis 8: Cap vs cooperative-stop tie resolves differently per harness)
• 🟡 The redefined TurnRecord.num_turns contract ('None when the agent crashed') is applied in Delegate only. Antigravity sends api_calls, which can be 0 and is set on a crash. Codex sends max(api_calls, 1), also set on a crash. OpenCode (step_count) and Pi (turn_count) were not checked against the new unit. (trigger: src/coder_eval/models/results.py) (restates: Axis 5: Redefined TurnRecord.num_turns contract is applied differently by each adapter)
• 🟡 The per-agent pages that link to the new parity section were not updated. docs/agents/CODEX.md:220 and docs/agents/ANTIGRAVITY.md:193-195 still say max_turns counts visible turns / tool calls. .claude/notes/agents.md:62-73 still points at EventCollector.visible_turn_count, which this PR deletes. (trigger: docs/agents/HARNESS_PARITY.md) (restates: Axis 7: Docs, docstrings and notes still describe the old max_turns / total_turns unit)

Tests:

• 🟡 No test covers the non-model guards (Codex _NON_MODEL_ITEM_TYPES userMessage/hookPrompt/contextCompaction; Antigravity non-MODEL step source). A mutation that removes both guards passes the full suite (6071 passed). (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 3: The guards that stop non-model items/steps from counting as API calls are never exercised)
• 🟡 There is no cap-vs-cooperative-stop tie test for Claude Code or Delegate. Codex and Antigravity have test_cooperative_stop_outranks_the_cap, but the other two harnesses do not. (trigger: tests/test_agent.py) (restates: Axis 8: Cap vs cooperative-stop tie resolves differently per harness)
• 🟡 There is no test for a Codex capped run where tokenUsage arrives before the last allowed call's item/completed (a capped variant of golden scenario d_cross_flush_is_error). All TestMaxTurnsApiCallCap cases send item/completed first. (trigger: tests/test_codex_agent.py) (restates: Axis 8: Codex cap can fire while the last allowed call's own tool is still running)
• 🟡 There is no Antigravity test that interleaves sub-agent (different trajectory_id) MODEL steps that carry usage_metadata and checks that api_calls and max_turns_reached do not change. (trigger: tests/test_antigravity_agent.py) (restates: Axis 8: Antigravity max_turns counts sub-agent MODEL steps as main-thread API calls)
• 🟡 No Delegate test sends a late or duplicate tool_result after the new force-close branch. This leaves the unmatched-result api_calls overcount uncovered. (trigger: tests/test_delegate_agent.py) (restates: Axis 6: Delegate counts an unmatched tool_result (late after force-close, or duplicate) as a new API call)
• 🔵 The PR deletes tests/test_visible_turn_cap.py, the only cross-agent cap test, and does not replace it. The documented 'num_turns is the cap plus one on every backend' is not checked for OpenCode (test_opencode_agent.py:1376) or Pi (test_pi_agent.py:457). A parametrized contract test for every harness would replace it. (trigger: tests/test_visible_turn_cap.py) (restates: Axis 3: The new 'num_turns is the cap plus one' contract for every backend is not asserted for OpenCode/Pi)
• 🔵 No test asserts Codex num_turns on a turn that crashes or is interrupted right after a tool-running call's usage event. In that case the pre-opened phantom call is counted. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 6: Codex reports num_turns one too high when a turn ends right after a tool call's usage event)

Downstream consumers:

• 🟡 num_turns changes unit (Codex constant 1 -> api_calls, Antigravity _assistant_turns -> api_calls, Delegate message_events -> api_calls). run_record.py:88-89 sums it into run.json total_turns, which evalboard/lib/runs.ts:896/1280 reads. The stale descriptions at result_metrics.py:149 ('SDK num_turns counts assistant messages') and run_record.py:182 ('SDK num_turns') were not updated. (trigger: src/coder_eval/models/results.py) (restates: Axis 8: num_turns / run.json total_turns changes unit for codex, antigravity and delegate)
• 🔵 Consumers that compare turn counts across harnesses or against a budget read total_turns, which now has a different unit per harness: the evalboard turns-efficiency ratio (evalboard/lib/turns.ts, overview.ts) and html.py expected_turns_overage. Codex rows jump from 1 per iteration to N. Nobody has checked whether these charts and badges still mean the same thing. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 8: num_turns / run.json total_turns changes unit for codex, antigravity and delegate)

Nightly pipeline:

• 🟡 The PR does not say what happens to the nightly run or the coder-eval-uipath dashboards. For the same trajectory, max_turns now caps model API calls instead of tool calls or visible turns on Codex and Antigravity, so rows can change from SUCCESS to MAX_TURNS_EXHAUSTED or the reverse. total_turns also shows a step change at this commit. docs/REPORT_SCHEMA.md (total_turns line 86, num_turns line 209) has no note of either change. (trigger: docs/agents/HARNESS_PARITY.md) (restates: Axis 8: num_turns / run.json total_turns changes unit for codex, antigravity and delegate)
• 🔵 Existing task YAMLs under tasks/ and experiments/ that set max_turns were tuned against the old per-harness unit (tool calls / visible turns). Nobody has audited whether their caps are still suitable now that one API call can carry many parallel tool calls. (trigger: src/coder_eval/models/limits.py)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE068 break-without-kill in the Claude Code message pump: an AST rule over ClaudeCodeAgent._pump_messages (tests/lint/rules/ce068_pump_break_kills_transport.py, wired in tests/lint/runner.py). Every break statement inside the async for over the SDK query stream must be preceded, in the same if body, by a call to self._kill_transport(...). A # noqa: CE068 with a reason is the exception. Blind spot: it cannot prove the transport exists. Also flag _build_claude_query when it builds the transport under a condition that does not name every argument that can make the pump break (timeout, max_turns, should_stop). Prevents: Axis 6 high: the cooperative-stop break at claude_code_agent.py:1147-1150 leaves the CLI running. The PR added a kill only to the sibling max_turns break at 1145, and the transport is built only when timeout is not None or max_turns is not None (line 1234).
• [ce-lint] CE069 cap-vs-stop order: in each agent pump loop that calls both should_stop() and a cap check (max_turns_reached() or api_calls > max_turns / len(main_turn_ids) > max_turns), the should_stop() check must come first in source order. Scan src/coder_eval/agents/*.py with AST for If nodes in the same loop body. This makes 'an armed early stop wins a tie' a mechanical invariant, and HARNESS_PARITY.md can state it once. Prevents: Axis 8/3 medium: on a tie, the claude-code backstop (claude_code_agent.py:1139) and delegate (delegate_agent.py:673) report MAX_TURNS_EXHAUSTED, but Codex and Antigravity report STOPPED_EARLY.
• [ce-lint] CE070 TurnRecord num_turns argument shape: every num_turns= keyword passed to the collector/TurnRecord finalize in src/coder_eval/agents/*.py must be a call to one shared helper, e.g. num_turns_for(api_calls, crashed=...) in agents/ (floor and crash rule defined once). A bare self.api_calls, max(...) or literal fails. This follows 'delete before you guard': the helper removes the drift, and the rule stops a new adapter from inlining its own floor. Prevents: Axis 5 medium: num_turns is 0, 1 or None for a zero-call or crashed turn depending on the adapter (antigravity_agent.py:1025, codex_agent.py:767, delegate_agent.py:922). Also Axis 6 low: Codex reports a phantom pending call on a crash (codex_agent.py:689, 767).
• [ce-lint] Extend CE028/CE030-style doc-drift checks with a removed-symbol grep: a whole-tree @pytest.mark.lint test in tests/test_custom_lint.py that collects every public attribute/property name of EventCollector and fails when docs/ or .claude/notes/ name EventCollector.<x> for an x that no longer exists. Also add a fixed denylist of retired phrases for the max_turns unit ('visible turns', 'Visible-turn cap', 'counts resolved tool calls') in docs/agents/*.md. Prevents: Axis 7 medium: docs/agents/CODEX.md:220, docs/agents/ANTIGRAVITY.md:193-195 and .claude/notes/agents.md:63-65 still describe the removed visible-turn unit and the deleted EventCollector.visible_turn_count.
• [ruff] Lower ruff max-branches in pyproject.toml (now 25, line 315) toward 20 in steps, following the 'cap below the current value' rule, and add C901 with max-complexity = 20. Existing offenders keep a visible # noqa debt marker. A method that goes from C to D grade (DelegateAgent._handle_event 16->24) then fails make check unless the author adds the marker, which makes the reviewer see it. Prevents: Axis 1 medium: DelegateAgent._handle_event rises to D(24) with an inline call-counting state machine (delegate_agent.py:727). Also Axis 1 medium: _AntigravityTurnState.process_step goes from C(13) to C(18) (antigravity_agent.py:819).
• [ce-lint] CE071 no redundant bool flags set beside the value they summarize: when an assignment state.<flag> = True is followed in the same block by state.<field> = <expr> and finalize already tests <field> > <cap>, the flag adds nothing. This needs semantic judgment in general, so limit the rule to a narrow form: flag any *_hit attribute on an agent turn-state class that has exactly one reader, and that reader is an or term next to a comparison on the same guard. Record the blind spot in the docstring. If this is too narrow to earn a rule, record it in .claude/harness-candidates.md instead. Prevents: Axis 1 medium: max_turns_hit in claude_code_agent.py:226/642/1140 repeats the num_turns > max_turns condition.
• [ce-lint] pyright/CE rule: forbid Any inside the type arguments of a set[...] / dict[...] annotation on agent turn-state attributes (self._x: set[tuple[str, Any]]) in src/coder_eval/agents/. Coerce SDK values at the boundary instead. An AST check on AnnAssign targets that start with self. in agents/*.py, with an allowlist for existing offenders. Prevents: Axis 2 low: _seen_steps: set[tuple[str, Any]] at antigravity_agent.py:728.

Harness improvements (not statically reachable):

• Add a cross-agent parametrized max_turns contract test (tests/test_turn_cap_contract.py), like the CE036 ContractCase pattern. Each adapter (claude-code, codex, antigravity, delegate, opencode, pi) gives a scripted event stream per case: (a) the run is capped at N, so assert max_turns_exhausted and num_turns == N+1; (b) a final reply on call N followed by trailing non-model items (Codex userMessage/hookPrompt/contextCompaction, an Antigravity SYSTEM step), so assert not exhausted and num_turns == N; (c) a tie between cap and should_stop, so assert STOPPED_EARLY; (d) a zero-call or crashed turn, so assert the one shared num_turns value; (e) sub-agent model calls, so assert they do not count. Replace the deleted tests/test_visible_turn_cap.py with it, and make CE047 or a sibling check require a case per built-in AgentKind. Why not static: Each harness counts calls from its own runtime event stream. Only replaying a stream through the adapter shows whether guards, ordering and floors give the documented values. Prevents: Axis 3 medium (untested non-model guards, codex_agent.py:562 / antigravity_agent.py:794), Axis 3 low (no num_turns check on OpenCode/Pi), Axis 5 medium (num_turns drift), Axis 8 medium (tie order), Axis 8 high (Antigravity sub-agent steps counted, antigravity_agent.py:794).
• Add capped variants of the golden-master out-of-order scenarios to tests/test_agent_golden_master.py. Run scenario d_cross_flush_is_error (tokenUsage before item/completed) with max_turns set to the call that owns the tool. Assert that tool completes with its real status, not unknown. Add a Delegate stream with a late or duplicate tool_result after a force-close, and assert api_calls does not rise. Why not static: The defect depends on the order events arrive at run time. Only a fixture that replays the order can catch it. Prevents: Axis 8 medium: the Codex cap interrupts the last allowed call's own tool (codex_agent.py:689). Axis 6 medium: Delegate counts an unmatched tool_result as a call (delegate_agent.py:755).
• Add a process-leak assertion to the Claude Code agent tests. For every early-exit path in _pump_messages (max_turns backstop, cooperative stop, timeout), run with a fake transport and assert _kill_transport was called and the fake process got a terminate/kill. Include a stop_early case with no timeout and no max_turns. Why not static: Whether a live subprocess handle exists and is terminated is run-time state. CE068 checks the code shape, but only a test proves the handle was built and killed. Prevents: Axis 6 high: the cooperative-stop break leaves the Claude CLI running (claude_code_agent.py:1147).
• Add a mutation-check step for new counting guards to the review workflow (and optionally a make mutate-agents target with mutmut limited to the changed agent lines). A guard whose removal fails no test is reported as untested. Why not static: Only running the suite against mutated code shows whether a branch is asserted. Coverage alone marks the line as run. Prevents: Axis 3 medium: removing the non-model guards at codex_agent.py:562 and antigravity_agent.py:794 fails no test (6071 passed).
• PR template / review checklist item: 'Does this change the meaning of a run.json / task.json field (for example num_turns, total_turns)? If yes, update docs/REPORT_SCHEMA.md with the change and its effective date, and tell the coder-eval-uipath eval-runner owners.' As an option, add a schema-changelog section to REPORT_SCHEMA.md that a docs test requires to change when the description of a field that feeds run.json changes (hash the Field descriptions in models/results.py). Why not static: A change of meaning with no wire-format change cannot be seen from the code shape. A description-hash test can only ask for a changelog entry; a person must judge whether the entry is right. Prevents: Axis 8 medium: the num_turns / total_turns unit changed with no REPORT_SCHEMA.md note (results.py:398, run_record.py:89/182).
• Extract a shared ApiCallCounter (open / close_on_usage / count) and a main-trajectory filter helper in agents/. Add unit tests on those helpers alone, so each adapter's counting rule can be tested apart from its event dispatcher. Why not static: This is a refactor for design and testability. Whether two state machines are the same needs semantic judgment, and a lint rule cannot detect that reliably. Prevents: Axis 1 medium: duplicated counters in Codex and Antigravity (antigravity_agent.py:819, codex_agent.py:522). Axis 1 medium: the complexity of Delegate._handle_event (delegate_agent.py:727).

Top 5 Priority Actions

1. Count only main-trajectory steps in the Antigravity counter at src/coder_eval/agents/antigravity_agent.py:794 and :818 (use the first trajectory_id seen, which is the SDK's convention), so sub-agent MODEL/usage steps do not add to api_calls, trigger MAX_TURNS_EXHAUSTED early or inflate num_turns; add a test that interleaves sub-agent steps that carry usage.
2. At src/coder_eval/agents/codex_agent.py:689-690, wait to open call N+1 until self.open_tools is empty (or make max_turns_reached() at :520 require no open tools), so the cap does not interrupt call N's own tool and record it as unknown; add a capped variant of golden scenario d_cross_flush_is_error.
3. Use one tie rule for all harnesses: move the max_turns backstop after the should_stop check in src/coder_eval/agents/claude_code_agent.py:1139 and src/coder_eval/agents/delegate_agent.py:673 (Codex and Antigravity already let early stop win), add a claude-code/delegate test_cooperative_stop_outranks_the_cap, and state the rule in docs/agents/HARNESS_PARITY.md:572.
4. Kill the Claude CLI on the cooperative-stop break at src/coder_eval/agents/claude_code_agent.py:1147-1150, as the backstop does at :1145, and also build the transport when should_stop is set (:1234), so a stopped-early CLI cannot keep spending tokens and change the sandbox while grading runs; add a test that asserts the kill.
5. Add to Delegate api_calls only when a matched tool_result closes the last open tool (src/coder_eval/agents/delegate_agent.py:754-756), and add the missing mutation-proof tests for the non-model guards (codex_agent.py:562, antigravity_agent.py:794), so a trailing contextCompaction/hookPrompt item or SYSTEM step cannot turn an in-cap run into MAX_TURNS_EXHAUSTED; then make num_turns use one floor/crash rule in all adapters (models/results.py:398) and record the total_turns unit change in docs/REPORT_SCHEMA.md.

---

Stats: 0 🔴 · 2 🟠 · 10 🟡 · 3 🔵 across 8 axes reviewed.

[uipreliga]

Fix what you agree with and 🚢

[uipreliga]

max_turns — this PR vs. the harness rewrite

Date: 2026-09-22

• A — this PR: ~/src/exp/coder_eval, branch bai/claude-code-turn-cap-backstop
  (commits ff566de1 … 8a6f1fd9). Diff: 16 files, +407 / −223.
• B — the rewrite: ~/src/coder_eval, branch feat/harness-turn-emitter-and-ports
  (46 commits over origin/main @ 396c22cc, 342 files, +21 144 / −13 521). The
  max_turns part is 3221e300 … 636c5ddd, 98d4c538 and a07fd67a.

Short answer

• Yes, B changed max_turns too. Your memory is half right. B did not change
  max_turns to count tool calls. B split the old limit into two limits:
  • run_limits.max_tool_calls (+ expected_tool_calls) — the OLD visible-turn cap
    (resolved main-thread tool calls). This is the "count tool calls" part.
  • run_limits.max_turns (+ expected_turns) — a NEW cap on main-thread model
    turns (one model response = one turn). This is the same unit as A.
• A and B agree on the unit: one main-thread model API call, sub-agents excluded,
  stop when call N+1 starts.
• They disagree on the scope, the mechanism and the recorded status (table below).
• B's architecture is better. It has one counter (TurnMonitor) on the
  should_stop channel, a harness contract that rejects the limit where it cannot be
  counted, and a separate tool-call cap.
• A's per-harness boundary detection is better in two places. A fixes two
  real defects that B still has: the Claude Code CLI keeps running after a cooperative
  stop, and Codex runs one tool of call N+1 before the cap fires. Port these two fixes
  to B.

Side-by-side

Aspect	A — this PR	B — rewrite
Unit of max_turns	Main-thread model API call	Main-thread model turn (one inner turn per model response) — same thing
Old "visible turn" cap (resolved tool calls)	Deleted (EventCollector.visible_turn_count removed)	Kept, renamed to max_tool_calls; expected_tool_calls added
expected_turns unit	Still visible turns (tool calls) — not the unit of max_turns	Model turns — same unit as max_turns
Scope	Per communicate() iteration. A retry or a dialog turn gets a fresh N	Cumulative per task: all retries and all dialog turns
Where it is counted	In each agent, ad hoc (api_calls counters, main_turn_ids)	Once, in TurnMonitor, from TurnStartEvent with parent_thread_id is None, deduplicated per turn id per communicate()
Where it is enforced	Each agent's own loop check (max_turns_reached(), api_calls > max_turns)	should_stop() poll; StopReason.MODEL_TURN_CAP, latched, precedence after EARLY_CRITERION and TOOL_CALL_CAP
claude-code	CLI native --max-turns plus a harness backstop that counts distinct main-thread message_ids and kills the CLI at call N+1	No native CLI cap. Only the TurnMonitor poll after each message. sdk_options.max_turns still passes through if a user sets it
codex	Call N+1 opens at thread/tokenUsage/updated of a call that ran tools, so the cap fires before call N+1 can start a tool	Call N+1 opens at its first item/started (or delta, or reasoning). A tool item starts as the tool runs, so one tool of call N+1 runs before the stop
antigravity	Call opens at the first new MODEL step (new step_index), closes at the step with usage_metadata	Generation opens on a MODEL step with new content, new usage, or an unseen tool call. Equivalent
opencode / pi	One step_start / turn_start per call (unchanged)	Same, through the emitter
delegate	Heuristic: call N+1 opens when all tools of call N have returned (or at the next text / new tool call if a tool never returns)	Agent does not exist on this branch
Harness with no boundary	Not possible to express	HarnessContract.counts_model_turns is False → HarnessContractError at resolution (only noop today)
Final status	AgentEndStatus.MAX_TURNS_EXHAUSTED, max_turns_exhausted: true	StopReason.MODEL_TURN_CAP → AgentEndStatus.TOOL_CALLS_EXHAUSTED, tool_calls_exhausted: true. MAX_TURNS_EXHAUSTED is deleted
Recorded count	TurnRecord.num_turns = calls; N+1 when capped, on every harness	EvaluationResult.model_turns (task-cumulative, None when the harness does not count). TurnRecord.num_turns keeps the old SDK meaning
Docs	Hand-written HARNESS_PARITY section	Generated parity table (make parity-table, CE069)
Fixtures	Unit tests per agent	Unit tests + tasks/run_limits/max_turns_cap.yaml fixture, resolution-gate tests

Pros and cons

A — this PR

Pros:

• Small and local. It is easy to review and easy to merge on main today.
• One unit on every harness, including delegate.
• The claude-code backstop keeps the CLI's own cap. The CLI stops before it sends
  call N+1, so on the routes where the CLI applies the cap there is no extra spend.
  The backstop only acts on routes where the CLI ignores the cap.
• Kills the CLI on a backstop stop even without turn_timeout (8a6f1fd9): it builds
  the SubprocessCLITransport when max_turns is set.
• Codex boundary is correct for tools: the cap fires at the tokenUsage of call N,
  before call N+1 can start a tool (on_token_usage_updated, call_ran_tools).
• num_turns means the same thing on every harness.

Cons:

• Five separate counters in five agents. There is no single source of truth. Each
  agent can drift.
• The old visible-turn cap is gone and there is no replacement. A task that wants to
  cap tool calls has no way to do it.
• expected_turns still counts visible turns (tool calls), while max_turns counts
  API calls. The two fields have the same name family and different units.
• Scope is per iteration. A crash retry or a dialog turn resets the count, so a
  dialog task with max_turns: 10 can make 10 × (dialog turns) × (attempts) calls.
• No resolution-time gate. A future harness with no call boundary would silently
  approximate or ignore the cap.
• The delegate count is a heuristic (see the bugs below).

B — rewrite

Pros:

• One counter, one enforcement point (TurnMonitor.should_stop). Every harness
  gets the cap by emitting TurnStartEvent; no agent has cap logic.
• Two caps with clear units: max_tool_calls and max_turns, each with a soft
  expected_* twin in the same unit.
• Task-cumulative scope. Retries and dialog turns cannot multiply the budget.
• The harness contract rejects max_turns / expected_turns where the harness cannot
  count model turns (_check_model_turn_limits). Fail loud, not fail open.
• Generated parity table (CE069); it cannot drift from the code.
• Sub-agent scoping is structural (parent_thread_id), not per-agent.

Cons:

• Very large change (342 files). The max_turns change cannot merge alone.
• claude-code no longer uses the CLI's native cap, so every capped run pays for the
  start of call N+1 (its input tokens at least) before the poll sees it.
• A model-turn stop is recorded as TOOL_CALLS_EXHAUSTED. A report cannot tell a
  tool-call cap from a model-turn cap (see the bugs below).
• No delegate agent on this branch, so no coverage for it.

Bugs that remain

In A (this PR)

1. expected_turns unit mismatch. max_turns counts API calls, expected_turns
   counts visible turns (tool calls). The docs say "visible turns" for one and "model
   API calls" for the other. A user who sets expected_turns: 8, max_turns: 10
   compares two different units.
2. No tool-call cap. Deleting EventCollector.visible_turn_count removed the only
   tool-call cap on Codex and Antigravity. No field replaces it.
3. Per-iteration scope. A crash retry and each dialog turn start a new count.
4. Codex item-type list is closed. _NON_MODEL_ITEM_TYPES is
   {userMessage, hookPrompt, contextCompaction}. A new non-model item type from a
   Codex upgrade opens a phantom call and can fire the cap early.
5. Codex overcount on a tool-ending turn. A call that ran tools opens call N+1 at
   its tokenUsage. If Codex then ends the turn without another model call (for
   example an interrupt or an error after the tools), num_turns is one too high, and
   at exactly N calls the cap fires on a call that never started.
6. Delegate heuristic. A reply that announces several tools counts once only if
   all tool calls arrive before their results. A reply that interleaves (call A,
   result A, call B) counts as two calls. A tool that never returns keeps the count
   frozen until the next text or new tool call.
7. Shared race (also in B). Claude Code streams tool execution. The
   AssistantMessage of call N+1 can arrive after its tool already started, so the
   backstop can still let one tool of call N+1 run on a route that ignored the cap.

In B (rewrite)

1. Claude Code CLI is not killed on a cooperative stop without turn_timeout.
   _build_claude_query builds the transport only when timeout is not None
   (claude_code_agent.py:1044-1046), and _pump_messages only breaks on a stop
   reason. Closing the SDK's query() stream does not end the CLI (the finding behind
   A's 8a6f1fd9). So on MODEL_TURN_CAP, TOOL_CALL_CAP, a budget or an early stop,
   the CLI keeps working and spending with no turn_timeout set. This affects every
   should_stop reason, not only max_turns. Port A's fix.
2. Codex runs one tool of call N+1. on_item_started calls _open_generation()
   for the first item of call N+1. For a tool item, item/started arrives when the
   tool already runs, so the monitor latches after the tool started. The interrupt is
   best effort. Port A's tokenUsage boundary (open the next call at the
   tokenUsage of a call that ran tools).
3. Codex phantom turns from non-model items. on_item_started opens a generation
   before it checks _CONTENT_ITEM_TYPES, so a mid-turn hookPrompt or
   contextCompaction item opens a new inner turn and counts as a model turn. (A's
   item 4 is the same class of problem, with an explicit list.)
4. Wrong end status for a model-turn stop. _END_STATUS_FOR_STOP maps
   MODEL_TURN_CAP to TOOL_CALLS_EXHAUSTED (streaming/events.py:93), and
   MAX_TURNS_EXHAUSTED no longer exists. Reports and the run record cannot tell which
   cap fired. Add a distinct status (or record the StopReason).
5. TurnRecord.num_turns keeps the old meaning. Its description is still "from
   ResultMessage.num_turns" (models/results.py:389-395), which is a Claude Code
   concept. The real count is EvaluationResult.model_turns. Two fields with near
   names and different meanings.
6. Extra spend on claude-code. B passes no native --max-turns, so the CLI always
   sends call N+1 before the poll can stop it. A keeps the CLI cap and uses the
   backstop only when the CLI ignores it.
7. Shared race — the same as A's item 7.

Recommendation

• Keep B's design (split caps, TurnMonitor, contract gate, cumulative scope,
  generated table).
• Port three things from A into B:
  1. Build the Claude Code transport when any stop can fire (not only on
     turn_timeout), and kill the CLI on every should_stop break (B bug 1).
  2. Codex: open the next model turn at the tokenUsage of a call that ran tools
     (B bug 2), and skip non-model item types before _open_generation() (B bug 3).
  3. Optionally pass run_limits.max_turns to the CLI as --max-turns as well, so
     the CLI stops before call N+1 where it can; the TurnMonitor stays the backstop.
     Note: B's cap is task-cumulative, so the CLI value must be the remaining turns
     for this communicate(), not N.
• Fix B bug 4 (distinct end status for a model-turn stop) before B merges.
• If A merges first as a stop-gap, fix A bug 1 (make expected_turns count API calls,
  or document the unit difference) and accept A bugs 2–3 as known until B lands.