Skip to content

fix: retry an interrupted Run like Cursor CLI - #103

Merged
oakimov merged 11 commits into
mainfrom
fix/cli-run-retry
Oct 18, 2026
Merged

oakimov merged 11 commits into
mainfrom
fix/cli-run-retry

Conversation

@oakimov

@oakimov oakimov commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Fixes #100.

Interrupted output-only Runs could end the turn, and plugin-appended reminders could reset the retry budget. I researched Cursor CLI's turn runner, stream errors and stall detector, and mimicked their retry eligibility, separate count limits and silence deadline.

Changes:

  • Share 10 transport retries and 3 server retries across the user turn, including host retries and tool steps. A stream ending without turn_ended uses the transport budget, as the CLI explicitly marks it. Server failures and the read-idle deadline use the server budget.
  • Keep OpenCode 2's canonical interruption notice on the original budget after a provider-requested retry, including trailing recognized reminder parts. Normalize identity only; Cursor still receives the actual notes. Attachments, ordinary follow-ups and genuine later requests remain distinct.
  • Treat an unfinished partial tool display row as output only until an exec or started tool lifecycle arrives. Close the unfinished row without executing a tool. Actual execs, opaque interactions and unknown frames retain their replay barriers.
  • Use the CLI's 30-second inbound-silence default. Every inbound frame renews it; outbound heartbeats do not. Retain the separate 120-second transport watchdog and explicit SDK override.
  • Resume the latest eligible checkpoint; capture tool-result cutoffs at actual serialized writes before drain. Track exec ownership across resumed Runs, including reused numeric ids. Rebuild from complete host history only for the guarded drop immediately after completed tools, preserving every result once.
  • Keep ambiguous uploads, final/auth errors, the two-resume no-progress cutoff and the 180-second consecutive-stall window guarded. CURSOR_ENDLESS_RETRIES=1 or true bypasses count limits; it does not bypass those safeguards or the host's own policy.
  • Document behavior in README and both host guides; keep changelog entries under the existing unreleased 0.8.3 section.

Validation:

  • Full coverage suite: 2,452 pass, zero failures. Typecheck/build, built architecture checks, Node HTTP/2 regressions and all 54 pricing fixtures pass.
  • Regressions cover reminder identity, physical EOF limits, unfinished rows, checkpoint/result ordering and preserved stateful barriers. The new failure cases were reproduced before their fixes.
  • Native OpenCode 1.19.3 and 2.0.32: CB-08 recovers in two Runs, with the original request once per Run and zero tool executions. OpenCode 2 also recovers at the actual 120-second transport cut with the semantic deadline explicitly raised.
  • Native EOF with changing reminders: OpenCode 2 reaches 11 Runs, ten retryable errors followed by status 400 with isRetryable: false and x-should-retry: false. OpenCode 1 stops after six Runs while provider errors remain retryable, under its own policy. Each Run carries the original request once; zero execs. Native retry delays were unchanged.

These native checks use the rebuilt provider through the hosts' AI SDK adapters and a local Cursor protocol stand-in. They do not validate the production plugin bootstrap or live Cursor service. The stock OpenCode 2 CLI exits 1 after successful recovery because of upstream #50062, with a pending fix in #52885. A no-row EOF control also recovers in two Runs and exits 1; a successful no-error baseline exits 0. We leave that CLI bug upstream.

@codecov

codecov Bot commented Oct 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking issue remains.

Summary

The PR adds turn-wide retry limits and recovery for interrupted Cursor Runs.

  • Interrupted Runs can retry safely and resume from eligible checkpoints.
  • Complete host tool results can rebuild a dropped Run from history.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Run fails] --> B{Safe to retry?}
  B -->|No| C[End turn]
  B -->|Yes| D{Failure type}
  D -->|Bare EOF or connection failure| E[Transport budget: 10 retries]
  D -->|Server error or read-idle| F[Server budget: 3 retries]
  E --> G{Budget available?}
  F --> G
  G -->|No| C
  G -->|Yes| H[Resume checkpoint, rebuild history, or request host retry]
  H --> A
Loading

Reviews (10) · Last reviewed commit: "fix: count incomplete streams as CLI tra..." · Reviewed by Greptile

Comment thread src/language-model.ts Outdated
Comment thread src/language-model.ts Outdated
@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Reran the #100 stub repro on a27f1d6 with OpenCode 2.0.31: a dropped stream and a read-idle cut after visible reasoning now continue the step, and the retry carries the lost request once (4 Runs, SAW lost=1/1 next=0/0 then lost=0/0 next=1/1, where c2a30d8 still ends turn 2 with the 400 after 3 Runs), the #97 error table is unchanged, and a stream that drops on every attempt stops after 11 attempts. From OpenCode's source: 1.17.13 to 1.18.16, inside the ^1.17.13 peer range, retry an isRetryable APIError with no attempt cap (RETRY_MAX_RETRIES = 5 arrived in 1.18.17), so on those versions a Run that drops after output on every attempt keeps retrying until the user stops it.

@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

One more case from our fork's runs: when the stream drops right after a tool result, before that attempt shows any text or reasoning, #103 keeps it final. On our fork the host retries it and the retry carries the lost request once without running the tool again. Not sure whether you want that covered here.

@oakimov

oakimov commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner Author

I researched Cursor CLI's retry behavior and mimicked its limits and safety cutoffs in 654e9fb. The CLI allows 10 transport retries and 3 server retries by default; output and checkpoints do not reset those counters. Letting OpenCode retry partial output without sharing that budget could otherwise loop indefinitely on older hosts.

Changes:

  • Share retry counts across provider recoveries, tool steps and host retries, including replacement connection failures.
  • Add CURSOR_ENDLESS_RETRIES=1 or true to bypass count limits. Stop, final errors and tool replay guards remain enforced, as do the CLI's two-resume no-progress cutoff and 180-second consecutive-stall retry window. OpenCode can still impose its own retry limit (mimics --endless-retries Cursor CLI option).
  • Document the behavior in the README and both host guides; keep the changelog under 0.8.3.

Regression coverage checks finite and endless retries, checkpoint progress, host retries and the safety cutoffs.

Comment thread src/language-model.ts
@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Reran on 4cce01a with OpenCode 2.0.31: the #100 drop and read-idle cases still continue with the lost request once, the error table still ends final (the capacity and undecodable custom-message refusals now take 4 Runs instead of 3), and run through 1.18.16's retry rules every exhausted-budget error comes out as a 400 with isRetryable: false, so 1.17.13 to 1.18.16 stop once the budget is spent. On 2.0.31 the every-attempt drop still stops after 11 attempts from OpenCode's own limit rather than the budget: each continuation adds OpenCode's "previous response was interrupted" user message, so retryTurnKey changes and every host retry starts a fresh budget. The drop right after a tool result is still final (automatic retry unsafe (attempt 1/14), 3 Runs), where our fork recovers it.

A Run that dropped or stalled after showing only text or reasoning, before
Cursor saved a checkpoint, ended the turn with a final error since #97, and
its request reached Cursor only with the next message.

Cursor CLI's Run retry loop sends the same action again when an attempt saved
no checkpoint, resumes the latest checkpoint otherwise, and stops after two
resumed attempts that stream without a new one. The provider cannot resend
after output the host already shows, so it hands that case to the host:

- An output-only failure with no checkpoint in the turn reaches the host as a
  retryable APICallError (x-should-retry: true, no status). OpenCode 1.x
  retries the step; OpenCode 2 continues it. The lost-request record carries
  the request once. Any stateful barrier stays final.
- The replay barrier records every reason, so text before a stateful frame is
  not mistaken for output only.
- An attempt without its own checkpoint resumes the turn's latest one, unless
  a later user message revoked it; two resumes without progress end the turn.

Fixes #100
@oakimov

oakimov commented Oct 18, 2026

Copy link
Copy Markdown
Owner Author

Fixed in 3d23261 and f9620d6, rebased onto main after #104 and #105.

Cursor CLI's retry budgets now survive OpenCode 2's actual plain-text interruption notice. A provider-requested retry identifies that continuation; genuine later requests, including the same literal text, keep a fresh budget.

A drop immediately after completed tools can rebuild from all completed host results, without resuming their stale checkpoint. Missing results, unknown activity, new output and uncheckpointed follow-ups remain final. A later tool result cannot erase a follow-up's barrier.

Validation: 2,432 tests pass, plus typecheck/build, architecture and Node HTTP/2 checks. With a local protocol stand-in, native OpenCode 2.0.32 stops repeated drops after 4 Runs instead of 11. OpenCode 1.19.3 and 2.0.32 both complete the post-tool recovery in 2 Runs with one shell side effect. These are native-host checks against a stand-in, not live Cursor service validation.

Comment thread src/language-model.ts Outdated
Comment thread src/language-model.ts Outdated
@oakimov

oakimov commented Oct 18, 2026

Copy link
Copy Markdown
Owner Author

@nkoynov ready for a retest

@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

This is from f9620d6, before 640393e; rerunning on 640393e now. With OpenCode 2.0.31: the drop and read-idle cases still continue with the lost request once, a drop right after a completed shell now rebuilds and finishes with the shell run once, a later user message that repeats the interruption text verbatim gets a fresh budget, the error table is unchanged, and the every-attempt drop now ends on the budget ("Cursor Run failed after 11 attempts", since our stand-in's reset counts as transport). Two things we still see: the notice is only recognized as a lone text part, so with a plugin that appends a reminder part to the last user message (we run one) every continuation starts a fresh budget and only OpenCode's own limit applies; and a stall after reasoning plus a partial_tool_call row with no exec (our CB-08 case: thinking, the row, then silence until the read-idle cut) still ends the turn final, where our fork has OpenCode retry it and the request comes through.

@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Reran the same set on 640393e with and without our plugins and got the same results as on f9620d6: drop and read-idle continue with the lost request once, the post-tool drop recovers with the shell run once, the repeated notice gets a fresh budget, the error table is unchanged, and the every-attempt drop ends on the budget after 11 attempts without the plugins. Both open points still stand: with a plugin that appends a reminder part to the last user message, OpenCode's own limit ends the every-attempt drop instead of the budget, and the CB-08 stall (reasoning, a partial_tool_call row with no exec, then the read-idle cut) still ends final.

@oakimov

oakimov commented Oct 18, 2026

Copy link
Copy Markdown
Owner Author

For the plugin-appended reminder case, could you share a redacted example of the final user message’s content parts? Is the extra text part wrapped in <system-reminder>…</system-reminder>, or does it use another format? This will help distinguish retry continuations from genuine follow-up requests while preserving the shared retry budget.

@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Sure. It's our own plugin, a Claude Code compatibility layer: on every model call it appends one text part to the last real user message, wrapped in <system-reminder> tags (the user's output-style reminder, the same text for the whole session). So on an OpenCode 2 continuation the final user message's content is:

[
  { "type": "text", "text": "The previous response was interrupted. Continue from where you left off without repeating completed content." },
  { "type": "text", "text": "<system-reminder>\nConcise output style is active. Be concise: lead with the result, skip preamble and narration, keep only what the user needs.\n</system-reminder>" }
]

The notice is always the first part and the reminder is the only addition. The same plugin also inserts host notes, but as separate user messages wrapped in <system-update>…</system-update> right after tool results, not as extra parts of this message, so they shouldn't affect the match.

@oakimov

oakimov commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner Author

Fixed in e29d69c.
I ran into an upstream bug: the OpenCode 2 CLI exit-code result is an upstream bug: anomalyco/opencode#50062, with a pending fix in #52885. Its CLI retains exit code 1 after an intermediate session.step.failed, even when the execution later succeeds.

Our native test of your CB-08 recovered in two Runs on OpenCode 1.19.3 and 2.0.32: the original request appeared once per Run, with zero tool executions. OpenCode 2 also recovered at the actual 120-second read-idle cut with the semantic deadline explicitly raised. The stock OpenCode 2 CLI still exited 1 because of the upstream behavior above.

These checks used the rebuilt provider through the native hosts' AI SDK adapters against a local Cursor protocol stand-in; they were not live Cursor service tests. We will leave the CLI exit-status fix upstream.

@nkoynov

nkoynov commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Reran on e29d69c with OpenCode 2.0.31, with and without our plugins: the every-attempt drop now ends on the budget in both ("Cursor Run failed after 11 attempts", 13 Runs), and the CB-08 stall recovers in 2 Runs with the request once and nothing executed (the shown write row closes as "Provider did not return a tool result"). Drop and read-idle (now cut at 30 s), the post-tool drop with the shell run once, the fresh budget for a repeated notice and the error table are unchanged, so both points we raised are resolved for us.

@oakimov
oakimov merged commit d73175e into main Oct 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A stall or read-idle cut after Cursor's output ends the turn again since #97

2 participants