feat: resilient LLM streaming & recovery, --continue-project, grouped help

[Emasoft]

Description of Changes

Four features plus a hardening round addressing the owner review — all ten review items (see the item map below):

Resilient LLM streaming & recovery

• Two-tier retry with independent budgets: infrastructure failures (transport errors, mid-stream disconnects/resets, HTTP 408/429/5xx) draw from -max-stream-retries / LATE_MAX_STREAM_RETRIES (default 10; 0 disables); HTTP 400 body rejections get a small dedicated tier (3) because strict gateways often fail transiently while reading the request body. Backoff: 500 ms doubling, 30 s cap, full jitter. Permanent network causes — TLS certificate/trust failures, unsupported schemes, HTTP-on-HTTPS — fail fast instead of burning the budget. Server Retry-After (delta-seconds or HTTP-date) is honored as a backoff floor, capped at 5 minutes, fully cancelable.
• Typed, errors.As-able errors: client.StatusError (provider type/code preserved; error bodies read once, bounded at 8 KB, with a sanitized 1024-byte text fallback when no JSON message exists) and client.StreamInterruptedError for mid-body transport failures (HTTP/2 RST_STREAM, GOAWAY, resets, truncated bodies).
• A real-world failure motivated this: Error: stream error: stream interrupted: stream error: stream ID 1; INTERNAL_ERROR; received from peer killed sessions with zero retries, because body-read errors never surface as *url.Error and the stdlib's bundled HTTP/2 error type is unexported. The session relay now drains the terminal error before returning — a select race could previously commit a truncated attempt as a clean turn.
• Rejected turns are not fatal: on a terminal 400 the last user message is rolled back and persisted (stale history file removed on pop-to-empty; the .meta.json sidecar is kept so --continue still finds the session), with an accurate, actionable message. Per request, history is sanitized (dangling assistant tool_calls get synthesized results; empty-ID tool calls are stripped) — the saved history is untouched.
• Honest UI: retry status reads "retry N/M after Xs backoff" (no implied live countdown); a dedicated RecoveryEvent fires the moment a retried attempt actually produces a response, with failure-class-matched toasts ("connection restored" / "request accepted after retry"); stale retry state is cleared on error/stop/close.

Owner follow-up (5736731620): --max-stream-retries=0/negative now silences both tiers — the bad-body tier previously kept its own budget and still retried HTTP 400s (fixed in 2605313); golangci-lint (v2 config) reports 0 findings (a3414b4).

--continue restored; new --continue-project

• --continue resumes the latest session globally ("do what I was last doing").
• --continue-project resumes the latest session of the current project: it resolves the repo root (works from subdirectories; falls back to CWD outside a repo) and matches recorded project directories by identity (os.SameFile after a lexical fast path), so symlinked roots resolve. The two flags are mutually exclusive. Session lookup loads the exact enumerated sidecar, skips vanished/unreadable metadata, and guards nil meta (no panic, no prefix-collision fallback).

Help & status bar

• late -h groups flags by scope and always shows true defaults (even when flags are mutated before -h); the status bar shows the focused agent's type with a stable color per category.

Owner review (5716775820) — item map

1. Accumulator reset on retry — done (integration-covered end-to-end). 2. Mid-body ECONNRESET retries — done + pinned classifier row. 3. Permanent URL/TLS causes fail fast. 4. Dedicated RecoveryEvent with tested sequences (recovery→final response; exhaustion→new request). 5. Nil-meta deref fixed; exact-file sidecar loads. 6. os.SameFile directory identity. 7. Default budget 10, documented in both quickstarts. 8. Retry-After honored (floor, 5-min cap, cancelable). 9. Non-countdown status wording. 10. Bounded, sanitized error diagnostics.

Per the owner review, the OTP permission mode (-force-revaluate-dangerous-commands, OTP registry, otp_code bash parameter) and install-dev.sh were split into their own independent draft PR #129 (pending the testing/benchmarking pass requested in the review); a symlink dev-install can return later as a makefile target. This PR is fully independent of it — no shared commits.

Behavior changes

• -force-revaluate-dangerous-commands and install-dev.sh moved to the independent draft PR feat: OTP gate for dangerous commands + install-dev.sh (split from #127) #129; permission-mode keeps its other two values and the two remaining permission flags are still mutually exclusive.
• --continue no longer filters by directory; use the new --continue-project for project-scoped resume.
• The default stream retry budget is now 10 (0 disables); a terminal HTTP 400 rolls back the last user message and persists the rollback.

Test plan

• go test -race ./... — all packages pass (11 stream-retry httptest SSE integration scenarios; orchestrator event-sequence tests; session lookup/sanitizer/rollback tables; plugin tests de-flaked)
• go vet ./... clean; gofmt clean on all changed files; stress-verified: flagship mid-stream retry tests 150/150, 50/50 and 30× under -race
• golangci-lint (v2 config) — 0 findings on the whole tree
• ./test/late-podman-test.sh — Linux-only (the wrapper hard-requires a Linux host per uname -s, and the test uses sha256sum), so it cannot run locally on macOS; validated by CI on Linux runners
• Manual smoke checks (kill LLM server mid-stream → retry → recovery toast; always-400 → rollback message)

Known limitations / follow-ups

• In-band SSE error payloads (data: {"error": ...}, event: error) are not parsed yet — a provider erroring mid-stream can still end as a clean-looking stream (pre-existing; follow-up).
• Historical assistant tool_calls serialize an index field, and reasoning_content rides on historical messages — tolerated by common endpoints; strict ones now fail gracefully via the bad-body tier + rollback.
• The image_unsupported rollback remains memory-only (self-healing; pre-existing).
• --continue swallows corrupt-history parse errors into an empty session (atomic writes make this unlikely; follow-up: warn vs not-exist).
• No client TLS knob (blocks a live TLS+HTTP/2 e2e test) and no --force-http1 escape hatch (less needed now that h2 mid-stream failures retry).
• A symlink-based dev install can return as a makefile target per the review.

Contributor License Agreement (CLA)

To accept your code, we legally need you to agree to our CLA so we can maintain the project's Business Source License (BSL) and future open-source transitions.

• By checking this box, I confirm that I have read and agree to the terms of the CLA.md in this repository. (To check the box, put an x between the brackets like this: [x])

[Emasoft]

I have read and agree to the terms of the CLA.md in this repository, and I agree to grant the copyright and patent licenses described therein for my contributions to this project. — Emasoft

[mlhher]

Thank you for your contribution. Please allow for some time to go over it and evaluate it.

[mlhher]

@Emasoft Thank you again for the PR. I went over it and noticed some things. Please take a look at them.

Also note that I strongly suggest splitting the OTP permission mode into its own PR and for now removing the install-dev.sh as these are unrelated. Specifically for the OTP path I think it might require some proper testing and benchmarking to ensure it works as expected.

If you want a symlink-based dev install, we should look at it separately e.g. as a makefile target.

Further one thing I'd split is the --continue logic. I do get the reasoning and it seems valuable but this seems to be imprecise. Continue to me means "do what i was last doing" while your behavior is more "resume the latest session in the directory". For that specifically I'd instead suggest using a new flag e.g. --continue-project. Ideally if possible, --continue-project should identify the repo root so it also works from inside a subdirectory.

For the remaining changes, these are the things I did notice:

1. Reset the orchestrator's accumulated response between retry attempts. In internal/orchestrator/base.go, both retry callbacks leave o.acc intact. The executor resets its own accumulator, but subsequent UI events still contain the failed attempt's text followed by the successful response. A temporary integration test produced FAILEDSUCCESS in the event stream while the returned response correctly contained only SUCCESS. Reset the orchestrator accumulator on retry in both execution paths, and cover this through an integration test.

2. Retry connection resets that happen while reading the response body. In internal/executor/stream_retry.go, a non-timeout net.OpError wrapping ECONNRESET falls through as non-retryable. Unlike failures from the initial HTTP request, response-body read failures need not have a url.Error wrapper. A targeted test confirmed this classification error. Recognize transient read failures while preserving the cancellation exclusions.

3. Stop retrying permanent URL and TLS errors. The same classifier treats every url.Error as transient. A certificate trust failure is therefore retried until the budget runs out; a targeted test confirmed this with x509.UnknownAuthorityError. Unsupported schemes can also arrive through this wrapper. Classify the underlying cause rather than treating the wrapper itself as sufficient evidence for a retry. Malformed URLs rejected before the request is sent may already fail immediately, so this does not affect every malformed URL.

4. Trigger recovery when the retry actually produces a response. In internal/tui/update.go, clearing WasRetrying and showing the recovery toast depend on a thinking event. The executor calls the turn-start callback outside the retry loop, so a successful retry emits content without another thinking event. A final text response leaves the flag set; the next user request can show a misleading recovery toast. Handle recovery on successful stream activity, or add a dedicated recovery event. Test the actual event sequence, including recovery followed by a final response and retry exhaustion followed by a new request.

5. Avoid a nil dereference when a session disappears during lookup. In internal/session/models.go, GetLatestSessionForDir accesses meta.WorkingDir without checking meta == nil. If the metadata file disappears after entry.Info() but before LoadSessionMeta checks it, the loader can return (nil, nil) and startup panics. A deterministic test reproduced this. Add the nil check, or load the exact enumerated file with loadMetaFile(filepath.Join(sessionsDir, entry.Name())) and skip read failures. Loading the exact file also avoids accidentally falling back to a different session with a matching prefix.

6. Match equivalent project directories. The new directory filter compares only cleaned path strings. A session recorded under a real directory is not found when the same directory is supplied through a symlink; a temporary test reproduced this. Case differences on case-insensitive filesystems also deserve coverage. After the lexical fast path, consider comparing directory identity with os.Stat and os.SameFile, with sensible handling for missing directories. Avoid blindly lowercasing paths, since filesystem case sensitivity varies.

These additional changes would improve the feature, but are separate from the confirmed ones above:

7. Reconsider the default retry budget and document it. With full jitter, 100 retries mean about 23.8 minutes of expected backoff and up to roughly 47.5 minutes, excluding request time. A smaller interactive default or elapsed-time budget would be easier to understand. For reference, five retries average 7.75 seconds of backoff; ten average 75.75 seconds. Add --max-stream-retries, LATE_MAX_STREAM_RETRIES, their precedence, and the zero-to-disable behavior to both quickstarts. They currently appear in CLI help but are absent from those guides.

8. Respect server-provided retry timing. StatusError currently discards response headers, so retry logic cannot honor Retry-After on a throttling or unavailable response. Preserve valid retry timing and combine it with the local backoff without retrying before the requested delay. Handle invalid values and keep the wait cancelable. This is an improvement to the new retry behavior, rather than a regression in the previous client.

9. Make the waiting message accurate over time. The UI formats “retrying in 24.8s” once and never updates it. It can remain visible after the wait ends and while the next request is running. Either render a countdown from a retry deadline and change the message when the attempt starts, or use wording that does not imply a live countdown.

10. Improve HTTP error diagnostics. Plain-text or HTML error bodies currently collapse to status: 502, for example. This limitation already exists on main; it was not introduced by this PR. If addressed here, read a bounded body once, decode the JSON from those bytes, and use a bounded, sanitized text fallback when no structured message exists. Reading the body only after a failed JSON decode can lose the bytes already consumed by the decoder. Also, the new Body field comment says “truncated,” but the implementation does not enforce a limit.

[Emasoft]

I agree on almost everything (except maybe the retry limits, since it happened to me often that during the night a service was down for 1 hour and then resumed, so recovering from that is important for continuity. But i will do as you say anyway). I'm working on all the issues you reported, and also on the improvements. I will slowly add the updates to the PR.

[mlhher]

(except maybe the retry limits, since it happened to me often that during the night a service was down for 1 hour and then resumed, so recovering from that is important for continuity. But i will do as you say anyway)

I do see your reasoning. I think there are arguments for both sides but I do see where you are coming from. As you noted this never happened to me so I might be biased, if this happens to you often then I understand it more. I think it might be best to let this be user configurable in the end but this is rather minor and should be rather easy to adjust from what I have seen so far.

Thank you again for the quick updates! Feel free to ping me if you want me to review again otherwise I will do so when I see newer commits.

[mlhher]

The changes look good. Seems like nearly everything has been fixed great work! One remaining thing I did notice was that --max-stream-retries=0 still retries HTTP 400 error responses, despite advertising that it disables retries entirely. Please take a look at that. Please also take a look at the golangci-lint results (these are very minor).

In the meantime I will do some more local testing to ensure everything works well and if nothing new comes up this will be merged. Thank you again!

[mlhher]

@Emasoft

I noticed one more thing, other than that this is merge-ready and will likely be merged once resolved.

The new agent-type label in the status bar seems redundant and doesn’t quite conform to the existing color scheme. Subagents are already identified by the breadcrumb on the right. If you want to show the focused agent consistently, including the orchestrator, I suggest adapting that existing label and placing it before the token counter using the current muted grey, since this information doesn’t need to demand attention. Otherwise, feel free to remove the new label and I’ll revisit it in a later pass.