diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 1d6046d..6153a0c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -77,7 +77,9 @@ The mechanism is deliberately generic: this plugin exports a capability, not a p OpenCode prepares auxiliary requests with the same session identity as the agent loop: `compaction` (context summarization inside a turn), `generate` (stateless text generation), and `title` (automatic chat naming). Compaction and generate share the turn's fate: if their hold is interrupted, the primary recovery continuation re-runs the step and the auxiliary request is prepared again. An automatic title does not: OpenCode generates it once when the first input of an untitled root session is promoted, and a lost title request is not retried by OpenCode itself. -A held `title` request therefore persists a semantic `pending-title` recovery marker (same minimal shape as a deferral: session ID, provider/model, location, timestamp, origin) instead of the primary continuation. A session can hold a primary continuation and a pending-title marker at the same time. When the allowed window arrives the marker is delivered as a check, never as a replay or a mutation: +A held `compaction` request (manual `/compact` or automatic context compaction — both are the same `model.request` kind) persists a semantic `pending-compaction` marker. It is not a replay: as soon as the recovered session is settled and idle, the plugin admits OpenCode's own compaction again (`session.compact`), and the re-triggered request holds at the pre-model guard through the peak window exactly like the original. OpenCode's runner keeps a pending compaction ahead of queued prompts, so a compact-first workflow survives a lost hold with its order intact: compaction first, then the prompt that was waiting behind it. Earlier-started work is different: if a compaction is recovered alongside a primary continuation for the same session, both are admitted in the same pass — compaction first, continuation behind it — and the continuation is not treated as stale just because the compaction completed first. + +A held `title` request therefore persists a semantic `pending-title` recovery marker (same minimal shape as a deferral: session ID, provider/model, location, timestamp, origin) instead of the primary continuation. A session can hold a primary continuation, a pending-title marker, and a pending-compaction marker at the same time; each class is superseded only by a fresh request of its own kind, and each transport-guard class (`title`, `compaction`, `primary`) clears only its own entry — a compact request can never clear a primary deferral or a pending title, and a `generate` request clears nothing. When the allowed window arrives the title marker is delivered as a check, never as a replay or a mutation: - A session that already has a title — generated by OpenCode, generated by another process, or renamed by the user — clears the marker without any request, so a title is never overwritten and never generated twice. - A session that still needs a title is left to OpenCode's own generation. Every path that loses a title hold (inactivity, supersede, shutdown, reload) also resumes the session — the plugin's same-session continuation or OpenCode's restart recovery — and that input promotion is OpenCode's native trigger for automatic naming. The marker waits for the title, clears once it appears, and is dropped with a diagnostic if no title appears within the settle cap; a later input promotion still names the session normally. @@ -118,6 +120,8 @@ OpenCode's v2 recovery is at-least-once, not exactly-once. If the plugin finds a A `WAITING_OFFPEAK` record is written before the guarded request is allowed. That state is safe for automatic same-session resume in `auto` mode. A hold for a live request (interactive or queue-owned) is intentionally not persisted as a request: it belongs to a running OpenCode session, so a plugin/server shutdown fails it closed instead of replaying it; the OpenCode server's own execution-claim recovery can resume sessions interrupted by a crash, and the plugin enforces pricing when they come back. Retry recoveries and inactivity-released holds are persisted (minimal metadata: session ID, provider/model, location) because their provider request is already finished or the session was deliberately interrupted; a reload or restart during peak keeps the pending continuation, which is delivered only when the session has settled with a failed/interrupted outcome and pricing allows. A running OpenCode server is required for automatic timers; the plugin does not install or operate a separate daemon. The supported OpenCode background service can remain running when the TUI terminal closes. Machine-restart startup is delegated to the host OS service manager if desired. +Recovery delivery is guarded against live work so it can never duplicate it. `session.outcome` only changes at settlement, so a running turn still reports its previous outcome; the engine tracks holds that are still parked in this instance and the execution state reported by OpenCode's own `session.execution.started` / terminal events, and it skips (rather than delivers) any recovery whose session is currently owned by a live hold or a running execution — ending the hold or settling the execution wakes the engine again. That closes the race where a boundary release and a recovery admission could otherwise both fire for one request. Delivery itself stays at-most-once per entry (the entry is removed before the request is sent) and per class: a compaction marker is delivered as the semantic compaction re-trigger, a primary marker as the continuation prompt, a title marker as a check. + ## Override `/offpeak run-now ` is a task-scoped, 15-minute override. It is persisted with an absolute expiry and shown in status while active. OFF pauses its use; it does not create or extend it. Expiry returns the next provider request to normal policy enforcement. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1242359..35fcfd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## 0.1.9 - 2026-10-16 + +- **A compaction held for peak pricing is no longer lost when its hold dies.** Manual compact-first is a normal workflow: compact the session, then send the next prompt. During PEAK that compaction waits at the pre-model hook like any other request — but its hold was memory-only, so a plugin reload, a server restart, or the inactivity sweep ended it with a terminal failed compaction and, worse, left the prompt queued behind it stranded: nothing resumed at off-peak and the compaction never ran. A held compaction now persists a semantic `pending-compaction` recovery entry (with `origin: "pending-compaction"`; the raw provider request is never replayed). As soon as the recovered session is settled and idle, the plugin admits OpenCode's own compaction again; the runner keeps a pending compaction ahead of queued prompts so compact-first order survives, and the re-triggered request holds at the pre-model guard through the peak window exactly like the original. A user cancel clears the marker; inactivity and supersede keep it. +- **Recovery classes are independently owned.** A session can carry a primary `deferred`, a `pending-title` and a `pending-compaction` entry at the same time. Each `model.request` hold supersedes only its own class, and each transport-guard class (`title`, `compaction`, `primary`) removes only its own entry — a compact request can no longer clear a primary deferral or a pending title, and a `generate` request clears nothing. At the allowed window the pending compaction is admitted first, then primary continuations, then title markers, so the native compact-first order is preserved. `/offpeak status` reports `compactionRecoveryWaiting`. +- **Recovery cannot race a live execution into a duplicate.** `session.outcome` only updates at settlement, so a still-running turn reports its previous outcome; the engine now tracks live holds and the execution state from OpenCode's own `session.execution.*` events, and recovery delivery waits while either owns the session. A boundary release and a recovery admission can no longer both fire for one request (previously reachable with a stale failed outcome), and every delivery is still at-most-once. +- Evidence: 115/115 tests (99 existing + 7 cancellation tests from v0.1.7/0.1.8 + new deterministic pending-compaction coverage: the exact compact-first-then-prompt workflow, reload and restart recovery, interrupt semantics, coexistence of all three recovery classes, no class clobbering, at-most-once delivery, ordering, and the live-hold/no-duplicate guard). Real OpenCode 2.0.30 runs with a local fixture provider: compact-first through a peak window completes exactly once with the prompt behind it and zero peak-priced traffic; the same workflow with a full server restart recovers the compaction, runs it before the prompt, and completes with clean recovery state (the same harness against v0.1.8 loses the compaction and duplicates the queued prompt's continuation). + ## 0.1.8 - 2026-10-16 - **A session can hold several requests at once, and cancelling cancels all of them.** OpenCode prepares auxiliary requests (the automatic title, generate, compaction) concurrently with the agent loop, so one session can have more than one request parked. The held-request bookkeeping is now per request rather than per session: `cancelHold` cancels every request of that session that is still waiting, each gets its own one-shot refusal at the transport boundary, and none can be left behind to dispatch at the next allowed window. A refusal is matched by provider, model, and request kind, and only a fresh request *of the same kind* supersedes an un-consumed refusal. diff --git a/DECISIONS.md b/DECISIONS.md index 029ddc5..92a63ee 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -95,3 +95,7 @@ Another plugin can decide that a parked request is obsolete — the most importa ## 2026-10-13: Peak-aware retry recovery uses OpenCode's retry hook A request that starts during OFF_PEAK can cross into PEAK and then fail with a retryable transport error (`Timed out waiting for response headers`). OpenCode's own retries would either reach the provider during peak or exhaust and fail the turn, and the guard's boundary hold cannot outlast the header-timeout clock, which starts at attempt start. The plugin now registers OpenCode's supported `retry` hook: during PEAK, with the plugin ON in `auto` mode and a retryable failure, it sets `decision = { retry: false }` and marks the work to continue at the next allowed window (a queue task moves to `WAITING_OFFPEAK`; an interactive session is recorded with minimal persisted recovery metadata scoped to the plugin location). The continuation uses the existing safe continuation prompt in the same session; the failed provider request is never replayed. Non-retryable failures stay `FAILED`, OFF/`queue`/`guard` modes and the override semantics are unchanged, and no new runtime, scheduler, database, or polling loop is added. Evidence: OpenCode 2.0.27 retry hook events (attempts 2–5; `provider.transport`/`provider.internal`/`provider.rate-limit` retryable, `provider.invalid-request` terminal at attempt 2) plus isolated real-binary end-to-end runs proving zero provider traffic during peak and automatic same-session continuation. + +## 2026-10-16: A held compaction is recovered semantically, admitted as soon as the session is settled + +Manual compact-first — compact the session, then send the next prompt — is a normal workflow, and during PEAK the compaction waits at the pre-model hook like any other request. That hold was memory-only, so a plugin reload, a full server restart, or the 60-minute inactivity sweep ended it as a terminal failed compaction and left the prompt queued behind it stranded; nothing resumed at the off-peak window. The plugin now persists a `pending-compaction` marker for a held compaction (manual and automatic compaction are the same `model.request` kind, so both are covered) and recovers it by admitting OpenCode's own compaction again — never by replaying the provider request. Three source facts shaped the design. First, `session.compact` is the supported semantic trigger, and OpenCode's runner keeps a pending compaction ahead of queued prompts (`packages/core/src/session/inbox.ts` reorders earlier steers behind a pending compaction "so their text stays verbatim after the checkpoint"), so the recovered compaction lands before the prompt that waited behind it. Second, the re-triggered request holds at the same pre-model guard, so recovery during peak still produces zero provider traffic. Third, because of that native priority there is no reason to wait for the allowed window to admit the recovery: it is delivered as soon as the session is settled and idle (any settled outcome, including `succeeded` — unlike a primary continuation, a requested compaction that never ran is not made stale by another completed turn), which is what restores compact-first order after a restart instead of letting a later prompt win the inbox. When a primary continuation coexists with a recovered compaction for one session, both are admitted in the same pass in native order (compaction, then continuation) so the continuation is not dropped by the compaction's own success. The v0.1.7/v0.1.8 transport guard cleared primary recoveries for any non-title kind, which could have let a compact request erase a primary deferral; each request kind now clears only its own recovery class. Evidence: deterministic tests for the exact workflow, reload/restart/inactivity/user-cancel semantics, three-class coexistence, class independence, at-most-once delivery and ordering, and real-CLI fixture runs where a restart during a held compaction still ends with exactly one completed compaction, the prompt behind it running once, and clean recovery state. diff --git a/HANDOFF.md b/HANDOFF.md index 93d33b7..6b49816 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,5 +1,39 @@ # Handoff — 2026-10-16 +## 2026-10-16 pending-compaction recovery (v0.1.9) + +**Reported workflow.** Compact an existing session first, then send the next prompt. During OFF_PEAK that is uneventful; during PEAK the compaction was held like any other request, but after a plugin reload, a full server restart, or the 60-minute inactivity sweep the compaction ended as a terminal **failed** compaction (the red terminal state) and the prompt queued behind it was stranded: at OFF_PEAK nothing resumed and the compaction never ran. + +**Live evidence on this host (reconstructed, not assumed).** Two real manual compactions created at 03:36:45Z and 03:40:44Z inside the 01:00–04:00Z peak window were held correctly (zero provider traffic) and failed with `PricingBlockedError: Offpeak plugin is shutting down; request blocked.` at exactly 03:49:38Z, when the 2.0.27→2.0.30 upgrade replaced the background service. Both sessions (`ses_ebfcd1ef4ffewnnZSpAD2OSx2c`, `ses_ebfe2b131ffeHv3UWb8nK8J7YT`) had **no** recovery entry — the compaction hold was memory-only — so OpenCode's own recovery could not resume them: the sessions sat idle after the restart and were only continued manually at 04:02Z. The primary holds that failed in the same shutdown were continued at the 04:00Z boundary by their persisted deferrals, which is exactly the difference the fix closes. + +**Root cause (source-traced).** A held compaction goes through `SessionModelRequest.prepare` with `kind: "compaction"` (`packages/core/src/session/model-request.ts`; manual and automatic compaction share that kind). Until v0.1.8 the engine persisted recovery only for `primary` (`deferred`), `retry-failure`, and `title` (`pending-title`) holds; a compaction hold existed only in the plugin instance's memory, so any disposal failed it closed with nothing to recover. Two ordering facts shaped the fix: OpenCode's runner keeps a pending compaction ahead of queued prompts (`packages/core/src/session/inbox.ts` reorders earlier steers behind a pending compaction "so their text stays verbatim after the checkpoint"), and `session.compact` is the supported semantic trigger for OpenCode's own compaction. + +**Fix (v0.1.9).** + +- A held compaction (any settled-recovery session) persists a **`pending-compaction`** marker (same minimal shape as the other classes; no raw provider replay). As soon as the recovered session is settled and idle, the plugin admits OpenCode's own compaction again; the re-triggered request holds at the pre-model guard through the peak window, so recovery still sends zero peak-priced traffic. Delivery does not wait for the boundary: admitting it early is what restores compact-first order after a restart (the marker may be admitted while the session is idle even in peak), and unlike a primary continuation the marker is **not** dropped merely because another turn succeeded — the user asked for a compaction that never ran. +- **Native ordering is preserved.** When a compaction is recovered alongside a primary continuation for the same session, both are admitted in the same pass — compaction first, continuation behind it — so the continuation is not considered stale by the compaction's success. The earlier-started prompt that was waiting in the inbox follows the recovered compaction by OpenCode's own priority. +- **Recovery classes are independently owned.** `model.request` holds supersede only a marker of their own kind, and the transport guard now clears only its own class (`title` → title; `compaction` → pending-compaction; `primary`/legacy → primary origins; `generate` → none). A compact request can no longer clear a primary deferral or a pending title. The flush order at the allowed window is compaction → primary → title. `/offpeak status` adds `compactionRecoveryWaiting` (and the TUI renders `Compaction recovery: N session(s) waiting for off-peak`). +- **Duplicate-race guard.** `session.outcome` only changes at settlement, so a running turn still reports its previous outcome; the engine now tracks live holds (per session) and the execution state from OpenCode's own `session.execution.started` / terminal events, and recovery delivery skips any entry whose session is owned by a live hold or a running execution (ending either wakes the engine through the existing pump). This closes the race — reproducible in isolation, and visible in the v0.1.8 comparison run — where a boundary release and a recovery admission both fired for one request, producing a duplicate continuation. +- **Cancel semantics unchanged and preserved.** A user cancel (Escape) clears the marker and nothing continues; the v0.1.7/v0.1.8 `cancelHold` path removes the marker through the same `deferralOrigin` it already removes deferrals with. Inactivity/supersede keep the marker; OFF keeps it without delivery; queue-owned holds keep their task semantics. + +**Tests.** `npm run lint`, `npm run typecheck`, `npm test` — **115/115** (99 v0.1.6 + 7 hold-cancellation + new pending-compaction coverage). New deterministic coverage includes: the compaction hold persists and removes its marker; a lost hold re-admits OpenCode's compaction once at recovery; reload; the exact compact-first → prompt fixture semantics (including a fake compaction that completes before the continuation is evaluated); running-hold and running-execution guards (no duplicate, no drop-on-success); inactivity/supersede keep the marker while user cancel clears it; three-class coexistence and per-class guard clearing (compaction never clears a primary deferral; generate clears nothing); at-most-once delivery across repeated flushes. + +**Real OpenCode 2.0.30 fixture validation (local fixture provider; zero paid traffic).** + +| Scenario | Result | +|---|---| +| `compact-first` (real TUI, `/compact` then prompt during PEAK) | compaction held with zero peak-priced traffic; boundary ran the compaction (one summary call) and then the prompt (one primary); same session; recovery clean | +| `compact-reload` (plugin reload while held) | same outcome: one compaction + one prompt at the boundary; clean recovery | +| `compact-restart` (full server restart while held, prompt re-sent via `opencode run --session` after the location woke) | the lost compaction failed closed on the old process; the new instance re-admitted OpenCode's compaction as soon as the location woke; at the boundary the compaction ran exactly once (one summary call), then the prompt ran exactly once (one primary); same session; recovery clean | +| `compact-restart` against v0.1.8 (control, identical harness) | compaction never ran (0 summary calls, no completed compaction row) and the queued prompt produced a duplicate: two user messages (the prompt plus a plugin continuation), two primary calls, plus one failed compaction row | +| `go-peak` animation regression (real TUI) | hold/resume toasts, zero peak-priced traffic, boundary release, exactly one hold and one release notification, single session | + +Evidence-integrity note: the fixture harness now uses a local-only provider id (`offpeak-fixture`) because models.dev today began serving a real `opencode-go` provider entry, which shadowed the fixture endpoint in fresh isolated runs; the fix is confined to the harness and prevents any fixture run from resolving a real provider (no fixture run ever reached a paid endpoint: the requests were rejected without credentials). + +**Release and live state.** v0.1.9 (PR, CI, GitHub release tarball) installed live on the shared service; `/api/plugin` reports the v0.1.9 tag commit; `/offpeak status` ON/`auto` with queue, recovery, titleRecovery, compactionRecovery and failed all 0; production policies and provider configuration restored; no fixture/test state remains in the live environment. The work was developed on branch `pending-compaction` rebased onto v0.1.8 (v0.1.7/v0.1.8 shipped concurrently by the hold-cancellation pass; both feature sets are present in v0.1.9, and both suites pass together). + +**Remaining limitation.** A compaction that finished its failed attempt on a disposed instance still records that one failed attempt (fail-closed by design); the recovery then completes the compaction semantically at the next opportunity. A plugin/server shutdown cannot avoid that terminal record, because OpenCode offers no "parked" execution state (see the held-request UI investigation above). + ## 2026-10-16 hold cancellation (`cancelHold`, v0.1.7) **What changed.** A held request can now be cancelled generically and without ending the session. New RPC method on the existing channel: `opencode-offpeak.cancelHold { sessionID, reason? } -> { cancelled, state, model? }`. The held-request state (`holding` / `committed` / `refused`) is a single synchronous transition, so a cancellation and an allowed-window release can never both win. A cancelled hold returns from the loop and arms a one-shot refusal; the transport-boundary hooks registered *before* the pricing guards consume it and replace the request with a credential-free placeholder aimed at a closed loopback endpoint (`127.0.0.1:1`, marker header `x-opencode-offpeak-refused`). OpenCode classifies that as a retryable transport failure, so its own retry machinery continues the same execution on the route the caller selects — no `session.interrupt`, no synthetic continuation, no lost turn. A refused placeholder is skipped by the pricing guard, expires after 60s, and is dropped when a fresh hold starts. `noteRetryFailure` ignores a refusal this plugin produced, so the v0.1.3 retry recovery cannot duplicate the caller's resumed work. `/offpeak status` adds `interactiveHolds`, `armedRefusals`, `cancelledHolds`, `refusedRequests`. @@ -12,6 +46,24 @@ **Coexistence note.** This work was developed in a git worktree (`/root/agent-tmp/opencode/offpeak-hold-cancel`) because another agent was concurrently editing the canonical checkout's `src/store.ts`/`src/engine.ts` for the pending-compaction recovery. Nothing of that in-flight work was touched; the canonical checkout was restored to exactly the concurrent agent's state before this branch was cut. +## 2026-10-16 held-request UI investigation (no code change; v0.1.6 unchanged) + +**Reported symptom:** during a peak window, an interactive prompt is correctly held (toast shown, zero provider traffic), but OpenCode's normal "building/working" animation keeps running; later it stops by itself; the persisted hold remains healthy; at off-peak the same session continues. + +**Baseline verified live (2026-10-16):** OpenCode CLI and background service **2.0.30** — the host was upgraded from 2.0.27 at 03:49:24Z (`updated OpenCode from=2.0.27 to=2.0.30 method=curl`, service replaced with `reason=version-mismatch previousVersion=2.0.27`; the old 2.0.27 service process still runs and was not touched). Plugin spec `git+https://github.com/lutzkind/opencode-offpeak.git#v0.1.6`, live commit `1fe61d2f5979f6f4df32208df7ed3f1cea8c9ae6` active (server+tui). `/offpeak status`: ON / `auto`, queue/recovery/titleRecovery/interactive all 0; Go DeepSeek PEAK policy and Zen flat-priced pass-through correct. Repository `d287f5b` clean; v0.1.6 the latest release; no open PRs/issues; lint, typecheck, **99/99 tests**, and `npm pack --dry-run` all pass. + +**Exact cause (source-traced against upstream `anomalyco/opencode` v2.0.27 and v2.0.30 — every relevant file byte-identical between the tags):** OpenCode's TUI shows the animated spinner + `esc interrupt` in the prompt footer exactly while `data.session.status(sessionID) === "running"` (`packages/tui/src/component/prompt/index.tsx`); that status is set only by `session.execution.started` and cleared only by `session.execution.succeeded|failed|interrupted` (`packages/client/src/solid/data.ts`). The plugin's hold runs inside `SessionModelRequest.prepare` (`packages/core/src/session/model-request.ts`), i.e. **after** the drain started, and no settlement event can exist while the hook promise waits — so the animation is OpenCode truthfully reporting an active execution with zero provider traffic. `DataSessionStatus` is exactly `"idle" | "running"`; no waiting/parked session state exists in the client, plugin API, protocol, or TUI. + +**Why it eventually stops:** (a) normally the boundary release continues the **same execution** — the animation does not stop, it flows into the real request; (b) OpenCode's 60-minute location-activity sweep interrupts the execution with `reason=inactivity` (`packages/core/src/location-activity.ts`, TTL 60 min, sweep 1 min) — the "stops after some time by itself" path — and the plugin keeps the deferral, delivering the continuation at the next allowed window; (c) a service shutdown/reload ends the execution (live incident below); (d) a user cancel (double Escape) ends it and clears the deferral. All four paths were reproduced with the real TUI against a local fixture provider (zero paid traffic) on 2.0.30 and cross-checked on the real 2.0.27 binary; details and raw evidence in the shared handoff. + +**Live incident reconstruction:** session `ses_ebd3c5c20ffeLytWmmLXcXMO0i` ("Greeting", prompt `say hi` at 03:31:20Z inside the 01:00–04:00Z peak window) was held; the 2.0.27→2.0.30 upgrade replaced the service at 03:49:38Z and 4 held sessions failed closed (`PricingBlockedError: Offpeak plugin is shutting down; request blocked.`); the persisted deferrals continued the same sessions at the 04:00:00Z boundary. The observed "animation stops by itself" was the service replacement, followed by the correct same-session continuation. + +**Decision — no code change (v0.1.6 remains canonical).** Every supported mechanism was inspected in source before rejection: `session.interrupt` is the only API that ends the `running` state but it terminates the turn with a `reason=user` interrupt (indistinguishable from a user cancel; would clear the deferral) and requires the synthetic `Continue the task…` prompt — a cancel/restart, not a parking state; holding the `prompt` admission hook would keep the session idle but the prompt is not durable before `admission.admit` and its prepare region is interruption-sensitive (`Effect.uninterruptibleMask(restore)`), regressing durable deferral and reload/restart recovery; a TUI `replace` slot takeover of `prompt.footer.status` is public API but suppresses the boundary's default content *and every claim inside it* for **every mounted footer** while registered, including unrelated sessions' location/`esc`/move state and the built-in `/btw` indicator, with no way to re-render the suppressed originals — a broad UI takeover, not a state transition. No fragile workaround was implemented; the hold toast (15 s, by design) and `/offpeak status` (`Interactive: N request(s) waiting for off-peak`) remain the authoritative waiting indicators. + +**Documentation updated:** the "Interactive message waits at peak" section of `docs/troubleshooting.md` now states the platform limitation, the transient-toast/persistent-status guidance, and the exact stop paths. No source, test, configuration, or release change; v0.1.6 tag and artifacts untouched. + +**Remaining limitation (platform, not plugin):** OpenCode 2.0.26–2.0.30 has no supported waiting session/execution state, so a held request necessarily looks "running" in the TUI until its execution settles. Revisit if OpenCode adds a parked/waiting execution state or a per-instance footer override. + ## Release status `opencode-offpeak` v0.1.6 is the canonical release: @@ -297,7 +349,7 @@ The implementation commit at the v0.1.0 release was `2782893c2e09eff6116f77c62b6 - Canonical checkout: `/root/opencode-offpeak` - Default branch: `main` - Package: `opencode-offpeak@0.1.6`, MIT -- Verified CLI: OpenCode v2.0.26 and v2.0.27 +- Verified CLI: OpenCode v2.0.26, v2.0.27 and v2.0.30 - Declared supported range: `>=2.0.26 <2.1.0`, enforced at startup and declared in `engines.opencode` - GitHub install after release: `opencode plugin add git+https://github.com/lutzkind/opencode-offpeak.git#v0.1.6` - Package exports OpenCode v2 server and TUI plugins. The TUI registers `/offpeak` and uses OpenCode plugin RPC to control the server-side queue. diff --git a/README.md b/README.md index a845c56..b7d0fef 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,8 @@ In `auto` mode, a supported request that reaches peak pricing first waits at Ope A queued `auto` task behaves the same way: if it reaches peak pricing before its next provider request, the guard stores `WAITING_OFFPEAK` and the same session waits and continues. Local filesystem and other tool operations already in flight are allowed to settle; the plugin does not kill them when the clock changes. +The same protection covers a manual compact-first workflow: if you compact a session during peak and then send your next prompt, both wait in order — the compaction first, the prompt behind it — and run the same way at the boundary, with nothing reaching the provider before then. If the compaction's hold is lost instead (plugin reload, full server restart, or the inactivity sweep), the plugin records a semantic `pending-compaction` marker and admits OpenCode's own compaction again as soon as the session is idle, keeping it ahead of the prompt that was waiting behind it. A user cancel clears the marker; an interrupted or restarted session recovers it. `/offpeak status` reports it as `Compaction recovery: N session(s) waiting for off-peak`. + The HTTP and experimental WebSocket transport guard still runs immediately before OpenCode sends anything, as the final hard barrier. Requests that arrive while the plugin is OFF, or in `queue`/`guard` mode, are blocked by that guard exactly as before. If the plugin shuts down or the server restarts while a request is waiting, the request fails closed as blocked instead of being sent under an uncertain pricing state; the OpenCode server's own execution-claim recovery resumes sessions interrupted by a crash, and the plugin continues to enforce pricing when they come back. If OpenCode itself interrupts a waiting session (its location-activity sweep ends a session that has had no durable events for 60 minutes), the plugin keeps the persisted deferral and continues the same session automatically at the next allowed window. A user cancel clears the pending deferral instead, so nothing continues after you cancel. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9f29cce..bd64b6e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -22,6 +22,12 @@ If you intentionally accept peak pricing for a queued task, turn the plugin ON a In `auto` mode, a supported interactive request sent during a peak window is deferred at OpenCode's pre-model hook — before any provider attempt (and its response-header timeout) starts — instead of being discarded or timing out, and the same session proceeds automatically when off-peak returns; no second message is needed. `/offpeak status` shows it under `Interactive` while it waits. In `queue` or `guard` mode, and while the plugin is OFF, interactive peak requests are blocked immediately by the transport guard. If a deferred request fails closed because the plugin or server shut down, resend it after the server is back. +While the request is held, the TUI keeps its normal animated "working" indicator even though nothing has been sent to the provider. OpenCode 2.0.26–2.0.30 reports a session as either `idle` or `running` and exposes no "waiting" state to plugins (the hold happens inside the request hook, before any provider attempt), so the indicator cannot be switched off without ending the turn; the hold toast (transient, about 15 seconds) and `/offpeak status` (persistent) are the authoritative waiting signals. The indicator stops when the execution settles: at the off-peak release it flows straight into the real request, and after a user cancel, a service shutdown/reload, or OpenCode's 60-minute inactivity sweep the session goes idle — except after a user cancel, the persisted deferral continues the same session at the next allowed window. + +## Manual compaction during peak + +Compacting a session during peak pricing (the `/compact` command or the "Compact session" action) waits like any other request: nothing is sent to the provider, and both the compaction and the prompt you send behind it run in order when off-peak returns — compaction first, then the prompt. If the compaction's hold is lost instead (plugin reload, full server restart, or the inactivity sweep), `/offpeak status` reports `Compaction recovery: N session(s) waiting for off-peak`; the plugin admits OpenCode's own compaction again as soon as the session is idle, keeps it ahead of a prompt that was already waiting, and the compaction runs exactly once at or after the allowed window. A compaction you cancel (Escape) does not come back. + ## A request was refused at the transport boundary (`x-opencode-offpeak-refused`) This is not a network failure. A plugin cancelled a parked request because the routing decision behind it changed (for example a free route became usable again), and the request was replaced by a credential-free placeholder aimed at a closed loopback endpoint so OpenCode could treat the attempt as a retryable transport failure and retry it on the new route. One line in the server log names the sentinel URL `http://127.0.0.1:1/opencode-offpeak-hold-cancelled`; the caller is expected to continue the same execution on another model, which is why the turn normally completes with no visible error. `/offpeak status` reports `interactiveHolds`, `armedRefusals`, `cancelledHolds`, and `refusedRequests`; a growing `refusedRequests` with a shrinking `armedRefusals` simply means cancellations are being consumed as designed. diff --git a/package.json b/package.json index c42558a..82e857c 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "opencode-offpeak", - "version": "0.1.8", + "version": "0.1.9", "description": "Queue OpenCode agent tasks and guard model requests against expensive pricing windows.", "author": "Lutz Kind", "license": "MIT", diff --git a/src/engine.ts b/src/engine.ts index 5e91e58..d95055d 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -35,6 +35,8 @@ export interface SessionDriver { wait(sessionID: string): Promise; get(sessionID: string): Promise; interrupt(sessionID: string): Promise; + /** Admits OpenCode's own compaction for the session (semantic recovery, never a replay). */ + compact(sessionID: string): Promise; } export interface EngineClock { @@ -81,6 +83,8 @@ export interface EngineStatus { recoveryWaiting: number; /** Subset of recoveryWaiting: sessions whose automatic title is still pending. */ titleRecoveryWaiting: number; + /** Subset of recoveryWaiting: sessions whose held compaction is still pending. */ + compactionRecoveryWaiting: number; failed: number; tasks: OffpeakTask[]; configPath: string; @@ -94,8 +98,8 @@ export interface GuardInput { transport: "http" | "websocket" | "model"; /** * OpenCode session request kind ("primary", "compaction", "title", "generate") - * when the guard runs from the pre-model `model.request` hook. Transport - * guards do not carry a kind. + * when the host provides it. The pre-model `model.request` hook always + * carries it; the transport hooks forward the kind in OpenCode 2.0.26+. */ kind?: string; } @@ -142,6 +146,8 @@ const RECOVERY_SETTLE_CAP_MS = 5 * 60_000; const PRIMARY_RECOVERY_ORIGINS = [undefined, "retry-failure", "deferred"] as const; /** The semantic "this session still needs its automatic title" marker. */ const TITLE_RECOVERY_ORIGIN = "pending-title" as const; +/** The semantic "this session still needs its held compaction" marker. */ +const COMPACTION_RECOVERY_ORIGIN = "pending-compaction" as const; export interface RetryFailureInput { sessionID: string; @@ -288,6 +294,19 @@ export class OffpeakEngine { #nextHoldID = 0; #cancelledHolds = 0; #refusedRequests = 0; + /** + * Sessions with a live hold in this instance. Recovery delivery waits for + * the hold to end: a session whose request is still parked owns its turn, + * and admitting a recovery beside a live release could duplicate it. + */ + readonly #activeHolds = new Map(); + /** + * Sessions OpenCode reports as executing (between `session.execution.started` + * and its terminal event). `session.outcome` only updates at settlement, so + * a running turn still reports its previous outcome; recovery waits for the + * turn to settle before it evaluates an entry again. + */ + readonly #running = new Set(); readonly #recoveriesRepo: RecoveryRepository | undefined; readonly #location: string; /** @@ -548,6 +567,9 @@ export class OffpeakEngine { titleRecoveryWaiting: [...this.#recoveries.values()].filter( (entry) => entry.origin === TITLE_RECOVERY_ORIGIN, ).length, + compactionRecoveryWaiting: [...this.#recoveries.values()].filter( + (entry) => entry.origin === COMPACTION_RECOVERY_ORIGIN, + ).length, failed: tasks.filter((task) => task.state === "FAILED").length, tasks, configPath: settings.path, @@ -557,12 +579,17 @@ export class OffpeakEngine { async guardRequest(input: GuardInput): Promise { // Any fresh request for this session supersedes a pending continuation of - // the same class. A title request only supersedes a pending title; it must - // not clear a primary deferral that another held request still owns, and a - // primary request must not clear a title that is still being held. + // the same class. A title request only supersedes a pending title; a + // compaction request only supersedes a pending compaction; neither may + // clear a primary deferral that another held request still owns. A + // transport guard without a kind is a legacy host and keeps the original + // primary-only behavior. if (input.kind === "title") await this.#removeRecovery(input.sessionID, [TITLE_RECOVERY_ORIGIN]); - else await this.#removeRecovery(input.sessionID, PRIMARY_RECOVERY_ORIGINS); + else if (input.kind === "compaction") + await this.#removeRecovery(input.sessionID, [COMPACTION_RECOVERY_ORIGIN]); + else if (input.kind === "primary" || input.kind === undefined) + await this.#removeRecovery(input.sessionID, PRIMARY_RECOVERY_ORIGINS); await this.#guardPricing(input, undefined); } @@ -583,6 +610,13 @@ export class OffpeakEngine { * OpenCode's own title generation for a session that still needs a title, * and it is cleared without any request when the session was titled (or * manually renamed) while it waited. + * + * A held compaction request (kind `compaction`, manual or automatic) + * persists a semantic `pending-compaction` marker. If the hold is lost + * (plugin or server shutdown, inactivity), recovery admits OpenCode's own + * compaction again at the next allowed window; the runner keeps a pending + * compaction ahead of queued prompts, so a prompt sent behind a + * compact-first workflow still runs after the compaction. */ async guardModelRequest(input: GuardInput): Promise { const origin: RecoveryOrigin | undefined = @@ -590,7 +624,9 @@ export class OffpeakEngine { ? "deferred" : input.kind === "title" ? TITLE_RECOVERY_ORIGIN - : undefined; + : input.kind === "compaction" + ? COMPACTION_RECOVERY_ORIGIN + : undefined; if (origin !== undefined) await this.#removeRecovery(input.sessionID, [origin]); await this.#guardPricing(input, origin); } @@ -679,6 +715,7 @@ export class OffpeakEngine { this.#requestPump(); } const hold = task === undefined ? this.#beginHold(input) : undefined; + this.#trackActiveHold(input.sessionID); const abortEpoch = this.#holdAborts.get(input.sessionID) ?? 0; try { while (!this.#disposed) { @@ -745,9 +782,38 @@ export class OffpeakEngine { if (hold !== undefined) this.#endHold(hold); if (waiter !== undefined) this.#interactiveWaiters.delete(waiter); this.#announcedHolds.delete(input.sessionID); + this.#untrackActiveHold(input.sessionID); } } + #trackActiveHold(sessionID: string): void { + this.#activeHolds.set(sessionID, (this.#activeHolds.get(sessionID) ?? 0) + 1); + } + + #untrackActiveHold(sessionID: string): void { + const count = (this.#activeHolds.get(sessionID) ?? 0) - 1; + if (count > 0) this.#activeHolds.set(sessionID, count); + else this.#activeHolds.delete(sessionID); + // A finished hold can free a recovery entry for this session. + this.#requestPump(); + } + + /** + * OpenCode started executing the session. Recovery treats a running session + * as unsettled until its terminal event arrives. + */ + noteExecutionRunning(sessionID: string): void { + if (this.#disposed) return; + this.#running.add(sessionID); + } + + /** OpenCode settled the session's execution; recovery may evaluate it again. */ + noteExecutionSettled(sessionID: string): void { + this.#running.delete(sessionID); + // The settlement can free a recovery entry for this session. + this.#requestPump(); + } + /** * Park an interactive request. Registration is what makes the request * discoverable and cancellable before it is dispatched. @@ -1000,8 +1066,9 @@ export class OffpeakEngine { /** * Called when OpenCode reports that a session's execution was interrupted. * - * - "user": the user cancelled. A pending deferral is cleared and the hold is - * aborted so nothing continues automatically. + * - "user": the user cancelled. Pending deferrals (primary, title, and + * compaction) are cleared and the hold is aborted so nothing continues + * automatically. * - "inactivity": OpenCode's location-activity sweep ended the execution. The * hold is aborted and a deferral (if present) is kept so the same session * continues once pricing allows. @@ -1011,10 +1078,15 @@ export class OffpeakEngine { async noteExecutionInterrupted(input: { sessionID: string; reason: string }): Promise { if (this.#disposed) return; if (input.reason === "user") { - // The user cancelled the work. Neither the same-session continuation nor - // a pending automatic title is delivered for abandoned work; OpenCode - // itself also generates no title without new input. - await this.#removeRecovery(input.sessionID, ["deferred", TITLE_RECOVERY_ORIGIN]); + // The user cancelled the work. Neither the same-session continuation, a + // pending automatic title, nor a pending compaction is delivered for + // abandoned work; OpenCode itself also generates no title without new + // input. + await this.#removeRecovery(input.sessionID, [ + "deferred", + TITLE_RECOVERY_ORIGIN, + COMPACTION_RECOVERY_ORIGIN, + ]); } if ( input.reason !== "user" && @@ -1099,10 +1171,13 @@ export class OffpeakEngine { * so the existing signal and boundary timer drive it; there is no polling * loop. * - * Primary continuations are delivered first so a same-session continuation - * makes the session busy again before any pending title is considered: the - * continuation's input promotion triggers OpenCode's own title generation, - * and the title marker is then cleared without sending anything. + * Groups flush in native order: a pending compaction is admitted first, so + * OpenCode's runner keeps it ahead of any prompt already queued behind it + * (compact-first stays compact-first across the loss). Primary continuations + * follow so a same-session continuation makes the session busy again before + * any pending title is considered: the continuation's input promotion + * triggers OpenCode's own title generation, and the title marker is then + * cleared without sending anything. */ async #flushRecoveries(): Promise { if (this.#disposed) return; @@ -1116,16 +1191,62 @@ export class OffpeakEngine { const entries = [...this.#recoveries.values()].filter( (entry) => entry.directory === this.#location, ); - for (const entry of entries) { - if (entry.origin === TITLE_RECOVERY_ORIGIN) continue; + const pendingCompactions = entries.filter( + (entry) => entry.origin === COMPACTION_RECOVERY_ORIGIN, + ); + const primaryEntries = entries.filter( + (entry) => + entry.origin !== COMPACTION_RECOVERY_ORIGIN && entry.origin !== TITLE_RECOVERY_ORIGIN, + ); + const titleEntries = entries.filter((entry) => entry.origin === TITLE_RECOVERY_ORIGIN); + // A compaction admitted in this pass and a primary continuation for the + // same session belong together: the runner admits the compaction first and + // the continuation behind it, so the earlier unfinished work resumes in + // native order instead of being considered stale once the compaction + // succeeds. + const compactionAdmitted = new Set(); + for (const entry of pendingCompactions) { const decision = evaluatePricing( settings.config.policies, entry.providerID, entry.modelID, this.#clock.now(), ); - if (settings.config.enabled && decision.allowed) { - const retryAt = await this.#deliverRecovery(entry); + // A compaction recovery is admitted as soon as the session is settled: + // the re-triggered compaction request itself holds at the pre-model + // guard through the peak window (zero provider traffic), and OpenCode's + // runner keeps a pending compaction ahead of queued prompts, so + // compact-first ordering survives without waiting for the boundary. In + // queue/guard modes a re-trigger during peak would fail closed, so it + // waits for the allowed window. + const deliverable = + settings.config.enabled && (settings.config.mode === "auto" || decision.allowed); + if (deliverable) { + const result = await this.#deliverCompactionRecovery(entry); + if (result.retryAt !== undefined) + next = next === undefined ? result.retryAt : Math.min(next, result.retryAt); + if (result.admitted) compactionAdmitted.add(entry.sessionID); + continue; + } + if (settings.config.enabled && decision.nextTransition !== undefined) { + next = + next === undefined ? decision.nextTransition : Math.min(next, decision.nextTransition); + } + } + for (const entry of primaryEntries) { + const decision = evaluatePricing( + settings.config.policies, + entry.providerID, + entry.modelID, + this.#clock.now(), + ); + // A continuation that belongs with a compaction admitted in this pass is + // delivered with it: the runner keeps the compaction ahead, so the + // continuation waits behind the checkpoint while the peak window holds + // both. + const forced = compactionAdmitted.has(entry.sessionID); + if (settings.config.enabled && (forced || decision.allowed)) { + const retryAt = await this.#deliverRecovery(entry, { force: forced }); if (retryAt !== undefined) next = next === undefined ? retryAt : Math.min(next, retryAt); continue; } @@ -1134,8 +1255,7 @@ export class OffpeakEngine { next === undefined ? decision.nextTransition : Math.min(next, decision.nextTransition); } } - for (const entry of entries) { - if (entry.origin !== TITLE_RECOVERY_ORIGIN) continue; + for (const entry of titleEntries) { const decision = evaluatePricing( settings.config.policies, entry.providerID, @@ -1156,7 +1276,10 @@ export class OffpeakEngine { } /** Returns a timestamp to retry at when the session has not settled yet. */ - async #deliverRecovery(entry: RecoveryEntry): Promise { + async #deliverRecovery( + entry: RecoveryEntry, + options?: { force?: boolean }, + ): Promise { let session: SessionSnapshot; try { session = await this.#sessions.get(entry.sessionID); @@ -1165,21 +1288,33 @@ export class OffpeakEngine { await this.#removeRecovery(entry.sessionID, [entry.origin]); return undefined; } - if (session.outcome === "succeeded") { - await this.#removeRecovery(entry.sessionID, [entry.origin]); - return undefined; - } - if (session.outcome === undefined) { - const firstCheck = this.#recoveryChecks.get(entry.sessionID) ?? this.#clock.now(); - this.#recoveryChecks.set(entry.sessionID, firstCheck); - if (this.#clock.now() - firstCheck < RECOVERY_SETTLE_CAP_MS) { - return this.#clock.now() + RECOVERY_SETTLE_DELAY_MS; + // A forced delivery belongs to a session whose compaction was just + // admitted in the same pass: the session is intentionally busy again, so + // the pending continuation is not stale. + if (!options?.force) { + // A running session or a live hold owns its turn: session.outcome only + // updates at settlement, so waiting avoids racing a live release into a + // duplicate continuation. Ending the hold or settling the execution + // wakes the pump again, so this pass can simply skip the entry. + if (this.#running.has(entry.sessionID) || this.#activeHolds.has(entry.sessionID)) { + return undefined; + } + if (session.outcome === "succeeded") { + await this.#removeRecovery(entry.sessionID, [entry.origin]); + return undefined; + } + if (session.outcome === undefined) { + const firstCheck = this.#recoveryChecks.get(entry.sessionID) ?? this.#clock.now(); + this.#recoveryChecks.set(entry.sessionID, firstCheck); + if (this.#clock.now() - firstCheck < RECOVERY_SETTLE_CAP_MS) { + return this.#clock.now() + RECOVERY_SETTLE_DELAY_MS; + } + await this.#removeRecovery(entry.sessionID, [entry.origin]); + this.#onError( + new Error(`Offpeak recovery for ${entry.sessionID} skipped: the session did not settle.`), + ); + return undefined; } - await this.#removeRecovery(entry.sessionID, [entry.origin]); - this.#onError( - new Error(`Offpeak recovery for ${entry.sessionID} skipped: the session did not settle.`), - ); - return undefined; } // Remove before sending so the continuation is delivered at most once. await this.#removeRecovery(entry.sessionID, [entry.origin]); @@ -1194,6 +1329,64 @@ export class OffpeakEngine { return undefined; } + /** + * Delivers a pending-compaction recovery. This is a semantic watch, not a + * replay: OpenCode's own compaction is admitted again for the session, and + * its runner keeps a pending compaction ahead of any prompt already queued + * behind it, so compact-first ordering survives a lost hold (shutdown, + * reload, or the inactivity sweep). + * + * The entry is removed before admission so the compaction is requested at + * most once. A session that already succeeded after the lost hold, or never + * settles, is dropped without admission, exactly like a primary + * continuation. + */ + async #deliverCompactionRecovery( + entry: RecoveryEntry, + ): Promise<{ retryAt?: number; admitted: boolean }> { + let session: SessionSnapshot; + try { + session = await this.#sessions.get(entry.sessionID); + } catch (error) { + this.#onError(error); + await this.#removeRecovery(entry.sessionID, [COMPACTION_RECOVERY_ORIGIN]); + return { admitted: false }; + } + // A running session or a live hold owns its turn: session.outcome only + // updates at settlement, so waiting avoids racing a live release into a + // duplicate compaction. Ending the hold or settling the execution wakes + // the pump again, so this pass can simply skip the entry. + if (this.#running.has(entry.sessionID) || this.#activeHolds.has(entry.sessionID)) { + return { admitted: false }; + } + // Unlike a primary continuation, the marker is not dropped when the + // session has since succeeded: the user asked for a compaction that never + // ran, so it is admitted as soon as the session is settled and idle. + if (session.outcome === undefined) { + const firstCheck = this.#recoveryChecks.get(entry.sessionID) ?? this.#clock.now(); + this.#recoveryChecks.set(entry.sessionID, firstCheck); + if (this.#clock.now() - firstCheck < RECOVERY_SETTLE_CAP_MS) { + return { retryAt: this.#clock.now() + RECOVERY_SETTLE_DELAY_MS, admitted: false }; + } + await this.#removeRecovery(entry.sessionID, [COMPACTION_RECOVERY_ORIGIN]); + this.#onError( + new Error( + `Offpeak compaction recovery for ${entry.sessionID} skipped: the session did not settle.`, + ), + ); + return { admitted: false }; + } + // Remove before admitting so the compaction is requested at most once. + await this.#removeRecovery(entry.sessionID, [COMPACTION_RECOVERY_ORIGIN]); + try { + await this.#sessions.compact(entry.sessionID); + } catch (error) { + this.#onError(error); + return { admitted: false }; + } + return { admitted: true }; + } + /** * Delivers a pending-title recovery. This is a semantic watch, not a replay: * diff --git a/src/server.ts b/src/server.ts index 6cc1a90..437db3c 100644 --- a/src/server.ts +++ b/src/server.ts @@ -2,14 +2,14 @@ import { Plugin } from "@opencode/plugin"; import { assertSupportedOpenCode } from "./compat.js"; import { ConfigRepository } from "./config.js"; import { type EngineNotification, OffpeakEngine, type SessionDriver } from "./engine.js"; -import { OFFPEAK_RPC, type OffpeakRpcInput, type OffpeakRpcOutput } from "./rpc.js"; import { isRefusedHandshakeUrl, isRefusedRequestUrl, - REFUSED_HEADER, REFUSED_HANDSHAKE_URL, + REFUSED_HEADER, refusedRequest, } from "./refused.js"; +import { OFFPEAK_RPC, type OffpeakRpcInput, type OffpeakRpcOutput } from "./rpc.js"; import { QueueRepository, RecoveryRepository } from "./store.js"; export default Plugin.define({ @@ -58,6 +58,11 @@ export default Plugin.define({ interrupt: async (sessionID) => { await context.session.interrupt({ sessionID }); }, + compact: async (sessionID) => { + // Semantic compaction recovery: admit OpenCode's own compaction + // again. It is never a replay of the lost request. + await context.session.compact({ sessionID }); + }, }; // Hold/resume notifications travel as plugin RPC events on OpenCode's own // event stream; a TUI renders them as toasts, and headless runs simply have @@ -173,7 +178,22 @@ export default Plugin.define({ try { for await (const event of context.event.subscribe({ signal: eventAbort.signal })) { if (eventAbort.signal.aborted) break; + if (event.type === "session.execution.started") { + // Recovery delivery waits while the session is executing: + // session.outcome only updates at settlement, so a running turn + // still reports its previous outcome. + engine.noteExecutionRunning(event.data.sessionID); + continue; + } + if ( + event.type === "session.execution.succeeded" || + event.type === "session.execution.failed" + ) { + engine.noteExecutionSettled(event.data.sessionID); + continue; + } if (event.type === "session.execution.interrupted") { + engine.noteExecutionSettled(event.data.sessionID); await engine.noteExecutionInterrupted({ sessionID: event.data.sessionID, reason: event.data.reason, diff --git a/src/store.ts b/src/store.ts index fc4ee6c..4294c62 100644 --- a/src/store.ts +++ b/src/store.ts @@ -50,6 +50,11 @@ export interface QueueDocument { * title. Recovery is semantic (OpenCode's own title generation is invoked * again; no provider request is replayed) and never overwrites a title the * session already has. + * - "pending-compaction": a compaction request deferred before its provider + * attempt. Recovery is semantic — OpenCode's own compaction is requested + * again at the next allowed window, and its runner keeps a pending + * compaction ahead of any queued prompt, so compact-first ordering is + * preserved without replaying the original request. */ export interface RecoveryEntry { sessionID: string; @@ -60,7 +65,7 @@ export interface RecoveryEntry { origin?: RecoveryOrigin; } -export type RecoveryOrigin = "retry-failure" | "deferred" | "pending-title"; +export type RecoveryOrigin = "retry-failure" | "deferred" | "pending-title" | "pending-compaction"; export interface RecoveryDocument { schemaVersion: 1; @@ -236,7 +241,8 @@ function validateRecoveryEntry(value: unknown, index: number): RecoveryEntry { value.origin !== undefined && value.origin !== "retry-failure" && value.origin !== "deferred" && - value.origin !== "pending-title" + value.origin !== "pending-title" && + value.origin !== "pending-compaction" ) { throw new Error(`Stored recovery entry ${index} has an invalid origin.`); } diff --git a/src/tui.ts b/src/tui.ts index 5aec2d5..90d26f5 100644 --- a/src/tui.ts +++ b/src/tui.ts @@ -142,6 +142,7 @@ function summarize(output: OffpeakRpcOutput): string { waiting?: number; interactiveWaiting?: number; recoveryWaiting?: number; + compactionRecoveryWaiting?: number; running?: { id?: string; sessionID?: string; state?: string }; failed?: number; configError?: string; @@ -181,6 +182,9 @@ function summarize(output: OffpeakRpcOutput): string { status.recoveryWaiting ? `Recovery: ${status.recoveryWaiting} session${status.recoveryWaiting === 1 ? "" : "s"} waiting for off-peak` : "", + status.compactionRecoveryWaiting + ? `Compaction recovery: ${status.compactionRecoveryWaiting} session${status.compactionRecoveryWaiting === 1 ? "" : "s"} waiting for off-peak` + : "", `Running: ${status.running?.id ?? "none"}${status.running?.sessionID ? ` (session ${status.running.sessionID})` : ""}`, status.configError ? `Config error: ${status.configError}` : "", status.enabled diff --git a/test/engine.test.mjs b/test/engine.test.mjs index 355d53f..5b666d1 100644 --- a/test/engine.test.mjs +++ b/test/engine.test.mjs @@ -78,6 +78,7 @@ class ManualClock { class FakeSessions { next = 0; prompts = []; + compactions = []; workers = new Map(); outcomes = new Map(); behavior = async () => undefined; @@ -119,6 +120,9 @@ class FakeSessions { async interrupt(sessionID) { this.outcomes.set(sessionID, "interrupted"); } + async compact(sessionID) { + this.compactions.push(sessionID); + } } async function until(predicate, message = "condition was not reached") { @@ -1432,7 +1436,7 @@ test("an auxiliary model.request hold waits without a primary deferral; a title } }); -test("compaction and generate model.request holds register no recovery of any kind", async () => { +test("a compaction model.request hold persists a pending-compaction marker; a generate hold registers nothing", async () => { const clock = new ManualClock("2026-10-12T01:30:00Z"); const h = await harness("2026-10-12T01:30:00Z", { clock }); const holds = [ @@ -1455,10 +1459,371 @@ test("compaction and generate model.request holds register no recovery of any ki ]; try { await until(async () => (await h.engine.status(selectedModel)).interactiveWaiting === 2); - assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 0); + // A held compaction persists a semantic pending-compaction marker; a held + // generate request has no recovery of its own. + assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 1); + assert.equal((await h.engine.status(selectedModel)).compactionRecoveryWaiting, 1); await clock.advanceTo("2026-10-12T04:00:00Z"); assert.deepEqual(await Promise.all(holds), ["released", "released"]); assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 0); + assert.equal((await h.engine.status(selectedModel)).compactionRecoveryWaiting, 0); + } finally { + await h.close(); + } +}); + +test("a held compaction that fails closed keeps its marker and admits OpenCode's compaction once at the boundary", async () => { + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock }); + const sessionID = "compaction-recovery-session"; + const pending = h.engine + .guardModelRequest({ sessionID, ...selectedModel, kind: "compaction", transport: "model" }) + .catch(() => "blocked"); + try { + await until(async () => (await h.engine.status(selectedModel)).compactionRecoveryWaiting === 1); + // The plugin reloads: the held compaction fails closed, the marker is kept. + await h.engine.dispose(); + assert.equal(await pending, "blocked"); + const restarted = new OffpeakEngine({ + repository: new QueueRepository(h.storage), + recoveries: new RecoveryRepository(h.storage), + location: "test-location", + config: h.config, + sessions: h.sessions, + clock, + }); + // The reloaded turn failed while its compaction was deferred. + h.sessions.outcomes.set(sessionID, "failed"); + await restarted.start(); + // A settled session is recovered as soon as the instance loads: the + // pending compaction is admitted again without waiting for the boundary, + // and it holds at the pre-model guard through the window like any other + // compaction. Recovery is semantic: no continuation prompt is sent. + await until(async () => h.sessions.compactions.length === 1); + assert.equal(h.sessions.compactions[0], sessionID); + assert.equal(h.sessions.prompts.length, 0); + assert.equal((await restarted.status(selectedModel)).compactionRecoveryWaiting, 0); + await clock.advanceTo("2026-10-12T04:00:00Z"); + await clock.advanceTo("2026-10-13T00:00:00Z"); + await new Promise((resolve) => setTimeout(resolve, 25)); + assert.equal(h.sessions.compactions.length, 1); + assert.equal(h.sessions.prompts.length, 0); + await restarted.dispose(); + } finally { + await h.close(); + } +}); + +test("a live compaction hold with a stale failed outcome releases once without a duplicate admission", async () => { + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock }); + const sessionID = "compaction-race-session"; + let released = false; + const hold = h.engine + .guardModelRequest({ sessionID, ...selectedModel, kind: "compaction", transport: "model" }) + .then( + () => { + released = true; + }, + () => { + released = true; + }, + ); + try { + // The session's last settled turn failed, but its live hold still owns the + // boundary release; recovery must not race it into a second compaction. + h.sessions.outcomes.set(sessionID, "failed"); + await until(async () => (await h.engine.status(selectedModel)).compactionRecoveryWaiting === 1); + await clock.advanceTo("2026-10-12T04:00:00Z"); + await until(() => released); + await new Promise((resolve) => setTimeout(resolve, 30)); + assert.equal(h.sessions.compactions.length, 0); + assert.equal(h.sessions.prompts.length, 0); + assert.equal((await h.engine.status(selectedModel)).compactionRecoveryWaiting, 0); + await hold; + } finally { + await h.close(); + } +}); + +test("a live primary hold with a stale failed outcome releases without a duplicate continuation", async () => { + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock }); + const sessionID = "primary-race-session"; + let released = false; + const hold = h.engine + .guardModelRequest({ sessionID, ...selectedModel, kind: "primary", transport: "model" }) + .then( + () => { + released = true; + }, + () => { + released = true; + }, + ); + try { + h.sessions.outcomes.set(sessionID, "failed"); + await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 1); + await clock.advanceTo("2026-10-12T04:00:00Z"); + await until(() => released); + await new Promise((resolve) => setTimeout(resolve, 30)); + assert.equal(h.sessions.prompts.length, 0); + assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 0); + await hold; + } finally { + await h.close(); + } +}); + +test("a compaction recovery and a primary continuation for one session are admitted together in compact-first order", async () => { + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock }); + const sessionID = "compact-first-session"; + const order = []; + h.sessions.compact = async (id) => { + order.push(`compact:${id}`); + h.sessions.compactions.push(id); + // The recovered compaction completes quickly; the continuation must still + // be admitted for the unfinished earlier work instead of being considered + // stale. + h.sessions.outcomes.set(id, "succeeded"); + }; + try { + const first = h.engine + .guardModelRequest({ sessionID, ...selectedModel, kind: "primary", transport: "model" }) + .catch(() => "blocked"); + await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 1); + await h.engine.dispose(); + assert.equal(await first, "blocked"); + + const second = new OffpeakEngine({ + repository: new QueueRepository(h.storage), + recoveries: new RecoveryRepository(h.storage), + location: "test-location", + config: h.config, + sessions: h.sessions, + clock, + }); + await second.start(); + const held = second + .guardModelRequest({ sessionID, ...selectedModel, kind: "compaction", transport: "model" }) + .catch(() => "blocked"); + await until(async () => (await second.status(selectedModel)).compactionRecoveryWaiting === 1); + await second.dispose(); + assert.equal(await held, "blocked"); + + const third = new OffpeakEngine({ + repository: new QueueRepository(h.storage), + recoveries: new RecoveryRepository(h.storage), + location: "test-location", + config: h.config, + sessions: h.sessions, + clock, + }); + h.sessions.outcomes.set(sessionID, "failed"); + await third.start(); + // The settled session's compaction is admitted first, and the primary + // continuation is admitted with it — the runner keeps the compaction + // ahead, so the earlier unfinished work resumes after the checkpoint. + await until(async () => h.sessions.prompts.length === 1); + assert.deepEqual(order, [`compact:${sessionID}`]); + assert.equal(h.sessions.prompts.length, 1); + assert.equal(h.sessions.prompts[0].sessionID, sessionID); + assert.equal(h.sessions.prompts[0].text, CONTINUATION); + assert.equal((await third.status(selectedModel)).recoveryWaiting, 0); + await third.dispose(); + } finally { + await h.close(); + } +}); + +test("a pending compaction waits while the session executes and is admitted once the session is idle", async () => { + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock }); + const sessionID = "compaction-running-session"; + const pending = h.engine + .guardModelRequest({ sessionID, ...selectedModel, kind: "compaction", transport: "model" }) + .catch(() => "blocked"); + try { + await until(async () => (await h.engine.status(selectedModel)).compactionRecoveryWaiting === 1); + await h.engine.dispose(); + assert.equal(await pending, "blocked"); + const restarted = new OffpeakEngine({ + repository: new QueueRepository(h.storage), + recoveries: new RecoveryRepository(h.storage), + location: "test-location", + config: h.config, + sessions: h.sessions, + clock, + }); + h.sessions.outcomes.set(sessionID, "failed"); + // OpenCode reports the session as executing before the instance loads: + // the marker waits without being admitted. + restarted.noteExecutionRunning(sessionID); + await restarted.start(); + await new Promise((resolve) => setTimeout(resolve, 30)); + assert.equal(h.sessions.compactions.length, 0); + assert.equal((await restarted.status(selectedModel)).compactionRecoveryWaiting, 1); + // The turn settled (even successfully): the marker is still admitted as + // soon as the settlement wakes the engine — the requested compaction + // never ran and is not dropped by another completed turn. + h.sessions.outcomes.set(sessionID, "succeeded"); + restarted.noteExecutionSettled(sessionID); + await until(async () => h.sessions.compactions.length === 1); + assert.equal(h.sessions.compactions[0], sessionID); + assert.equal((await restarted.status(selectedModel)).compactionRecoveryWaiting, 0); + await restarted.dispose(); + } finally { + await h.close(); + } +}); + +test("inactivity and superseded interrupts keep a compaction marker; the settled session is then admitted once", async () => { + const storage = new MemoryStorage(); + await storage.set("opencode-offpeak/recovery-v1", { + schemaVersion: 1, + entries: [ + { + sessionID: "compaction-interrupt-session", + providerID: selectedModel.providerID, + modelID: selectedModel.modelID, + directory: "test-location", + heldAt: "2026-10-12T01:30:00.000Z", + origin: "pending-compaction", + }, + ], + }); + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const sessions = new FakeSessions(); + const root = await mkdtemp(path.join(os.tmpdir(), "opencode-offpeak-test-")); + const engine = new OffpeakEngine({ + repository: new QueueRepository(storage), + recoveries: new RecoveryRepository(storage), + location: "test-location", + config: new ConfigRepository({ configPath: path.join(root, "offpeak.json") }), + sessions, + clock, + }); + const sessionID = "compaction-interrupt-session"; + try { + sessions.outcomes.set(sessionID, "failed"); + // Block delivery while the interrupt semantics are exercised. + engine.noteExecutionRunning(sessionID); + await engine.start(); + assert.equal((await engine.status(selectedModel)).compactionRecoveryWaiting, 1); + await engine.noteExecutionInterrupted({ sessionID, reason: "inactivity" }); + assert.equal((await engine.status(selectedModel)).compactionRecoveryWaiting, 1); + await engine.noteExecutionInterrupted({ sessionID, reason: "superseded" }); + assert.equal((await engine.status(selectedModel)).compactionRecoveryWaiting, 1); + // The settled session is admitted exactly once. + engine.noteExecutionSettled(sessionID); + await until(async () => sessions.compactions.length === 1); + assert.equal(sessions.compactions[0], sessionID); + assert.equal((await engine.status(selectedModel)).compactionRecoveryWaiting, 0); + await engine.dispose(); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); + +test("primary, title, and compaction recoveries coexist for one session and no guard clears another class", async () => { + const storage = new MemoryStorage(); + const entry = (origin) => ({ + sessionID: "coexist-session", + providerID: selectedModel.providerID, + modelID: selectedModel.modelID, + directory: "test-location", + heldAt: "2026-10-12T01:30:00.000Z", + origin, + }); + await storage.set("opencode-offpeak/recovery-v1", { + schemaVersion: 1, + entries: [entry("deferred"), entry("pending-title"), entry("pending-compaction")], + }); + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock, storage }); + try { + let status = await h.engine.status(selectedModel); + assert.equal(status.recoveryWaiting, 3); + assert.equal(status.titleRecoveryWaiting, 1); + assert.equal(status.compactionRecoveryWaiting, 1); + // A transport guard for a compaction clears only the compaction marker. + const compactionGuard = h.engine + .guardRequest({ + sessionID: "coexist-session", + ...selectedModel, + kind: "compaction", + transport: "http", + }) + .catch(() => "blocked"); + await until(async () => (await h.engine.status(selectedModel)).compactionRecoveryWaiting === 0); + status = await h.engine.status(selectedModel); + assert.equal(status.recoveryWaiting, 2); + assert.equal(status.titleRecoveryWaiting, 1); + // A generate request has no recovery class of its own and clears nothing. + const generateGuard = h.engine + .guardRequest({ + sessionID: "coexist-session", + ...selectedModel, + kind: "generate", + transport: "http", + }) + .catch(() => "blocked"); + await new Promise((resolve) => setTimeout(resolve, 30)); + assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 2); + // A primary request clears only the primary classes. + const primaryGuard = h.engine + .guardRequest({ + sessionID: "coexist-session", + ...selectedModel, + kind: "primary", + transport: "http", + }) + .catch(() => "blocked"); + await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 1); + status = await h.engine.status(selectedModel); + assert.equal(status.titleRecoveryWaiting, 1); + // A title request clears only the title marker. + const titleGuard = h.engine + .guardRequest({ + sessionID: "coexist-session", + ...selectedModel, + kind: "title", + transport: "http", + }) + .catch(() => "blocked"); + await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 0); + for (const guard of [compactionGuard, generateGuard, primaryGuard, titleGuard]) { + guard.catch(() => undefined); + } + } finally { + await h.close(); + } +}); + +test("a user interrupt clears primary, title, and compaction recoveries together", async () => { + const storage = new MemoryStorage(); + const entry = (origin) => ({ + sessionID: "cancel-session", + providerID: selectedModel.providerID, + modelID: selectedModel.modelID, + directory: "test-location", + heldAt: "2026-10-12T01:30:00.000Z", + origin, + }); + await storage.set("opencode-offpeak/recovery-v1", { + schemaVersion: 1, + entries: [entry("deferred"), entry("pending-title"), entry("pending-compaction")], + }); + const clock = new ManualClock("2026-10-12T01:30:00Z"); + const h = await harness("2026-10-12T01:30:00Z", { clock, storage }); + try { + assert.equal((await h.engine.status(selectedModel)).recoveryWaiting, 3); + await h.engine.noteExecutionInterrupted({ sessionID: "cancel-session", reason: "user" }); + const status = await h.engine.status(selectedModel); + assert.equal(status.recoveryWaiting, 0); + assert.equal(status.titleRecoveryWaiting, 0); + assert.equal(status.compactionRecoveryWaiting, 0); } finally { await h.close(); } @@ -1899,6 +2264,9 @@ test("a delivered primary continuation lets OpenCode name the session; the title try { await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 2); assert.equal((await h.engine.status(selectedModel)).titleRecoveryWaiting, 1); + // Both held requests are lost (plugin reload); both markers survive. + await h.engine.dispose(); + assert.deepEqual(await Promise.all(holds), ["blocked", "blocked"]); // OpenCode's own promotion of the continuation names the session. h.sessions.outcomes.set(sessionID, "interrupted"); h.sessions.workers.set(sessionID, Promise.resolve()); @@ -1906,13 +2274,22 @@ test("a delivered primary continuation lets OpenCode name the session; the title h.sessions.snapshots.set(id, { title: "Continuation title", created }); return { outcome: "succeeded" }; }; + const restarted = new OffpeakEngine({ + repository: new QueueRepository(h.storage), + recoveries: new RecoveryRepository(h.storage), + location: "test-location", + config: h.config, + sessions: h.sessions, + clock, + }); + await restarted.start(); await clock.advanceTo("2026-10-12T04:00:00Z"); await until(async () => h.sessions.prompts.length === 1); - await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 0); + await until(async () => (await restarted.status(selectedModel)).recoveryWaiting === 0); // The continuation carried the title; the marker never generated one itself. assert.equal(h.sessions.titles.length, 0); assert.equal(h.sessions.snapshots.get(sessionID).title, "Continuation title"); - await Promise.all(holds); + await restarted.dispose(); } finally { await h.close(); } @@ -2099,7 +2476,9 @@ test("auxiliary title, compaction, and generate holds never announce a notificat ), ); try { - await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 1); + // The title hold persists a pending-title marker; the compaction hold + // persists a pending-compaction marker; the generate hold has none. + await until(async () => (await h.engine.status(selectedModel)).recoveryWaiting === 2); assert.deepEqual(h.notifications, []); // A title hold is released immediately (it has no peak window for this model // and never waits); the others stay held until the boundary.