Skip to content

Resume accepted inside a delete's commit window waits the delete out instead of exiting - #12

Merged
ben-vargas merged 2 commits into
mainfrom
fix/resume-inside-delete-window
Oct 10, 2026
Merged

ben-vargas merged 2 commits into
mainfrom
fix/resume-inside-delete-window

Conversation

@ben-vargas

Copy link
Copy Markdown
Owner

Fixes the production follow-up flagged in #10. A resume accepted while a delete of the same run was committing could fail quietly.

The problem

A launcher can accept a detached resume inside a delete's commit window. That is after the delete's checks and before its rename. The launcher installs .resuming and spawns the engine, and the delete then finds the marker and rolls back (§7.3.3: delete loses). But the spawned engine is a separate process, and it can reach the run before that rollback finishes:

  • Before the rename, or after the restore but before release: the delete still holds run.lock, so the engine refuses with run is already being executed by pid <delete's pid>.
  • Between the rename and the restore: the run is in the trash, so the child's journal read fails with no journal.

Either way the engine exits and does not retry. The accepted resume never runs, and its marker ages out after 30s. Safety held, since the run was never both resumed and deleted, but the resume was lost.

The fix

  • retention marks its run lock op: "delete" (src/run-lock.js acquireRunLock(dir, { op })).

  • deleteInProgress(dir, trashRoot, runId) checks for a lock marked op: "delete", held by a live process, written within the last minute. It looks at the run's directory and at the run's trash entries, since the lock travels into the trash with a committing delete.

  • A resume waits up to 15s (settleDelete, polling every 50ms) while a delete is in progress on its run, then proceeds:

    • in the CLI's run --resume and resume paths, before the journal read (these are the paths the CLI, MCP and viewer launchers' children enter);
    • in the engine's ownership step (claimRun), around the directory check and lock acquisition.

    If the delete rolled back, the run is back and the resume takes it. If it committed, the resume fails exactly as before.

  • Unchanged:

    • A resume never creates anything under runs/<id> before it owns the run. The single linearization point (the delete's rename) holds.
    • Fresh runs and resumes with no delete in progress behave as before, with no wait.

Tests (test/retention.test.js)

  • New, ×2: an accepted detached resume (the shipped launcher, its engine held by Fix flaky retention test: resume launched inside the delete's commit window #10's spawn gate) is released while the delete holds its lock, and again while the run is in the trash. The test then asserts that the engine takes ownership after the rollback. Both fail on main.
  • Changed: a resume running INSIDE the post-rename window cannot resurrect the run. The in-process resume now waits instead of refusing. The test still asserts that no resume path recreated runs/<id> and that the rollback restored the run byte for byte. It then asserts that the waiting resume takes the run. This test fails on main.
  • Root suite: 553/553 locally (node 24). Retention: 33/33.

Docs: ARCHITECTURE.md (run.lock), CHANGELOG (Unreleased → Fixed).

…te out instead of exiting

A launcher can accept a resume by installing .resuming after a delete's checks and
before its rename; the delete then finds the marker and rolls back. But an engine that
started before the rollback finished found the run in the trash (no journal) or under
the delete's lock (already being executed) and exited, so the accepted resume never ran.

retention now marks its run lock op: 'delete'. A resume (the CLI's run --resume and
resume paths, which the CLI, MCP and viewer launchers' children enter, and the engine's
ownership step) waits up to 15s while a live delete holds the run, at its path or in a
trash entry, then proceeds: the run is back if the delete rolled back, gone if it
committed. Tests cover the engine starting under the delete's lock and while the run is
in the trash (both failed before), and the post-rename test now expects the waiting
resume to take the run after the rollback without ever recreating runs/<id>.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T03:10:42.152361Z eab6d90 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eab6d90210

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/engine.js
@ben-vargas
ben-vargas merged commit a025b53 into main Oct 10, 2026
8 checks passed
@ben-vargas

Copy link
Copy Markdown
Owner Author

@codex Thanks, valid. Fixed in a follow-up PR (#13): once no delete is in progress, the engine makes one final claim on the settled state, with a deterministic regression test.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

ben-vargas added a commit that referenced this pull request Oct 10, 2026
Follow-up to #12 from its Codex review. claimRun checked for a delete in progress only after a claim failed, so a delete that finished in between left the resume exiting with a stale error although the run was back. Once no delete is in progress, one more claim decides. Deterministic regression test in test/retention.test.js.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant