Skip to content

Close interrupted runs during restart recovery - #9

Draft
yubnsbski wants to merge 1 commit into
mainfrom
agent/interrupted-runs
Draft

Close interrupted runs during restart recovery#9
yubnsbski wants to merge 1 commit into
mainfrom
agent/interrupted-runs

Conversation

@yubnsbski

Copy link
Copy Markdown
Owner

Summary

  • close every in-flight Run associated with an interrupted active task as interrupted
  • set ended_at, error_class, and a structured interruption reason
  • re-queue the task, close its Run, and write task.recovered evidence in one SQLite transaction
  • include the exact interrupted Run IDs in the recovery audit event
  • leave already-completed Runs unchanged
  • keep restart recovery idempotent

Root cause

Restart recovery moved active tasks back to queued, but their Run records stayed running forever. The dashboard and audit history could therefore claim a model was still active after the process had already died, and a later retry would create another Run beside the orphan.

There was also a crash window between committing the recovered task state and writing its audit event because those operations used separate transactions.

Behavior

When WBZ starts and finds a task in planning, executing, verifying, or reviewing, it atomically:

  1. marks that task's running Runs as interrupted;
  2. records a common recovery timestamp and reason;
  3. moves the task back to queued;
  4. writes task.recovered with the interrupted Run IDs.

Calling recovery again makes no further changes.

Validation

  • python -m pytest -q — 23 passed
  • python -m compileall -q src tests
  • git diff --check
  • regression coverage verifies Run closure, completed-Run preservation, audit linkage, and idempotent second recovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant