Skip to content

fix(orchestrator): record failed stage in state.json when a handler raises - #11

Closed
wangzhengzhuo05 wants to merge 1 commit into
cdxiaodong:mainfrom
wangzhengzhuo05:fix/orchestrator-stage-exception-state
Closed

wangzhengzhuo05 wants to merge 1 commit into
cdxiaodong:mainfrom
wangzhengzhuo05:fix/orchestrator-stage-exception-state

Conversation

@wangzhengzhuo05

Copy link
Copy Markdown

What

Orchestrator.run_stage now records a failed stage in state.json before re-raising, so a handler crash during recon / test / report no longer leaves the workspace with stage artifacts on disk but no record of what happened.

Why

Fixes #10. As reported, when a stage handler raised, run_stage exited before _save_state, so:

  • completed_stages stayed empty and history never saw the attempt;
  • partial artifacts from the failed stage were orphaned with no trace;
  • the caller (cmd_run) got an exception and then a load_state() that printed nothing useful.

How

  • src/cain_agent/orchestrator.py — wrap the handler call in try / except Exception; on failure persist current_stage, the unchanged completed_stages, updated_at, history (with a {"status": "failed", "error": "..."} entry) plus failed_stage and error, then re-raise the original exception unchanged.
  • The success path is deliberately unchanged: same four state keys and the same history entry shape. A later successful run_stage rewrites the state dict, so failed_stage / error clear themselves — that is the recovery path covered by test 3.
  • run() needed no change; it inherits the recording because run_stage re-raises.

Tests

  • .venv/bin/python -m pytest tests/test_orchestrator.py -q → 14 passed
  • full suite .venv/bin/python -m pytest -q → 1118 passed, 3 skipped
  • ruff check on both changed files → clean

Three new tests:

  1. test_failed_stage_is_recorded_in_state — record present, completed_stages empty, history has the failed entry, state.json on disk.
  2. test_run_records_state_when_a_handler_raises — the run() path persists the failure too.
  3. test_failure_marker_is_cleared_after_a_successful_stage — a successful retry drops failed_stage.

Mutation check

Reverted only the fix's behaviour (except Exceptionexcept ZeroDivisionError) with the new tests left in place:

FAILED tests/test_orchestrator.py::test_failed_stage_is_recorded_in_state - KeyError: 'failed_stage'
FAILED tests/test_orchestrator.py::test_run_records_state_when_a_handler_raises - KeyError: 'failed_stage'
2 failed, 1 passed

Restored → 14 passed. The tests fail for the intended reason, so they guard the new behaviour instead of passing vacuously.

AI disclosure

This change was implemented with AI assistance (OpenCode + muse-spark 1.3) and reviewed, tested, and mutation-checked by the submitter.

Fixes #10

@wangzhengzhuo05

Copy link
Copy Markdown
Author

Closing this out.

It has been waiting 7 days without any maintainer review, and we are winding down this contribution effort — a stale PR sitting in the queue costs maintainers attention without giving them a decision to make, so it is better closed than left open indefinitely.

If the direction is useful, the branch is intact on the fork (fix/orchestrator-stage-exception-state, +66/−3 across 2 file(s)) and can be picked up as-is.

What it contains: fix(orchestrator): record failed stage in state.json when a handler raisesOrchestrator.run_stage now records a failed stage in state.json before re-raising, so a handler crash during recon / test / report no longer leaves the workspace with sta

It was verified locally before opening (target test suite green, plus a mutation check that reverts only the fix and confirms the new regression test fails). Thanks for the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Orchestrator.run does not handle stage handler exceptions, leaving state inconsistent

1 participant