Skip to content

fix(lifecycle): treat cancellation during actor startup as graceful shutdown - #17

Merged
dorothyyzh merged 1 commit into
mainfrom
fix/lifecycle-cancel-during-startup
Jul 8, 2026
Merged

dorothyyzh merged 1 commit into
mainfrom
fix/lifecycle-cancel-during-startup

Conversation

@dorothyyzh

@dorothyyzh dorothyyzh commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Slack: https://theplant.slack.com/archives/C09H5U13GJY/p1783419772718129

Problem

A shutdown request (context cancellation) that arrives while an actor is still starting is reported as a startup failure. The startup loop in Serve returns the actor's error unconditionally:

if err := actor.Start(ctx); err != nil {
    logger.ErrorContext(...)
    return err
}

…while the run phase already treats context.Canceled as a clean exit:

err := g.Wait()
if err == nil || errors.Is(err, context.Canceled) || errors.Is(err, errServiceCompleted) {
    return nil
}

So the same shutdown signal is graceful if it lands one millisecond later, but a "failure" if it races the startup sequence.

Real-world impact

  • Flaky TestHealthyEndpoints in ciam-next (failing run): the test cancels the serve context right after health checks pass; when the cancel lands inside the challenge janitor's ~1.6ms seed-job enqueue transaction, database/sql aborts with context canceled, Serve reports it as Failed to start service, and the test fails.
  • In production, a SIGTERM landing in the startup window makes the process exit non-zero, which K8s rolling updates read as a crash instead of a normal termination.

Fix

Align the startup loop with the run-phase semantics: if the serve context has been cancelled, log the aborted actor at info level and return nil — the deferred cleanup stops already-started actors as usual.

Two deliberate choices:

  • Detect via ctx.Err(), not the returned error. The cancellation may be masked by intermediate layers (database drivers can surface it as e.g. driver: bad connection), while "the serve context is cancelled" is unambiguous: once shutdown is requested, whatever Start returned is moot.
  • context.DeadlineExceeded still surfaces as an error, matching the run phase (TestServeConvenienceFunction relies on this distinction).

Testing

  • New TestServeCancelDuringActorStartup reproduces the race deterministically (fails on main, passes with the fix) and pins that a real startup failure is still reported.
  • Full suite + go vet + golangci-lint run (v2.11.3, same as CI): clean.
  • Verified end-to-end against ciam-next via a local replace: TestHealthyEndpoints passes.

🤖 Generated with Claude Code

…hutdown

A shutdown request (context cancellation) that arrives while an actor is
still starting was reported as a startup failure: the startup loop
returned the actor's error unconditionally, so Serve exited non-nil even
though the run phase already treats context.Canceled as a clean exit.

In practice this races e.g. a SIGTERM (or a test's cancel()) against a
service whose Start performs I/O — seen as a flaky TestHealthyEndpoints
in ciam-next where the challenge janitor's seed-job enqueue was aborted
mid-transaction by the test's shutdown.

Align the startup loop with the run-phase semantics: if the serve
context is cancelled, log the aborted actor at info level and return nil
so the deferred cleanup stops already-started actors. Detection uses
ctx.Err() rather than the returned error because drivers may mask the
cancellation; context.DeadlineExceeded still surfaces as an error, as in
the run phase.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@molon
molon requested a review from Copilot July 8, 2026 06:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dorothyyzh
dorothyyzh merged commit 1080aa5 into main Jul 8, 2026
2 checks passed
@shenzhaoyan
shenzhaoyan deleted the fix/lifecycle-cancel-during-startup branch September 3, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants