Skip to content

apr serve: drain on SIGTERM/SIGINT — keep listening, 503 + Retry-After for new requests, finish in-flight; today SIGTERM cuts requests to EOF #4449

Open

Description

@noahgift

Found by paiml/infra#1150 (P0). The resident apr serve review lane on gx10 flapped about 20 times in 4h, and every stop cut in-flight requests to EOF.

Defect. apr serve run shuts down gracefully only on ctrl_c. shutdown_signal() in crates/apr-cli/src/commands/serve/server.rs awaits tokio::signal::ctrl_c() alone. As a result:

  • SIGTERM, which systemd sends by default on systemctl stop, kills the process mid-request, and clients read EOF.
  • The WGPU path (handlers.rs ~751) calls axum::serve with no graceful shutdown at all.
  • Even on SIGINT, axum's with_graceful_shutdown closes the listener, so a request that arrives during the drain is refused. It never gets an answer it can act on.

infra works around the first point with KillSignal=SIGINT (infra#1150).

Asked (cop ruling on infra#1150). On SIGTERM or SIGINT:

  1. Enter drain.
  2. Keep the listener open.
  3. Answer every NEW request 503 with Retry-After: <s>. /health answers 503 too.
  4. Let in-flight requests (including streams) finish.
  5. Exit once in-flight reaches 0, or at a --drain-timeout.

Apply the same behaviour on every axum::serve site: CUDA, WGPU and CPU.

Falsifier.

  • A request in flight at SIGTERM completes 200.
  • A request sent after SIGTERM gets 503 with Retry-After, never a reset or EOF.
  • Reverting the handler makes both rows red.

Activity

  1. added this to the 0.71.0 milestone on Oct 8, 2026
  2. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  3. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5329 (apr serve request path: BPE tokenization, graceful drain, no 500 on oversize prompts), split out of #5312 by theme.

  4. modified the milestones: 0.71.0, 0.73.0 on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityowner:la-0.71Owned by the 0.71 look-ahead slotsize:MK̂ 3-4 h

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions