Skip to content

Streamable HTTP: failed resume returns 200 + empty stream, causing rmcp's own client to reconnect-loop at 1Hz forever with the same Last-Event-Id #958

Description

@jackzampolin

Summary

When a streamable-HTTP session's event replay fails server-side (e.g. the session's channel is closed but the session entry still exists), the rmcp server answers the client's resume GET with HTTP 200 + an immediately-closed empty SSE stream. rmcp's own Rust client treats that as a successful reconnect followed by a graceful stream close — which resets its retry counter — and reconnects with the same Last-Event-Id after retry(0) (1s by default). The result is a permanent 1 Hz request loop per wedged stream that never backs off, never terminates, and burns CPU + log volume on both ends indefinitely.

Both halves behave reasonably in isolation; together they form a closed loop.

Server side

src/transport/streamable_http_server/tower.rs (~line 975 in 1.8.0):

Err(e) => {
    // Return 200 with an immediately-closed empty stream.
    // Returning an HTTP error would cause EventSource to retry
    // with the same Last-Event-ID in an infinite loop. ...
    tracing::warn!("Resume failed ({e}), returning empty stream");

The comment's rationale is browser EventSource semantics — but the empty-200 produces exactly the feared infinite same-Last-Event-Id loop in rmcp's own Rust client, just disguised as a successful reconnect (and it emits one unthrottled WARN per iteration).

Client side

src/transport/common/client_side_sse.rs, SseAutoReconnectStream::poll_next, graceful-close arm:

None => {
    // Per SEP-1699, a graceful stream close is reconnectable. ...
    let interval = this.server_retry_interval.take()
        .or_else(|| this.retry_policy.retry(0));   // <-- retry_times hardcoded to 0
    ...
    SseAutoReconnectStreamState::WaitingNextRetry { sleep, retry_times: 0 }
  • retry_times is reset to 0 on every graceful close; the counter only increments when the reconnect itself errors. A 200-empty response therefore never counts as a failure: with the default ExponentialBackoff { max_times: None, base_duration: 1s }, the loop runs at retry(0) = 1s forever.
  • last_event_id is preserved across iterations, so every reconnect re-sends the same Last-Event-Id, guaranteeing the server's resume fails again.

Production impact (how we found this)

Fleet of ~17 long-running rmcp clients (defra-agent runtimes) against an rmcp streamable-HTTP server (rmcp 0.15.0, behavior unchanged in 1.8.0). A handful of long-lived idle connections' sessions became unresumable; each entered the loop. Over ~4.7 days: 14.8M Resume failed WARN lines (2.9 GB of server logs), sustained CPU on both ends, and the server's real signals drowned. Observed WARN rate ≈ 36/sec ≈ number of wedged streams × 1 Hz, matching the analysis exactly.

Suggested fixes (either breaks the loop; both seem right)

  1. Server: when resume fails but the session is still alive, fall back to create_standalone_stream for that session (fresh live stream, no replay) instead of an empty stream — the client gets keep-alives and stops looping. If the session is terminated/unknown, return 404 per the spec's session-management rules so clients re-initialize (same spirit as Streamable HTTP: Session not found returns 401 instead of 404 (MCP spec violation) #689). Note 404 is safe for this client: a failed reconnect does increment retry_times, and exponential backoff self-quiets.
  2. Client: treat a graceful close that delivered zero events since connect as a failed attempt (don't reset retry_times), and honor max_times on the graceful-close path. Optionally clear last_event_id after N consecutive failed resumes and re-establish a fresh stream.
  3. At minimum, rate-limit the server's Resume failed WARN.

Happy to contribute a PR for either side if maintainers agree on the direction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions