[Rust] No backoff before reconnect after a mid-stream failure (hot-loops a flapping server)

[zlata-stefanovic-db]

Summary

After a mid-stream failure, the supervisor re-enters its loop and immediately calls RetryIf::spawn, whose first create attempt fires with zero delay (tokio_retry's FixedInterval inserts backoff only between retries within an episode, not before the first). Combined with the fresh-budget-per-episode behavior (#640), this means a flapping server is reconnected in a tight hot loop with no pacing.

Location

rust/sdk/src/stream/grpc/supervisor.rs:78-79,149

let strategy = FixedInterval::from_millis(options.recovery_backoff_ms)
    .take(options.recovery_retries as usize);
// ...
let creation = RetryIf::spawn(strategy, create_attempt, should_retry).await;

FixedInterval delays are applied between attempts, so the first attempt of each new recovery episode is immediate. There is no unconditional recovery_backoff_ms sleep before re-establishing a stream that just died.

Why it matters

A server that accepts and then immediately drops streams triggers reconnect attempts with no delay between episodes — CPU burn and a connection storm against the service. recovery_backoff_ms is effectively skipped for the first attempt of every episode.

Suggested fix

Sleep recovery_backoff_ms before re-establishing a stream after any failure (not only between within-episode retries). Optionally add jitter — neither core currently jitters, which is a separate consideration.

Reference: how the pure-Go core handles this

The pure-Go reimplementation sleeps RecoveryBackoff before every reconnect:

purego/internal/stream/supervisor.go:51-55

select {
case <-time.After(cs.cfg.RecoveryBackoff):
case <-ctx.Done():
    return
}

This runs at the top of the loop whenever failedAttempts > 0, i.e. before the next runOnce (which includes the Open call).