fix: bound client writes so a wedged peer cannot stall the server; 0.11.1:17 → 0.11.1:18

[stupleb]

One Electrum client that stops draining its socket freezes the whole server —
no wallet served, no block indexed — until the kernel gives up on it. Observed
in the field from 19 minutes to 8h39m, on two unrelated servers.

The defect

Peer::send writes responses with a blocking write_all, and no socket in the
electrs tree sets a timeout — set_write_timeout/SO_SNDTIMEO appears nowhere,
in v0.11.1 or in master. That write happens in handle_peer_events, reached
from handle_events, which is a plain sequential for loop running inline on
the single serve() loop — the same loop that calls rpc.sync().

So a client whose receive window closes and stays closed — gone without a FIN,
suspended, or behind a stalled proxy — holds that loop for as long as the kernel
keeps retransmitting. Indexing stops with it, and the process is too wedged to
answer SIGTERM, so a stop falls through to SIGKILL after the grace period.

It self-heals only when the kernel finally errors the socket, at which point the
existing path logs disconnecting due to failed to send response and drops that
peer. That signature is the fingerprint: every long stall ends in a burst of
those, immediately followed by a catch-up batch of every block missed.

Field evidence

Two unrelated servers, different architectures, one on :16 and one on :17:

Server	Stall	Recovery
x86_64	8h 39m	17 peers dropped, then indexing 58 blocks
x86_64	3h 32m	12 peers dropped, then indexing 27 blocks
x86_64	1h 55m	5 peers dropped, then indexing 8 blocks
x86_64	1h 43m	3 peers dropped, then indexing 15 blocks
aarch64	19m 22s	ended by a manual restart; SIGTERM → SIGKILL at 60s

During the 3h32m stall bitcoind accepted 28 blocks and electrs indexed none —
not one log line for the whole window. Short dips (under ~40 min) carry no
disconnect burst; those are the ordinary indexing blips :17's retry addressed.

Both users read the frozen server as a stuck resync. One reinstalled to fix it,
destroying a 62 GB index and triggering a 23-hour rebuild that was never needed.
The other restarted by hand and reported that Electrs "doesn't come back on its
own" after a power cut — it had come back, reached Fully synced in 52 seconds,
and wedged a minute later.

The fix

Set a 60s SO_SNDTIMEO on each accepted socket in accept_loop, and let the
error path that already exists drop just that peer.

The timeout bounds one write syscall, not a whole response, so a client that
keeps draining resets it on every partial write — a slow link cannot trip it
however large the response. What trips it is a receive window that stays shut
for the full minute. try_clone dups the fd and the option lives on the socket,
so setting it at accept also covers the clone handed to Peer.

Carried as a build-time patch rather than a submodule bump because there is no
upstream fix to bump to: v0.11.1 is the newest tag, and master sets no socket
timeout either. patches/README.md records the retirement condition. The
Dockerfile applies patches/*.patch with --fuzz=0, so a submodule bump that
moves the context fails the build rather than applying somewhere subtly wrong.

Blast radius

• Only Electrum client sockets. accept_loop serves electrum_rpc_addr and
  nothing else. bitcoind p2p is an outbound TcpStream::connect in p2p.rs;
  bitcoind RPC and the Prometheus endpoint are separate. None are touched.
• SO_SNDTIMEO affects sends only, so recv_loop's reads are unchanged.
• No index impact. No RocksDB or schema change, migrations: {}, upstream
  still 0.11.1 — nobody reindexes.
• Partial-write-then-disconnect is not a new failure mode. It is the existing
  path, which fired 42 times in one user's log; this reaches it sooner. A
  truncated line carries no newline, so clients discard rather than misparse it.
• On a server with no stalling client the timeout never fires and behaviour
  is identical to :17.

The one new outcome: a client that is alive but has stopped reading for a full
minute now gets disconnected where before it did not. It reconnects — Electrum
clients handle this routinely. The trade is that client losing its own
connection instead of freezing the server for everyone.

Verification

• docker build --target builder — exit 0, Finished release profile in 16m 43s
• compiled source in the image confirmed to carry the change
• tsc --noEmit exit 0; prettier exit 0, no formatting drift
• submodule left pristine at 35216c6; the patch is a build-time delta only

Not verified: that this resolves the stall on a live server — a wedged client
can't be manufactured locally. Worth confirming on one of the affected boxes
before release. The 60s value is a judgement call, not a measured one: ~19×
shorter than the shortest observed stall, and it takes a full minute of zero
bytes accepted to trip.

Follow-up

Every electrs user has this, not just StartOS. Worth an upstream issue at
romanz/electrs with the same evidence.

🤖 Generated with Claude Code

[helix-nine]

Pushed two maintainer commits on top:

• 132e05d folds in a cleanup that was parked for the next release: a migration clearing the task replay keys abandoned when bitcoind renamed its config action (bitcoind:config, bitcoind:other-config). They're harmless today only because they demand the same prune: 0 the live bitcoind:autoconfig key does; a stale key of exactly this kind fought datum-gateway's live one and stopped the service. clearTask is a no-op where a key is absent, so it ships unconditionally. One correction to the description this implies: migrations is no longer {} — the "no index impact, nobody reindexes" conclusion is unchanged.
• 7735f15 — doc corrections: UPDATING.md now carries the patch-revalidation step in "Applying the bump"; patches/README.md no longer claims a live client cannot trip the write timeout (one whose application stops reading long enough to fill its receive window can, and is disconnected by design — matching the patch's own comment and the description here); and "context has moved" → "context has changed" in the --fuzz=0 prose, since relocated-but-intact context still applies at an offset — changed context is what fails.

Independently verified while reviewing: the patch applies clean with patch -p1 --fuzz=0 against pristine 35216c6, upstream v0.11.1 and master both confirmed timeout-free, blast-radius claims check out, and 0.11.1:18 is free on the reference registry. The earlier red CI run was a runner-acquisition infrastructure failure (zero steps executed) — superseded by this push.