infra(etcd): quorum has a 24ms voter (nan-eu001) — raft agreement latency degrades the API server #134

Summary

The etcd quorum spans two Hetzner locations. Private-network RTT measured today (2026-09-30, VLAN 10.10.0.0/24):

etcd itself is healthy on disk — WAL fsync avg 0.26 ms, backend commit avg 0.34 ms (99.9% of observations < 8 ms), proposals_failed_total = 0. The latency is raft agreement, not disk:

• etcd_server_slow_apply_total = 894,217 (lifetime, nan-eu006)
• apiserver journal on 2026-09-29 shows constant apply request took too long (~893/h) with traces of agreement among raft nodes before linearized reading at 100-150 ms

Every linearizable read pays a read-index round-trip to the leader. With one voter 24 ms away, an unlucky leader placement makes the whole API surface pay 24-150 ms per read.

Impact (observed)

• 2026-09-29 incident: a marketplace Postgres instance went NotReady after losing API-server connectivity (instance-manager context deadline exceeded to 10.245.0.1); user apps got ECONNREFUSED, and the CNPG operator stayed wedged ~23 h — a user-issued rollout restart was only applied the next morning, after the operator upgrade.
• Watch-based components inherit the lag (kube-router syncing 4,576 NetworkPolicies across 1,124 namespaces). Related: security: member-*/space-* pods run ~40s without NetworkPolicy enforcement at start (kube-router sync window) #123 (kube-router sync window, security angle).

Contributing factor

Control-plane nodes run ~90 customer pods (35 / 35 / 20 on eu001 / eu006 / eu007, no taints) — Postgres/Qdrant workloads competing with etcd and the apiserver for CPU and I/O.

Proposal — requires careful review before applying anything

1. Immediate, no downtime: identify the current leader (etcd_server_is_leader in http://127.0.0.1:2381/metrics on each control plane) and move-leader to eu006/eu007 if it is eu001.
2. Structural: re-home the etcd membership of nan-eu001 to a node in the fast DC (join new member → remove eu001) so all 3 voters share sub-millisecond RTT.
3. Taint the 3 control-planes and move customer workloads to workers (PDBs, CNPG switchover before drains).
4. Hygiene: periodic defrag (db 149 MB / 62 MB in use), Grafana alerts on slow_apply rate and peer RTT, throttle platform-api reconcile churn.

⚠️ Nothing here should be applied without a reviewed runbook and a maintenance window. Etcd topology changes and workload migration on the shared prod cluster are high-blast-radius operations. This issue is the problem statement and proposal — not an approved change.