networking: new space pods get ECONNREFUSED to marketplace databases for ~15-20s after start (kube-router source sync lag) #135

Summary

New pods in space namespaces get ECONNREFUSED when connecting to marketplace CNPG databases (and other in-namespace services) for roughly 15-20 s after pod start: the new pod's source IP is not yet in the destination node's kube-router KUBE-SRC ipsets, so the destination's default-deny ingress rejects it.

Measured on 2026-09-30 with a fresh pod TCP-probing a marketplace Postgres every ~1.5 s:

Reproduced from three different spaces (same-namespace client → CNPG -rw service and direct pod IP). Pods started earlier connect fine, so the failure only hits apps that deploy/start with short retry loops — a real user app failed to connect this way on 2026-09-29 and kept failing until it was restarted the next day.

Root cause (shared with #123)

Same underlying kube-router sync window from the other side: #123 covers the new pod being unprotected during the window; this issue covers the new pod being rejected as a source during it. Policy state for a new pod only propagates after kube-router's sync, currently 15-40 s across nodes. The scale (4,576 NetworkPolicies / 1,124 namespaces) and etcd raft latency (see the etcd quorum issue) amplify it.

Proposal

1. Cheap, robust fix in platform-api: add a wait-for-db initContainer to the marketplace app templates (loop nc -z $DB_HOST 5432 up to 90 s) so app containers only start after the policy window closes — the availability-side mirror of the wait-network-isolation gate from security: member-*/space-* pods run ~40s without NetworkPolicy enforcement at start (kube-router sync window) #123. Needs its own tests (init gate timeout, DB down behavior).
2. Re-measure after the etcd topology fix: target window < 2 s, tracked with the same timing probe.
3. If still > 5 s: evaluate the CNI-level options already listed in security: member-*/space-* pods run ~40s without NetworkPolicy enforcement at start (kube-router sync window) #123 (pre-programmed per-namespace default-deny or CNI migration).

Review before rolling out: the initContainer change affects every marketplace deployment.