Skip to content

security: member-*/space-* pods run ~40s without NetworkPolicy enforcement at start (kube-router sync window) #123

Open

Description

@sre-helmcode

Summary

kube-router enforces NetworkPolicies for a NEW pod only after its sync, roughly 35-40 s after the pod starts. Until then, the host accepts all of 10.244.0.0/16 in INPUT.

I measured it on 2026-09-29 on nan-eu009 with a fresh pod probing every 10 s:

t after start apiserver ClusterIP 10.245.0.1:443 node public IP :6443 node public IP :22
0-30 s open open open
40 s onwards blocked blocked blocked

Builds are protected: cloud-api 0.8.187 (#262) added the wait-network-isolation initContainer, so member code starts only after isolation. That took 31 s on the first real build.

Member app pods (member-*) and Space pods (space-*) have no such gate. Member-controlled code in those pods can reach the node's host ports, the apiserver and other in-cluster services for ~40 s after every (re)start. A crash-looping pod gets that window on every restart.

Options

  1. A shared isolation-gate initContainer, like the builds one, injected into every workload cloud-api deploys into member-*/space-*. It covers the Helm SDK app charts and the Spaces resources. The cost is that each start is delayed by the isolation time.
  2. A mutating admission webhook or Kyverno policy that injects the gate into every pod in member-*/space-*. This also covers workloads not created by cloud-api.
  3. Fix it at the CNI level: make kube-router pre-program per-namespace default-deny before the pod's own chain exists, or evaluate Cilium or another CNI with atomic policy on pod creation. This is the most robust option but a large change to the shared cluster.

Notes

  • The window matters most for crash-looping pods. See the four hermes-agent pods in member-* with 11k-38k restarts.
  • The builds gate measures and logs the elapsed time (network isolation enforced after Ns), which gives a distribution for sizing a deadline.
  • References: memory feedback_kube_router_new_pod_unisolated_40s, helmcode/nan-devops#615, helmcode/nan-cloud-api#262.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions