Member pods can reach inference backends directly, bypassing LiteLLM #89

Activity

1. sre-helmcode

   assigned

   barckcode

   on Sep 16, 2026
2. sre-helmcode

   closed this as completedon Sep 17, 2026
3. sre-helmcode commented on Sep 17, 2026

   

   sre-helmcode

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Remediation complete — summary of what shipped.

   Confirmed exploitable before fixing, not just reported: from a pod carrying the member policy, GET http://10.10.0.9:8090/v1/models returned 200 with the model list.

   829 namespaces	swept to the RFC1918 except list — 632 vulnerable → 0
   nan-cloud-api#212	the Go generator, merged and deployed — verified the running binary contains 10.0.0.0/8 and no longer 10.244.0.0/16
   nan-devops#544	the second copy of the same policy (k8s/templates/networkpolicy.yaml, applied by create-member.sh) — found by the QA reviewer; unfixed it would have undone the sweep
   link-local	169.254.0.0/16 added to both generators: measured, the policy was allowing it and only the absence of a listener stopped it

   The fix ports the shape internal/spaces/resources.go has had since Spaces shipped: whole-RFC1918 except list, plus one rule opening the VLAN for TCP 30000-32767 only (the marketplace tenant-DB NodePorts, which resolve to VLAN addresses since the host firewall dropped NodePorts on the public NIC, nan-devops#384). The backends listen below that range.

   Verified after the fix:

   whisper 10.10.0.9:8090    http=000 in 0.0005s   (policy rejection, not timeout)
   kokoro  10.10.0.5:8880    http=000 in 0.0002s
   metadata 169.254.169.254  http=000 in 0.0002s   (was a 7s timeout: allowed, just unanswered)
   internet 1.1.1.1          http=301 in 0.05s     (unaffected)
   

   The sweep alone was not enough, and the QA review was right to insist. Production ran pre-fix code throughout, so two namespaces were minted vulnerable while remediating — member-juanoliver during the sweep and member-vicent-sd a minute before the audit that caught it. Both are closed; the deploy is what stopped the leak.

   Reviewers cleared the adjacent escape routes: the node's public IP does not reach the backends (they bind to the VLAN only), a member cannot create Endpoints to DNAT around the carve-out (nan-member grants services, not endpoints), hostNetwork is blocked by baseline PodSecurity, and the cluster is IPv4 single-stack.

   Two follow-ups worth their own issue

   1. The carve-out is by port range, not by service. Every NodePort is reachable from every member pod — today six tenant databases (credential-protected) and ssh-gateway. A future NodePort on something sensitive would re-open a path silently. Noted in the code and KUBERNETES.md.
   2. The backends answer an unauthenticated GET at all — speaches, Kokoro-FastAPI and vLLM ship no auth, which is why the network boundary is the only control.

   Also out of scope here: the host firewall accepts everything from the pod CIDR toward the node itself, so wildcard-bound services (kubelet :10250, apiserver :6443, sshd) remain reachable from a member pod via the node's public IP. All authenticated, and the old policy allowed the same — but it is the same class.