security: builds can forge Cf-Connecting-Ip via direct Traefik access, weakening key-sharing detection #114

Summary

A member build pod can reach Traefik directly: builds-system must reach Traefik pods on 8000/8443 for the registry push. From there it can send a request for api.nan.builders or api.helmcode.com with a forged Cf-Connecting-Ip. The LiteLLM usage hooks copy that header straight into the client_ip tag (("cf-connecting-ip", "client_ip", None) in both the community and enterprise *-usage-hook ConfigMaps). API-KEY-SHARING-DETECTION.md treats client_ip as the real client IP, so a key-sharer can make every request look like it comes from one residential IP, or from rotating ones, and weaken key-sharing detection. The request still needs a valid API key, so this is a detection bypass, not an auth bypass.

Tenants: both (shared Traefik). Found in the builds-system security review on 2026-09-29. Related: helmcode/nan-devops#615 (builds netpol), #616 (edge strip of LiteLLM control fields).

Why it wasn't fixed in the review PR

The fix changes the Traefik entrypoint config shared by both tenants. The last Traefik config change (2026-09-26) was reverted by eu001's on-disk HelmChartConfig and 404'd both tenants. So this ships as its own reviewed change with a rollout plan, not as a side effect of the builds PR.

Facts (verified 2026-09-29)

• Every cloudflared request enters Traefik on web (:8000). The HelmChartConfig comment says so, and the tunnel targets the web Service port. Nothing from Cloudflare arrives on websecure.
• websecure (:8443) carries only in-cluster clients: Kaniko pushes to registry.nan.builders via the CoreDNS rewrite to traefik.kube-system.
• websecure terminates TLS for every router. Ingresses without tls: still attach to all entrypoints. So a build pod can do curl -k --resolve api.nan.builders:443:<traefik> -H 'Cf-Connecting-Ip: 1.2.3.4' … and reach LiteLLM through the normal router.
• Member-/space- pods can't reach Traefik in-cluster (RFC1918 is excluded). builds-system is the only tenant-controlled namespace that can.
• forwardedHeaders.trustedIPs doesn't help: it governs X-Forwarded-* only, never Cf-*.

Proposed fix (low blast radius)

1. Strip Cf- on websecure at the entrypoint level.* Nothing legitimate on that entrypoint carries them.

   # HelmChartConfig valuesContent (+ traefik-config-sync copy + eu001 disk)
   ports:
     websecure:
       middlewares:
         - kube-system-strip-client-ip-headers@kubernetescrd
   ---
   apiVersion: traefik.io/v1alpha1
   kind: Middleware
   metadata: {name: strip-client-ip-headers, namespace: kube-system}
   spec:
     plugin:
       stripstream:            # NOT headers.customRequestHeaders: the built-in
         requestHeaders:       # middleware buffers response bodies (multi-GB
           Cf-Connecting-Ip: ""   # registry blob downloads)
           Cf-Connecting-Ipv6: ""
           True-Client-Ip: ""
           Cf-Ipcountry: ""
           Cf-Ray: ""
           X-Real-Ip: ""

   Test: tests/traefik asserts that websecure has the middleware, web does not, and the Middleware uses the stripstream plugin, not headers.
2. Drop port 8000 from the builds-system Traefik egress rule. Kaniko pushes over HTTPS (443 → 8443), so builds never need web. Without this, a build can reach web directly, where Cf-* is trusted, and step 1 is moot. Update tests/builds-system::test_only_in_cluster_peers_are_dns_and_traefik to {8443}. Verify with one staging or test build that the push still completes before merging.
3. (Defence in depth, optional) Have the usage hook set client_ip only when the request also carries Cf-Ray, and tag a mismatch. This doesn't replace 1+2, since Cf-Ray is forgeable too.

Rollout

Follow the Traefik runbook from the 2026-09-26 postmortem. Apply the HelmChartConfig change in git, in traefik-config-sync, and on eu001's disk. Watch the rollout and confirm with kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik and a curl on both API hosts. Then verify:

• From a builds-system-like pod: curl -sk --resolve api.nan.builders:443:<traefik-pod-ip> -H 'Cf-Connecting-Ip: 203.0.113.9' https://api.nan.builders/v1/models, then check the Traefik access log. The request must show no Cf-Connecting-Ip, and LiteLLM SpendLogs must show no client_ip:203.0.113.9 tag. Use a test key, never a member's.
• A real request through Cloudflare still carries the correct client_ip.