infra(traefik): move Traefik from k3s addon to ArgoCD (follow-up to 2026-09-26 outage) #106

Context

2026-09-26, 03:04-06:21 UTC (~3h15m): api.nan.builders and api.helmcode.com answered 404 on every inference route. LiteLLM was healthy. Root cause: nan-eu001 took the k3s controller lease and re-applied its stale on-disk /var/lib/rancher/k3s/server/manifests/traefik-config.yaml (no stripstream plugin); Traefik dropped every LiteLLM router (plugin: unknown plugin type: stripstream).

Short-term fix (nan-devops PR "fix(traefik): sync traefik-config.yaml to every k3s server from git"): a DaemonSet keeps that file identical to git on the three servers, the Traefik image is pinned, and new alerts cover routers going missing and config drift.

That still leaves Traefik managed by the k3s addon/helm-controller machinery: config lives on node disks, whichever server holds the lease re-renders the HelmChart from its own k3s binary, and deleting the file garbage-collects the live config. This issue removes that machinery.

Goal

Traefik is an ordinary ArgoCD-managed Helm release (pinned chart + image, values in git), with k3s no longer deploying or reconciling it.

Plan (maintenance window, both tenants affected)

1. Pre-work (no prod impact)
   • Align k3s: upgrade nan-eu001 v1.34.6 -> v1.35.4 (same as eu006/eu007), one server at a time, etcd health checked between steps.
   • Build the ArgoCD app kube-system/traefik (upstream traefik/traefik chart, pinned version matching the running one; values = the current HelmChartConfig values + what the k3s HelmChart injects: priorityClassName, tolerations, publishedService, image). helm template diff against the live Deployment/Service/RBAC/IngressClass must be empty or explained.
   • Decide CRD ownership (traefik-crd chart today): Argo app with Replace=true/SSA, or keep CRDs static.
   • Rehearse the whole cutover on a throwaway k3s (3 servers) incl. rollback.
2. Cutover (window)
   • Take over the existing Helm release in place (same release name traefik, namespace kube-system) so Deployment/Service are adopted, not recreated: no ClusterIP change, cloudflared keeps working.
   • Add --disable=traefik to the k3s config on all 3 servers and restart k3s one at a time. Before that, confirm on the rehearsal cluster what k3s does to the existing release/HelmChart objects when disabled (it deletes the HelmChart; the helm-delete job must NOT uninstall the release Argo now owns).
   • Remove traefik.yaml / traefik-config.yaml from the manifests dirs and retire the traefik-config-sync DaemonSet and its alert.
3. Verification
   • Gatus all green; traefik-router-missing-* silent; SSE stream test (no coalescing, no 60s cut) on both hosts; access logs still in VictoriaLogs; stripstream headers still stripped.
   • Force a lease change (restart k3s on the current leader) and confirm Traefik does not roll.
4. Rollback: re-enable traefik in k3s config, restore traefik-config.yaml from git to the manifests dirs, restart k3s on one server.

Acceptance

• k3s servers on the same version
• Traefik deployed only by ArgoCD; no Traefik files in any server/manifests
• Lease change drill does not touch Traefik (2026-10-02: the leader restart re-took the lease, so the lease never moved; the manifests re-apply did not touch Traefik)
• Runbook + postmortem updated (platform/devops/docs/postmortems/, docs/INCIDENTS.md)

Target: week of 2026-09-28.