docs(operations): rewrite tenant Kubernetes OIDC page for cozy-realm flow

Companion to the cozystack PR pivot away from per-tenant Keycloak realms.
The implementation now authenticates tenant kube-apiservers against the
platform cozy realm and enforces cross-cluster isolation via a per-cluster
audience claim rather than realm separation. Per-tenant realms are
deferred to a follow-up that also ships tenant-delegated identity
management (the missing piece that would justify the realm-per-tenant
operational surface area).

Page changes:

* Drop the per-tenant realm Overview framing; describe the cozy-realm
  flow with audience-claim isolation as the cross-cluster guard.
* Update object names everywhere to the new globally-unique shape:
  KeycloakClient / KeycloakRealmGroup are now
  `<namespace>-<release-name>` (e.g. `tenant-acme-kubernetes-prod-a`),
  paralleling the existing `tenant-<X>-<level>` group pattern for the
  management cluster.
* Update issuer URL examples to `https://keycloak.<root-host>/realms/cozy`.
* Drop the "orphan realm cleanup workaround" Limitations section — no
  per-tenant realm means no orphan realm.
* Drop the up-to-10-minute cascade timing + awaiting-oidc-realm beacon
  description — the cozy realm is always provisioned by install-cozystack
  before any tenant Kubernetes CR can request OIDC, so there is no race.
* Keep: self-signed Keycloak unsupported, hard-coded JWT claims, runtime
  toggle does not clean up CRB, CI/headless KeycloakClient patch, the
  full troubleshooting section.
* Add an explicit "Future work" callout flagging per-tenant realms +
  tenant-delegated identity management as planned follow-up so the
  reader understands today's design tradeoff.
* Update troubleshooting cleanup recipe selectors to the new label
  value `cozystack.io/oidc-cluster=tenant-acme-kubernetes-prod-a`.

Pairs with cozystack PR #3044.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

Author: IvanHunters

content/en/docs/next/operations/oidc/tenant_clusters.md

@@ -1,7 +1,7 @@
 ---
 title: "OIDC for tenant Kubernetes clusters"
 linkTitle: "Tenant Kubernetes OIDC"
-description: "Per-tenant Keycloak realms and OIDC authentication on tenant kube-apiservers"
+description: "OIDC authentication on tenant kube-apiservers via the platform cozy Keycloak realm"
 weight: 40
 aliases:
   - /docs/next/oidc/tenant-kubernetes
@@ -11,7 +11,8 @@ This page covers OIDC authentication on the tenant Kubernetes clusters
 managed by Cozystack (the per-tenant kube-apiservers backed by Kamaji).
 It complements [Enable OIDC Server]({{< relref "enable_oidc.md" >}}),
 which covers OIDC for the **management** cluster (dashboard, kubeapps,
-mgmt kubectl) using the platform `cozy` realm.
+mgmt kubectl). Both flows authenticate against the same platform `cozy`
+Keycloak realm.
 
 In all examples below the tenant is `acme` (so its namespace is
 `tenant-acme`), the tenant Kubernetes CR is `prod-a` (so its release
@@ -20,36 +21,42 @@ name is `kubernetes-prod-a`), and the platform root host is
 
 ## Overview
 
-Each Cozystack tenant gets its own dedicated Keycloak realm for its
-kube-apiservers and tenant-scoped applications. The management
-identity domain (the `cozy` realm) stays separated from per-tenant
-identity. Users granted access to a tenant cluster live in that
-tenant's realm only — they cannot accidentally log into the management
-cluster or another tenant.
-
-When the `Kubernetes` CR opts into OIDC (`spec.oidc.enabled: true`):
-
-1. The `apps/tenant` chart auto-provisions a `ClusterKeycloakRealm`
-   named after the tenant (`tenant-acme`) and the standard `groups`
-   `KeycloakClientScope` inside it. The realm name is published as
-   `_namespace.oidc-realm` in the tenant's `cozystack-values` Secret
-   so descendants and apps pick it up.
-2. The `apps/kubernetes` chart creates a per-cluster public
-   `KeycloakClient kubernetes-prod-a` (with its own audience scope so
-   cross-cluster token replay fails inside the same realm), the
-   `KeycloakRealmGroup prod-a`, and wires the tenant kube-apiserver
-   via `KamajiControlPlane.spec.apiServer.extraArgs`.
-3. A post-install Job (`kubernetes-prod-a-oidc-rbac`) applies a
-   `ClusterRoleBinding` inside the tenant cluster, binding the realm
-   group `prod-a` to the built-in `cluster-admin` ClusterRole.
-   Operators grant or revoke access by adding or removing users from
-   the Keycloak group.
-
-Operators do **not** need to pre-toggle `Tenant.spec.oidc.enabled`
-during normal operation — the parent `apps/tenant` chart auto-detects
-child Kubernetes CRs with `oidc.enabled: true` and provisions the
-realm automatically. The explicit `Tenant.spec.oidc.enabled=true` is
-only useful as a manual cleanup workaround (see Limitations below).
+Tenant kube-apiservers authenticate users against the platform `cozy`
+Keycloak realm. Cross-cluster isolation is enforced via a per-cluster
+**audience claim** — a token minted for cluster A's apiserver fails
+audience validation when replayed against cluster B's apiserver, even
+though both consume the same realm.
+
+When the `Kubernetes` CR opts into OIDC (`spec.oidc.enabled: true`),
+the `apps/kubernetes` chart provisions the following objects:
+
+1. A per-cluster public `KeycloakClient`
+   `tenant-acme-kubernetes-prod-a` in the cozy realm, with its own
+   `KeycloakClientScope` audience mapper
+   `tenant-acme-kubernetes-prod-a-audience` so cross-cluster token
+   replay fails.
+2. A `KeycloakRealmGroup tenant-acme-kubernetes-prod-a` in the cozy
+   realm. The name parallels the existing `tenant-<X>-<level>` group
+   pattern used for the management cluster's per-tenant access tiers.
+3. OIDC arguments on the tenant kube-apiserver via
+   `KamajiControlPlane.spec.apiServer.extraArgs` (`--oidc-issuer-url=
+   https://keycloak.acme.example.com/realms/cozy`, `--oidc-client-id=
+   tenant-acme-kubernetes-prod-a`, `--oidc-username-claim=
+   preferred_username`, `--oidc-groups-claim=groups`).
+4. A post-install/post-upgrade Job
+   (`kubernetes-prod-a-oidc-rbac`) that applies a `ClusterRoleBinding`
+   inside the tenant cluster, binding the realm group to the built-in
+   `cluster-admin` ClusterRole.
+
+Operators grant or revoke access by adding or removing users from the
+`tenant-acme-kubernetes-prod-a` group in the cozy realm.
+
+**Future work**: dedicated per-tenant Keycloak realms (with
+tenant-delegated identity management — the tenant owner manages their
+own users without involving the platform admin) are a planned follow-up.
+This iteration keeps identity ownership with the platform operator to
+avoid the bootstrap-credentials and realm-lifecycle complexity that
+realm-per-tenant introduces.
 
 ## Prerequisites
 
@@ -84,30 +91,17 @@ spec:
     enabled: true
 ```
 
-Each chart in the chain reconciles on its own loop (default interval
-5 minutes), so the full cascade takes up to ~10 minutes worst case
-from a cold start:
-
-1. The `apps/tenant` reconcile creates `ClusterKeycloakRealm tenant-acme`
-   and publishes `_namespace.oidc-realm=tenant-acme` to the tenant's
-   `cozystack-values` Secret.
-2. The `apps/kubernetes` reconcile picks up the realm, provisions the
-   per-cluster `KeycloakClient kubernetes-prod-a`, the realm group
-   `prod-a`, and adds OIDC flags to the kube-apiserver.
-3. The post-install Job binds `Group prod-a` to `cluster-admin` inside
-   the tenant cluster.
-
-Until step 1 completes, `apps/kubernetes` renders a
-`kubernetes-prod-a-awaiting-oidc-realm` ConfigMap beacon in the tenant
-namespace and the kube-apiserver runs without OIDC arguments — the
-client-cert (mTLS) admin kubeconfig stays usable throughout.
+On the next `apps/kubernetes` reconcile (≤ 5 min) the kube-apiserver
+gets OIDC flags, the per-cluster `KeycloakClient` + audience scope +
+`KeycloakRealmGroup` appear in the cozy realm, and the post-install
+Job creates the in-cluster `ClusterRoleBinding`.
 
 ## Create a user and grant access
 
-In Keycloak (the tenant realm — `tenant-acme`):
+In Keycloak, on the platform `cozy` realm:
 
 1. Create a user, set a non-temporary password, mark email verified.
-2. Add the user to the realm group named after the cluster (`prod-a`).
+2. Add the user to the realm group `tenant-acme-kubernetes-prod-a`.
    One membership = full `cluster-admin` access to that cluster.
 
 To revoke access, remove the user from the group.
@@ -161,8 +155,8 @@ users:
       args:
       - oidc-login
       - get-token
-      - --oidc-issuer-url=https://keycloak.acme.example.com/realms/tenant-acme
-      - --oidc-client-id=kubernetes-prod-a
+      - --oidc-issuer-url=https://keycloak.acme.example.com/realms/cozy
+      - --oidc-client-id=tenant-acme-kubernetes-prod-a
 ```
 
 Running `kubectl get pods` opens the browser, logs the user into
@@ -171,29 +165,6 @@ on the `groups` claim.
 
 ## Limitations
 
-### Realm cleanup is not automatic after the last child OIDC cluster is removed
-
-The `apps/tenant` chart uses Helm's `lookup` function to discover
-whether any child `Kubernetes` CR has `spec.oidc.enabled=true`.
-Helm-controller does **not** re-render a chart when a `lookup` result
-changes — it only re-renders when the chart source artifact or the
-HelmRelease values change. Consequently, deleting the last
-`Kubernetes` CR with OIDC enabled does **not** trigger an
-`apps/tenant` re-render, and the orphan `ClusterKeycloakRealm` stays
-in the tenant namespace.
-
-To force cleanup, the operator can:
-
-- Toggle `Tenant.spec.oidc.enabled=true` and then back to `false`.
-  Each toggle changes the HelmRelease values, which triggers a
-  re-render with the up-to-date lookup result. After the second
-  toggle, the chart no longer renders the realm and Helm prunes it.
-  This is the only legitimate use of `Tenant.spec.oidc.enabled` —
-  during normal operation the field stays at its default `false`.
-- Or wait for the next platform upgrade that bumps any
-  chart-affecting source — the realm cleanup happens for free as a
-  side effect of the new render.
-
 ### Self-signed Keycloak is not supported
 
 The tenant apiserver validates the OIDC issuer over HTTPS using the
@@ -228,7 +199,7 @@ match it once OIDC is off. Manual cleanup:
 
 ```bash
 KUBECONFIG=/tmp/prod-a-admin.kubeconfig kubectl delete clusterrolebinding \
-  --selector cozystack.io/oidc-cluster=prod-a
+  --selector cozystack.io/oidc-cluster=tenant-acme-kubernetes-prod-a
 ```
 
 (reuse the admin kubeconfig extracted in the "Wire kubectl" section).
@@ -241,7 +212,8 @@ browser-flow logins. For CI pipelines that need a non-interactive
 token, the cluster-admin can patch the client on the live cluster:
 
 ```bash
-kubectl --context=mgmt -n tenant-acme patch keycloakclient kubernetes-prod-a \
+kubectl --context=mgmt -n tenant-acme patch keycloakclient \
+  tenant-acme-kubernetes-prod-a \
   --type=merge --patch '{"spec":{"directAccess":true}}'
 ```
 
@@ -262,10 +234,11 @@ kubectl --context=mgmt -n tenant-acme get pod \
   tr ',' '\n' | grep oidc
 ```
 
-Confirm the issuer URL matches the realm — decode the id_token and
-compare `iss` against the `--oidc-issuer-url` flag. Confirm `aud` in
-the token equals `--oidc-client-id`; mismatch is the most common
-cause when running multiple clusters in the same realm.
+Confirm the issuer URL matches the cozy realm — decode the id_token
+and compare `iss` against the `--oidc-issuer-url` flag. Confirm `aud`
+in the token equals `--oidc-client-id`; mismatch is the most common
+cause when running multiple clusters in the same realm (the per-cluster
+audience scope is what enforces isolation here).
 
 ### Apiserver returns 403 for a user that is in the right group
 
@@ -279,18 +252,14 @@ kubectl --context=mgmt -n tenant-acme \
   > /tmp/prod-a-admin.kubeconfig
 
 KUBECONFIG=/tmp/prod-a-admin.kubeconfig kubectl get clusterrolebinding \
-  --selector cozystack.io/oidc-cluster=prod-a
+  --selector cozystack.io/oidc-cluster=tenant-acme-kubernetes-prod-a
 ```
 
 The bootstrap Job runs as a `post-install` / `post-upgrade` hook;
-check its logs in the management cluster:
+check its logs in the management cluster (the Job is garbage-collected
+on success — if you don't see it, that's the happy path):
 
 ```bash
 kubectl --context=mgmt -n tenant-acme logs \
   job/kubernetes-prod-a-oidc-rbac
 ```
-
-### Realm or scope objects stuck after CR deletion
-
-See "Realm cleanup is not automatic" under Limitations. Operator
-intervention required (toggle `Tenant.spec.oidc`).