From c2124009280a61ba26e51d90d4b7dbab05544de8 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 4 Sep 2026 00:48:52 +0300 Subject: [PATCH 1/6] docs(unified-tls-pki): fix the canonical trust-anchor name and its owner Implementation surfaced errors in the accepted design that the text now corrects. The canonical projection cannot be named -ca-cert or -tenant-ca: Percona PSMDB claims the first with a key-bearing Secret, and the second collides across releases, because for an application foo-tenant CloudNativePG's own CA Secret is postgres-foo-tenant plus -ca, the same string as the projection of application foo. The name is now .tenant-ca, disjoint by character class rather than by survey: application names are DNS-1035 labels and release prefixes are dot-free, while Secret names are DNS-1123 subdomains where a dot is legal. The projection cannot be owner-referenced to the application CR. The apps.cozystack.io kinds are virtual, the HelmRelease is the only real owner, and deleting the application deletes it. Record what has merged: the edge tier, the per-tenant wildcard propagation, the wildcard-secret reconciler, and the TenantProjection API with its group, version, short name, namespace scope and single CACert projection type. State the controller's RBAC as the chart grants it, and the fact that the release is derived from the Flux release label, with NoRelease reported when it is absent. Drop the promised ValidatingAdmissionPolicy on sentinel writes, which was never built; RBAC denying tenants any verb on internal.cozystack.io is the control. Describe the content guard as the merged controller has it: each PEM block is decoded and the chain rebuilt only from blocks that parse as an x509 certificate. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 8 +- design-proposals/unified-tls-pki/README.md | 120 +++++++++++------- 2 files changed, 78 insertions(+), 50 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 742352d..6c1a6a2 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -3,7 +3,7 @@ - **Title:** `External database exposure via Gateway API TLS-passthrough (SNI) and end-to-end TLS` - **Author(s):** `@lexfrei` -- **Date:** `2026-06-24` +- **Date:** `2026-06-24`; revised `2026-09-02` - **Status:** Accepted ## Overview @@ -16,7 +16,7 @@ This is the design-proposal artifact required by `cozystack/cozystack#2816`, and ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract. The engines reach that point by one path: the extraction controller reads each engine's CA source and writes the canonical `.tenant-ca` — copying with nothing to strip from redis and kafka's own key-free CA object, and stripping the key from postgres and mongodb's key-bearing Secret — rather than any engine publishing that canonical name itself. Either way the external client verifies against the one canonical `.tenant-ca`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. - **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -107,7 +107,7 @@ flowchart TB P2["postgres db2
presents CNPG server cert"] P3["redis cache
presents operator server cert"] end - CA["<release>-ca-cert (ca.crt only)
projected via tenantsecrets"] + CA["<release>.tenant-ca (ca.crt only)
projected via tenantsecrets"] C1 -->|"raw TLS, SNI"| L1 C2 -->|"raw TLS, SNI"| L1 C3 -->|"raw TLS, SNI"| L2 @@ -136,7 +136,7 @@ One subtlety the "single certificate" framing hides: an external client doing `s WS5 adds nothing beyond two hooks that already exist: -1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. +1. **Trust-anchor delivery** — the `.tenant-ca` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. 2. **SAN coverage** — the chart already injects the external hostname into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated on TLS and external). Under this proposal the hook injects **both** name shapes — `..` and `.` — so the certificate is valid for the subdomain scheme shipping now and the flat scheme arriving with Cilium 1.20 (§6). Stated plainly, where this does **not** work: diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 5203902..f056105 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -3,26 +3,25 @@ - **Title:** `Unified TLS and PKI model for managed applications` - **Author(s):** `@lexfrei` -- **Date:** `2026-06-24` +- **Date:** `2026-06-24`; revised `2026-09-02` (the §5 delivery mechanism, revised after the API-owner review in `cozystack/cozystack#3299`) - **Status:** Accepted ## Overview -Certificate handling in Cozystack grew out of four uncoordinated mechanisms: per-application issuance inside each chart, per-host ACME on the default ingress path, an opt-in Gateway API path that mints a wildcard via DNS-01, and no supported way to bring an externally-issued wildcard. The epic `cozystack/cozystack#2811` set out to converge them, and the edge of that convergence has largely landed. What has not landed — and what this proposal exists to pin down before it does — is the part that lives *inside* the engines: who owns the PKI for each managed application, and how a tenant receives the trust anchor it needs to verify a TLS connection. +Certificate handling in Cozystack grew out of four uncoordinated mechanisms: per-application issuance inside each chart, per-host ACME on the default ingress path, an opt-in Gateway API path that mints a wildcard via DNS-01, and no supported way to bring an externally-issued wildcard. The epic `cozystack/cozystack#2811` set out to converge them, and the edge of that convergence has landed. What this proposal exists to pin down is the part that lives *inside* the engines: who owns the PKI for each managed application, and how a tenant receives the trust anchor it needs to verify a TLS connection. This proposal records a written target design for the whole model, so that the two architectural forks at its center are decided on paper rather than discovered one pull request at a time. The first fork is the issuance abstraction: rather than "cert-manager as the single issuer", the target is **a single operator-facing interface to choose the certificate source, plus a uniform contract for consuming `ca.crt`**. The second fork is mint-versus-consume: rather than forcing every application to stop minting and consume a central certificate, the target is an explicitly **two-tier** model — cert-manager (or a bring-your-own wildcard) at the edge, and operator-owned PKI inside the engines — where what is unified is the *consume contract*, not the certificate authority itself. The second fork is the answer to the open question the epic body still carries as a goal ("should applications stop minting certs entirely and only consume cert-manager output?"): no — two-tier, because forcing pure-consume breaks the engines that rotate their own CA. ## Scope and related proposals -This proposal is the umbrella design for the work tracked by epic `cozystack/cozystack#2811`. It does not re-specify the edge work that has already merged; it states the target the whole model converges on and focuses on the interior contract that is still open. +This proposal is the umbrella design for the work tracked by epic `cozystack/cozystack#2811`. It does not re-specify the edge work that has already merged; it states the target the whole model converges on and focuses on the interior contract, where the per-engine work still sits. -- **Edge, merged:** `cozystack/cozystack#2988` (ACME wildcard on the default ingress-nginx path), `cozystack/cozystack#2989` (the CA-only trust-anchor helper). -- **Edge, open:** `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`). +- **Edge, merged:** `cozystack/cozystack#2988` (ACME wildcard on the default ingress-nginx path), `cozystack/cozystack#2989` (the CA-only trust-anchor helper), and `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`, merged 2026-07-15). The whole edge tier has landed. - **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (end-to-end TLS for databases); `cozystack/cozystack#2977` (opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. - **Per-app TLS series (open):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). These are the pull requests that should land *after* this contract is accepted, not before; several currently propose handing tenants a key-bearing Secret, which this contract exists to correct (see Security). -- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `-ca-cert` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. +- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `.tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. -All repository paths below refer to the `cozystack/cozystack` repository; paths attributed to an open PR (for example the wildcard-secret reconciler in PR `cozystack/cozystack#2990`) are not yet on `main`. +All repository paths below refer to the `cozystack/cozystack` repository. Every path cited here is on `main` except those attributed to the still-open per-app TLS series (`cozystack/cozystack#2729`, `cozystack/cozystack#2692`, `cozystack/cozystack#2683`, `cozystack/cozystack#2682`, `cozystack/cozystack#2680`), whose chart templates are not. ## Context @@ -40,7 +39,7 @@ The second class **consumes** PKI that its operator owns end-to-end — but, cru - `kafka` (Strimzi) is the **clean** reference: it exposes `-cluster-ca-cert` and `-clients-ca-cert`, each a CA-certificate-only object with no private key (`packages/apps/kafka/templates/dashboard-resourcemap.yaml`). Strimzi can serve a public certificate on its external listener while keeping internal broker mTLS on its own CA (`generateCertificateAuthority: false` + per-listener `brokerCertChainAndKey`). - `postgres` (CloudNativePG) renders no cert-manager objects; the CNPG operator auto-generates a self-signed CA and signs the server certificate. The CA lives in the operator-created `-ca` Secret, which **carries `ca.key`** and is created **asynchronously**; it is **not** delivered to tenants. The only tenant-facing object is `-credentials` (`packages/apps/postgres/templates/init-script.yaml`), which is a chart-rendered Opaque Secret holding **only `user: password` pairs** — no `ca.crt`, no `tls.key` — surfaced through the dashboard resource map (`packages/apps/postgres/templates/dashboard-resourcemap.yaml`). So postgres is a **trust-anchor delivery gap** (the tenant gets passwords but never `ca.crt`), the same shape as nats/qdrant — *not* a private-key coupling. The comment at `packages/apps/postgres/templates/db.yaml:20-22` claiming `ca.crt` rides in `-credentials` contradicts what the chart actually renders and is a documentation bug. -- `mongodb` (Percona PSMDB) is operator-owned, but the PSMDB operator mints a cert-manager chain **at runtime** and publishes `-ca-cert` as a **key-bearing** cert-manager `isCA` Secret (`tls.crt` + `tls.key` + `ca.crt`). The chart itself renders no TLS objects on `main` today — the whole PKI is operator-created after deploy, which is exactly why it needs the same asynchronous-source treatment as CNPG. Same gap as the rest, plus a trap: the name `-ca-cert` is key-free for the redis fork and key-bearing for PSMDB — opposite shapes under one name. +- `mongodb` (Percona PSMDB) is operator-owned, but the PSMDB operator mints a cert-manager chain **at runtime** and publishes `-ca-cert` as a **key-bearing** cert-manager `isCA` Secret (`tls.crt` + `tls.key` + `ca.crt`). The chart itself renders no TLS objects on `main` today — the whole PKI is operator-created after deploy, which is exactly why it needs the same asynchronous-source treatment as CNPG. Same gap as the rest, plus a trap: the name `-ca-cert` is key-free for the patched redis operator and key-bearing for PSMDB — opposite shapes under one name. And one engine is the proof of the whole framing rather than a mere exclusion. `kubernetes` (Kamaji) is the **fourth independent PKI owner** — alongside cert-manager chart graphs, CNPG, and Strimzi — and the one that is fundamentally non-swappable: Kamaji owns the control-plane CA, the kubeconfig it hands out *pins* that CA, and every client authenticates against it by construction, so a public edge certificate is not just unnecessary there but meaningless. Kamaji is the strongest single argument for "unify the interface, not the CA": any design that homogenized the certificate authority would break it outright, whereas the consume contract below (a key-free trust anchor, delivered uniformly) is something even Kamaji already satisfies in spirit — the kubeconfig *is* its trust-anchor delivery. It needs no unification work and is excluded from the rollout, but it belongs in the enumeration as the boundary case that fixes the model's shape. @@ -96,7 +95,7 @@ flowchart TB end subgraph interior["Interior tier — per-engine operator-owned PKI"] OP["DB operator owns CA + server cert
(CNPG / Strimzi / PSMDB / cert-manager chart graph)"] - CACERT["<release>-ca-cert
(ca.crt only, no private key)"] + CACERT["<release>.tenant-ca
(ca.crt only, no private key)"] OP --> CACERT end subgraph tenant["Tenant"] @@ -119,60 +118,88 @@ The contract is a per-engine table. It is the artifact the per-app pull requests | Engine | PKI owner | CA-bearing Secret today | Key in that Secret? | Self-publishes key-free `ca.crt`? | `ca.crt` to tenant today? | Operator certificate capability | | --- | --- | --- | --- | --- | --- | --- | | kafka (Strimzi) | operator | `-cluster-ca-cert`, `-clients-ca-cert` | no | **yes** | yes — clean reference | `generateCertificateAuthority: false` BYO-CA + per-listener `brokerCertChainAndKey`; public cert on external listener, own-CA mTLS internally | -| redis (forked operator) | operator fork | `-ca-tls` (chart cert-manager) | yes | **yes** → operator emits `-ca-cert` | yes | forked `redis-operator` `spec.tls.caCertSecretName` outputs a key-free `ca.crt`-only Opaque Secret (v1.4.0+) | +| redis | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2729` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca-tls` is key-bearing | n/a | **not yet** — under `cozystack/cozystack#2729` the patched operator does publish a key-free `-ca-cert`, but nothing on the platform reads it: the chart's sentinel sources the cert-manager CA instead | no | stock freshworks-oss `redis-operator` v3.3.5 has no TLS support; `cozystack/cozystack#2729` carries it as a downstream patch over v3.3.5 (`packages/system/redis-operator/images/redis-operator/patches/tls.diff`, cut from freshworks-oss/redis-operator#85 commit 38b9e3732), which adds `spec.tls` including `caCertSecretName` | | postgres (CloudNativePG) | operator | `-ca` (operator-created, async) | **yes — `ca.key`** | no | no — tenant gets passwords-only `-credentials` (gap) | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | | mongodb (Percona PSMDB) | operator | `-ca-cert` (operator-created at runtime; the chart renders no TLS objects) | **yes — cert-manager `isCA`** | no | no (gap; name collides with redis, opposite shape) | `spec.tls.issuerConf` BYO issuer; auto-mints a cert-manager chain | | nats | chart + cert-manager | `-ca` | yes | no | no | — | | qdrant | chart + cert-manager | `-ca` | yes | no | no | — | -| mariadb (open PR) | chart + cert-manager | `-ca-tls` | yes | no | no | mariadb-operator `serverCertSecretRef` / `serverCASecretRef` / `serverCertIssuerRef` | -| opensearch (open PR) | chart + cert-manager | `-http-ca` | yes | no | no | opster `security.tls.http.secret.name`; transport operator-generated | -| rabbitmq (open PR) | chart + cert-manager | `-ca` | yes | no | no | cluster-operator `spec.tls.secretName` / `caSecretName` | +| mariadb | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2680` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca-tls` is key-bearing | n/a | no | no | mariadb-operator `serverCertSecretRef` / `serverCASecretRef` / `serverCertIssuerRef` | +| opensearch | operator — opster generates its own TLS | none chart-rendered — on `main` the chart sets `security.tls.transport.generate` and `security.tls.http.generate`, so opster mints the material and the chart renders no cert-manager objects; `cozystack/cozystack#2682` (open, in review) adds a chart cert-manager chain whose CA Secret `-http-ca` is key-bearing | n/a | no | no | opster `security.tls.http.secret.name`; transport operator-generated | +| rabbitmq | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2683` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca` is key-bearing | n/a | no | no | cluster-operator `spec.tls.secretName` / `caSecretName` | | clickhouse (Altinity) | — (BYO-cert) | supplied cert mount | n/a | no | no (not in TLS series yet) | mounts a supplied cert; no native issuance | | kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | no — out of model | control-plane CA, kubeconfig-pinned, non-swappable | -The takeaway: only **kafka** (native) and **redis** (forked operator) are at the target today — they self-publish a key-free `ca.crt`. **Every other engine has a key-bearing CA Secret and delivers no `ca.crt` to the tenant** — one uniform delivery gap. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. +The takeaway: **kafka** (native) is the only engine that publishes a key-free `ca.crt` under its own name today, and it is still not at the canonical `.tenant-ca` — it alone needs the controller to copy that output across under that name (§5). **Every other engine that ships TLS on `main` today — postgres, mongodb, nats, qdrant — has a key-bearing CA Secret and delivers no `ca.crt` to the tenant**, one uniform delivery gap, and the four whose TLS is still in review (redis, mariadb, rabbitmq, opensearch) all mint a key-bearing CA the same way. **redis** is the one that already closes the gap inside its own pull request: `cozystack/cozystack#2729` declares a `TenantProjection` sentinel naming that key-bearing chart CA, so the anchor lands directly at `.tenant-ca` with the key stripped, and the key-free `-ca-cert` its patched operator also publishes goes unread. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. ### 4. The uniform consume contract -Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `-ca-cert`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenantresource: "true"`. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. +Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `.tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenant-ca: "true"` and made tenant-visible through the lineage webhook's `tenantresource` verdict. + +The name took three revisions, and the reason is worth recording because it is the same mistake twice. `-ca-cert` was rejected because Percona PSMDB already creates a Secret of that exact name and puts a private key in it — a collision across **engines**. Its replacement `-tenant-ca` was justified as "claimed by no operator this platform ships", and that was false: it collides across **releases**. For an application `foo` the projection is `postgres-foo-tenant-ca`; for a sibling application `foo-tenant`, CloudNativePG's own CA is `postgres-foo-tenant` + `-ca` — the same string. Both names are legal and several Postgres instances in one namespace is the normal model. Nor is it merely a guard problem: if the projection lands first, CNPG rejects the key-free Secret (`missing ca.key secret data`) and the sibling's PKI never completes, so the sibling application never comes up — silently, blaming a Secret its owner never created. That is a regression, since before this contract existed the name was unclaimed and both applications worked, and no guard can refuse it, because the controller that wrote first was legitimate. Only the first of the three was ever published — `-ca-cert` is the name the version of this proposal on `main` still carries — while `-tenant-ca` lived entirely inside this document's own revision history, so no release ever shipped it. + +The canonical name is therefore `.tenant-ca`, and it is collision-free by **character class** rather than by survey. Application names are validated as DNS-1035 labels (`[a-z]([-a-z0-9]*[a-z0-9])?`), so a dot is structurally impossible in one; every release prefix is dot-free; and Secret names are DNS-1123 subdomains, where a dot is legal. So `` is dot-free by construction and can never equal a dotted name — for any application name, any operator, and any suffix an operator may add in future. That is a proof. "No operator claims it today" is a guess with an expiry date, and it expired twice. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. -### 5. Delivery: one engine-agnostic extraction controller +### 5. Delivery: a declared sentinel and one engine-agnostic controller + +The contract in §4 fixes the output object; this section fixes the input path — how the key-free `.tenant-ca` gets written for an engine whose operator does not already emit it under that name. Everything in this section is **internal**: no tenant and no consumer observes any of it. A tenant still reads exactly the object §4 defines — `.tenant-ca`, containing only `ca.crt` — and nothing here changes what that object is, how it is labelled, or how it is consumed. The delivery mechanism lives entirely below the line §4 draws. + +The input path splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** — and the platform's standing preference (source from a key-free object where one exists; strip only when forced to) follows from it: + +- **Engines whose operator already publishes a key-free CA under its own name.** `kafka` (Strimzi, native) is here today: its CA already lands key-free, under `-clients-ca-cert` rather than the canonical `.tenant-ca`, so the controller still copies it across through a declared projection — a straight copy with nothing to strip. `redis` is **not** in this group, despite appearances: its open pull request does teach the patched operator to publish a key-free `-ca-cert`, but nothing reads that Secret, because the chart's sentinel names the key-bearing CA cert-manager writes instead — so redis belongs with the engines below. +- **Every other engine** — nats, qdrant, redis, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output the platform reads. None can feed the helper at chart-render time (see "The problem"). The projection sources from the key-bearing CA Secret, and the controller strips everything but `ca.crt`. -The contract in §4 fixes the output object. The remaining work is the input path, and it splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** +The alternative to one controller is to change each upstream operator so it emits a key-free CA of its own, the way redis's open pull request adds `caCertSecretName` to the operator it patches. Carrying an equivalent change for CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than a single small controller — and redis is the cautionary case rather than the model, because the key-free output its patch adds goes unread: the chart projects from the CA cert-manager writes, which exists as soon as the certificate is issued rather than only after the operator has reconciled. So the controller is the target for every non-self-publishing engine. -- **Engines that self-publish a key-free CA** need no platform machinery. `kafka` (Strimzi, native) and `redis` (forked operator via `caCertSecretName`) are here: they converge by matching the canonical name and stamping the tenant-resource label. They opt **out** of the controller below simply by not stamping the source label. -- **Every other engine** — nats, qdrant, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output. None can feed the helper at chart-render time (see "The problem"). These opt **in** via an explicit source label and are served by **one engine-agnostic extraction controller**. +The mechanism is two pieces: a **declaration** the chart renders, and a small **controller** that reconciles it. An earlier revision of this proposal delivered the input path with a *label* on the CA Secret — the owner stamped `publish-ca-cert` and the controller selected sources by that label. An API-owner review of the implementation (`cozystack/cozystack#3299`) found that this emulates object semantics with a label: the thing being declared is "project this Secret's `ca.crt` as a trust anchor", a first-class intent that deserves a first-class object with a name, a status, and its own admission rules — not a boolean smuggled onto someone else's Secret. The mechanism below is that object. -The alternative to one controller is to fork each upstream operator the way redis was forked (adding a `caCertSecretName`-style key-free output). Forking CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than a single small controller, and redis's fork was a reaction to a review blocker, not a designed-up-front pattern. So the controller is the target for every non-self-publishing engine. +**(a) The owner declares the projection with a namespaced sentinel.** The app chart renders a small namespaced custom resource — `TenantProjection`, group `internal.cozystack.io` — that names the source Secret explicitly instead of stamping a label on it: -The controller is deliberately **engine-agnostic**: it does not branch on cert-manager-vs-operator-owned, and it does not key off Secret names. Its contract is three pieces, only the middle of which is new code. +```yaml +apiVersion: internal.cozystack.io/v1alpha1 +kind: TenantProjection +metadata: + name: -ca +spec: + projections: + - type: CACert + sourceSecretName: -ca # the engine's CA-bearing Secret + sourceKey: ca.crt # key to lift (defaults to ca.crt) +``` + +The API is settled: `TenantProjection` in group `internal.cozystack.io`, version `v1alpha1`, short name `tproj`, namespaced, with `spec.projections[].type` admitting exactly one value, `CACert` (`api/internalapi/v1alpha1/tenantprojection_types.go`; CRD at `packages/system/cozystack-controller/definitions/internal.cozystack.io_tenantprojections.yaml`). The projected Secret lands in the sentinel's own namespace, one per release, alongside the application. The sentinel's own name is free-form — the shipped postgres chart calls it `-ca` after the source it names (`packages/apps/postgres/templates/tenant-projection.yaml`) — because the controller never parses it: it derives the release from the `helm.toolkit.fluxcd.io/name` label Flux stamps on every rendered object (`internal/controller/cacert/reconciler.go:332`), and a sentinel that reaches the cluster without that label publishes nothing and reports `Ready=False, Reason=NoRelease`. -**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). For the cert-manager-minting charts this rides `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries the marker from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. +The sentinel is a **declaration**, not a copy of the data: the chart renders it at template time, so it exists in the Flux digest and is reconciled like any other chart object, even though the source Secret it names is created asynchronously by cert-manager or the DB operator long after render. Naming the source explicitly — rather than relying on a name convention or a label the operator must be coaxed into stamping — is what keeps the mechanism engine-agnostic: the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and `-ca-cert` is overloaded (key-free for the patched redis operator, key-bearing for PSMDB), and the declaration accommodates all of them because the chart author who knows the source name simply writes it down. It also dissolves the label design's release-attribution problem: cert-manager runs with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its application and the label leg had to add a separate `-release` label to say which release it belonged to; the sentinel is itself a chart object in the release's namespace, attributed to its release the same way every chart object is (see (c)), so no such marker is needed. -**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each label-selected source Secret it upserts a `type: Opaque` Secret named `-ca-cert` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: +**(b) A small extraction controller reconciles the sentinel.** The controller watches `TenantProjection` objects (not Secret labels) and the source Secrets they name. For each `CACert` projection it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. Its watch/upsert shape follows the merged wildcard-secret reconciler (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — ownership and lineage provide those (part (c)). It does four security-load-bearing things and nothing more: -- **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, verify the lifted value is a CA certificate and carries no `-----BEGIN … PRIVATE KEY-----` header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. -- **Tolerate operator-created, asynchronous sources.** The source may be created by the CNPG or PSMDB operator after the chart renders (not just by cert-manager). The controller waits on the watch event for the labeled source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. +- **Read the source the sentinel names, and check its content.** Read only the Secret named in `sourceSecretName`, lift only `sourceKey`, and before writing decode it in a `pem.Decode` loop, parse each block with `x509.ParseCertificate`, and rebuild the projected value only from blocks that validate as a certificate — never copying the input verbatim, so no byte the parser did not accept as a certificate can reach the tenant. That verifies the lifted value is a certificate, not that it is specifically a CA certificate. The sentinel says *which* Secret; the content check says *whether to strip* — which is what lets one controller straight-copy a key-free kafka source and strip a key-bearing PSMDB source without confusing the two, even though the key-free and key-bearing shapes both turn up under `-ca-cert`-adjacent names. +- **Tolerate operator-created, asynchronous sources.** The named source may be created by the CNPG or PSMDB operator (or by cert-manager) after the chart renders. The controller waits on the watch event for that source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. The sentinel exists from render time, so there is always a durable declaration to reconcile against — and to carry status (part (e)) — while the source is still missing. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. -- **Owner-ref the projected Secret to the application instance CR**, resolved from the `app.kubernetes.io/instance` label already on the source Secret. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid: Kubernetes garbage-collects the `-ca-cert` on app deletion for free, and the controller needs no prune logic of its own. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) +- **Owner-ref the projected Secret to the sentinel.** The projection takes an `ownerReference` to the `TenantProjection` that declared it — not to the application-instance CR, and not to the `HelmRelease` directly. The application-instance CR cannot be referenced: the `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, with a tenant application *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and no UID to point at. The sentinel, by contrast, is a real chart-rendered object in the same namespace, so a real owner-reference is valid and garbage collection is native (part (c)). `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (The projection does not owner-ref the *source* CA Secret: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before reaching the app.) + +**(c) Attribution, lineage, and garbage collection: the sentinel is a plain chart object.** The sentinel itself carries **no** `ownerReferences`. Flux stamps every object it renders with the `helm.toolkit.fluxcd.io/name` label naming the release, and that label — not an owner-reference — is how the sentinel is attributed to its release, exactly as for any chart-rendered object. So the lineage walk resolves the chain **Secret → sentinel → HelmRelease → application** with no change to the webhook: the projected Secret owner-refs the sentinel, the sentinel's release label names the `HelmRelease`, and the `HelmRelease` backs the application. Garbage collection is native and needs no delete path in the controller: deleting the application deletes the `HelmRelease`, Flux prunes the chart's objects including the sentinel, and Kubernetes then garbage-collects the projection the sentinel owns. Marking the projection tenant-visible is unchanged from §4 — the controller stamps `internal.cozystack.io/tenant-ca: "true"` on it, one generic `ApplicationDefinition.spec.secrets.include` entry (`matchLabels: {internal.cozystack.io/tenant-ca: "true"}`) matches it by label (`internal/lineagecontrollerwebhook/matcher.go`), and the lineage webhook then walks the projection's owner chain and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). + +**(d) Admission: RBAC is the control, not a ValidatingAdmissionPolicy.** No tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot create, update, or delete a `TenantProjection` to aim the controller at a Secret of its choosing — the only way a sentinel enters the cluster is by being rendered into a release. The extraction controller holds `get`/`list`/`watch` on `tenantprojections` and `get`/`update`/`patch` on `tenantprojections/status` — enough to read a declaration and write the `Ready` condition part (e) needs, and nothing beyond it (`packages/system/cozystack-controller/templates/rbac.yaml`). It never creates or deletes a sentinel, since that stays the chart's (helm-controller's) job. + +**(e) A broken declaration is observable.** The sentinel carries `status.conditions`, and the controller sets `Ready`. When a declaration cannot be satisfied — the named source Secret does not exist, or an upstream bump renames the CA Secret out from under `sourceSecretName` — the controller records `Ready=False, Reason=SourceNotFound` on the sentinel. This is a concrete advantage over the label design, which had no object to carry status: there, a source that never appeared could only make the controller requeue silently, invisible until someone noticed the trust anchor was missing. With the sentinel the failure is a queryable condition on a named object — alertable, and pointing at the exact broken release. -**(c) Marking stays in `spec.secrets`, by label not by name.** Because `ApplicationDefinition.spec.secrets.include` accepts a label selector (`internal/lineagecontrollerwebhook/matcher.go`), one generic entry — `matchLabels: {internal.cozystack.io/tenant-ca: "true"}`, stamped by the controller on every projected Secret — covers every engine with no per-release `resourceName` templating. The lineage admission webhook then does the rest: on admission it walks the projected Secret's `ownerReferences` to the owning application and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). +**(f) Revocation is part of the mechanism, and the canonical name is guarded.** The `tenantresource` verdict in (c) has to be able to go back to `false`, and that does not happen for free. Each projection records a digest of the owning `ApplicationDefinition`'s `spec.secrets` in `internal.cozystack.io/ca-cert-selectors`, and the digest is part of the drift check, so a change to those selectors forces exactly one write; that write also drops the `internal.cozystack.io/managed-by-cozystack` marker, which is what hands the projection back to the lineage webhook — the webhook's `objectSelector` skips any object already carrying the marker, so forcing a write without dropping it would leave the verdict frozen at whatever the selectors said on create. Both halves together are what lets a definition that stops selecting the anchor actually take the tenant's read access away, instead of leaving it reading a trust anchor the platform withdrew. The canonical name carries its own guards: a Secret already sitting at `.tenant-ca` that holds private key material is refused loudly rather than overwritten or adopted (`Ready=False, Reason=CanonicalNameOccupied`, plus a Warning Event on the Secret), and a release whose anchor is declared more than once publishes nothing — `MultipleCACertProjections` when a single sentinel declares two `CACert` entries, `ReleaseContested` when two sentinels in one namespace claim the same release. All of these live in `internal/controller/cacert/reconciler.go`. -What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`). +What this *reuses* rather than rebuilds: Flux's release-name label and its native pruning (for attribution and garbage collection), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), and the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`). The irreducibly new work is the `TenantProjection` sentinel and the extraction step that reconciles it — read one key from the named source, write a key-free copy, re-copy on rotation, report readiness — the same job an operator does natively on the one engine that already ships a key-free CA object (kafka's `-clients-ca-cert`). Kafka still declares a sentinel, because its CA lands under an engine-specific name rather than the canonical one; for it alone the extraction step is a straight copy, with nothing to strip. ### 6. Per-engine application order -`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `-ca-cert` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` is already at the target and needs only documentation. `redis` (`cozystack/cozystack#2729`) converges by shape adaptation: its forked operator self-publishes a key-free `-ca-cert` via `caCertSecretName`, so it does not stamp the source label and the controller leaves it alone. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) adopt the controller via the source label. `kubernetes` (Kamaji) is explicitly out. +`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-clients-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) is a stripping case rather than a straight copy: its chart mints a cert-manager chain and declares a sentinel naming the key-bearing `-ca-tls` that chain produces, so the controller lifts `ca.crt` and drops the key, and the key-free `-ca-cert` the patched operator also publishes goes unread. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) converge by declaring a `TenantProjection` sentinel that names their CA Secret. `kubernetes` (Kamaji) is explicitly out. ## User-facing changes -A tenant sees one canonical, key-free trust-anchor object per managed application — `-ca-cert`, carrying only `ca.crt` — through the dashboard resource map and the `tenantsecrets` projection, in the same shape across every engine. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. +A tenant sees one canonical, key-free trust-anchor object per managed application — `.tenant-ca`, carrying only `ca.crt` — through the dashboard resource map and the `tenantsecrets` projection, in the same shape across every engine. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. ## Upgrade and rollback compatibility -This document changes nothing at runtime; it records a target. The pull requests that implement it are individually backward-compatible in the engine's own PKI: per-app TLS is opt-in (tri-state), external exposure is opt-in, and the extraction controller adds a new object without altering existing Secrets. postgres convergence is purely additive — it publishes a new key-free `-ca-cert` and leaves the passwords-only `-credentials` in place (removing it would drop the tenant's connection passwords). Reverting any implementing PR removes the new `-ca-cert` object and the controller that maintains it, leaving the engine's existing PKI untouched. +This document changes nothing at runtime; it records a target. The pull requests that implement it are individually backward-compatible in the engine's own PKI: per-app TLS is opt-in (tri-state), external exposure is opt-in, and the extraction controller adds a new object without altering existing Secrets. postgres convergence is purely additive — it publishes a new key-free `.tenant-ca` and leaves the passwords-only `-credentials` in place (removing it would drop the tenant's connection passwords). Reverting any implementing PR removes the new `.tenant-ca` object and the controller that maintains it, leaving the engine's existing PKI untouched. ## Security @@ -180,35 +207,35 @@ The trust boundary is precise: a tenant receives `ca.crt` and never receives `tl No merged engine leaks a private key to a tenant today. The merged charts withhold their key-bearing Secrets, and the tenant-facing objects (postgres `-credentials`, the nats/qdrant credentials Secrets) are passwords-only. The live instance of the risk is not in `main` — it is in the **in-flight per-app PRs**, several of which currently propose labelling a key-bearing Secret to tenants: mariadb (`cozystack/cozystack#2680`) projects the CA private key `-ca-tls`, mongodb (`cozystack/cozystack#2692`) projects the key-bearing `-ca-cert`, and rabbitmq (`cozystack/cozystack#2683`) and opensearch (`cozystack/cozystack#2682`) project the leaf key. This contract exists to stop them landing that way; the consume object and the extraction controller are what let those PRs deliver `ca.crt` without the key. -Two consequences for the controller. First, it writes at runtime, after chart-render, so it cannot lean on the helper's render-time guard alone: it must whitelist the single `ca.crt` key and re-assert the no-private-key check itself on every write. Second, it adds a new trust surface — read access to per-release CA Secrets, write access for the key-free copy — and must never overwrite a Secret it did not create, surfacing any name collision as a Warning Event rather than failing silently. +Two consequences for the controller. First, it writes at runtime, after chart-render, so it cannot lean on the helper's render-time guard alone: it must whitelist the single `ca.crt` key and re-assert the no-private-key check itself on every write. Second, it adds a new trust surface — read access to per-release CA Secrets, write access for the key-free copy — and must never overwrite a Secret it did not create, surfacing any name collision as a Warning Event rather than failing silently. The declaration that drives it is itself locked down: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a `TenantProjection` aiming the controller at a Secret of their choosing — RBAC alone is the control, with no admission policy narrowing the writer further. ## Failure and edge cases - The helper's input PEM contains a private-key header → render fails closed; the chart does not deploy a key-bearing Secret. -- The source `ca.crt` somehow carries a private-key header at runtime → the controller refuses to write the copy (runtime whitelist plus guard); no key-bearing Secret is ever projected. -- The labeled source is a key-free Secret that should be left alone (a self-publishing engine mis-stamped) → the content check sees no private key to strip and the controller still writes only `ca.crt`; no key is ever exposed. -- The per-release CA Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on the labeled source; it does not error or busy-loop. +- The source `ca.crt` somehow carries a block that does not parse as an `x509.Certificate` at runtime → the controller refuses to write the copy (the `pem.Decode`/`x509.ParseCertificate` guard rebuilds the projection only from validated blocks); no key-bearing Secret is ever projected. +- The sentinel names a key-free source that should be left alone (a self-publishing engine) → the content check sees no private key to strip and the controller still writes only `ca.crt`; no key is ever exposed. +- The named source Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on that source and records `Ready=False, Reason=SourceNotFound` on the sentinel meanwhile; it does not error or busy-loop. +- An upstream bump renames the CA Secret out from under `sourceSecretName` → the source never reappears under the declared name, and the sentinel surfaces `Ready=False, Reason=SourceNotFound` — a queryable, alertable condition rather than a silent requeue. - A foreign Secret already occupies the target name → the controller leaves it untouched (management-label guard) and emits a Warning Event on the application, so the operator sees the collision. - The CA rotates → the controller re-copies `ca.crt` on the next source change; no chart re-render is required. -- The application is deleted → the projected `-ca-cert` is garbage-collected by Kubernetes via its owner reference to the application instance; the controller needs no delete path. +- The application is deleted → Flux prunes the sentinel with the rest of the chart's objects, and Kubernetes garbage-collects the projected `.tenant-ca` via its owner reference to the sentinel; the controller needs no delete path. ## Testing - The helper is already covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`, including the fail-closed assertions. -- The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a labeled source appearing after the consumer (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), and owner-reference-driven garbage collection on app deletion. -- Each per-app pull request adds helm-unittest fixtures asserting the `-ca-cert` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a sentinel whose named source appears after it (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), a sentinel whose source never appears (asserting `Ready=False, Reason=SourceNotFound` on `status.conditions`), and owner-reference-driven garbage collection of the projection when the sentinel is pruned. +- Each per-app pull request adds helm-unittest fixtures asserting the chart renders the `TenantProjection` sentinel with the right `sourceSecretName`, and asserting the projected `.tenant-ca` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- No dedicated admission fixture is needed: RBAC is the control, and no tenant-facing ClusterRole grants any verb on `internal.cozystack.io`, so a tenant cannot write a `TenantProjection` by any path. ## Rollout -1. Edge — `cozystack/cozystack#2988` and `cozystack/cozystack#2989` merged; per-tenant propagation `cozystack/cozystack#2990` still open. +1. Edge — `cozystack/cozystack#2988`, `cozystack/cozystack#2989`, and the per-tenant propagation `cozystack/cozystack#2990` are all merged. 2. This contract — accepted. -3. Extraction controller — implemented and tested. +3. Sentinel CRD and extraction controller — implemented and tested. 4. Per-app convergence — postgres first (tracked by `cozystack/cozystack#2814`), then the remaining per-app TLS pull requests onto the contract. ## Open questions -- **Labelling operator-created sources.** The cert-manager charts stamp the source label via `Certificate.spec.secretTemplate.labels`. Operators that create the CA Secret themselves (CNPG `-ca`, PSMDB `-ca-cert`) may not let the chart label their output. For those, the controller needs either an operator that supports output labels or a small per-engine source-name configuration as a fallback — this is the one place the engine-agnostic label is not yet sufficient, and it should be resolved before postgres/mongodb convergence. -- The exact namespace convention for the `-ca-cert` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. - How this intersects with per-tenant wildcard propagation (issue `cozystack/cozystack#2820`, implemented by PR `cozystack/cozystack#2990`). That is a *cross-namespace* replication problem; the extraction controller here is *intra-namespace, per-release*, so the two do not share a mechanism — but they should share the management-label and foreign-collision conventions. Note that `clustersecret-operator` is packaged in the repo but is not wired into a default bundle, so it is not an installed primitive this design can assume. ## Alternatives considered @@ -216,12 +243,13 @@ Two consequences for the controller. First, it writes at runtime, after chart-re - **Pure-consume (applications stop minting, consume a central cert-manager output).** Rejected: it breaks the rotation lifecycle of CloudNativePG and Strimzi, whose own-CA management is mutually exclusive with an externally-supplied server certificate. - **`lookup` for the asynchronous CA.** Rejected: `lookup` runs at render time and is invisible to the Flux digest, so the chart would not re-render when the CA appears. (`cozystack/cozystack#1787` already moved the global-values channel off `lookup` onto `valuesFrom` for the same digest reason.) - **A custom `valuesFrom` pointing at `-ca`.** Rejected: `expectedValuesFrom()` pins every application HelmRelease to the single `cozystack-values` Secret and overwrites drift. -- **Forking each upstream operator (the redis path) to emit a key-free CA.** Rejected as the general mechanism: it works for redis because the fork is small, but applying it to CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than one engine-agnostic controller. +- **Changing each upstream operator to emit a key-free CA.** Rejected as the general mechanism, and retired even for the engine that pioneered it. Redis was going to carry a forked operator; its pull request now carries a downstream patch over freshworks-oss `redis-operator` v3.3.5 instead, and the key-free Secret that patch produces goes unread, because the chart projects from the CA cert-manager writes. Carrying an equivalent change for CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator would be far more surface to own than one engine-agnostic controller. - **cert-manager `trust-manager`.** Considered and rejected as the mechanism, recorded as the validating prior art. trust-manager is the canonical "watch a CA source, materialize a key-free copy, never touch `tls.key`" operator and confirms the copy-model is the mainstream choice for trust-anchor distribution. But it does not fit operationally: its `Bundle`/`ClusterBundle` is cluster-scoped and fans out to namespaces by selector (not one object per release); it reads its sources from its own trust namespace, not from an arbitrary per-release tenant namespace; and it targets ConfigMaps first (Secret targets are opt-in). Modelling per-release, intra-namespace extraction on it would mean one cluster-scoped Bundle per database release reading a source it cannot natively see — against the grain. The small intra-namespace controller is a better fit. -- **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `-ca-cert` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. -- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's explicit source label plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. +- **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `.tenant-ca` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. +- **A source-selection label on the CA Secret (the earlier design of this proposal, superseded).** The owner stamped `internal.cozystack.io/publish-ca-cert` on its CA Secret and the controller selected sources by that label. It was adopted first, then replaced by the declared `TenantProjection` sentinel of §5 after an API-owner review (`cozystack/cozystack#3299`) found it emulates object semantics: a label carries no name, no status, and no admission surface of its own, so a broken declaration could only requeue silently, and the release a cert-manager Secret belonged to had to be supplied by a second `-release` label because the Secret carried no ownerReference. The declared sentinel is the chosen mechanism — a first-class namespaced object with `status.conditions`, RBAC-restricted writes, and a source named **by name** rather than by label. Declaring the source by name also closes this proposal's former open question "Labelling operator-created CA sources": CNPG and PSMDB need not label their operator-created output at all, because the chart names that Secret in the sentinel's `sourceSecretName`. The implementation briefly carried a second discovery leg alongside the label — a `spec.caCert` field on the cluster-scoped, per-kind `ApplicationDefinition` — with the mirror-image flaw: a cluster-scoped object standing in for a per-namespace, per-release fact. Both legs collapse into the single namespaced sentinel. +- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's declared sentinel plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; the sentinel names the source explicitly instead, and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. - **A field filter on the projection itself (the principled root fix — deferred to its own proposal).** Today `tenantsecrets` delivers the whole Secret `Data`, which is the only reason a key-free *copy* must be materialized at all. A per-key field filter on the projection (or the `spec.secrets` selector) would let a single `ca.crt` key be projected straight out of a key-bearing Secret — no controller, no second object — and would generalize to the operator-owned engines. It is **not** a local change, which is why it is deferred rather than adopted here: the projection is **writable** at the registry level (`pkg/registry/core/tenantsecret/rest.go` implements Create/Update/Patch/Delete), and the write path (`tenantToSecret`) replaces the underlying Secret's `Data` **wholesale** (`out.Data = ts.Data`). A read-side key filter without a matching write-side filter would, the moment any principal with write access used the filtered view, silently drop the keys it could not see. Tenant roles grant only `get/list/watch` today, but that read-only posture lives in a different package (`packages/system/cozystack-basics` clusterroles), so a field filter's safety is entangled with a write path and an RBAC posture defined elsewhere. That makes it a redesign of the tenant-secret API with its own blast-radius analysis — a separate design proposal — not a rider on the TLS rollout. Recorded here as the future simplification the extraction controller could later retire, explicitly **not** a dependency of this proposal. -- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `-ca-cert` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. +- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `.tenant-ca` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. - **A general-purpose cluster-secret replication operator / a copy-issuer webhook.** Rejected during the edge work (`cozystack/cozystack#2990`, `cozystack/cozystack#2812`) in favor of native references and a purpose-built reconciler with a tight ownership guard that move only the Secret name, never key material. --- From 13714f0988f6989b261413e07818611ba1e488da Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Thu, 3 Sep 2026 09:52:18 +0300 Subject: [PATCH 2/6] docs(unified-tls-pki): anchor the engine table and fix three facts The engine table described `main` in the present tense, so it goes stale every time the contract lands somewhere new. Anchor it to one dated moment and drop "today" from the two column headers and the takeaway; the table is a record of the gap this contract closes, not a tracker of `main`. Three claims were stale under that anchor. postgres, nats and qdrant all render a TenantProjection sentinel on `main` and deliver `.tenant-ca`, so their "ca.crt to tenant?" cells and the takeaway now say so, and section 6 and the rollout list no longer carry nats and qdrant as convergence still to come. Two identifiers were wrong outright: redis's CA Secret is `.ca-tls`, with a dot for the same collision-across-releases reason section 4 gives for the canonical name, and the carried operator patch is cut from freshworks-oss/redis-operator#85 commit 2ad2ed8d1. mariadb's `-ca-tls` is hyphenated and is left alone. Assisted-by: LLM Signed-off-by: Timofei Larkin Signed-off-by: Aleksei Sviridkin --- design-proposals/unified-tls-pki/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index f056105..7b864cd 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -113,23 +113,23 @@ The edge is largely done and is recorded here only to fix the framing. An operat ### 3. Interior PKI-ownership contract -The contract is a per-engine table. It is the artifact the per-app pull requests must conform to, and it makes the two-tier reality explicit: the operator owns the CA; the platform unifies only how `ca.crt` is consumed. The decisive column is **"self-publishes a key-free `ca.crt`?"** — that, not "mint vs consume", is what determines whether an engine needs a delivery mechanism. +The contract is a per-engine table. It is the artifact the per-app pull requests must conform to, and it makes the two-tier reality explicit: the operator owns the CA; the platform unifies only how `ca.crt` is consumed. The decisive column is **"self-publishes a key-free `ca.crt`?"** — that, not "mint vs consume", is what determines whether an engine needs a delivery mechanism. The table describes `main` as of 2026-09-02. -| Engine | PKI owner | CA-bearing Secret today | Key in that Secret? | Self-publishes key-free `ca.crt`? | `ca.crt` to tenant today? | Operator certificate capability | +| Engine | PKI owner | CA-bearing Secret | Key in that Secret? | Self-publishes key-free `ca.crt`? | `ca.crt` to tenant? | Operator certificate capability | | --- | --- | --- | --- | --- | --- | --- | | kafka (Strimzi) | operator | `-cluster-ca-cert`, `-clients-ca-cert` | no | **yes** | yes — clean reference | `generateCertificateAuthority: false` BYO-CA + per-listener `brokerCertChainAndKey`; public cert on external listener, own-CA mTLS internally | -| redis | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2729` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca-tls` is key-bearing | n/a | **not yet** — under `cozystack/cozystack#2729` the patched operator does publish a key-free `-ca-cert`, but nothing on the platform reads it: the chart's sentinel sources the cert-manager CA instead | no | stock freshworks-oss `redis-operator` v3.3.5 has no TLS support; `cozystack/cozystack#2729` carries it as a downstream patch over v3.3.5 (`packages/system/redis-operator/images/redis-operator/patches/tls.diff`, cut from freshworks-oss/redis-operator#85 commit 38b9e3732), which adds `spec.tls` including `caCertSecretName` | -| postgres (CloudNativePG) | operator | `-ca` (operator-created, async) | **yes — `ca.key`** | no | no — tenant gets passwords-only `-credentials` (gap) | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | +| redis | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2729` (open, in review) adds a chart cert-manager chain whose CA Secret `.ca-tls` is key-bearing | n/a | **not yet** — under `cozystack/cozystack#2729` the patched operator does publish a key-free `-ca-cert`, but nothing on the platform reads it: the chart's sentinel sources the cert-manager CA instead | no | stock freshworks-oss `redis-operator` v3.3.5 has no TLS support; `cozystack/cozystack#2729` carries it as a downstream patch over v3.3.5 (`packages/system/redis-operator/images/redis-operator/patches/tls.diff`, cut from freshworks-oss/redis-operator#85 commit 2ad2ed8d1), which adds `spec.tls` including `caCertSecretName` | +| postgres (CloudNativePG) | operator | `-ca` (operator-created, async) | **yes — `ca.key`** | no | yes — `.tenant-ca`, alongside the unchanged passwords-only `-credentials` | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | | mongodb (Percona PSMDB) | operator | `-ca-cert` (operator-created at runtime; the chart renders no TLS objects) | **yes — cert-manager `isCA`** | no | no (gap; name collides with redis, opposite shape) | `spec.tls.issuerConf` BYO issuer; auto-mints a cert-manager chain | -| nats | chart + cert-manager | `-ca` | yes | no | no | — | -| qdrant | chart + cert-manager | `-ca` | yes | no | no | — | +| nats | chart + cert-manager | `-ca` | yes | no | yes, when TLS is on | — | +| qdrant | chart + cert-manager | `-ca` | yes | no | yes, when TLS is on | — | | mariadb | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2680` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca-tls` is key-bearing | n/a | no | no | mariadb-operator `serverCertSecretRef` / `serverCASecretRef` / `serverCertIssuerRef` | | opensearch | operator — opster generates its own TLS | none chart-rendered — on `main` the chart sets `security.tls.transport.generate` and `security.tls.http.generate`, so opster mints the material and the chart renders no cert-manager objects; `cozystack/cozystack#2682` (open, in review) adds a chart cert-manager chain whose CA Secret `-http-ca` is key-bearing | n/a | no | no | opster `security.tls.http.secret.name`; transport operator-generated | | rabbitmq | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2683` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca` is key-bearing | n/a | no | no | cluster-operator `spec.tls.secretName` / `caSecretName` | | clickhouse (Altinity) | — (BYO-cert) | supplied cert mount | n/a | no | no (not in TLS series yet) | mounts a supplied cert; no native issuance | | kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | no — out of model | control-plane CA, kubeconfig-pinned, non-swappable | -The takeaway: **kafka** (native) is the only engine that publishes a key-free `ca.crt` under its own name today, and it is still not at the canonical `.tenant-ca` — it alone needs the controller to copy that output across under that name (§5). **Every other engine that ships TLS on `main` today — postgres, mongodb, nats, qdrant — has a key-bearing CA Secret and delivers no `ca.crt` to the tenant**, one uniform delivery gap, and the four whose TLS is still in review (redis, mariadb, rabbitmq, opensearch) all mint a key-bearing CA the same way. **redis** is the one that already closes the gap inside its own pull request: `cozystack/cozystack#2729` declares a `TenantProjection` sentinel naming that key-bearing chart CA, so the anchor lands directly at `.tenant-ca` with the key stripped, and the key-free `-ca-cert` its patched operator also publishes goes unread. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. +The takeaway: **kafka** (native) is the only engine that publishes a key-free `ca.crt` under its own name, and it is still not at the canonical `.tenant-ca` — it alone needs the controller to copy that output across under that name (§5). **Every other engine that ships TLS — postgres, mongodb, nats, qdrant — has a key-bearing CA Secret**, one uniform delivery gap; postgres, nats and qdrant have since closed it by declaring a sentinel, leaving mongodb, and the four whose TLS is still in review (redis, mariadb, rabbitmq, opensearch) all mint a key-bearing CA the same way. **redis** is the one that already closes the gap inside its own pull request: `cozystack/cozystack#2729` declares a `TenantProjection` sentinel naming that key-bearing chart CA, so the anchor lands directly at `.tenant-ca` with the key stripped, and the key-free `-ca-cert` its patched operator also publishes goes unread. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. ### 4. The uniform consume contract @@ -191,7 +191,7 @@ What this *reuses* rather than rebuilds: Flux's release-name label and its nativ ### 6. Per-engine application order -`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-clients-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) is a stripping case rather than a straight copy: its chart mints a cert-manager chain and declares a sentinel naming the key-bearing `-ca-tls` that chain produces, so the controller lifts `ca.crt` and drops the key, and the key-free `-ca-cert` the patched operator also publishes goes unread. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) converge by declaring a `TenantProjection` sentinel that names their CA Secret. `kubernetes` (Kamaji) is explicitly out. +`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-clients-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) is a stripping case rather than a straight copy: its chart mints a cert-manager chain and declares a sentinel naming the key-bearing `.ca-tls` that chain produces, so the controller lifts `ca.crt` and drops the key, and the key-free `-ca-cert` the patched operator also publishes goes unread. `nats` and `qdrant` have since converged the same way. `mongodb`, then rabbitmq, mariadb and opensearch, converge by declaring a `TenantProjection` sentinel that names their CA Secret. `kubernetes` (Kamaji) is explicitly out. ## User-facing changes @@ -232,7 +232,7 @@ Two consequences for the controller. First, it writes at runtime, after chart-re 1. Edge — `cozystack/cozystack#2988`, `cozystack/cozystack#2989`, and the per-tenant propagation `cozystack/cozystack#2990` are all merged. 2. This contract — accepted. 3. Sentinel CRD and extraction controller — implemented and tested. -4. Per-app convergence — postgres first (tracked by `cozystack/cozystack#2814`), then the remaining per-app TLS pull requests onto the contract. +4. Per-app convergence — postgres (tracked by `cozystack/cozystack#2814`), nats and qdrant merged; the remaining per-app TLS pull requests onto the contract. ## Open questions From 86b3fceff93cd05a54cb95f34c28d77b507e6228 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 4 Sep 2026 00:48:52 +0300 Subject: [PATCH 3/6] docs(unified-tls-pki): describe the trust anchor as the merged controller delivers it The delivery mechanism in section 5 is the one that shipped: a chart renders a TenantProjection sentinel naming the engine's CA-bearing Secret, the controller copies ca.crt alone into .tenant-ca, and the ResourceDefinition selects that copy by the tenant-ca label. The label leg that earlier revisions described was never merged and is gone from the text. The engine table carries structural facts only: who owns the PKI, which Secret holds the CA and whether it carries a key, how the anchor is delivered under this contract. Which engines have converged is tracked in cozystack/cozystack#2814, not here, so the table cannot go stale with the next merge. clickhouse serves no TLS at all; its only certificate input is the outbound trust bundle of the backup sidecar. postgres is described on its own, before the contract, which is what the whole Context section does. kafka's key-free CA Secret is cited from Strimzi's split between the CA certificate and CA key Secrets. The twelve Ready reasons match the reconciler one for one, and the content gate is described as all-or-nothing: a value is refused or passed whole, and the single-key copy strips unconditionally. The dot in the canonical name rests on two assumptions the platform does not enforce, and both are stated: no per-release suffix in use ends with .tenant-ca, and the release name the controller derives is dot-free, which the apps API guarantees and the Flux name label the controller actually reads does not. Dotted suffixes are already in use, so the argument is conditional on that set. References to work that never shipped are corrected: community#29 was closed, and the ExposureClass/ServiceExposure implementation was removed before v1.6.0 after riding its release candidates. Closed workstreams are marked as such. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/unified-tls-pki/README.md | 181 +++++++++++++-------- 1 file changed, 112 insertions(+), 69 deletions(-) diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 7b864cd..625d090 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -1,9 +1,8 @@ - # Unified TLS and PKI model for managed applications - **Title:** `Unified TLS and PKI model for managed applications` - **Author(s):** `@lexfrei` -- **Date:** `2026-06-24`; revised `2026-09-02` (the §5 delivery mechanism, revised after the API-owner review in `cozystack/cozystack#3299`) +- **Date:** `2026-06-24`; revised `2026-09-03` (the §5 delivery mechanism, revised after the review of `cozystack/cozystack#3299` and its split into `cozystack/cozystack#3407`, `#3408` and `#3411`, and with §3's engine table, §4's naming and separator rule, §6, Testing and Alternatives rewritten alongside it) - **Status:** Accepted ## Overview @@ -14,38 +13,45 @@ This proposal records a written target design for the whole model, so that the t ## Scope and related proposals -This proposal is the umbrella design for the work tracked by epic `cozystack/cozystack#2811`. It does not re-specify the edge work that has already merged; it states the target the whole model converges on and focuses on the interior contract, where the per-engine work still sits. +This proposal is the umbrella design for the work tracked by epic `cozystack/cozystack#2811`. It does not re-specify the edge work; it states the target the whole model converges on and focuses on the interior contract, where the per-engine work sits. -- **Edge, merged:** `cozystack/cozystack#2988` (ACME wildcard on the default ingress-nginx path), `cozystack/cozystack#2989` (the CA-only trust-anchor helper), and `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`, merged 2026-07-15). The whole edge tier has landed. -- **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (end-to-end TLS for databases); `cozystack/cozystack#2977` (opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. -- **Per-app TLS series (open):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). These are the pull requests that should land *after* this contract is accepted, not before; several currently propose handing tenants a key-bearing Secret, which this contract exists to correct (see Security). -- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `.tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. +- **Edge (PRs):** `cozystack/cozystack#2988` (ACME wildcard on the default ingress-nginx path), `cozystack/cozystack#2989` (the CA-only trust-anchor helper), and `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`). +- **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (closed, end-to-end TLS for databases); `cozystack/cozystack#2977` (closed, opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. +- **Per-engine TLS series (PRs):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). Each carries one engine onto this contract, and each is meant to land against it rather than ahead of it — the failure mode they exist to avoid is handing a tenant a key-bearing Secret (see Security). Which of them have landed is tracked in `cozystack/cozystack#2814`, not here. +- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `.tenant-ca` object defined here. The structured `expose` model carries an "external implies TLS" rule that likewise points clients at this trust anchor. It has not landed: the proposal (`cozystack/community#29`) was closed, and the `ExposureClass` / `ServiceExposure` implementation (`cozystack/cozystack#3081`) was removed again by `cozystack/cozystack#3218` in favour of a native `Service` `loadBalancerClass`, so it was gone before `v1.6.0`. Both consume this proposal's output and define no PKI of their own. -All repository paths below refer to the `cozystack/cozystack` repository. Every path cited here is on `main` except those attributed to the still-open per-app TLS series (`cozystack/cozystack#2729`, `cozystack/cozystack#2692`, `cozystack/cozystack#2683`, `cozystack/cozystack#2682`, `cozystack/cozystack#2680`), whose chart templates are not. +All repository paths below refer to the `cozystack/cozystack` repository. A path belonging to an engine's TLS series is cited from that series' branch and named with it; everything else is cited from the platform. This document states the contract, not the state of a branch — [`design-proposals/README.md`](../README.md) is explicit that the codebase is the final source of truth for what the system does. + +## Decisions + +- [0002. mongodb sources the trust anchor from its leaf Secret, not from the operator's CA Secret](./decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md) — the three properties that beat the obvious source. +- [0001. The trust anchor is declared by a namespaced `TenantProjection` and published as `.tenant-ca`](./decisions/0001-the-trust-anchor-is-declared-by-a-namespaced-sentinel.md) — why the source-selection label was set aside, and why both `-ca-cert` and `-tenant-ca` were rejected as the canonical name. ## Context -### Edge today +This section describes each engine on its own — what its chart and its operator provide before the §5 contract reaches it — not the tenant-visible result after it has. Which engines have converged onto the contract is tracked in `cozystack/cozystack#2814`. + +### The edge System ingresses (dashboard, grafana, keycloak, harbor) get a per-host certificate via the cert-manager `cluster-issuer` annotation and ingress-shim. The `letsencrypt-prod`, `letsencrypt-stage`, and `selfsigned-cluster-issuer` ClusterIssuers exist with HTTP-01 and DNS-01 solvers. Gateway API is present but opt-in, and in DNS-01 mode the `TenantGateway` controller renders a per-apex wildcard. With `cozystack/cozystack#2988` an operator can now drop in a wildcard Secret and have the default ingress path serve it via `--default-ssl-certificate`; `cozystack/cozystack#2990` extends that to per-tenant termination points. -### Interior today +### The interior -The managed engines fall into two classes by *who mints*, but — and this is the load-bearing correction — that axis is **not** the axis that decides who needs a delivery mechanism. Verified against the current charts on `main` and the open per-app PRs. +The managed engines fall into two classes by *who mints*, but — and this is the load-bearing correction — that axis is **not** the axis that decides who needs a delivery mechanism. Verified against the charts and against each engine's TLS series. -The first class **mints** its own chain from a chart-rendered cert-manager graph (self-signed Issuer → CA Certificate → CA Issuer → leaf Certificate). `nats` and `qdrant` do this in `main` (`packages/apps/nats/templates/certmanager.yaml`, `packages/apps/qdrant/templates/certmanager.yaml`); the open per-app pull requests for redis, rabbitmq, mariadb, and opensearch add the same shape. In this class the CA Secret (`-ca` or `-ca-tls`) is a cert-manager CA-certificate Secret and therefore **carries the private key**, and the `ca.crt` is not delivered to tenants at all today. +The first class **mints** its own chain from a chart-rendered cert-manager graph (self-signed Issuer → CA Certificate → CA Issuer → leaf Certificate). `nats` and `qdrant` do this (`packages/apps/nats/templates/certmanager.yaml`, `packages/apps/qdrant/templates/certmanager.yaml`), and the series for redis, rabbitmq and opensearch adds the same shape. mariadb's does too, but only under its non-default `tls.issuer: cert-manager`; by default that engine's CA is the operator's own. In this class the CA Secret is a cert-manager CA-certificate Secret and therefore **carries the private key**, and it is not an object a tenant may be handed. Its name is not uniform, and the whole set is worth reading once, because §4 builds a rule on the separator: `-ca` for nats, qdrant and rabbitmq, `.ca-tls` for mariadb and for redis, and `.http-ca` for opensearch. -The second class **consumes** PKI that its operator owns end-to-end — but, crucially, operator-owned does **not** imply already-converged. Only one engine in it actually delivers a key-free trust anchor today: +The second class **consumes** PKI that its operator owns end-to-end — but, crucially, operator-owned does **not** imply already-converged. Being operator-owned says nothing about whether the trust anchor is already key-free and reachable: -- `kafka` (Strimzi) is the **clean** reference: it exposes `-cluster-ca-cert` and `-clients-ca-cert`, each a CA-certificate-only object with no private key (`packages/apps/kafka/templates/dashboard-resourcemap.yaml`). Strimzi can serve a public certificate on its external listener while keeping internal broker mTLS on its own CA (`generateCertificateAuthority: false` + per-listener `brokerCertChainAndKey`). -- `postgres` (CloudNativePG) renders no cert-manager objects; the CNPG operator auto-generates a self-signed CA and signs the server certificate. The CA lives in the operator-created `-ca` Secret, which **carries `ca.key`** and is created **asynchronously**; it is **not** delivered to tenants. The only tenant-facing object is `-credentials` (`packages/apps/postgres/templates/init-script.yaml`), which is a chart-rendered Opaque Secret holding **only `user: password` pairs** — no `ca.crt`, no `tls.key` — surfaced through the dashboard resource map (`packages/apps/postgres/templates/dashboard-resourcemap.yaml`). So postgres is a **trust-anchor delivery gap** (the tenant gets passwords but never `ca.crt`), the same shape as nats/qdrant — *not* a private-key coupling. The comment at `packages/apps/postgres/templates/db.yaml:20-22` claiming `ca.crt` rides in `-credentials` contradicts what the chart actually renders and is a documentation bug. -- `mongodb` (Percona PSMDB) is operator-owned, but the PSMDB operator mints a cert-manager chain **at runtime** and publishes `-ca-cert` as a **key-bearing** cert-manager `isCA` Secret (`tls.crt` + `tls.key` + `ca.crt`). The chart itself renders no TLS objects on `main` today — the whole PKI is operator-created after deploy, which is exactly why it needs the same asynchronous-source treatment as CNPG. Same gap as the rest, plus a trap: the name `-ca-cert` is key-free for the patched redis operator and key-bearing for PSMDB — opposite shapes under one name. +- `kafka` (Strimzi) is the **clean** reference: it exposes `-cluster-ca-cert` and `-clients-ca-cert`, each a CA-certificate-only object with no private key. The split is upstream naming, not a chart choice: Strimzi keeps each CA's private key in a separate Secret without the `-cert` suffix, `-cluster-ca` beside `-cluster-ca-cert` (`KafkaResources.clusterCaKeySecretName` and `clusterCaCertificateSecretName` in strimzi-kafka-operator `0.45.1-rc1`, the version `packages/system/kafka-operator` vendors). The chart grants the tenant the two `-cert` objects and neither key Secret (`packages/apps/kafka/templates/dashboard-resourcemap.yaml`). Strimzi can serve a public certificate on its external listener while keeping internal broker mTLS on its own CA (`generateCertificateAuthority: false` + per-listener `brokerCertChainAndKey`). +- `postgres` (CloudNativePG) renders no cert-manager objects; the CNPG operator auto-generates a self-signed CA and signs the server certificate. The CA lives in the operator-created `-ca` Secret, which **carries `ca.key`** and is created **asynchronously**; it is **not** delivered to tenants. On its own the engine hands the tenant one object, `-credentials` (`packages/apps/postgres/templates/init-script.yaml`), a chart-rendered Opaque Secret holding **only `user: password` pairs** — no `ca.crt`, no `tls.key` — surfaced through the dashboard resource map (`packages/apps/postgres/templates/dashboard-resourcemap.yaml`). So the engine-native picture is a **trust-anchor delivery gap** — the tenant gets passwords, and nothing in the engine hands it `ca.crt` — the same shape as nats/qdrant, *not* a private-key coupling, and it is the gap §5 closes. +- `mongodb` (Percona PSMDB) is operator-owned, but the PSMDB operator mints a cert-manager chain **at runtime** and publishes `-ca-cert` as a **key-bearing** cert-manager `isCA` Secret (`tls.crt` + `tls.key` + `ca.crt`). The chart names the leaf Secret in `spec.secrets.ssl` and pins the operator's `tls.mode`, but renders no cert-manager objects of its own: the whole PKI is operator-created after deploy, which is exactly why it needs the same asynchronous-source treatment as CNPG. Same gap as the rest, plus a trap for anyone matching CA Secrets by name: PSMDB's key-bearing `-ca-cert` and the key-free `.ca-cert` the patched redis operator publishes are one separator apart and hold opposite shapes. -And one engine is the proof of the whole framing rather than a mere exclusion. `kubernetes` (Kamaji) is the **fourth independent PKI owner** — alongside cert-manager chart graphs, CNPG, and Strimzi — and the one that is fundamentally non-swappable: Kamaji owns the control-plane CA, the kubeconfig it hands out *pins* that CA, and every client authenticates against it by construction, so a public edge certificate is not just unnecessary there but meaningless. Kamaji is the strongest single argument for "unify the interface, not the CA": any design that homogenized the certificate authority would break it outright, whereas the consume contract below (a key-free trust anchor, delivered uniformly) is something even Kamaji already satisfies in spirit — the kubeconfig *is* its trust-anchor delivery. It needs no unification work and is excluded from the rollout, but it belongs in the enumeration as the boundary case that fixes the model's shape. +And one engine is the proof of the whole framing rather than a mere exclusion. `kubernetes` (Kamaji) is an **independent PKI owner** like the others this table lists, and the one that is fundamentally non-swappable: Kamaji owns the control-plane CA, the kubeconfig it hands out *pins* that CA, and every client authenticates against it by construction, so a public edge certificate is not just unnecessary there but meaningless. Kamaji is the strongest single argument for "unify the interface, not the CA": any design that homogenized the certificate authority would break it outright, whereas the consume contract below (a key-free trust anchor, delivered uniformly) is something even Kamaji already satisfies in spirit — the kubeconfig *is* its trust-anchor delivery. It needs no unification work and is excluded from the rollout, but it belongs in the enumeration as the boundary case that fixes the model's shape. ### Platform mechanisms this proposal builds on -- **The CA-only helper.** `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`, from `cozystack/cozystack#2989`) renders a Secret containing only `ca.crt`. It fails closed if the input PEM contains any private-key header, and it always stamps the label `internal.cozystack.io/tenantresource: "true"`. It is covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`. The merged PR frames the situation it addresses as exactly this — a delivery gap: "the cert-manager apps grant tenants no access to those Secrets at all, so a tenant currently cannot obtain `ca.crt`". +- **The CA-only helper.** `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`, from `cozystack/cozystack#2989`) renders a Secret containing only `ca.crt`. It fails closed if the input PEM contains any private-key header, and it stamps two labels: `internal.cozystack.io/tenant-ca`, the selector §5(g) names, so a helper-rendered anchor converges with a projected one, and `internal.cozystack.io/tenantresource`, which is rendered for shape and which the lineage webhook recomputes on admission anyway. It is covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`. That PR frames the situation it addresses as exactly this — a delivery gap: "the cert-manager apps grant tenants no access to those Secrets at all, so a tenant currently cannot obtain `ca.crt`". - **Why the helper is needed but not sufficient.** The CA+leaf chain is rendered per-app by each chart's own cert-manager graph (for example `packages/apps/nats/templates/certmanager.yaml`), and the resulting CA Secret carries `ca.key` — so it is not itself a `ca.crt`-only object and cannot be handed to a tenant. The helper fixes the *output shape* (a key-free `ca.crt` Secret) but takes the CA cert as a Helm value, and nothing in the current architecture can feed it that value for an asynchronously-issued, per-release CA (see "The problem"). - **The tenant projection.** Secrets carrying the `internal.cozystack.io/tenantresource` label are exposed to tenants as the virtual resource `core.cozystack.io/tenantsecrets` (`pkg/registry/core/tenantsecret/rest.go`; the label constant lives in `pkg/apis/core/v1alpha1/tenantresource_types.go`; the RBAC grant is on the virtual resource, not on raw `core/v1` Secrets — `packages/system/cozystack-basics/templates/clusterroles.yaml`). The projection is **label-filtered, not field-filtered**: the entire Secret `Data` is delivered (`secretToTenant` copies `sec.Data` whole). This is the security pivot of the whole model — a labeled Secret must contain only safe material. - **The label's authority is the lineage webhook, not the chart.** The `tenantresource` label is not honored just because something stamped it. The lineage admission webhook (`internal/lineagecontrollerwebhook/webhook.go`) is its authority: it walks a Secret's `ownerReferences` to the owning application and sets `tenantresource` to `true` or `false` from that application's `spec.secrets` selector on **every** admission. A statically-stamped label therefore does not survive on its own — any Secret meant to be tenant-visible must also match a `spec.secrets.include` entry, or the webhook overwrites the label to `false`. This is why the consume contract below marks the CA object through `spec.secrets` (by label), not by stamping the label alone. @@ -55,9 +61,9 @@ And one engine is the proof of the whole framing rather than a mere exclusion. ` The epic's original headline — "consume not mint, cert-manager as the single issuance abstraction" — is not realizable as stated, for two reasons. -First, there is no written contract for the interior. Nothing records, per engine, who owns the PKI and how `ca.crt` reaches the tenant. As a result the per-app TLS pull requests each re-derive the answer, and the answer differs between them — to the point that some now propose projecting a key-bearing Secret to tenants (see Security). +First, there is no written contract for the interior. Nothing records, per engine, who owns the PKI and how `ca.crt` reaches the tenant. As a result each engine's TLS series re-derives the answer, and the answers differ — to the point of projecting a key-bearing Secret to tenants (see Security). -Second, most engines have **no path to deliver a key-free `ca.crt` today**. The cert-manager-minting engines (nats, qdrant, and the four open per-app pull requests) cannot consume the helper because of three compounding constraints that are real and verified — and the operator-owned engines whose operator emits a key-bearing CA (postgres/CNPG, mongodb/PSMDB) hit the same wall, because their CA Secret is also key-bearing and (for CNPG) asynchronous: +Second, most engines have **no path of their own to deliver a key-free `ca.crt`**. The cert-manager-minting engines (nats, qdrant, and the four engines whose series adds the same chain) cannot consume the helper because of three compounding constraints that are real and verified — and the operator-owned engines whose operator emits a key-bearing CA (postgres/CNPG, mongodb/PSMDB) hit the same wall, because their CA Secret is also key-bearing and (for CNPG) asynchronous: - `valuesFrom` is pinned. `expectedValuesFrom()` (`internal/controller/applicationdefinition_helmreconciler.go:99-107`) hardcodes a single `{Kind: Secret, Name: cozystack-values}` reference, and the reconciler overwrites any drift. An application chart cannot add a sideways `valuesFrom` pointing at its own `-ca`. - `lookup` cannot drive it. PR `cozystack/cozystack#1787` moved the global-values channel off `lookup` onto `valuesFrom`; `lookup` itself is still available and is used by several charts to read a pre-existing per-release Secret. But it runs at template-render time and is invisible to the Flux digest, so a chart that reads an asynchronously-created Secret via `lookup` does not re-render when that Secret appears — it would need a manual `helm upgrade`. @@ -78,7 +84,7 @@ So the helper, by itself, closes the *output* shape (a key-free `ca.crt` Secret) - This proposal does **not** force pure-consume on engines that own their PKI; doing so would break CloudNativePG and Strimzi certificate rotation, which are mutually exclusive with an externally-supplied server certificate. - It does **not** homogenize the certificate authority. CA ownership stays the operator's choice — one corporate CA for everything, per-engine self-signed, or a cert-manager issuer are all legitimate. - It does **not** make Cozystack a public/WebPKI certificate authority, and it does not issue certificates for a tenant's own external domain. -- It does **not** redesign the edge, which already merged (`cozystack/cozystack#2988`, `cozystack/cozystack#2989`) or is in flight (`cozystack/cozystack#2990`); it only references it and reframes the top-line goal. +- It does **not** redesign the edge (`cozystack/cozystack#2988`, `cozystack/cozystack#2989`, `cozystack/cozystack#2990`); it only references it and reframes the top-line goal. ## Design @@ -107,39 +113,47 @@ flowchart TB The two tiers are independent. The edge tier answers "what certificate does a public client see when it reaches the platform", and is satisfied by an operator-chosen source (ACME HTTP-01, ACME DNS-01 wildcard, or a bring-your-own wildcard). The interior tier answers "what does a client that connects directly to a managed engine need to trust", and is satisfied by the engine's own operator-owned PKI. The only thing that crosses between them is the *shape* of the trust-anchor object a tenant consumes. -### 2. Edge tier (largely landed) +### 2. Edge tier -The edge is largely done and is recorded here only to fix the framing. An operator selects the certificate source once; the default ingress path serves a supplied wildcard via `--default-ssl-certificate` (`cozystack/cozystack#2988`), and the Gateway path consumes the same Secret via an `existingSecret` mode. Propagating that wildcard to per-tenant termination points (`cozystack/cozystack#2990`) is the one open remainder. The reframed top-line goal applies here: this is "a single interface to choose the source", not "a single issuer for everything". +The edge is recorded here only to fix the framing. An operator selects the certificate source once; the default ingress path serves a supplied wildcard via `--default-ssl-certificate` (`cozystack/cozystack#2988`), and the Gateway path consumes the same Secret via an `existingSecret` mode. Propagating that wildcard to per-tenant termination points is `cozystack/cozystack#2990`. The reframed top-line goal applies here: this is "a single interface to choose the source", not "a single issuer for everything". ### 3. Interior PKI-ownership contract -The contract is a per-engine table. It is the artifact the per-app pull requests must conform to, and it makes the two-tier reality explicit: the operator owns the CA; the platform unifies only how `ca.crt` is consumed. The decisive column is **"self-publishes a key-free `ca.crt`?"** — that, not "mint vs consume", is what determines whether an engine needs a delivery mechanism. The table describes `main` as of 2026-09-02. +The contract is a per-engine table. It is the artifact the per-app pull requests must conform to, and it makes the two-tier reality explicit: the operator owns the CA; the platform unifies only how `ca.crt` is consumed. The decisive column is **"self-publishes a key-free `ca.crt`?"** — that, not "mint vs consume", is what decides whether the controller's extraction is a straight copy or a strip. Every engine in the table declares a sentinel either way, except clickhouse, which serves no TLS, and Kamaji. Filling in the last column takes one more question, the one §5 asks: whether the chart's sentinel names that key-free object, or the key-bearing one beside it. The table is deliberately free of convergence state: it records who owns each engine's CA, whether that object carries a key, and what the controller must therefore do, all of which outlive any particular branch. A series number in a cell is the provenance of a fact, not a status. Which engines have already converged is tracked in `cozystack/cozystack#2814`. -| Engine | PKI owner | CA-bearing Secret | Key in that Secret? | Self-publishes key-free `ca.crt`? | `ca.crt` to tenant? | Operator certificate capability | +| Engine | PKI owner | CA-bearing Secret | Key in that Secret? | Self-publishes key-free `ca.crt`? | Delivery under this contract | Operator certificate capability | | --- | --- | --- | --- | --- | --- | --- | -| kafka (Strimzi) | operator | `-cluster-ca-cert`, `-clients-ca-cert` | no | **yes** | yes — clean reference | `generateCertificateAuthority: false` BYO-CA + per-listener `brokerCertChainAndKey`; public cert on external listener, own-CA mTLS internally | -| redis | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2729` (open, in review) adds a chart cert-manager chain whose CA Secret `.ca-tls` is key-bearing | n/a | **not yet** — under `cozystack/cozystack#2729` the patched operator does publish a key-free `-ca-cert`, but nothing on the platform reads it: the chart's sentinel sources the cert-manager CA instead | no | stock freshworks-oss `redis-operator` v3.3.5 has no TLS support; `cozystack/cozystack#2729` carries it as a downstream patch over v3.3.5 (`packages/system/redis-operator/images/redis-operator/patches/tls.diff`, cut from freshworks-oss/redis-operator#85 commit 2ad2ed8d1), which adds `spec.tls` including `caCertSecretName` | -| postgres (CloudNativePG) | operator | `-ca` (operator-created, async) | **yes — `ca.key`** | no | yes — `.tenant-ca`, alongside the unchanged passwords-only `-credentials` | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | -| mongodb (Percona PSMDB) | operator | `-ca-cert` (operator-created at runtime; the chart renders no TLS objects) | **yes — cert-manager `isCA`** | no | no (gap; name collides with redis, opposite shape) | `spec.tls.issuerConf` BYO issuer; auto-mints a cert-manager chain | -| nats | chart + cert-manager | `-ca` | yes | no | yes, when TLS is on | — | -| qdrant | chart + cert-manager | `-ca` | yes | no | yes, when TLS is on | — | -| mariadb | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2680` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca-tls` is key-bearing | n/a | no | no | mariadb-operator `serverCertSecretRef` / `serverCASecretRef` / `serverCertIssuerRef` | -| opensearch | operator — opster generates its own TLS | none chart-rendered — on `main` the chart sets `security.tls.transport.generate` and `security.tls.http.generate`, so opster mints the material and the chart renders no cert-manager objects; `cozystack/cozystack#2682` (open, in review) adds a chart cert-manager chain whose CA Secret `-http-ca` is key-bearing | n/a | no | no | opster `security.tls.http.secret.name`; transport operator-generated | -| rabbitmq | none — no TLS on `main` | none — the chart renders no TLS objects on `main`; `cozystack/cozystack#2683` (open, in review) adds a chart cert-manager chain whose CA Secret `-ca` is key-bearing | n/a | no | no | cluster-operator `spec.tls.secretName` / `caSecretName` | -| clickhouse (Altinity) | — (BYO-cert) | supplied cert mount | n/a | no | no (not in TLS series yet) | mounts a supplied cert; no native issuance | -| kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | no — out of model | control-plane CA, kubeconfig-pinned, non-swappable | - -The takeaway: **kafka** (native) is the only engine that publishes a key-free `ca.crt` under its own name, and it is still not at the canonical `.tenant-ca` — it alone needs the controller to copy that output across under that name (§5). **Every other engine that ships TLS — postgres, mongodb, nats, qdrant — has a key-bearing CA Secret**, one uniform delivery gap; postgres, nats and qdrant have since closed it by declaring a sentinel, leaving mongodb, and the four whose TLS is still in review (redis, mariadb, rabbitmq, opensearch) all mint a key-bearing CA the same way. **redis** is the one that already closes the gap inside its own pull request: `cozystack/cozystack#2729` declares a `TenantProjection` sentinel naming that key-bearing chart CA, so the anchor lands directly at `.tenant-ca` with the key stripped, and the key-free `-ca-cert` its patched operator also publishes goes unread. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. +| kafka (Strimzi) | operator | `-cluster-ca-cert`, `-clients-ca-cert` | no | **yes** | straight copy — the sentinel names the key-free CA object and there is nothing to strip | `generateCertificateAuthority: false` BYO-CA + per-listener `brokerCertChainAndKey`; public cert on external listener, own-CA mTLS internally | +| redis | chart + cert-manager (`cozystack/cozystack#2729`) | `.ca-tls` | yes | the patched operator writes a key-free `.ca-cert`, which nothing on the platform reads | strip — the sentinel names the chart's key-bearing CA, because it exists as soon as the certificate is issued | stock freshworks-oss `redis-operator` v3.3.5 has no TLS support; `cozystack/cozystack#2729` carries it downstream over v3.3.5 in two halves that have to move together — the operator binary (`packages/system/redis-operator/images/redis-operator/patches/tls.diff`, cut from freshworks-oss/redis-operator#85 commit 2ad2ed8d1) and the CRD schema carrying `spec.tls` and its `x-kubernetes-validations` rule (`packages/system/redis-operator/patches/crd-tls.diff`, alongside `rbac-tls.diff`), without which the field is pruned on admission and the patched binary waits for something it never sees — together giving `spec.tls` including `caCertSecretName` | +| postgres (CloudNativePG) | operator | `-ca` (operator-created, asynchronous) | **yes — `ca.key`** | no | strip — alongside the unchanged passwords-only `-credentials` | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | +| mongodb (Percona PSMDB) | operator | `-ca-cert` (operator-created at runtime), with `ca.crt` carried on the leaf `-ssl` as well | **yes — cert-manager `isCA`** | no | strip — and the sentinel names the leaf `-ssl`, not the CA Secret (§6) | `spec.tls.issuerConf` BYO issuer; auto-mints a cert-manager chain | +| nats | chart + cert-manager | `-ca` | yes | no | strip | — | +| qdrant | chart + cert-manager | `-ca` | yes | no | strip | — | +| mariadb | operator, with an optional chart cert-manager chain under `cozystack/cozystack#2680` | `.ca-tls` on the cert-manager path | yes | **yes** — the operator maintains a key-free `-ca-bundle` under either issuer | straight copy — the sentinel names that bundle, so there is nothing to strip | mariadb-operator `serverCertSecretRef` / `serverCASecretRef` / `serverCertIssuerRef` | +| opensearch | chart + cert-manager (`cozystack/cozystack#2682`), over the transport TLS opster generates for itself | `.http-ca` | yes | no | strip | opster `security.tls.http.secret.name`; transport operator-generated | +| rabbitmq | chart + cert-manager (`cozystack/cozystack#2683`) | `-ca` | yes | no | strip | cluster-operator `spec.tls.secretName` / `caSecretName` | +| clickhouse (Altinity) | — (no server TLS) | none | n/a | no | out of the series — the chart serves no TLS, so no server certificate exists to project from | no native issuance and no server certificate; the chart's only certificate input is `backup.endpointCA`, an outbound trust bundle the backup sidecar uses to verify an S3 endpoint | +| kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | out of model — the kubeconfig is its trust-anchor delivery | control-plane CA, kubeconfig-pinned, non-swappable | + +The takeaway: **kafka and mariadb** are the two engines the contract answers with a straight copy — their operator publishes a key-free `ca.crt` under its own name *and* their sentinel names that object, rather than the key-bearing one beside it. Neither publishes it at the canonical `.tenant-ca`, so the controller still has to write it there (§5). **Every other engine gives the controller a key-bearing Secret to source from**, one uniform delivery gap, whether that CA comes from a chart-rendered cert-manager graph (nats, qdrant, redis, rabbitmq, opensearch) or from the engine's own operator (postgres, mongodb). The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing and operator-created, hence asynchronous, so they need the same key-free projection as the cert-manager-minting engines, which additionally never reach the tenant at all. **redis and mariadb together** show that the split is about which object the platform reads rather than which one exists. Both have an operator that publishes a key-free CA, and their charts decide differently: mariadb's sentinel names the operator's `-ca-bundle`, because it exists under either issuer and carries the previous CA alongside the new one across a rollover, while redis's names the key-bearing cert-manager CA, because that Secret exists as soon as the certificate is issued rather than only after the operator has reconciled — so redis strips and its operator's key-free output goes unread. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. So the contract is one sentence for every engine but clickhouse and Kamaji: converge by declaring a `TenantProjection` sentinel naming the Secret that carries your `ca.crt`, and let the controller publish `.tenant-ca` from it. Which engines have done so is tracked in `cozystack/cozystack#2814`, and is deliberately not recorded here. ### 4. The uniform consume contract -Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `.tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenant-ca: "true"` and made tenant-visible through the lineage webhook's `tenantresource` verdict. +Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `.tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenant-ca: "true"` and made tenant-visible through the lineage webhook's `tenantresource` verdict. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. + +The name took three revisions, and the reason is worth recording because it is the same mistake twice. `-ca-cert` was rejected because Percona PSMDB already creates a Secret of that exact name and puts a private key in it — a collision across **engines**. Its replacement `-tenant-ca` was justified as "claimed by no operator this platform ships", and that was false: it collides across **releases**. For an application `foo` the projection is `postgres-foo-tenant-ca`; for a sibling application `foo-tenant`, CloudNativePG's own CA is `postgres-foo-tenant` + `-ca` — the same string. Both names are legal and several Postgres instances in one namespace is the normal model. Nor is it merely a guard problem: if the projection lands first, CNPG rejects the key-free Secret (`missing ca.key secret data`) and the sibling's PKI never completes, so the sibling application never comes up — silently, blaming a Secret its owner never created. That would be a regression rather than a limitation: the name was unclaimed before this contract existed, so both applications worked, and no guard could refuse it, because the controller that wrote first was legitimate. Only the first of the three was ever published in a proposal; `-tenant-ca` lived entirely inside this document's revision history. No release ever wrote either of them: the extraction controller carried the `.tenant-ca` suffix in the revision that merged. + +The canonical name is therefore `.tenant-ca`, and the dot is what does the work. Two of its legs are machine-enforced, and both are worth naming because they are the reason the dot is available at all. Application names are validated as DNS-1035 labels (`k8svalidation.IsDNS1035Label`, `pkg/apis/apps/validation/validation.go`), so a dot is structurally impossible inside one. A release prefix is pinned to `^[a-z0-9-]*$` on `ApplicationDefinitionRelease` (`api/v1alpha1/applicationdefinitions_types.go`), which excludes the dot by construction. Secret names are DNS-1123 subdomains, where a dot is legal — so `.tenant-ca` is a name the platform can write and no `` the apps API admits can spell. -The name took three revisions, and the reason is worth recording because it is the same mistake twice. `-ca-cert` was rejected because Percona PSMDB already creates a Secret of that exact name and puts a private key in it — a collision across **engines**. Its replacement `-tenant-ca` was justified as "claimed by no operator this platform ships", and that was false: it collides across **releases**. For an application `foo` the projection is `postgres-foo-tenant-ca`; for a sibling application `foo-tenant`, CloudNativePG's own CA is `postgres-foo-tenant` + `-ca` — the same string. Both names are legal and several Postgres instances in one namespace is the normal model. Nor is it merely a guard problem: if the projection lands first, CNPG rejects the key-free Secret (`missing ca.key secret data`) and the sibling's PKI never completes, so the sibling application never comes up — silently, blaming a Secret its owner never created. That is a regression, since before this contract existed the name was unclaimed and both applications worked, and no guard can refuse it, because the controller that wrote first was legitimate. Only the first of the three was ever published — `-ca-cert` is the name the version of this proposal on `main` still carries — while `-tenant-ca` lived entirely inside this document's own revision history, so no release ever shipped it. +Two things are assumed rather than enforced, and both are worth stating. The first is that the release name is dot-free. The apps API makes it so, through the two legs above — but the controller derives the release from a `helm.toolkit.fluxcd.io/name` label value, where a dot is legal, so a hand-written HelmRelease carrying one is outside the guarantee. Under a dot-free release name the arithmetic is narrow: **`` can spell `.tenant-ca` only by supplying that dot itself — that is, only when the suffix ends with `.tenant-ca`**. The second assumption is that no suffix in use does. Nothing validates a suffix; chart authors and upstream operators pick them, and dotted ones are already in use here — under `cozystack/cozystack#2729` the redis chart names its CA Secret `.ca-tls` and its leaf `.tls`, the operator that series patches writes `.ca-cert`, `cozystack/cozystack#2680` gives mariadb the same pair, and `cozystack/cozystack#2682` names the opensearch CA Secret `.http-ca`. None of those ends with `.tenant-ca`, so none reaches the canonical name; but that is a claim about a set this platform keeps growing. -The canonical name is therefore `.tenant-ca`, and it is collision-free by **character class** rather than by survey. Application names are validated as DNS-1035 labels (`[a-z]([-a-z0-9]*[a-z0-9])?`), so a dot is structurally impossible in one; every release prefix is dot-free; and Secret names are DNS-1123 subdomains, where a dot is legal. So `` is dot-free by construction and can never equal a dotted name — for any application name, any operator, and any suffix an operator may add in future. That is a proof. "No operator claims it today" is a guess with an expiry date, and it expired twice. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. +What makes the name safe is that the residue is guarded rather than argued away. A Secret already occupying `.tenant-ca` that this controller did not create is never overwritten and never adopted — §5(f) — so a future suffix that did collide surfaces as a refusal, with a Warning Event on the colliding Secret and a `Ready=False` condition on the sentinel naming it. That is the difference from the two names before it, and it is a difference in failure direction rather than in confidence. `-tenant-ca` had no refusal available at all: a sibling application whose own CA the projection pre-empted would have stopped coming up, silently, and no guard could have refused the write, because the writer that got there first was legitimate. This one fails *closed* in the direction the controller controls: a stranger already at the canonical name is refused, loudly. The other direction is not symmetric and should not be claimed as such — if the projection is written first and some future operator suffix then wants that same name, what happens is that operator's behaviour, not this controller's, and CNPG's rejection of a key-free Secret is what that looks like. What has changed is the probability, not the shape: `-tenant-ca` collided with an operator the platform already ships, and `.tenant-ca` collides only with a suffix ending in `.tenant-ca`, which nobody has written. -This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. +The same reasoning applies one level down, to the source Secrets a chart renders, and the contract should say so rather than leave the next per-engine author to rediscover it. Two per-release Secrets collide across releases exactly when one of their suffixes is a tail of the other: if the suffix set holds both `-x` and `-y-x`, then release `foo`'s `-y-x` object and release `foo-y`'s `-x` object are the same string, and both release names are legal. The set to check is every per-release Secret suffix that lands in the namespace, the ones an operator writes included, not only the ones the chart renders. + +**A chart separates a per-release Secret suffix with a dot when, and only when, that pair exists in its set.** The check belongs to the chart author, over the whole set, and this document deliberately does not publish a verdict per engine: an enumeration that misses one operator-written name produces exactly the false clearance this section is warning about. Three worked pairs, because they are documented in the charts themselves. Under `cozystack/cozystack#2729` the redis set holds `-ca-tls` and `-tls`, so hyphenated, application `foo`'s CA Secret and application `foo-ca`'s leaf Secret would both be `redis-foo-ca-tls`; it uses `.ca-tls` and `.tls` for that reason and records it in a comment on the Certificate. Under `cozystack/cozystack#2680` the mariadb chart's cert-manager path renders that same pair and separates it the same way, restating the rule in a comment on its own Certificate. And opensearch is the case that shows why the operator's names count: opster writes its own transport CA at `-ca` unconditionally, and keeping the chart's CA apart within one release is not enough, because across releases the two meet anyway — with a hyphenated `-http-ca`, application `foo`'s chart CA and application `foo-http`'s operator CA would both be `opensearch-foo-http-ca`, which is why `cozystack/cozystack#2682` separates it with a dot. So the dot is neither a platform-wide convention nor a redis-local taste: it is what this contract asks of new work wherever the mechanism above finds the pair, and applying it anywhere else buys nothing. + +This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-cluster-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. ### 5. Delivery: a declared sentinel and one engine-agnostic controller @@ -147,12 +161,12 @@ The contract in §4 fixes the output object; this section fixes the input path The input path splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** — and the platform's standing preference (source from a key-free object where one exists; strip only when forced to) follows from it: -- **Engines whose operator already publishes a key-free CA under its own name.** `kafka` (Strimzi, native) is here today: its CA already lands key-free, under `-clients-ca-cert` rather than the canonical `.tenant-ca`, so the controller still copies it across through a declared projection — a straight copy with nothing to strip. `redis` is **not** in this group, despite appearances: its open pull request does teach the patched operator to publish a key-free `-ca-cert`, but nothing reads that Secret, because the chart's sentinel names the key-bearing CA cert-manager writes instead — so redis belongs with the engines below. -- **Every other engine** — nats, qdrant, redis, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output the platform reads. None can feed the helper at chart-render time (see "The problem"). The projection sources from the key-bearing CA Secret, and the controller strips everything but `ca.crt`. +- **Engines whose operator already publishes a key-free CA under its own name, and whose sentinel is to source from it.** `kafka` (Strimzi) lands its CA key-free under `-cluster-ca-cert` — the CA that signs the broker certificates, which is what a client verifies the endpoint against; the sibling `-clients-ca-cert` signs user certificates for client authentication and is the wrong direction of trust — and converges by naming that object; `mariadb` has the operator's `-ca-bundle`, which exists under either issuer and spans a CA rollover, and converges by naming it (`cozystack/cozystack#2680`). Neither is at the canonical `.tenant-ca`, so the controller copies it across through a declared projection — a straight copy with nothing to strip. Publishing a key-free CA is not by itself enough to put an engine here: `redis` is not, despite appearances, because although the operator patch its series carries does publish a key-free `.ca-cert`, the chart's sentinel names the key-bearing cert-manager CA instead, so nothing reads it and redis belongs with the engines below. +- **Every other engine** — nats, qdrant, redis, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output the platform reads. None can feed the helper at chart-render time (see "The problem"). The sentinel names whichever Secret carries a correct and stable `ca.crt` — usually the engine's CA Secret, and for mongodb the leaf `-ssl` instead, for the reasons in [decision 0002](./decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md) — and the controller strips everything but that key. The alternative to one controller is to change each upstream operator so it emits a key-free CA of its own, the way redis's open pull request adds `caCertSecretName` to the operator it patches. Carrying an equivalent change for CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than a single small controller — and redis is the cautionary case rather than the model, because the key-free output its patch adds goes unread: the chart projects from the CA cert-manager writes, which exists as soon as the certificate is issued rather than only after the operator has reconciled. So the controller is the target for every non-self-publishing engine. -The mechanism is two pieces: a **declaration** the chart renders, and a small **controller** that reconciles it. An earlier revision of this proposal delivered the input path with a *label* on the CA Secret — the owner stamped `publish-ca-cert` and the controller selected sources by that label. An API-owner review of the implementation (`cozystack/cozystack#3299`) found that this emulates object semantics with a label: the thing being declared is "project this Secret's `ca.crt` as a trust anchor", a first-class intent that deserves a first-class object with a name, a status, and its own admission rules — not a boolean smuggled onto someone else's Secret. The mechanism below is that object. +The mechanism is two pieces: a **declaration** the chart renders, and a small **controller** that reconciles it. An earlier revision of this proposal delivered the input path with a *label* on the CA Secret — the owner stamped `publish-ca-cert` and the controller selected sources by that label. Review of that implementation (`cozystack/cozystack#3299`) found that this emulates object semantics with a label: the thing being declared is "project this Secret's `ca.crt` as a trust anchor", a first-class intent that deserves a first-class object with a name, a status, and its own admission rules — not a boolean smuggled onto someone else's Secret. The mechanism below is that object. **(a) The owner declares the projection with a namespaced sentinel.** The app chart renders a small namespaced custom resource — `TenantProjection`, group `internal.cozystack.io` — that names the source Secret explicitly instead of stamping a label on it: @@ -168,34 +182,46 @@ spec: sourceKey: ca.crt # key to lift (defaults to ca.crt) ``` -The API is settled: `TenantProjection` in group `internal.cozystack.io`, version `v1alpha1`, short name `tproj`, namespaced, with `spec.projections[].type` admitting exactly one value, `CACert` (`api/internalapi/v1alpha1/tenantprojection_types.go`; CRD at `packages/system/cozystack-controller/definitions/internal.cozystack.io_tenantprojections.yaml`). The projected Secret lands in the sentinel's own namespace, one per release, alongside the application. The sentinel's own name is free-form — the shipped postgres chart calls it `-ca` after the source it names (`packages/apps/postgres/templates/tenant-projection.yaml`) — because the controller never parses it: it derives the release from the `helm.toolkit.fluxcd.io/name` label Flux stamps on every rendered object (`internal/controller/cacert/reconciler.go:332`), and a sentinel that reaches the cluster without that label publishes nothing and reports `Ready=False, Reason=NoRelease`. +The API is settled: `TenantProjection` in group `internal.cozystack.io`, version `v1alpha1`, short name `tproj`, namespaced, with `spec.projections[].type` admitting exactly one value, `CACert` (`api/internalapi/v1alpha1/tenantprojection_types.go`; CRD at `packages/system/cozystack-controller/definitions/internal.cozystack.io_tenantprojections.yaml`). The projected Secret lands in the sentinel's own namespace, one per release, alongside the application. The sentinel's own name is free-form — the shipped postgres chart calls it `-ca` after the source it names (`packages/apps/postgres/templates/tenant-projection.yaml`) — because the controller never parses it: it derives the release from the `helm.toolkit.fluxcd.io/name` label Flux stamps on every rendered object (`internal/controller/cacert/reconciler.go`), and a sentinel that reaches the cluster without that label publishes nothing and reports `Ready=False, Reason=NoRelease`. -The sentinel is a **declaration**, not a copy of the data: the chart renders it at template time, so it exists in the Flux digest and is reconciled like any other chart object, even though the source Secret it names is created asynchronously by cert-manager or the DB operator long after render. Naming the source explicitly — rather than relying on a name convention or a label the operator must be coaxed into stamping — is what keeps the mechanism engine-agnostic: the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and `-ca-cert` is overloaded (key-free for the patched redis operator, key-bearing for PSMDB), and the declaration accommodates all of them because the chart author who knows the source name simply writes it down. It also dissolves the label design's release-attribution problem: cert-manager runs with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its application and the label leg had to add a separate `-release` label to say which release it belonged to; the sentinel is itself a chart object in the release's namespace, attributed to its release the same way every chart object is (see (c)), so no such marker is needed. +The sentinel is a **declaration**, not a copy of the data: the chart renders it at template time, so it exists in the Flux digest and is reconciled like any other chart object, even though the source Secret it names is created asynchronously by cert-manager or the DB operator long after render. Naming the source explicitly — rather than relying on a name convention or a label the operator must be coaxed into stamping — is what keeps the mechanism engine-agnostic: the CA-bearing Secret names are non-uniform (`-ca`, `.ca-tls`, `.http-ca`, `-ca-cert`) and two of them differ only in a separator — PSMDB's `-ca-cert` is key-bearing while the `.ca-cert` the patched redis operator writes is key-free, and the declaration accommodates all of them because the chart author who knows the source name simply writes it down. It also dissolves the label design's release-attribution problem: cert-manager runs with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its application and the label leg had to add a separate `-release` label to say which release it belonged to; the sentinel is itself a chart object in the release's namespace, attributed to its release the same way every chart object is (see (c)), so no such marker is needed. -**(b) A small extraction controller reconciles the sentinel.** The controller watches `TenantProjection` objects (not Secret labels) and the source Secrets they name. For each `CACert` projection it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. Its watch/upsert shape follows the merged wildcard-secret reconciler (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — ownership and lineage provide those (part (c)). It does four security-load-bearing things and nothing more: +**(b) A small extraction controller reconciles the sentinel.** The controller watches `TenantProjection` objects (not Secret labels), the source Secrets they name, the projections it owns, and the `ApplicationDefinition`s whose `spec.secrets` decide tenant visibility — that last watch is what makes the re-admission in (f) happen when the selectors change rather than at the next resync. For each `CACert` projection it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. Its watch/upsert shape follows the wildcard-secret reconciler (`internal/controller/wildcardsecret/reconciler.go`). It marks its own copies with `internal.cozystack.io/ca-cert-copy` (part (g)) and keeps one narrow removal path, described in part (c); the bulk of the prune logic is unnecessary here because ownership and lineage provide it. Its security-load-bearing work is these four: -- **Read the source the sentinel names, and check its content.** Read only the Secret named in `sourceSecretName`, lift only `sourceKey`, and before writing decode it in a `pem.Decode` loop, parse each block with `x509.ParseCertificate`, and rebuild the projected value only from blocks that validate as a certificate — never copying the input verbatim, so no byte the parser did not accept as a certificate can reach the tenant. That verifies the lifted value is a certificate, not that it is specifically a CA certificate. The sentinel says *which* Secret; the content check says *whether to strip* — which is what lets one controller straight-copy a key-free kafka source and strip a key-bearing PSMDB source without confusing the two, even though the key-free and key-bearing shapes both turn up under `-ca-cert`-adjacent names. +- **Read the source the sentinel names, and check its content.** Read only the Secret named in `sourceSecretName`, lift only `sourceKey`, and before writing decode it in a `pem.Decode` loop and parse every block with `x509.ParseCertificate`. The check is **all-or-nothing, and not a filter**: the whole value is refused on the first block whose type is not `CERTIFICATE`, on the first `x509.ParseCertificate` failure, and on any trailing bytes that are not a PEM block. Nothing is published in any of those cases — the sentinel reports `Ready=False, Reason=SourceRejected` and a Warning Event lands on the source Secret. What is written on success is rebuilt from the parsed certificate DER rather than copied from the input, so a human-readable preamble, or key material tucked between blocks, cannot ride along. That verifies the lifted value is a certificate, not that it is specifically a CA certificate. The sentinel says *which* Secret; the content check refuses the value or passes it and never chooses between copying and stripping — the single-key whitelist below does that unconditionally, which is what lets one controller take a key-free kafka source and a key-bearing PSMDB source down the same path without confusing the two, even though the key-free and key-bearing shapes turn up under names one separator apart. - **Tolerate operator-created, asynchronous sources.** The named source may be created by the CNPG or PSMDB operator (or by cert-manager) after the chart renders. The controller waits on the watch event for that source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. The sentinel exists from render time, so there is always a durable declaration to reconcile against — and to carry status (part (e)) — while the source is still missing. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. -- **Owner-ref the projected Secret to the sentinel.** The projection takes an `ownerReference` to the `TenantProjection` that declared it — not to the application-instance CR, and not to the `HelmRelease` directly. The application-instance CR cannot be referenced: the `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, with a tenant application *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and no UID to point at. The sentinel, by contrast, is a real chart-rendered object in the same namespace, so a real owner-reference is valid and garbage collection is native (part (c)). `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (The projection does not owner-ref the *source* CA Secret: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before reaching the app.) +- **Owner-ref the projected Secret to the sentinel.** The projection takes an `ownerReference` to the `TenantProjection` that declared it — not to the application-instance CR, and not to the `HelmRelease` directly. The application-instance CR cannot be referenced: the `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, with a tenant application *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and no UID to point at. The sentinel, by contrast, is a real chart-rendered object in the same namespace, so a real owner-reference is valid and garbage collection is native (part (c)). `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (The projection does not owner-ref the *source* CA Secret. Some operators do own their output — CloudNativePG owns `-ca`, and the PSMDB operator patches an ownerReference onto `-ssl` after cert-manager creates it unowned, which `hack/e2e-chainsaw/mongodb` asserts — but the chain stops one hop later: it reaches the engine CR, which carries no ownerReference on to the `HelmRelease`, so a walk from the source still dead-ends before the application. The sentinel is attributed by Flux's release label instead, which is what makes the walk in (c) resolve.) -**(c) Attribution, lineage, and garbage collection: the sentinel is a plain chart object.** The sentinel itself carries **no** `ownerReferences`. Flux stamps every object it renders with the `helm.toolkit.fluxcd.io/name` label naming the release, and that label — not an owner-reference — is how the sentinel is attributed to its release, exactly as for any chart-rendered object. So the lineage walk resolves the chain **Secret → sentinel → HelmRelease → application** with no change to the webhook: the projected Secret owner-refs the sentinel, the sentinel's release label names the `HelmRelease`, and the `HelmRelease` backs the application. Garbage collection is native and needs no delete path in the controller: deleting the application deletes the `HelmRelease`, Flux prunes the chart's objects including the sentinel, and Kubernetes then garbage-collects the projection the sentinel owns. Marking the projection tenant-visible is unchanged from §4 — the controller stamps `internal.cozystack.io/tenant-ca: "true"` on it, one generic `ApplicationDefinition.spec.secrets.include` entry (`matchLabels: {internal.cozystack.io/tenant-ca: "true"}`) matches it by label (`internal/lineagecontrollerwebhook/matcher.go`), and the lineage webhook then walks the projection's owner chain and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). +**(c) Attribution, lineage, and garbage collection: the sentinel is a plain chart object.** The sentinel itself carries **no** `ownerReferences`. Flux stamps every object it renders with the `helm.toolkit.fluxcd.io/name` label naming the release, and that label — not an owner-reference — is how the sentinel is attributed to its release, exactly as for any chart-rendered object. So the lineage walk resolves the chain **Secret → sentinel → HelmRelease → application** with no change to the webhook: the projected Secret owner-refs the sentinel, the sentinel's release label names the `HelmRelease`, and the `HelmRelease` backs the application. Garbage collection is native for the case that matters: deleting the application deletes the `HelmRelease`, Flux prunes the chart's objects including the sentinel, and Kubernetes then garbage-collects the projection the sentinel owns. One case native collection cannot reach, and for which the controller keeps a removal path: a sentinel that goes on existing while its `CACert` entry is gone owns a projection nothing will collect, and it would serve a stale trust anchor indefinitely — so the controller withdraws that projection, and only one it owns. Marking the projection tenant-visible is unchanged from §4 — the controller stamps `internal.cozystack.io/tenant-ca: "true"` on it, one generic `ApplicationDefinition.spec.secrets.include` entry (`matchLabels: {internal.cozystack.io/tenant-ca: "true"}`) matches it by label (`internal/lineagecontrollerwebhook/matcher.go`), and the lineage webhook then walks the projection's owner chain and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). **(d) Admission: RBAC is the control, not a ValidatingAdmissionPolicy.** No tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot create, update, or delete a `TenantProjection` to aim the controller at a Secret of its choosing — the only way a sentinel enters the cluster is by being rendered into a release. The extraction controller holds `get`/`list`/`watch` on `tenantprojections` and `get`/`update`/`patch` on `tenantprojections/status` — enough to read a declaration and write the `Ready` condition part (e) needs, and nothing beyond it (`packages/system/cozystack-controller/templates/rbac.yaml`). It never creates or deletes a sentinel, since that stays the chart's (helm-controller's) job. **(e) A broken declaration is observable.** The sentinel carries `status.conditions`, and the controller sets `Ready`. When a declaration cannot be satisfied — the named source Secret does not exist, or an upstream bump renames the CA Secret out from under `sourceSecretName` — the controller records `Ready=False, Reason=SourceNotFound` on the sentinel. This is a concrete advantage over the label design, which had no object to carry status: there, a source that never appeared could only make the controller requeue silently, invisible until someone noticed the trust anchor was missing. With the sentinel the failure is a queryable condition on a named object — alertable, and pointing at the exact broken release. -**(f) Revocation is part of the mechanism, and the canonical name is guarded.** The `tenantresource` verdict in (c) has to be able to go back to `false`, and that does not happen for free. Each projection records a digest of the owning `ApplicationDefinition`'s `spec.secrets` in `internal.cozystack.io/ca-cert-selectors`, and the digest is part of the drift check, so a change to those selectors forces exactly one write; that write also drops the `internal.cozystack.io/managed-by-cozystack` marker, which is what hands the projection back to the lineage webhook — the webhook's `objectSelector` skips any object already carrying the marker, so forcing a write without dropping it would leave the verdict frozen at whatever the selectors said on create. Both halves together are what lets a definition that stops selecting the anchor actually take the tenant's read access away, instead of leaving it reading a trust anchor the platform withdrew. The canonical name carries its own guards: a Secret already sitting at `.tenant-ca` that holds private key material is refused loudly rather than overwritten or adopted (`Ready=False, Reason=CanonicalNameOccupied`, plus a Warning Event on the Secret), and a release whose anchor is declared more than once publishes nothing — `MultipleCACertProjections` when a single sentinel declares two `CACert` entries, `ReleaseContested` when two sentinels in one namespace claim the same release. All of these live in `internal/controller/cacert/reconciler.go`. +**(f) Revocation is part of the mechanism, and the canonical name is guarded.** The `tenantresource` verdict in (c) has to be able to go back to `false`, and that does not happen for free. Each projection records a digest of the owning `ApplicationDefinition`'s `spec.secrets` in `internal.cozystack.io/ca-cert-selectors`, and the digest is part of the drift check, so a change to those selectors forces exactly one write; that write also drops the `internal.cozystack.io/managed-by-cozystack` marker, which is what hands the projection back to the lineage webhook — the webhook's `objectSelector` skips any object already carrying the marker, so forcing a write without dropping it would leave the verdict frozen at whatever the selectors said on create. Both halves together are what lets a definition that stops selecting the anchor actually take the tenant's read access away, instead of leaving it reading a trust anchor the platform withdrew. The canonical name carries its own guards, and they are two distinct refusals. A Secret at `.tenant-ca` that this controller did not create is never overwritten and never adopted: adoption keys on the projection's **owner reference** back to a `TenantProjection` of the sentinel's name, not on any label — a label is something any writer in the namespace can forge or strip, and keying adoption on one would let an actor who strips it make the controller disown its own object and keep serving whatever bytes that actor wrote. A stranger at the name is refused as `Ready=False, Reason=ProjectionCollision`, with a Warning Event on the colliding Secret. Separately, when the sentinel's own declared source *is* the canonical name and carries private key material, the answer is `Ready=False, Reason=CanonicalNameOccupied` and a Warning Event on that Secret. There is a third outcome that is neither refusal: when the declared source sits at the canonical name and is key-free, it already *is* the trust anchor, so the controller writes nothing and reports `Ready=True, Reason=Projected` — and if that Secret departs from the shape the name promises, it emits a `CACertCanonicalNameContract` Warning Event rather than silently implying it checked out. It is the only path on which the object a tenant reads at that name is not one the controller wrote. A release whose anchor is declared more than once publishes no new anchor, and one an earlier uncontested reconcile already published is left in place rather than withdrawn, because taking a trust anchor away is worse than leaving a stale one visible beside the condition that explains it — `MultipleCACertProjections` when a single sentinel declares two `CACert` entries, `ReleaseContested` when two sentinels in one namespace claim the same release. All of these live in `internal/controller/cacert/reconciler.go`. + +**(g) Which label is written by whom, on the projection.** Five labels appear below; four are stamped on the projection and the last is Flux's, on the sentinel. An anchor a chart renders through `cozy-lib.tls.caCertSecret` instead of through this controller carries its two selector labels from the helper — the table is about the projected object, not about every route to a trust anchor. Two of the five both read as ownership markers and are easy to reach for interchangeably: `internal.cozystack.io/ca-cert-copy`, the controller's own marker, and `internal.cozystack.io/managed-by-cozystack`, the webhook's. Making that mistake would be a real bug in one specific direction, though the shipped controller does not: the lineage webhook is registered with `objectSelector` `managed-by-cozystack` `DoesNotExist`, so a projection created already carrying that marker never reaches the webhook — it is admitted, the webhook simply skips it — and therefore never receives a `tenantresource` verdict, and no tenant ever reads the trust anchor. Neither name is what adoption keys on — see (f). + +| Label | Written by | What it means | +| --- | --- | --- | +| `internal.cozystack.io/tenant-ca` | the extraction controller | this Secret is a trust anchor; it is the single engine-agnostic selector an `ApplicationDefinition.spec.secrets.include` entry matches | +| `internal.cozystack.io/ca-cert-copy` | the extraction controller | this projection was controller-created. It is part of the drift check, and it is **not** what adoption keys on — that is the owner reference (see (f)) | +| `internal.cozystack.io/tenantresource` | the lineage admission webhook | the authoritative verdict on whether a tenant may read the Secret, recomputed from the owning application's `spec.secrets` on every admission | +| `internal.cozystack.io/managed-by-cozystack` | the lineage admission webhook | the webhook's own re-admission marker. The controller never writes it; it only deletes it, on a selectors-digest change, to hand the projection back for a fresh verdict (see (f)) | +| `helm.toolkit.fluxcd.io/name` | Flux | the release the sentinel belongs to, and the string the canonical projection name is built from | + +The controller also writes two annotations on the projection: `internal.cozystack.io/ca-cert-source`, recording the `/` it read, and `internal.cozystack.io/ca-cert-selectors`, the selectors digest (f) turns into a re-admission. -What this *reuses* rather than rebuilds: Flux's release-name label and its native pruning (for attribution and garbage collection), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), and the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`). The irreducibly new work is the `TenantProjection` sentinel and the extraction step that reconciles it — read one key from the named source, write a key-free copy, re-copy on rotation, report readiness — the same job an operator does natively on the one engine that already ships a key-free CA object (kafka's `-clients-ca-cert`). Kafka still declares a sentinel, because its CA lands under an engine-specific name rather than the canonical one; for it alone the extraction step is a straight copy, with nothing to strip. +What this *reuses* rather than rebuilds: Flux's release-name label and its native pruning (for attribution and garbage collection), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), and the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`). The irreducibly new work is the `TenantProjection` sentinel and the extraction step that reconciles it — read one key from the named source, write a key-free copy, re-copy on rotation, report readiness — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-cluster-ca-cert`). Kafka still declares a sentinel, because its CA lands under an engine-specific name rather than the canonical one; for it and for mariadb the extraction step is a straight copy, with nothing to strip. ### 6. Per-engine application order -`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-clients-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) is a stripping case rather than a straight copy: its chart mints a cert-manager chain and declares a sentinel naming the key-bearing `.ca-tls` that chain produces, so the controller lifts `ca.crt` and drops the key, and the key-free `-ca-cert` the patched operator also publishes goes unread. `nats` and `qdrant` have since converged the same way. `mongodb`, then rabbitmq, mariadb and opensearch, converge by declaring a `TenantProjection` sentinel that names their CA Secret. `kubernetes` (Kamaji) is explicitly out. +`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-cluster-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) is a stripping case rather than a straight copy: its chart mints a cert-manager chain and declares a sentinel naming the key-bearing `.ca-tls` that chain produces, so the controller lifts `ca.crt` and drops the key, and the key-free `.ca-cert` the patched operator also publishes goes unread. `nats` and `qdrant` converge the same way, over the `-ca` their own cert-manager graph renders. `mongodb` is the case where the sentinel names no CA Secret at all: it names the **leaf** Secret `-ssl` with `sourceKey: ca.crt`, for the three reasons in [decision 0002](./decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md): it is the name the chart itself sets in `spec.secrets.ssl`, it is what the operator's self-signed fallback writes when cert-manager is absent, and during a CA rotation the operator merges the old and the new CA into its `ca.crt`. PSMDB's `-ca-cert` has none of those three properties. `mariadb` is the other straight copy: like kafka it names a key-free object its operator already maintains, `-ca-bundle`. `rabbitmq` and `opensearch` converge by declaring a sentinel over the key-bearing CA their chart's cert-manager chain produces. `kubernetes` (Kamaji) is explicitly out. This is the intended order, not a progress report; `cozystack/cozystack#2814` tracks how far it has got. ## User-facing changes -A tenant sees one canonical, key-free trust-anchor object per managed application — `.tenant-ca`, carrying only `ca.crt` — through the dashboard resource map and the `tenantsecrets` projection, in the same shape across every engine. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. +A tenant sees one canonical, key-free trust-anchor object per managed application — `.tenant-ca`, carrying only `ca.crt` — through the `tenantsecrets` projection, in the same shape across every engine. It reaches the tenant by label rather than by name: no chart's dashboard resource map grants the anchor, because a name grant would convey whatever occupies that name, where the label path conveys only what the platform has vouched for. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. ## Upgrade and rollback compatibility @@ -205,34 +231,51 @@ This document changes nothing at runtime; it records a target. The pull requests The trust boundary is precise: a tenant receives `ca.crt` and never receives `tls.key` or `ca.key`. The label-filtered, full-object nature of the `tenantsecrets` projection makes this a **preventive invariant** rather than the fix of a live leak: because any labeled Secret is delivered in full, the platform's standing rule must be that no key-bearing Secret is ever labeled tenant-facing — and the way to honor it is to remove the key from the *object* (a separate `ca.crt`-only Secret), not to rely on field filtering at projection time. -No merged engine leaks a private key to a tenant today. The merged charts withhold their key-bearing Secrets, and the tenant-facing objects (postgres `-credentials`, the nats/qdrant credentials Secrets) are passwords-only. The live instance of the risk is not in `main` — it is in the **in-flight per-app PRs**, several of which currently propose labelling a key-bearing Secret to tenants: mariadb (`cozystack/cozystack#2680`) projects the CA private key `-ca-tls`, mongodb (`cozystack/cozystack#2692`) projects the key-bearing `-ca-cert`, and rabbitmq (`cozystack/cozystack#2683`) and opensearch (`cozystack/cozystack#2682`) project the leaf key. This contract exists to stop them landing that way; the consume object and the extraction controller are what let those PRs deliver `ca.crt` without the key. +No engine leaks a private key to a tenant: the charts withhold their key-bearing Secrets, and the tenant-facing objects (postgres `-credentials`, the nats/qdrant credentials Secrets) are passwords-only. The risk is structural rather than historical, and it is why this contract is written down. Every engine's TLS series puts a key-bearing CA Secret in the tenant's own namespace, and the shortest path from there to "the tenant needs `ca.crt`" is to label that Secret — at which point the projection delivers it whole, private key included, because the projection is label-filtered and not field-filtered. So the rule this contract imposes on the series is: **a chart never labels a key-bearing Secret tenant-facing; it declares a `TenantProjection` over that Secret instead**, and the controller publishes the key-free anchor. The consume object and the extraction controller exist so that no per-engine pull request has to reach for the shortcut. Two consequences for the controller. First, it writes at runtime, after chart-render, so it cannot lean on the helper's render-time guard alone: it must whitelist the single `ca.crt` key and re-assert the no-private-key check itself on every write. Second, it adds a new trust surface — read access to per-release CA Secrets, write access for the key-free copy — and must never overwrite a Secret it did not create, surfacing any name collision as a Warning Event rather than failing silently. The declaration that drives it is itself locked down: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a `TenantProjection` aiming the controller at a Secret of their choosing — RBAC alone is the control, with no admission policy narrowing the writer further. ## Failure and edge cases - The helper's input PEM contains a private-key header → render fails closed; the chart does not deploy a key-bearing Secret. -- The source `ca.crt` somehow carries a block that does not parse as an `x509.Certificate` at runtime → the controller refuses to write the copy (the `pem.Decode`/`x509.ParseCertificate` guard rebuilds the projection only from validated blocks); no key-bearing Secret is ever projected. +- The source `ca.crt` carries a block that does not parse as an `x509.Certificate`, a block of some other PEM type, or trailing bytes after the last block → the whole value is refused, nothing is published, and the sentinel reports `Ready=False, Reason=SourceRejected` with a Warning Event on the source. A partly valid value is never partly projected. - The sentinel names a key-free source that should be left alone (a self-publishing engine) → the content check sees no private key to strip and the controller still writes only `ca.crt`; no key is ever exposed. - The named source Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on that source and records `Ready=False, Reason=SourceNotFound` on the sentinel meanwhile; it does not error or busy-loop. - An upstream bump renames the CA Secret out from under `sourceSecretName` → the source never reappears under the declared name, and the sentinel surfaces `Ready=False, Reason=SourceNotFound` — a queryable, alertable condition rather than a silent requeue. -- A foreign Secret already occupies the target name → the controller leaves it untouched (management-label guard) and emits a Warning Event on the application, so the operator sees the collision. +- A foreign Secret already occupies the target name → the controller leaves it untouched, because the Secret carries no owner reference back to the sentinel, and emits a Warning Event on that Secret so the operator sees the collision. - The CA rotates → the controller re-copies `ca.crt` on the next source change; no chart re-render is required. -- The application is deleted → Flux prunes the sentinel with the rest of the chart's objects, and Kubernetes garbage-collects the projected `.tenant-ca` via its owner reference to the sentinel; the controller needs no delete path. +- The application is deleted → Flux prunes the sentinel with the rest of the chart's objects, and Kubernetes garbage-collects the projected `.tenant-ca` via its owner reference to the sentinel. +- The sentinel survives but stops declaring its `CACert` entry → nothing is pruned and nothing is collected, so the controller withdraws the projection it published rather than leave a stale trust anchor behind; a foreign Secret at that name, or another sentinel's projection, is left untouched. + +The sentinel's `Ready` condition carries exactly these reasons, and they are the whole set an operator can meet: + +- `Projected` — every declared projection is published. +- `NoRelease` — the sentinel reached the cluster without Flux's `helm.toolkit.fluxcd.io/name` label, so no release, and therefore no canonical name, can be derived from it. +- `SourceInvalid` — the entry declares an empty `sourceSecretName`. The CRD pins a minimum length, so this is only reachable for an object admitted by an older CRD. +- `SourceNotFound` — the named source Secret does not exist yet. +- `SourceNotReady` — the source exists but does not carry the named key, or carries it empty; operators routinely create the Secret before populating it. +- `SourceRejected` — the lifted value carries private key material, or fails the certificate gate. +- `ProjectionCollision` — a Secret this controller did not create occupies the canonical name. +- `ProjectionTerminating` — the previous projection is still being garbage-collected, so writing to it would land and then be thrown away. +- `CanonicalNameOccupied` — the declared source is itself at the canonical name and carries private key material. +- `MultipleCACertProjections` — one sentinel declares more than one `CACert` entry. +- `ReleaseContested` — two sentinels in one namespace claim the same release. +- `UnsupportedProjectionType` — the entry's type is not `CACert`. Unreachable while the CRD enum admits only that value, and kept as the explicit answer if the enum is widened before the code that handles a new type lands. ## Testing - The helper is already covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`, including the fail-closed assertions. -- The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a sentinel whose named source appears after it (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), a sentinel whose source never appears (asserting `Ready=False, Reason=SourceNotFound` on `status.conditions`), and owner-reference-driven garbage collection of the projection when the sentinel is pruned. -- Each per-app pull request adds helm-unittest fixtures asserting the chart renders the `TenantProjection` sentinel with the right `sourceSecretName`, and asserting the projected `.tenant-ca` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- The extraction controller's suite is plain Go, one `func Test…` per case over a fake client in `internal/controller/cacert/reconciler_test.go`, the same shape as `internal/controller/wildcardsecret/reconciler_test.go`, with no envtest and no Ginkgo in either. It covers the source that has not appeared yet and the source that exists without the key; both refusals of the content gate, including a value that is certificate armour around something else and a valid certificate followed by trailing bytes; the foreign Secret at the canonical name and the forged-marker variant of it; two sentinels contesting one release, and two `CACert` entries in one sentinel; the stale-owner re-home after an application is deleted and recreated under the same name; and the selectors-digest write that hands the projection back to the lineage webhook. +- Two properties are out of that suite's reach by construction, because a fake client runs neither a garbage collector nor an admission chain: the webhook's `tenantresource` verdict, and owner-reference-driven collection of the projection. The verdict is asserted in the live end-to-end fixtures (`hack/e2e-chainsaw/postgres`, `hack/e2e-chainsaw/mongodb`), which is the only place it can be — they read the anchor back through `core.cozystack.io/tenantsecrets` and assert the key-bearing source is not served that way. Collection is asserted nowhere: the fixtures let chainsaw tear the application down during cleanup, so no step deletes it and then checks that `.tenant-ca` is gone. Rotation is asserted in half. The re-copy branch is driven by the tamper-heal case, where an existing projection holding foreign bytes is rewritten from the source on a single reconcile; what no case does is mutate a source between two reconciles, so the watch-to-write path a real rotation actually takes is never exercised end to end. Nor does anything close the loop the anchor exists for: of the engines that have converged, only postgres and mongodb assert the projection end to end at all — qdrant's fixture does not touch it and nats has none — and neither of those two completes a handshake validated against the projected `ca.crt`. +- Each per-engine pull request adds helm-unittest fixtures asserting the chart renders the `TenantProjection` sentinel with the right `sourceSecretName` (`packages/apps/*/tests/tenant_projection_test.yaml`) — the projection itself is written at runtime, so no chart-render test can see it — plus an end-to-end check that a tenant can read `ca.crt` and cannot read any object carrying `tls.key`. - No dedicated admission fixture is needed: RBAC is the control, and no tenant-facing ClusterRole grants any verb on `internal.cozystack.io`, so a tenant cannot write a `TenantProjection` by any path. ## Rollout -1. Edge — `cozystack/cozystack#2988`, `cozystack/cozystack#2989`, and the per-tenant propagation `cozystack/cozystack#2990` are all merged. -2. This contract — accepted. -3. Sentinel CRD and extraction controller — implemented and tested. -4. Per-app convergence — postgres (tracked by `cozystack/cozystack#2814`), nats and qdrant merged; the remaining per-app TLS pull requests onto the contract. +1. Edge — the wildcard on the default ingress path, the CA-only helper, and per-tenant propagation (`cozystack/cozystack#2988`, `cozystack/cozystack#2989`, `cozystack/cozystack#2990`). +2. This contract. +3. The `TenantProjection` CRD and the extraction controller that reconciles it. +4. Per-engine convergence, in the order §6 gives, tracked by `cozystack/cozystack#2814`. ## Open questions @@ -246,10 +289,10 @@ Two consequences for the controller. First, it writes at runtime, after chart-re - **Changing each upstream operator to emit a key-free CA.** Rejected as the general mechanism, and retired even for the engine that pioneered it. Redis was going to carry a forked operator; its pull request now carries a downstream patch over freshworks-oss `redis-operator` v3.3.5 instead, and the key-free Secret that patch produces goes unread, because the chart projects from the CA cert-manager writes. Carrying an equivalent change for CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator would be far more surface to own than one engine-agnostic controller. - **cert-manager `trust-manager`.** Considered and rejected as the mechanism, recorded as the validating prior art. trust-manager is the canonical "watch a CA source, materialize a key-free copy, never touch `tls.key`" operator and confirms the copy-model is the mainstream choice for trust-anchor distribution. But it does not fit operationally: its `Bundle`/`ClusterBundle` is cluster-scoped and fans out to namespaces by selector (not one object per release); it reads its sources from its own trust namespace, not from an arbitrary per-release tenant namespace; and it targets ConfigMaps first (Secret targets are opt-in). Modelling per-release, intra-namespace extraction on it would mean one cluster-scoped Bundle per database release reading a source it cannot natively see — against the grain. The small intra-namespace controller is a better fit. - **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `.tenant-ca` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. -- **A source-selection label on the CA Secret (the earlier design of this proposal, superseded).** The owner stamped `internal.cozystack.io/publish-ca-cert` on its CA Secret and the controller selected sources by that label. It was adopted first, then replaced by the declared `TenantProjection` sentinel of §5 after an API-owner review (`cozystack/cozystack#3299`) found it emulates object semantics: a label carries no name, no status, and no admission surface of its own, so a broken declaration could only requeue silently, and the release a cert-manager Secret belonged to had to be supplied by a second `-release` label because the Secret carried no ownerReference. The declared sentinel is the chosen mechanism — a first-class namespaced object with `status.conditions`, RBAC-restricted writes, and a source named **by name** rather than by label. Declaring the source by name also closes this proposal's former open question "Labelling operator-created CA sources": CNPG and PSMDB need not label their operator-created output at all, because the chart names that Secret in the sentinel's `sourceSecretName`. The implementation briefly carried a second discovery leg alongside the label — a `spec.caCert` field on the cluster-scoped, per-kind `ApplicationDefinition` — with the mirror-image flaw: a cluster-scoped object standing in for a per-namespace, per-release fact. Both legs collapse into the single namespaced sentinel. -- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's declared sentinel plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; the sentinel names the source explicitly instead, and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. -- **A field filter on the projection itself (the principled root fix — deferred to its own proposal).** Today `tenantsecrets` delivers the whole Secret `Data`, which is the only reason a key-free *copy* must be materialized at all. A per-key field filter on the projection (or the `spec.secrets` selector) would let a single `ca.crt` key be projected straight out of a key-bearing Secret — no controller, no second object — and would generalize to the operator-owned engines. It is **not** a local change, which is why it is deferred rather than adopted here: the projection is **writable** at the registry level (`pkg/registry/core/tenantsecret/rest.go` implements Create/Update/Patch/Delete), and the write path (`tenantToSecret`) replaces the underlying Secret's `Data` **wholesale** (`out.Data = ts.Data`). A read-side key filter without a matching write-side filter would, the moment any principal with write access used the filtered view, silently drop the keys it could not see. Tenant roles grant only `get/list/watch` today, but that read-only posture lives in a different package (`packages/system/cozystack-basics` clusterroles), so a field filter's safety is entangled with a write path and an RBAC posture defined elsewhere. That makes it a redesign of the tenant-secret API with its own blast-radius analysis — a separate design proposal — not a rider on the TLS rollout. Recorded here as the future simplification the extraction controller could later retire, explicitly **not** a dependency of this proposal. -- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `.tenant-ca` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. +- **A source-selection label on the CA Secret (the earlier design of this proposal, superseded).** The owner stamped `internal.cozystack.io/publish-ca-cert` on its CA Secret and the controller selected sources by that label. It was adopted first, then replaced by the declared `TenantProjection` sentinel of §5 after review of `cozystack/cozystack#3299` found it emulates object semantics: a label carries no name, no status, and no admission surface of its own, so a broken declaration could only requeue silently, and the release a cert-manager Secret belonged to had to be supplied by a second `-release` label because the Secret carried no ownerReference. The declared sentinel is the chosen mechanism — a first-class namespaced object with `status.conditions`, RBAC-restricted writes, and a source named **by name** rather than by label. Declaring the source by name also closes this proposal's former open question "Labelling operator-created CA sources": CNPG and PSMDB need not label their operator-created output at all, because the chart names that Secret in the sentinel's `sourceSecretName`. The implementation briefly carried a second discovery leg alongside the label — a `spec.caCert` field on the cluster-scoped, per-kind `ApplicationDefinition` — with the mirror-image flaw: a cluster-scoped object standing in for a per-namespace, per-release fact. Both legs collapse into the single namespaced sentinel. +- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's declared sentinel plus lineage reuse: the CA Secret names are non-uniform and two of them differ only in a separator (`-ca-cert` key-bearing, `.ca-cert` key-free), so a name convention would mis-handle at least one; the sentinel names the source explicitly instead, and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. +- **A field filter on the projection itself (the principled root fix — deferred to its own proposal).** `tenantsecrets` delivers the whole Secret `Data`, which is the only reason a key-free *copy* must be materialized at all. A per-key field filter on the projection (or the `spec.secrets` selector) would let a single `ca.crt` key be projected straight out of a key-bearing Secret — no controller, no second object — and would generalize to the operator-owned engines. It is **not** a local change, which is why it is deferred rather than adopted here: the projection is **writable** at the registry level (`pkg/registry/core/tenantsecret/rest.go` implements Create/Update/Patch/Delete), and the write path (`tenantToSecret`) replaces the underlying Secret's `Data` **wholesale** (`out.Data = ts.Data`). A read-side key filter without a matching write-side filter would, the moment any principal with write access used the filtered view, silently drop the keys it could not see. Tenant roles grant only `get/list/watch` today, but that read-only posture lives in a different package (`packages/system/cozystack-basics` clusterroles), so a field filter's safety is entangled with a write path and an RBAC posture defined elsewhere. That makes it a redesign of the tenant-secret API with its own blast-radius analysis — a separate design proposal — not a rider on the TLS rollout. Recorded here as the future simplification the extraction controller could later retire, explicitly **not** a dependency of this proposal. +- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `.tenant-ca` as a named Secret object through `tenantsecrets`, which the dashboard surfaces. Different consumer model; a copy object is what an object-by-name consumer needs. - **A general-purpose cluster-secret replication operator / a copy-issuer webhook.** Rejected during the edge work (`cozystack/cozystack#2990`, `cozystack/cozystack#2812`) in favor of native references and a purpose-built reconciler with a tight ownership guard that move only the Secret name, never key material. --- From c8b15a4ec08358253642863e4c018af0a73a7c2e Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 4 Sep 2026 00:48:53 +0300 Subject: [PATCH 4/6] docs(unified-tls-pki): add the decision records this revision owes The repository convention asks for a decision record when an accepted design changes course, and this revision changes it twice. 0001 records that the trust anchor is declared by a namespaced TenantProjection sentinel and published as .tenant-ca. It was decided in the rework of cozystack/cozystack#3299 that merged as cozystack/cozystack#3407, with #3408 and #3411 completing it. The record states the argument against the label mechanism in its own voice, lists the rejected alternatives with where each was argued, and names the two assumptions the dotted name rests on together with the mitigation: a Secret already sitting at the canonical name is refused, never adopted. 0002 records that mongodb sources the anchor from its leaf Secret rather than from the operator's CA Secret. The three reasons are verified against the pinned PSMDB operator v1.22.0: the self-signed fallback never writes the CA Secret, and a CA rotation merges old and new CA into the leaf's ca.crt while the CA Secret carries only the current one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...or-is-declared-by-a-namespaced-sentinel.md | 43 +++++++++++++++++++ ...s-the-trust-anchor-from-the-leaf-secret.md | 39 +++++++++++++++++ 2 files changed, 82 insertions(+) create mode 100644 design-proposals/unified-tls-pki/decisions/0001-the-trust-anchor-is-declared-by-a-namespaced-sentinel.md create mode 100644 design-proposals/unified-tls-pki/decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md diff --git a/design-proposals/unified-tls-pki/decisions/0001-the-trust-anchor-is-declared-by-a-namespaced-sentinel.md b/design-proposals/unified-tls-pki/decisions/0001-the-trust-anchor-is-declared-by-a-namespaced-sentinel.md new file mode 100644 index 0000000..d51f446 --- /dev/null +++ b/design-proposals/unified-tls-pki/decisions/0001-the-trust-anchor-is-declared-by-a-namespaced-sentinel.md @@ -0,0 +1,43 @@ +# 0001. The trust anchor is declared by a namespaced `TenantProjection` and published as `.tenant-ca` + +- **Number:** `0001` +- **Date:** `2026-07-22` +- **Status:** Accepted +- **Deciders:** `@lexfrei, @lllamnyp` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** the rework of [`cozystack/cozystack#3299`](https://github.com/cozystack/cozystack/pull/3299) that merged as [`cozystack/cozystack#3407`](https://github.com/cozystack/cozystack/pull/3407), recorded in the proposal by [`cozystack/community#36`](https://github.com/cozystack/community/pull/36) +- **Implemented in:** [`cozystack/cozystack#3407`](https://github.com/cozystack/cozystack/pull/3407), [`cozystack/cozystack#3408`](https://github.com/cozystack/cozystack/pull/3408), and [`cozystack/cozystack#3411`](https://github.com/cozystack/cozystack/pull/3411), which added the release-prefix pattern the Consequences below rely on + +## Context + +The accepted proposal delivered the input path with a **label**: an application chart stamped `internal.cozystack.io/publish-ca-cert` on its CA Secret, a controller selected sources by that label, and the key-free copy it wrote was named `-ca-cert`. Both halves of that — the selection mechanism and the object name — were settled on paper and neither survived the implementation in [#3299](https://github.com/cozystack/cozystack/pull/3299). + +The label could not carry attribution. cert-manager runs on this platform with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret holds no ownerReference back to the application that asked for it, and the controller had no way to tell which release a labelled Secret belonged to. The design answered that with a second label carrying the release name, which is where it stopped being a label and started being an object with the fields spelled as strings. It also had nowhere to put a status: a declaration whose source never appears could only requeue, silently, until somebody noticed the trust anchor was missing. An intermediate revision moved selection onto a `spec.caCert` field of the cluster-scoped `ApplicationDefinition` and inherited the mirror-image flaw — a cluster-scoped, per-kind object standing in for a per-namespace, per-release fact. + +The name failed twice, and each failure was found by an operator that already claimed it. `-ca-cert` collides across **engines**: Percona PSMDB creates a Secret of exactly that name and puts a private key in it. Its replacement `-tenant-ca` was justified as claimed by no operator the platform ships, and collides across **releases**: for an application `foo` the projection is `postgres-foo-tenant-ca`, and for a sibling application `foo-tenant` CloudNativePG's own CA Secret is `postgres-foo-tenant` plus `-ca` — the same string. That collision has a direction no guard can refuse: if the projection is written first, CNPG rejects the key-free Secret with `missing ca.key secret data`, the sibling's PKI never completes, and that application never starts, blaming a Secret its owner never created. The controller that wrote first was within its rights, so there is nothing to refuse. + +The shape of the first problem is a category error rather than a missing fix: what the chart is declaring — project this Secret's `ca.crt` as a trust anchor — is a first-class intent, and emulating an object with a label pays for it in every direction at once. The rework that followed was split out of that branch and merged as [#3407](https://github.com/cozystack/cozystack/pull/3407), which @lllamnyp approved as the API owner. + +## Decision + +The chart declares the trust anchor by rendering a namespaced `TenantProjection` (group `internal.cozystack.io`, version `v1alpha1`, short name `tproj`) that names the source Secret by name in `sourceSecretName` and the key to lift in `sourceKey`; the controller derives the release from the `helm.toolkit.fluxcd.io/name` label Flux stamps on the sentinel, publishes a key-free Secret named `.tenant-ca` carrying only `ca.crt`, owner-references it to the sentinel, and reports the outcome as a `Ready` condition on the sentinel. + +## Why not the alternatives + +- **The source-selection label, which is what the proposal originally said** ([#3299](https://github.com/cozystack/cozystack/pull/3299)). A label has no name to reference, no status to carry, and no admission surface of its own, so a broken declaration is invisible; and because cert-manager Secrets carry no ownerReference under `enableCertificateOwnerRef: false`, it needed a second label to answer a question the sentinel answers by being a chart object in the release's namespace. +- **A `spec.caCert` field on `ApplicationDefinition`** (an intermediate revision of the same pull request). Cluster-scoped and per-kind, standing in for a fact that is per-namespace and per-release. +- **`-ca-cert` as the canonical name.** Claimed by Percona PSMDB, with a private key in it — the source and the target would collide on the engine that most needs the projection. The argument that retired the name is recorded in [§4 of the proposal](../README.md). +- **`-tenant-ca` as the canonical name.** Collides with CloudNativePG's own CA across sibling releases, in the one direction that cannot be guarded; it would have stopped an application that worked before this contract existed from coming up at all. The sibling-release arithmetic is worked through in the same section. +- **Owner-referencing the projection to the application instance.** Not possible rather than unwise: the `apps.cozystack.io` kinds are virtual, an application is stored as a HelmRelease whose `spec.values` is the application spec, so there is no object in etcd and no UID to reference; the storage shape this rests on is described in [§5 of the proposal](../README.md). +- **Owner-referencing the projection to the HelmRelease directly.** Workable, and rejected for lifetime rather than for lineage: the lineage walk resolves either chain through unmodified code, but the sentinel is the object whose existence *is* the declaration, so pruning the chart collects the projection natively (`internal/controller/cacert/reconciler.go`, "Why the owner is the sentinel, not the HelmRelease"). It does not remove the delete path outright: `withdrawProjection` stays for the one case garbage collection never sees, a sentinel that goes on existing with its `CACert` entry removed. + +## Consequences + +- A broken declaration is now queryable. `kubectl get tproj` shows a source that never appeared, a source renamed out from under the sentinel, a value that failed the certificate gate, and a release two sentinels are contesting. +- Every converging chart renders one more object, and the platform owns a CRD it did not have. +- Nothing had to be migrated. [#3407](https://github.com/cozystack/cozystack/pull/3407) carried the `.tenant-ca` suffix in the revision that merged, so no release ever wrote a projected `-ca-cert` or `-tenant-ca` into a cluster. +- The dot in the canonical name rests on two assumptions the platform does not enforce. First, that the release name is dot-free: the apps API guarantees it through the release prefix pattern and the DNS-1035 application name, but the controller derives the release from a `helm.toolkit.fluxcd.io/name` label value, where a dot is legal, so a hand-written HelmRelease is outside the guarantee. Under a dot-free release name a suffix reaches the canonical name only by ending with `.tenant-ca` — which is the second assumption, that none in use does; the dotted names already in use under [#2729](https://github.com/cozystack/cozystack/pull/2729) end otherwise and are no threat to it. The residue is covered by refusing, rather than adopting, a Secret already sitting at the canonical name. + +## Revisit if + +A validating rule constrains the suffixes charts and operators may append to per-release Secret names, which would retire the second assumption; or the `tenantsecrets` projection gains a per-key field filter, which would retire the copied object and the controller with it. diff --git a/design-proposals/unified-tls-pki/decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md b/design-proposals/unified-tls-pki/decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md new file mode 100644 index 0000000..7c17bd1 --- /dev/null +++ b/design-proposals/unified-tls-pki/decisions/0002-mongodb-sources-the-trust-anchor-from-the-leaf-secret.md @@ -0,0 +1,39 @@ +# 0002. mongodb sources the trust anchor from its leaf Secret, not from the operator's CA Secret + +- **Number:** `0002` +- **Date:** `2026-09-01` +- **Status:** Accepted +- **Deciders:** `@Arsolitt, @lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/cozystack#2692`](https://github.com/cozystack/cozystack/pull/2692), recorded in the proposal by [`cozystack/community#36`](https://github.com/cozystack/community/pull/36) +- **Implemented in:** [`cozystack/cozystack#2692`](https://github.com/cozystack/cozystack/pull/2692) + +## Context + +The contract said the sentinel names the engine's CA-bearing Secret and the controller strips everything but `ca.crt`, and for the engines converged at the time that was unambiguous: nats and qdrant have the CA their chart's cert-manager graph renders, and postgres has the one CloudNativePG creates. Each had exactly one object worth naming. + +mongodb had two. The Percona PSMDB operator mints a cert-manager chain at runtime and writes both a key-bearing CA Secret at `-ca-cert` and a leaf Secret at `-ssl` that carries `ca.crt` alongside its own key pair. Reading the contract literally pointed at `-ca-cert`, and three properties of that name argued against it, none of which is visible in the resulting template without the comment that now sits above it. + +## Decision + +The mongodb chart's `TenantProjection` names the leaf Secret `{{ .Release.Name }}-ssl` with `sourceKey: ca.crt`, ungated — the PSMDB operator issues the chain unconditionally, so the source exists for every release. + +## Why not the alternatives + +These three are the chart's own stated reasons, recorded in the comment above [`packages/apps/mongodb/templates/tenant-projection.yaml`](https://github.com/cozystack/cozystack/blob/main/packages/apps/mongodb/templates/tenant-projection.yaml) as it merged in [#2692](https://github.com/cozystack/cozystack/pull/2692). The second and third are claims about PSMDB's behaviour, and both hold at the operator version the platform pins, `v1.22.0` (`packages/system/mongodb-operator/charts/psmdb-operator/Chart.yaml`). In the operator's own source, `createSSLManually` (`pkg/controller/perconaservermongodb/ssl.go`) writes the leaf and the internal Secret and nothing else, so the self-signed fallback never produces `-ca-cert`; that name belongs to the cert-manager path alone, reached through `applyCertManagerCertificates` and `CertificateCA` (`pkg/psmdb/tls/certificate.go`). `mergeNewCA` (same `ssl.go`) merges the old and the new CA into the leaf's `ca.crt` with `MergePEM` (`pkg/psmdb/tls/pem.go`) and leaves `-ca-cert` untouched. + +**Why not `-ca-cert`, the operator's CA Secret:** + +- **It is not a name the chart controls.** `-ssl` is the name the chart itself sets in the PSMDB CR's `spec.secrets.ssl`, so chart and sentinel move together; `-ca-cert` is the operator's internal choice and can change under an operator bump with nothing in the chart to notice. +- **It does not exist on every path that produces TLS.** When cert-manager is absent the operator falls back to its own self-signed material and writes the leaf, but no cert-manager CA Secret — so a sentinel over `-ca-cert` would sit at `Ready=False, Reason=SourceNotFound` for that entire configuration while TLS is working. +- **It carries the wrong bundle during a rotation.** The operator merges the old and the new CA into the leaf's `ca.crt`, so the leaf is the object that stays verifiable across a CA rotation; the CA Secret carries only the current CA, and a client that fetched the anchor mid-rotation would hold half of it. + +## Consequences + +- The contract's sentence about sourcing is narrower than it read. The sentinel names whichever Secret carries a correct and stable `ca.crt`; the engine's CA Secret is the usual answer, not the definition. +- Nothing about the security boundary changes. The leaf is key-bearing too — it holds `tls.key` — so the controller's strip is doing the same work it does everywhere else, and the guard sees the same shape. +- The next per-engine author has three properties to check rather than a name to copy, and this record is where the checklist lives. Reaching for the CA Secret by reflex is the failure mode it exists to prevent. + +## Revisit if + +PSMDB stops merging the old and new CA into the leaf's `ca.crt` during a rotation, or its self-signed fallback starts writing a CA Secret of its own — either removes one of the three reasons, and the second one removes the strongest. From ac5fa3a77d3ce4c2227a98166c344f4e5892e98b Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 4 Sep 2026 00:48:53 +0300 Subject: [PATCH 5/6] docs(external-database-exposure): record why exposure is the native Service The proposal's orchestration story pointed at a layer that reconciled expose entries into ServiceExposure objects. That API group was removed in favour of native Service type: LoadBalancer with loadBalancerClass, on the ground that managed-application charts own their Service, so the document named an integration point the platform does not have. Write the decision down where its alternatives can be weighed once: what the removal settled, the three shapes it turned down and where each was argued, and what follows for a design that wanted an object in between. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...sure-is-the-native-loadbalancer-service.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md diff --git a/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md new file mode 100644 index 0000000..6d2e604 --- /dev/null +++ b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md @@ -0,0 +1,37 @@ +# 0001. External exposure is the native LoadBalancer Service, not a Cozystack exposure API + +- **Number:** `0001` +- **Date:** `2026-07-16` +- **Status:** Accepted +- **Deciders:** `@kvaps, @lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** the argument in [`cozystack/cozystack#3164`](https://github.com/cozystack/cozystack/issues/3164), which `cozystack/cozystack#3218` names as its rationale, settled by merging [`cozystack/cozystack#3218`](https://github.com/cozystack/cozystack/pull/3218) +- **Implemented in:** [`cozystack/cozystack#3218`](https://github.com/cozystack/cozystack/pull/3218) + +## Context + +Two tenant-facing exposure surfaces were in play at once, one landed and one still pending. The pending one was the structured `expose` proposal ([`cozystack/community#29`](https://github.com/cozystack/community/pull/29)), open at the time and set to replace the chart-level `external` boolean with an additive `expose` list layered on `ServiceExposure` — pendency that [`cozystack/cozystack#3164`](https://github.com/cozystack/cozystack/issues/3164) gives as the reason to settle the question before the proposal extended the exposure half rather than after. The implementation had landed ahead of it: `network.cozystack.io/v1alpha1`, a cluster-scoped `ExposureClass` and a namespaced `ServiceExposure` reconciled by a controller in `cozystack-controller`, merged in [`cozystack/cozystack#3081`](https://github.com/cozystack/cozystack/pull/3081) on 2026-06-30. + +@lllamnyp filed that issue the next day and later approved the removal; the objection in it is what forced the call, and the removal followed two weeks after the filing. The vendor-neutral LoadBalancer selection the new group offered is a native Kubernetes field, `Service.spec.loadBalancerClass`. And two API groups over one domain — `network.cozystack.io` for how an application is reachable, beside `sdn.cozystack.io` for who may reach it — is a fault unless the split is written down, which for the exposure surface it never was, the group having skipped the design process the policy surface went through. + +This proposal was written against that group while it existed. Its section 5 handed both the tenant-facing trigger and the per-release `TLSRoute` to the layer that reconciled `expose` entries into `ServiceExposure` objects. + +## Decision + +External exposure is the native Kubernetes primitive, with no Cozystack object in front of it. `cozystack/cozystack#3218` removed the `network.cozystack.io` group; the host ingress, its only consumer, went on rendering `type: LoadBalancer` and gained an optional `publishing.loadBalancerClass` to pick the LoadBalancer controller, leaving the `externalIPs` node-IP default path unchanged. The ground given in the removal is that managed-application charts own their Service, so native `type: LoadBalancer` plus `loadBalancerClass` and an admin-provisioned address pool cover external exposure without a dedicated API group and a controller to reconcile it. It reached the v1.6.0 release candidates but not the release: `docs/changelogs/v1.6.0.md` records it as introduced and removed inside one cycle and not part of v1.6.0. + +## Why not the alternatives + +- **Keep `ServiceExposure` as the object charts point at instead of rendering a Service.** This is the kind `cozystack/cozystack#3164` puts its open question to — what it provides that `Service.spec.loadBalancerClass` plus an admin-provisioned pool per class does not — and `cozystack/cozystack#3218` answers it: managed-application charts already render their own Service, so the indirection served one consumer, the host ingress, with no second one in prospect. +- **Keep `ExposureClass` and remove only `ServiceExposure`.** `cozystack/cozystack#3164` grants that this half is defensible on its own, an admin-owned named config object on the StorageClass analogy, and aims its open question elsewhere. The removal took both because the class had no reader left: every reader of the kind lived inside the `serviceexposure` controller that went with it, `Service.spec.loadBalancerClass` names the LoadBalancer controller directly, and the pool behind a class is provisioned by an administrator rather than by an object. +- **Leave both kinds in place and write the missing justification for two API groups.** `cozystack/cozystack#3164`'s second concern is that two tenant-facing networking groups over one domain is a fault unless the split is written down, and that the exposure surface skipped the design process the policy surface went through. A justification can defend a split that buys something; what was left after the first bullet did not. + +## Consequences + +- Nothing mediates between a chart and its external endpoint, so a design that wants an object in between has to earn it rather than assume it. This proposal's section 5 does not ask for one: the per-release `TLSRoute` is rendered by the release's own chart, the shape `packages/apps/harbor/templates/httproute.yaml` uses for an `HTTPRoute` and `packages/system/cozystack-api/templates/api-tlsroute.yaml` for a `TLSRoute`, and the shared engine listener is a `TenantGateway` field specified in [`cozystack/cozystack#3342`](https://github.com/cozystack/cozystack/pull/3342). That split is derived from this decision in the proposal, not argued in `cozystack/cozystack#3218` — the removal thread does not discuss Gateway routing at all. +- The class stops short of the databases. `publishing.loadBalancerClass` reaches charts as `_cluster.load-balancer-class`, and `packages/extra/ingress/templates/nginx-ingress.yaml` is what reads it, so a managed database's LoadBalancer Service carries no class and lands on the cluster's default LoadBalancer implementation. +- The migration cost fell only on clusters tracking `main` that had set `publishing.exposureClass`: provision an address pool, switch to `publishing.loadBalancerClass`, delete any orphaned `cozystack-` pool by hand. The v1.6.0 release itself never carried the kinds. + +## Revisit if + +A successor to the structured `expose` model lands and needs an object of its own. The question to answer then is what that object does beyond selecting a class and naming a pool — which is what `cozystack/cozystack#3164` asked, and what the removed shape had no answer for. From 48ed9e822f8fd25e48fb2934dda2c542b350ff37 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 4 Sep 2026 00:48:53 +0300 Subject: [PATCH 6/6] docs(external-database-exposure): orchestrate exposure on the objects that exist Section 5 handed the trigger and the per-release route to a layer that reconciled expose entries into ServiceExposure objects. Reground it on what the tenant Gateway offers: one tls- passthrough listener per tlsPassthroughListeners entry, nothing routed until a TLSRoute attaches by sectionName and names a backend, and the release's own chart rendering that route from values it already reads. Native LoadBalancer Services remain the path for the engines that cannot be SNI-routed. Claims the same reading falsifies go with it: the listeners declare no route kinds rather than restricting them to TLSRoute, declaring an entry withdraws termination from every name its SNI covers, route attachment is confined to the Gateway's own namespace, the listener cap fails the whole render rather than one listener, and the shared listener does not disappear with an engine's last release. The flat-hostname phase gains the second gate that rule puts on it, in each place that describes the phase. Who declares the per-engine entry, the app chart or the tenant chart, is stated as an open question with the cost on each side rather than answered. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 94 +++++++++++-------- 1 file changed, 57 insertions(+), 37 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 6c1a6a2..c39d828 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -3,7 +3,7 @@ - **Title:** `External database exposure via Gateway API TLS-passthrough (SNI) and end-to-end TLS` - **Author(s):** `@lexfrei` -- **Date:** `2026-06-24`; revised `2026-09-02` +- **Date:** `2026-06-24`; revised `2026-09-03` - **Status:** Accepted ## Overview @@ -12,17 +12,21 @@ Today every managed database a tenant exposes externally gets its own `LoadBalan The design ships on the Cilium the platform runs today: passthrough listeners use distinct per-engine subdomain hostnames (`*..`), which avoid the listener-hostname overlap that Cilium isolates correctly only from 1.20 onward. The flat `*.` scheme — shorter `.` connection hostnames — is recorded as the refinement to adopt once the platform is on Cilium 1.20, and the SAN plan below makes that adoption additive rather than client-breaking. -This is the design-proposal artifact required by `cozystack/cozystack#2816`, and it records the decision that issue asks for. The trade-off that issue frames is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. +This is the design-proposal artifact `cozystack/cozystack#2816` (closed) required, and it records the decision that issue asked for. The trade-off it framed is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (closed, PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract. The engines reach that point by one path: the extraction controller reads each engine's CA source and writes the canonical `.tenant-ca` — copying with nothing to strip from redis and kafka's own key-free CA object, and stripping the key from postgres and mongodb's key-bearing Secret — rather than any engine publishing that canonical name itself. Either way the external client verifies against the one canonical `.tenant-ca`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. -- **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. -- **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion proposal under the same epic, and both are on `main`. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract. The engines reach that point by one path: the extraction controller reads each engine's CA source and writes the canonical `.tenant-ca` — a straight copy with nothing to strip where the operator already maintains a key-free CA object for the chart's sentinel to name (kafka, mariadb), and a strip from the key-bearing Secret every other engine has — rather than any engine publishing that canonical name itself. redis is on the stripping side despite appearances: the operator it patches does publish a key-free object, but the chart's sentinel names the key-bearing cert-manager CA instead, so redis strips exactly like postgres and mongodb. Either way the external client verifies against the one canonical `.tenant-ca`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Related:** the structured `expose` model — it would replace the chart-level `external` boolean with a structured, additive `expose` list, and lists Gateway/SNI consolidation as the forward-compatible future path. None of it has landed: the proposal (`cozystack/community#29`) was closed, and the `ExposureClass` / `ServiceExposure` implementation (`cozystack/cozystack#3081`) was removed again by `cozystack/cozystack#3218`, which replaced it with the native `Service` field `loadBalancerClass` surfaced as `publishing.loadBalancerClass`. Treat every reference to `expose` below as a design this proposal composes with if it returns, not as a platform mechanism. Nothing here waits on it: the trigger is a chart value on the database beside `external`, and the per-release route is rendered by the database's own chart against the tenant Gateway (§5), the shape `packages/apps/harbor` already uses for its `HTTPRoute`. A structured exposure model returning later would select this mode in place of that chart value; it would not move the route. +- **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, closed, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). All repository paths below refer to the `cozystack/cozystack` repository. +## Decisions + +- [0001. External exposure is the native LoadBalancer Service, not a Cozystack exposure API](./decisions/0001-external-exposure-is-the-native-loadbalancer-service.md) — why the removed `ExposureClass` / `ServiceExposure` layer is not the orchestration point. + ## Context ### TLS-passthrough already exists @@ -118,7 +122,7 @@ flowchart TB CA -.->|"verifies"| C3 ``` -A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a per-engine wildcard hostname: `tls-postgres` on 5432 with hostname `*.postgres.`, `tls-redis` on 6379 with `*.redis.`, `tls-mongos` on 27017 with `*.mongo.` — each `mode: Passthrough`, `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: [".."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. (Restricting the new listeners to `TLSRoute` is safe where the 443 listeners could not be: the `cilium/cilium#45559` same-port kind-divergence collapse applies per port, and each engine listener is alone on its port.) +A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a per-engine wildcard hostname: `tls-postgres` on 5432 with hostname `*.postgres.`, `tls-redis` on 6379 with `*.redis.`, `tls-mongos` on 27017 with `*.mongo.` — each `mode: Passthrough`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: [".."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. (These listeners declare no route kinds at all, where the 443 listeners declare both kinds together; the two settings answer different constraints of the same Cilium. On the pinned version a listener that spells its kinds out has them applied to every route on the Gateway rather than to the listener that declared them, so naming `TLSRoute` here would reject every `HTTPRoute` the tenant publishes. What an unset field then admits is the implementation's call rather than the spec's, and `cozystack/cozystack#3342` states both halves: Gateway API leaves the protocol-to-kind mapping to the implementation and its conventional table pairs `TLS` with `TCPRoute` as well, so what leaves `TLSRoute` as the whole set here is the pinned Cilium implementing neither `TCPRoute` nor `UDPRoute`.) This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it. @@ -172,41 +176,54 @@ The ceiling, stated honestly: ### 5. API and controller extension -The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. - Two objects implement the model, at different cardinalities: one shared listener per engine type, and one `TLSRoute` per database release. -The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility: +The split between them follows the platform's own convention rather than introducing one: the tenant Gateway's listeners are rendered by the `TenantGateway` controller, and a release's route is rendered by that release's own chart. Listener synthesis belongs to the controller because `Gateway.spec.listeners` has exactly one writer by design — `packages/extra/gateway` renders the `TenantGateway` CR and deliberately renders neither the `Gateway` nor its `Certificate` objects, which is what keeps Helm and the controller from racing on that array. Route rendering belongs to the chart because a chart needs no discovery beyond values it already reads: `_namespace.gateway` names the namespace of the nearest ancestor tenant that owns a Gateway and `_namespace.host` the apex the hostname is built from, both arriving in the release's `cozystack-values` Secret, while the Gateway in that namespace is named `cozystack` for every tenant. Two charts write that Secret and a database chart sees no difference between them: `packages/apps/tenant/templates/namespace.yaml` renders it for every tenant it creates and skips `tenant-root` by name, and `packages/system/cozystack-basics/templates/cozystack-values-secret.yaml` is the sole source for `tenant-root` itself, where `gateway` appears only once `_cluster.gateway-enabled` is set. `packages/apps/harbor/templates/httproute.yaml` is exactly that with an `HTTPRoute`, and `packages/system/cozystack-api/templates/api-tlsroute.yaml` is the same attachment with a `TLSRoute` naming its listener by `sectionName` — reading the namespace from `_cluster.expose-ingress` instead, being a platform component rather than a tenant app. A database chart is that second shape with a tenant app's discovery: `_namespace.gateway` for the namespace, `sectionName: tls-` for the listener. + +The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak for this design — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. `cozystack/cozystack#3342` specifies one structured field alongside it. The two passthrough forms stay independent and neither replaces the other, but the older field does not come through unchanged: the same specification bounds it at 62 entries of at most 249 characters each, refuses a new entry whose name collides with one of its own, and withdraws the HTTPS-terminate listener from every hostname its listeners answer — a behaviour change that reaches clusters which never declare the new field at all. Everything below is the new field's contract, cited from there rather than restated as a second design; the block abridges it, keeping the field names, types and the four list-level markers verbatim, shortening the comments, and omitting the per-field markers the comments describe: ```go -// TLSPassthroughListener declares one shared passthrough listener for a -// database engine type, on its native port, with a wildcard SNI hostname. -// Every release of that engine attaches its own TLSRoute by SNI hostname. +// TLSPassthroughListener declares one layer-4 TLS-passthrough listener +// on the tenant Gateway: mode Passthrough, on a native port, matched by +// SNI. Every release of that engine attaches its own TLSRoute. // -// Validation contract (enforced by CEL / admission): -// - Name: DNS-1123 label; unique within the list; renders as "tls-". -// - Port: 1..65535; unique within the list; must not collide with a -// synthesized layer-7 listener port (443) or another entry. -// - Hostname: optional; a valid DNS wildcard or exact hostname; defaults -// to "*.." when empty (the per-engine subdomain -// scheme; see §1 for why the flat "*." form is gated -// on Cilium 1.20). +// Validation contract (schema and CEL on the CRD, so a bad entry is +// refused on the write rather than stalling the tenant's reconcile): +// - Name: DNS-1123 label, and the sectionName a TLSRoute attaches +// to. The list is a map keyed by name, so names are unique +// by schema; a name must also differ from every +// TLSPassthroughServices entry, since both forms render a +// "tls-" listener. +// - Port: 1..65535, distinct from every other entry, and neither +// 80 nor 443 — the Gateway's own http and TLS-terminate +// listeners own those. +// - Hostname: an exact hostname or a left-most-label wildcard, on +// Gateway API's own Hostname pattern, and within the +// tenant apex (equal to it or a subdomain of it). type TLSPassthroughListener struct { - Name string `json:"name"` // listener suffix -> "tls-" (e.g. "postgres") - Port int32 `json:"port"` // native port (5432/6379/27017) - Hostname string `json:"hostname,omitempty"` // wildcard SNI match, default "*.." + Name string `json:"name"` // renders listener "tls-" + Port int32 `json:"port"` // native port (5432/6379/27017) + Hostname string `json:"hostname"` // SNI match, e.g. "*.postgres." } -// TLSPassthroughListeners renders one Passthrough listener "tls-" on -// .Port per entry, AllowedRoutes restricted to TLSRoute. Independent of the -// layer-7 TLSPassthroughServices field. +// TLSPassthroughListeners renders one "tls-" Passthrough listener +// per entry, independent of the layer-7 TLSPassthroughServices field. // +optional +// +listType=map +// +listMapKey=name +// +kubebuilder:validation:MaxItems=62 TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"` ``` -The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: [".."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation. +Some rules of that contract have no admission form and fall to the controller instead, so they surface as `Ready=False` on the `TenantGateway` rather than rejecting the write. Hostname overlap is the one that bears on this design: no two passthrough listeners may match the same SNI, counting the port-443 listeners `tlsPassthroughServices` renders and counting wildcards, which is why a listener hostname as broad as `*.` is refused while those defaults are present. It sits in the controller on cost grounds rather than for want of expressiveness — wildcard-aware matching across two lists is a nested CEL scan whose estimated cost pushes the CRD past the apiserver's install-time budget. The per-engine wildcards this design uses stay clear of that rule; the flat `*.` scheme of §1 does not, so that phase is gated on this rule as well as on the Cilium fix. An apex no listener hostname can sit inside is judged the same way, and only while the field is in use: a hostname must be lowercase and within the apex, so an apex carrying upper case — which reaches the `TenantGateway` verbatim from the `namespace.cozystack.io/host` label, normalised by nothing — leaves every entry unsatisfiable. The assembled Gateway's listener total against the Gateway API cap (§3) is judged there too — that one on the finished object rather than on anything declared on a field — as is a repeated entry in the sibling `tlsPassthroughServices` list. One restriction runs the other way, enforced at admission and pinned to the certificate mode rather than to a hostname: the field is refused under `certMode: dns01` or `existingSecret`, where the tenant is served from one wildcard terminate listener that the pinned Cilium cannot keep apart from a passthrough listener under the same apex — so a connection arriving on 443 would reach the database backend. The controller carries its own copy of the admission rules, because the CRD and the controller roll out separately and it cannot assume the spec it reads was checked against the rules it knows. + +The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` names the `cozystack` Gateway in the `_namespace.gateway` namespace with `sectionName: tls-`, `spec.hostnames: [".."]` carries the SNI match, and `spec.rules[].backendRefs` points at the database Service on its native port. Attachment to these listeners is narrower than to the port-443 ones: the contract pins their `allowedRoutes.namespaces` to the `TenantGateway`'s own namespace by the `kubernetes.io/metadata.name` label kube-apiserver writes, instead of selecting on `namespace.cozystack.io/gateway`, which every inheriting child tenant namespace carries. So a release renders a route that attaches when it lives in the tenant that owns the Gateway — the case where `_namespace.gateway` names its own namespace — and a release in an inheriting child tenant is out of reach until that attach set widens. Route and backend in one namespace also means no `ReferenceGrant`; a cross-namespace backend would need one. `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation (the platform's own route is on `v1alpha2`). + +Per entry the field renders one `mode: Passthrough` listener named `tls-` on the declared `Port` with the entry's `Hostname`, alongside the layer-7 terminate listeners. Nothing is routed until a `TLSRoute` attaches by `sectionName: tls-` and names a backend, and because the listener never terminates TLS the backend — not the Gateway — holds the certificate for the listener's hostname. Declaring the entry does one thing past creating the listener, and for this design it is not a detail: the hostname is reserved against termination, so every name the entry's SNI covers loses its HTTPS-terminate listener and the ACME certificate that came with it, and an `HTTPRoute` claiming such a name is told so in its own status. Under per-engine wildcards that is the whole `*..` subtree, for as long as the entry exists — a reason of its own to settle who owns the entry, since the withdrawal outlives the last release. One consequence of the pinned Cilium is worth stating where the port is declared: the whole Gateway translates into a single Envoy listener whose filter chains match on transport protocol and SNI alone, so a handshake carrying the engine's SNI selects that chain on any port the tenant's load balancer forwards, 443 included. The declared port is where the listener is published, not a boundary on where the backend can be reached; the hostname is the access surface, the backend keeps its own authentication, and the SNI is public. + +What the two objects do not settle is who declares the listener entry. The route's owner follows from the convention above — the database's own chart, once per release, gated on the same `external`-adjacent value that selects this mode. The entry has two candidate owners with different costs, and this proposal leaves the choice open rather than answering it (see Open questions). `cozystack/cozystack#3342` exposes no chart value for the field, so an entry there is reachable only by editing the `TenantGateway` directly — enough to validate the listener, not a path for wiring an engine. -The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human. The natural shape of that orchestration is the structured `expose` model of `design-proposals/structured-external-exposure` (community pull request #29): an `expose` entry whose class or scope selects SNI-passthrough is the tenant-facing trigger, and the layer that reconciles `expose` entries into `ServiceExposure` objects is the same layer that renders the per-release `TLSRoute` and reference-counts the shared engine listener. Until that model lands, the interim trigger is the engine's `external`-adjacent toggle, with the same rendering responsibility held by the Tenant / HelmRelease orchestration. Either way, the engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. +Everything off the SNI path keeps the exposure primitive the platform ships. `cozystack/cozystack#3218` removed the `network.cozystack.io` group — `ExposureClass` and `ServiceExposure` — on the ground that managed-application charts own their Service, so native `type: LoadBalancer` with `loadBalancerClass` and an admin-provisioned pool cover external exposure without a dedicated API group and controller ([decision 0001](./decisions/0001-external-exposure-is-the-native-loadbalancer-service.md)). Each engine's chart renders that Service itself under `external: true` (`packages/apps/postgres/templates/external-svc.yaml`, `packages/apps/redis/templates/service.yaml`), and the engines the matrix excludes or defers stay there (§4). The class the removal introduced, `publishing.loadBalancerClass`, reaches charts as `_cluster.load-balancer-class` and is read by the host ingress alone (`packages/extra/ingress/templates/nginx-ingress.yaml`), so a database's LoadBalancer Service carries no class and lands on the cluster's default LoadBalancer implementation. Surfacing the class on database Services is a chart change independent of this proposal. ### 6. Trust-anchor and SAN flow @@ -214,7 +231,7 @@ End to end: the chart injects the external hostnames into the operator-issued ce ## User-facing changes -A database gains an `external`-adjacent toggle to select passthrough/SNI mode (expected to become an `expose` entry once the structured-exposure model lands; see §5). Per-engine connection recipes are documented against the subdomain hostnames: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=.postgres."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt --sni .redis. -h .redis.` (the explicit `--sni` matters: redis-cli does not derive SNI from `-h`; library clients do), `mongosh --tls --tlsCAFile ca.crt --host .mongo.`. Kafka, MariaDB, and non-sharded MongoDB keep today's per-LoadBalancer behavior. +A database gains an `external`-adjacent toggle to select passthrough/SNI mode — a value on the app beside `external`, which its chart also reads to render the route (§5). Per-engine connection recipes are documented against the subdomain hostnames: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=.postgres."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt --sni .redis. -h .redis.` (the explicit `--sni` matters: redis-cli does not derive SNI from `-h`; library clients do), `mongosh --tls --tlsCAFile ca.crt --host .mongo.`. Kafka, MariaDB, and non-sharded MongoDB keep today's per-LoadBalancer behavior. ## Upgrade and rollback compatibility @@ -230,17 +247,20 @@ Exposing a database externally with TLS explicitly off is not silently corrected - A pre-PG17 (client or server) or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). - Any MariaDB/MySQL client dials a passthrough listener → mutual deadlock (client waits for the server greeting, listener waits for a ClientHello) → timeout; prevented by never rendering a MariaDB listener (matrix exclusion). -- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway (until ListenerSet lands with Cilium 1.20). -- A flat `*.` passthrough listener is created on Cilium <1.20 → hostname overlap with the terminate listeners; routing does not isolate correctly. Prevented by the subdomain default; the flat scheme is gated on the platform reaching Cilium 1.20 (Rollout). +- The assembled listener total exceeds the Gateway API cap of 64 → the controller renders no Gateway at all and the tenant Gateway reports `Ready=False`; it is not the one listener over the line that is refused. Because listeners are one per engine type, the cap is reached through child-apex fan-out rather than database fan-out, and the mitigation is to split that subtree onto its own Gateway until `ListenerSet` lifts the cap. +- A flat `*.` passthrough listener is declared → the controller's overlap rule refuses it while the default `tlsPassthroughServices` listeners hold `api.`, and the tenant Gateway reports `Ready=False` instead of rendering; on the pinned Cilium that hostname would in any case overlap the terminate listeners and routing would not isolate correctly. The per-engine subdomain hostnames every entry declares here avoid both, and the flat scheme is a later phase (Rollout). - An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy on the typed kind, evaluated in `cozystack-api`'s admission chain (see Security), not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). -- A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. +- A release selects passthrough mode before its engine has a listener → the chart renders the `TLSRoute`, no `tls-` section exists to attach to, and the route reports that in its status; nothing serves the hostname until the entry is declared. The reverse ordering of the deletion case, and reachable for the same reason: the route follows the release and the entry does not. +- An engine listener is declared for `*..` → every hostname under that subtree stops being terminated by the Gateway, since a passthrough hostname earns no HTTPS-terminate listener and no certificate. Harmless while the subtree holds only database hostnames, which is what the per-engine scheme buys; publishing an HTTP app under `..` is what to avoid. +- A database in an inheriting child tenant selects passthrough mode → its chart renders the `TLSRoute` and the engine listener exists on the ancestor's Gateway, but the route cannot attach: the native-port listeners admit only the namespace that owns the Gateway, while the port-443 listeners admit the whole inheriting subtree. The refusal lands on the route's status, not on the database. +- A database is deleted → its per-release `TLSRoute` goes with the release, because the release's own chart renders it. The engine listener does not: it is an entry on the `TenantGateway`, so deleting the last release of an engine leaves a listener matching that engine's SNI with nothing to forward to. Which layer removes the entry is the open question below. ## Testing - Helm-template assertions that the certificate SAN includes both `..` and `.` per engine, mirroring the existing TLS test fixtures. -- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough`, the `*..` default hostname, and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. -- An admission test that `tls.enabled: false` with `external: true` is rejected by the typed-kind ValidatingAdmissionPolicy on the aggregated path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. +- A controller unit test that a `tlsPassthroughListeners` entry renders a shared listener on the declared port with `mode: Passthrough`, the entry's hostname, and route attachment confined to the Gateway's own namespace, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. +- An admission test that `tls.enabled: false` with `external: true` is rejected by the typed-kind ValidatingAdmissionPolicy on the aggregated path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name and port uniqueness, port range, the reserved 80 and 443, hostname within the apex, and the refusal under `certMode: dns01` or `existingSecret`) is enforced at admission, while hostname overlap, an apex no listener hostname fits inside, a repeated `tlsPassthroughServices` entry and the assembled listener total surface as `Ready=False`. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed. @@ -249,15 +269,15 @@ Exposing a database externally with TLS explicitly off is not silently corrected 1. API field plus controller listener rendering, no engine wired. **Gate:** on the deployed Cilium (1.19.x), stand up a `mode: Passthrough` listener on a native database port (for example 5432) with a per-engine subdomain hostname (`*.postgres.`) alongside the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. The gate validates the subdomain scheme on the version actually deployed; no part of Phase 1 depends on an unreleased Cilium. 2. Redis (the cleanest fit) behind an opt-in; default-on candidate once the gate and library-client SNI behavior are confirmed in practice. 3. PostgreSQL (direct-TLS) and sharded MongoDB, opt-in. -4. Flat-hostname refinement, gated on the platform's Cilium carrying the isolation fix (see Open questions): add `*.` passthrough listeners, extend per-release routes with the flat hostname, re-run the Phase 1 gate against the overlap case (a passthrough listener sharing `*.` with the terminate listeners, exactly one filter chain). Certificates already carry both SANs, so this phase changes no client. +4. Flat-hostname refinement, gated on two things rather than one (see Open questions): the platform's Cilium carrying the isolation fix, and the controller's own overlap rule, which refuses a `*.` listener hostname against the `api.` the default `tlsPassthroughServices` seeds — lifting that is a change to the controller, not a consequence of the bump. Then add `*.` passthrough listeners, extend per-release routes with the flat hostname, re-run the Phase 1 gate against the overlap case (a passthrough listener sharing `*.` with the terminate listeners, exactly one filter chain). Certificates already carry both SANs, so this phase changes no client. Kafka, MariaDB, and non-sharded MongoDB are explicitly out of this rollout. Each phase ships documentation, a connection recipe, and an end-to-end gate. ## Open questions -- **Ship now on subdomains, or wait and ship flat-only** — the flat `*.` hostnames need a Cilium release carrying the isolation fix `cilium/cilium#44889`: 1.20 (final proposed for late July 2026), or a 1.19.x patch if the open v1.19 backport (`cilium/cilium#46826`) merges first. That leaves a genuine fork: implement the subdomain scheme now on 1.19.x and adopt flat later as the additive phase, or wait for the fixed Cilium and ship flat-only from day one — no dual scheme, no per-engine subdomain legacy to carry. Waiting may well be free in calendar terms, because engine wiring is gated on `unified-tls-pki` convergence anyway (Scope), which lands on a similar horizon. This document specifies the subdomain path as the no-wait default; the fork is a scheduling decision to settle at the next sync, not a design blocker. +- **Ship now on subdomains, or wait and ship flat-only** — the flat `*.` hostnames need a Cilium release carrying the isolation fix `cilium/cilium#44889`: 1.20 (final proposed for late July 2026), or a 1.19.x patch if the open v1.19 backport (`cilium/cilium#46826`) merges first. That leaves a genuine fork: implement the subdomain scheme now on 1.19.x and adopt flat later as the additive phase, or ship flat-only from day one — no dual scheme, no per-engine subdomain legacy to carry. The second branch costs more than waiting: the overlap rule refuses a `*.` listener hostname while the default `tlsPassthroughServices` entries hold `api.`, so flat-only also needs that rule relaxed or those defaults emptied, neither of which a Cilium version delivers. Waiting may well be free in calendar terms, because engine wiring is gated on `unified-tls-pki` convergence anyway (Scope), which lands on a similar horizon. This document specifies the subdomain path as the no-wait default; the fork is a scheduling decision to settle at the next sync, not a design blocker. - **Direct-TLS client/server floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct` on either client (libpq PG17+, driver support) or server (PG17+)? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? -- **TLSRoute ownership and the `expose` model** — the natural owner of per-release route rendering is the orchestration layer that reconciles `expose` entries (community pull request #29), which also answers who reference-counts the shared listener down to zero. If this proposal is implemented before `expose` lands, does the interim Tenant / HelmRelease orchestration carry that logic temporarily, or do we sequence this proposal's engine wiring after `expose`? +- **Who declares the `tlsPassthroughListeners` entry for an engine** — the per-release `TLSRoute` belongs to the database's own chart, the shape `packages/apps/harbor` and `packages/system/cozystack-api` already use (§5); the listener entry does not follow from that. Rendered by the app chart, the first exposed release of an engine adds one element to an array on a `TenantGateway` that `packages/extra/gateway` renders in full, including its `tlsPassthroughServices` list — two Helm releases writing one array, which no template can do on its own and which needs a controller or a patch mechanism under it. Rendered by the tenant chart, ownership is clean and the cost moves: a tenant declares an engine's listener before any release of that engine exists, which opens the port and matches the SNI with nothing to forward the stream to. A routeless passthrough listener is not unheard of — a tenant that owns a Gateway but is not the one the platform's own `TLSRoute`s attach to already carries the three default `tlsPassthroughServices` listeners with none behind them — but those sit on 443 among the terminate listeners, and whether the pinned Cilium programs one alone on a native port is not established anywhere: `TenantGateway` readiness folds in every listener's `Accepted` and `Programmed` conditions, so if it declines, the entry holds the tenant's `Ready` down until its first release arrives. A stored entry also pins the tenant's certificate mode, since the CEL rule that refuses the field under `dns01` or `existingSecret` sits on the `spec` node and stops being suppressed by ratcheting the moment anything on that node changes, and the gateway release retries the refused write without end. Whoever owns the array owns removing its last entry, which settles reference-counting with it. - **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling on 1.19.x, and should the controller surface a status condition as the budget nears 64? (ListenerSet dissolves the cap once the platform is on Cilium 1.20.) - **ListenerSet adoption** — Cilium's implementation merged upstream (2026-06-26) and ships in 1.20; Gateway API v1.5.0 carries it in Standard. Do we design this field and its status to be ListenerSet-ready now (listeners as per-namespace objects merged onto a shared Gateway, lifting both the 64 cap and the one-IP-per-tenant ceiling), or revisit after the platform's Cilium bump? This determines whether one-IP-per-tenant is a permanent or temporary ceiling.