From f05d4bd468df552bea3c4af3837fee5ae2428033 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 16 Jul 2026 17:05:40 +0300 Subject: [PATCH 01/14] Add design proposal: Distributed tracing via OTLP and VictoriaTraces Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 design-proposals/distributed-tracing/README.md diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md new file mode 100644 index 0000000..f86c260 --- /dev/null +++ b/design-proposals/distributed-tracing/README.md @@ -0,0 +1,211 @@ + +# Distributed tracing in the Cozystack monitoring stack + +- **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` +- **Author(s):** `@scooby87` +- **Date:** `2026-07-16` +- **Status:** Draft + +## Overview + +Cozystack ships two of the three observability signals out of the box — metrics (VictoriaMetrics) and logs (VictoriaLogs) — but has no supported way to collect distributed **traces**. An operator who wants to see how a request flows through a managed database or messaging cluster, or to correlate a slow span with the logs and metrics it produced, has nothing to turn on. This proposal adds the third signal: an OTLP ingest path, a VictoriaTraces backend that mirrors the existing VictoriaLogs deployment one-for-one, a Grafana traces datasource wired for trace↔logs↔metrics correlation, and a per-application opt-in toggle so a tenant enables tracing on exactly the workloads that need it. + +The design is deliberately conservative: it reuses the multi-tenant topology, the operator-driven provisioning, the Grafana datasource pattern, and the values surface that metrics and logs already established, so tracing lands as "the same thing again for a third signal" rather than a new subsystem with new conventions. The backend choice — VictoriaTraces — falls out of that principle: the `VTCluster`/`VTSingle` CRDs already ship in the victoria-metrics-operator that Cozystack deploys today, so no new operator is introduced. + +## Scope and related proposals + +In scope: a traces backend (platform-wide and per-tenant), an OTLP ingest gateway, a Grafana datasource with correlation, and a per-app opt-in surface. Out of scope: automatic instrumentation of arbitrary tenant workloads, and tracing the internals of Virtual Machines or the Kubernetes control plane (VMs and Kubernetes are not traced by this proposal — only Cozystack-managed applications that can emit OTLP are). + +- **Sibling stack:** the platform monitoring stack `packages/system/monitoring` and the per-tenant stack `packages/extra/monitoring` (wired by `packages/apps/tenant/templates/monitoring.yaml`). This proposal extends both. +- **Collection agents:** `packages/system/monitoring-agents` (fluent-bit, vmagent) — the deployment pattern the OTLP collector follows. +- **Prior art in-repo:** Harbor already exposes an internal trace config (`packages/system/harbor/charts/harbor/values.yaml`, provider `jaeger` or `otel`). It is app-local and not a platform backend; this proposal supersedes ad-hoc per-app trace endpoints with a shared destination. +- **Driver:** requested by a client (hidora) who needs request-level visibility across managed DBaaS and messaging services. + +## Context + +Cozystack's observability is multi-tenant with a central backend. The platform stack runs in `tenant-root` (namespace `cozy-monitoring`) and hosts VictoriaMetrics, VictoriaLogs, Grafana (via grafana-operator), Alerta and vmalert. Tenants either ship signals to the central backend (ExternalName redirects in tenant namespaces point at the root stack; vmagent stamps a `tenant:` external label) or run their own isolated stack from `packages/extra/monitoring`. + +Metrics storage is declared as a list of tiers in values and rendered into VictoriaMetrics CRs — `metricsStorages` in `packages/system/monitoring/values.yaml` defines a `shortterm` (3d) and a `longterm` (14d) tier. Logs storage follows the identical shape: `logsStorages` renders one `VLCluster` per entry in `packages/system/monitoring/templates/vlogs/vlogs.yaml`, with the retention period set on `vlstorage.retentionPeriod`, `managedMetadata` labels for application ownership, a label stamped on the storage PVC claim template so the post-delete cleanup hook can find it, and a load-bearing guard that **fails the render** when the list is empty rather than silently shipping to a non-existent endpoint (the fix for issue `cozystack/cozystack#3181`). + +Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. + +Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. + +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. + +### The problem + +- "A query against my managed Postgres is slow and I can't see where the time goes — which statement, which replica, which downstream call." There is no trace to open. +- "I have a log line for a failed request and a latency spike on a dashboard, but no way to jump from either to the actual request span." The signals don't correlate. +- "My application already emits OTLP spans, but Cozystack gives me nowhere to send them." There is no OTLP endpoint and no backend. + +## Goals + +- Accept traces over **OTLP** (gRPC `4317` and HTTP `4318`), the standard cloud-native tracing protocol. +- Store traces in a **VictoriaTraces** backend with a **configurable retention period, defaulting to 14 days**. +- Provide a **per-application opt-in** `tracing.enabled` toggle; tracing is **off by default** and adds zero overhead until enabled. +- Provision a **Grafana traces datasource** and wire **trace↔logs↔metrics** correlation so an operator can pivot between all three signals. +- Preserve **per-tenant isolation and the central-backend topology** exactly as metrics and logs do today. +- Introduce **no new operator** and no new provisioning convention — reuse VictoriaTraces (already in vm-operator), the storage-list values shape, and the grafana-operator datasource pattern. + +### Non-goals + +- Auto-instrumenting arbitrary tenant workloads. This proposal wires the *transport and storage*; emitting spans is the application's job (native where the engine supports it, sidecar/agent otherwise). +- Tracing Virtual Machines or Kubernetes control-plane internals. +- Changing the existing metrics or logs pipelines. +- Mandating a sampling policy for tenant applications (the platform sets a safe default and exposes a knob). + +## Design + +### 1. Backend: VictoriaTraces (`tracingStorages`) + +Add a `tracingStorages` list to the monitoring values, parallel to `metricsStorages` and `logsStorages`, in both `packages/system/monitoring/values.yaml` and `packages/extra/monitoring/values.yaml`: + +```yaml +tracingStorages: +- name: generic + retentionPeriod: "14d" # configurable; default 14 days per the requirement + storage: 10Gi + storageClassName: "" +``` + +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and — reusing the `cozystack/cozystack#3181` lesson — a guard that **fails the render on an empty list** so a misconfiguration is loud, not a silent black hole for spans. + +```yaml +{{- range .Values.tracingStorages }} +--- +apiVersion: operator.victoriametrics.com/v1 +kind: VTCluster +metadata: + name: {{ .name }} +spec: + managedMetadata: + labels: + apps.cozystack.io/application.kind: Monitoring + apps.cozystack.io/application.name: {{ $.Release.Name }} + vtinsert: { replicaCount: 2 } + vtselect: { replicaCount: 2 } + vtstorage: + retentionPeriod: {{ .retentionPeriod | quote }} + replicaCount: 2 + storage: + volumeClaimTemplate: + metadata: + labels: + apps.cozystack.io/application.name: {{ $.Release.Name }} + spec: + {{- with .storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .storage }} +{{- end }} +``` + +The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today. + +### 2. OTLP ingest via an OpenTelemetry Collector + +VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern: + +- Receivers: `otlp` on gRPC `4317` and HTTP `4318`. +- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. +- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. + +Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces. + +### 3. Grafana datasource and correlation + +Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: + +- **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. +- **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. + +### 4. Per-application opt-in + +Add a `tracing` struct to each participating app's `values.yaml` using the cozyvalues-gen annotation conventions, modelled on foundationdb's `monitoring.enabled` toggle and postgres's `backup` struct: + +```yaml +## @typedef {struct} Tracing - OpenTelemetry (OTLP) tracing configuration. +## @field {bool} enabled - Enable OTLP trace export from this application. +## @field {string} [endpoint] - OTLP collector endpoint. Defaults to the platform collector in cozy-monitoring. +## @field {string} [samplingRatio] - Head-sampling ratio 0.0..1.0. Defaults to the platform policy. + +## @param {Tracing} tracing - OpenTelemetry tracing configuration. +tracing: + enabled: false + endpoint: "" + samplingRatio: "" +``` + +How an app emits spans depends on the engine: + +- **Native OTLP**, wired by chart config: ClickHouse (`opentelemetry_span_log`, currently disabled in `packages/apps/clickhouse/templates/clickhouse.yaml`) and NATS (native OTLP in recent versions). +- **Sidecar/agent OTLP**: Kafka and RabbitMQ (JVM/plugin agents), MariaDB, Redis and Postgres (an OpenTelemetry agent/exporter sidecar). The `tracing.enabled` toggle gates the sidecar and the `OTEL_EXPORTER_OTLP_ENDPOINT` env, defaulting to the platform collector. + +### 5. Data flow + +```mermaid +flowchart LR + app["Managed app
(tracing.enabled)"] -- OTLP 4317/4318 --> col["OpenTelemetry Collector
cozy-monitoring"] + col -- OTLP --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + gr["Grafana"] -- Jaeger query --> vt + gr -. trace_id .-> vl["VictoriaLogs"] + gr -. span metrics .-> vm["VictoriaMetrics"] +``` + +## User-facing changes + +- A new `tracingStorages` block in the monitoring values (system and per-tenant), with `retentionPeriod` defaulting to 14 days. +- A new per-app `tracing.*` block; off by default. +- A Traces datasource and Explore/Traces view in Grafana, with pivot links to logs and metrics. +- A docs entry point (`docs/observability/distributed-tracing.md` in `cozystack/cozystack`) covering how to enable tracing and point an app at the collector. + +## Upgrade and rollback compatibility + +The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An empty/absent `tracingStorages` renders no `VTCluster` (guarded, so it fails loudly only if a downstream component is told to expect one — matching the logs behaviour). No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). + +## Security + +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend. +- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. +- **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. + +## Failure and edge cases + +- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. +- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. +- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app. +- `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. +- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising. + +## Testing + +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured, and that an empty `tracingStorages` fails the render. +- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` Jaeger API and visible in Grafana. +- **Manual**: verify trace→logs and trace→metrics pivots in Grafana. + +## Rollout + +1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet. +2. **Grafana**: traces datasource + correlation links. +3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. +4. **Docs**: enablement guide under `docs/observability/`. + +## Open questions + +- **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. +- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? +- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? +- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit. +- **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? + +## Alternatives considered + +- **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap. +- **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have. +- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly. +- **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise. From bd75a12b80fc0289c513a99804e39ef4c9db3338 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 16 Jul 2026 17:25:42 +0300 Subject: [PATCH 02/14] =?UTF-8?q?Refine=20tracing=20design:=20staged=20A?= =?UTF-8?q?=E2=86=92B=20ingest=20grounded=20in=20current=20architecture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direct-to-vtinsert MVP (mirrors logs) then an opt-in OpenTelemetry Collector gateway; ExternalName redirect via cozystack-basics; poller readiness gate on VTCluster; clarify WorkloadMonitor is operational-only so tracing opt-in lives in app values. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index f86c260..c511e69 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -29,7 +29,7 @@ Metrics storage is declared as a list of tiers in values and rendered into Victo Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. -Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. +Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`. Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. @@ -103,17 +103,21 @@ spec: {{- end }} ``` -The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today. +The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as it does for `VLCluster` today: `waitStrategy: poller` plus a `healthCheckExprs` entry that waits for the CR's `status.updateStatus == 'operational'` (see `packages/extra/monitoring/templates/helmrelease.yaml`). Without the poller gate the release flips Ready as soon as Helm applies the CR — the exact silent-black-hole failure mode that motivated `cozystack/cozystack#3181` for logs. -### 2. OTLP ingest via an OpenTelemetry Collector +### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway) -VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern: +`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**. + +**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. + +**Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318`. - Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. - Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. -Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces. +Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation @@ -149,8 +153,10 @@ How an app emits spans depends on the engine: ```mermaid flowchart LR - app["Managed app
(tracing.enabled)"] -- OTLP 4317/4318 --> col["OpenTelemetry Collector
cozy-monitoring"] - col -- OTLP --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + app["Managed app
(tracing.enabled)"] -- "OTLP 4317/4318
(stable svc endpoint)" --> ext["ExternalName redirect
cozy-monitoring → tenant-root"] + ext -- "Stage A: direct" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + ext -. "Stage B: via gateway" .-> col["OpenTelemetry Collector
(sampling / rate-limit / tenant)"] + col -- OTLP --> vt gr["Grafana"] -- Jaeger query --> vt gr -. trace_id .-> vl["VictoriaLogs"] gr -. span metrics .-> vm["VictoriaMetrics"] @@ -169,7 +175,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Security -- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend. +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. - **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. @@ -177,10 +183,10 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Failure and edge cases - Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. -- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app. +- OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. +- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. -- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising. +- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. ## Testing @@ -190,22 +196,26 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Rollout -1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet. +1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. -4. **Docs**: enablement guide under `docs/observability/`. +4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution. +5. **Docs**: enablement guide under `docs/observability/`. ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. - **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? -- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? -- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit. +- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.) - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? +- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default? + +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered - **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap. - **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have. -- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly. +- **No collector, app → backend directly** (ingest): adopted as **Stage A**, not rejected — `vtinsert` is OTLP-native, so direct ingest is the fastest correct MVP and an exact mirror of the logs path. Its limitations (no shared trust boundary, per-tenant attribution, or central sampling/rate-limiting) are exactly what **Stage B**'s collector gateway adds later, without changing the app-facing endpoint. +- **Collector as a DaemonSet agent** (ingest topology): rejected — that node-local shape fits fluent-bit tailing log files, but OTLP traces are pushed over the network by the apps themselves, so a centralized gateway Deployment (the `vtinsert`/vmagent role) is the right shape. - **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise. From 39e0205a05a29b6e3d12a4e1f02dc41f6244f478 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 17 Jul 2026 10:25:38 +0300 Subject: [PATCH 03/14] Address review: tenant headers, OTLP port/path, tail-sampling affinity, disk retention - Stage A: document AccountID/ProjectID tenant-header injection and the real vtinsert OTLP contract (port 10481, /insert/opentelemetry/v1/traces); ExternalName aliases DNS only. - Stage B: otlphttp exporter sets tenant headers; tail-sampling trace affinity (single replica or loadbalancingexporter routing_key=traceID). - Backend: configurable replicaCount; retentionDiskSpaceUsage vs age retention; durability caveat (replicaCount != data replication); absent-vs-empty contract. - Grafana: span-metrics need a spanmetrics connector; read-side tenant isolation. - Security/Failure/Testing/Open questions updated accordingly. - Drop leftover authoring comment; add application.group managedMetadata label. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 68 ++++++++++++------- 1 file changed, 45 insertions(+), 23 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index c511e69..e9a7138 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -1,4 +1,3 @@ - # Distributed tracing in the Cozystack monitoring stack - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` @@ -64,12 +63,18 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora ```yaml tracingStorages: - name: generic - retentionPeriod: "14d" # configurable; default 14 days per the requirement + retentionPeriod: "14d" # age-based retention; default 14 days per the requirement + retentionDiskSpaceUsage: "" # optional disk-based cap (e.g. "80%" / bytes); see note below storage: 10Gi storageClassName: "" + replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and — reusing the `cozystack/cozystack#3181` lesson — a guard that **fails the render on an empty list** so a misconfiguration is loud, not a silent black hole for spans. +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, configurable replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). + +**Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. + +**Durability note** (raised in review): `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml {{- range .Values.tracingStorages }} @@ -81,13 +86,17 @@ metadata: spec: managedMetadata: labels: + apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - vtinsert: { replicaCount: 2 } - vtselect: { replicaCount: 2 } + vtinsert: { replicaCount: {{ .replicaCount | default 2 }} } + vtselect: { replicaCount: {{ .replicaCount | default 2 }} } vtstorage: retentionPeriod: {{ .retentionPeriod | quote }} - replicaCount: 2 + {{- with .retentionDiskSpaceUsage }} + retentionMaxDiskSpaceUsagePercent: {{ . | quote }} + {{- end }} + replicaCount: {{ .replicaCount | default 2 }} storage: volumeClaimTemplate: metadata: @@ -111,22 +120,30 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. +**OTLP service contract** (raised in review — the exact ports/path matter): an `ExternalName` Service only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and OTLP/gRPC only when `-otlpGRPCListenAddr` is set on the component (conventionally `4317`). So Stage A must be precise: either (a) point apps at `vtinsert`'s real OTLP/HTTP endpoint (`…:10481/insert/opentelemetry/v1/traces`) and enable the gRPC listener on the `VTCluster` if gRPC is wanted, redirecting via a `ClusterIP`/`ExternalName` Service that exposes those ports, or (b) front `vtinsert` with the Stage-B collector, which owns the canonical `4317/4318` OTLP listeners. The earlier "stable `…:4317`" shorthand assumes the collector or an explicit port-mapped Service — plain DNS aliasing alone is not enough. + +**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `otlphttp` exporter `headers:` block — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. + **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. -- Receivers: `otlp` on gRPC `4317` and HTTP `4318`. -- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. -- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. +- Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). +- Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. +- Exporter: `otlphttp` to the `vtinsert` service of the `tracingStorages` backend, with the `headers:` block setting `AccountID`/`ProjectID` from the authenticated tenant identity (not just the resource attribute). + +**Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. -Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. +Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317` once the collector fronts ingest); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. -- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them (raised in review). So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. +**Read-side tenant isolation** (raised in review — the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. + ### 4. Per-application opt-in Add a `tracing` struct to each participating app's `values.yaml` using the cozyvalues-gen annotation conventions, modelled on foundationdb's `monitoring.enabled` toggle and postgres's `backup` struct: @@ -171,28 +188,30 @@ flowchart LR ## Upgrade and rollback compatibility -The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An empty/absent `tracingStorages` renders no `VTCluster` (guarded, so it fails loudly only if a downstream component is told to expect one — matching the logs behaviour). No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). +The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An **absent** `tracingStorages` renders no `VTCluster` and succeeds (tracing stays off); an **explicitly empty** list fails the render per the absent-vs-empty contract. No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). ## Security -- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. -- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. +- **Tenant attribution must be trusted**: `AccountID`/`ProjectID` (and the `tenant` attribute) must be injected from authenticated workload identity, never accepted verbatim from a tenant that could spoof another's IDs. On the shared backend this injection belongs at a boundary the tenant cannot bypass — the per-tenant proxy or the collector — not purely in tenant-controlled app config. +- **Isolation (write and read)**: write-side attribution alone is not isolation. The **read** path must scope each tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so a shared `vtselect` cannot leak spans across tenants; central-backend mode keeps the existing tenant-labelling model, isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. ## Failure and edge cases -- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. +- `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` (or `-retention.maxDiskSpaceUsageBytes`) via `retentionDiskSpaceUsage`, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. +- App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured, and that an empty `tracingStorages` fails the render. -- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` Jaeger API and visible in Grafana. -- **Manual**: verify trace→logs and trace→metrics pivots in Grafana. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskSpaceUsage` maps to the disk-retention field when set. +- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. +- **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. ## Rollout @@ -205,12 +224,15 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. -- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? -- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.) +- **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? +- **Grafana traces datasource type (confirm before implementation)**: the design assumes `type: jaeger` against `vtselect` (VictoriaTraces' Jaeger-compatible query API), with the dedicated VictoriaTraces datasource plugin as the alternative. This is the one load-bearing claim not verifiable against the cozystack tree — the whole trace↔logs↔metrics UX depends on which actually lands, so it must be confirmed against the pinned VictoriaTraces version before building. +- **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) +- **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. +- **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? -- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default? +- **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered From 441b4723c654a16c0316ec9df35417e30e421840 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sat, 18 Jul 2026 21:13:53 +0300 Subject: [PATCH 04/14] =?UTF-8?q?Move=20proposal=20status=20Draft=20?= =?UTF-8?q?=E2=86=92=20Review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index e9a7138..58c6db8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -3,7 +3,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` - **Date:** `2026-07-16` -- **Status:** Draft +- **Status:** Review ## Overview From 65f5f0c7678a30a0ff251aa14dda6cedf2137cf5 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 12:32:46 +0300 Subject: [PATCH 05/14] Address 2nd CodeRabbit pass: percent-only disk cap, stable gRPC contract, headers_setter - retentionDiskUsagePercent is percent-only (maps to retentionMaxDiskSpaceUsagePercent); drop the ambiguous bytes-or-percent field. - Fix the Stage A->B endpoint contradiction: fix the app-facing contract to a port-explicit OTLP/gRPC :4317 Service (bare ExternalName can't carry port/path); gRPC chosen because vtinsert and collector OTLP/HTTP paths differ. - Stage B tenant headers via headers_setter extension (from_context), not static otlphttp.headers which cannot derive per-tenant AccountID/ProjectID. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 58c6db8..4478c2c 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -64,7 +64,7 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora tracingStorages: - name: generic retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskSpaceUsage: "" # optional disk-based cap (e.g. "80%" / bytes); see note below + retentionDiskUsagePercent: "" # optional disk-based cap, percent only (e.g. "80"); see note below storage: 10Gi storageClassName: "" replicaCount: 2 # component scaling, NOT data replication (see durability note) @@ -93,7 +93,7 @@ spec: vtselect: { replicaCount: {{ .replicaCount | default 2 }} } vtstorage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskSpaceUsage }} + {{- with .retentionDiskUsagePercent }} retentionMaxDiskSpaceUsagePercent: {{ . | quote }} {{- end }} replicaCount: {{ .replicaCount | default 2 }} @@ -116,23 +116,23 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as ### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway) -`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**. +`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, behind **one stable app-facing endpoint** — an explicit in-cluster Service on OTLP/gRPC `4317` — so promoting Stage A→B swaps what that Service targets (backend → collector), not the app's configuration. -**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. +**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP/gRPC to the stable Service `otel-traces.cozy-monitoring.svc:4317`, which in Stage A resolves to `vtinsert`'s OTLP/gRPC listener (enabled by setting `-otlpGRPCListenAddr` on the `VTCluster`). Tenant redirection to the central stack reuses the mechanism in `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` (gated on `_cluster.monitoring-enabled` from `monitoring.rootEnabled`) that logs and metrics already use — but as a **port-explicit** Service, not a bare `ExternalName`. -**OTLP service contract** (raised in review — the exact ports/path matter): an `ExternalName` Service only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and OTLP/gRPC only when `-otlpGRPCListenAddr` is set on the component (conventionally `4317`). So Stage A must be precise: either (a) point apps at `vtinsert`'s real OTLP/HTTP endpoint (`…:10481/insert/opentelemetry/v1/traces`) and enable the gRPC listener on the `VTCluster` if gRPC is wanted, redirecting via a `ClusterIP`/`ExternalName` Service that exposes those ports, or (b) front `vtinsert` with the Stage-B collector, which owns the canonical `4317/4318` OTLP listeners. The earlier "stable `…:4317`" shorthand assumes the collector or an explicit port-mapped Service — plain DNS aliasing alone is not enough. +**OTLP service contract** (raised in review — the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `otlphttp` exporter `headers:` block — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). - Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. -- Exporter: `otlphttp` to the `vtinsert` service of the `tracingStorages` backend, with the `headers:` block setting `AccountID`/`ProjectID` from the authenticated tenant identity (not just the resource attribute). +- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** (raised in review) and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. **Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. -Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317` once the collector fronts ingest); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. +Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, promoting Stage A→B changes only what that Service targets (`vtinsert` → collector) — a platform-side change, with no app reconfiguration (this holds for gRPC; OTLP/HTTP is not path-interchangeable, so HTTP users adopt the collector from the start). A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation @@ -170,9 +170,9 @@ How an app emits spans depends on the engine: ```mermaid flowchart LR - app["Managed app
(tracing.enabled)"] -- "OTLP 4317/4318
(stable svc endpoint)" --> ext["ExternalName redirect
cozy-monitoring → tenant-root"] - ext -- "Stage A: direct" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] - ext -. "Stage B: via gateway" .-> col["OpenTelemetry Collector
(sampling / rate-limit / tenant)"] + app["Managed app
(tracing.enabled)"] -- "OTLP/gRPC :4317
(stable Service)" --> svc["otel-traces Service
cozy-monitoring (port-explicit)"] + svc -- "Stage A: → vtinsert gRPC" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + svc -. "Stage B: → collector" .-> col["OpenTelemetry Collector
(sampling / rate-limit / AccountID+ProjectID)"] col -- OTLP --> vt gr["Grafana"] -- Jaeger query --> vt gr -. trace_id .-> vl["VictoriaLogs"] @@ -202,14 +202,14 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` (or `-retention.maxDiskSpaceUsageBytes`) via `retentionDiskSpaceUsage`, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` via the percent-only `retentionDiskUsagePercent` values field (upstream also offers a mutually-exclusive `-retention.maxDiskSpaceUsageBytes`, deliberately not exposed here to keep the field unambiguous), and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskSpaceUsage` maps to the disk-retention field when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsagePercent` maps to `retentionMaxDiskSpaceUsagePercent` when set. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. @@ -218,7 +218,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c 1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. -4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution. +4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the `otel-traces:4317` Service from `vtinsert` to the collector (no app change for gRPC clients). Ships sampling/rate-limiting and `AccountID`/`ProjectID` header injection via `headers_setter`. 5. **Docs**: enablement guide under `docs/observability/`. ## Open questions From f4d552c77adc0574a63a0c858cc181873f561ed9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 14:32:38 +0300 Subject: [PATCH 06/14] Correct VTCluster spec keys, disk-cap field, and topology per CRD verification Blocking fixes from /branch-review, verified against the pinned operator CRD: - VTCluster sub-spec keys are spec.insert/select/storage (prefix dropped), NOT vtinsert/vtselect/vtstorage; rendered workloads/services keep the vt prefix. - Disk cap field is spec.storage.retentionMaxDiskSpaceUsageBytes (bytes); no percent CR field exists. values field is now retentionDiskUsageBytes (bytes); earlier percent-only mapping was inverted. - Topology: monitoring installs into cozy-monitoring; ExternalName redirects live in cozy-monitoring and resolve to *.tenant-root.svc.cluster.local. - Nits: storageSize values key (avoid triple-storage stutter); WorkloadMonitor wording tightened + redis listed; confirm-before-impl caveat lists OTLP wire details + service-name prefix. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 4478c2c..e0b8e4b 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -22,15 +22,15 @@ In scope: a traces backend (platform-wide and per-tenant), an OTLP ingest gatewa ## Context -Cozystack's observability is multi-tenant with a central backend. The platform stack runs in `tenant-root` (namespace `cozy-monitoring`) and hosts VictoriaMetrics, VictoriaLogs, Grafana (via grafana-operator), Alerta and vmalert. Tenants either ship signals to the central backend (ExternalName redirects in tenant namespaces point at the root stack; vmagent stamps a `tenant:` external label) or run their own isolated stack from `packages/extra/monitoring`. +Cozystack's observability is multi-tenant with a central backend. The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana via grafana-operator, Alerta, vmalert) installs into the `cozy-monitoring` namespace (`packages/core/platform/sources/monitoring.yaml`), and its backing insert/select services resolve to `*.tenant-root.svc.cluster.local`. The redirect wiring lives in `cozy-monitoring`: `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` creates `ExternalName` services *there* (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) whose `externalName` targets are in `tenant-root` — so `cozy-monitoring` holds the aliases and `tenant-root` holds what they resolve to, gated on `_cluster.monitoring-enabled`. vmagent stamps a `tenant:` external label; a tenant can instead run its own isolated stack from `packages/extra/monitoring`. Metrics storage is declared as a list of tiers in values and rendered into VictoriaMetrics CRs — `metricsStorages` in `packages/system/monitoring/values.yaml` defines a `shortterm` (3d) and a `longterm` (14d) tier. Logs storage follows the identical shape: `logsStorages` renders one `VLCluster` per entry in `packages/system/monitoring/templates/vlogs/vlogs.yaml`, with the retention period set on `vlstorage.retentionPeriod`, `managedMetadata` labels for application ownership, a label stamped on the storage PVC claim template so the post-delete cleanup hook can find it, and a load-bearing guard that **fails the render** when the list is empty rather than silently shipping to a non-existent endpoint (the fix for issue `cozystack/cozystack#3181`). Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. -Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`. +Applications expose metrics today but there is no tracing surface. Most managed engines declare a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb, redis) and/or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis's `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status (and query Prometheus for resource usage) for the dashboard and billing surfaces — the actual metric scrape configs are rendered by the individual app charts, not by this controller, so tracing opt-in belongs in each app's `values.yaml`, not in `WorkloadMonitor`. -Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services still carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*`. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. ### The problem @@ -64,13 +64,13 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora tracingStorages: - name: generic retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskUsagePercent: "" # optional disk-based cap, percent only (e.g. "80"); see note below - storage: 10Gi + retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below + storageSize: 10Gi # PVC size for vtstorage storageClassName: "" replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, configurable replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. @@ -89,12 +89,12 @@ spec: apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - vtinsert: { replicaCount: {{ .replicaCount | default 2 }} } - vtselect: { replicaCount: {{ .replicaCount | default 2 }} } - vtstorage: + insert: { replicaCount: {{ .replicaCount | default 2 }} } + select: { replicaCount: {{ .replicaCount | default 2 }} } + storage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskUsagePercent }} - retentionMaxDiskSpaceUsagePercent: {{ . | quote }} + {{- with .retentionDiskUsageBytes }} + retentionMaxDiskSpaceUsageBytes: {{ . | quote }} {{- end }} replicaCount: {{ .replicaCount | default 2 }} storage: @@ -108,7 +108,7 @@ spec: {{- end }} resources: requests: - storage: {{ .storage }} + storage: {{ .storageSize }} {{- end }} ``` @@ -202,20 +202,20 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` via the percent-only `retentionDiskUsagePercent` values field (upstream also offers a mutually-exclusive `-retention.maxDiskSpaceUsageBytes`, deliberately not exposed here to keep the field unambiguous), and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsagePercent` maps to `retentionMaxDiskSpaceUsagePercent` when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. ## Rollout -1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. +1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the port-explicit `otel-traces:4317` gRPC redirect Service in `packages/system/cozystack-basics` (targeting `vtinsert`'s gRPC listener). Apps can push OTLP/gRPC directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. 4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the `otel-traces:4317` Service from `vtinsert` to the collector (no app change for gRPC clients). Ships sampling/rate-limiting and `AccountID`/`ProjectID` header injection via `headers_setter`. @@ -225,14 +225,14 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. - **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? -- **Grafana traces datasource type (confirm before implementation)**: the design assumes `type: jaeger` against `vtselect` (VictoriaTraces' Jaeger-compatible query API), with the dedicated VictoriaTraces datasource plugin as the alternative. This is the one load-bearing claim not verifiable against the cozystack tree — the whole trace↔logs↔metrics UX depends on which actually lands, so it must be confirmed against the pinned VictoriaTraces version before building. +- **Confirm-before-implementation (upstream VictoriaTraces surface)**: the `VTCluster` spec keys (`insert`/`select`/`storage`) and the disk-cap field (`retentionMaxDiskSpaceUsageBytes`) are verified against the pinned operator CRD. Still taken on upstream faith and to be confirmed against the pinned VictoriaTraces version before building: the Grafana datasource type (`type: jaeger` against `vtselect`, or the dedicated VictoriaTraces plugin — the whole correlation UX depends on which lands); the OTLP wire details (`vtinsert` HTTP `:10481` + `/insert/opentelemetry/v1/traces`, gRPC via `-otlpGRPCListenAddr`); and the exact rendered service-name prefix (`vtinsert-*`/`vtselect-*` vs `insert-*`). - **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) - **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. - **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? - **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered From dd44d1cccf7d2f8180dfd76ff8239f6ff8ea6045 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 15:59:01 +0300 Subject: [PATCH 07/14] docs: drop 'raised in review' meta-annotations from the proposal Keep the substantive points; remove the review-attribution asides so the document reads as a standalone design rather than a review-response artifact. Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index e0b8e4b..d2d9b8f 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -72,9 +72,9 @@ tracingStorages: Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). -**Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. +**Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. -**Durability note** (raised in review): `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. +**Durability note:** `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml {{- range .Values.tracingStorages }} @@ -120,17 +120,17 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP/gRPC to the stable Service `otel-traces.cozy-monitoring.svc:4317`, which in Stage A resolves to `vtinsert`'s OTLP/gRPC listener (enabled by setting `-otlpGRPCListenAddr` on the `VTCluster`). Tenant redirection to the central stack reuses the mechanism in `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` (gated on `_cluster.monitoring-enabled` from `monitoring.rootEnabled`) that logs and metrics already use — but as a **port-explicit** Service, not a bare `ExternalName`. -**OTLP service contract** (raised in review — the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. +**OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). - Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. -- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** (raised in review) and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. +- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. -**Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. +**Tail-sampling affinity:** tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, promoting Stage A→B changes only what that Service targets (`vtinsert` → collector) — a platform-side change, with no app reconfiguration (this holds for gRPC; OTLP/HTTP is not path-interchangeable, so HTTP users adopt the collector from the start). A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. @@ -139,10 +139,10 @@ Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. -- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them (raised in review). So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them. So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. -**Read-side tenant isolation** (raised in review — the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. +**Read-side tenant isolation** (the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. ### 4. Per-application opt-in @@ -202,7 +202,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. From 7c9394cbd90b46659eb935477c39d3cefad8a404 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 18:55:58 +0300 Subject: [PATCH 08/14] Address 2026-07-20 review: drop unverified query-string tenant precedence Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index d2d9b8f..d304ad2 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -122,7 +122,7 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. From 3cb278cd76f6650bed2166cba46466dce2f6aa07 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 19:25:38 +0300 Subject: [PATCH 09/14] Address branch review: resolve read-side isolation, VTSingle mode, fix ExternalName wording - Resolve read-side tenant isolation open question: structural isolation (isolated per-tenant stack + same-namespace vtselect datasource + NetworkPolicy, account 0), matching logs/metrics; vtselect has no authz so a datasource header pin is routing, not a security boundary; shared backend needs vmauth (deferred). Reconcile Grafana + Security sections. - Resolve VTCluster-vs-VTSingle: per-entry mode: cluster|single (default cluster). - Fix contradiction on the no-backend failure mode: Service has no backing endpoints, not 'ExternalName resolves to nothing' (design uses a port-explicit Service, not a bare ExternalName). - Soften the vtinsert-* service-name prefix to confirm-before-implementation. - Align write-path AccountID/ProjectID framing with the current account-0 + tenant-label convention. - Add revised-date suffix. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index d304ad2..996eac8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -2,7 +2,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` +- **Date:** `2026-07-16` (revised 2026-07-20) - **Status:** Review ## Overview @@ -30,7 +30,7 @@ Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operat Applications expose metrics today but there is no tracing surface. Most managed engines declare a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb, redis) and/or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis's `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status (and query Prometheus for resource usage) for the dashboard and billing surfaces — the actual metric scrape configs are rendered by the individual app charts, not by this controller, so tracing opt-in belongs in each app's `values.yaml`, not in `WorkloadMonitor`. -Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services still carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*`. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services are *expected* to carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*` — this prefix is load-bearing for the Stage A `otel-traces:4317` target, so it stays a **confirm-before-implementation** item (see Open questions) rather than an asserted fact. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. ### The problem @@ -63,6 +63,7 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora ```yaml tracingStorages: - name: generic + mode: cluster # "cluster" -> VTCluster (default), "single" -> VTSingle (edge/dev/small) retentionPeriod: "14d" # age-based retention; default 14 days per the requirement retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below storageSize: 10Gi # PVC size for vtstorage @@ -70,13 +71,14 @@ tracingStorages: replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). +Render one backend CR per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with the VictoriaTraces prefix-less sub-spec keys). The per-entry `mode` selects the CR kind: `cluster` (default) renders a `VTCluster` — the multi-component variant used below and the same choice `metricsStorages`/`logsStorages` make with `VMCluster`/`VLCluster` in every stack today; `single` renders a single-binary `VTSingle` for edge/dev/small clusters where the cluster footprint is unwarranted (its sub-spec is flatter — no separate insert/select/storage components — so `replicaCount` and the storage/retention fields apply to the one workload). Both carry: `managedMetadata` labels for application ownership, configurable replica counts, `retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. **Durability note:** `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml +{{- /* sketch shows mode: cluster; mode: single renders kind: VTSingle with the flatter single-binary spec */}} {{- range .Values.tracingStorages }} --- apiVersion: operator.victoriametrics.com/v1 @@ -122,7 +124,7 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Note the current Cozystack convention is precisely account `0` everywhere plus a `tenant` external label (see Context) — the same one metrics/logs use — so per-tenant `AccountID`/`ProjectID` is *not* today's convention but the option a **shared** trace backend would adopt to separate tenants by account. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`), the default isolation model, sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. @@ -136,13 +138,13 @@ Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, ### 3. Grafana datasource and correlation -Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: +Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API under the `/select/jaeger` prefix on `vtselect`, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) with a same-namespace URL — `http://vtselect-{{ .name }}.{{ $.Release.Namespace }}.svc:10471/select/jaeger` (cluster; `:10428` for `VTSingle`) — exactly as the logs datasource points at its in-namespace `vlselect`. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. - **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them. So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. -**Read-side tenant isolation** (the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. +**Read-side tenant isolation** works exactly as it does for logs and metrics today — **structurally, not by a query-time tenant filter**. Cozystack does not scope reads with `AccountID`/`ProjectID` anywhere: every monitoring backend uses account `0`, tenants that need isolation run their own self-contained stack in their own namespace (`packages/extra/monitoring`, wired by `packages/apps/tenant/templates/monitoring.yaml`), and the Grafana datasource URL is pinned to that namespace's own select service, fenced by NetworkPolicy. Traces inherit this model unchanged: the isolated per-tenant trace stack's datasource points at its own in-namespace `vtselect`, so tenant A physically cannot reach tenant B's spans. This is important because `vtselect` performs **no per-tenant authorization** and trusts `AccountID`/`ProjectID` verbatim (upstream: "use vmauth for per-tenant authorization") — so a datasource header pin is a routing selector, never a security boundary. On a *shared* central backend, traces get the same weak, label-only cross-tenant read property that shared metrics/logs already have; making that a real boundary would require an authenticating proxy (vmauth) that forces the headers from identity and strips client-supplied ones — deferred as future work, not part of the MVP (see Resolved / Open questions). ### 4. Per-application opt-in @@ -194,7 +196,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. - **Tenant attribution must be trusted**: `AccountID`/`ProjectID` (and the `tenant` attribute) must be injected from authenticated workload identity, never accepted verbatim from a tenant that could spoof another's IDs. On the shared backend this injection belongs at a boundary the tenant cannot bypass — the per-tenant proxy or the collector — not purely in tenant-controlled app config. -- **Isolation (write and read)**: write-side attribution alone is not isolation. The **read** path must scope each tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so a shared `vtselect` cannot leak spans across tenants; central-backend mode keeps the existing tenant-labelling model, isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **Isolation (write and read)**: write-side attribution alone is not isolation, and `vtselect` performs no per-tenant authorization — it trusts the `AccountID`/`ProjectID` read header verbatim, so pinning that header on a per-tenant Grafana datasource is a routing selector, not a security boundary. Read isolation is therefore delivered the same way logs/metrics deliver it today: **either** (a) an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource can only reach its own in-namespace `vtselect`, fenced by NetworkPolicy — the recommended default; **or** (b) an authenticating proxy (vmauth) in front of a shared `vtselect` that derives `AccountID`/`ProjectID` from authenticated identity and strips any client-supplied headers. A per-tenant datasource header pin is safe only in combination with (a) or (b), because the endpoint itself remains reachable and unauthenticated. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. @@ -204,7 +206,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. - Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. -- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. +- App emits OTLP but no backend deployed → the `otel-traces` Service has no backing endpoints, so the exporter's connection fails and it drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing @@ -224,15 +226,13 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. -- **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? - **Confirm-before-implementation (upstream VictoriaTraces surface)**: the `VTCluster` spec keys (`insert`/`select`/`storage`) and the disk-cap field (`retentionMaxDiskSpaceUsageBytes`) are verified against the pinned operator CRD. Still taken on upstream faith and to be confirmed against the pinned VictoriaTraces version before building: the Grafana datasource type (`type: jaeger` against `vtselect`, or the dedicated VictoriaTraces plugin — the whole correlation UX depends on which lands); the OTLP wire details (`vtinsert` HTTP `:10481` + `/insert/opentelemetry/v1/traces`, gRPC via `-otlpGRPCListenAddr`); and the exact rendered service-name prefix (`vtinsert-*`/`vtselect-*` vs `insert-*`). -- **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) - **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. - **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? - **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). Both `VTCluster` and `VTSingle` are supported via a per-entry `mode` field (default `cluster`, matching how `metricsStorages`/`logsStorages` render the cluster variant in every stack today; `single` for edge/dev/small). Read-side tenant isolation is **structural**, exactly as for logs/metrics: account `0` everywhere, isolation by an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource reaches only its own in-namespace `vtselect`, fenced by NetworkPolicy — not a query-time tenant filter, since `vtselect` has no per-tenant authorization and trusts the `AccountID`/`ProjectID` header verbatim. A shared central backend would need vmauth (forcing headers from authenticated identity, stripping client-supplied ones) to become a real read boundary; that is deferred future work, not the MVP. ## Alternatives considered From a538e0b59bd6139947c39f2f5c3fb6ffbc97e517 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 23 Jul 2026 16:49:32 +0300 Subject: [PATCH 10/14] Address Mattia + IvanHunters review: disk-bounded default, per-tier replicas, storageSize rationale, PSS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mattia #1: default is always disk-bounded — retentionMaxDiskSpaceUsageBytes derived from storageSize (~85%) when retentionDiskUsageBytes is unset, so the out-of-the-box config can't silently fill the PVC. Update Failure + Testing. - Mattia #2: replicaCount documented as a uniform shortcut with optional per-tier insert/select/storage overrides (as VMCluster), storage tier durability-adjacent. - IvanHunters: inline rationale for the storageSize key name (avoids spec.storage.storage stutter with the VTCluster CRD's nested storage object). - IvanHunters: add Pod Security Standards posture for the new Collector Deployment and per-app OTLP sidecars (restricted: non-root, drop ALL, seccomp RuntimeDefault, no privilege escalation). - Bump revised-date to 2026-07-23. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 30 +++++++++++-------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 996eac8..2df86b8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -2,7 +2,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` (revised 2026-07-20) +- **Date:** `2026-07-16` (revised 2026-07-23) - **Status:** Review ## Overview @@ -65,12 +65,17 @@ tracingStorages: - name: generic mode: cluster # "cluster" -> VTCluster (default), "single" -> VTSingle (edge/dev/small) retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below - storageSize: 10Gi # PVC size for vtstorage + retentionDiskUsageBytes: "" # disk-based cap, byte quantity (e.g. "50GB"); when empty, derived from storageSize (~85%) so the default is always disk-bounded (see note) + storageSize: 10Gi # PVC size for vtstorage (named storageSize, not storage — see naming note) storageClassName: "" - replicaCount: 2 # component scaling, NOT data replication (see durability note) + replicaCount: 2 # uniform shortcut for all tiers; NOT data replication (see durability note) + # insert: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) + # select: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) + # storage: { replicaCount: 2 } # optional per-tier override; durability-adjacent, still NOT replication ``` +The per-entry `replicaCount` is a **uniform shortcut**: it seeds all three cluster tiers so a simple config stays one line. Because the tiers are rarely sized identically in practice — `insert` scales with ingest, `select` with query load, `storage` with retention/throughput, and `storage.replicaCount` is durability-adjacent rather than a query-scaling knob — each tier accepts an optional explicit override (`insert`/`select`/`storage` `replicaCount`), mirroring how `VMCluster` exposes per-component counts; an unset tier falls back to the shortcut. The PVC-size key is named `storageSize` rather than the sibling `storage` (used by `metricsStorages`/`logsStorages`) on purpose: the `VTCluster` CRD already nests a second `storage` object under `spec.storage` (`spec.storage.storage.volumeClaimTemplate`), so reusing `storage` as the values key would read as `spec.storage.storage` stutter — the rename keeps the values surface unambiguous. + Render one backend CR per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with the VictoriaTraces prefix-less sub-spec keys). The per-entry `mode` selects the CR kind: `cluster` (default) renders a `VTCluster` — the multi-component variant used below and the same choice `metricsStorages`/`logsStorages` make with `VMCluster`/`VLCluster` in every stack today; `single` renders a single-binary `VTSingle` for edge/dev/small clusters where the cluster footprint is unwarranted (its sub-spec is flatter — no separate insert/select/storage components — so `replicaCount` and the storage/retention fields apply to the one workload). Both carry: `managedMetadata` labels for application ownership, configurable replica counts, `retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. @@ -91,14 +96,14 @@ spec: apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - insert: { replicaCount: {{ .replicaCount | default 2 }} } - select: { replicaCount: {{ .replicaCount | default 2 }} } + {{- /* per-tier replicaCount falls back to the uniform .replicaCount shortcut (fallback plumbing elided in this sketch) */}} + insert: { replicaCount: {{ .insert.replicaCount | default .replicaCount | default 2 }} } + select: { replicaCount: {{ .select.replicaCount | default .replicaCount | default 2 }} } storage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskUsageBytes }} - retentionMaxDiskSpaceUsageBytes: {{ . | quote }} - {{- end }} - replicaCount: {{ .replicaCount | default 2 }} + {{- /* always disk-bounded: explicit bytes, else ~85% of storageSize so the default cannot silently fill the PVC (exact byte math elided) */}} + retentionMaxDiskSpaceUsageBytes: {{ .retentionDiskUsageBytes | default (include "vtraces.defaultDiskCap" .) | quote }} + replicaCount: {{ .storage.replicaCount | default .replicaCount | default 2 }} storage: volumeClaimTemplate: metadata: @@ -199,19 +204,20 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **Isolation (write and read)**: write-side attribution alone is not isolation, and `vtselect` performs no per-tenant authorization — it trusts the `AccountID`/`ProjectID` read header verbatim, so pinning that header on a per-tenant Grafana datasource is a routing selector, not a security boundary. Read isolation is therefore delivered the same way logs/metrics deliver it today: **either** (a) an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource can only reach its own in-namespace `vtselect`, fenced by NetworkPolicy — the recommended default; **or** (b) an authenticating proxy (vmauth) in front of a shared `vtselect` that derives `AccountID`/`ProjectID` from authenticated identity and strips any client-supplied headers. A per-tenant datasource header pin is safe only in combination with (a) or (b), because the endpoint itself remains reachable and unauthenticated. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. +- **Pod Security Standards**: the new OTLP Collector Deployment and the per-app OTLP sidecars/agents (Kafka, RabbitMQ, MariaDB, Redis, Postgres) run inside `tenant-*` namespaces, which enforce PSS **restricted**. They must ship a compliant pod-security posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, and `seccompProfile.type: RuntimeDefault` — like every other new workload class in this stack; no privileged or host-namespace access is required for OTLP ingest. ## Failure and edge cases - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk, so an age-only config lets a full PVC block ingest *before* old traces are evicted. The default is therefore **always disk-bounded**: `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) is set from the byte-quantity `retentionDiskUsageBytes` values field when given, and otherwise **derived from `storageSize` (~85%)** so no entry ever ships without a disk bound — matching the "uncrashable default, don't just document the knife" spirit of the `#3181` guard. A PVC/disk-usage capacity alert is added on top, and the behaviour is covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the `otel-traces` Service has no backing endpoints, so the exporter's connection fails and it drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set, and that when it is unset the field is still rendered, derived from `storageSize` (~85%), so no rendered `VTCluster`/`VTSingle` is ever disk-unbounded. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. From 8c0786110ff7b22b56c92c09a33fdfa7719b0e32 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 29 Jul 2026 16:01:23 +0300 Subject: [PATCH 11/14] Reframe proposal for @lllamnyp review: tenancy/network model, backend pluggability Address the CHANGES_REQUESTED review by shifting the doc from 'mirror logs/metrics one-for-one' to an architecture-first proposal focused on multitenancy, security, and vendor-neutrality: - Tenancy+network (core): traces are pushed, not scraped, so the metrics/logs central-agent model does not transfer under Cilium tenant isolation. Collector now lives inside the tenant namespace (zero NetworkPolicy change); shared-central backend is opt-in and needs an explicit, narrow egress rule (designed, not deferred). - Backend pluggability: OTLP+datasource is a backend-agnostic seam; VictoriaTraces default, Tempo/Jaeger selectable. - Per-app model: drop foundationdb monitoring.enabled as the template (it gates a billing WorkloadMonitor); tracing follows the platform-configured metrics model; endpoint is platform-decided. - Correct WorkloadMonitor description (billing/ownership meta-resource). - Routing: address Ingress/Gateway-API alternative. - Strip premature Helm-values/CRD-field detail into a non-normative Implementation notes appendix; keep principles in the body. - Drop the unverified query-string tenant-precedence claim; keep PSS. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 280 ++++++++---------- 1 file changed, 123 insertions(+), 157 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 2df86b8..472b983 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -1,36 +1,45 @@ # Distributed tracing in the Cozystack monitoring stack -- **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` +- **Title:** `Distributed tracing for managed applications via OTLP` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` (revised 2026-07-23) +- **Date:** `2026-07-16` (revised 2026-07-29) - **Status:** Review ## Overview -Cozystack ships two of the three observability signals out of the box — metrics (VictoriaMetrics) and logs (VictoriaLogs) — but has no supported way to collect distributed **traces**. An operator who wants to see how a request flows through a managed database or messaging cluster, or to correlate a slow span with the logs and metrics it produced, has nothing to turn on. This proposal adds the third signal: an OTLP ingest path, a VictoriaTraces backend that mirrors the existing VictoriaLogs deployment one-for-one, a Grafana traces datasource wired for trace↔logs↔metrics correlation, and a per-application opt-in toggle so a tenant enables tracing on exactly the workloads that need it. +Cozystack ships two of the three observability signals out of the box — metrics (VictoriaMetrics) and logs (VictoriaLogs) — but has no supported way to collect distributed **traces**. An operator who wants to see how a request flows through a managed database or messaging cluster, or to correlate a slow span with the logs and metrics it produced, has nothing to turn on. This proposal adds the third signal. -The design is deliberately conservative: it reuses the multi-tenant topology, the operator-driven provisioning, the Grafana datasource pattern, and the values surface that metrics and logs already established, so tracing lands as "the same thing again for a third signal" rather than a new subsystem with new conventions. The backend choice — VictoriaTraces — falls out of that principle: the `VTCluster`/`VTSingle` CRDs already ship in the victoria-metrics-operator that Cozystack deploys today, so no new operator is introduced. +The proposal is deliberately **architecture-first**: it settles the hard questions — multitenancy, network isolation, the trust boundary, and vendor neutrality — and leaves the exact Helm value shapes, field names, and port numbers to the implementation PRs in `cozystack/cozystack`. The three load-bearing decisions are: + +1. **Tracing is ingested inside the tenant boundary.** Unlike metrics and logs, traces are *pushed* by the application, so the collection point is a per-tenant OpenTelemetry Collector that lives in the tenant's own namespace. This keeps traces from punching a hole through the Cilium tenant isolation that metrics/logs never had to cross (they are *scraped* centrally). See [Design §2](#2-multitenancy-and-network-model-the-core-decision). +2. **The backend is pluggable.** OTLP ingest plus a Grafana datasource is a backend-agnostic contract. VictoriaTraces is the recommended default (its `VTCluster`/`VTSingle` CRDs already ship in the victoria-metrics-operator Cozystack deploys, so no new operator), but Grafana Tempo and Jaeger are first-class alternatives an admin can select. See [Design §1](#1-backend-pluggable-victoriatraces-default). +3. **Enablement follows the metrics model, not a per-app toggle.** For a supported engine, tracing is configured by the platform the same way metric scrapes are — not gated behind a copied `enabled` flag. See [Design §4](#4-per-application-enablement). + +This proposal does *not* mechanically mirror the logs/metrics stack. Where their conventions genuinely fit (grafana-operator datasources, the operator-driven readiness gate) it reuses them; where tracing is fundamentally different (push vs. scrape, and therefore the network path) it diverges on purpose and says why. One idea flows the other way, too: the `VTSingle` single-binary mode this proposal wants for edge/dev clusters is something the metrics and logs stacks could usefully adopt. ## Scope and related proposals -In scope: a traces backend (platform-wide and per-tenant), an OTLP ingest gateway, a Grafana datasource with correlation, and a per-app opt-in surface. Out of scope: automatic instrumentation of arbitrary tenant workloads, and tracing the internals of Virtual Machines or the Kubernetes control plane (VMs and Kubernetes are not traced by this proposal — only Cozystack-managed applications that can emit OTLP are). +In scope: a traces backend (per-tenant and, as an opt-in, shared-central), an OTLP ingest path that respects tenant network isolation, a Grafana datasource with trace↔logs↔metrics correlation, and platform-side enablement for supported engines. Out of scope: automatic instrumentation of arbitrary tenant workloads, and tracing the internals of Virtual Machines or the Kubernetes control plane. - **Sibling stack:** the platform monitoring stack `packages/system/monitoring` and the per-tenant stack `packages/extra/monitoring` (wired by `packages/apps/tenant/templates/monitoring.yaml`). This proposal extends both. -- **Collection agents:** `packages/system/monitoring-agents` (fluent-bit, vmagent) — the deployment pattern the OTLP collector follows. -- **Prior art in-repo:** Harbor already exposes an internal trace config (`packages/system/harbor/charts/harbor/values.yaml`, provider `jaeger` or `otel`). It is app-local and not a platform backend; this proposal supersedes ad-hoc per-app trace endpoints with a shared destination. +- **Collection agents:** `packages/system/monitoring-agents` (fluent-bit, vmagent) — the *deployment* pattern the collector borrows, but note the traffic model differs (see [§2](#2-multitenancy-and-network-model-the-core-decision)). +- **Network policy:** `packages/apps/tenant/templates/networkpolicy.yaml` — the Cilium tenant-isolation policies this design must live within. +- **Prior art in-repo:** Harbor exposes an internal, app-local trace config (`packages/system/harbor/charts/harbor/values.yaml`, provider `jaeger`/`otel`); it is not a platform backend. This proposal supersedes ad-hoc per-app trace endpoints with a shared destination. - **Driver:** requested by a client (hidora) who needs request-level visibility across managed DBaaS and messaging services. ## Context -Cozystack's observability is multi-tenant with a central backend. The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana via grafana-operator, Alerta, vmalert) installs into the `cozy-monitoring` namespace (`packages/core/platform/sources/monitoring.yaml`), and its backing insert/select services resolve to `*.tenant-root.svc.cluster.local`. The redirect wiring lives in `cozy-monitoring`: `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` creates `ExternalName` services *there* (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) whose `externalName` targets are in `tenant-root` — so `cozy-monitoring` holds the aliases and `tenant-root` holds what they resolve to, gated on `_cluster.monitoring-enabled`. vmagent stamps a `tenant:` external label; a tenant can instead run its own isolated stack from `packages/extra/monitoring`. +Cozystack's observability is multi-tenant with a central backend, and it crosses the tenant boundary in a very specific way that traces cannot simply copy. -Metrics storage is declared as a list of tiers in values and rendered into VictoriaMetrics CRs — `metricsStorages` in `packages/system/monitoring/values.yaml` defines a `shortterm` (3d) and a `longterm` (14d) tier. Logs storage follows the identical shape: `logsStorages` renders one `VLCluster` per entry in `packages/system/monitoring/templates/vlogs/vlogs.yaml`, with the retention period set on `vlstorage.retentionPeriod`, `managedMetadata` labels for application ownership, a label stamped on the storage PVC claim template so the post-delete cleanup hook can find it, and a load-bearing guard that **fails the render** when the list is empty rather than silently shipping to a non-existent endpoint (the fix for issue `cozystack/cozystack#3181`). +**Where the backend runs.** The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana, Alerta, vmalert) is deployed by the *root* tenant into `tenant-root` (`packages/apps/tenant/templates/monitoring.yaml`). The `cozy-monitoring` namespace holds only `ExternalName` aliases (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) that CNAME to the real services in `tenant-root` (`packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, gated on `_cluster.monitoring-enabled`). The cluster-wide agents (vmagent, fluent-bit, kube-state-metrics, node-exporter) live in `cozy-monitoring` (`packages/system/monitoring-agents`). -Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. +**How metrics and logs cross the tenant boundary — by *scraping*, not by tenants pushing out.** vmagent runs centrally in `cozy-monitoring` and scrapes targets cluster-wide, then remote-writes to `vminsert-*.tenant-root.svc`. fluent-bit runs as a DaemonSet in `cozy-monitoring` and ships to `vlinsert-generic.tenant-root.svc`. Both cross into `tenant-root` because `cozy-monitoring` carries the `cozystack.io/system: "true"` label (stamped by the operator, `internal/operator/package_reconciler.go`) and `tenant-root`'s Cilium ingress policy trusts every `cozystack.io/system` namespace. **This is an ingress trust granted to a system namespace — it does not let a tenant workload egress toward monitoring.** -Applications expose metrics today but there is no tracing surface. Most managed engines declare a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb, redis) and/or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis's `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status (and query Prometheus for resource usage) for the dashboard and billing surfaces — the actual metric scrape configs are rendered by the individual app charts, not by this controller, so tracing opt-in belongs in each app's `values.yaml`, not in `WorkloadMonitor`. +**Tenant network isolation (Cilium).** `packages/apps/tenant/templates/networkpolicy.yaml` renders `CiliumClusterwideNetworkPolicy` per tenant; the posture is default-deny once selected. A tenant pod's egress is limited to: its own tenant subtree; endpoints labeled `app.kubernetes.io/name: vminsert` / `app.kubernetes.io/instance: etcd` / `cozystack.io/service: ingress` in **ancestor** namespaces; DNS; a fixed set of `cozy-*` namespaces (`cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-kubevirt-cdi`); and `world` (out-of-cluster). Notably **`cozy-monitoring` is not on that list**, and a generic pod in `tenant-root` is not reachable either — only the specifically-labeled `vminsert`/`etcd`/`ingress` endpoints are. A tenant that runs its own monitoring gets a per-tenant vmagent *inside its own namespace* that pushes up to the parent's `vminsert` — that is exactly the `vminsert`-labeled ancestor egress rule. -Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services are *expected* to carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*` — this prefix is load-bearing for the Stage A `otel-traces:4317` target, so it stays a **confirm-before-implementation** item (see Open questions) rather than an asserted fact. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. +**Application observability today.** Apps expose metrics out of the box: the charts render `VMServiceScrape`/`VMPodScrape` (or CNPG's `enablePodMonitor`) unconditionally — there is no per-app "enable metrics" toggle. Most engines also declare a `WorkloadMonitor` CR, but `WorkloadMonitor` is **not** a metrics-collection mechanism: it is a billing/ownership meta-resource holding a label selector that identifies the pods, services, and PVCs belonging to an app, reconciled into `Workload` objects (replicas/resources/status) for the dashboard and billing surfaces (`internal/controller/workloadmonitor_controller.go`). The one app that gates a `WorkloadMonitor` behind a tenant-facing `monitoring.enabled` flag is foundationdb — and that is a mistake we should not copy (it hides a billing meta-resource behind a tenant toggle), not a template for a tracing switch. + +**No new operator is needed for the default backend.** The victoria-metrics-operator Cozystack already runs (`packages/system/victoria-metrics-operator`) ships the `VTCluster` and `VTSingle` CRDs (`operator.victoriametrics.com/v1`), which decompose into insert/select/storage components analogous to `VLCluster`. ### The problem @@ -40,210 +49,167 @@ Crucially, the tracing backend needs no new operator. The victoria-metrics-opera ## Goals -- Accept traces over **OTLP** (gRPC `4317` and HTTP `4318`), the standard cloud-native tracing protocol. -- Store traces in a **VictoriaTraces** backend with a **configurable retention period, defaulting to 14 days**. -- Provide a **per-application opt-in** `tracing.enabled` toggle; tracing is **off by default** and adds zero overhead until enabled. -- Provision a **Grafana traces datasource** and wire **trace↔logs↔metrics** correlation so an operator can pivot between all three signals. -- Preserve **per-tenant isolation and the central-backend topology** exactly as metrics and logs do today. -- Introduce **no new operator** and no new provisioning convention — reuse VictoriaTraces (already in vm-operator), the storage-list values shape, and the grafana-operator datasource pattern. +- Accept traces over **OTLP** (the standard cloud-native tracing protocol) at an endpoint the application can reach **without breaching tenant network isolation**. +- Store traces in a **pluggable backend** with a configurable retention period (default 14 days); VictoriaTraces is the default, Tempo/Jaeger are selectable. +- Preserve **per-tenant isolation** — on both the write and the read path — as the default, and describe honestly what a shared-central backend would additionally require. +- Provision a **Grafana traces datasource** and wire **trace↔logs↔metrics** correlation. +- Enable tracing for a supported engine the way metrics are enabled — **platform-configured**, not a per-app opt-in flag copied from the wrong example. +- Introduce **no new operator** for the default backend. ### Non-goals -- Auto-instrumenting arbitrary tenant workloads. This proposal wires the *transport and storage*; emitting spans is the application's job (native where the engine supports it, sidecar/agent otherwise). +- Auto-instrumenting arbitrary tenant workloads. This proposal wires transport, storage, and tenancy; emitting spans is the engine's job (native where supported, sidecar/agent otherwise). - Tracing Virtual Machines or Kubernetes control-plane internals. - Changing the existing metrics or logs pipelines. -- Mandating a sampling policy for tenant applications (the platform sets a safe default and exposes a knob). +- Locking in the exact Helm value schema, CRD field names, or port numbers — those are settled in the implementation PRs (see [Implementation notes](#implementation-notes-non-normative)). ## Design -### 1. Backend: VictoriaTraces (`tracingStorages`) - -Add a `tracingStorages` list to the monitoring values, parallel to `metricsStorages` and `logsStorages`, in both `packages/system/monitoring/values.yaml` and `packages/extra/monitoring/values.yaml`: - -```yaml -tracingStorages: -- name: generic - mode: cluster # "cluster" -> VTCluster (default), "single" -> VTSingle (edge/dev/small) - retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskUsageBytes: "" # disk-based cap, byte quantity (e.g. "50GB"); when empty, derived from storageSize (~85%) so the default is always disk-bounded (see note) - storageSize: 10Gi # PVC size for vtstorage (named storageSize, not storage — see naming note) - storageClassName: "" - replicaCount: 2 # uniform shortcut for all tiers; NOT data replication (see durability note) - # insert: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) - # select: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) - # storage: { replicaCount: 2 } # optional per-tier override; durability-adjacent, still NOT replication -``` +### 1. Backend: pluggable, VictoriaTraces default -The per-entry `replicaCount` is a **uniform shortcut**: it seeds all three cluster tiers so a simple config stays one line. Because the tiers are rarely sized identically in practice — `insert` scales with ingest, `select` with query load, `storage` with retention/throughput, and `storage.replicaCount` is durability-adjacent rather than a query-scaling knob — each tier accepts an optional explicit override (`insert`/`select`/`storage` `replicaCount`), mirroring how `VMCluster` exposes per-component counts; an unset tier falls back to the shortcut. The PVC-size key is named `storageSize` rather than the sibling `storage` (used by `metricsStorages`/`logsStorages`) on purpose: the `VTCluster` CRD already nests a second `storage` object under `spec.storage` (`spec.storage.storage.volumeClaimTemplate`), so reusing `storage` as the values key would read as `spec.storage.storage` stutter — the rename keeps the values surface unambiguous. +OTLP ingest and a Grafana datasource form a **backend-agnostic contract**: the collector speaks OTLP outward, and Grafana reads through a datasource. Only two things change when the backend changes — the storage CR the platform renders, and the Grafana datasource type. Everything else in this design (the collector, the tenancy model, the correlation wiring, the per-app surface) is unaffected. The platform therefore exposes a **backend selector**, defaulting to VictoriaTraces: -Render one backend CR per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with the VictoriaTraces prefix-less sub-spec keys). The per-entry `mode` selects the CR kind: `cluster` (default) renders a `VTCluster` — the multi-component variant used below and the same choice `metricsStorages`/`logsStorages` make with `VMCluster`/`VLCluster` in every stack today; `single` renders a single-binary `VTSingle` for edge/dev/small clusters where the cluster footprint is unwarranted (its sub-spec is flatter — no separate insert/select/storage components — so `replicaCount` and the storage/retention fields apply to the one workload). Both carry: `managedMetadata` labels for application ownership, configurable replica counts, `retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). +- **VictoriaTraces** (default): VictoriaMetrics' own trace backend — the same-vendor counterpart to Tempo, on the same `victoria-metrics-operator` as VictoriaMetrics/VictoriaLogs. `VTCluster` for production, `VTSingle` (single binary) for edge/dev/small clusters where a multi-component cluster is unwarranted. Chosen as default because its CRDs already ship in the operator Cozystack runs — no new operator, one operational model shared with metrics/logs. (The `VTSingle`/`VTCluster` split is a mode the metrics and logs stacks would benefit from adopting too.) **Caveat: VictoriaTraces is pre-GA** per its upstream roadmap (data structure/backward-compat not yet frozen; the Grafana-facing query API is being delivered as Tempo Query-frontend-compatible HTTP APIs) — which is exactly why the backend seam and the Tempo fallback below matter, and why maturity is an [open question](#open-questions). +- **Grafana Tempo**: object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs) with the strongest Grafana-native correlation. The recommended fallback if VictoriaTraces is not ready at the pinned version. Because VictoriaTraces is converging on a Tempo-compatible query API, both can share the same Grafana datasource type — making this fallback nearly seamless. +- **Jaeger**: mature and OTLP-native, but its own UI and weaker Grafana integration cut against single-pane correlation. -**Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. +The vendor-neutrality question @lllamnyp raised is answered by *this* seam, not by adding operators: an admin picks a backend at install time; the OTLP/datasource contract keeps the choice from leaking into applications or the tenancy design. See [Alternatives](#alternatives-considered) for the full trade-off. -**Durability note:** `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. +The backend is provisioned by the operator, and the monitoring HelmRelease gates readiness on the storage CR reaching `operational` (the `waitStrategy: poller` + `healthCheckExprs` pattern `VLCluster` already uses in `packages/extra/monitoring/templates/helmrelease.yaml`). Without that gate the release flips Ready before the backend can accept writes — the silent-black-hole failure that motivated `cozystack/cozystack#3181` for logs. Tracing is opt-in: a cluster that configures no tracing backend renders nothing and stays healthy; a backend that is *requested but left empty* is a misconfiguration and fails the render. (The exact values shape is an implementation detail — [Implementation notes](#implementation-notes-non-normative).) -```yaml -{{- /* sketch shows mode: cluster; mode: single renders kind: VTSingle with the flatter single-binary spec */}} -{{- range .Values.tracingStorages }} ---- -apiVersion: operator.victoriametrics.com/v1 -kind: VTCluster -metadata: - name: {{ .name }} -spec: - managedMetadata: - labels: - apps.cozystack.io/application.group: apps.cozystack.io - apps.cozystack.io/application.kind: Monitoring - apps.cozystack.io/application.name: {{ $.Release.Name }} - {{- /* per-tier replicaCount falls back to the uniform .replicaCount shortcut (fallback plumbing elided in this sketch) */}} - insert: { replicaCount: {{ .insert.replicaCount | default .replicaCount | default 2 }} } - select: { replicaCount: {{ .select.replicaCount | default .replicaCount | default 2 }} } - storage: - retentionPeriod: {{ .retentionPeriod | quote }} - {{- /* always disk-bounded: explicit bytes, else ~85% of storageSize so the default cannot silently fill the PVC (exact byte math elided) */}} - retentionMaxDiskSpaceUsageBytes: {{ .retentionDiskUsageBytes | default (include "vtraces.defaultDiskCap" .) | quote }} - replicaCount: {{ .storage.replicaCount | default .replicaCount | default 2 }} - storage: - volumeClaimTemplate: - metadata: - labels: - apps.cozystack.io/application.name: {{ $.Release.Name }} - spec: - {{- with .storageClassName }} - storageClassName: {{ . }} - {{- end }} - resources: - requests: - storage: {{ .storageSize }} -{{- end }} -``` +### 2. Multitenancy and network model (the core decision) -The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as it does for `VLCluster` today: `waitStrategy: poller` plus a `healthCheckExprs` entry that waits for the CR's `status.updateStatus == 'operational'` (see `packages/extra/monitoring/templates/helmrelease.yaml`). Without the poller gate the release flips Ready as soon as Helm applies the CR — the exact silent-black-hole failure mode that motivated `cozystack/cozystack#3181` for logs. +This is where tracing genuinely departs from metrics and logs, and where the design must be explicit rather than deferred. -### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway) +**Why the metrics/logs model does not transfer.** Metrics and logs cross the tenant boundary because central agents *scrape/tail* workloads and forward — the tenant never egresses toward monitoring (see [Context](#context)). OTLP is the opposite: the **application pushes** spans. Under the current Cilium policy a tenant pod *cannot* reach a collector in `cozy-monitoring` (not in the egress allow-list) nor a generic collector in `tenant-root` (only `vminsert`/`etcd`/`ingress`-labeled endpoints are reachable). So a naive "app → central OTLP endpoint" design would either be blocked by policy or require punching a broad hole in tenant isolation. That is precisely the concern @lllamnyp flagged, and it is real. -`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, behind **one stable app-facing endpoint** — an explicit in-cluster Service on OTLP/gRPC `4317` — so promoting Stage A→B swaps what that Service targets (backend → collector), not the app's configuration. +**Decision: the collector lives inside the tenant namespace.** Each participating tenant gets an OpenTelemetry Collector Deployment in its *own* namespace — the same placement as a per-tenant vmagent. Applications push OTLP to that in-namespace collector, so app→collector traffic is intra-namespace and covered by the existing `allow-internal-communication` policy: **zero NetworkPolicy change, and tenant isolation is never breached.** The collector is the per-tenant trust boundary: it stamps the tenant's identity, applies sampling and rate-limits, and forwards onward. What it forwards *to* defines the two supported topologies: -**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP/gRPC to the stable Service `otel-traces.cozy-monitoring.svc:4317`, which in Stage A resolves to `vtinsert`'s OTLP/gRPC listener (enabled by setting `-otlpGRPCListenAddr` on the `VTCluster`). Tenant redirection to the central stack reuses the mechanism in `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` (gated on `_cluster.monitoring-enabled` from `monitoring.rootEnabled`) that logs and metrics already use — but as a **port-explicit** Service, not a bare `ExternalName`. +| | **Per-tenant backend (default)** | **Shared central backend (opt-in)** | +|---|---|---| +| Backend location | tenant's own namespace (like `packages/extra/monitoring`) | `tenant-root` | +| Collector → backend hop | intra-namespace | crosses into `tenant-root` | +| NetworkPolicy change | none | **new Cilium egress rule required** (see below) | +| Write isolation | physical (never leaves namespace) | by injected `AccountID`/`ProjectID`, enforced at the collector | +| Read isolation | physical: datasource → in-namespace select service, fenced by policy | needs an authenticating proxy (vmauth); a bare shared select is **not** a boundary | +| When to choose | default; strongest isolation | central retention/cost consolidation, accepted trade-off | -**OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. +**The shared-central egress rule (designed here, not deferred).** For the opt-in shared topology, the collector→central-backend hop needs an explicit allowance — modeled exactly on the existing ancestor-`vminsert` egress block in `packages/apps/tenant/templates/networkpolicy.yaml`: add a `toEndpoints` rule matching the central OTLP-ingest endpoint's label (e.g. `app.kubernetes.io/name: `) in the ancestor namespace, gated on the tenant having tracing enabled. This is a deliberate, narrow widening of the tenant egress surface, and it is a security decision the operator opts into — not a silent default. We explicitly **reject** the alternative of mislabeling the collector as `vminsert` to ride the existing rule: that is a label hack that erodes the meaning of the policy. -**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Note the current Cozystack convention is precisely account `0` everywhere plus a `tenant` external label (see Context) — the same one metrics/logs use — so per-tenant `AccountID`/`ProjectID` is *not* today's convention but the option a **shared** trace backend would adopt to separate tenants by account. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`), the default isolation model, sidestep this by not sharing a backend. +**Identity, write and read.** +- *Write:* `AccountID`/`ProjectID` (and any `tenant` resource attribute) are injected by the per-tenant collector from the identity it is deployed with — never from tenant-controlled application config, which a tenant could spoof. Because the collector is per-tenant, it already knows its tenant. +- *Read:* in the default per-tenant topology the Grafana datasource points only at the tenant's own in-namespace select service, fenced by NetworkPolicy — tenant A physically cannot reach tenant B's spans. On a shared backend the select service performs no per-tenant authorization (it trusts the read headers verbatim), so real read isolation there requires an authenticating proxy (vmauth) that derives the tenant from identity and strips client-supplied headers. This is stated as part of the design, with the honest caveat that a shared backend without vmauth has only the same weak, label-only cross-tenant read property that shared metrics/logs have today. -**Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. +### 3. Ingest topology and routing -- Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). -- Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. -- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. +Applications target a **stable in-namespace endpoint** — the per-tenant collector's OTLP Service. Keeping that endpoint fixed means the platform can evolve what sits behind it (sampling policy, backend target) without any application reconfiguration. -**Tail-sampling affinity:** tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. +**Why a Collector and not direct-to-backend.** Direct ingest to the backend's insert component is possible (VictoriaTraces' insert speaks OTLP natively) and is a reasonable first implementation step, but it provides no place to stamp tenant identity, rate-limit a noisy tenant, or sample. Since [§2](#2-multitenancy-and-network-model-the-core-decision) already puts a per-tenant component in the namespace for the trust boundary, that component *is* the collector — so the collector is core to the design, not a later add-on. If a phased rollout ships direct ingest first, the app-facing endpoint stays the collector's Service so the switch is platform-side only. -Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, promoting Stage A→B changes only what that Service targets (`vtinsert` → collector) — a platform-side change, with no app reconfiguration (this holds for gRPC; OTLP/HTTP is not path-interchangeable, so HTTP users adopt the collector from the start). A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. +**A gateway Deployment, not a DaemonSet.** OTLP is push-based over the network — the applications are the agents — so the collector plays the centralized `vmagent`/insert role, not the node-local fluent-bit role. -### 3. Grafana datasource and correlation +**Tail sampling needs trace affinity.** If tail sampling is used, all spans of a trace must reach one collector instance; a plain Service round-robins and would sample on partial traces. So the sampling tier runs single-replica, or behind a first tier using consistent-hash routing by trace ID. Head sampling has no such constraint. The default sampling policy is an [open question](#open-questions). -Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API under the `/select/jaeger` prefix on `vtselect`, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) with a same-namespace URL — `http://vtselect-{{ .name }}.{{ $.Release.Namespace }}.svc:10471/select/jaeger` (cluster; `:10428` for `VTSingle`) — exactly as the logs datasource points at its in-namespace `vlselect`. Configure: +**Routing: why an in-namespace Service, and the Ingress/Gateway alternative (@lllamnyp :130).** Because the collector is *in the tenant namespace*, the app-facing hop is plain in-cluster service traffic — an Ingress or Gateway-API `HTTPRoute` would add an L7 hop with no benefit for intra-namespace OTLP, and OTLP/gRPC in particular is a poor fit for a typical HTTP Ingress. An internal Gateway/`HTTPRoute` becomes relevant only for the shared-central topology (a single in-cluster address fronting the central backend) or for exposing OTLP to workloads *outside* the cluster; both are folded into the shared-backend egress design and the external-exposure [open question](#open-questions) rather than the default path. If Cozystack standardizes on Gateway API for in-cluster L7, the shared-central front-end should use an `HTTPRoute` there rather than a bespoke proxy. -- **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. -- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them. So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. -- **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. +### 4. Per-application enablement -**Read-side tenant isolation** works exactly as it does for logs and metrics today — **structurally, not by a query-time tenant filter**. Cozystack does not scope reads with `AccountID`/`ProjectID` anywhere: every monitoring backend uses account `0`, tenants that need isolation run their own self-contained stack in their own namespace (`packages/extra/monitoring`, wired by `packages/apps/tenant/templates/monitoring.yaml`), and the Grafana datasource URL is pinned to that namespace's own select service, fenced by NetworkPolicy. Traces inherit this model unchanged: the isolated per-tenant trace stack's datasource points at its own in-namespace `vtselect`, so tenant A physically cannot reach tenant B's spans. This is important because `vtselect` performs **no per-tenant authorization** and trusts `AccountID`/`ProjectID` verbatim (upstream: "use vmauth for per-tenant authorization") — so a datasource header pin is a routing selector, never a security boundary. On a *shared* central backend, traces get the same weak, label-only cross-tenant read property that shared metrics/logs already have; making that a real boundary would require an authenticating proxy (vmauth) that forces the headers from identity and strips client-supplied ones — deferred as future work, not part of the MVP (see Resolved / Open questions). +Tracing enablement follows **how metrics actually work**, not the foundationdb `monitoring.enabled` example (which, per [Context](#context), gates a billing meta-resource and is the wrong model — @lllamnyp :31, :156). Metrics are configured by the platform for supported engines with no per-app opt-in flag; tracing aims for the same: for an engine Cozystack knows how to trace, enabling the tenant's tracing stack wires that engine's spans to the in-namespace collector. -### 4. Per-application opt-in +The universal-field problem (@lllamnyp :169) — "add the same `tracing` struct to every app" is not a clean solution, and simply copy-pasting a schema across charts is what we want to avoid. The design commitments here are about *mechanism*, and the concrete implementation is deliberately left to follow-up work: -Add a `tracing` struct to each participating app's `values.yaml` using the cozyvalues-gen annotation conventions, modelled on foundationdb's `monitoring.enabled` toggle and postgres's `backup` struct: +- **The endpoint is platform-decided, never app-decided.** An application does not carry an OTLP endpoint in its values (that would let it address another namespace and is the metrics analogy @lllamnyp drew): the platform points every traced engine at the tenant's in-namespace collector Service. +- **Enablement is a platform capability, not per-app config duplication.** Rather than stamping an identical `tracing` block into each Application spec, the enablement lives with the tenant's tracing stack; an app participates by virtue of being a supported engine, mapped once from the tenant/monitoring configuration down to the engine's HelmRelease. The exact mechanism for that Application→HelmRelease mapping — and which engines are in the first cut — is an [open question](#open-questions) to be resolved with a clean implementation, not by schema copy-paste. +- **How an engine emits spans** still varies: native OTLP where the engine supports it (e.g. ClickHouse `opentelemetry_span_log`, NATS), a sidecar/agent otherwise (Kafka, RabbitMQ, MariaDB, Redis, Postgres). That is an engine-integration detail, not a tenant-facing surface. -```yaml -## @typedef {struct} Tracing - OpenTelemetry (OTLP) tracing configuration. -## @field {bool} enabled - Enable OTLP trace export from this application. -## @field {string} [endpoint] - OTLP collector endpoint. Defaults to the platform collector in cozy-monitoring. -## @field {string} [samplingRatio] - Head-sampling ratio 0.0..1.0. Defaults to the platform policy. +### 5. Grafana datasource and correlation -## @param {Tracing} tracing - OpenTelemetry tracing configuration. -tracing: - enabled: false - endpoint: "" - samplingRatio: "" -``` +A `GrafanaDatasource` CR per backend, attached through `instanceSelector: { matchLabels: { dashboards: grafana } }` (the logs/metrics datasource pattern genuinely fits here). For VictoriaTraces the datasource reads its query API — upstream is stabilizing this as **Tempo Query-frontend-compatible HTTP APIs** (so a `type: tempo` datasource, shared with the Tempo fallback), with a Jaeger-compatible surface and a dedicated plugin also in the picture; which lands is a [confirm-before-implementation](#open-questions) item, since the correlation UX depends on it. Correlation: -How an app emits spans depends on the engine: +- **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. +- **Trace → metrics** and **metrics → trace**: Grafana only *links* to pre-existing metrics — it does not generate them. RED/span metrics need a producer: the collector's `spanmetrics` connector (recommended) or native app instrumentation, exporting to VictoriaMetrics; exemplars linking metric→trace require the metric producer to emit `trace_id` exemplars. +- **Logs → trace**: derived fields so a `trace_id` in a log opens the trace. -- **Native OTLP**, wired by chart config: ClickHouse (`opentelemetry_span_log`, currently disabled in `packages/apps/clickhouse/templates/clickhouse.yaml`) and NATS (native OTLP in recent versions). -- **Sidecar/agent OTLP**: Kafka and RabbitMQ (JVM/plugin agents), MariaDB, Redis and Postgres (an OpenTelemetry agent/exporter sidecar). The `tracing.enabled` toggle gates the sidecar and the `OTEL_EXPORTER_OTLP_ENDPOINT` env, defaulting to the platform collector. +Read-side datasource scoping follows the isolation model of [§2](#2-multitenancy-and-network-model-the-core-decision): in the default topology the datasource URL is pinned to the tenant's own in-namespace select service. -### 5. Data flow +### 6. Data flow ```mermaid flowchart LR - app["Managed app
(tracing.enabled)"] -- "OTLP/gRPC :4317
(stable Service)" --> svc["otel-traces Service
cozy-monitoring (port-explicit)"] - svc -- "Stage A: → vtinsert gRPC" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] - svc -. "Stage B: → collector" .-> col["OpenTelemetry Collector
(sampling / rate-limit / AccountID+ProjectID)"] - col -- OTLP --> vt - gr["Grafana"] -- Jaeger query --> vt - gr -. trace_id .-> vl["VictoriaLogs"] - gr -. span metrics .-> vm["VictoriaMetrics"] + subgraph tns["tenant-<x> namespace"] + app["Managed app
(supported engine)"] -- "OTLP (in-namespace)" --> col["OTLP Collector
(tenant trust boundary:
identity + sampling + rate-limit)"] + col -- "default: local backend" --> vt["Traces backend
(VTSingle / VTCluster)"] + end + col -. "opt-in: cross-boundary
(new Cilium egress rule)" .-> central["Shared central backend
tenant-root (+ vmauth on read)"] + gr["Grafana"] -- "trace query (Tempo-compat)" --> vt + gr -. "trace_id" .-> vl["VictoriaLogs"] + gr -. "span metrics" .-> vm["VictoriaMetrics"] ``` ## User-facing changes -- A new `tracingStorages` block in the monitoring values (system and per-tenant), with `retentionPeriod` defaulting to 14 days. -- A new per-app `tracing.*` block; off by default. -- A Traces datasource and Explore/Traces view in Grafana, with pivot links to logs and metrics. -- A docs entry point (`docs/observability/distributed-tracing.md` in `cozystack/cozystack`) covering how to enable tracing and point an app at the collector. +- A tracing backend selector and retention in the monitoring configuration (system and per-tenant), defaulting to VictoriaTraces / 14 days. +- Traces enabled per tenant (not per app); supported engines emit spans automatically once the tenant's tracing stack is on. +- A Traces datasource and Explore/Traces view in Grafana, with pivots to logs and metrics. +- A docs entry point (`docs/observability/distributed-tracing.md` in `cozystack/cozystack`). ## Upgrade and rollback compatibility -The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An **absent** `tracingStorages` renders no `VTCluster` and succeeds (tracing stays off); an **explicitly empty** list fails the render per the absent-vs-empty contract. No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). +Purely additive and opt-in. Existing clusters see no change until a tracing backend is configured. No data migration. Rollback is removing the tracing configuration; stored spans in the backend's PVCs are discarded on backend removal (irreversible for already-stored spans, like logs — flagged). The opt-in shared-central egress rule is only rendered when that topology is selected, so default clusters see no NetworkPolicy change. ## Security -- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. -- **Tenant attribution must be trusted**: `AccountID`/`ProjectID` (and the `tenant` attribute) must be injected from authenticated workload identity, never accepted verbatim from a tenant that could spoof another's IDs. On the shared backend this injection belongs at a boundary the tenant cannot bypass — the per-tenant proxy or the collector — not purely in tenant-controlled app config. -- **Isolation (write and read)**: write-side attribution alone is not isolation, and `vtselect` performs no per-tenant authorization — it trusts the `AccountID`/`ProjectID` read header verbatim, so pinning that header on a per-tenant Grafana datasource is a routing selector, not a security boundary. Read isolation is therefore delivered the same way logs/metrics deliver it today: **either** (a) an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource can only reach its own in-namespace `vtselect`, fenced by NetworkPolicy — the recommended default; **or** (b) an authenticating proxy (vmauth) in front of a shared `vtselect` that derives `AccountID`/`ProjectID` from authenticated identity and strips any client-supplied headers. A per-tenant datasource header pin is safe only in combination with (a) or (b), because the endpoint itself remains reachable and unauthenticated. -- **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. -- **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. -- **Pod Security Standards**: the new OTLP Collector Deployment and the per-app OTLP sidecars/agents (Kafka, RabbitMQ, MariaDB, Redis, Postgres) run inside `tenant-*` namespaces, which enforce PSS **restricted**. They must ship a compliant pod-security posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, and `seccompProfile.type: RuntimeDefault` — like every other new workload class in this stack; no privileged or host-namespace access is required for OTLP ingest. +- **New tenant-supplied input:** the OTLP endpoint accepts spans from tenant workloads. The per-tenant collector is the trust boundary — it enforces tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust a shared backend. +- **Tenant attribution must be trusted:** `AccountID`/`ProjectID` (and the `tenant` attribute) are injected at the collector from the identity it is deployed with, never accepted from tenant-controlled app config. +- **Isolation (write and read):** the default per-tenant topology isolates physically (nothing leaves the namespace); the shared-central topology requires the new egress rule for writes and an authenticating vmauth proxy for reads, because the backend's select component performs no per-tenant authorization on its own. +- **Network policy:** the default path needs no change to Cilium policy; the shared-central path adds exactly one narrow, tracing-gated egress rule, described in [§2](#2-multitenancy-and-network-model-the-core-decision). +- **Transport:** OTLP endpoints should be TLS-terminated; align with `design-proposals/unified-tls-pki` rather than minting bespoke certs. +- **RBAC:** the new backend/datasource/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents have. +- **Pod Security Standards:** the OTLP Collector Deployment and any per-engine OTLP sidecars run in `tenant-*` namespaces, which enforce PSS **restricted**. They must ship a compliant posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `seccompProfile.type: RuntimeDefault`; no privileged or host-namespace access is required for OTLP ingest. ## Failure and edge cases -- `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. -- OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk, so an age-only config lets a full PVC block ingest *before* old traces are evicted. The default is therefore **always disk-bounded**: `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) is set from the byte-quantity `retentionDiskUsageBytes` values field when given, and otherwise **derived from `storageSize` (~85%)** so no entry ever ships without a disk bound — matching the "uncrashable default, don't just document the knife" spirit of the `#3181` guard. A PVC/disk-usage capacity alert is added on top, and the behaviour is covered by a PVC-exhaustion test. -- `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. -- App emits OTLP but no backend deployed → the `otel-traces` Service has no backing endpoints, so the exporter's connection fails and it drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. -- App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. +- No tracing backend configured → tracing disabled, renders nothing, cluster stays healthy. A backend requested but left empty → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. +- OTLP endpoint unreachable (collector/backend down) → the app's exporter drops/retries per OTLP defaults; the app does not crash and serving is unaffected. +- Storage full → the backend's retention must be **disk-bounded, not only age-bounded**, so a full PVC cannot block ingest before old traces are evicted; a capacity alert is added on top. (The specific retention field is an implementation detail.) +- Tenant with tracing off → no collector, no sidecar, no CR: zero overhead. +- Missing tenant identity on a shared backend → spans would land in the default tenant; the collector's write-boundary injection prevents this by construction. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set, and that when it is unset the field is still rendered, derived from `storageSize` (~85%), so no rendered `VTCluster`/`VTSingle` is ever disk-unbounded. -- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. -- **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. +- **Unit/lint:** `helm template` + `helm lint` for the backend, collector, and datasource; assert they render only when tracing is configured and that a requested-but-empty backend fails the render; assert the rendered backend is always disk-bounded. +- **e2e** (Chainsaw): deploy the tenant tracing stack, enable a native-OTLP engine (e.g. ClickHouse), generate activity, assert a trace is queryable and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); assert app→collector works with **no** NetworkPolicy change, and that the shared-central egress rule is required for the cross-boundary hop. +- **Manual:** verify trace→logs and (with the spanmetrics connector) trace→metrics pivots in Grafana. ## Rollout -1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the port-explicit `otel-traces:4317` gRPC redirect Service in `packages/system/cozystack-basics` (targeting `vtinsert`'s gRPC listener). Apps can push OTLP/gRPC directly. No collector yet. -2. **Grafana**: traces datasource + correlation links. -3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. -4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the `otel-traces:4317` Service from `vtinsert` to the collector (no app change for gRPC clients). Ships sampling/rate-limiting and `AccountID`/`ProjectID` header injection via `headers_setter`. -5. **Docs**: enablement guide under `docs/observability/`. +1. **Per-tenant backend + collector:** the tracing backend and OTLP Collector inside the tenant stack (`packages/extra/monitoring` and the root path via `packages/system/monitoring`), with the poller readiness gate. Apps push OTLP to the in-namespace collector; no NetworkPolicy change. +2. **Grafana:** traces datasource + correlation links. +3. **Engine integrations:** native-OTLP engines first (ClickHouse, NATS), then sidecar-based (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per engine. +4. **Shared-central topology (opt-in):** add the narrow Cilium egress rule and, for reads, the vmauth authenticating proxy. +5. **Docs:** enablement guide under `docs/observability/`. ## Open questions -- **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. -- **Confirm-before-implementation (upstream VictoriaTraces surface)**: the `VTCluster` spec keys (`insert`/`select`/`storage`) and the disk-cap field (`retentionMaxDiskSpaceUsageBytes`) are verified against the pinned operator CRD. Still taken on upstream faith and to be confirmed against the pinned VictoriaTraces version before building: the Grafana datasource type (`type: jaeger` against `vtselect`, or the dedicated VictoriaTraces plugin — the whole correlation UX depends on which lands); the OTLP wire details (`vtinsert` HTTP `:10481` + `/insert/opentelemetry/v1/traces`, gRPC via `-otlpGRPCListenAddr`); and the exact rendered service-name prefix (`vtinsert-*`/`vtselect-*` vs `insert-*`). -- **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. -- **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? -- **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? -- **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. - -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). Both `VTCluster` and `VTSingle` are supported via a per-entry `mode` field (default `cluster`, matching how `metricsStorages`/`logsStorages` render the cluster variant in every stack today; `single` for edge/dev/small). Read-side tenant isolation is **structural**, exactly as for logs/metrics: account `0` everywhere, isolation by an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource reaches only its own in-namespace `vtselect`, fenced by NetworkPolicy — not a query-time tenant filter, since `vtselect` has no per-tenant authorization and trusts the `AccountID`/`ProjectID` header verbatim. A shared central backend would need vmauth (forcing headers from authenticated identity, stripping client-supplied ones) to become a real read boundary; that is deferred future work, not the MVP. +- **Backend maturity:** VictoriaTraces is **pre-GA** upstream (data structure/backward-compat not yet frozen; Tempo Query-frontend-compatible query API still landing). Is it production-ready at the version Cozystack pins, or does the first cut ship on Tempo and switch to VictoriaTraces once GA? Only the backend CR and datasource type change either way. +- **Application→HelmRelease enablement mechanism:** what is the clean way to map a tenant's tracing enablement onto supported engines' HelmReleases without copy-pasting a `tracing` schema into every chart, and which engines are in the first cut? +- **Sampling default:** head sampling at the source (scales freely) vs. tail sampling at the collector (needs trace affinity)? +- **Durability:** VictoriaTraces cluster mode does not replicate spans across storage nodes; is single-backend durability acceptable, or is collector replication into two independent backends warranted? +- **External OTLP exposure:** should tenants push spans from outside the cluster, and through which ingress/Gateway path? (This is where an internal Gateway/`HTTPRoute` would earn its place — see [§3](#3-ingest-topology-and-routing).) +- **Confirm-before-implementation (upstream backend surface):** the Grafana datasource type for VictoriaTraces (Tempo Query-frontend-compatible API, expected primary, vs. Jaeger-compatible surface vs. dedicated plugin) and the exact OTLP wire/CRD field details — settled against the pinned version in the implementation PR. ## Alternatives considered -- **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap. -- **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have. -- **No collector, app → backend directly** (ingest): adopted as **Stage A**, not rejected — `vtinsert` is OTLP-native, so direct ingest is the fastest correct MVP and an exact mirror of the logs path. Its limitations (no shared trust boundary, per-tenant attribution, or central sampling/rate-limiting) are exactly what **Stage B**'s collector gateway adds later, without changing the app-facing endpoint. -- **Collector as a DaemonSet agent** (ingest topology): rejected — that node-local shape fits fluent-bit tailing log files, but OTLP traces are pushed over the network by the apps themselves, so a centralized gateway Deployment (the `vtinsert`/vmagent role) is the right shape. -- **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise. +- **Mirror the logs/metrics stack one-for-one** (framing): rejected as the *guiding principle* (@lllamnyp :61). The scrape-based network model of metrics/logs does not transfer to push-based OTLP, so copying it wholesale would have designed straight into the tenant-isolation problem in [§2](#2-multitenancy-and-network-model-the-core-decision). Conventions are reused only where they genuinely fit. +- **Single-vendor VictoriaTraces, no backend seam** (backend): rejected in favor of the pluggable OTLP/datasource contract (@lllamnyp :33). VictoriaTraces remains the default (no new operator), but Tempo and Jaeger are selectable, and the choice never leaks into applications or the tenancy design. +- **Central OTLP endpoint apps push to directly** (ingest/tenancy): rejected — blocked by Cilium tenant isolation and would require a broad egress hole. The per-tenant in-namespace collector avoids the breach entirely. +- **Per-app `tracing.enabled` toggle modeled on foundationdb** (enablement): rejected (@lllamnyp :31, :156) — foundationdb's toggle gates a billing meta-resource, and metrics enablement is not per-app in the first place. Tracing follows the platform-configured metrics model. +- **Internal Ingress / Gateway-API `HTTPRoute` for the app-facing hop** (routing): rejected for the default path (@lllamnyp :130) — the app→collector hop is intra-namespace, so an L7 hop adds cost without benefit and fits OTLP/gRPC poorly. Retained as the right tool for the shared-central front-end and external exposure. +- **Collector as a DaemonSet agent** (topology): rejected — the node-local shape fits fluent-bit tailing files, but OTLP is pushed over the network, so a gateway Deployment is correct. +- **Always-on tracing** (enablement default): rejected — tracing overhead and storage cost stay a tenant's explicit choice. + +--- + +## Implementation notes (non-normative) + +These are pointers for the implementation PRs, **not** part of the design contract; exact names/values are confirmed against the pinned versions there. VictoriaTraces `VTCluster` decomposes into insert/select/storage (spec keys drop the `vl`-style prefix); rendered workloads/services are expected to carry a `vt` prefix. `vtinsert` defaults to OTLP/HTTP on `:10481` at `/insert/opentelemetry/v1/traces`, with OTLP/gRPC enabled via `-otlpGRPCListenAddr` — so gRPC (path-less) is the more portable app-facing contract. Disk-bounded retention uses the CRD's disk-space field (a byte quantity), defaulted from the PVC size when unset. Tenant routing on a shared backend uses `AccountID`/`ProjectID` **request headers only** (there is no query-string precedence rule for VictoriaTraces/VictoriaLogs — that is a different VictoriaMetrics-cluster path mechanism); on a shared backend the collector's `headers_setter` extension (`from_context`, receiver `include_metadata: true`) injects them, since static exporter headers cannot vary per tenant. From 648eb8f0e1cc0f60b60c0105acb05fa86e567055 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 31 Jul 2026 18:46:19 +0300 Subject: [PATCH 12/14] Address 2026-07-30 review: platform facts, PII, sampling, drop review citations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documentation pass on the tracing proposal per @lllamnyp's 2026-07-30 review (architecture accepted; these are fact/coverage fixes): - Context: correct the agents' write target — chart default is cozy-monitoring, the platform bundle overrides global.target to tenant-root (bundles/system.yaml); resolve the alias self-contradiction. - Context: mark the Cilium egress list as a non-exhaustive snapshot; fix apiserver (label-gated) and ingress (any namespace) specifics. - Security: PSS restricted is NOT enforced on tenant namespaces — make restricted-compliance a design requirement of the new workloads. - Security: add span-content / PII handling (collector redaction as the control point; who can read spans on a shared backend) + tests. - §4: honest per-engine breakdown — NATS is client-side (not native OTLP); ClickHouse span_log is disabled and export needs wiring; Postgres statement-level needs pg_tracing/client, not a sidecar. - §3: concrete default (10% head sampling) + rough volume model. - Editorial: drop all unresolvable '@lllamnyp :NN' review citations (as dd44d1c did before the reframe reintroduced them). - Non-blocking: seam 'plus operational wiring'; shared-central egress endpoint-selects only the collector; future-proof note for the unwired monitoring.yaml PackageSource; grammar fix. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 59 ++++++++++++------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 472b983..d4e48b7 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -2,7 +2,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` (revised 2026-07-29) +- **Date:** `2026-07-16` (revised 2026-07-31) - **Status:** Review ## Overview @@ -31,15 +31,15 @@ In scope: a traces backend (per-tenant and, as an opt-in, shared-central), an OT Cozystack's observability is multi-tenant with a central backend, and it crosses the tenant boundary in a very specific way that traces cannot simply copy. -**Where the backend runs.** The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana, Alerta, vmalert) is deployed by the *root* tenant into `tenant-root` (`packages/apps/tenant/templates/monitoring.yaml`). The `cozy-monitoring` namespace holds only `ExternalName` aliases (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) that CNAME to the real services in `tenant-root` (`packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, gated on `_cluster.monitoring-enabled`). The cluster-wide agents (vmagent, fluent-bit, kube-state-metrics, node-exporter) live in `cozy-monitoring` (`packages/system/monitoring-agents`). +**Where the backend runs.** The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana, Alerta, vmalert) is deployed by the *root* tenant into `tenant-root` (`packages/apps/tenant/templates/monitoring.yaml`). The `cozy-monitoring` namespace holds `ExternalName` aliases (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) that CNAME to the real services in `tenant-root` (`packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, gated on `_cluster.monitoring-enabled`). The cluster-wide agents (vmagent, fluent-bit, kube-state-metrics, node-exporter) live in `cozy-monitoring` (`packages/system/monitoring-agents`). (A currently-unwired `packages/core/platform/sources/monitoring.yaml` PackageSource could instead install monitoring *into* `cozy-monitoring`; if that mode ever lands, the alias topology here and the ancestor-based shared-central egress in [§2](#2-multitenancy-and-network-model-the-core-decision) both need revisiting.) -**How metrics and logs cross the tenant boundary — by *scraping*, not by tenants pushing out.** vmagent runs centrally in `cozy-monitoring` and scrapes targets cluster-wide, then remote-writes to `vminsert-*.tenant-root.svc`. fluent-bit runs as a DaemonSet in `cozy-monitoring` and ships to `vlinsert-generic.tenant-root.svc`. Both cross into `tenant-root` because `cozy-monitoring` carries the `cozystack.io/system: "true"` label (stamped by the operator, `internal/operator/package_reconciler.go`) and `tenant-root`'s Cilium ingress policy trusts every `cozystack.io/system` namespace. **This is an ingress trust granted to a system namespace — it does not let a tenant workload egress toward monitoring.** +**How metrics and logs cross the tenant boundary — by *scraping*, not by tenants pushing out.** vmagent and fluent-bit run centrally in `cozy-monitoring`: vmagent scrapes targets cluster-wide and remote-writes, fluent-bit tails node logs and ships them. Their destination is `global.target`, which the chart defaults to `cozy-monitoring` but the platform bundle overrides to `tenant-root` (`packages/core/platform/templates/bundles/system.yaml`) — so the agents write to `vminsert-*` / `vlinsert-generic`, resolved either directly in `tenant-root` or through the `cozy-monitoring` ExternalName aliases that CNAME there (the aliases are the chart-default path and what any `cozy-monitoring`-local reference resolves through). Either way the write crosses into `tenant-root` because `cozy-monitoring` carries the `cozystack.io/system: "true"` label (stamped by the operator, `internal/operator/package_reconciler.go`) and `tenant-root`'s Cilium ingress policy trusts every `cozystack.io/system` namespace. **This is an ingress trust granted to a system namespace — it does not let a tenant workload egress toward monitoring.** -**Tenant network isolation (Cilium).** `packages/apps/tenant/templates/networkpolicy.yaml` renders `CiliumClusterwideNetworkPolicy` per tenant; the posture is default-deny once selected. A tenant pod's egress is limited to: its own tenant subtree; endpoints labeled `app.kubernetes.io/name: vminsert` / `app.kubernetes.io/instance: etcd` / `cozystack.io/service: ingress` in **ancestor** namespaces; DNS; a fixed set of `cozy-*` namespaces (`cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-kubevirt-cdi`); and `world` (out-of-cluster). Notably **`cozy-monitoring` is not on that list**, and a generic pod in `tenant-root` is not reachable either — only the specifically-labeled `vminsert`/`etcd`/`ingress` endpoints are. A tenant that runs its own monitoring gets a per-tenant vmagent *inside its own namespace* that pushes up to the parent's `vminsert` — that is exactly the `vminsert`-labeled ancestor egress rule. +**Tenant network isolation (Cilium).** `packages/apps/tenant/templates/networkpolicy.yaml` renders `CiliumClusterwideNetworkPolicy` per tenant; the posture is default-deny once selected. A tenant pod's egress is limited to an explicit allow-list — the snapshot below is **representative, not exhaustive** (the file is authoritative; the list has grown over time): its own tenant subtree; endpoints labeled `app.kubernetes.io/name: vminsert` / `app.kubernetes.io/instance: etcd` in **ancestor** namespaces; `cozystack.io/service: ingress` endpoints in **any** namespace; DNS; a **label-gated** `kube-apiserver` egress (pods carrying `policy.cozystack.io/allow-to-apiserver: "true"`); a set of shared `cozy-*` namespaces (`cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-kubevirt-cdi`, …); and `world` (out-of-cluster). The load-bearing fact for §2 is stable under any addition to this list: **`cozy-monitoring` is not on it**, and a generic pod in `tenant-root` is not reachable either — only the specifically-labeled `vminsert`/`etcd` endpoints are. A tenant that runs its own monitoring gets a per-tenant vmagent *inside its own namespace* that pushes up to the parent's `vminsert` — that is exactly the `vminsert`-labeled ancestor egress rule. **Application observability today.** Apps expose metrics out of the box: the charts render `VMServiceScrape`/`VMPodScrape` (or CNPG's `enablePodMonitor`) unconditionally — there is no per-app "enable metrics" toggle. Most engines also declare a `WorkloadMonitor` CR, but `WorkloadMonitor` is **not** a metrics-collection mechanism: it is a billing/ownership meta-resource holding a label selector that identifies the pods, services, and PVCs belonging to an app, reconciled into `Workload` objects (replicas/resources/status) for the dashboard and billing surfaces (`internal/controller/workloadmonitor_controller.go`). The one app that gates a `WorkloadMonitor` behind a tenant-facing `monitoring.enabled` flag is foundationdb — and that is a mistake we should not copy (it hides a billing meta-resource behind a tenant toggle), not a template for a tracing switch. -**No new operator is needed for the default backend.** The victoria-metrics-operator Cozystack already runs (`packages/system/victoria-metrics-operator`) ships the `VTCluster` and `VTSingle` CRDs (`operator.victoriametrics.com/v1`), which decompose into insert/select/storage components analogous to `VLCluster`. +**No new operator is needed for the default backend.** The victoria-metrics-operator that Cozystack already runs (`packages/system/victoria-metrics-operator`) ships the `VTCluster` and `VTSingle` CRDs (`operator.victoriametrics.com/v1`), which decompose into insert/select/storage components analogous to `VLCluster`. ### The problem @@ -47,6 +47,8 @@ Cozystack's observability is multi-tenant with a central backend, and it crosses - "I have a log line for a failed request and a latency spike on a dashboard, but no way to jump from either to the actual request span." The signals don't correlate. - "My application already emits OTLP spans, but Cozystack gives me nowhere to send them." There is no OTLP endpoint and no backend. +The first scenario is the aspiration, but honesty about depth matters up front: statement-level DB visibility is not something the platform can conjure with a sidecar — it depends on an in-engine extension or client-side instrumentation. What the platform *guarantees* is the transport, storage, tenancy, and correlation; how deep each engine can see is spelled out per engine in [§4](#4-per-application-enablement). + ## Goals - Accept traces over **OTLP** (the standard cloud-native tracing protocol) at an endpoint the application can reach **without breaching tenant network isolation**. @@ -67,13 +69,13 @@ Cozystack's observability is multi-tenant with a central backend, and it crosses ### 1. Backend: pluggable, VictoriaTraces default -OTLP ingest and a Grafana datasource form a **backend-agnostic contract**: the collector speaks OTLP outward, and Grafana reads through a datasource. Only two things change when the backend changes — the storage CR the platform renders, and the Grafana datasource type. Everything else in this design (the collector, the tenancy model, the correlation wiring, the per-app surface) is unaffected. The platform therefore exposes a **backend selector**, defaulting to VictoriaTraces: +OTLP ingest and a Grafana datasource form a **backend-agnostic contract**: the collector speaks OTLP outward, and Grafana reads through a datasource. What changes when the backend changes is mostly the storage CR the platform renders and the Grafana datasource type — plus the backend-specific operational wiring (the readiness gate's `healthCheckExprs`, the retention knobs, which differ between PVC and object-storage backends, and the read-side auth proxy). What is *unaffected* is the load-bearing part: the collector, the tenancy model, the correlation wiring, and the per-app surface. The platform therefore exposes a **backend selector**, defaulting to VictoriaTraces: - **VictoriaTraces** (default): VictoriaMetrics' own trace backend — the same-vendor counterpart to Tempo, on the same `victoria-metrics-operator` as VictoriaMetrics/VictoriaLogs. `VTCluster` for production, `VTSingle` (single binary) for edge/dev/small clusters where a multi-component cluster is unwarranted. Chosen as default because its CRDs already ship in the operator Cozystack runs — no new operator, one operational model shared with metrics/logs. (The `VTSingle`/`VTCluster` split is a mode the metrics and logs stacks would benefit from adopting too.) **Caveat: VictoriaTraces is pre-GA** per its upstream roadmap (data structure/backward-compat not yet frozen; the Grafana-facing query API is being delivered as Tempo Query-frontend-compatible HTTP APIs) — which is exactly why the backend seam and the Tempo fallback below matter, and why maturity is an [open question](#open-questions). - **Grafana Tempo**: object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs) with the strongest Grafana-native correlation. The recommended fallback if VictoriaTraces is not ready at the pinned version. Because VictoriaTraces is converging on a Tempo-compatible query API, both can share the same Grafana datasource type — making this fallback nearly seamless. - **Jaeger**: mature and OTLP-native, but its own UI and weaker Grafana integration cut against single-pane correlation. -The vendor-neutrality question @lllamnyp raised is answered by *this* seam, not by adding operators: an admin picks a backend at install time; the OTLP/datasource contract keeps the choice from leaking into applications or the tenancy design. See [Alternatives](#alternatives-considered) for the full trade-off. +Vendor neutrality is answered by *this* seam, not by adding operators: an admin picks a backend at install time; the OTLP/datasource contract keeps the choice from leaking into applications or the tenancy design. See [Alternatives](#alternatives-considered) for the full trade-off. The backend is provisioned by the operator, and the monitoring HelmRelease gates readiness on the storage CR reaching `operational` (the `waitStrategy: poller` + `healthCheckExprs` pattern `VLCluster` already uses in `packages/extra/monitoring/templates/helmrelease.yaml`). Without that gate the release flips Ready before the backend can accept writes — the silent-black-hole failure that motivated `cozystack/cozystack#3181` for logs. Tracing is opt-in: a cluster that configures no tracing backend renders nothing and stays healthy; a backend that is *requested but left empty* is a misconfiguration and fails the render. (The exact values shape is an implementation detail — [Implementation notes](#implementation-notes-non-normative).) @@ -81,7 +83,7 @@ The backend is provisioned by the operator, and the monitoring HelmRelease gates This is where tracing genuinely departs from metrics and logs, and where the design must be explicit rather than deferred. -**Why the metrics/logs model does not transfer.** Metrics and logs cross the tenant boundary because central agents *scrape/tail* workloads and forward — the tenant never egresses toward monitoring (see [Context](#context)). OTLP is the opposite: the **application pushes** spans. Under the current Cilium policy a tenant pod *cannot* reach a collector in `cozy-monitoring` (not in the egress allow-list) nor a generic collector in `tenant-root` (only `vminsert`/`etcd`/`ingress`-labeled endpoints are reachable). So a naive "app → central OTLP endpoint" design would either be blocked by policy or require punching a broad hole in tenant isolation. That is precisely the concern @lllamnyp flagged, and it is real. +**Why the metrics/logs model does not transfer.** Metrics and logs cross the tenant boundary because central agents *scrape/tail* workloads and forward — the tenant never egresses toward monitoring (see [Context](#context)). OTLP is the opposite: the **application pushes** spans. Under the current Cilium policy a tenant pod *cannot* reach a collector in `cozy-monitoring` (not in the egress allow-list) nor a generic collector in `tenant-root` (only `vminsert`/`etcd`/`ingress`-labeled endpoints are reachable). So a naive "app → central OTLP endpoint" design would either be blocked by policy or require punching a broad hole in tenant isolation. **Decision: the collector lives inside the tenant namespace.** Each participating tenant gets an OpenTelemetry Collector Deployment in its *own* namespace — the same placement as a per-tenant vmagent. Applications push OTLP to that in-namespace collector, so app→collector traffic is intra-namespace and covered by the existing `allow-internal-communication` policy: **zero NetworkPolicy change, and tenant isolation is never breached.** The collector is the per-tenant trust boundary: it stamps the tenant's identity, applies sampling and rate-limits, and forwards onward. What it forwards *to* defines the two supported topologies: @@ -94,7 +96,7 @@ This is where tracing genuinely departs from metrics and logs, and where the des | Read isolation | physical: datasource → in-namespace select service, fenced by policy | needs an authenticating proxy (vmauth); a bare shared select is **not** a boundary | | When to choose | default; strongest isolation | central retention/cost consolidation, accepted trade-off | -**The shared-central egress rule (designed here, not deferred).** For the opt-in shared topology, the collector→central-backend hop needs an explicit allowance — modeled exactly on the existing ancestor-`vminsert` egress block in `packages/apps/tenant/templates/networkpolicy.yaml`: add a `toEndpoints` rule matching the central OTLP-ingest endpoint's label (e.g. `app.kubernetes.io/name: `) in the ancestor namespace, gated on the tenant having tracing enabled. This is a deliberate, narrow widening of the tenant egress surface, and it is a security decision the operator opts into — not a silent default. We explicitly **reject** the alternative of mislabeling the collector as `vminsert` to ride the existing rule: that is a label hack that erodes the meaning of the policy. +**The shared-central egress rule (designed here, not deferred).** For the opt-in shared topology, the collector→central-backend hop needs an explicit allowance — modeled on the existing ancestor-`vminsert` egress block in `packages/apps/tenant/templates/networkpolicy.yaml`, but tightened on **both** ends: its `toEndpoints` matches the central OTLP-ingest endpoint's label (e.g. `app.kubernetes.io/name: `) in the ancestor namespace, and — unlike the `vminsert` block, whose `endpointSelector` selects *all* tenant pods — its `endpointSelector` matches **only the collector pods** (the way the `{{- if .Values.monitoring }}` virt-handler rule selects just vmagent), so only the collector, not every workload, can cross the boundary. It is gated on the tenant having tracing enabled. This is a deliberate, narrow widening of the tenant egress surface, and a security decision the operator opts into — not a silent default. We explicitly **reject** the alternative of mislabeling the collector as `vminsert` to ride the existing rule: that is a label hack that erodes the meaning of the policy. **Identity, write and read.** - *Write:* `AccountID`/`ProjectID` (and any `tenant` resource attribute) are injected by the per-tenant collector from the identity it is deployed with — never from tenant-controlled application config, which a tenant could spoof. Because the collector is per-tenant, it already knows its tenant. @@ -108,19 +110,29 @@ Applications target a **stable in-namespace endpoint** — the per-tenant collec **A gateway Deployment, not a DaemonSet.** OTLP is push-based over the network — the applications are the agents — so the collector plays the centralized `vmagent`/insert role, not the node-local fluent-bit role. -**Tail sampling needs trace affinity.** If tail sampling is used, all spans of a trace must reach one collector instance; a plain Service round-robins and would sample on partial traces. So the sampling tier runs single-replica, or behind a first tier using consistent-hash routing by trace ID. Head sampling has no such constraint. The default sampling policy is an [open question](#open-questions). +**Sampling and volume.** Traces are the burstiest signal, and a busy app can fill the disk cap in hours if every span is kept — bounded, but so short-lived it is useless. So tracing ships with a **non-trivial default: head sampling at 10%** (`0.1`), applied at the collector, with the ratio exposed as a knob. As a rough order-of-magnitude for sizing (not a guarantee — span size varies with attributes): stored spans land around a few hundred bytes each, so ~1,000 spans/s sustained is on the order of tens of GiB/day at 100%, i.e. a small default PVC would fill in about a day unsampled and roughly ten days at the 10% default. Head sampling scales freely and is the Stage-1 default; **tail sampling** (keep-errors/keep-slow) is more useful but needs trace affinity — all spans of a trace must reach one collector instance, so the sampling tier runs single-replica or behind a first tier using consistent-hash routing by trace ID. Tail sampling is deferred to the collector-gateway step; the head/tail trade-off is tracked in [open questions](#open-questions). -**Routing: why an in-namespace Service, and the Ingress/Gateway alternative (@lllamnyp :130).** Because the collector is *in the tenant namespace*, the app-facing hop is plain in-cluster service traffic — an Ingress or Gateway-API `HTTPRoute` would add an L7 hop with no benefit for intra-namespace OTLP, and OTLP/gRPC in particular is a poor fit for a typical HTTP Ingress. An internal Gateway/`HTTPRoute` becomes relevant only for the shared-central topology (a single in-cluster address fronting the central backend) or for exposing OTLP to workloads *outside* the cluster; both are folded into the shared-backend egress design and the external-exposure [open question](#open-questions) rather than the default path. If Cozystack standardizes on Gateway API for in-cluster L7, the shared-central front-end should use an `HTTPRoute` there rather than a bespoke proxy. +**Routing: why an in-namespace Service, and the Ingress/Gateway alternative.** Because the collector is *in the tenant namespace*, the app-facing hop is plain in-cluster service traffic — an Ingress or Gateway-API `HTTPRoute` would add an L7 hop with no benefit for intra-namespace OTLP, and OTLP/gRPC in particular is a poor fit for a typical HTTP Ingress. An internal Gateway/`HTTPRoute` becomes relevant only for the shared-central topology (a single in-cluster address fronting the central backend) or for exposing OTLP to workloads *outside* the cluster; both are folded into the shared-backend egress design and the external-exposure [open question](#open-questions) rather than the default path. If Cozystack standardizes on Gateway API for in-cluster L7, the shared-central front-end should use an `HTTPRoute` there rather than a bespoke proxy. ### 4. Per-application enablement -Tracing enablement follows **how metrics actually work**, not the foundationdb `monitoring.enabled` example (which, per [Context](#context), gates a billing meta-resource and is the wrong model — @lllamnyp :31, :156). Metrics are configured by the platform for supported engines with no per-app opt-in flag; tracing aims for the same: for an engine Cozystack knows how to trace, enabling the tenant's tracing stack wires that engine's spans to the in-namespace collector. +Tracing enablement follows **how metrics actually work**, not the foundationdb `monitoring.enabled` example (which, per [Context](#context), gates a billing meta-resource and is the wrong model). Metrics are configured by the platform for supported engines with no per-app opt-in flag; tracing aims for the same: for an engine Cozystack knows how to trace, enabling the tenant's tracing stack wires that engine's spans to the in-namespace collector. -The universal-field problem (@lllamnyp :169) — "add the same `tracing` struct to every app" is not a clean solution, and simply copy-pasting a schema across charts is what we want to avoid. The design commitments here are about *mechanism*, and the concrete implementation is deliberately left to follow-up work: +The universal-field problem — "add the same `tracing` struct to every app" is not a clean solution, and simply copy-pasting a schema across charts is what we want to avoid. The design commitments here are about *mechanism*, and the concrete implementation is deliberately left to follow-up work: -- **The endpoint is platform-decided, never app-decided.** An application does not carry an OTLP endpoint in its values (that would let it address another namespace and is the metrics analogy @lllamnyp drew): the platform points every traced engine at the tenant's in-namespace collector Service. +- **The endpoint is platform-decided, never app-decided.** An application does not carry an OTLP endpoint in its values (that would let it address another namespace, and it mirrors how metrics work — the scrape target is not something the app declares): the platform points every traced engine at the tenant's in-namespace collector Service. - **Enablement is a platform capability, not per-app config duplication.** Rather than stamping an identical `tracing` block into each Application spec, the enablement lives with the tenant's tracing stack; an app participates by virtue of being a supported engine, mapped once from the tenant/monitoring configuration down to the engine's HelmRelease. The exact mechanism for that Application→HelmRelease mapping — and which engines are in the first cut — is an [open question](#open-questions) to be resolved with a clean implementation, not by schema copy-paste. -- **How an engine emits spans** still varies: native OTLP where the engine supports it (e.g. ClickHouse `opentelemetry_span_log`, NATS), a sidecar/agent otherwise (Kafka, RabbitMQ, MariaDB, Redis, Postgres). That is an engine-integration detail, not a tenant-facing surface. + +**How an engine emits spans varies, and the level of visibility differs per engine** — this is worth an honest breakdown rather than a flat "native vs. sidecar" split, because a sidecar cannot see *inside* a process: + +| Engine | Span production | Reality | +|---|---|---| +| ClickHouse | native (`opentelemetry_span_log` system table) | span *production* is native, but Cozystack currently **disables** it (`clickhouse.yaml` renders ``), and *export* still needs wiring — a shipper or materialized-view → OTLP pipeline. The integration PR must re-enable the log and add the exporter. | +| Postgres | needs an in-engine extension (e.g. `pg_tracing`) or client-side instrumentation | a sidecar **cannot** produce statement-execution spans from outside the process. Statement-level visibility (the flagship "which statement" case) requires the extension or app-side tracing, not an agent. | +| NATS | client-side instrumentation | OTLP tracing for NATS is emitted by instrumented clients (context propagated in message headers); NATS 2.11 server-side message tracing emits **NATS-format** events to a subject, **not** OTLP. Not a server-native OTLP source. | +| Kafka, RabbitMQ, MariaDB, Redis | client-side / proxy / JVM-agent | broker- or protocol-level spans via an agent or instrumented client; depth depends on the engine. | + +The takeaway: the platform delivers the transport, storage, tenancy, and correlation; the *depth* of what an engine can emit is an engine-integration property, and statement-level DB tracing in particular is an extension/client concern, not something a sidecar conjures. ### 5. Grafana datasource and correlation @@ -160,12 +172,13 @@ Purely additive and opt-in. Existing clusters see no change until a tracing back ## Security - **New tenant-supplied input:** the OTLP endpoint accepts spans from tenant workloads. The per-tenant collector is the trust boundary — it enforces tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust a shared backend. +- **Span content (PII / secrets):** spans carry application data, and the flagship slow-SQL case will reliably put statement text — literal values, bound parameters, sometimes connection strings — into span attributes. Redaction is owned at the **collector**: the per-tenant collector is the control point for an `attributes`/`redaction`/`transform` processor that masks or drops sensitive keys before export, shipped with a conservative default and a per-tenant override. On the default per-tenant backend, span content never leaves the tenant namespace; on a shared-central backend, whoever can query the tenant's spans can read that content, so the read-isolation controls below (per-tenant stack, or vmauth) are also the PII boundary. The engine-integration PRs must document what each engine emits so redaction defaults can be set sensibly. - **Tenant attribution must be trusted:** `AccountID`/`ProjectID` (and the `tenant` attribute) are injected at the collector from the identity it is deployed with, never accepted from tenant-controlled app config. - **Isolation (write and read):** the default per-tenant topology isolates physically (nothing leaves the namespace); the shared-central topology requires the new egress rule for writes and an authenticating vmauth proxy for reads, because the backend's select component performs no per-tenant authorization on its own. - **Network policy:** the default path needs no change to Cilium policy; the shared-central path adds exactly one narrow, tracing-gated egress rule, described in [§2](#2-multitenancy-and-network-model-the-core-decision). - **Transport:** OTLP endpoints should be TLS-terminated; align with `design-proposals/unified-tls-pki` rather than minting bespoke certs. - **RBAC:** the new backend/datasource/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents have. -- **Pod Security Standards:** the OTLP Collector Deployment and any per-engine OTLP sidecars run in `tenant-*` namespaces, which enforce PSS **restricted**. They must ship a compliant posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `seccompProfile.type: RuntimeDefault`; no privileged or host-namespace access is required for OTLP ingest. +- **Pod Security Standards:** the OTLP Collector Deployment and any per-engine OTLP sidecars run in `tenant-*` namespaces. Cozystack does **not** currently *enforce* PSS `restricted` there — the tenant namespace template stamps no `pod-security.kubernetes.io/*` labels, and the cluster-wide default is distro-dependent (`baseline` on Talos, unlabeled on vanilla kubeadm/kind/k3s). So restricted-compliance is a **design requirement of these new workloads**, not an existing platform guarantee to lean on: they must ship a compliant posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `seccompProfile.type: RuntimeDefault` — so they are safe regardless of the namespace's enforcement level. No privileged or host-namespace access is required for OTLP ingest. ## Failure and edge cases @@ -179,13 +192,15 @@ Purely additive and opt-in. Existing clusters see no change until a tracing back - **Unit/lint:** `helm template` + `helm lint` for the backend, collector, and datasource; assert they render only when tracing is configured and that a requested-but-empty backend fails the render; assert the rendered backend is always disk-bounded. - **e2e** (Chainsaw): deploy the tenant tracing stack, enable a native-OTLP engine (e.g. ClickHouse), generate activity, assert a trace is queryable and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); assert app→collector works with **no** NetworkPolicy change, and that the shared-central egress rule is required for the cross-boundary hop. +- **Redaction / PII:** assert the collector's redaction processor masks or drops the configured sensitive attributes (e.g. SQL bind values) before export, so a span queried from the backend does not carry the raw secret. +- **Sampling / volume:** assert the default head-sampling ratio is applied at the collector, and that with a disk cap set a sustained span flood degrades safely (oldest-evicted, ingest not blocked) rather than filling the PVC. - **Manual:** verify trace→logs and (with the spanmetrics connector) trace→metrics pivots in Grafana. ## Rollout 1. **Per-tenant backend + collector:** the tracing backend and OTLP Collector inside the tenant stack (`packages/extra/monitoring` and the root path via `packages/system/monitoring`), with the poller readiness gate. Apps push OTLP to the in-namespace collector; no NetworkPolicy change. 2. **Grafana:** traces datasource + correlation links. -3. **Engine integrations:** native-OTLP engines first (ClickHouse, NATS), then sidecar-based (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per engine. +3. **Engine integrations:** start with the engine closest to native — ClickHouse (re-enable `opentelemetry_span_log` + add the exporter) — then client-instrumented / agent engines (NATS, Kafka, RabbitMQ, MariaDB, Redis) and extension-based DB tracing (Postgres via `pg_tracing`), one PR per engine; see the [per-engine breakdown](#4-per-application-enablement). 4. **Shared-central topology (opt-in):** add the narrow Cilium egress rule and, for reads, the vmauth authenticating proxy. 5. **Docs:** enablement guide under `docs/observability/`. @@ -193,18 +208,18 @@ Purely additive and opt-in. Existing clusters see no change until a tracing back - **Backend maturity:** VictoriaTraces is **pre-GA** upstream (data structure/backward-compat not yet frozen; Tempo Query-frontend-compatible query API still landing). Is it production-ready at the version Cozystack pins, or does the first cut ship on Tempo and switch to VictoriaTraces once GA? Only the backend CR and datasource type change either way. - **Application→HelmRelease enablement mechanism:** what is the clean way to map a tenant's tracing enablement onto supported engines' HelmReleases without copy-pasting a `tracing` schema into every chart, and which engines are in the first cut? -- **Sampling default:** head sampling at the source (scales freely) vs. tail sampling at the collector (needs trace affinity)? +- **Sampling policy beyond the default:** the default is 10% head sampling at the collector (see [§3](#3-ingest-topology-and-routing)); the open question is the Stage-2 tail-sampling policy (keep-errors / keep-slow thresholds) and whether the head default should differ per engine or per tenant. - **Durability:** VictoriaTraces cluster mode does not replicate spans across storage nodes; is single-backend durability acceptable, or is collector replication into two independent backends warranted? - **External OTLP exposure:** should tenants push spans from outside the cluster, and through which ingress/Gateway path? (This is where an internal Gateway/`HTTPRoute` would earn its place — see [§3](#3-ingest-topology-and-routing).) - **Confirm-before-implementation (upstream backend surface):** the Grafana datasource type for VictoriaTraces (Tempo Query-frontend-compatible API, expected primary, vs. Jaeger-compatible surface vs. dedicated plugin) and the exact OTLP wire/CRD field details — settled against the pinned version in the implementation PR. ## Alternatives considered -- **Mirror the logs/metrics stack one-for-one** (framing): rejected as the *guiding principle* (@lllamnyp :61). The scrape-based network model of metrics/logs does not transfer to push-based OTLP, so copying it wholesale would have designed straight into the tenant-isolation problem in [§2](#2-multitenancy-and-network-model-the-core-decision). Conventions are reused only where they genuinely fit. -- **Single-vendor VictoriaTraces, no backend seam** (backend): rejected in favor of the pluggable OTLP/datasource contract (@lllamnyp :33). VictoriaTraces remains the default (no new operator), but Tempo and Jaeger are selectable, and the choice never leaks into applications or the tenancy design. +- **Mirror the logs/metrics stack one-for-one** (framing): rejected as the *guiding principle*. The scrape-based network model of metrics/logs does not transfer to push-based OTLP, so copying it wholesale would have designed straight into the tenant-isolation problem in [§2](#2-multitenancy-and-network-model-the-core-decision). Conventions are reused only where they genuinely fit. +- **Single-vendor VictoriaTraces, no backend seam** (backend): rejected in favor of the pluggable OTLP/datasource contract. VictoriaTraces remains the default (no new operator), but Tempo and Jaeger are selectable, and the choice never leaks into applications or the tenancy design. - **Central OTLP endpoint apps push to directly** (ingest/tenancy): rejected — blocked by Cilium tenant isolation and would require a broad egress hole. The per-tenant in-namespace collector avoids the breach entirely. -- **Per-app `tracing.enabled` toggle modeled on foundationdb** (enablement): rejected (@lllamnyp :31, :156) — foundationdb's toggle gates a billing meta-resource, and metrics enablement is not per-app in the first place. Tracing follows the platform-configured metrics model. -- **Internal Ingress / Gateway-API `HTTPRoute` for the app-facing hop** (routing): rejected for the default path (@lllamnyp :130) — the app→collector hop is intra-namespace, so an L7 hop adds cost without benefit and fits OTLP/gRPC poorly. Retained as the right tool for the shared-central front-end and external exposure. +- **Per-app `tracing.enabled` toggle modeled on foundationdb** (enablement): rejected — foundationdb's toggle gates a billing meta-resource, and metrics enablement is not per-app in the first place. Tracing follows the platform-configured metrics model. +- **Internal Ingress / Gateway-API `HTTPRoute` for the app-facing hop** (routing): rejected for the default path — the app→collector hop is intra-namespace, so an L7 hop adds cost without benefit and fits OTLP/gRPC poorly. Retained as the right tool for the shared-central front-end and external exposure. - **Collector as a DaemonSet agent** (topology): rejected — the node-local shape fits fluent-bit tailing files, but OTLP is pushed over the network, so a gateway Deployment is correct. - **Always-on tracing** (enablement default): rejected — tracing overhead and storage cost stay a tenant's explicit choice. From 159140ea14f3b1c0955e898ca2fc97c95700c2cd Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 2 Aug 2026 19:23:38 +0300 Subject: [PATCH 13/14] Fix ingress egress scope: ancestor-pinned in -egress block, cluster-wide via allow-to-ingress branch-review caught that the Cilium snapshot conflated two rules: the per-tenant -egress block pins cozystack.io/service: ingress to ancestor namespaces (like vminsert/etcd), while the separate allow-to-ingress policy (endpointSelector: {}) reaches ingress endpoints cluster-wide. Word both accurately. Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index d4e48b7..124713e 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -35,7 +35,7 @@ Cozystack's observability is multi-tenant with a central backend, and it crosses **How metrics and logs cross the tenant boundary — by *scraping*, not by tenants pushing out.** vmagent and fluent-bit run centrally in `cozy-monitoring`: vmagent scrapes targets cluster-wide and remote-writes, fluent-bit tails node logs and ships them. Their destination is `global.target`, which the chart defaults to `cozy-monitoring` but the platform bundle overrides to `tenant-root` (`packages/core/platform/templates/bundles/system.yaml`) — so the agents write to `vminsert-*` / `vlinsert-generic`, resolved either directly in `tenant-root` or through the `cozy-monitoring` ExternalName aliases that CNAME there (the aliases are the chart-default path and what any `cozy-monitoring`-local reference resolves through). Either way the write crosses into `tenant-root` because `cozy-monitoring` carries the `cozystack.io/system: "true"` label (stamped by the operator, `internal/operator/package_reconciler.go`) and `tenant-root`'s Cilium ingress policy trusts every `cozystack.io/system` namespace. **This is an ingress trust granted to a system namespace — it does not let a tenant workload egress toward monitoring.** -**Tenant network isolation (Cilium).** `packages/apps/tenant/templates/networkpolicy.yaml` renders `CiliumClusterwideNetworkPolicy` per tenant; the posture is default-deny once selected. A tenant pod's egress is limited to an explicit allow-list — the snapshot below is **representative, not exhaustive** (the file is authoritative; the list has grown over time): its own tenant subtree; endpoints labeled `app.kubernetes.io/name: vminsert` / `app.kubernetes.io/instance: etcd` in **ancestor** namespaces; `cozystack.io/service: ingress` endpoints in **any** namespace; DNS; a **label-gated** `kube-apiserver` egress (pods carrying `policy.cozystack.io/allow-to-apiserver: "true"`); a set of shared `cozy-*` namespaces (`cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-kubevirt-cdi`, …); and `world` (out-of-cluster). The load-bearing fact for §2 is stable under any addition to this list: **`cozy-monitoring` is not on it**, and a generic pod in `tenant-root` is not reachable either — only the specifically-labeled `vminsert`/`etcd` endpoints are. A tenant that runs its own monitoring gets a per-tenant vmagent *inside its own namespace* that pushes up to the parent's `vminsert` — that is exactly the `vminsert`-labeled ancestor egress rule. +**Tenant network isolation (Cilium).** `packages/apps/tenant/templates/networkpolicy.yaml` renders `CiliumClusterwideNetworkPolicy` per tenant; the posture is default-deny once selected. A tenant pod's egress is limited to an explicit allow-list — the snapshot below is **representative, not exhaustive** (the file is authoritative; the list has grown over time): its own tenant subtree; endpoints labeled `app.kubernetes.io/name: vminsert` / `app.kubernetes.io/instance: etcd` / `cozystack.io/service: ingress` in **ancestor** namespaces (the per-tenant `-egress` block pins each to an ancestor namespace), plus a separate `allow-to-ingress` policy that reaches `cozystack.io/service: ingress` endpoints **cluster-wide**; DNS; a **label-gated** `kube-apiserver` egress (pods carrying `policy.cozystack.io/allow-to-apiserver: "true"`); a set of shared `cozy-*` namespaces (`cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-kubevirt-cdi`, …); and `world` (out-of-cluster). The load-bearing fact for §2 is stable under any addition to this list: **`cozy-monitoring` is not on it**, and a generic pod in `tenant-root` is not reachable either — only the specifically-labeled `vminsert`/`etcd` endpoints are. A tenant that runs its own monitoring gets a per-tenant vmagent *inside its own namespace* that pushes up to the parent's `vminsert` — that is exactly the `vminsert`-labeled ancestor egress rule. **Application observability today.** Apps expose metrics out of the box: the charts render `VMServiceScrape`/`VMPodScrape` (or CNPG's `enablePodMonitor`) unconditionally — there is no per-app "enable metrics" toggle. Most engines also declare a `WorkloadMonitor` CR, but `WorkloadMonitor` is **not** a metrics-collection mechanism: it is a billing/ownership meta-resource holding a label selector that identifies the pods, services, and PVCs belonging to an app, reconciled into `Workload` objects (replicas/resources/status) for the dashboard and billing surfaces (`internal/controller/workloadmonitor_controller.go`). The one app that gates a `WorkloadMonitor` behind a tenant-facing `monitoring.enabled` flag is foundationdb — and that is a mistake we should not copy (it hides a billing meta-resource behind a tenant toggle), not a template for a tracing switch. From 6d91fae2f70ed6548f463ea5bc454ed76b142898 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 3 Aug 2026 15:08:47 +0300 Subject: [PATCH 14/14] Reframe driver as Cozystack adopters, not a named client Drop the specific client reference (hidora); the motivation is request-level visibility requested by Cozystack adopters generally. Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 124713e..b5e26ea 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -25,7 +25,7 @@ In scope: a traces backend (per-tenant and, as an opt-in, shared-central), an OT - **Collection agents:** `packages/system/monitoring-agents` (fluent-bit, vmagent) — the *deployment* pattern the collector borrows, but note the traffic model differs (see [§2](#2-multitenancy-and-network-model-the-core-decision)). - **Network policy:** `packages/apps/tenant/templates/networkpolicy.yaml` — the Cilium tenant-isolation policies this design must live within. - **Prior art in-repo:** Harbor exposes an internal, app-local trace config (`packages/system/harbor/charts/harbor/values.yaml`, provider `jaeger`/`otel`); it is not a platform backend. This proposal supersedes ad-hoc per-app trace endpoints with a shared destination. -- **Driver:** requested by a client (hidora) who needs request-level visibility across managed DBaaS and messaging services. +- **Driver:** requested by adopters of Cozystack who need request-level visibility across managed DBaaS and messaging services. ## Context