You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
kyverno: Add Prometheus-backed policy engine health metrics - #1310
Adds a "Policy Engine Health" section to the existing Kyverno Dashboard, backed by live Prometheus queries against Kyverno's own metrics. Auto-discovers a reachable Prometheus server the same way the official prometheus plugin does, and degrades to a clean "not detected" message (reusing the existing NotInstalledBanner) when one isn't found, rather than crashing or showing a blank section.
What Kyverno actually exposes,
Port-forwarded to all 4 of Kyverno's metrics services on a real cluster and read the raw /metrics output directly, since several metric families are lazily registered and don't appear until first observed. A cold check showed almost nothing under kyverno_*. Triggered real admission traffic (a real ValidatingPolicy, a passing and a failing ConfigMap) and re-checked, which surfaced the real set this PR uses: kyverno_admission_requests_total, kyverno_admission_review_duration_seconds, kyverno_breaker_total, kyverno_policy_rule_info_total, kyverno_controller_requeue_total, with their real label dimensions (request_allowed, status_ready, circuit_name, etc.), not just names copied from a doc page. Cross-checked against Kyverno's own metrics reference and its official community Grafana dashboard to confirm these are the standard, expected observability surface for this project.
The 5 signals
Admission Rate: current request throughput (rate(kyverno_admission_requests_total[5m]))
Denied (5m): how many requests were blocked recently (request_allowed="false")
Breaker Trips (1h): Kyverno tripping its own overload protection (kyverno_breaker_total)
Rules Not Ready (now): rules that exist but never became ready (kyverno_policy_rule_info_total{status_ready!="true"}), a correctness gap the existing report-based Dashboard can't see, since a not-ready rule produces no reports at all
Admission Review Latency (P95, last hour): the one chart, webhook e2e latency. Slow here risks admission timeouts blocking real cluster operations, not just Kyverno's own correctness
Deliberately not a full observability dashboard, Grafana already does that well. Just enough to answer "is the policy engine healthy right now" without leaving Headlamp.
Discovery reuses the prometheus plugin's proven pattern, not its code
The official prometheus plugin (same monorepo) already solved this: search for a pod or service labeled headlamp-prometheus=true (a Headlamp-specific opt-in) or the common app.kubernetes.io/name=prometheus labels, validate candidates with a real query, then proxy through the Kubernetes API server's built-in subresource proxy. That logic lives inside prometheus's own bundle and isn't exported for other plugins to import. Rather than copying its file structure, built usePrometheus() mirroring this plugin's own existing useKyvernoCRDs() hook exactly: same module-level cache keyed by cluster, same listener/notify pattern. Reusing the same label conventions means a user who already configured the prometheus plugin gets this working with zero extra setup.
Scope decision: no new Settings UI
The prometheus plugin ships a full Settings page for manually overriding namespace/service/port. Deliberately skipped that here, auto-discovery only, with a clean "not detected" message when it fails. A configuration subsystem for one secondary Dashboard section would be scope creep this feature never needed.
Two real correctness issues found and fixed during verification
PromQL's empty-vector gotcha.sum()/increase() over zero matching series returns an empty result in PromQL, not a 0, so a perfectly healthy cluster (nothing denied, no breaker trips) would have shown "no data" on every stat tile instead of an honest 0, reading as "unknown" rather than "nothing wrong". Fixed by appending or vector(0) to every instant query. Verified the fix against a real denial: applied a Deny-mode policy, triggered a real blocked kubectl create, and confirmed the "Denied" tile went from a misleading dash to a real, correct number once enough Prometheus scrapes had accumulated to compute increase() over.
Uncalibrated severity thresholds would have shipped false alarms. The first version color-coded "Breaker Trips" and "Reconcile Errors" with fixed red/amber thresholds. Testing surfaced a real problem with that: kyverno_breaker_total, summed with increase() over an hour on a cluster whose pods have restarted many times (any long-lived kind cluster, not unique to this test setup), showed 460 trips despite the raw counter only being at 24, because Prometheus's counter-reset compensation correctly sums each restart's segment. That's mathematically correct, not a bug, but it also means there's no real production baseline available to justify a specific "5 trips is bad" threshold. Rather than guess and risk a false alarm on a healthy cluster, redesigned the color logic: only assert what's actually unambiguous (0 is good), leave any positive count neutral instead of presumptively red, for every metric except "Rules Not Ready", where a nonzero count genuinely has no other honest interpretation.
Verification
tsc --noEmit and eslint: clean across all changed and new files.
No real Prometheus was installed on the local test cluster (checked: no ServiceMonitor CRD, no matching labeled pod/service). Deployed a real one (plain Deployment + Service + a static scrape config targeting all 4 Kyverno metrics services, no Operator/CRDs needed), labeled to match the same discovery convention the code searches for.
Generated real admission traffic, including a real Deny-mode policy rejection (confirmed via the actual webhook error and a non-zero kubectl exit code), so the metrics being charted reflect real cluster events, not synthetic data.
Confirmed the happy path renders real data end to end: all 5 stat tiles and the latency chart populated with genuine scraped values.
Proved the degradation path for real: scaled the Prometheus deployment to zero, forced a fresh discovery probe with a full page reload (the discovery cache is per-session), and confirmed the "Prometheus was not detected on this cluster" banner renders cleanly instead of a crash or a blank section. Then restored Prometheus and reverified the real path still works.
Checked the browser console throughout: no new errors introduced. The only warning present (react-i18next: Trans: You need to pass in an i18next instance) is pre-existing and not introduced by this PR, it's already explained by this plugin never having declared headlamp.i18n in package.json, confirmed by checking this branch's own package.json still lacks that declaration.
This sums one gauge series per scrape target, so an HA Kyverno deployment reports the same unready rule once for every replica and inflates the tile. Kyverno's gauge is emitted from each controller's full policy list, while Prometheus distinguishes targets with labels such as instance; aggregate replicas with max by over the rule identity before summing.
kyverno_policy_rule_info_total is an observable gauge exposed by both Kyverno's admission and background-controller processes, and by every replica. Summing the raw series therefore counts the same not-ready policy/rule once per scrape target. Collapse by policy identity before summing so the displayed value is stable across controller and replica counts.
Preserve gaps when latency quantiles have no observations
connectNulls draws a continuous latency line across intervals where histogram_quantile returned no observation. That visually invents continuity during no-traffic gaps and defeats the null preservation above; render those intervals as gaps instead.
setInterval can launch another seven-query batch before the previous batch finishes. Since responses are not sequenced, an older request can overwrite newer data (or switch a successful view to an error), and stalled proxy requests can accumulate. Schedule the next refresh only after the current load settles.
Cache invalidation leaves queries targeting dead pod endpoints
kyverno/src/hooks/usePrometheus.ts:125
The cache permanently stores a dynamic pod/service endpoint, including failed discovery. Because pod candidates are preferred, a Prometheus rollout can replace the cached pod name and leave this section querying a dead proxy path for the rest of the session; a transient initial outage likewise remains “not detected” after recovery. Prefer stable services and/or invalidate and re-run discovery after query failures or a bounded TTL.
These five hooks run before either the CEL-CRD or Prometheus checks resolve. Because the Overview route is guarded only by the legacy CRD, every legacy-only cluster—and even clusters with no Prometheus—will issue five list requests to optional API resources that do not exist. Handling their errors in celPoliciesNotReady does not prevent that traffic. Move the CEL list hooks into a child component that is mounted only when crds.cel is true and the health section needs the count; use 0 directly when the CEL group is absent.
The PR description defines exactly five health signals and omits controller requeues, but this adds a fifth tile in addition to those four tiles and the latency chart (six signals total). Please either document Requeues as an intentional sixth signal—including why it belongs in the deliberately minimal scope—or remove the requeue query/card so the implementation matches the stated scope.
kyverno_breaker_total counts every breaker invocation, not trips: Kyverno's pkg/breaker/breaker.go increments it before checking whether the breaker is open. Normal traffic therefore increases “Breaker Trips” even when no work is dropped. Query kyverno_breaker_drops_total, the counter incremented when the breaker fails open and drops work, instead.
Treat CEL discovery failures as unknown, not absent
crds.cel === false does not confirm that CEL policies are absent. checkAPIGroup in useKyvernoCRDs.ts returns false for every request failure, including permission denials and transient errors, and caches that result. This branch then reports zero even if the policy lists contain unready CEL policies, potentially showing a green readiness tile for the session. Distinguish confirmed API absence from failed discovery; return zero only for confirmed absence and preserve an unknown value when discovery fails.
Discover CEL availability per policy resource kind
A discoverable CEL API group does not guarantee that all five policy kinds are installed. Kyverno's Helm chart allows individual CEL CRDs to be disabled. One absent kind therefore makes its list fail and keeps the entire readiness tile unknown, hiding counts from installed CEL kinds and legacy policies. Discover availability per resource and exclude confirmed-absent kinds from this loop, while retaining unknown for authorization or transient failures.
Scope discovery requests and validation to the captured cluster
kyverno/src/hooks/usePrometheus.ts:144
The captured cluster scopes only the cache, not the discovery requests. If the user switches from A to B while a list request is pending, later probes and fallback searches can target B and cache its endpoint under A. Returning to A then uses the wrong endpoint until the five-minute cache expires. Pass the captured cluster through discovery and the query helpers, and explicitly scope both resource-list requests and candidate-validation requests to it.
A range query can omit evaluation timestamps entirely when no series exists, for example during a sufficiently long scrape outage. rangeSeries only maps returned samples, so the chart connects the valid points on either side even with connectNulls disabled. Fill the requested time grid with null-valued points wherever a sample is missing, including leading and trailing gaps.
This total excludes CleanupPolicy and ClusterCleanupPolicy, which the plugin supports and which expose ready in resources/cleanupPolicy.ts:62–78. An unready cleanup policy therefore leaves “Policies Not Ready” showing a green 0 when the legacy/CEL policies are healthy. Include both cleanup kinds with the same absent/error handling, or explicitly label the tile as covering only legacy/CEL policies and correct the full-coverage comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
illume
left a comment
Summary
Adds a "Policy Engine Health" section to the existing Kyverno Dashboard, backed by live Prometheus queries against Kyverno's own metrics. Auto-discovers a reachable Prometheus server the same way the official
prometheusplugin does, and degrades to a clean "not detected" message (reusing the existingNotInstalledBanner) when one isn't found, rather than crashing or showing a blank section.What Kyverno actually exposes,
Port-forwarded to all 4 of Kyverno's metrics services on a real cluster and read the raw
/metricsoutput directly, since several metric families are lazily registered and don't appear until first observed. A cold check showed almost nothing underkyverno_*. Triggered real admission traffic (a realValidatingPolicy, a passing and a failingConfigMap) and re-checked, which surfaced the real set this PR uses:kyverno_admission_requests_total,kyverno_admission_review_duration_seconds,kyverno_breaker_total,kyverno_policy_rule_info_total,kyverno_controller_requeue_total, with their real label dimensions (request_allowed,status_ready,circuit_name, etc.), not just names copied from a doc page. Cross-checked against Kyverno's own metrics reference and its official community Grafana dashboard to confirm these are the standard, expected observability surface for this project.The 5 signals
rate(kyverno_admission_requests_total[5m]))request_allowed="false")kyverno_breaker_total)kyverno_policy_rule_info_total{status_ready!="true"}), a correctness gap the existing report-based Dashboard can't see, since a not-ready rule produces no reports at allDeliberately not a full observability dashboard, Grafana already does that well. Just enough to answer "is the policy engine healthy right now" without leaving Headlamp.
Discovery reuses the
prometheusplugin's proven pattern, not its codeThe official
prometheusplugin (same monorepo) already solved this: search for a pod or service labeledheadlamp-prometheus=true(a Headlamp-specific opt-in) or the commonapp.kubernetes.io/name=prometheuslabels, validate candidates with a real query, then proxy through the Kubernetes API server's built-in subresource proxy. That logic lives insideprometheus's own bundle and isn't exported for other plugins to import. Rather than copying its file structure, builtusePrometheus()mirroring this plugin's own existinguseKyvernoCRDs()hook exactly: same module-level cache keyed by cluster, same listener/notify pattern. Reusing the same label conventions means a user who already configured theprometheusplugin gets this working with zero extra setup.Scope decision: no new Settings UI
The
prometheusplugin ships a full Settings page for manually overriding namespace/service/port. Deliberately skipped that here, auto-discovery only, with a clean "not detected" message when it fails. A configuration subsystem for one secondary Dashboard section would be scope creep this feature never needed.Two real correctness issues found and fixed during verification
sum()/increase()over zero matching series returns an empty result in PromQL, not a0, so a perfectly healthy cluster (nothing denied, no breaker trips) would have shown "no data" on every stat tile instead of an honest0, reading as "unknown" rather than "nothing wrong". Fixed by appendingor vector(0)to every instant query. Verified the fix against a real denial: applied aDeny-mode policy, triggered a real blockedkubectl create, and confirmed the "Denied" tile went from a misleading dash to a real, correct number once enough Prometheus scrapes had accumulated to computeincrease()over.kyverno_breaker_total, summed withincrease()over an hour on a cluster whose pods have restarted many times (any long-lived kind cluster, not unique to this test setup), showed 460 trips despite the raw counter only being at 24, because Prometheus's counter-reset compensation correctly sums each restart's segment. That's mathematically correct, not a bug, but it also means there's no real production baseline available to justify a specific "5 trips is bad" threshold. Rather than guess and risk a false alarm on a healthy cluster, redesigned the color logic: only assert what's actually unambiguous (0is good), leave any positive count neutral instead of presumptively red, for every metric except "Rules Not Ready", where a nonzero count genuinely has no other honest interpretation.Verification
tsc --noEmitandeslint: clean across all changed and new files.Deny-mode policy rejection (confirmed via the actual webhook error and a non-zerokubectlexit code), so the metrics being charted reflect real cluster events, not synthetic data.react-i18next: Trans: You need to pass in an i18next instance) is pre-existing and not introduced by this PR, it's already explained by this plugin never having declaredheadlamp.i18ninpackage.json, confirmed by checking this branch's ownpackage.jsonstill lacks that declaration.Policy Engine Health, real data
Degradation path, Prometheus not detected