HealthChecks failing with "no matches for kind..." whereas CRD is installed.

[zioc]

Using kustomize controller v1.8.5, we have a kustomization running this healthcheck (among others):

healthChecks:
  - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: mgmt-2620952448-rke2-capo
    namespace: sylva-system

that fails to reconcile with following error (first occurence at 2026-06-22T20:09:10.247Z):

health check failed after 30.034806535s: timeout waiting for: [OpenStackCluster/sylva-system/mgmt-2620952448-rke2-capo status: 'Unknown': no matches for kind "OpenStackCluster" in group "infrastructure.cluster.x-k8s.io"]

whereas the crd is already installed by another kustomization, and has a single version:

kubectl get customresourcedefinitions.apiextensions.k8s.io openstackclusters.infrastructure.cluster.x-k8s.io -o yaml | yq -r '.metadata.creationTimestamp, .spec.versions[].name'
2026-06-22T20:06:23Z
v1beta1

and the custom resource already exists in cluster:

❯ kubectl get openstackclusters.infrastructure.cluster.x-k8s.io mgmt-2620952448-rke2-capo -o yaml | yq .metadata.creationTimestamp
2026-06-22T20:06:47Z

This looks similar to that issue fluxcd/flux2#531, but this issue is fixed in v1.8.5, and the CRD has a single version installed...

Side note: we don't have the DisableStatusPollerCache feature enabled in our kustomize controller deployments, maybe it could help fixing that issue, as it looks like a cache miss. But this shouldn't have occurred anyway...

[zioc]

After digging a bit further, it turns out that this issue comes from a caching issue in controller-runtime RESTMapper (see upstream issue)

This issue is fairly critical, since the kustomize controller may endlessly fail to perform healthChecks on a groupKind that has been installed in a cluster after the initial discovery of the group, as it has been observed in sylva project.

As a short-term workaround, we've configured the failing Kustomizations with a serviceAccount. In that case the kustomize controller creates a new RESTMapper during each reconciliation, and avoids the cache invalidation issue.

This is issue could also be fixed in kstatus library by adding the group version to RESTMapping requests when it is available in kustomization healthCheck's NamespacedObjectKindReference: When version is provided to the restmapper, it is not affected by the cache invalidation issue described above, and it is much more efficient as it uses serverResourcesForGroupVersion under the hood that fetches a single group version in the API instead of ServerGroupsAndResources that requests all api groups and versions.

[stefanprodan]

I think no one hit this issue because most people deploy Flux with multi-tenancy enabled. Running each reconciliation under a dedicated SA should not be just "a short-term workaround", it's how Flux should run in production system.

[zioc]

Running each reconciliation under a dedicated SA should not be just "a short-term workaround", it's how Flux should run in production system.

Sure, but for now it also implies that statusPoller cache won't be shared across reconciliations, which can also lead to some some performance issues as described in #1436.

[stefanprodan]

A shared cache comes with downsides too, as many reconciliation run in parallel and they all block on the cache write mutex. The tradeoff is goroutine contention vs increased memory usage. Anyway if this gets fixed upstream, we'll update controller-runtime to fix this bug for users who don't run Flux in multi-tenancy mode.

[matheuscscp]

A shared cache implies RBAC cannot gate tenants.

[zioc]

Thanks for your feedbacks. In sylva project we have a fairly specific use of flux where all flux resource are managed by a single entity, so we were no urged to use multi-tenancy (but still plan to do so, it's a work in progress).

I was not aware of the shared cache write mutex thought, it makes a point indeed.