Add opt-in preflight admission readiness check before kubeadm upgrade apply

[felipe88alves]

What would you like to be added

I'd like to add an opt-in preflight check that verifies the API server's admission pipeline is ready before kubeadm upgrade apply runs on the first control plane node.

We run kubespray across multiple production clusters and have been hitting intermittent [ERROR CreateJob] failures during upgrades. Our clusters use ValidatingAdmissionPolicy with a CRD-based paramKind, and after the API server restarts mid-upgrade, the VAP param controller's RESTMapper takes ~30 seconds to refresh its CRD discovery cache (kubernetes/kubernetes#124237). During that window, kubeadm upgrade apply's preflight health-check Job gets rejected by the stale admission controller.

Here's the relevant error from a real upgrade run (sanitized):

TASK [kubernetes/control-plane : Kubeadm | Upgrade first control plane node]
...
[ERROR CreateJob]: could not create Job "upgrade-health-check-xxxxx" in the namespace
"kube-system": jobs.batch "upgrade-health-check-xxxxx" is forbidden:
ValidatingAdmissionPolicy '<vap-policy>' denied request: failed to configure policy:
failed to find resource referenced by paramKind: '<crd-group/version, Kind=ParamCRD>'

The VAP param controller can't resolve the CRD paramKind reference because its RESTMapper cache is stale after the API server restart. The Job itself is fine; the admission pipeline is temporarily broken.

The existing check-api.yml only checks /healthz, which returns 200 while the admission pipeline is still broken. A potential fix would be to add an opt-in task between check-api.yml and kubeadm upgrade apply that dry-runs a batch/v1 Job (the same resource kind kubeadm creates for its preflight) through the server-side admission pipeline, retrying until admission controllers are ready. This would follow existing patterns like the configurable health-check approach from #12448/PR #12452.

I have a working implementation on our fork and would be happy to submit a PR.

Why is this needed

I've been hitting this failure intermittently across our production clusters during Kubernetes upgrades with kubespray. Our clusters use ValidatingAdmissionPolicy with CRD-based paramKind references, and the upgrade fails on the first control plane node, where kubeadm upgrade apply is called.

The root cause is a timing issue in the upgrade flow:

1. Tasks in kubeadm-setup.yml (audit policy, admission control configs, tracing) and kubeadm-fix-apiserver.yml (component kubeconfigs) notify the Restart apiserver / restart kubelet handlers
2. The handlers fire and restart the API server
3. The VAP param controller's RESTMapper cache is invalidated and takes ~30 seconds to refresh (kubernetes/kubernetes#124237 - still open, no fix merged; related: kubernetes/kubernetes#122658)
4. check-api.yml passes because /healthz returns 200 while the admission pipeline is still stale
5. kubeadm upgrade apply creates its preflight health-check Job, which gets rejected with HTTP 422
6. Upgrade fails with [ERROR CreateJob]

In this case, we traced the failure to the VAP param controller failing to resolve a custom CRD paramKind after the restart. API server audit logs showed the first 422 error ~30 seconds before kubeadm upgrade apply ran, consistent with the RESTMapper refresh window.

This is different from #12006/#12009, which fixed secondary control plane nodes by switching to kubeadm upgrade node (PR #12015). Our issue is on the first control plane node where kubeadm upgrade apply is correct. It's also different from #12448 (PR #12452), which made the /healthz retry count configurable. The /healthz check passes fine, it's the admission pipeline that's broken.

I think anyone running VAP with CRD paramKind references will eventually hit this during upgrades. The preflight check adds a direct probe of the admission pipeline to catch the staleness window. It should be deprecated once kubernetes/kubernetes#124237 is fixed in all supported Kubernetes versions.

[VannTen]

I really do not like creating ourselves a Job which basically do the exact same thing that kubeadm preflight check already does.

A configurable retry with configurable delay on the kubeadm upgrade should work, no ?

[felipe88alves]

It doesn't create the Job.
It's a dry-run deployment, it's just validating that it's able to deploy it.
Admission validation occurs in the API, so a server-side dry-run should block it if that's the case.

[VannTen]

Hum, sorry, I missed that. Still, I'd prefer that we'd do a kubeadm retry (less churn, and this could help with a broader class of problem). Is there something making this difficult / inappropriate ?

[felipe88alves]

The kubeadm retry solution feels like using a sledghammer to kill a fly. The kubeadm upgrade failure I've reported is very specific (and unsolved), the solution is tailored to solve it.

There are a number of reasons why a kubeadm upgrade execution would fail, and I fear the blanket retry mechanism would just cause the system to delay execution failures that wouldn't be fixed by retries anyway.

TLDR: I'd prefer a targeted fix in this scenario. A blanket retry mechanism can be added in another task, but I don't see a real need for it.

[VannTen]

I understand what you're saying, but precisely, IMO, the solution it too much tailored to that particular problem, and essentially recreates a part of the kubeadm process. (the CreateJob preflight check).
As every task adds overhead when trying to read / understand the codebase, adding something which another component already does is a really hard sell from the maintenance POV.

I'm not advocating for a blanket retry, though, but for a opt-in one.

[felipe88alves]

My core issue with a retry logic here is that it would apply to the kubeadm upgrade apply operation which is a fairly heavy operation (static pod restarts, etcd, component reconfig). We haven't seen any other flaky/strange issues with the kubeadm upgrade apply operation except for this, that means that a real failure would likely lead to an unnecessarily expensive and time-consuming retry.

Two other points are that:
1 - It doesn't really re-create the CreateJob preflight check, many other issues could cause the CreateJob check to fail even if this commit is merged.
2 - this is a workaround for a reported issue (kubernetes/kubernetes#124237) which was opened in 2024 and still hasn't been fixed, I'll add a comment there to see if we can push for a more permanent fix, but this will handle it in the meantime.

I get your point on the maintenance overhead, but I'd like to insist on this one if you don't mind.