Summary
ResourceManager.Apply / ApplyAll cleans up field managers (cleanupMetadata → PatchReplaceFieldsManagers) only after a successful server-side dry-run. When a stale/foreign field manager co-owns a field such that the merged object is invalid, the dry-run fails first and Apply returns an error. As a result, --override-manager in kustomize-controller can resolve valid-but-wrong co-ownership, but cannot recover an object that is wedged at dry-run validation, even though stripping the offending manager would make the merge valid.
A workload is managed by Flux via server-side apply. During an incident an operator disables Flux and patches the workload with client-side PUTs from a legacy controller (operation: Update). Re-enabling Flux reconciliations leaves a stale legacy-controller / Update managedFields entry co-owning a container's identity ("." + f:name) but NOT its required f:image. When a later build removes that container, server-side apply cannot prune it (the legacy Update manager still owns the element) but drops the field Flux solely owned (f:image), leaving a half-removed, invalid container: spec.template.spec.containers[N].image: Required value
This fails at the dry-run on every reconcile, before any write. The standard post-dry-run FieldManagers cleanup never runs, so --override-manager cannot recover it.
Potential fix
Add a separate, opt-in list to run the field-manager takeover before dryRunApply, reusing the existing PatchReplaceFieldsManagers.
The sequence of steps that lead to the issue and the proposed fix are in the draft PR fluxcd/pkg#1276.
Why a separate list
We discussed this in the dev meeting on August 6th with @matheuscscp and @stealthybox , and running the takeover before the dry-run sacrifices a real property of the current design where a failed/invalid dry-run, Apply returns at NewDryRunErr and cleanupMetadata never runs, so managedFields are left untouched when the apply doesn't proceed. A pre-dry-run takeover mutates ownership first, so if the dry-run then fails for an unrelated reason (a different missing required field, a webhook rejection), ownership has already been reassigned on an object whose desired state never applied.
Ther always-on default managers that kustomize-controller already reclaims post-dry-run (kubectl Apply/Update, before-first-apply, and the controller's own Update) stay on the post-dry-run path to minimize the blast radius of the change.
kustomize-controller wiring
A dedicated flag --override-manager-before-dry-run=<manager> populates FieldManagersBeforeDryRun with only those managers (both Apply and Update ops). It is kept separate from --override-manager so operators opt specific known-problematic managers into the stronger, pre-validation behavior; the default managers and ordinary --override-manager entries keep today's post-dry-run semantics.
Summary
ResourceManager.Apply/ApplyAllcleans up field managers (cleanupMetadata→PatchReplaceFieldsManagers) only after a successful server-side dry-run. When a stale/foreign field manager co-owns a field such that the merged object is invalid, the dry-run fails first andApplyreturns an error. As a result,--override-managerin kustomize-controller can resolve valid-but-wrong co-ownership, but cannot recover an object that is wedged at dry-run validation, even though stripping the offending manager would make the merge valid.A workload is managed by Flux via server-side apply. During an incident an operator disables Flux and patches the workload with client-side PUTs from a legacy controller (operation: Update). Re-enabling Flux reconciliations leaves a stale
legacy-controller / UpdatemanagedFields entry co-owning a container's identity ("." + f:name) but NOT its required f:image. When a later build removes that container, server-side apply cannot prune it (the legacy Update manager still owns the element) but drops the field Flux solely owned (f:image), leaving a half-removed, invalid container:spec.template.spec.containers[N].image: Required valueThis fails at the dry-run on every reconcile, before any write. The standard post-dry-run FieldManagers cleanup never runs, so --override-manager cannot recover it.
Potential fix
Add a separate, opt-in list to run the field-manager takeover before
dryRunApply, reusing the existingPatchReplaceFieldsManagers.The sequence of steps that lead to the issue and the proposed fix are in the draft PR fluxcd/pkg#1276.
Why a separate list
We discussed this in the dev meeting on August 6th with @matheuscscp and @stealthybox , and running the takeover before the dry-run sacrifices a real property of the current design where a failed/invalid dry-run,
Applyreturns atNewDryRunErrandcleanupMetadatanever runs, so managedFields are left untouched when the apply doesn't proceed. A pre-dry-run takeover mutates ownership first, so if the dry-run then fails for an unrelated reason (a different missing required field, a webhook rejection), ownership has already been reassigned on an object whose desired state never applied.Ther always-on default managers that kustomize-controller already reclaims post-dry-run (
kubectlApply/Update,before-first-apply, and the controller's ownUpdate) stay on the post-dry-run path to minimize the blast radius of the change.kustomize-controller wiring
A dedicated flag
--override-manager-before-dry-run=<manager>populatesFieldManagersBeforeDryRunwith only those managers (both Apply and Update ops). It is kept separate from--override-managerso operators opt specific known-problematic managers into the stronger, pre-validation behavior; the default managers and ordinary--override-managerentries keep today's post-dry-run semantics.