bug: forced upgrades fail with strict decoding errors when chart schemas contain minor warnings

[gaurishchaddha]

Description

When executing forced upgrades on a HelmRelease (spec.upgrade.force: true), Helm v4's underlying client hardcodes fieldValidationDirective=Strict during the HTTP PUT (replace) operation.

If a chart manifest contains minor schema discrepancies (such as deprecated fields, misplaced annotation keys, or third-party vendor metadata), the Kubernetes API server rejects the forced upgrade with an HTTP 400 Bad Request strict decoding error:
Plaintext

Helm upgrade failed: HTTP 400 Bad Request (strict decoding error)

This behavior causes helm-controller reconciliation loops to get permanently stuck on affected charts, breaking backward compatibility for workloads that deployed and reconciled cleanly under Helm v3.
Root Cause & Upstream Context

Helm v3 Behavior: Omits the fieldValidation URL query parameter on HTTP PUT requests during forced replaces. The Kubernetes API server falls back to its native Warn behavior, pruning unhandled fields and returning an HTTP 200 OK response with a 299 warning header.

Helm v4 Behavior: Explicitly appends &fieldValidation=Strict to the HTTP PUT request URL. Because FieldValidationDirective was not exposed on Helm's action.Upgrade struct, helm-controller had no mechanism to override this directive back to Warn.

An upstream PR has been submitted to helm/helm to expose FieldValidationDirective on the Go SDK:
👉 Upstream PR: helm/helm#32477
Proposed Fix

Once the upstream SDK change is available, the proposal is an introducing an opt-in feature gate to helm-controller: RelaxFieldValidationOnForceReplace.

When enabled, helm-controller explicitly passes upgrade.FieldValidationDirective = kube.FieldValidationDirectiveWarn specifically for the force-replace upgrade path.
Proposed Controller Changes

1. internal/action/upgrade.go

if upgrade.ForceReplace {
	if allow, _ := features.Enabled(features.RelaxFieldValidationOnForceReplace); allow {
		upgrade.FieldValidationDirective = kube.FieldValidationDirectiveWarn
	}
}

2. internal/features/features.go

const (
	// RelaxFieldValidationOnForceReplace makes the controller relax Kubernetes API
	// server field validation to "Warn" specifically for the force-replace PUT used
	// during upgrades with spec.upgrade.force: true. Helm v4 defaults this path to
	// strict validation (unlike Helm v3, which sent no directive at all and so
	// inherited the API server's own lenient Warn default), causing upgrades to
	// hard-fail on stray or misplaced fields in rendered chart manifests that were
	// previously silently pruned. Warn, not Ignore, is the deliberate choice: both
	// directives have the identical effect on the persisted object, but Warn also
	// returns a response Warning header, which client-go's default WarningHandler
	// logs via klog — giving an audit trail for silently-dropped fields that Ignore
	// would not, and matching v3's actual server-default value, not just its effect.
	// This is disabled by default; enable per-cluster/component via --feature-gates
	// for HelmReleases affected by such chart bugs.
	RelaxFieldValidationOnForceReplace = "RelaxFieldValidationOnForceReplace"
)

var features = map[string]bool{
	// ...
	RelaxFieldValidationOnForceReplace: false,
}

Rationale: Why Warn Over Ignore?

The choice of kube.FieldValidationDirectiveWarn over kube.FieldValidationDirectiveIgnore is deliberate:

Identical Persistence Effect: Both Warn and Ignore instruct the API server to prune unknown or misplaced fields identically in the stored Kubernetes object.

Auditability: Warn returns an HTTP response warning header that client-go's default WarningHandler logs via klog. This preserves an explicit audit trail in controller logs for pruned fields, whereas Ignore would drop them silently without notification.

Parity: It restores exact behavioral parity with Helm v3's original server-default handling.

Steps to Reproduce

• Render or deploy a Helm chart containing a misplaced key in a manifest (e.g., an annotations key incorrectly nested inside ConfigMap.metadata.annotations).
  

• Create a HelmRelease resource with spec.upgrade.force: true.
  

• Trigger a controller reconciliation and observe the strict decoding failure in helm-controller logs.
  

Minimal Reproducible Chart Manifests

To make reproduction as easy as possible, here is a minimal test chart that triggers the strict decoding error during forced upgrades.

1. Chart.yaml

apiVersion: v2
name: strict-validation-test
version: 0.1.0
description: A minimal chart to reproduce fieldValidation=Strict failures

2. templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: helm-annotation-test
  namespace: default
# Intentional schema discrepancy: misplaced top-level 'annotations' key outside 'metadata'
annotations:
  helm.sh/hook: pre-install
  helm.sh/hook-weight: "-1"
data:
  test: misaligned

---

Reproduction Steps

1. Initial Install / Helm 3 Upgrade (Succeeds with HTTP 299 Warning):
   Standard installs or Helm 3 forced upgrades permit the API server to prune the misplaced top-level annotations field, returning a 299 warning header:

   helm install helm-annotation-test ./strict-validation-test

2. Helm 4 / helm-controller Forced Upgrade (Fails with HTTP 400 Bad Request):
   Triggering a forced upgrade (spec.upgrade.force: true or helm upgrade --force-replace) appends fieldValidation=Strict, causing the API server to reject the update:

   helm upgrade --force-replace helm-annotation-test ./strict-validation-test

   Resulting Error:

   Error: UPGRADE FAILED: cannot patch "helm-annotation-test" with kind ConfigMap: 
   ConfigMap in version "v1" cannot be handled as a ConfigMap: 
   v1.ConfigMap.annotations: unknown field "annotations"