[patrick-hermann-sva]

Problem

Follow-up to #49 ("Befund nach der Flux-Übernahme"). kro applies every RGD change to every existing instance, including the Job of a CR that finished long ago. A Job's spec.template is immutable. So any new trigger script in the Job's command (e.g. the retried GET from #50) puts every existing ClusterBuildWatch / BackstageTemplateRun into ERROR:

ResourcesReady=False resource reconciliation failed: apply results contain errors:
Job.batch "app-old" is invalid: spec.template: Invalid value: ... field is immutable

kro v0.9.4 (the version on test5) has no create-only, ignore-field or skip-update option for a managed resource. The applyset always does SSA with Force: true, and the only per-resource switches are includeWhen, readyWhen and externalRef.

Fix (both RGDs)

1. The script leaves the Job. start.sh, input.json and not-after live in the <name>-input ConfigMap, which kro updates in place. The Job runs only sh /input/start.sh. A script change touches no Job and runs nothing.
2. What can still change goes into the Job name. That is the CR's triggerImage plus a trigger-v1 marker for the template itself. The name becomes <name>-<first 8 hex of sha256>. A change creates a new Job and kro prunes the old one. The new Job's GET finds the instance and exits already exists, not starting it again, the same path as a Job deleted by hand. A new triggerImage default never reaches existing CRs, because the API server stored the default in each CR.
3. Start-once is unchanged: GET first (retried), POST only on a clean ERR_INSTANCE_ID_NOT_FOUND, POST never retried, notAfter checked first. notAfter now reaches the script as a file instead of an env var (still free text, never spliced into the script).
4. Migration is automatic. Existing instances still have Job <name>, which is no longer desired, so kro prunes it and creates <name>-<hash>. That Job finds the instance and exits 0. CRs already in ERROR recover by themselves.

READMEs and RGD comments explain the rule: changing anything else in the Job's pod template requires bumping trigger-v1.

Rejected alternatives

• Hash of the whole template (script included) in the name: every release would start one pod per existing CR and send one GET to the sidecar. A Job that fails because the sidecar is down would turn a successful run's status red. The ConfigMap approach needs neither for a script change.
• includeWhen on instance or Job status: a resource cannot reference its own status. A flip to false prunes the Job, and status.jobName/succeeded lose their source. The kro docs warn against flip-flop.
• Pod/Deployment instead of Job: a Pod template is just as immutable. A Deployment would keep one long-lived pod per CR.

Trade-offs

• CR names may be at most 54 characters (Job name + label ≤ 63). Longer names fail cleanly at creation (must be no more than 63 bytes) and start nothing. Backstage request-vm produces vm-<≤40> / delete-vm-<…>, which fits.
• A pruned Job leaves its Completed pod behind without an owner: kro deletes without a propagation policy, so the Job default is orphan. The same happens today when a CR is deleted. This is cosmetic.
• A renamed Job (trigger-v1 bump or triggerImage edit) asks the sidecar once more. Like any recreated Job, it starts again only if the state store has forgotten the instance and notAfter has not passed.

Test evidence (throwaway kind cluster kro-rgd-test, kro v0.9.4 = test5, mock dapr endpoint counting start POSTs)

Step	Result
main RGD, CR app-old created	Job ran, POST #1
main RGD, script changed (simulated release)	reproduced: app-old → ERROR, Job.batch "app-old" is invalid: spec.template ... immutable
new RGD applied over it	app-old → ACTIVE; Job app-old pruned, app-old-c5d8a89e logs instance cbw_app-old already exists, not starting it again; starts still 1
new CR app-new	POST #1, ACTIVE
script-only release on the new RGD	both CRs ACTIVE, no new Job, no new pod, no new GET/POST; ConfigMap carries the new script
CR spec edit (notAfter, pollSeconds)	ACTIVE, only the ConfigMap changes
triggerImage default bumped	existing CRs unaffected (default stored in CR)
CR triggerImage edited	new Job app-old-c5d8a79e, old pruned, already exists, starts still 1
hash marker changed (all Jobs renamed)	all CRs ACTIVE, every new Job already exists, starts still 1 each
CR deleted + re-applied, same name	already exists, no start
54-char name / 55-char name	ACTIVE / ERROR at creation, must be no more than 63 bytes, nothing started
backstage RGD: same sequence (old ERROR → new ACTIVE, script release, new CR, past notAfter, with watch)	vm-demo-1 recovered, POST count 1; past notAfter → has passed, not starting; vm-demo-3 POST body incl. watch as before

Final mock counters: {"cbw_app-old": 1, "cbw_app-new": 1, "cbw_aaaa…": 1, "bst_vm-demo-1": 1, "bst_vm-demo-3": 1}. Every instance was started exactly once.

Checks: pre-commit run --files on the changed files (check-yaml, detect-secrets, …) passed; YAML parses; the extracted start.sh passes shellcheck -s sh with the kro expressions stubbed. Both RGDs: GraphAccepted=True, revision compiled on kro v0.9.4.

Rollout note

Live on test5 right now: 1 instance (cluster-build-watch/app-dev-homerun, ACTIVE). After the next release it gets Job app-dev-homerun-<hash> (logs already exists), and the old Job app-dev-homerun is pruned. Its pod remains as an ownerless Completed pod.

Refs #49

🤖 Generated with Claude Code