cloud-api deploy hangs: platform-api-migrate Job stuck terminating with argocd hook-finalizer #171

2026-10-11: three cloud-api prod deploys (0.8.245/0.8.246/0.8.247) hung for 40+ min at the first line of 'Deploy to prod': kubectl delete job platform-api-migrate -n nan-system --ignore-not-found. The Job (created 11:05 UTC by the previous deploy, completed OK) had deletionTimestamp=11:15Z and finalizers=[argocd.argoproj.io/hook-finalizer]: the migrate Job is also an Argo CD hook of the manually-synced prod-platform-api app, and nothing removed the hook finalizer, so kubectl delete (which waits) blocked forever.

Resolution applied by the Lead: cancelled the two older runs, removed the finalizer from the completed Job (kubectl -n nan-system patch job platform-api-migrate --type json -p '[{"op":"remove","path":"/metadata/finalizers"}]'). One run then raced the newest (AlreadyExists) but the rollout completed: prod on 0.8.247, schema 109.

Fix options (pick one):

• In deploy.yml: kubectl delete job ... --wait=false, then strip a leftover argocd hook finalizer and wait with a timeout before apply; fail loudly after N seconds instead of hanging.
• Stop managing the migrate Job as an Argo CD hook (the pipeline already owns it), or make it a Job the app ignores.
• Add concurrency: { group: deploy-prod, cancel-in-progress: false } so consecutive merges queue instead of racing (they raced on AlreadyExists today).