Skip to content

Deploy backend Container App revisions using immutable image references #552

Description

@bostdiek

User story

As a PantryPilot operator, I want production deployments to reference an immutable backend image digest or commit tag so that every deployment creates a traceable Azure Container Apps revision running the intended code.

Problem

The production workflow currently builds and pushes pantrypilot-backend:prod, then updates the Container App using that same mutable tag. Because the configured image string does not change between deployments, a deployment can complete and its health check can pass while an existing revision continues serving an older image digest.

This occurred on July 22, 2026. Production served backend source from before the async-session concurrency fix even though the fix had been merged to production and the deployment workflow had run. The stale revision produced:

sqlalchemy.exc.InterfaceError: cannot perform operation: another operation is in progress

Production was recovered manually by resolving the current prod tag to its digest and creating a new revision with az containerapp revision copy. The recovered revision was pinned to:

pantrypilotacrprodprod001.azurecr.io/pantrypilot-backend@sha256:67e85c928ff21164d58addda006e0dfc20955e62271ee5dc69e5895f9f14197b

Acceptance criteria

  • Backend images are published with an immutable commit-SHA tag.
  • Production deployment references that SHA tag or its resolved digest, never :prod.
  • Each deployment creates a Container Apps revision with a SHA-derived revision suffix.
  • Deployment waits for the new revision to become ready and confirms it receives 100% traffic.
  • Deployment fails when the active revision digest differs from the digest produced by the triggering build.
  • The backend exposes its build commit SHA and image version through a health or version endpoint.
  • Deployment verification confirms the endpoint reports the expected commit SHA.
  • The previous healthy revision remains available according to the rollback retention policy.
  • Bicep deployment does not restore or retain the mutable :prod image reference.
  • Development and production workflow validation covers immutable image selection.

Implementation notes

  • Pass the image digest or workflow_run.head_sha from the image-build workflow into deployment.
  • Prefer deploying registry/repository@sha256:....
  • Update .github/workflows/build-images.yml, .github/workflows/deploy-prod.yml, and the Container App Bicep image configuration.
  • Do not treat a generic HTTP 200 health response as proof that the expected build is running; verify build identity explicitly.

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions