[searledan]

Closes #171

The nightly release train now deploys each tag to staging first, then production, from one build. Staging stays current instead of drifting; it was 22 days behind in #166.

The pre-production gate is a small side-effect, not the point. Staging's smoke test is the same set production's own deploy already runs and self-rolls-back on. It saves a few minutes of a broken production revision, and it catches no functional regression.

Shape

release ──► staging-gate ──► deploy-staging ──► deploy (production)
             │                  (builds, imports       (no build: imports
             │                   ghcr@digest)           staging's digest)
             ├─ deploy=true   staging up and readable
             ├─ deploy=false  STAGING_PINNED set, or staging torn down (script exit 3)
             └─ FAILS         anything else — the train stops

• staging-gate.yml (new, reusable) decides per tag. It uses the staging OIDC credential with environment: { deployment: false }, so it records no phantom staging deployment. I verified that against the environment's deployment history.

• check-serving-revisions.sh gains exit 3, ABSENT. It is returned only for az's exact (ResourceNotFound) The Resource 'Microsoft.App/containerApps/<app>' under resource group '<rg>' was not found, matched as a fixed string. A missing resource group, a missing role, a lapsed login, or another app's not-found all stay NOT ASSESSED (exit 2), and the gate fails on them.

• release.yml: production requires the gate to have succeeded and staging to have done what the gate decided:

  • the gate said deploy, and staging succeeded; or
  • the gate said skip, and staging was skipped.

  A staging job skipped for any other reason can't let production through.

• deploy.yml builds once and imports by digest. The import pulls ghcr…@sha256:<digest>, never :tag. With image_digest set, the build is skipped.

  • The train's production call passes staging's digest along with require_image_digest, so production fails rather than silently rebuilding if the digest doesn't arrive.
  • Revisions still reference the readable tag (vitally-mcp:v4.x.y), as decided.
  • Manual dispatch is unchanged by default.

• STAGING_PINNED is an environment variable on staging. While it is non-empty, staging is skipped and production still ships.

  • The notice gives the first line of the reason, with any :: defanged, on every tagged run.
  • A comment beside it explains why this is not the AUTO_DEPLOY pattern.

Azure change (already applied)

Role	built-in ContainerApp Reader
Scope	resource group vitally-prod-rg-uksouth
Principal	vitally-prod-id-uksouth (c57b35e7-…)
Assignment	fcffb5d1-5cdc-47e8-88f6-8578dd6d5139, 2026-09-29
Captured	infra/terraform/identity.tf + an import block in imports.tf

Why it was needed. Without the grant, absence could not be observed. Run 36581965911 checked this as the deploy identity before the grant:

• revision list on a nonexistent app → AuthorizationFailed, the same answer as a lost role;
• containerapp list -g RG → exit 0, listing only the apps the identity can see, which is fail-open.

After the grant (run 36582385414): (ResourceNotFound).

What it adds. It adds read at resource-group scope, plus Microsoft.Insights/alertRules/* (write on classic alert rules). The identity is also both apps' runtime identity, so the running containers get that read too. identity.tf lists each action. The built-in role was chosen over a custom one (decision: dsearle).

Verification

The real gate, called by a temporary self-test (since deleted).

Case	Run	Result
Live staging	36584985171 attempt 1	deploy=true
Nonexistent app	36584985171 attempt 1	deploy=false, torn-down notice
Bogus client id	36584985171 attempt 1	gate failed (AADSTS700038)
STAGING_PINNED set for the run, then deleted	36584985171 attempt 2	deploy=false on all three, with the reason in the notice

Attempt 2 was a deliberately pinned re-run, so read attempt 1 for the unpinned cases. All three pre-review agents misread attempt 2 as the only evidence.

The self-test also caught a real bug in run 36584594351. Under Actions' bash -e, exit 3 ended the step before the case statement saw it, so a torn-down staging would have failed the train instead of skipping. Fixed with rc=0; … || rc=$?.

The production if:, evaluated by GitHub over stub jobs (in both self-test runs):

Gate / staging	Production
deploy / staging ok	runs
deploy / staging failed	skipped
skip / staging skipped	runs
gate failed	skipped

deploy.yml, by staging-only manual dispatches of this branch (agreed beforehand):

Run	Path	Result
36703664882	builds	sha-dedad9e = sha256:7910a7ef…b015. GHCR's package version for that tag has the same name. Smoke and metadata checks passed
36704781085	image_digest given, build skipped	the build and GHCR login steps were skipped, and given by the caller (build skipped) is in the summary. Smoke passed

After both, the spin-up check on revision --0000018 gave ReadOnly=true, App Insights set, PASS.

Other checks

• Script tests: 82 passed, 0 failed. The 8 new ABSENT cases fail against main's script and pass against this branch.
• actionlint clean on all workflows.

Not yet proven — needs the real train, agreed with you per run

• A train run with a new tag deploys staging, then production, on the same digest; I'll show both summaries. (The squash-merge of this PR produces that tag.)
• STAGING_PINNED set during a train run: staging is skipped with the reason, and production deploys. Needs a further new tag.
• After a train run, the spin-up check passes on the train-created staging revision.
• A staging smoke failure stopping production is covered by the stub scenario above rather than by breaking staging on purpose.

Known edges, documented rather than fixed

• Concurrency is unverified. GitHub's reusable-workflow docs don't say whether deploy.yml's top-level concurrency applies when it is called. If it does, a queued manual staging dispatch can cancel the train's pending staging call. Production then stops (safe), and recovery is gh run rerun <id> --failed.
• A failed or cancelled train leaves its tag and Release unshipped. Before this change that happened only when production failed. The recovery is recorded in release.yml and CLAUDE.md: re-run the failed jobs, not a fresh train, which finds no new tag and does nothing.
• Outside the train, reusing a tag overwrites it. A manual dispatch that reuses a tag without image_digest still rebuilds over it. The freeze route in CLAUDE.md now says to pass the digest.
• A wrong CONTAINER_APP on the staging environment would read as torn down. The notice names the app it looked for.

Docs

The drift was swept by subject across CLAUDE.md, infra/terraform/README.md, the runbooks and the workflow comments. CLAUDE.md changes:

• the Staging section is rewritten around "tracks the train unless pinned or torn down";
• the pin/unpin commands, including that unpinning does not catch staging up;
• the CI/CD and image-registry rows;
• a Freezing deploys skipped/cancelled table with the right recovery;
• the shared-identity section gains the "considered, does not trigger a revisit" line;
• the teardown table says to delete STAGING_PINNED.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GEqmJd34vDn6rLCMHzZCpG

[Copilot]

Copilot review overview

🟡 Changes recommended

The required-digest failure path can still rebuild and overwrite the GHCR release tag before failing.

Review effort: Balanced
Findings: 1

Open (1)

• Avoid rebuilding and pushing when the required image digest is missing · New

What changed in this PR

Updates the release train to deploy one image to staging before production, while safely handling pinned or absent staging.

Changes:

• Adds staging gating and digest-based deployment reuse.
• Distinguishes absent Container Apps from lookup failures.
• Captures the required Azure role and updates operational documentation.

File	Description
.github/​workflows/​staging-gate.yml	Determines whether staging should deploy.
.github/​workflows/​release.yml	Sequences staging before production.
.github/​workflows/​deploy.yml	Adds digest-based build reuse.
.github/​scripts/​check-serving-revisions.sh	Adds explicit absent-app handling.
.github/​scripts/​tests/​check-serving-revisions.test.sh	Tests absence detection.
infra/​terraform/​identity.tf	Captures the resource-group reader grant.
infra/​terraform/​imports.tf	Adds the role-assignment import.
infra/​terraform/​README.md	Documents deployment prerequisites.
docs/​runbooks/​entra-cutover-staging-validation.md	Clarifies the stale-image cutoff.
CLAUDE.md	Documents the new release process and recovery.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The production release path and digest handoff have not yet been exercised by a real release-train run.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Avoid rebuilding and pushing when the required image digest is missing

[Copilot]

Copilot review overview

🟡 Changes recommended

Cancellation recovery incorrectly uses --failed, which does not rerun cancelled jobs.

Review effort: Balanced
Findings: 2

Open (2)

• Document explicit rerun for cancelled staging jobs · New
• Rerun cancelled staging jobs explicitly, not with --failed · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

It changes the production release path and its complete staging-to-production flow cannot be exercised until the first real post-merge train run.

Review effort: Balanced
Findings: None

Resolved since last review (2)

• Rerun cancelled staging jobs explicitly, not with --failed
• Document explicit rerun for cancelled staging jobs