Deploy every merge to staging, then production; require a staging run before merging

[demegire]

What and why

Merging to main now deploys to production with no manual step, and a change to the running service is run on staging before it can merge.

• deploy (new): when ci passes on main, it deploys that commit to staging. If staging starts cleanly, the same commit goes to production. It can also be run by hand with any commit on main, to redeploy or roll back. PROD_AUTO_DEPLOY=false stops it at staging.
• deploy-staging: now for trying a branch before merging, by pushing it to staging or running the workflow with a ref. The scripts that run as root on the host come from main; the branch only supplies the release. A separate job with no branch code marks the deployed commit with the staging status.
• staging-check (new): sets the staging status on each PR head. Docs-, tests-, evals- and Markdown-only changes pass on their own. Anything else stays pending until that commit is deployed to staging, and a new push resets it. It runs on pull_request_target but never checks out PR code.
• activate_release.sh, used by every deploy:
  • a host lock (/run/tin-lite-deploy.lock), so people and CI never deploy at the same time;
  • stops if inode use is ≥ 90%;
  • reinstalls host setup only when its scripts change, because that restarts the proxy in-flight sandboxes use;
  • migrates before switching;
  • waits for the new process's "Application startup complete" and fails if it restarts instead (/healthz stays 200 from the old process during the drain);
  • keeps the newest 20 releases;
  • never rolls back on its own.
• deploy.sh: prod now uses the release env kept on the host (/etc/tin-lite/release.env), like staging. TIN_LITE_UPLOAD_ENV=1 keeps the old upload path.
• Policy: AGENTS.md, docs/repository-policy.md, CONTRIBUTING and docs/staging.md now say that merging deploys. The PR template gains a Tested on staging section.

Verification

• Deployed this branch to staging four times with deploy.sh:
  • the first run did the host install;
  • the next runs skipped it (no proxy restart);
  • a run with TIN_LITE_RELEASE_SOURCE released from a separate checkout using this checkout's scripts;
  • every run: migrations, the new process reported startup, and both health checks passed.
• test_secret_fences, test_staging_isolation; YAML parses.
• Prod's host setup matches what install_switchboard.sh generates (Caddyfile, units, squid, proxy helpers; the only difference is the stop timeout living in a drop-in, which has the same effect). Its install fingerprint is recorded, so the first automatic prod deploy won't restart the proxy.
• A background security review found three issues in an earlier commit, all fixed here: the branch supplied its own root-run scripts, a status-writing token ran alongside branch code, and the lock was in world-writable /run/lock.

Tested on staging

The staging deploys above. The deploy workflow itself can only run once this is on main: merging this PR is its first run, and it deploys #406–#408 plus this PR to production (prod is at d9d7870; no migrations or catalog executor changes since).

Notes

Operator setup already done:

• tin-lite-prod-deployer service account, trusted only for this repo's production environment (by its ID-based subject), with OS Login and viewer on the prod VM only.
• OS Login turned on for the prod VM. Owners SSH through gcloud compute ssh as <user>_emotionmachine_ai; raw SSH with the old project keys no longer works there.
• /etc/tin-lite/release.env seeded on prod from the current release.
• The production environment is limited to main; the staging environment to main and staging.
• PROD_AUTO_DEPLOY=true.

After merge: add staging as a required status check on main. It isn't required yet, because staging-check has to be on main first.

The maintainer bypass also covers the new check; the policy text says skipping staging for a service change is deploying it untested.

🤖 Generated with Claude Code