Add a staging deployment that cannot touch production's state

.github/workflows/deploy-staging.yml

@@ -0,0 +1,48 @@
+name: deploy-staging
+
+# Staging runs every merge to main, and any branch on request. Production deploys stay manual.
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch, tag or SHA to deploy to staging
+        required: true
+        default: main
+
+permissions:
+  contents: read
+
+# One release at a time; a newer request waits instead of interrupting a deploy midway.
+concurrency:
+  group: deploy-staging
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    # The staging environment is the only identity Google Cloud trusts for this deployer.
+    environment:
+      name: staging
+      url: https://app.staging.tin.computer
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ inputs.ref || github.sha }}
+          persist-credentials: false
+      - uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
+        with:
+          workload_identity_provider: ${{ vars.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ vars.GCP_DEPLOY_SERVICE_ACCOUNT }}
+      - uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1
+      - name: Deploy
+        env:
+          TIN_LITE_DEPLOY_TARGET: staging
+          TIN_LITE_GCP_PROJECT: ${{ vars.GCP_PROJECT }}
+          TIN_LITE_STATIC_IP: ${{ vars.STAGING_STATIC_IP }}
+        run: bash infra/deploy.sh

docs/README.md

@@ -18,6 +18,7 @@ remain authoritative; older contract versions are retained for compatibility.
 ## Self-hosting and operations
 
 - [Engineering reference](engineering.md): configuration, local commands and service APIs.
+- [Staging](staging.md): deploying a change end to end before production, and what it shares.
 - [Optional billing](self-hosted-billing.md), [billing coverage](workflow-billing-coverage.md)
   and [model accounting](model-service-accounting.md).
 - [Authentication and integrations](auth-and-integrations.md),

docs/staging.md

@@ -0,0 +1,82 @@
+# Staging
+
+Hosted Tin has a staging deployment at `https://app.staging.tin.computer` for running a
+change end to end, as production would, before it reaches `app.tin.computer`. Local
+development does not reproduce production's providers, callbacks or release process;
+staging does, with its own state.
+
+A deploy to staging is not a production deployment. Production stays a separate,
+explicitly authorized step.
+
+## Deploying
+
+- Every push to `main` deploys to staging (`.github/workflows/deploy-staging.yml`).
+- Any branch, tag or SHA deploys on request: **Actions → deploy-staging → Run workflow**,
+  or `gh workflow run deploy-staging -f ref=<branch>`. One deploy runs at a time; a newer
+  request waits. A branch deploy stays until the next one, so say in the team channel
+  when you take staging for a branch.
+- From a workstation: `TIN_LITE_DEPLOY_TARGET=staging TIN_LITE_GCP_PROJECT=<project>
+  bash infra/deploy.sh`.
+
+A staging deploy installs the host scripts, unpacks the release, runs `tin-lite migrate`
+from the new release and only then switches `current` and restarts. Its database is
+long-lived and never reset, so it applies every migration and catalog change in the same
+order production will. A release that would crash production's boot (a changed executor,
+a reused built-in ID) crashes staging's first.
+
+The workflow authenticates to Google Cloud with workload identity. Google trusts only
+jobs from this repository in the `staging` GitHub environment, and the deployer can log
+in to the staging host only. The environment's variables name the provider, service
+account, project and static IP.
+
+## What is separate and what is shared
+
+| | Staging |
+|---|---|
+| Host | Its own VM, installed by the same `install_switchboard.sh` |
+| Postgres | Local to the staging host (`install_staging_postgres.sh`) |
+| Temporal | Its own namespace |
+| code.storage | Production's organization, its own key, every repository prefixed `staging-` |
+| Clerk | A development instance |
+| GitHub | Its own GitHub App and founder-account OAuth App |
+| Stripe | Test mode, with its own webhook endpoint |
+| Keyword Planner (gak) | Its own service on the staging host |
+| Models, E2B, data providers | Production's accounts and keys |
+| Google and X OAuth | Production's clients, once the staging callbacks are registered |
+
+`TIN_LITE_ENVIRONMENT=staging` refuses to boot without `TIN_LITE_CODE_STORAGE_REPO_PREFIX`,
+outside Stripe test mode, or when the database host or Temporal namespace equals
+production's (`TIN_LITE_PRODUCTION_DATABASE_HOST`, `TIN_LITE_PRODUCTION_TEMPORAL_NAMESPACE`).
+
+The prefix applies wherever Tin names a repository upstream: lookups, creation, deletion,
+sandbox git remotes and scoped tokens. Stored references keep the unprefixed ID, so
+staging writes `staging-registry/workflows` and `staging-projects/<id>` and never
+production's repositories. The key is still organization-wide; rotate it alone if the
+staging host is compromised.
+
+## Where its configuration lives
+
+On the staging host, readable by root and the service group only:
+
+- `/etc/tin-lite/release.env`: secrets and settings, copied into each release as `.env`.
+  Nothing is uploaded from a workstation.
+- `/etc/tin-lite/clerk.env` and `/etc/tin-lite/runtime.env`: the service environment.
+  `install_switchboard.sh` regenerates the hosts and proxy keys in `runtime.env` and keeps
+  every other line.
+- `/etc/tin-lite/github-app.pem` and `/etc/tin-lite/postgres-password`.
+
+## How it differs from production
+
+- Production's Postgres refuses `DELETE` and `UPDATE` without `WHERE` (`pg_strict`);
+  staging's does not. `tests/test_sql_requires_where.py` catches such statements in CI.
+- The database is on the same host: no network latency, failover or connection drops.
+  Its settings otherwise follow production's (version, `max_connections`, time zone,
+  memory, a role that is not a superuser).
+- Clerk's development instance keeps sessions differently and is shared with the Tin
+  website's staging site, so its sign-in paths point there. Tin's own sign-in works; an
+  MCP connection's sign-in and consent use the instance's hosted pages.
+- Real model, sandbox and data-provider calls cost real money. Keep test projects few and
+  their spending limits low; pause schedules you are not testing.
+
+Fixture tests prove the contract and staging proves the release; production still needs
+its one real run after a production deploy.

infra/activate_release.sh

@@ -26,12 +26,29 @@ if [[ -f /tmp/tin-lite-github-app.pem ]]; then
     's#^TIN_LITE_GITHUB_APP_PRIVATE_KEY_PATH=.*#TIN_LITE_GITHUB_APP_PRIVATE_KEY_PATH=/etc/tin-lite/github-app.pem#' \
     /tmp/tin-lite.env
 fi
-install -m 640 -o tinlite -g tinlite /tmp/tin-lite.env "${release_dir}/.env"
+# An uploaded .env wins; otherwise the release env kept on the host (staging) is used.
+release_env=/tmp/tin-lite.env
+if [[ ! -f "${release_env}" ]]; then
+  release_env=/etc/tin-lite/release.env
+fi
+if [[ ! -f "${release_env}" ]]; then
+  echo "no release env: upload .env or keep /etc/tin-lite/release.env on the host" >&2
+  exit 66
+fi
+install -m 640 -o tinlite -g tinlite "${release_env}" "${release_dir}/.env"
 chown -R tinlite:tinlite "${release_dir}"
 
 sudo -u tinlite env UV_CACHE_DIR=/var/cache/tin-lite-uv \
   /usr/local/bin/uv sync --frozen --no-dev --directory "${release_dir}"
 
+# Migrations run from the new release, under the service env, before it goes live.
+if [[ "${TIN_LITE_RUN_MIGRATIONS:-0}" == "1" ]]; then
+  systemd-run --wait --pipe --collect -p User=tinlite -p WorkingDirectory="${release_dir}" \
+    -p EnvironmentFile=/etc/tin-lite/clerk.env -p EnvironmentFile=/etc/tin-lite/runtime.env \
+    -E UV_CACHE_DIR=/var/cache/tin-lite-uv \
+    /usr/local/bin/uv run --frozen --no-dev tin-lite migrate
+fi
+
 ln -sfn "${release_dir}" /opt/tin-lite/current
 systemctl enable tin-lite-switchboard
 systemctl restart tin-lite-switchboard

infra/deploy.sh

@@ -3,22 +3,53 @@ set -euo pipefail
 
 : "${TIN_LITE_GCP_PROJECT:?Set TIN_LITE_GCP_PROJECT to your deployment project}"
 
+# prod: the release env is uploaded from a local .env. staging: it stays on the host in
+# /etc/tin-lite/release.env, and migrations run before the release goes live.
+target="${TIN_LITE_DEPLOY_TARGET:-prod}"
+case "${target}" in
+  prod)
+    instance=tin-lite-switchboard
+    address=tin-lite-switchboard-ip
+    default_host=app.tin.computer
+    upload_env=1
+    run_migrations="${TIN_LITE_RUN_MIGRATIONS:-0}"
+    health_retries=20
+    ;;
+  staging)
+    instance=tin-lite-staging
+    address=tin-lite-staging-ip
+    default_host=app.staging.tin.computer
+    upload_env=0
+    run_migrations=1
+    # A first boot publishes the whole catalog and a restart drains running workflows.
+    health_retries=100
+    ;;
+  *)
+    echo "TIN_LITE_DEPLOY_TARGET must be prod or staging" >&2
+    exit 64
+    ;;
+esac
+zone=us-central1-a
+region=us-central1
+
 root="$(git rev-parse --show-toplevel)"
 cd "${root}"
 if [[ -n "$(git status --porcelain --untracked-files=no)" ]]; then
   echo "tracked worktree changes must be committed before deployment" >&2
   exit 65
 fi
-if [[ ! -f .env ]]; then
-  echo "local .env is required" >&2
-  exit 66
-fi
-if rg -q '^OPENAI_API_KEY=' .env; then
-  echo "OPENAI_API_KEY is forbidden" >&2
-  exit 67
+github_private_key_path=""
+if [[ "${upload_env}" == "1" ]]; then
+  if [[ ! -f .env ]]; then
+    echo "local .env is required" >&2
+    exit 66
+  fi
+  if grep -q '^OPENAI_API_KEY=' .env; then
+    echo "OPENAI_API_KEY is forbidden" >&2
+    exit 67
+  fi
+  github_private_key_path="$(sed -n 's/^TIN_LITE_GITHUB_APP_PRIVATE_KEY_PATH=//p' .env | tail -n 1)"
 fi
-
-github_private_key_path="$(sed -n 's/^TIN_LITE_GITHUB_APP_PRIVATE_KEY_PATH=//p' .env | tail -n 1)"
 if [[ -n "${github_private_key_path}" ]]; then
   if [[ ! -f "${github_private_key_path}" ]]; then
     echo "TIN_LITE_GITHUB_APP_PRIVATE_KEY_PATH must name a readable local file" >&2
@@ -35,45 +66,57 @@ temporary_dir="$(mktemp -d)"
 trap 'rm -rf "${temporary_dir}"' EXIT
 git archive --format=tar.gz --output="${temporary_dir}/tin-lite-release.tar.gz" HEAD
 
+release_files=("${temporary_dir}/tin-lite-release.tar.gz")
+if [[ "${upload_env}" == "1" ]]; then
+  release_files+=(.env)
+fi
 gcloud compute scp \
-  "${temporary_dir}/tin-lite-release.tar.gz" \
-  .env \
+  "${release_files[@]}" \
   infra/install_switchboard.sh \
   infra/install_forward_proxy.sh \
   infra/proxy_policy.py \
   src/tin_lite/proxy_grants.py \
   infra/activate_release.sh \
-  tin-lite-switchboard:/tmp/ \
-  --project "${TIN_LITE_GCP_PROJECT}" --zone us-central1-a --quiet
+  "${instance}:/tmp/" \
+  --project "${TIN_LITE_GCP_PROJECT}" --zone "${zone}" --quiet
 
 if [[ -n "${github_private_key_path}" ]]; then
   gcloud compute scp \
     "${github_private_key_path}" \
-    tin-lite-switchboard:/tmp/tin-lite-github-app.pem \
-    --project "${TIN_LITE_GCP_PROJECT}" --zone us-central1-a --quiet
-  gcloud compute ssh tin-lite-switchboard \
-    --project "${TIN_LITE_GCP_PROJECT}" --zone us-central1-a --quiet \
+    "${instance}:/tmp/tin-lite-github-app.pem" \
+    --project "${TIN_LITE_GCP_PROJECT}" --zone "${zone}" --quiet
+  gcloud compute ssh "${instance}" \
+    --project "${TIN_LITE_GCP_PROJECT}" --zone "${zone}" --quiet \
     --command="sudo chown root:root /tmp/tin-lite-github-app.pem && \
 sudo chmod 600 /tmp/tin-lite-github-app.pem"
 fi
 
-static_ip="$(gcloud compute addresses describe tin-lite-switchboard-ip \
-  --project "${TIN_LITE_GCP_PROJECT}" --region us-central1 --format='value(address)')"
-product_host="${TIN_LITE_PRODUCT_HOST:-app.tin.computer}"
+static_ip="${TIN_LITE_STATIC_IP:-}"
+if [[ -z "${static_ip}" ]]; then
+  static_ip="$(gcloud compute addresses describe "${address}" \
+    --project "${TIN_LITE_GCP_PROJECT}" --region "${region}" --format='value(address)')"
+fi
+if [[ ! "${static_ip}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+  echo "the ${target} static IP is not an IPv4 address" >&2
+  exit 64
+fi
+product_host="${TIN_LITE_PRODUCT_HOST:-${default_host}}"
 if [[ ! "${product_host}" =~ ^[a-zA-Z0-9.-]+$ ]]; then
   echo "TIN_LITE_PRODUCT_HOST must be a DNS hostname" >&2
   exit 64
 fi
-gcloud compute ssh tin-lite-switchboard \
-  --project "${TIN_LITE_GCP_PROJECT}" --zone us-central1-a --quiet \
+move_env=""
+if [[ "${upload_env}" == "1" ]]; then
+  move_env="sudo mv /tmp/.env /tmp/tin-lite.env && "
+fi
+gcloud compute ssh "${instance}" \
+  --project "${TIN_LITE_GCP_PROJECT}" --zone "${zone}" --quiet \
   --command="sudo env TIN_LITE_STATIC_IP=${static_ip} TIN_LITE_PRODUCT_HOST=${product_host} bash /tmp/install_switchboard.sh && \
-sudo mv /tmp/.env /tmp/tin-lite.env && \
-sudo bash /tmp/activate_release.sh ${release_sha}"
+${move_env}sudo env TIN_LITE_RUN_MIGRATIONS=${run_migrations} bash /tmp/activate_release.sh ${release_sha}"
 
 public_host="tin-lite-switchboard.${static_ip//./-}.sslip.io"
-curl --http1.1 --fail --silent --show-error --retry 20 --retry-all-errors --retry-delay 3 \
+curl --http1.1 --fail --silent --show-error --retry "${health_retries}" --retry-all-errors --retry-delay 3 \
   "https://${public_host}/healthz"
 
-product_host="${TIN_LITE_PRODUCT_HOST:-app.tin.computer}"
-curl --http1.1 --fail --silent --show-error --retry 20 --retry-all-errors --retry-delay 3 \
+curl --http1.1 --fail --silent --show-error --retry "${health_retries}" --retry-all-errors --retry-delay 3 \
   "https://${product_host}/healthz"

infra/install_staging_postgres.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# Staging's database: stock Postgres on the staging host, set up to behave like production's
+# managed Postgres where Tin can tell the difference. Production's pg_strict (no DELETE or
+# UPDATE without WHERE) is not available here; tests/test_sql_requires_where.py covers it.
+set -euo pipefail
+
+if [[ "${EUID}" -ne 0 ]]; then
+  echo "run as root" >&2
+  exit 64
+fi
+
+major="${TIN_LITE_POSTGRES_MAJOR:-18}"
+# Production's exact build when the archive still has it; otherwise the latest minor.
+version="${TIN_LITE_POSTGRES_VERSION:-18.6-1.pgdg12+2}"
+database=tin
+role=tin_app
+password_file=/etc/tin-lite/postgres-password
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get update
+apt-get install -y ca-certificates curl gnupg postgresql-common
+if [[ ! -f /etc/apt/sources.list.d/pgdg.list ]]; then
+  install -d /usr/share/postgresql-common/pgdg
+  curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
+    -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
+  echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release && echo "${VERSION_CODENAME}")-pgdg main" \
+    > /etc/apt/sources.list.d/pgdg.list
+  apt-get update
+fi
+if apt-cache madison "postgresql-${major}" | awk -F'|' '{gsub(/ /, "", $2); print $2}' |
+  grep -qxF "${version}"; then
+  apt-get install -y "postgresql-${major}=${version}"
+  apt-mark hold "postgresql-${major}"
+else
+  echo "postgresql-${major} ${version} is not available; installing the latest ${major}.x" >&2
+  apt-get install -y "postgresql-${major}"
+fi
+
+install -d -m 755 "/etc/postgresql/${major}/main/conf.d"
+cat > "/etc/postgresql/${major}/main/conf.d/tin-staging.conf" <<'EOF'
+# Production's values where they shape behavior Tin can observe.
+listen_addresses = 'localhost'
+max_connections = 25
+ssl = on
+timezone = 'UTC'
+log_timezone = 'UTC'
+work_mem = '2MB'
+maintenance_work_mem = '16MB'
+shared_buffers = '64MB'
+idle_in_transaction_session_timeout = 0
+random_page_cost = 1.1
+effective_io_concurrency = 32
+default_toast_compression = 'lz4'
+EOF
+systemctl enable "postgresql@${major}-main"
+systemctl restart "postgresql@${major}-main"
+
+install -d -m 750 -o root -g tinlite /etc/tin-lite 2>/dev/null || install -d -m 750 /etc/tin-lite
+if [[ ! -s "${password_file}" ]]; then
+  umask 077
+  openssl rand -hex 32 > "${password_file}"
+fi
+chmod 600 "${password_file}"
+password="$(cat "${password_file}")"
+
+# Like production's application role: no superuser, cannot create databases or roles.
+sudo -u postgres psql -v ON_ERROR_STOP=1 -q <<SQL
+DO \$\$
+BEGIN
+  IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${role}') THEN
+    CREATE ROLE ${role} LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE;
+  END IF;
+END
+\$\$;
+ALTER ROLE ${role} PASSWORD '${password}';
+SQL
+if ! sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname = '${database}'" | grep -q 1; then
+  sudo -u postgres createdb --owner "${role}" "${database}"
+fi
+echo "postgres ready: database ${database}, role ${role}, password in ${password_file}"

scripts/complete_organic_audit.py

@@ -79,6 +79,7 @@ async def main():
             storage = CodeStorage(
                 organization=settings.code_storage_org,
                 private_key=settings.code_storage_api_key.get_secret_value(),
+                repo_prefix=settings.code_storage_repo_prefix,
             )
             definition = json.loads(
                 await storage.read_canonical_artifact(