Key Rotation

Key Rotation

Procedures for rotating all credentials associated with a strut stack — SSH keys, API keys, database passwords, GitHub secrets, and environment variables.

Key Types

Category	Storage	CLI Prefix	Affects
SSH keys	VPS authorized_keys + keys/ssh-keys.json	keys ssh:*	VPS access, CI/CD
API keys	.env SEMANTIC_API_KEYS + keys/api-keys.json	keys api:*	External consumers
Database passwords	.env + live DB	keys db:rotate	All services
GitHub secrets	GitHub repo secrets	keys github:*	CI/CD pipelines
Env var secrets	.env file	keys env:*	All services

The keys/ directory in each stack stores metadata only (fingerprints, masked values, dates) — never actual secrets.

Quick Reference

# Inventory all tracked keys
strut my-stack keys inventory --env prod

# Discover secrets (local + VPS + GitHub)
strut my-stack keys discover --env prod

# Validate env file completeness
strut my-stack keys env:validate --env prod

# Audit SSH keys
strut my-stack keys ssh:audit --env prod

Full Stack Rotation

Work through each step in order — database passwords require a redeploy, so batch changes.

Step 1: Backup

strut my-stack keys env:backup --env prod
strut my-stack backup all --env prod

Step 2: SSH Keys

strut my-stack keys ssh:rotate <username> --env prod
strut my-stack keys ssh:audit --env prod

# Push new VPS deploy key to repos
strut my-stack keys github:rotate-vps-key \
  --repos "org/repo1,org/repo2,org/repo3"

Step 3: Database Passwords

strut my-stack keys db:rotate postgres --env prod
strut my-stack keys db:rotate neo4j --env prod

# Redeploy immediately (services need new passwords)
strut my-stack deploy --env prod
strut my-stack health --env prod

Step 4: Env Var Secrets

# Auto-rotatable secrets
strut my-stack keys env:rotate --env prod

# Third-party keys (rotate in external service first, then update)
strut my-stack keys env:set MISTRAL_API_KEY "new-key" --env prod
strut my-stack keys env:set GH_PAT "ghp_new..." --env prod

Step 5: API Keys

strut my-stack keys api:list
strut my-stack keys api:rotate <key-name>

Step 6: Sync GitHub Secrets

while IFS= read -r repo; do
  [[ "$repo" =~ ^# ]] && continue
  [[ -z "$repo" ]] && continue
  strut my-stack keys github:sync --repo "$repo" --from .prod.env
done < stacks/my-stack/repos.conf

Step 7: Verify

strut my-stack keys env:validate --env prod
strut my-stack keys env:diff --local .prod.env --remote --env prod
strut my-stack health --env prod --json
strut my-stack keys ssh:audit --env prod

Deploy Keys & CI Bootstrap (since v0.27.0)

For new stacks or CI pipeline setup, use the dedicated commands to generate a deploy keypair and push all required secrets to your CI provider in one step.

Generate a Deploy Key

strut my-stack ssh:keygen --name ci --env prod

This will:

1. Generate an ED25519 keypair named ci (stored locally at ~/.ssh/strut_<host>_ci and ~/.ssh/strut_<host>_ci.pub)
2. Append the public key to the VPS host's authorized_keys
3. Display the key fingerprint for verification

The --name flag lets you maintain multiple deploy keys per stack (e.g. ci, backup-agent, monitoring).

Bootstrap CI Secrets

strut my-stack ci:init --provider github --env prod

This reads the stack's required_vars, the generated deploy key, and host connection details, then pushes them as repository secrets (or environment secrets) to the specified provider. Supported providers:

Provider	Secrets Target
github	GitHub Actions repository secrets (via gh CLI) — default
gitlab	GitLab CI/CD variables (via glab CLI)
manual	Prints the secrets for manual entry (no provider write)

Use --dry-run to preview without writing:

strut my-stack ci:init --dry-run --env prod

Output shows each secret name and its source (env file, generated key, topology) without exposing values.

Typical Workflow

# 1. Generate the deploy keypair
strut my-stack ssh:keygen --name ci --env prod

# 2. Preview what ci:init would set
strut my-stack ci:init --provider github --dry-run --env prod

# 3. Push secrets to GitHub Actions
strut my-stack ci:init --provider github --env prod

# 4. Verify the pipeline can connect
# (trigger a CI run or use `strut my-stack keys ssh:audit`)
strut my-stack keys ssh:audit --env prod

Relationship to Key Rotation

When rotating SSH keys (Step 2 above), you can regenerate the CI deploy key and re-run ci:init to update the provider in one pass:

strut my-stack ssh:keygen --name ci --env prod
strut my-stack ci:init --provider github --env prod

This replaces the manual loop over keys github:sync for the CI-specific keypair.

Registry Credential Rotation (since v0.27.0)

Rotate Docker registry pull tokens (e.g. GitHub Container Registry PATs) across all VPS hosts in one pass.

Rotate Pull Credentials

strut my-stack keys rotate-registry --registry ghcr.io --env prod

This will:

1. Prompt for the new token (input hidden, never logged)
2. Deliver it to each configured host via docker login --password-stdin
3. Record rotation metadata in keys/registry-credentials.json

Options:

• --hosts h1,h2 — target specific hosts (default: all from VPS_HOST)
• --username <user> — registry username (default: GHCR_USER from env)
• --revoke-old — print guidance for revoking the old PAT after rotation
• --dry-run — preview without executing

Check Registry Status

strut my-stack keys registry-status --env prod
strut my-stack keys registry-status --json

Shows per-host: reachability, login state (from ~/.docker/config.json), last rotation date, and username.

Typical Workflow

# 1. Generate a new fine-grained PAT in GitHub UI
# 2. Rotate credentials across all hosts
strut my-stack keys rotate-registry --env prod

# 3. Verify
strut my-stack keys registry-status --env prod

# 4. Revoke old PAT in GitHub settings
#    (strut prints the URL with --revoke-old)

Rotation Schedule

Secret Type	Interval	Also Rotate On
SSH keys	90 days	Team member departure
GH_PAT	90 days	GitHub expiry warning
Database passwords	90 days	Suspected breach
API keys	90 days	Consumer offboarding
Third-party API keys	Per provider	Suspected exposure

Audit log: stacks/<stack>/keys/key-audit.log

Troubleshooting

Services Down After Rotation

strut my-stack logs --tail 100 --env prod
# Most common: services didn't pick up new DB password → redeploy
strut my-stack deploy --env prod

SSH Key Not Working

strut my-stack keys ssh:audit --env prod
ssh -i ~/.ssh/strut-<stack>-vps ubuntu@<VPS_HOST> "echo ok"

Rotation Failed Mid-Process

# Restore from backup
ls .env.backup-*
cp .env.backup-YYYYMMDD-HHMMSS .prod.env
strut my-stack deploy --env prod