◆ orchestrator

A local CLI to spin up Hetzner Cloud dev VMs from a single cloud-config template, verify each one is dev-ready, and manage their lifecycle. Each VM is reachable at https://<name>.<base-domain> via Caddy with a Let's Encrypt cert.

Single operator, single workstation. State lives in ~/.dev-orchestrator/. No daemon, no remote DB, no async — concurrent.futures.ThreadPoolExecutor with hard caps (5 create, 8 test, 3 power/destroy).

_{▸ The workshop admin dashboard — live pool, claims, costs, and one-click re-test.}

▸ Install

With uv (recommended)

uv handles the Python version and virtualenv automatically. Install uv once (curl -LsSf https://astral.sh/uv/install.sh | sh), then:

git clone git@github.com:casushq/orchestrator.git
cd orchestrator
uv sync

uv sync reads pyproject.toml, fetches Python 3.11 if needed, creates .venv, and installs all dependencies plus the dev console script. Run it as .venv/bin/dev … or activate first (source .venv/bin/activate).

To include the test dependencies:

uv sync --extra dev

With pip

Requires Python 3.11+.

git clone git@github.com:casushq/orchestrator.git
cd orchestrator
python3.11 -m venv .venv
.venv/bin/pip install -e .

The console script dev is installed in .venv/bin/. Either activate the venv (source .venv/bin/activate) or call it directly (.venv/bin/dev …).

On macOS with pyenv (python3.11: command not found even though pyenv versions lists 3.11.x), the global pyenv version is older. Either pin pyenv for this directory:

pyenv local 3.11.13              # writes .python-version
python -m venv .venv
.venv/bin/pip install -e .

…or call the binary directly:

~/.pyenv/versions/3.11.13/bin/python -m venv .venv
.venv/bin/pip install -e .

▸ Prerequisites

Before dev init, have these set up once on the Hetzner side:

1. Hetzner Cloud project with an SSH key uploaded.
2. A DNS zone in the Hetzner Console whose name is BASE_DOMAIN or any parent of it (e.g. casus.org works for a base of dev.casus.org, and a delegated dev.casus.org zone works too). DNS lives on the unified Cloud API now (api.hetzner.cloud); the legacy dns.hetzner.com console + token are deprecated as of May 2026, so the orchestrator uses the same HETZNER_TOKEN for both compute and DNS.

The orchestrator manages A records inside the matching zone — dev create adds <name>.<base-domain> → VM IPv4, dev destroy removes it. Zone discovery is automatic: it picks the longest-suffix-match against BASE_DOMAIN from your project's zones.

▸ First-run wizard

dev init

Interactively prompts for every value below, validates the Hetzner Cloud token, the SSH key existence, and the Hetzner DNS token + zone, then writes ~/.dev-orchestrator/.env with mode 0600 and creates an empty state.db.

Field	Notes
HETZNER_TOKEN	Hetzner Cloud API token. Used for both compute and DNS — DNS lives on the unified api.hetzner.cloud now. Validated against the API.
BASE_DOMAIN	Suffix for VM FQDNs (e.g. dev.casus.org). VMs reachable as <name>.<base-domain>. The orchestrator picks the matching DNS zone automatically.
CADDY_EMAIL	Email registered with Let's Encrypt for the wildcard cert account. Receives expiry notifications.
SSH_KEY_NAME	Name of an SSH key already uploaded to your Hetzner project. Existence is verified.
SSH_PRIVATE_KEY_PATH	Path on your workstation; used by dev ssh, dev logs, dev sync, and the SSH-based health checks. ~ is expanded.
USERNAME	Login user provisioned inside each VM. Default human. Used as the SSH login account, sudoers entry, and home-directory owner.
USER_PASSWORD_HASH	SHA-512 ($6$…) password hash for the USERNAME account inside each VM. dev init asks for the password as plaintext (entered twice, masked) and hashes it for you — this field stores the resulting hash, never the plaintext.
SSH_PUBKEY	Contents of the public key matching SSH_PRIVATE_KEY_PATH. The wizard auto-detects this if <key>.pub exists.
OPENROUTER_API_KEY	Baked into ~/.pi/agent/auth.json on each VM.
GITHUB_TOKEN	Optional; only needed if BOILERPLATE_REPO is private.
OFF_S3_ENDPOINT	Optional. Hetzner Object Storage endpoint, e.g. https://nbg1.your-objectstorage.com. When set, VMs download parquet files from S3 instead of HuggingFace.
OFF_S3_BUCKET	Optional. S3 bucket name, e.g. openfoodfacts-dataharvest-2026. Pushed to VMs as a public download URL (no credentials).
OFF_S3_ACCESS_KEY	Optional. Hetzner Object Storage access key ID. Required by scripts/setup_object_storage.py and scripts/upload_parquet.py on the operator machine. Not pushed to VMs.
OFF_S3_SECRET_KEY	Optional. Hetzner Object Storage secret key. Required by the upload scripts. Not pushed to VMs.
BOILERPLATE_REPO	Format: github.com/owner/repo — no https:// prefix, no .git suffix, no git@github.com: SSH form. The cloud-config builds the HTTPS URL from this.
DEFAULT_SERVER_TYPE, DEFAULT_LOCATION, DEFAULT_IMAGE, DEFAULT_MODEL	Defaults: cx23, nbg1, ubuntu-24.04, anthropic/claude-sonnet-4.6. DEFAULT_IMAGE accepts a system/app image name (e.g. ubuntu-24.04, docker-ce), a numeric image id, or the description of a snapshot/backup in your project (e.g. claude-pink).
DEV_PORT	Port Caddy reverse-proxies to (i.e. where the participant runs their dev server). Default 3000.

Output:

• ~/.dev-orchestrator/.env (mode 0600)
• ~/.dev-orchestrator/state.db (servers + test_runs schema)
• ~/.dev-orchestrator/logs/ (per-day append-only logs)
• ~/.dev-orchestrator/certs/ (wildcard cert + key + ACME account key, 0600)

At the end of dev init, the wizard offers to obtain the wildcard cert inline (~30–60s). Decline and run dev cert init later if you prefer. dev create hard-fails without the cert, so it must exist before you provision any VMs.

▸ Wildcard TLS cert

Per-VM Let's Encrypt issuance hits the 5-certs-per-week-per-identifier rate limit fast in workshop scenarios (a few destroy/recreate cycles for the same hostname is enough). The orchestrator instead obtains a single *.<base-domain> + <base-domain> cert via DNS-01 against your Hetzner DNS zone, and pushes it into every VM's cloud-config. Caddy on each VM serves the static cert — it never talks to ACME, so it can't be rate limited regardless of how many times you provision the same name.

dev cert init                 # obtain a fresh cert (overwrites any existing)
dev cert init --staging       # use Let's Encrypt staging (untrusted, no rate limit)
dev cert renew                # renew if expiring within 30 days
dev cert renew --force        # renew unconditionally
dev cert show                 # print expiry + SAN list

The cert lives at ~/.dev-orchestrator/certs/fullchain.pem (mode 0644) and privkey.pem (mode 0600). The ACME account key is in the same directory at account-key.pem. Renewal is manual — Let's Encrypt issues 90-day certs; renew at 60 (or dev create will warn at 30).

The DNS-01 challenge places a multi-value TXT record at _acme-challenge.<base-domain> (one validation per name in the cert), waits ~30s for propagation, answers the challenges, and tears the TXT record down. Uses the same HETZNER_TOKEN as everything else, the unified Hetzner Cloud DNS API.

▸ Workshop lifecycle

One-time setup on a fresh laptop, then the workshop sequence top to bottom.

Phase	Command	When	Notes
One-time	dev init	Once per laptop	Writes ~/.dev-orchestrator/.env, validates Hetzner token, picks the DNS zone.
One-time	dev cert init	After dev init	Issues the wildcard cert via DNS-01. Repeat every 60 days with dev cert renew --force.
One-time	dev snapshot <prep-vm>	Optional, before any workshop	Bakes the stable installs into a Hetzner snapshot. Cuts cloud-init from 15 minutes to ~4.
Pre-workshop	dev admin deploy --workshop <slug>	Day before	Provisions the admin VM at the BASE_DOMAIN apex. Prompts for admin password, workshop password, and the plaintext SSH password shown to participants. The SSH password must match the VM password configured by dev init.
Pre-workshop	dev create --count N --workshop <slug>	Day before	Provisions the pool, 10% over-warmed so broken VMs and re-claims don't deplete it.
Pre-workshop	dev test all --workshop <slug>	Day before	Confirms every VM is 18/18 green. Anything below 18 gets dev destroy <name> plus a replacement.
Live	Admin dashboard	Workshop day	Monitor + one-click re-test.
Post-workshop	dev destroy all --workshop <slug>	After	Deletes DNS records first, then VMs, then state rows.
Post-workshop	dev admin destroy	After	Removes the admin VM and its DNS record.

Workshop-day playbook

T minus 1 day.

1. dev cert renew --force if the wildcard is within 30 days of expiry. dev cert show to verify.
2. dev admin deploy --workshop <slug>. Prompts for three secrets; write them down.
3. dev create --workshop <slug> --count <participants + 10%>. Wait for cloud-init (~15 min on stock, ~4 min from a snapshot).
4. dev test all --workshop <slug>. Anything below 18/18: dev destroy <name> plus a replacement.
5. Open https://<base>/admin, log in, set the workshop name and (optionally) logo from settings.

T minus 1 hour.

1. Open https://<base>/admin on a second monitor. Leave it open during the workshop.
2. Walk the room with the workshop password.

During the workshop.

1. Dashboard refreshes every 10 s. Filter by name to find specific participants.
2. If a participant reports a broken VM: click the row, hit Re-test. If it stays red, tell them to re-submit at /name (they get a fresh VM from the over-warm).
3. If a participant is stuck on SSH: walk them through the workspace page step by step. Pi auto-attaches a tmux session, so reconnects pick up where they left off.

Post-workshop.

1. dev destroy all --workshop <slug>. Confirms with the count.
2. dev admin destroy. Confirms with the VM name.
3. Hetzner project is back to baseline. ~/.dev-orchestrator/state.db is empty for this workshop.

▸ Admin server (workshop variant)

For workshops (35 participants self-claiming pre-warmed VMs from a browser), provision a small admin VM in the same Hetzner project:

dev admin deploy --workshop dataharvest-2026

This creates <base-domain> (e.g. dev.casus.org), fronted by Caddy with the same wildcard cert workshop VMs serve. Prompts once for the admin UI password, the workshop password (participants type this), and the plaintext SSH password shown on the per-VM workspace page. Configuration is stored in /etc/orchestrator-admin/env; orchestrator source is rsynced into /opt/orchestrator-admin/; the FastAPI app runs under a systemd unit orchestrator-admin.service.

Once running:

• https://<base-domain>/ is the participant claim flow (password, name, atomic claim, workspace page at /server/<vm-name>). The SSH password prompt is validated against the VM login hash from dev init so the page cannot show a stale password.
• https://<base-domain>/admin is the dashboard (HTTP Basic): pool table with 10-second auto-refresh, name filter, Hetzner + OpenRouter cost summary (60-second cache).
• https://<base-domain>/admin/server/<name> is the per-VM page with a Re-test button (sync, 15-second budget, runs every check except cloud_init_done).
• https://<base-domain>/admin/settings rotates the workshop password without redeploy.

Lifecycle: provision the day before, destroy after.

dev admin deploy --workshop X    # day before
# ... workshop happens ...
dev admin destroy                # after

dev admin update rsyncs your local source and restarts the unit without recreating the VM. dev admin logs streams journalctl -u orchestrator-admin. Re-running dev admin deploy --workshop X against an existing admin VM falls through to the update path and prints a yellow warning if X differs from the slug baked into the admin's env file (rotate slugs by destroying

• redeploying).

dev admin deploy is idempotent on a stale apex DNS record. If a prior deploy was interrupted between VM create and state.add_server, the apex A record can be left dangling in your zone — pointing at a destroyed VM. The deploy detects this, deletes the stale record, prints a yellow Replaced stale apex A record at ... warning, and proceeds with the new IP. No manual hcloud dns rrset delete needed.

The pool itself is still provisioned with the CLI, just with the workshop label set so the admin can find them:

dev create --workshop dataharvest-2026 --count 40 --name-prefix dh

(Over-warm by ~10% so broken VMs + accidental re-claims don't deplete the pool.)

Admin dashboard

https://<BASE_DOMAIN>/admin after HTTP Basic login.

• Stats row. Three tiles, 60-second cache: Claims (N / M participants claimed), Hetzner (VM hours since creation in EUR), OpenRouter (lifetime usage on the shared key in USD).
• Filter input. Live filters the pool table by VM name or claimed participant name (HTMX debounced 200 ms).
• Pool table. One row per VM, refreshing every 10 seconds: VM name (linked to detail), status badge, IPv4, claimed-by, last-test verdict. Click a VM name to open the detail page.
• Per-VM detail (/admin/server/<vm>): Live /status.json from the VM plus a Re-test button that re-runs the admin-safe 17-check subset synchronously within a 15-second budget.

Admin settings

/admin/settings. Three sections; each saves independently without redeploying the admin server.

Section	What it sets	Takes effect
Workshop name	Human-readable label shown on participant pages. Empty falls back to the slug.	Next request
Workshop logo	URL to a square logo image. Empty falls back to the bundled logo. Non-square images are center-cropped.	Next request
Workshop password	bcrypt-hashed password participants type at the front page.	Next request. Existing 4-hour sessions stay valid.

Participant flow

1. GET / → password page.
2. POST /auth with workshop password → sets 4-hour cookie, redirects to /name.
3. POST /claim with typed name → atomically picks oldest unclaimed VM, labels it, redirects to /server/<vm-name>.
4. /server/<vm-name> → workspace page: SSH command, plaintext SSH password, recovery URL, tmux quickstart.
5. ssh human@<vm>.<base> → tmux session main auto-attached.

tmux sessions

Every SSH login automatically attaches to a persistent tmux session named main (created on first login, reattached on every subsequent one). This keeps work intact across disconnections — participants never lose their terminal state.

If a session is unresponsive but the terminal still accepts input: type exit or Ctrl+D inside the tmux session to kill it. The login shell is still alive (no exec replacement), so you land at a bare bash prompt; log out and reconnect to get a fresh session.

If the terminal is completely frozen: run this from your local machine:

ssh <user>@<host> 'tmux kill-session -t main'

Non-interactive SSH bypasses the auto-attach (no TTY), kills the stuck session, and exits. The next normal login starts a clean new one.

CLI usage telemetry

If your operator .env has ADMIN_URL=https://<base> set, every dev invocation fires a fire-and-forget POST to /admin/heartbeat (2-second timeout, daemon thread, silent on failure). The admin appends one JSON line per call to /var/log/orchestrator-admin/cli-usage.log. Sensitive flag values (--password, --*-token, --secret, --api-key) are redacted before send. The endpoint is unauthenticated in v1 by design.

To disable, omit ADMIN_URL from your .env.

▸ Common workflows

Create a batch

dev create --count 3

Walks you through a defaults table, lets you override any field, asks to confirm, then for each server: submits a Hetzner create (concurrency 5), waits until it reports running, allocates a DNS A record <name>.<base-domain> → IPv4 in your Hetzner DNS zone. Cloud-init (which installs pi, Claude, Caddy, the status page generator and clones the boilerplate) continues asynchronously after the command returns — dev test is how you confirm bootstrap is finished.

Non-interactive:

dev create --count 5 --no-prompt
dev create --count 1 --location fsn1 --server-type cax21 --no-prompt

Timing. The Hetzner create call returns in ~30 seconds (server status running). DNS A record creation adds ~1 second. Cloud-init plus bootstrap.sh then take roughly 10–15 minutes on a cax11: ~3 min for apt-get dist-upgrade, then serial installs of NodeSource Node, GitHub CLI, pi, Claude, Caddy, plus the boilerplate clone and the first cert issuance from Let's Encrypt. Run dev logs <name> --follow to watch in real time.

Server type selection

dev create builds its candidate list from the live Hetzner catalog, not from DEFAULT_SERVER_TYPE / DEFAULT_LOCATION. The selection policy is intentionally hardcoded and not exposed in any settings menu:

1. Locations: nbg1 (Nuremberg), fsn1 (Falkenstein), hel1 (Helsinki). EU only, both DE and FI. Latency between EU datacenters is single-digit ms — irrelevant for a workshop VM.
2. Family: architecture == "x86" and cpu_type == "shared", read from the structured API fields. Today this matches cx (Intel) and cpx (AMD); tomorrow it auto-includes any new shared-x86 family Hetzner adds. Excludes ARM (cax, trips the boilerplate's NodeSource fallback uncomfortably often) and dedicated x86 (ccx, ~3-5x the price for no workshop benefit).
3. Availability: every candidate is checked against the per-location available: true flag the API attaches to each server type. Combos the API knows are unsupported never even make the list — no more "unsupported location for server type" from a stale default.
4. Disk floor: every candidate must have disk >= max(requested-type.disk, image.disk_size). The operator's --server-type works as a floor, never a ceiling: pick cpx32 and the ladder will not silently fall back to a 40 GB box for cost savings. When booting a snapshot, the source VM's disk_size is read and combined with that floor — snapshots from a 160 GB VM refuse to boot on smaller disks at the Hetzner API level (returns non-retryable service_error), so the ladder filters those out before any attempt.
5. Order: cheapest gross-hourly EUR first, walking up the ladder. Mixing across locations and across vendors (Intel vs AMD) is fine — cx22 in fsn1 beats cx32 in nbg1 if it's cheaper, and cpx22 is just as good as cx22 for the workshop's x86_64 workload. The fallback retries the next cheapest only on Hetzner resource_unavailable / placement_error (genuine capacity issues).

The operator's DEFAULT_SERVER_TYPE / DEFAULT_LOCATION and the --server-type / --location CLI flags are honored as the first attempt only, and only if that combo is in the valid (and disk-qualifying) set. If the operator asks for something the API says is invalid (e.g. cx23 in nbg1 when Hetzner has marked that combo unavailable, or a 40 GB type for a 160 GB snapshot), the ladder skips straight to the cheapest valid combo instead of failing.

To override the policy entirely (e.g. to test ARM, or to force a specific size for benchmarking):

dev create --server-type cax21 --location fsn1 --no-fallback

--no-fallback disables the ladder; the create attempts exactly the requested (server_type, location) and fails with whatever Hetzner returns. The pure helper eu_x86_shared_price_ladder in src/dev_orchestrator/hetzner.py is the single source of truth for the policy.

Verify a batch

dev test all                      # parallel up to 8 servers, sequential per server
dev test dev-2026-05-09-01

Runs 17 checks per server (see Health checks below). Most are derived from a single fetch of /status.json per VM, so the suite is fast even on big batches. Results are persisted to state.db and surface in dev list and dev show.

Pre-bake a snapshot (recommended for data-heavy workshops)

Workshop skills that need large datasets (e.g. the 7.5 GB Open Food Facts parquet) add 30–90 minutes to each VM's first-boot time. Instead of waiting on 35 VMs in parallel, stage the data on one prep VM and snapshot it:

dev create --count 1 --no-prompt --server-type cpx32   # 160 GB disk, fits OFF dataset
# ... wait for bootstrap + data download (dev test will show datasets_ready: fail → pass) ...
dev snapshot <prep-vm>                                 # captures full disk incl. staged data
# prints: DEFAULT_IMAGE=<image-id>

Set DEFAULT_IMAGE=<image-id> in ~/.dev-orchestrator/.env, then dev create --count 35 clones from the snapshot. Each clone boots with data already in place — cloud-init re-runs (new instance-id) but the prep-service sentinel is already there so datasets_ready passes immediately.

The orchestrator picks an appropriate server type for the clones automatically: the ladder enforces server.disk >= snapshot.disk_size, so a 160 GB snapshot will never be attempted on (and silently fail against) a 40 GB box. You don't need to pin --server-type on the clones unless you want more RAM than the cheapest qualifying type.

--image accepts three forms; use whichever is convenient:

• Numeric image id (e.g. 390582140) — fastest, no description lookup.
• Snapshot description (e.g. dh-prep-01-snapshot-20260526) — what dev snapshot prints. Not the source VM's name.
• System image name (e.g. ubuntu-24.04) — for fresh boots.

Snapshot disk size matters: the snapshot is tied to its source VM's disk size, not the bytes actually used. A 160 GB prep VM snapshot forces clones onto 160 GB+ server types regardless of how empty the filesystem is. Pick the smallest viable prep VM size for the data you need to stage.

Destroy the prep VM after snapshotting:

dev destroy <prep-vm>

Iterate on the boilerplate

dev sync all                       # git pull + scripts/bootstrap.sh on each server

Lets you change skills/prompts in your boilerplate repo without rebuilding VMs. If the repo doesn't ship a scripts/bootstrap.sh, sync just runs the pull and reports ok.

Tear down

dev destroy dev-2026-05-09-01      # confirm by typing the name back
dev destroy all                     # confirm by typing the count back

Destroy deletes the DNS A record first, then the Hetzner VM, then the local state row. If the VM teardown fails the DNS record is still removed (so a future tenant doesn't inherit a stale record pointing at their IP).

Day-to-day

dev list                           # everything you manage, with status + last test
dev show dev-...-01                # one server, full detail + per-check breakdown + fqdn
dev power off all                  # off-all needs typed-count confirmation
dev power on dev-...-01
dev ssh dev-...-01                 # exec into ssh as human
dev logs dev-...-01 -f             # follow /var/log/dev-bootstrap.log

▸ Per-VM status page

Every VM publishes a status page at https://<name>.<base-domain>/status (HTML) and /status.json (machine-readable). The Caddyfile serves both from /var/www/status regardless of whether the user's dev server on DEV_PORT is up — so you can always tell what's happening on the box.

A shell script at /usr/local/bin/dev-status-gen.sh regenerates both files. It runs:

• Once at the end of bootstrap.sh so /status exists immediately.
• Every 60 seconds thereafter via the dev-status.timer systemd unit (with OnBootSec=30s).

The JSON payload contains, with no secrets:

{
  "generated_at": "2026-05-10T...Z",
  "hostname": "dataharvest-01",
  "fqdn": "dataharvest-01.dev.casus.org",
  "uptime_seconds": 600,
  "bootstrap_complete": true,
  "services": { "caddy": "active", "fail2ban": "active", "ufw": "active" },
  "tools": {
    "pi": "0.74.0", "claude": "2.1.138", "node": "24.15.0",
    "rg": "14.0.3", "fd": "9.0.0", "gh": "2.40.0"
  },
  "repo": { "present": true, "sha": "abc123def..." },
  "skill": { "present": true, "sha": "def456abc..." },
  "parquet": { "count": 54, "expected": 54, "empty": 0, "complete": true },
  "dev_port": 3000,
  "dev_port_listening": false,
  "datasets": [{"name": "openfoodfacts", "state": "ready", "done": true}]
}

Requests to https://<name>.<base-domain>/output/* serve static files from ~/output directly. This is the default path for presentation HTML: save ~/output/report.html, then open https://<name>.<base-domain>/output/report.html.

Anything else hitting https://<name>.<base-domain>/<anything> is reverse-proxied to localhost:DEV_PORT — that's where the participant runs their app.

▸ Health checks (dev test)

Each check is a function in checks.py that returns a CheckResult (name, passed, duration_ms, message). They run sequentially against one server, and dev test all parallelises across servers up to 8 at a time. Most checks read from a single /status.json fetch per server, so they're cheap and don't add SSH load.

#	Name	Source	Asserts	Typical failure
1	ssh_reachable	SSH	human can run a command	sshd still restarting from apt-get dist-upgrade; key not authorised; fail2ban ban (rare on a fresh VM)
2	cloud_init_done	SSH	cloud-init status --wait exits 0 within 180s	package_upgrade still running; write_files errors (older template versions before defer:true)
3	dns_resolves	DNS	<fqdn> resolves locally to the IPv4 in state	DNS record never created; record stale after manual rename; resolver caching
4	https_reachable	HTTPS	GET https://<fqdn>/status.json returns 200	Caddy not up; cert issuance still in progress; firewall blocking 443
5	cert_valid	TLS	TLS handshake succeeds against system trust store (static wildcard cert from orchestrator)	Cert files missing on disk; key permissions wrong; cert expired (dev cert renew)
6	status_fresh	status.json	generated_at within the last 5 minutes	systemd timer not enabled; generator script crashed
7	bootstrap_complete	status.json	bootstrap_complete: true	An installer step failed under set -euo pipefail; check dev logs <name>
8	caddy_running	status.json	services.caddy == "active"	Caddyfile syntax error; cert issuance loop
9	fail2ban_running	status.json	services.fail2ban == "active"	Package install failed during cloud-init
10	ufw_active	status.json	services.ufw == "active"	runcmd didn't run cleanly
11	pi_installed	status.json	tools.pi non-empty	NodeSource setup failed; pi installer aborted
12	claude_installed	status.json	tools.claude non-empty	claude.ai/install.sh network fetch failed
13	core_tools	status.json	rg, fd, gh, node all reported	One of those installers failed; fd symlink not created
14	repo_cloned	status.json	repo.present == true	BOILERPLATE_REPO format wrong; private repo without valid GITHUB_TOKEN
15	skill_loaded	status.json	skills/openfoodfacts/SKILL.md exists	boilerplate checkout is stale or the skill was not wired into the repo
16	datasets_ready	status.json	every entry in datasets[] has state == "ready"	background data-prep service still running; download failed; a parquet file is zero bytes — check systemctl status off-data-prep on the VM
17	pi_auth_present	SSH	~/.pi/agent/auth.json mode is 600	defer:true missing on the write_files entry (file never written)
18	pi_calls_openrouter	SSH	pi -p 'reply with the single word ok' returns "ok"	OPENROUTER_API_KEY invalid; chosen DEFAULT_MODEL doesn't exist on OpenRouter; model with heavy preamble obscures "ok"

Result table is sorted alphabetically by (server, check). Persisted to state.db.test_runs so dev list and dev show can show the last verdict (pass (N/18)) without re-running.

A failing check exits non-zero so dev test is CI-friendly; a single failure does not abort the rest of the suite.

▸ What's installed on each VM

The cloud-config template at templates/cloud-config.yaml.tmpl provisions:

• human user with sudo, SSH-key auth, password auth for participants, and NOPASSWD:ALL sudoers entry
• ufw firewall (22, 80, 443 only), fail2ban, unattended-upgrades
• Node.js (NodeSource LTS, with bundled npm), GitHub CLI, ripgrep, fd, jq, tmux
• pi coding agent (~/.pi/agent/auth.json points at OpenRouter, the default model is ${DEFAULT_MODEL} from your .env)
• Claude Code CLI
• Caddy reverse-proxy on https://<hostname>.<base-domain>, serving the static wildcard cert obtained by the orchestrator (no per-VM ACME)
• Static publishing at /output/* from ~/output, alongside the normal / reverse proxy to DEV_PORT
• Status page generator + 60s systemd timer
• Your boilerplate repo cloned into ~/dev/workshop-boilerplate
• A participant front door at ~/workshop with links to assignments, skill docs, data, and output

Bootstrap output is captured to /var/log/dev-bootstrap.log and the marker Bootstrap complete. is what the bootstrap_complete check looks for.

▸ Architecture

src/dev_orchestrator/
  cli.py        Typer entrypoints; wires modules together
  config.py     dataclass over ~/.dev-orchestrator/.env
  state.py      sqlite3 wrapper (servers + test_runs; lazy column upgrades)
  template.py   string.Template renderer (safe_substitute, preserves
                bash $HOME / ${FOO:-} and apt's $${distro_id} escapes)
  hetzner.py    hcloud SDK wrapper (create/list/delete/power)
  dns.py        Hetzner DNS API wrapper (create_a_record/delete_record)
  ssh.py        paramiko connect (with retry) + run helpers
  checks.py     18 health checks + parallel runner
  ui.py         Rich tables + questionary prompts
templates/
  cloud-config.yaml.tmpl

▸ Running tests

With uv:

uv sync --extra dev
uv run pytest

With pip:

.venv/bin/pip install -e ".[dev]"
.venv/bin/pytest

Unit tests are hermetic (no network, no Hetzner, no SSH). The suite runs in ~30s on a laptop.

Testing strategy

Two layers, by design:

• Unit tests (pytest) cover the parts that fit in a function: the cloud-config template renderer, the sqlite state DB (including the lazy column-upgrade for older state files), and the pure-logic check helpers (parse_cloud_init_status, parse_bootstrap_log, the status.json-derived checks, the registry tripwires). These run in milliseconds, gate every commit, and don't need credentials or network.

• Live integration (dev test against a real VM) is how the SSH-dependent half of the codebase is exercised: paramiko connect, the SSH-based checks, the actual Caddy + cert flow, the dev create → DNS → bootstrap → dev destroy lifecycle. These are not pytest because (a) they cost real money each run and (b) they're inherently slow (~10–15 min for one VM to bootstrap). Run them after non-trivial changes to checks.py, hetzner.py, dns.py, ssh.py, cli.py, or the cloud-config template.

The hcloud and Hetzner DNS SDKs are not mocked anywhere — calls go to Hetzner directly. This was a conscious choice: a mocked Hetzner that returns "server created, status=running" would have masked every bug found during live verification (cloud-init's defer:true write_files ordering, Ubuntu 24.04's needrestart whiptail prompt, NodeSource vs Ubuntu npm conflict on ARM64, paramiko's stdin-channel hang in pi). Trust the SDK; verify the integration on a real VM.

What live verification has caught that unit tests didn't

Bug	Where	Fix
write_files with owner: human:human failed because cloud-init runs write_files before users-groups	templates/cloud-config.yaml.tmpl	defer: true on every human-owned entry
Ubuntu 24.04's needrestart showed an interactive whiptail kernel-upgrade prompt mid-apt install, killing bootstrap	bootstrap.sh in template	export NEEDRESTART_MODE=a + DEBIAN_FRONTEND=noninteractive at the top
NodeSource's setup script silently no-op'd on ARM64; falling back to Ubuntu's nodejs package broke because Ubuntu's npm depends on Ubuntu's node-* packages that conflict with NodeSource's nodejs when the repo IS configured	bootstrap.sh in template	If nodesource.* apt source exists, install only nodejs (NodeSource bundles npm); otherwise install nodejs npm
pi installer dropped the binary in ~/.npm-global/bin/pi, but bash -lc doesn't pick up env.sh because Ubuntu's ~/.bashrc returns early on non-interactive shells, so PATH didn't include ~/.npm-global/bin	checks.py:_shell() and /etc/profile.d/dev-env.sh from cloud-init	_shell() explicitly sources env.sh; /etc/profile.d/dev-env.sh lands during cloud-init's early write_files so PATH is correct from the very first SSH login
pi -p '...' via paramiko hung 45s waiting for stdin (paramiko's exec_command leaves the channel open), but the same call via real ssh returned in 4s	checks.py:_shell()	Append </dev/null to every wrapped command

Test files

• tests/test_template.py — cloud-config rendering: every EXPECTED_KEYS value substitutes; apt's $${distro_id} preserved; bash $HOME / ${GITHUB_TOKEN:-} preserved; Caddyfile block renders with the actual DEV_PORT; missing-key error names every absent key; none of the _REMOVED_TERMS substrings appear in the rendered output (tripwire against accidental re-introduction of dependencies that were migrated out).

• tests/test_state.py — state DB: full CRUD, partial updates, both UNIQUE constraints, test-run record/fetch, recency order, cascade delete, fqdn + dns_record_id round-trip, lazy column upgrade for legacy DBs (a state.db created before the DNS migration is opened, the new columns get added, existing rows survive).

• tests/test_checks_unit.py — check registry + pure-logic helpers: registered set matches the current spec (tripwire); none of the REMOVED_TERMS appear in any check name (tripwire); parse_cloud_init_status, parse_bootstrap_log; every status.json-derived check (bootstrap, services, tools, repo, status_fresh) covered for both pass and fail paths; https_reachable and dns_resolves exercised over their pure context (no sockets opened).

Adding a new check

When adding a new entry to ALL_CHECKS in checks.py:

1. Decide which source the check reads from: SSH (ctx.client), status.json (ctx.status_json), or HTTPS/DNS (ctx.https_status, ctx.fqdn). Most new checks should be status-derived — add a field to dev-status-gen.sh and read it here.
2. Write the check function (returns CheckResult). If the check has parsing logic, factor the parser into a top-level helper so it's unit-testable.
3. Add unit tests covering the pass case, the obvious fail case, and the "no status.json" / "no SSH" path.
4. Add the new name to test_registered_checks_match_current_spec's expected set — this assertion is your tripwire against accidental removal or rename.
5. Run dev test <name> against a real VM to validate the SSH or network side.

Removing a dependency

When a tool is migrated out of the cloud-config or check suite (e.g. swapping a tunneling provider), add its name to REMOVED_TERMS in tests/test_checks_unit.py and _REMOVED_TERMS in tests/test_template.py. The tripwires there will fail any future change that re-introduces the term in a check name or in the rendered cloud-config — the cheapest way to catch a half-revert.

▸ Recovery and troubleshooting

The admin VM crashed mid-workshop

Claimed VMs keep working: participants are connected via SSH direct to VM IPs, not through the admin server. New claims fail until the admin comes back. The CLI is the failover for everything except the claim flow.

dev list --workshop <slug>          # see the pool
dev test all --workshop <slug>      # health-check
dev ssh dh-04                       # SSH directly

dev admin destroy                   # cleans DNS + VM (idempotent)
dev admin deploy --workshop <slug>  # redeploy

Cookie secret rotates on redeploy; participants log in fresh. VM names and claims survive because they live on Hetzner labels.

Recovery patterns for participants

Scenario	Recovery
Cookie and bookmark intact	Open /server/<vm> directly. Works.
Cookie expired, VM name remembered	Type https://<base>/server/<vm>. Server redirects to /?next=/server/<vm>. Log in. Cookie set. Land back on the workspace page.
Cookie intact, forgot VM name	Re-submit at /name. Get a new VM from the pool over-warm. Old claim orphans on the prior VM until teardown.
Forgot VM name, had real work on the old one	Operator filters the dashboard by participant's typed name, reads the old FQDN out-of-band, hands it to the participant. Participant SSHs directly.

dev admin deploy healthz times out

The VM is usually fine; it's a DNS resolution problem on your laptop. Verify:

dig +short @1.1.1.1 <base-domain>
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

If 1.1.1.1 has the record but your local resolver doesn't, you're on a VPN with a stale negative cache. Bypass with --resolve:

curl --resolve <base-domain>:443:<ip> https://<base-domain>/healthz

TLS handshake works but SAN doesn't match

The wildcard cert on disk is for a different BASE_DOMAIN. Re-issue and push:

dev cert renew --force
dev cert show              # confirms SANs match BASE_DOMAIN
dev sync-certs all         # pushes to every running VM including admin

A pool VM is wedged

dev test <name>            # confirms the failure
dev destroy <name>         # removes DNS + VM + state row
dev create --count 1 --workshop <slug> --name-prefix <prefix>

Pool over-warm absorbs the gap until the replacement boots.

Participants can't log in

1. Check the workshop password against /admin/settings. If you rotated mid-workshop and forgot, set a new one and tell the room.
2. Check that <base>/admin responds. If it doesn't, the admin VM is down — see above.
3. Check Caddy logs on the admin VM: dev admin logs.

error during placement (resource_unavailable, …) on dev create

By default, dev create will automatically fall back when Hetzner reports the requested (server_type, location) is at capacity. The search order is per-location size ladder: it walks up the size ladder in the requested location first, then moves to the next location and walks the ladder again there. Within a family the ladder is cx23 → cx33 → cx43 → cx53 (and similarly cax11 → cax21 → cax31 → cax41, cpx11 → cpx21 → …).

The ladder never crosses CPU vendor or architecture. A cax11 request stays on ARM; a cx23 request stays on Intel — silent vendor swaps are never performed.

Each VM in a batch falls back independently, so one batch can land on a mix of sizes. The actually-provisioned (server_type, location) is recorded in state.db and shown in dev show. Each fallback decision is logged to ~/.dev-orchestrator/logs/<date>.log, and a yellow notice prints to the terminal when a VM ends up on a different size or location than requested.

The ladder also auto-skips any size Hetzner has marked deprecated past its unavailable_after date — the orchestrator queries /v1/server_types once per dev create and filters the live list rather than relying on hardcoded defaults.

If you'd rather fail fast on capacity issues (and pick the alternative yourself), pass --no-fallback:

dev create --count 5 --no-fallback

cert_valid fails with TLSV1_ALERT_INTERNAL_ERROR

Caddy is alerting on the TLS handshake — almost always because the static wildcard cert isn't on disk where it expects, or its key is unreadable by the caddy user. Check on the VM: sudo ls -la /etc/caddy/wildcard.{crt,key} (key should be 0640, caddy group). If the files are missing, either the cloud-config render got an empty cert (run dev cert show locally — if no cert, run dev cert init) or the b64 decoding failed at boot (sudo journalctl -u cloud-final --no-pager | grep -i wildcard).

https_reachable returns http=502

Caddy is up and got a cert (otherwise this would be a TLS error), but nothing is listening on DEV_PORT to reverse-proxy to. That's expected for an idle VM — /status is served from disk regardless. Per spec, https_reachable checks /status.json specifically (which Caddy serves from disk), so a 502 here means Caddy itself is in a bad state, not that the dev server is missing.

dns_resolves fails with gaierror

Either the DNS record was never created (check dev show <name> — dns_record should not be -), or your local resolver is caching a stale negative response. Try dscacheutil -flushcache on macOS or resolvectl flush-caches on Linux.

pi_calls_openrouter returns the wrong text

The check expects ok (case-insensitive) in the response. Some models preface with formatting; if you change DEFAULT_MODEL to a model with heavy preamble behavior, the check will be flaky. The default anthropic/claude-sonnet-4.6 is well-behaved for this prompt.

Bootstrap takes longer than 15 minutes

SSH in (dev ssh <name>) and check sudo cloud-init status --long. If package_upgrade is still running, just wait. If it's stuck on a needrestart prompt, the cloud-config sets NEEDRESTART_MODE=a to avoid this — but only for bootstrap.sh, not for cloud-init's own package upgrade. On rare images you may need to reboot the VM (dev power off <name>; dev power on <name>) and re-run dev sync.

Private boilerplate repo

Set GITHUB_TOKEN during dev init. The cloud-config exports it into the VM's env.sh and uses it for the https://${GITHUB_TOKEN}@… clone. The token also lets you gh from inside the VM.

A dev destroy left a DNS record behind

Shouldn't happen — destroy deletes DNS first, then VM, then state. But if it did (e.g. you killed the process mid-destroy): the record is in your Hetzner DNS zone, you can delete it from the dashboard, or re-dev create with the same name (which will collide on DNS create and surface the existing record_id).

▸ Command reference

Command	Options / arguments	Purpose
dev init		First-run wizard. Writes .env, validates token, picks DNS zone.
dev cert init	--staging	Obtain the wildcard cert via DNS-01.
dev cert renew	--force	Renew if expiring within 30 days (--force renews unconditionally).
dev cert show		Print expiry date and SAN list.
dev create	--count N --name-prefix PREFIX --server-type TYPE --location LOC --image IMAGE --model MODEL --repo REPO --workshop SLUG --no-prompt --no-fallback	Provision N servers. Interactive defaults table unless --no-prompt or flags are set.
dev list	--workshop SLUG	Pool overview: status badge, IPv4, last-test verdict.
dev show NAME		Full detail for one server: all fields + per-check breakdown.
dev test NAME|all	--workshop SLUG	Run the 18-check health suite. Exits non-zero on any failure.
dev snapshot NAME	--description DESC	Snapshot a server into a reusable image. Prints the image id to paste into DEFAULT_IMAGE.
dev sync NAME|all		git pull + scripts/bootstrap.sh on each server (iterate on boilerplate without rebuilding).
dev sync-certs NAME|all		Push current wildcard cert to one (or all) VMs and reload Caddy.
dev power on NAME|all		Power on server(s).
dev power off NAME|all		Power off server(s). all requires typed-count confirmation.
dev destroy NAME|all	--workshop SLUG	Delete DNS record, then VM, then state row. Requires typed confirmation.
dev ssh NAME		Exec into a server as the configured user.
dev logs NAME	--follow / -f	Stream /var/log/dev-bootstrap.log.
dev admin deploy	--workshop SLUG	Provision the admin VM at the BASE_DOMAIN apex.
dev admin update		Rsync local source onto the admin VM and restart the unit.
dev admin logs	--follow / -f	Stream journalctl -u orchestrator-admin.
dev admin destroy		Tear down the admin VM and its DNS record.

▸ Out of scope

Per the original PRD: no YAML config (.env is enough), no plugin system for checks, no abstraction over hcloud, no asyncio, no TUI, no auto-update of pi/claude/Caddy versions.