Bind deployed executors before requiring client certificates

[vincent10400094]

Problem

Turning on dispatcher_require_client_cert disconnected every executor deployed with generate-certs.sh + deploy-certs.yml (executor ID is not enrolled, then control session ownership mismatch). That caused a ~7 minute production outage, which was fixed by inserting binding rows by hand. A reissued client certificate would break the same way again. deploy/README.md suggested that giving executors certificates was enough.

Cause

When client certificates are required, cmd/dispatcher/main.go enables EnforceEnrollment. enrollment.Store.Admit/Bound then admits only executor IDs that have an executor_enrollments row binding them to the SHA-256 of their leaf certificate. Nothing in the deployment created those rows. The executor role also had no way to pass an enrollment token.

Fix

• debuglet-dispatcher -bind-executor ID -bind-certificate PATH (new enrollment.CertificateFingerprint and Store.Bind, which reuses SetExecutorEnrollment):
  • checks the PEM leaf chains to tls.ca_file with clientAuth EKU, is not a CA, and has CN == executor ID (that is what generate-certs.sh and the enrollment signer issue)
  • refuses non-certificate PEM blocks such as private keys
  • records the lowercase hex SHA-256 of the leaf DER, the same value as chainFingerprint
  • is idempotent ("already bound"), only replaces that ID's own binding (it prints the old fingerprint), and leaves outstanding tokens alone
  • cannot be combined with other admin flags or with -check-database/-upgrade-database/-init-database
• Ansible (tasks/bind-executors.yml, when dispatcher_require_client_cert and the new dispatcher_bind_inventory_executors (default true) are on):
  • copies each inventory executor's public client.crt (never a key) to {{ config_dir }}/dispatcher/executors/<id>.crt
  • runs the installed dispatcher with -bind-executor as the service user. It uses runuser the same way upgrade-database.yml does, and runs the command directly when the play already runs as that user.
  • Runs in:
    • the dispatcher role (site.yml): after rendering config and DB, before start/restart handlers
    • update-config.yml: before restarting with the new config
    • deploy-certs.yml: on a dispatcher that already enforces, before the executors install their certs, so a reissued cert gets re-bound. It reports and skips when the dispatcher isn't installed yet or doesn't enforce yet.
• executor_enrollment_token: per-host secret, rendered as [credentials] enrollment_token only when set (the template task is already no_log). Executors that have a token are not bound from a certificate.
• Preflight: refuses enforcement while any inventory executor has neither a certificate to bind nor a token. The check also runs on --limit dispatcher. With binding disabled, every executor needs a token. Token format is checked under no_log.
• Docs:
  • new "Requiring client certificates" section in deploy/README.md
  • corrected "Self-service executor enrollment" and dispatcher_require_client_cert text
  • docs/operations/remote-deployment.md and executor-onboarding.md: enforcement binds IDs to fingerprints, the deployment binds inventory executors automatically, and reissuing a cert re-binds
• CHANGELOG entries under Added and Fixed referencing Requiring client certificates disconnects every deployed executor: no enrollment binding, no Ansible path, docs wrong #415.

Rollout notes for an existing fleet

1. Deploy this release with site.yml first. deploy-certs.yml/update-config.yml bind with the installed dispatcher binary, and older binaries don't have -bind-executor.
2. In production the bindings were inserted by hand. If they match the deployed certificates, the first run prints "already bound" and changes nothing. If they don't match, the run replaces them with the deployed certificates.
3. To enable enforcement: make sure every executor has its cert (make deploy-certs), set dispatcher_require_client_cert: true, then run update-config.yml. It binds before it restarts the dispatcher. Check the "Report the executor bindings" output.
4. Reissuing a cert: delete certs/executors/<id>/ and run make deploy-certs. The dispatcher re-binds before the executor installs and restarts with the new cert. The old cert is refused from the moment it is re-bound, so that executor is offline until its own restart later in the same run.
5. Removing an executor from the inventory does not revoke its binding. Use -revoke-executor.

Tests

• go build, go vet, gofmt for cmd/dispatcher and internal/dispatcher/enrollment. go test ./cmd/dispatcher ./internal/dispatcher/... ./internal/executor/config pass.
  • New: internal/dispatcher/enrollment/bind_test.go covers bind, idempotence, replace, other IDs untouched, tokens untouched, malformed input, and fingerprint verification (foreign CA, server-only EKU, CN mismatch, expired, CA leaf, private key in file, garbage, chain file).
  • New: cmd/dispatcher/bind_executor_test.go runs the command end to end against a fresh database (bound, already bound, reissued replaces, refusals record nothing) and checks the flag pairing.
  • The -init-database exclusivity test now includes the new flags.
• deploy/test/ansible-render.sh through deploy/test/provisioner-check.sh (release built with ./deploy/scripts/build-linux.sh on a clean commit): all checks pass. New checks cover:
  • the preflight refusals and acceptance with a token
  • token rendering, and that the installed executor accepts it
  • the token not appearing in logs
  • on a separate fixture tree with a real dispatcher database:
    • deploy-certs.yml defers binding before install
    • deploy-dispatcher.yml binds the inventory executor (fingerprint checked with openssl)
    • only the public cert reaches the dispatcher
    • update-config.yml reports "already bound"
    • a reissued cert gets re-bound by deploy-certs.yml
    • a foreign-CA cert is refused

Not exercised: a real systemd host, the runuser path as root, and a live executor reconnecting after a bind. The fixture runs as the service user without systemd.

Fixes #415

🤖 Generated with Claude Code