M7: security hardening (cross-cutting) - #75
Open
MJohnson459 wants to merge 1 commit into
Open
Conversation
Through M3 the fleet had one credential on one path — an operator token on
dispatch — and everything else leaned on the tailnet. The argument for changing
that is not that WireGuard is weak but that a single boundary makes every robot
as trusted as the fleet server: a robot is physically reachable and carries the
largest dependency tree in the system, so one compromised robot would own the
fleet. M7 does not replace the tailnet; it stops the tailnet being the only
thing in the way.
The broker is no longer anonymous. credentials.py generates mosquitto's
password_file and acl_file from the registry into $MOTE_FLEET_HOME/broker, and
the broker re-reads them on SIGHUP (broker.sh reload, which knows whether it
started a local process or a container) — so who may connect is derived, never
hand-maintained, and a revoked operator disappears because the query no longer
returns them. Three principals, disjoint by construction: a robot publishes
only under its own robot_id and reads only its own task/command; an operator
subscribes fleet-wide and publishes nothing; the server alone may write
task/command. A robot_id is a lowercase DNS label, so it can never contain the
underscore the other two carry — no runtime collision check, and a test asserts
the property rather than the paragraph.
Robot credentials are issued inside the enrollment transaction and returned
once, so rotation is just running enroll again; only the hash is stored. Every
/v1 route now needs an operator token, checked by one gate in front of routing
so a new route is authenticated by default and an unauthenticated caller gets
401 rather than a 404 that would disclose which routes exist. /healthz and the
static UI stay open, the latter because the page must load before it can ask
for a token. /v1/config hands the browser that operator's subscribe-only broker
credential, which is why it needs one — so "the browser cannot publish" is now
the broker's rule and not only our client's missing PUBLISH encoder.
Gotcha that shapes every ACL test: mosquitto denies silently. A forbidden
publish is accepted and dropped; a forbidden subscribe is granted at SUBACK and
never delivers. Denial is therefore asserted on delivery, since a suite
checking return codes would pass with no ACL loaded at all.
The network half is mote_bringup/tailscale/policy.hujson — committed source of
truth, pasted into the admin console — whose own tests block asserts robots
cannot reach each other; Tailscale refuses to save a policy that fails it.
Packages: provisioning now runs pixi install --locked, so a robot installs the
dependency set that was tested. Signing waits for M5, when there is a package
to sign.
Breaking change for a deployed fleet, by design: a robot enrolled before M7 has
no credential and the broker will refuse it. Re-run pixi run enroll; the
agent's log says so.
Also fixes a defect from M3 that this milestone made visible: `hidden` was not
hiding anything, because .dispatch { display: flex } outranks the UA's
[hidden] { display: none } — the dispatch form and the Foxglove link were on
screen whenever the code believed otherwise.
Contract and measurements: docs/fleet/security.md (the fleet's third versioned
contract, alongside M1's MQTT one and M3's HTTP one) and
docs/fleet/m7-verification.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M7 security hardening: the broker stops being anonymous and the fleet API's read routes stop being open.
Why. Through M3 the fleet had one credential on one path — an operator token on dispatch — and everything else leaned on the tailnet. The argument for changing that is not that WireGuard is weak, but that a single boundary makes every robot as trusted as the fleet server: a robot is physically reachable and carries the largest dependency tree in the system, so one compromised robot would own the fleet. M7 does not replace the tailnet; it stops it being the only thing in the way.
What changed.
mote_fleet/server/credentials.pyrenders mosquitto'spassword_fileandacl_fileinto$MOTE_FLEET_HOME/broker/, and the broker re-reads them on SIGHUP (broker.sh reload, which records whether it started a local process or a container). Who may connect is therefore derived, never hand-maintained — a revoked operator disappears because the query no longer returns them. Three principals, disjoint by construction: a robot (username = itsrobot_id) publishes only under its own prefix and reads only its owntask/command; an operator (op_<slug>_<hex>) subscribes fleet-wide and publishes nothing;fleet_serveralone may writetask/command. Arobot_idis a lowercase DNS label so it can never contain the underscore the other two carry — no runtime collision check needed.enrollagain.$MOTE_HOME/fleet.yamland the registry are both0600./v1route needs an operator token, checked by one gate in front of routing — so a route added later is authenticated by default, and an unauthenticated caller gets 401 rather than a 404 that would disclose which routes exist./healthzand the static UI stay open, the latter because the page must load before it can ask for a token./v1/confighands the browser that operator's subscribe-only broker credential, which is why it needs one: "the browser cannot publish" is now the broker's rule, not only our client's missing PUBLISH encoder.mote_bringup/tailscale/policy.hujson(source of truth; pasted into the admin console), whose owntestsblock asserts robots cannot reach each other — Tailscale refuses to save a policy that fails it.pixi install --locked, so a robot installs the dependency set that was tested rather than silently re-solving. Signing waits for M5, when there is a package to sign.hiddenwas not hiding anything in the dashboard since M3 —.dispatch { display: flex }outranks the UA's[hidden] { display: none }, so the dispatch form and Foxglove link were on screen whenever the code believed otherwise.How it was verified.
pixi run test), pluspixi run -e dev test-fleetat 212/212 and lint clean.test_broker_acl.py) covering the milestone's criterion directly: anonymous refused, robot A cannot read robot B's commands or forge its health, operator cannot publish or read commands, server can dispatch to any robot.test_e2e_fleet.pynow runs against an authenticated, ACL-enforcing broker started with empty credential files, so the enroll → issue → SIGHUP → agent connects → dispatch → status chain only completes if every link works.disconnected: not authorised) and 401'd its next HTTP call — one act closing both paths.$7$hash verified against a hash written by the realmosquitto_passwd, so the reimplementation cannot drift from the broker.Key gotcha, recorded because it shapes every ACL test: mosquitto denies silently. A forbidden publish is accepted at the socket and dropped; a forbidden subscribe is granted at SUBACK and never delivers. Denial is therefore asserted on delivery — a suite checking return codes would pass with no ACL file loaded at all.
Breaking change for a deployed fleet, by design: a robot enrolled before M7 has no credential and the broker will refuse it.
pixi run enrollis the fix and the agent's log says so.Deliberately not done, each with reasons in
docs/fleet/security.md: mTLS (username/password inside WireGuard is proportionate, and the seam is unchanged), expiring operator sessions (Q7's OIDC answer replaces the minting, not the checking), package signing (nothing to sign until M5), Foxglove tokens (M2 does not exist yet — but the tailnet rule already scopes port 8765).Left open, filed as task #264: the tailnet policy is committed but not yet applied in the admin console; no robot has been re-enrolled against an M7 server; and
pixi run fleet-broker-wsitself was not exercised (a container broker was verified, but started by hand on non-default ports because the workstation already had one on 1883/9001).Docs:
docs/fleet/security.md(the fleet's third versioned contract, alongside M1's MQTT one and M3's HTTP one) anddocs/fleet/m7-verification.md, withcontrol-plane.md,fleet-api.md,README.md,CLAUDE.mdand the design doc's M7 milestone updated.Committed as
8b2e03bon branchfleet-m7-security(worktree). Not pushed and no PR opened — remote operations are yours.