diff --git a/CLAUDE.md b/CLAUDE.md index 40d2774..9ccbfe6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,7 @@ Milestone M2 of `docs/design/fleet.md`; the operator flow is `docs/fleet/README. ## Fleet: the operator view + dispatch API (M3) -Milestone M3 of `docs/design/fleet.md`, and the end of v0. The **HTTP** wire is specified as its own versioned contract in **`docs/fleet/fleet-api.md`** (M1's MQTT one is `control-plane.md`); the operator flow is `docs/fleet/README.md` §6–9 and the measurements are `m3-verification.md`. **The two directions of the loop take different paths on purpose.** *Reads* ride MQTT: the browser subscribes to `mote/v1/+/{presence,health,pose,task/status}` over WebSockets, and because all of those are retained it has the whole fleet's state within a second of loading — no polling, no service in the middle. *Writes* ride HTTP: `POST /v1/robots//dispatch` authorizes an operator token (`fleetctl operator new --name `; the name is what the audit row records), writes the audit row, then publishes to the same `task/command` topic. **The topic tree did not change — only who publishes to it**, and `fleetctl dispatch` moved to the API too, so there is one write path rather than one per client. The command grammar is still parsed only by the robot's task layer: a parser in the server would be a second grammar to keep in step. **The browser cannot publish**: `server/ui/mqtt.mjs` is a hand-rolled subscribe-only MQTT 3.1.1 client that implements no PUBLISH packet, so the split is enforced by omission (M7 makes it structural with a subscribe-only broker credential). The UI is static ES modules — no bundler, no npm, no vendored library — served by the same stdlib `http.server`; `map.mjs` holds the Q5 world→pixel transform (`px = (wx-origin_x)/res`, `py = height - (wy-origin_y)/res`) and a pan/zoom/follow canvas, and only draws robots on the *same* site+floor as the selected one because a pose from another floor is a different map frame. **Basemaps come from site bundles on the fleet box** (`--maps-dir`, default `$MOTE_FLEET_HOME/sites`, the layout `sites.py` writes, seeded by rsync until **M4** makes the registry canonical behind the same two routes). **M1's websockets blocker is settled**: `pixi run fleet-broker` runs `eclipse-mosquitto` under docker with the repo's own `mosquitto.conf`, because conda-forge's build has none; `pixi run -e fleet fleet-broker-local` is the conda binary for a box without docker, and it strips the WS stanza and says so. Two things that run in the same file (`test_ui.py` → `ui_test.mjs`) are the MQTT codec and the transform, tested under node against the very files the browser loads; `browser_check.mjs` drives a real headless Chrome over CDP against a running stack and is an operator's tool, not a CI test. **The phone is the realistic off-LAN client**, so below 760 px the three panes become one at a time behind a bottom tab bar (`server/ui/layout.mjs`), selecting a robot in the roster navigates to the map — what the desktop layout gets for free by showing both — and the canvas gained pinch-to-zoom (`pinchSpan`/`pinchUpdate` in `map.mjs`, pure and tested, because a division by a zero span puts NaN in the view scale and blanks the map for good) plus a fingertip-sized hit target. The breakpoint is a **silent** seam — CSS decides what is displayed, JS decides when a selection navigates, and disagreement yields a tab bar over stacked panes rather than an error — so it lives in `layout.mjs` and `ui_test.mjs` reads the stylesheet and holds it there, as it does for every pane having a tab and for `touch-action: none` on the canvas (without which the browser eats the drag and the pinch before a single pointer event arrives). Dispatch gained a **zone picker that writes `goto ` into the command box rather than sending it**: the grammar stays the robot's, and the keyboard leaves the common case. Three pre-existing bugs fell out, all of which a desk hides: `hidden` does not hide an element whose class sets `display` (the empty promote picker), the canvas backing store was resized on width alone so a height change left the previous frame's scale bar under the new one, and the scale bar was drawn in the dark theme's near-white on a white basemap — a canvas gets no cascade, so it now reads `--dim` off the element. Measurements, including `browser_check.mjs`'s phone pass, are `m3-verification.md` §9; **a real device is still the acceptance** — emulation gets the viewport and the touch points right and the thumb wrong. +Milestone M3 of `docs/design/fleet.md`, and the end of v0. The **HTTP** wire is specified as its own versioned contract in **`docs/fleet/fleet-api.md`** (M1's MQTT one is `control-plane.md`); the operator flow is `docs/fleet/README.md` §6–9 and the measurements are `m3-verification.md`. **The two directions of the loop take different paths on purpose.** *Reads* ride MQTT: the browser subscribes to `mote/v1/+/{presence,health,pose,task/status}` over WebSockets, and because all of those are retained it has the whole fleet's state within a second of loading — no polling, no service in the middle. *Writes* ride HTTP: `POST /v1/robots//dispatch` authorizes an operator token (`fleetctl operator new --name `; the name is what the audit row records), writes the audit row, then publishes to the same `task/command` topic. **The topic tree did not change — only who publishes to it**, and `fleetctl dispatch` moved to the API too, so there is one write path rather than one per client. The command grammar is still parsed only by the robot's task layer: a parser in the server would be a second grammar to keep in step. **The browser cannot publish**: `server/ui/mqtt.mjs` is a hand-rolled subscribe-only MQTT 3.1.1 client that implements no PUBLISH packet, so the split is enforced by omission (M7 makes it structural with a subscribe-only broker credential). The UI is static ES modules — no bundler, no npm, no vendored library — served by the same stdlib `http.server`; `map.mjs` holds the Q5 world→pixel transform (`px = (wx-origin_x)/res`, `py = height - (wy-origin_y)/res`) and a pan/zoom/follow canvas, and only draws robots on the *same* site+floor as the selected one because a pose from another floor is a different map frame. **Basemaps come from site bundles on the fleet box** (`--maps-dir`, default `$MOTE_FLEET_HOME/sites`, the layout `sites.py` writes, seeded by rsync until **M4** makes the registry canonical behind the same two routes). **M1's websockets blocker is settled**: `pixi run fleet-broker` runs `eclipse-mosquitto` under docker with the repo's own `mosquitto.conf`, because conda-forge's build has none; `pixi run -e fleet fleet-broker-local` is the conda binary for a box without docker, and it strips the WS stanza and says so. Two things that run in the same file (`test_ui.py` → `ui_test.mjs`) are the MQTT codec and the transform, tested under node against the very files the browser loads; `browser_check.mjs` drives a real headless Chrome over CDP against a running stack and is an operator's tool, not a CI test — `pixi run fleet-ui-check` is that stack in one command (broker on ephemeral ports, server, a temp `MOTE_FLEET_HOME`, the sim's `office_world` bundle as the basemap, and `test/fake_robots.py`, which publishes `protocol.py` payloads and answers `task/command` and is *not* a second robot implementation), torn down afterwards; `-- --keep` leaves it up for UI work. It stays out of CI because it needs docker (conda's mosquitto still has no websockets) *and* a chrome, which the arm runner has not — the decision, and what wiring it in would take, are recorded in `m3-verification.md` §2 rather than left looking like coverage. **The phone is the realistic off-LAN client**, so below 760 px the three panes become one at a time behind a bottom tab bar (`server/ui/layout.mjs`), selecting a robot in the roster navigates to the map — what the desktop layout gets for free by showing both — and the canvas gained pinch-to-zoom (`pinchSpan`/`pinchUpdate` in `map.mjs`, pure and tested, because a division by a zero span puts NaN in the view scale and blanks the map for good) plus a fingertip-sized hit target. The breakpoint is a **silent** seam — CSS decides what is displayed, JS decides when a selection navigates, and disagreement yields a tab bar over stacked panes rather than an error — so it lives in `layout.mjs` and `ui_test.mjs` reads the stylesheet and holds it there, as it does for every pane having a tab and for `touch-action: none` on the canvas (without which the browser eats the drag and the pinch before a single pointer event arrives). Dispatch gained a **zone picker that writes `goto ` into the command box rather than sending it**: the grammar stays the robot's, and the keyboard leaves the common case. Three pre-existing bugs fell out, all of which a desk hides: `hidden` does not hide an element whose class sets `display` (the empty promote picker), the canvas backing store was resized on width alone so a height change left the previous frame's scale bar under the new one, and the scale bar was drawn in the dark theme's near-white on a white basemap — a canvas gets no cascade, so it now reads `--dim` off the element. Measurements, including `browser_check.mjs`'s phone pass, are `m3-verification.md` §9; **a real device is still the acceptance** — emulation gets the viewport and the touch points right and the thumb wrong. ## Fleet: the map registry (M4) diff --git a/docs/fleet/README.md b/docs/fleet/README.md index 5818a65..69987ac 100644 --- a/docs/fleet/README.md +++ b/docs/fleet/README.md @@ -691,6 +691,15 @@ a second command language for the fleet server to keep in step. Between 760 and 1100 px the panes stack and scroll, as before. +**Working on the page itself?** `pixi run fleet-ui-check` builds a throwaway +fleet to point it at — a broker, a fleet server, a basemap and three robots that +exist only on the wire — runs the browser checks against it, and tears it all +down; `pixi run fleet-ui-check -- --keep` leaves it up and prints the URL and an +operator token instead. It uses ports and a state directory of its own, so it +runs beside the fleet you actually operate. Needs a docker and a chrome +([`m3-verification.md`](m3-verification.md) §2). The checks include the phone +layout above, so the emulated pass is one command too. + **What it does not do**, deliberately: no marker clustering, no basemap tiling, no 3D, no camera, no teleop. The first two are what `fleet.md` Q5 describes for large sites and would be unmeasured complexity at this fleet size; the last diff --git a/docs/fleet/m3-verification.md b/docs/fleet/m3-verification.md index 5ebf20f..230d5ee 100644 --- a/docs/fleet/m3-verification.md +++ b/docs/fleet/m3-verification.md @@ -48,38 +48,115 @@ instead, and serves robots and `fleetctl` unchanged. (Ms made the container the default and folded the two tasks into one: that fallback is now `pixi run -e fleet fleet-broker-local`.) -## 2. The operator view, in a real browser — **9/9 checks, off the ROS graph** +## 2. The operator view, in a real browser — **15/15 checks, off the ROS graph** -The dashboard was driven by headless Chrome over the DevTools protocol against a -live stack: the container broker, `fleet-server` (from the ROS-free `fleet` -environment), two enrolled robots, and an operator token. +The dashboard is driven by headless Chrome over the DevTools protocol against a +live stack: the container broker, `fleet-server`, a site bundle to draw on, +enrolled robots, and an operator token. **One command builds all of that, runs +the checks and takes it down again:** ```console -$ node mote_fleet/test/browser_check.mjs http://127.0.0.1:8088 +$ pixi run fleet-ui-check +broker: eclipse-mosquitto:2.1-alpine on 52619 (mqtt) / 46001 (ws) +server: http://127.0.0.1:54489 (state in /tmp/mote-ui-check-qhaatfpk) +robots: mote-01, mote-02, mote-03 on office_world/ground + mote-03 is offline — the broker published its will + ok the browser connected to the broker over WebSockets — broker connected -ok the roster came from retained MQTT state — mote-01,mote-02 -ok health states are rendered — ok,degraded +ok the roster came from retained MQTT state — mote-01,mote-02,mote-03 +ok health states are rendered — ok,degraded,offline ok a basemap was resolved for the selected robot — office_world/ground ok the map canvas has pixels on it — 467726 painted pixels ok the health roll-up lists subsystems — 4 rows -ok dispatch went through the fleet API — dispatched f6f07ef1809a4f18 -ok the robot answered on task/status — accepted,dispatched,rejected +ok dispatch went through the fleet API — dispatched c751a9e20c304c1e +ok the robot answered on task/status — succeeded,accepted,dispatched +screenshot: fleet-ui.png + +ok a coarse pointer is what the page thinks it has +ok one pane at a time, with a tab bar to move between them — {"tabs":"flex","shown":1} +ok no pane scrolls sideways on a phone +ok the canvas backing store follows the pane it is in — 1170x1674 for 1170x1674 +ok picking a robot in the roster shows it on the map — map +ok two fingers zoom the map +screenshot: fleet-ui-phone.png ok no uncaught page errors — 0 -9/9 checks passed +15/15 checks passed ``` ![The fleet dashboard](../images/fleet-ui.webp) -Nothing was polled: the roster, the health roll-up, both robot positions and the -task-status log are all retained MQTT state that arrived on the WebSocket within -a second of the page loading. Both colour schemes were rendered -(`Emulation.setEmulatedMedia`) and checked by eye. - -The robots behind it are the wire, not the hardware — a script publishing the -real `protocol.py` payloads and answering `task/command`. The **real** agent and -behaviour tree are covered by the end-to-end test in §4; what this run is for is -the half only a browser can answer. +That is the original M3 run's 9 assertions and §9's phone pass, in one command +against a stack it built itself — 6.2 s end to end. The painted-pixel count is +identical to the hand-assembled run's because it is the same basemap: the +harness serves +`mote_simulation/sim_home/sites/office_world`, the only real saved map +(`map.yaml` + PNG + zones) committed in the tree, so the world→pixel transform +and the zone overlay are exercised at a real scale rather than against a fixture +that agrees with them. + +Nothing is polled *by the page*: the roster, the health roll-up, all three robot +positions and the task-status log are retained MQTT state that arrives on the +WebSocket within a second of loading. Both colour schemes were rendered +(`Emulation.setEmulatedMedia`) and checked by eye during the original run; the +committed checks assert what a screenshot cannot, and leave the two themes to +the two screenshots they write. + +**The robots are the wire, not the hardware.** `mote_fleet/test/fake_robots.py` +publishes `protocol.py` payloads and answers `task/command`, and that is the +whole of it — not a second robot implementation but the contract itself, which +is exactly what the UI consumes. What it does model is what the UI renders +differently: an `ok` robot and a `degraded` one, a pose that moves, the task +transitions (`goto dropoff` → dispatched/accepted/succeeded, `wibble` → rejected +`unknown command 'wibble'`, `goto nowhere` → rejected `unknown zone 'nowhere'` — +the same shape as §3's real robot, measured through `fleetctl` against this +fixture), a redelivered command recognised rather than re-run, and one robot +that **drops its socket without a DISCONNECT** so the broker publishes its will. +The harness waits on that will rather than on a sleep: an offline row the +fixture published for itself would not be testing the Last Will path at all. The +**real** agent and behaviour tree are covered by the end-to-end test in §4; what +this run is for is the half only a browser can answer. + +Two things it deliberately does not touch. It never uses `~/.mote-fleet` — the +registry and the basemaps go in a temp directory that is deleted afterwards — +and it never uses 1883/9001, because the workstation this was measured on was +**already running a broker and a fleet server for a real robot on exactly those +ports**, and both were still serving when the run finished. Every process starts +in its own session, so teardown reaps this stack and nothing else (verified: +no container, no temp directory and no process left behind). Two runs at once on +different sites — `office_world` and `hospital_world` — both pass 15/15, sharing +no port, container or directory. + +The broker image is the compose file's pin, read at startup exactly as +`broker.sh` reads it: a tag of the harness's own would be a third broker able to +drift onto a mosquitto whose websockets support differs, which is the failure +`test_deploy_config.py` exists to prevent — so that test now watches this file +too. + +`-- --keep` skips the browser and leaves the stack up with its URL and operator +token printed, which is the loop for actually working on `server/ui/`. + +### Can it gate CI? — **no, and this is the decision rather than an oversight** + +It needs two things the `build` workflow's runners do not both have: + +- **docker**, because the browser's read path is MQTT-over-WebSockets and + conda-forge's mosquitto is still built without them (re-measured at 2.0.20: + `ldd` finds no libwebsockets). Hosted `ubuntu-latest` has docker, so this half + would be free there. +- **a chrome**. `ubuntu-latest` ships one; the matrix's other half, + `ubuntu-24.04-arm`, does not — so wiring it in buys one architecture's + coverage of a page that has no architecture-specific behaviour. + +The flakiness objection is at least answered: every assertion now polls to a +deadline instead of sleeping a guessed interval (`settle()` in +`browser_check.mjs`), so a loaded shared runner makes the job slower rather than +red. What is left is a judgement about a ~40 s job — image pull included — that +would gate every PR on a headless browser to protect the six files in +`server/ui/`. So it stays **a command an operator runs when touching `server/ui/`**, +one workflow step away from being a gate if that changes. Same shape as the sim +smoke test's answer (#51), and for one of the same reasons: hosted CI does not +have the machine the check needs. ## 3. Dispatch is mediated — **confirmed, including the refusals** @@ -361,7 +438,9 @@ seams that fail silently: the CSS/JS breakpoint, every pane having a tab, and `browser_check.mjs` grew a **phone pass** — 390x844 at device scale 3, mobile metrics and touch emulation on, driven with real `Input.dispatchTouchEvent` gestures rather than synthesised DOM events. Against a live stack (container -broker, fleet server, three scripted robots on the `office_world` bundle): +broker, fleet server, three scripted robots on the `office_world` bundle — since +this was measured, that stack is `pixi run fleet-ui-check` and those robots are +the committed `fake_robots.py`, so this pass is now the tail of §2's run): ``` ok a coarse pointer is what the page thinks it has diff --git a/mote_fleet/README.md b/mote_fleet/README.md index d825d4b..0ae2542 100644 --- a/mote_fleet/README.md +++ b/mote_fleet/README.md @@ -117,11 +117,19 @@ Four tiers, so the same files give full coverage wherever they run: single-in-flight rule and the full agent against an injected fake MQTT client, so CI covers it on both architectures without a broker; and the robot's map staging and symlink flip against a real fleet server, with no ROS at all. -- **browser** (`test_ui.py` → `ui_test.mjs`) — the MQTT packet codec and the - world→pixel transform under node, against the same `.mjs` files the browser - loads. Skips where there is no node. `browser_check.mjs` is the other half — - a real headless browser against a running stack, which needs more than CI has, - so it is an operator's tool rather than a test. +- **browser** (`test_ui.py` → `ui_test.mjs`, `test_fake_robots.py`) — the MQTT + packet codec and the world→pixel transform under node, against the same `.mjs` + files the browser loads (skips where there is no node); and the wire-only + robots the dashboard is checked against, held to `protocol.py` and to the task + layer's grammar, so the fixture can never become a second definition of the + wire. `browser_check.mjs` is the other half — + a real headless browser against a running stack, which needs a docker and a + chrome, so it is an operator's tool rather than a test (the reasoning, and + what wiring it into CI would take, are in `docs/fleet/m3-verification.md` §2). + It needs no stack of your own: **`pixi run fleet-ui-check`** builds one — + broker, server, basemap, and the wire-only robots of `fake_robots.py` — on + ports nobody else is using, runs the checks and tears it all down. `-- --keep` + leaves it up instead, which is the loop for working on `server/ui/`. - **end to end** (`test_e2e_fleet.py`, `test_e2e_map_registry.py`, `test_fleet_outage.py`) — a real mosquitto, the real fleet server, the `enroll` CLI, a real paho client, and diff --git a/mote_fleet/test/browser_check.mjs b/mote_fleet/test/browser_check.mjs index 19ef9e1..c3d21bc 100644 --- a/mote_fleet/test/browser_check.mjs +++ b/mote_fleet/test/browser_check.mjs @@ -7,25 +7,54 @@ // in CI; what only a browser can answer is whether the page actually connects // to the broker over WebSockets, draws the basemap, and dispatches. // +// `pixi run fleet-ui-check` builds that stack around this file — a broker, a +// server, a basemap and a fake fleet on ports nobody else is using — and is how +// to run these checks without one. Point it at a stack of your own with: +// // node mote_fleet/test/browser_check.mjs http://localhost:8080 [token] [out.png] // // It speaks the Chrome DevTools Protocol over node's built-in WebSocket, so it // needs no npm install: only a chrome/chromium on PATH. import { spawn } from 'node:child_process'; -import { mkdtempSync, writeFileSync } from 'node:fs'; +import { accessSync, constants, mkdtempSync, writeFileSync } from 'node:fs'; +import { createServer } from 'node:net'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; const url = process.argv[2] || 'http://localhost:8080'; const token = process.argv[3] || ''; const shot = process.argv[4] || 'fleet-ui.png'; + +function onPath(name) { + return (process.env.PATH || '').split(':').some((dir) => { + try { + accessSync(join(dir, name), constants.X_OK); + return true; + } catch { + return false; + } + }); +} + const CHROME = process.env.CHROME || - ['google-chrome', 'chromium', 'chromium-browser'].find(Boolean); + ['google-chrome', 'chromium', 'chromium-browser'].find(onPath) || + 'google-chrome'; const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); +// An ephemeral debugging port, so two of these can run at once — a fixed one +// makes a second run attach to the first run's browser. +const freePort = () => + new Promise((resolve) => { + const server = createServer(); + server.listen(0, '127.0.0.1', () => { + const { port } = server.address(); + server.close(() => resolve(port)); + }); + }); + async function devtools(port) { for (let attempt = 0; attempt < 50; attempt += 1) { try { @@ -81,10 +110,28 @@ function check(name, ok, detail = '') { console.log(`${ok ? 'ok ' : 'FAIL'} ${name}${detail ? ` — ${detail}` : ''}`); } +// Every assertion below is about something that arrives — a WebSocket +// handshake, a retained message, a basemap decode, a robot's reply — so each +// one polls to a deadline instead of sleeping a guessed interval. A fixed +// sleep is either longer than it needs to be or, on a loaded machine, a red +// result about code that is fine; that difference is what decides whether this +// could ever be a gate rather than an operator's tool. +const DEADLINE_MS = 20000; + +async function settle(session, expression, satisfied, timeout = DEADLINE_MS) { + const deadline = Date.now() + timeout; + for (;;) { + const value = await session.evaluate(expression); + if (satisfied(value) || Date.now() > deadline) return value; + await sleep(150); + } +} + const profile = mkdtempSync(join(tmpdir(), 'mote-ui-')); +const debugPort = await freePort(); const chrome = spawn(CHROME, [ '--headless=new', - '--remote-debugging-port=9333', + `--remote-debugging-port=${debugPort}`, `--user-data-dir=${profile}`, '--window-size=1600,900', '--no-first-run', @@ -97,7 +144,7 @@ chrome.on('error', (error) => { }); try { - const socket = new WebSocket(await devtools(9333)); + const socket = new WebSocket(await devtools(debugPort)); await new Promise((resolve) => socket.addEventListener('open', resolve)); const session = new Session(socket); await session.send('Runtime.enable'); @@ -106,64 +153,86 @@ try { if (token) { await session.send('Page.navigate', { url }); - await sleep(1000); + await settle(session, `!!document.getElementById('broker-state')`, (up) => up); await session.evaluate(`localStorage.setItem('mote.operator.token', '${token}')`); } await session.send('Page.navigate', { url }); - // Long enough for the config fetch, the WebSocket handshake, the retained - // messages and the basemap decode. - await sleep(4000); + const broker = await settle( + session, + `(document.getElementById('broker-state') || {}).className || ''`, + (className) => className.includes('connected'), + ); check( 'the browser connected to the broker over WebSockets', - (await session.evaluate(`document.getElementById('broker-state').className`)).includes( - 'connected', + broker.includes('connected'), + await session.evaluate( + `(document.getElementById('broker-state') || {}).textContent || 'no page'`, ), - await session.evaluate(`document.getElementById('broker-state').textContent`), ); - const roster = await session.evaluate( + const roster = await settle( + session, `[...document.querySelectorAll('.robot-id')].map(n => n.textContent).join(',')`, + (ids) => ids.includes('mote-01'), ); check('the roster came from retained MQTT state', roster.includes('mote-01'), roster); - const health = await session.evaluate( + const health = await settle( + session, `[...document.querySelectorAll('.robot-state')].map(n => n.textContent).join(',')`, + (states) => /ok|degraded|fault/.test(states), ); check('health states are rendered', /ok|degraded|fault/.test(health), health); - const mapLabel = await session.evaluate( + const mapLabel = await settle( + session, `document.getElementById('map-label').textContent`, + (label) => label.includes('/'), ); check('a basemap was resolved for the selected robot', mapLabel.includes('/'), mapLabel); - const drawn = await session.evaluate(`(() => { - const canvas = document.getElementById('map-canvas'); - const ctx = canvas.getContext('2d'); - const data = ctx.getImageData(0, 0, canvas.width, canvas.height).data; - let painted = 0; - for (let i = 3; i < data.length; i += 4) if (data[i] > 0) painted += 1; - return painted; - })()`); + const drawn = await settle( + session, + `(() => { + const canvas = document.getElementById('map-canvas'); + const ctx = canvas.getContext('2d'); + const data = ctx.getImageData(0, 0, canvas.width, canvas.height).data; + let painted = 0; + for (let i = 3; i < data.length; i += 4) if (data[i] > 0) painted += 1; + return painted; + })()`, + (painted) => painted > 10000, + ); check('the map canvas has pixels on it', drawn > 10000, `${drawn} painted pixels`); - const subsystems = await session.evaluate( + const subsystems = await settle( + session, `document.querySelectorAll('#subsystems .subsystem').length`, + (rows) => rows > 0, ); check('the health roll-up lists subsystems', subsystems > 0, `${subsystems} rows`); if (token) { - const dispatched = await session.evaluate(`(async () => { + await session.evaluate(`(() => { document.getElementById('command').value = 'goto dropoff'; document.getElementById('dispatch').requestSubmit(); - await new Promise(r => setTimeout(r, 1500)); - return document.getElementById('dispatch-note').textContent; })()`); + const dispatched = await settle( + session, + `document.getElementById('dispatch-note').textContent`, + (note) => note.startsWith('dispatched'), + ); check('dispatch went through the fleet API', dispatched.startsWith('dispatched'), dispatched); - await sleep(2500); - const statuses = await session.evaluate( + const statuses = await settle( + session, `[...document.querySelectorAll('#status-log .status-state')].map(n => n.textContent).join(',')`, + (states) => /succeeded|failed|rejected/.test(states), + // The assertion is `accepted`; this shorter deadline only buys the rest + // of the lifecycle when it is cheap (a fake robot's task takes seconds). + // A real robot's goto takes minutes and must not hold the run open. + 5000, ); check('the robot answered on task/status', statuses.includes('accepted'), statuses); } @@ -190,7 +259,14 @@ try { maxTouchPoints: 5, }); await session.send('Page.navigate', { url }); - await sleep(4000); + // The phone checks click through the tab bar and select a robot, so the + // reload has to have got as far as a populated roster — waited for, not + // slept through, for the reason `settle` exists. + await settle( + session, + `document.querySelectorAll('.robot').length`, + (rows) => rows > 0, + ); check( 'a coarse pointer is what the page thinks it has', @@ -252,20 +328,19 @@ try { const r = document.getElementById('map-canvas').getBoundingClientRect(); return { x: r.x + r.width / 2, y: r.y + r.height / 2 }; })()`); - const paint = () => - session.evaluate(`(() => { + const paintHash = `(() => { const c = document.getElementById('map-canvas'); const d = c.getContext('2d').getImageData(0, 0, c.width, c.height).data; let h = 0; for (let i = 0; i < d.length; i += 997) h = (h * 31 + d[i]) >>> 0; return h; - })()`); + })()`; const touch = (type, points) => session.send('Input.dispatchTouchEvent', { type, touchPoints: points.map(([x, y], id) => ({ x, y, id })), }); - const before = await paint(); + const before = await session.evaluate(paintHash); await touch('touchStart', [ [canvas.x - 40, canvas.y], [canvas.x + 40, canvas.y], @@ -279,8 +354,9 @@ try { [canvas.x + 160, canvas.y], ]); await touch('touchEnd', []); - await sleep(400); - check('two fingers zoom the map', (await paint()) !== before); + // The redraw is a frame away, not a fixed interval away. + const after = await settle(session, paintHash, (hash) => hash !== before, 3000); + check('two fingers zoom the map', after !== before); const phoneShot = shot.replace(/(\.png)?$/, '-phone.png'); const phonePng = await session.send('Page.captureScreenshot', { format: 'png' }); diff --git a/mote_fleet/test/fake_robots.py b/mote_fleet/test/fake_robots.py new file mode 100644 index 0000000..ed14b5d --- /dev/null +++ b/mote_fleet/test/fake_robots.py @@ -0,0 +1,437 @@ +"""A fleet that exists only on the wire. + +The dashboard consumes the control-plane contract and nothing else: presence, +health, pose and task status arriving over MQTT, and one command going back the +other way. So the cheapest honest thing to point it at is a script that +publishes exactly that contract — no ROS, no Nav2, no hardware — which is what +this is. It is **not** a second robot implementation: every payload is built by +``protocol.py``, the same module the real agent builds them with, so a change to +the wire changes this fixture or fails it. + +What it does model, because the UI renders each of them differently: + +* **presence, with a Last Will** — the ``offline`` profile connects, publishes + its retained state, then drops the socket *without* a DISCONNECT, which is the + only way to see the broker publish the will on the robot's behalf. +* **health with subsystems** — an ``ok`` robot and a ``degraded`` one, so the + roll-up and the per-subsystem rows both have something to draw. +* **a pose that moves** — retained, republished a few times a second, so the map + is live rather than a single dot. +* **task status transitions** — ``dispatched`` → ``accepted`` → ``succeeded`` + for a command the robot's grammar knows, ``dispatched`` → ``rejected`` for one + it does not, and a redelivery re-publishes the last status rather than + re-running the task. That is the agent's single-in-flight rule (``dispatch.py`` + owns it for real); here it is just enough of it to drive the status log. + +Run it against a broker of your own while working on ``server/ui/``:: + + python mote_fleet/test/fake_robots.py --host 127.0.0.1 --port 1883 + +or let ``ui_check.py`` bring up a whole private stack around it +(``pixi run fleet-ui-check``). +""" + +from __future__ import annotations + +import argparse +import math +import signal +import sys +import threading +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +import paho.mqtt.client as mqtt # noqa: E402 + +from mote_fleet import protocol # noqa: E402 + +#: Command words the robot's task layer knows (``mote_tasks/task_server.py``). +#: Anything else is rejected, which is a case the status log should be able to +#: show without an operator having to invent a broken robot. +VERBS = ("fetch", "goto") + +#: Zone names the sim worlds' ``zones.yaml`` files all carry, so a `goto` typed +#: into the dashboard against the shipped basemap succeeds. +ZONES = ("pickup", "dropoff", "home") + +PROFILES = ("ok", "degraded", "offline") + + +def subsystems(state: str) -> list[dict]: + """A health roll-up shaped like the health monitor's ``/diagnostics_agg``. + + The names are the real robot's; the degraded one is a real degraded case + (``slip_monitor`` reports slip as DEGRADED, never FAULT). + """ + rows = [ + protocol.subsystem("drive", protocol.OK, "2 servos, 50 Hz"), + protocol.subsystem("lidar", protocol.OK, "10.0 Hz"), + protocol.subsystem("localisation", protocol.OK, "icp residual 0.9 cm"), + protocol.subsystem("system", protocol.OK, "cpu 31%, 46 C, disk 38%"), + ] + if state == protocol.DEGRADED: + rows[2] = protocol.subsystem( + "localisation", protocol.DEGRADED, "slip: 0.14 m over 1.0 s" + ) + return rows + + +class FakeRobot: + """One robot, on the wire only. + + Publishing happens from the ticker thread and from paho's network thread + (a command reply), which is safe: ``paho`` serialises publishes internally, + and the only shared state is the in-flight command, guarded below. + """ + + def __init__( + self, + robot_id: str, + *, + host: str, + port: int, + profile: str = "ok", + site: str = "office_world", + floor: str = "ground", + centre: tuple[float, float] = (0.0, 0.0), + radius: float = 0.6, + period_s: float = 40.0, + task_seconds: float = 2.0, + zones: tuple[str, ...] = ZONES, + revision: str = "", + ): + if profile not in PROFILES: + raise ValueError(f"unknown profile {profile!r} (want {'/'.join(PROFILES)})") + self.id = robot_id + self.host = host + self.port = port + self.profile = profile + self.site = site + self.floor = floor + self.centre = centre + self.radius = radius + self.period_s = period_s + self.task_seconds = task_seconds + self.zones = zones + self.revision = revision + self.started = time.monotonic() + self.dropped = False + + self._lock = threading.Lock() + self._task = None # the in-flight command, or None + self._last_status = {} # command id -> the status last published for it + + self.client = mqtt.Client( + mqtt.CallbackAPIVersion.VERSION2, client_id=f"fake-{robot_id}" + ) + self.client.on_connect = self._on_connect + self.client.on_message = self._on_message + # The will is what makes "the robot dropped off" instant instead of a + # timeout — set before connecting, because the broker records it from + # the CONNECT packet. + self.client.will_set( + protocol.topic(robot_id, protocol.PRESENCE), + protocol.encode( + protocol.presence(robot_id, False, reason="connection lost") + ), + qos=protocol.QOS, + retain=True, + ) + + # -- lifecycle -------------------------------------------------------- + + def start(self): + self.client.connect(self.host, self.port, keepalive=30) + self.client.loop_start() + return self + + def _on_connect(self, client, _userdata, _flags, reason_code, _properties=None): + if getattr(reason_code, "is_failure", False): + print(f"{self.id}: broker refused the connection: {reason_code}") + return + # Subscribing here rather than after connect() is what keeps a + # reconnect from coming back deaf. + client.subscribe(protocol.topic(self.id, protocol.COMMAND), qos=protocol.QOS) + self._publish( + protocol.PRESENCE, protocol.presence(self.id, True, agent="fake_robots") + ) + self.tick() + + def drop(self): + """Die the way a robot on a failing link does: no DISCONNECT packet, so + the broker publishes the will. Closing the socket after stopping the + network loop is what makes it a drop rather than a clean goodbye.""" + self.dropped = True + self.client.loop_stop() + try: + self.client.socket().close() + except (AttributeError, OSError): + pass + + def close(self): + """Go offline politely — the agent's own shutdown path.""" + if self.dropped: + return + self._publish( + protocol.PRESENCE, protocol.presence(self.id, False, reason="stopped") + ) + self.client.loop_stop() + self.client.disconnect() + + # -- publishing ------------------------------------------------------- + + def _publish(self, leaf: str, payload: dict, retain: bool = True): + self.client.publish( + protocol.topic(self.id, leaf), + protocol.encode(payload), + qos=protocol.QOS, + retain=retain, + ) + + def health_payload(self) -> dict: + state = protocol.DEGRADED if self.profile == "degraded" else protocol.OK + summary = ( + "slip detected while turning" + if state == protocol.DEGRADED + else "all subsystems nominal" + ) + with self._lock: + task = dict(self._task["summary"]) if self._task else None + return protocol.health( + self.id, + state, + summary, + subsystems(state), + task=task, + site=self.site, + floor=self.floor, + version="fake-robots", + uptime_s=time.monotonic() - self.started, + # Which revision this robot is *running*. Left out unless told, so + # a fixture never claims a revision the registry then reports as + # out of date — that banner should mean a real robot behind a map. + map=( + {"site": self.site, "floor": self.floor, "revision": self.revision} + if self.revision + else None + ), + ) + + def pose_payload(self) -> dict: + """A slow circle. Retained, so the map is populated the instant the page + loads and moves afterwards.""" + angle = 2 * math.pi * ((time.monotonic() - self.started) / self.period_s) + return protocol.pose( + self.id, + self.centre[0] + self.radius * math.cos(angle), + self.centre[1] + self.radius * math.sin(angle), + angle + math.pi / 2, + site=self.site, + floor=self.floor, + ) + + def tick(self, health: bool = True): + """One round of the periodic publishes.""" + if self.dropped: + return + if health: + self._publish(protocol.HEALTH, self.health_payload()) + self._publish(protocol.POSE, self.pose_payload()) + self._finish_due_task() + + # -- commands --------------------------------------------------------- + + def _on_message(self, _client, _userdata, message): + try: + payload = protocol.decode(message.payload, protocol.COMMAND) + except protocol.ProtocolError as exc: + print(f"{self.id}: refusing a malformed command: {exc}") + return + self._handle(payload) + + def _handle(self, payload: dict): + command_id, text = payload["id"], payload["command"] + with self._lock: + previous = self._last_status.get(command_id) + if previous is not None: + # A redelivery is recognised, never re-run: the broker may + # redeliver a QoS-1 command, and the correlation id is what + # tells the two apart. + self._publish(protocol.STATUS, previous) + return + busy = self._task + if busy is not None: + self._reply( + command_id, text, protocol.REJECTED, f"busy with {busy['text']!r}" + ) + return + + self._reply(command_id, text, protocol.DISPATCHED) + refusal = self._refuse(text) + if refusal: + self._reply(command_id, text, protocol.REJECTED, refusal) + return + with self._lock: + self._task = { + "id": command_id, + "text": text, + "due": time.monotonic() + self.task_seconds, + "summary": { + "id": command_id, + "command": text, + "state": protocol.ACCEPTED, + }, + } + self._reply(command_id, text, protocol.ACCEPTED, "running") + + def _refuse(self, text: str) -> str: + """The task layer's grammar, as far as a wire fake can honour it.""" + words = text.split() + if not words: + return "empty command" + verb, rest = words[0], words[1:] + if verb not in VERBS: + return f"unknown command {verb!r}" + if verb == "goto": + if len(rest) != 1: + return "goto takes one zone" + if rest[0] not in self.zones: + return f"unknown zone {rest[0]!r}" + if verb == "fetch" and len(rest) != 2: + return "fetch takes a target and a drop zone" + return "" + + def _reply(self, command_id, text, state, detail=""): + payload = protocol.status(self.id, command_id, text, state, detail=detail) + with self._lock: + self._last_status[command_id] = payload + self._publish(protocol.STATUS, payload) + + def _finish_due_task(self): + with self._lock: + task = self._task + if task is None or time.monotonic() < task["due"]: + return + self._task = None + self._reply(task["id"], task["text"], protocol.SUCCEEDED, "arrived") + self._publish(protocol.HEALTH, self.health_payload()) + + +def run( + robots: list[FakeRobot], *, tick_s: float = 0.5, health_every: int = 6, until=None +): + """Tick every robot until interrupted (or until ``until()`` is true). + + Pose goes out every tick so the map moves; health every few, which is close + to the agent's own 5 s heartbeat and keeps the "health is current" rule in + the UI on the right side of its staleness window. + """ + stop = threading.Event() + + def _signal(_number, _frame): + stop.set() + + for name in (signal.SIGINT, signal.SIGTERM): + signal.signal(name, _signal) + + count = 0 + while not stop.is_set() and not (until and until()): + for robot in robots: + robot.tick(health=count % health_every == 0) + count += 1 + stop.wait(tick_s) + + +def build(args) -> list[FakeRobot]: + robots = [] + for index, spec in enumerate(args.robot): + robot_id, _, profile = spec.partition(":") + robots.append( + FakeRobot( + robot_id, + host=args.host, + port=args.port, + profile=profile or "ok", + site=args.site, + floor=args.floor, + # Spread the circles along the default site's corridor so two + # robots are not one dot, and none of them drives into a wall. + centre=(index * 3.0 - 3.0, 0.0), + radius=0.6, + task_seconds=args.task_seconds, + zones=tuple(args.zones.split(",")) if args.zones else ZONES, + revision=args.revision, + ) + ) + return robots + + +def main(argv=None): + parser = argparse.ArgumentParser( + prog="fake-robots", description=__doc__.split("\n\n")[0] + ) + parser.add_argument("--host", default="127.0.0.1", help="broker host") + parser.add_argument("--port", type=int, default=1883, help="broker MQTT port") + parser.add_argument( + "--robot", + action="append", + metavar="ID[:PROFILE]", + help=f"a robot to publish as; PROFILE is one of {'/'.join(PROFILES)} " + "(repeatable; default: mote-01:ok mote-02:degraded mote-03:offline)", + ) + parser.add_argument("--site", default="office_world") + parser.add_argument("--floor", default="ground") + parser.add_argument( + "--revision", + default="", + help="the map revision these robots are running (default: report none, " + "so the dashboard never shows a made-up out-of-date map)", + ) + parser.add_argument( + "--zones", + default=",".join(ZONES), + help="zone names `goto` will accept; anything else is rejected", + ) + parser.add_argument( + "--task-seconds", + type=float, + default=2.0, + help="how long an accepted task takes to succeed", + ) + parser.add_argument( + "--duration", + type=float, + default=0.0, + help="stop after this many seconds (default: run until interrupted)", + ) + args = parser.parse_args(argv) + args.robot = args.robot or ["mote-01:ok", "mote-02:degraded", "mote-03:offline"] + + robots = build(args) + for robot in robots: + robot.start() + print( + f"{robot.id}: {robot.profile}, publishing {protocol.topic(robot.id, '#')}" + ) + # Let the retained state land before the will-path robot drops: an offline + # robot the dashboard has never seen any health for is a less interesting + # (and less realistic) row than one that reported and then vanished. + time.sleep(1.0) + for robot in robots: + if robot.profile == "offline": + robot.drop() + print(f"{robot.id}: dropped the socket — the broker publishes its will") + + deadline = time.monotonic() + args.duration if args.duration else None + try: + run(robots, until=(lambda: time.monotonic() > deadline) if deadline else None) + finally: + for robot in robots: + robot.close() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/mote_fleet/test/test_deploy_config.py b/mote_fleet/test/test_deploy_config.py index 8bb8ce6..82821b7 100644 --- a/mote_fleet/test/test_deploy_config.py +++ b/mote_fleet/test/test_deploy_config.py @@ -24,6 +24,9 @@ COMPOSE = REPO / "mote_fleet" / "deploy" / "docker-compose.yml" BROKER_SH = REPO / "mote_fleet" / "server" / "broker.sh" MOSQUITTO_CONF = REPO / "mote_fleet" / "server" / "mosquitto.conf" +#: The third broker that could drift: the one `fleet-ui-check` runs to check the +#: dashboard. It reads the compose file's pin for the same reason broker.sh does. +UI_CHECK = REPO / "mote_fleet" / "test" / "ui_check.py" PIN = re.compile(r"^ *image: *\$\{MOTE_BROKER_IMAGE:-([^}]*)\}", re.M) @@ -40,7 +43,7 @@ def test_the_broker_image_is_pinned_exactly_once(): others = [ path - for path in (BROKER_SH, MOSQUITTO_CONF) + for path in (BROKER_SH, MOSQUITTO_CONF, UI_CHECK) if "eclipse-mosquitto:" in path.read_text() ] assert others == [], ( diff --git a/mote_fleet/test/test_fake_robots.py b/mote_fleet/test/test_fake_robots.py new file mode 100644 index 0000000..4672d23 --- /dev/null +++ b/mote_fleet/test/test_fake_robots.py @@ -0,0 +1,132 @@ +"""The wire-only robots are held to the wire. + +``fake_robots.py`` exists so the dashboard can be checked against something, and +its whole claim is that it publishes the control-plane contract and nothing +else. That claim is worth exactly as much as a test: without one, a payload the +UI has stopped understanding would still sail through ``fleet-ui-check`` and the +fixture would quietly become a second, wrong definition of the wire. + +Nothing here connects: ``FakeRobot`` builds its MQTT client but only touches the +network in ``start()``, so this tier runs anywhere paho imports — no broker, no +ROS, both architectures. +""" + +import pytest + +pytest.importorskip("paho.mqtt.client") + +import fake_robots # noqa: E402 + +from mote_fleet import protocol # noqa: E402 + + +@pytest.fixture +def robot(): + """One robot, with its publishes intercepted instead of sent.""" + subject = fake_robots.FakeRobot("mote-01", host="127.0.0.1", port=1883) + subject.published = [] + subject._publish = lambda leaf, payload, retain=True: subject.published.append( + (leaf, payload) + ) + return subject + + +def of(robot, leaf): + return [payload for name, payload in robot.published if name == leaf] + + +def states(robot): + return [payload["state"] for payload in of(robot, protocol.STATUS)] + + +@pytest.mark.parametrize("profile", fake_robots.PROFILES) +def test_health_and_pose_meet_the_contract(profile): + robot = fake_robots.FakeRobot("mote-01", host="h", port=1, profile=profile) + protocol.check(robot.health_payload(), protocol.HEALTH) + protocol.check(robot.pose_payload(), protocol.POSE) + + +def test_an_unknown_profile_is_refused(): + with pytest.raises(ValueError): + fake_robots.FakeRobot("mote-01", host="h", port=1, profile="haunted") + + +def test_health_carries_subsystems_and_only_claims_a_map_when_told(): + plain = fake_robots.FakeRobot("mote-01", host="h", port=1).health_payload() + assert [row["name"] for row in plain["subsystems"]] + # A fixture that invented a revision would show up on the dashboard as a + # robot running an out-of-date map, which is a real state and must not be + # faked into existence. + assert plain["map"] is None + + told = fake_robots.FakeRobot( + "mote-01", host="h", port=1, revision="20260708T000623" + ).health_payload() + assert told["map"]["revision"] == "20260708T000623" + + +def test_degraded_is_degraded_all_the_way_down(): + degraded = fake_robots.FakeRobot( + "mote-01", host="h", port=1, profile="degraded" + ).health_payload() + assert degraded["state"] == protocol.DEGRADED + assert protocol.DEGRADED in [row["state"] for row in degraded["subsystems"]] + + +def test_a_known_command_runs_to_success(robot): + command = protocol.command("goto dropoff") + robot._handle(command) + assert states(robot) == [protocol.DISPATCHED, protocol.ACCEPTED] + + robot._task["due"] = 0 # the task's time is up + robot.tick() + assert states(robot)[-1] == protocol.SUCCEEDED + for payload in of(robot, protocol.STATUS): + protocol.check(payload, protocol.STATUS) + assert payload["id"] == command["id"] + + +@pytest.mark.parametrize( + "text,reason", + [ + ("wibble", "unknown command"), + ("goto nowhere", "unknown zone"), + ("goto", "one zone"), + ("fetch box", "target and a drop zone"), + ("", "empty"), + ], +) +def test_the_grammar_refuses_what_the_task_layer_would(robot, text, reason): + robot._handle(protocol.command(text)) + assert states(robot) == [protocol.DISPATCHED, protocol.REJECTED] + assert reason in of(robot, protocol.STATUS)[-1]["detail"] + + +def test_a_redelivery_is_recognised_not_re_run(robot): + command = protocol.command("goto home") + robot._handle(command) + robot._handle(command) + # The same command id arriving twice re-publishes where it got to; it does + # not start a second task, and it is not rejected as "busy" with itself. + assert states(robot) == [ + protocol.DISPATCHED, + protocol.ACCEPTED, + protocol.ACCEPTED, + ] + + +def test_a_second_command_is_refused_while_one_is_in_flight(robot): + robot._handle(protocol.command("goto home")) + robot._handle(protocol.command("goto pickup")) + assert states(robot)[-1] == protocol.REJECTED + assert "busy" in of(robot, protocol.STATUS)[-1]["detail"] + + +def test_the_will_is_an_offline_presence(robot): + # paho keeps the will it was handed; this is the payload the *broker* + # publishes when the socket drops, which is the whole point of the offline + # profile and cannot be asserted from the robot's own publishes. + will = robot.client._will_payload + protocol.check(protocol.decode(will), protocol.PRESENCE) + assert protocol.decode(will)["online"] is False + assert robot.client._will_topic.decode() == "mote/v1/mote-01/presence" diff --git a/mote_fleet/test/ui_check.py b/mote_fleet/test/ui_check.py new file mode 100644 index 0000000..fb0c9d6 --- /dev/null +++ b/mote_fleet/test/ui_check.py @@ -0,0 +1,461 @@ +"""Bring up a private fleet, drive the dashboard in a real browser, tear it down. + + pixi run fleet-ui-check # assert; exit 0/1 + pixi run fleet-ui-check -- --keep # leave it up and print the URL + +M3 verified the dashboard against a real browser (``browser_check.mjs``), but +the stack it ran against was assembled by hand and the robots behind it were a +throwaway script — so the checks were repeatable only by whoever had the +scratch directory. This is that setup, committed: a broker, a fleet server, a +map, enrolled robots and an operator token, all on ports nobody else is using, +around the same browser assertions. + +**Nothing here touches the workstation's own fleet.** The broker is a container +on ports picked from the ephemeral range (the workstation usually already has +one on 1883 serving a real robot), the registry and the basemaps live in a +temporary directory rather than ``~/.mote-fleet``, and every process is started +in its own session so teardown reaps this stack and nothing else — the same +scoping rule the sim smoke test settled on. + +**Why it is not a pytest.** It needs a docker (conda-forge's mosquitto is built +without websockets, measured again at 2.0.20, and the browser's read path is +MQTT-over-WebSockets) and a chrome. See ``docs/fleet/m3-verification.md`` §2 for +where that leaves CI. + +The robots are ``fake_robots.py`` — the control-plane contract and nothing else, +which is exactly what the UI consumes. The *real* agent and behaviour tree are +covered by ``test_e2e_fleet.py``. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import socket +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.request +from pathlib import Path + +HERE = Path(__file__).resolve().parent +REPO = HERE.parents[1] +SERVER_DIR = REPO / "mote_fleet" / "server" + +# The registry and the wire contract, from the source tree — the same import +# the fleet server itself does, and the reason this needs no ROS. +sys.path.insert(0, str(SERVER_DIR)) +sys.path.insert(0, str(HERE.parent)) + +#: A committed site bundle to draw the robots on. The sim's, because it is the +#: only real map (map.yaml + PNG + zones, saved by ``sites.py``) in the tree — +#: a synthetic one would exercise neither the origin/resolution transform at a +#: real scale nor the zone overlay. +DEFAULT_SITE = "office_world" +SIM_SITES = REPO / "mote_simulation" / "sim_home" / "sites" + +#: What the fake fleet looks like: two robots reporting, one that dropped its +#: link so the broker published its will. +ROBOTS = ("ok", "degraded", "offline") + +#: Where the broker image tag is pinned — once, in the compose file, for every +#: broker the fleet runs (``broker.sh`` defers to it the same way). A tag of its +#: own here would be a third broker that could drift onto a mosquitto whose +#: websockets support differs, which is precisely the failure this check exists +#: to catch. ``test_deploy_config.py`` fails if one reappears. +COMPOSE = REPO / "mote_fleet" / "deploy" / "docker-compose.yml" +IMAGE_PIN = re.compile(r"^ *image: *\$\{MOTE_BROKER_IMAGE:-([^}]*)\}", re.M) + + +def broker_image() -> str: + override = os.environ.get("MOTE_BROKER_IMAGE") + if override: + return override + found = IMAGE_PIN.findall(COMPOSE.read_text()) + if not found: + raise RuntimeError( + f"cannot read the broker image pin from {COMPOSE} — set " + "MOTE_BROKER_IMAGE, or repair that file" + ) + return found[0] + + +def free_port() -> int: + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + return probe.getsockname()[1] + + +def wait_for_port(port: int, what: str, timeout: float = 30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + with socket.create_connection(("127.0.0.1", port), timeout=0.5): + return + except OSError: + time.sleep(0.2) + raise RuntimeError(f"{what} never came up on port {port}") + + +def wait_for_http(url: str, what: str, timeout: float = 30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(url, timeout=2) as response: + if response.status == 200: + return json.loads(response.read()) + except (urllib.error.URLError, OSError, ValueError): + time.sleep(0.2) + raise RuntimeError(f"{what} never answered {url}") + + +def post(url: str, payload: dict, token: str = "") -> dict: + request = urllib.request.Request( + url, + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + if token: + request.add_header("Authorization", f"Bearer {token}") + with urllib.request.urlopen(request, timeout=10) as response: + return json.loads(response.read()) + + +class Stack: + """The processes this run owns, and the one way to stop all of them.""" + + def __init__(self, root: Path): + self.root = root + self.processes: list[tuple[str, subprocess.Popen]] = [] + self.container = "" + + def spawn(self, name: str, argv: list[str], **kwargs) -> subprocess.Popen: + # start_new_session: the process group is the exact teardown scope, so + # stopping this stack can never reach another run's broker or server. + process = subprocess.Popen(argv, start_new_session=True, **kwargs) + self.processes.append((name, process)) + return process + + def broker(self, port: int, ws_port: int, image: str) -> str: + """A container mosquitto on ports of our own, from the shipped config. + + The config is the deployed one with its two listener lines rewritten, + rather than a second config that could drift from what a fleet box runs. + """ + conf = (SERVER_DIR / "mosquitto.conf").read_text() + conf = conf.replace("\nlistener 1883\n", f"\nlistener {port}\n") + conf = conf.replace("\nlistener 9001\n", f"\nlistener {ws_port}\n") + # Retained state is this run's alone: a fresh temp directory every time + # means yesterday's robots never appear in today's roster. + conf = conf.replace("persistence true", "persistence false") + path = self.root / "mosquitto.conf" + path.write_text(conf) + + self.container = f"mote-ui-check-{os.getpid()}" + self.spawn( + "broker", + [ + "docker", + "run", + "--rm", + "--name", + self.container, + "--network", + "host", + "-v", + f"{path}:/mosquitto/config/mosquitto.conf:ro", + image, + # Named explicitly, as broker.sh does, rather than trusting the + # image's default command to keep reading that path. + "mosquitto", + "-c", + "/mosquitto/config/mosquitto.conf", + ], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + wait_for_port(port, "the broker") + wait_for_port(ws_port, "the broker's websocket listener") + return self.container + + def stop(self): + if self.container: + subprocess.run( + ["docker", "rm", "-f", self.container], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + for _name, process in reversed(self.processes): + if process.poll() is not None: + continue + try: + os.killpg(process.pid, 15) + except (ProcessLookupError, PermissionError): + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + os.killpg(process.pid, 9) + + +def preflight(args) -> list[str]: + """What is missing, in the words of what to do about it.""" + missing = [] + if not shutil.which("docker"): + missing.append( + "docker — the browser's read path is MQTT-over-WebSockets and " + "conda-forge's mosquitto is built without them" + ) + if not shutil.which("node"): + missing.append("node — browser_check.mjs speaks the DevTools protocol") + if not ( + os.environ.get("CHROME") and shutil.which(os.environ["CHROME"]) + ) and not any( + shutil.which(name) for name in ("google-chrome", "chromium", "chromium-browser") + ): + missing.append("a chrome/chromium on PATH (or $CHROME pointing at one)") + if not (SIM_SITES / args.site).is_dir(): + missing.append(f"a site bundle at {SIM_SITES / args.site}") + return missing + + +def seed_maps(root: Path, site: str) -> Path: + """Copy the basemap in, rather than serving it out of the checkout. + + The registry writes to its maps directory (it re-announces floors, and an + operator can promote from the UI), and a verification run must not leave + anything in the git tree. + """ + maps = root / "sites" + maps.mkdir() + shutil.copytree(SIM_SITES / site, maps / site, symlinks=True) + return maps + + +def published_revision(maps: Path, site: str, floor: str) -> str: + """What the floor's ``map`` symlink points at — the revision the registry + will call canonical, and so the one the robots should report running.""" + link = maps / site / "floors" / floor / "map" + return os.path.basename(os.readlink(link)) if link.is_symlink() else "" + + +def enroll_fleet(url: str, registry, count: int) -> list[str]: + """Enrol through the real route, so the roster is the registry's own. + + The ids come back allocated (``mote-01``, ``mote-02``, …) rather than being + asserted here: dispatch 404s on a robot the registry has never seen, so a + fake fleet that skipped this would fail the one check that writes. + """ + ids = [] + for index in range(count): + answer = post( + f"{url}/v1/enroll", + { + "schema": 1, + "token": registry.new_token(note="fleet-ui-check"), + "fingerprint": f"ui-check-{index}", + "name": f"fake {index + 1}", + "facts": {"model": "wire-only", "harness": "fleet-ui-check"}, + }, + ) + ids.append(answer["robot_id"]) + return ids + + +def main(argv=None): + parser = argparse.ArgumentParser( + prog="fleet-ui-check", description=__doc__.split("\n\n")[0] + ) + parser.add_argument( + "--site", + default=DEFAULT_SITE, + help=f"site bundle to serve (default: {DEFAULT_SITE})", + ) + parser.add_argument("--floor", default="ground") + parser.add_argument( + "--keep", + action="store_true", + help="skip the browser checks; leave the stack up and print how to reach it", + ) + parser.add_argument( + "--screenshot", + default="fleet-ui.png", + help="where browser_check.mjs writes its screenshot", + ) + parser.add_argument( + "--image", + default=None, + help="broker container image (default: the compose file's pin, or " + "$MOTE_BROKER_IMAGE)", + ) + args = parser.parse_args(argv) + + args.image = args.image or broker_image() + + missing = preflight(args) + if missing: + print("fleet-ui-check needs:", file=sys.stderr) + for item in missing: + print(f" - {item}", file=sys.stderr) + return 2 + + from registry import Registry # noqa: E402 (server dir is on sys.path) + + mqtt_port, ws_port, http_port = free_port(), free_port(), free_port() + root = Path(tempfile.mkdtemp(prefix="mote-ui-check-")) + stack = Stack(root) + url = f"http://127.0.0.1:{http_port}" + try: + print(f"broker: {args.image} on {mqtt_port} (mqtt) / {ws_port} (ws)") + stack.broker(mqtt_port, ws_port, args.image) + + db = root / "registry.db" + maps = seed_maps(root, args.site) + stack.spawn( + "fleet-server", + [ + sys.executable, + "-u", + str(SERVER_DIR / "fleet_server.py"), + "--db", + str(db), + "--host", + "127.0.0.1", + "--port", + str(http_port), + "--broker-host", + "127.0.0.1", + "--broker-port", + str(mqtt_port), + "--broker-ws-port", + str(ws_port), + "--maps-dir", + str(maps), + ], + ) + wait_for_http(f"{url}/healthz", "the fleet server") + print(f"server: {url} (state in {root})") + + registry = Registry(str(db)) + ids = enroll_fleet(url, registry, len(ROBOTS)) + token = registry.new_operator(name="fleet-ui-check") + print(f"robots: {', '.join(ids)} on {args.site}/{args.floor}") + + stack.spawn( + "fake-robots", + [ + sys.executable, + "-u", + str(HERE / "fake_robots.py"), + "--host", + "127.0.0.1", + "--port", + str(mqtt_port), + "--site", + args.site, + "--floor", + args.floor, + "--revision", + published_revision(maps, args.site, args.floor), + *sum( + ( + ["--robot", f"{name}:{profile}"] + for name, profile in zip(ids, ROBOTS) + ), + [], + ), + ], + ) + # The dashboard reads retained state, so the robots must have published + # before the browser connects — otherwise the roster check is a race. + # Waiting on the broker rather than on a sleep also settles the will: + # the dropped robot's retained presence is the broker's own doing, and + # if it never arrives the fixture is not modelling what it claims to. + wait_for_retained(mqtt_port, ids, dropped=ids[ROBOTS.index("offline")]) + + if args.keep: + print(f"\n open {url}") + print(f" token {token}") + print(f" broker ws://127.0.0.1:{ws_port}") + print("\nCtrl-C to tear it all down.") + try: + while True: + time.sleep(3600) + except KeyboardInterrupt: + print() + return 0 + + print() + result = subprocess.run( + [ + "node", + str(HERE / "browser_check.mjs"), + url, + token, + str(Path(args.screenshot).resolve()), + ] + ) + return result.returncode + except (RuntimeError, urllib.error.URLError, KeyboardInterrupt) as exc: + print(f"fleet-ui-check: {exc}", file=sys.stderr) + return 2 + finally: + stack.stop() + shutil.rmtree(root, ignore_errors=True) + + +def wait_for_retained( + port: int, ids: list[str], *, dropped: str, timeout: float = 30.0 +): + """Wait for the state the page will be handed on connect. + + Subscribing here is the same thing the browser does a moment later, so what + this returns on is exactly what the roster, the health roll-up and the map + are about to be drawn from — including the *offline* presence the broker + publishes on the dropped robot's behalf. + """ + import paho.mqtt.client as mqtt + + from mote_fleet import protocol + + seen: dict[tuple[str, str], dict] = {} + + def collect(_client, _userdata, message): + parsed = protocol.parse_topic(message.topic) + if parsed: + seen[parsed] = json.loads(message.payload) + + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="ui-check-wait") + client.on_message = collect + client.connect("127.0.0.1", port, keepalive=30) + client.subscribe(f"{protocol.ROOT}/{protocol.VERSION}/#", qos=protocol.QOS) + client.loop_start() + try: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + healthy = all( + (robot, "health") in seen and (robot, "pose") in seen for robot in ids + ) + will = seen.get((dropped, "presence"), {}).get("online") is False + if healthy and will: + print(f" {dropped} is offline — the broker published its will") + return + time.sleep(0.25) + missing = [robot for robot in ids if (robot, "health") not in seen] + raise RuntimeError( + f"retained state never arrived: no health from {missing or 'nobody'}" + + ("" if will else f"; no will for {dropped}") + ) + finally: + client.loop_stop() + client.disconnect() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pixi.toml b/pixi.toml index 9b2d0c1..b98ab3a 100644 --- a/pixi.toml +++ b/pixi.toml @@ -48,6 +48,13 @@ fleetctl = "python -u mote_fleet/server/fleetctl.py" # operator CLI # `pixi run ` ambiguous. The no-docker fallback is `fleet-broker-local`, # below, which does need the environment: it runs conda's binary. fleet-broker = "bash mote_fleet/server/broker.sh" +# The dashboard, checked in a real browser against a fake fleet: broker, server, +# basemap, wire-only robots and the browser assertions, on ports of their own, +# torn down afterwards (docs/fleet/m3-verification.md §2). Needs docker and a +# chrome, so it lives here for the same reason `fleet-broker` does — a +# dependency on the machine, not on an environment. `-- --keep` leaves the stack +# up instead, which is the loop for working on server/ui/. +fleet-ui-check = "python -u mote_fleet/test/ui_check.py" # Server pipelines (docs/fleet/server-pipelines.md). The deployed servers need # docker and nothing else — these tasks are the convenience of running the same