Skip to content

Mirror Mender's fleet for registration and the admin Fleet page - #683

Open
Volyarosh94 wants to merge 1 commit into
fix/stub-pages-and-opsfrom
feat/mender-mirror
Open

Volyarosh94 wants to merge 1 commit into
fix/stub-pages-and-opsfrom
feat/mender-mirror

Conversation

@Volyarosh94

Copy link
Copy Markdown
Contributor

Based on #682 (stub pages and ops), which sits on the rest of the open chain, so merge those first.

What

  • Two tables, mender_devices and mender_sync_state (migration 0027). mender-auto-accept fills them with POST /api/internal/mender/snapshot: a bearer token (MENDER_SYNC_TOKEN, compared in constant time, 404 while unset), a 2 MiB cap, 409 for a generated_at that is not newer, and a whole-fleet upsert that marks devices missing from the body as not present.
  • MENDER_SOURCE picks where registration asks: api (live, the default), shadow (live answer decides, the mirror is compared in the log) or mirror. In mirror a stale snapshot refuses with mender_unreachable and mirror stale (...); the age limit is MENDER_MIRROR_MAX_AGE_S.
  • Two alerts: mender_snapshot_source_changed, and mender_snapshot_count_drop when a push lists more than 20% fewer devices than the last.
  • GET /api/admin/fleet and an admin Fleet page: one row per node with registry state, claim and contact email state, Mender state, presence (both, registry only, Mender only) and a banner when the mirror is stale.
  • nginx gets an exact location for the snapshot path, so the 2 MiB cap applies there and 1 MiB stays everywhere else.
  • docs/mender-snapshot.md is the contract for node-infra. The runbook gets a cutover section and alert entries, docs/alerting.md the three Mender alerts, and .env.example the new keys, with the MENDER_PAT block marked legacy.
  • Nothing changes under /v1/nodes, and contracts/nodes-v1.openapi.yaml is untouched. The api path stays until cutover step 7 in the runbook.

Calls I made

  • One bad device does not fail the push. It is dropped, or stored with a reason in snapshot_issue, and registration refuses that node_id only. 422 is for the envelope. Agreed with the node-infra side.
  • node_id is unique only among present rows without an issue, so a decommissioned device or a duplicate cannot block a push.
  • A snapshot dated more than 300 s ahead is refused. A last snapshot more than 30 s ahead of the server clock counts as stale (the clock stepped back).
  • A token shorter than 32 characters counts as unset. An unknown MENDER_SOURCE refuses registration with the alert.
  • A blank auth set id or public key counts as missing.
  • Shadow logs agreement as well as disagreement, so the soak has something to read. Refused requests are logged at most once a minute, and a standing device fault only when it is new or has changed.
  • The Fleet page also lists devices Mender has with no node_id.
  • No alert for flagged devices. The Fleet page marks them.
  • The published registration text still says the server asks Mender directly. The node contract is frozen here, so that goes in the step 7 PR.

Tested

just check green, new tests fail without their code, 0027 up/down/up on a copy of a dev database, and the Fleet page checked in a browser at desktop and phone width with a snapshot pushed by curl.

mender-auto-accept pushes the whole fleet to /api/internal/mender/snapshot.
MENDER_SOURCE (api, shadow, mirror) picks what registration reads.
The admin Fleet page shows registry, claim and Mender state side by side.
@claude

claude Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Claude finished @Volyarosh94's task in 3m 6s —— View job


Review: Mirror Mender's fleet for registration and the admin Fleet page

  • Gather context (PR description, diff vs origin/fix/stub-pages-and-ops, CLAUDE.md)
  • Review backend core (core/mender_mirror.py, routes/mender_sync.py, services/mender.py)
  • Review routes/node_register.py, routes/admin.py, services/fleet.py, migration 0027
  • Review frontend (FleetPage.tsx, types.ts, client.ts), nginx config, .env.example, docs cross-checks
  • Post findings

Summary

This is a careful, well-documented piece of work, and I did not find any correctness bugs. A few notes on what I specifically checked:

backend/routes/mender_sync.py / backend/core/mender_mirror.py (the trickiest part)

  • Verified the write ordering in _apply_devices (backend/routes/mender_sync.py:538-596): absent rows are flipped to present=False before flagged/clean writes, and node-id "moves" are cleared to NULL before the final write — so the partial unique index on (node_id) WHERE present=1 AND snapshot_issue IS NULL can never be violated by a device swap, a node_id moving off a decommissioned device, or two devices racing for the same id in one push. This held up under several scenarios I traced by hand (simultaneous swap of two node_ids, a device going absent while another claims its id, two devices in one payload sharing an id).
  • The _TAKE_WRITE_LOCK no-op update + read-after-lock pattern (mender_sync.py:496-501, 654-656) correctly serializes concurrent snapshot pushes against the generated_at monotonicity check.
  • stale_reason/snapshot_age_s (core/mender_mirror.py:145-176) correctly handle the clock-stepped-back case (negative age vs. MIRROR_FUTURE_TOLERANCE_S), and _lookup_mirror fails closed on both a stale mirror and a DB/parse error.
  • One deliberate (not a bug) design choice worth being aware of: in _sort_out (mender_sync.py:309-333), two devices sharing a node_id in the same push are both flagged as "shared", even if one of them already carries an unrelated field issue. That's a conservative choice (avoids a device silently becoming the sole clean claimant of an id another device still lists), consistent with the PR's stated call that "node_id is unique only among present rows without an issue."

routes/node_register.py

  • Confirmed the ordering requirements the docstring calls out still hold: rate limiter before the Mender/mirror lookup, config validation after identity resolution, and the shared 403 body covering unknown/unaccepted/unreachable cases identically.
  • MenderUnreachable → mender_unreachable alert path is wired correctly and fires with nothing written yet.

services/fleet.py

  • _rank/_mirror_rows pick a flagged present row over an unflagged one, and the most recently confirmed one otherwise, matching the module docstring.

Cross-checks

  • dashboard/src/types.ts's AdminFleet/AdminFleetRow/AdminFleetMirror line up field-for-field with services/fleet.py's _row/_header output.
  • docs/alerting.md's three Mender alert rows match the alert names and messages actually sent from the code.
  • nginx template: the /api/internal/mender/snapshot exact-match location with client_max_body_size 2m is added consistently across all four server blocks, ahead of the catch-all /api/ locations (correct precedence for an exact match).
  • backend/migrations/env.py and backend/migrations/versions/0027_mender_mirror.py register the new tables correctly for alembic check; the down migration drops things in FK-safe order.
  • Confirmed node_register.py's edits are docstring/comment only — no route signature or model changes, consistent with the PR's claim that contracts/nodes-v1.openapi.yaml needs no regeneration.

Nothing here blocks merge from my read. Nice work on the concurrency reasoning in particular — it's the part most PRs in this area get wrong.

@Volyarosh94

Copy link
Copy Markdown
Contributor Author

The sender is offworldlabs/node-infra#12. It follows docs/mender-snapshot.md here, and pushes nothing until RETINA_SYNC_TARGETS is set, so either can merge first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant