[FEAT] Auto-retry admin commands until verified successful

[Yeraze]

Feature Request

Add a mechanism for auto-retrying admin commands until verified successful, instead of a single fire-and-forget send.

Problem

For remote nodes with weak/lossy links, admin commands (config changes, etc.) often need several attempts before they land. Today the user has to manually notice the command didn't take effect and resend it themselves — repeatedly, for nodes that may take many tries.

Proposed behavior

MeshMonitor sends the admin command, then verifies whether it actually applied (e.g. by re-reading the relevant config/state from the node). If verification fails, it automatically retries — up to some bound (count and/or timeout) — instead of requiring the user to manually resend.

Relationship to other work

This depends on the async admin-command rework already proposed in #4482 (decoupling the HTTP request from the ACK/passkey wait) — that's the piece that would let a retry loop run in the background without blocking the request thread.

Requested by

@wilhel1812 (Discord: Wilhelm), via #admin-command-trouble on 2026-08-01.

---

Authored by NodeZero 0️⃣

[Yeraze]

Reopening — PR #4502 implemented ACK-based retry only. From the PR body: "This needs no per-command verification logic, so it applies uniformly to every command kind — including ones with no readable state, like reboot." It does not wait out a node reboot, and it never re-reads the setting from the node to confirm it was actually stored (adminOperationService.ts:57-63, adminRoutes.ts:1962-1976 — success comes from the mesh routing ACK on the original send, not a fresh config read). That's the part of this issue still open: reboot-safe, re-query-based verification.

Refined scope from @wilhel1812:

• Should be separate from normal remote admin controls, not baked into the same synchronous request path — for far-away nodes this could run for hours.
• Not something most users need, so likely opt-in / a distinct flow rather than default behavior.
• Users need a way to see progress on a running verification job. A "running jobs" view or notification center would cover that, but that's a bigger feature than this issue — flagging it as a related need, not scoping it in here.

---

Authored by NodeZero 0️⃣

[Yeraze]

Follow-up from a Discord discussion (credit to NullVoid for the question): does retrying until ACK risk a reboot loop on an asymmetric link (node can hear MeshMonitor better than it can transmit back), and does a retry resend the exact same packet or a new one?

Current behavior (verified against main):

• The retry loop only fires on ack.timedOut (adminRoutes.ts:1953-1969, adminOperationService.ts:44,62-63).
• reboot is not in ACK_AWAITED_COMMANDS (adminRoutes.ts:1483-1488), so it fire-and-forgets and is immediately marked succeeded (adminRoutes.ts:1959-1961) — it never enters the retry path today. No loop risk right now.
• If a command is retried, each attempt gets a freshly randomized packet ID (protobufService.ts:1917-1918) — no firmware-level dedup would catch a resend, so a retried destructive command would execute again for real.

Why this matters for this issue specifically: verification-based retry (the actual ask here) will likely want to await commitEditSettings (proto commit_edit_settings, adminRoutes.ts:1883-1884) to confirm config was applied. Per the begin_edit_settings comment in admin.proto (lines 460-464), committing edits triggers an implicit save-to-filesystem-and-reboot — so commitEditSettings has the same "duplicate destructive side effect" exposure as reboot itself, and is a much more likely candidate to get added to ACK_AWAITED_COMMANDS as this feature grows.

Suggest adding a warning comment above ACK_AWAITED_COMMANDS so this doesn't get missed later:

// WARNING: commands added here are retried on ack-timeout using a freshly
// generated packet ID (protobufService.ts:1917-1918) — firmware has no
// dedup for the resend. Before adding a command, confirm it is safe to
// receive twice. Commands with irreversible/destructive device-state
// side effects (reboot, factory reset, nodedb reset, commitEditSettings —
// which applies pending changes and can trigger an implicit reboot per
// admin.proto's begin_edit_settings comment) must not be added without an
// additional safeguard, e.g. verifying device state before resending.

Other commands checked for the same reboot-on-receipt exposure: purgeNodeDb (nodedb_reset) — proto comment doesn't mention reboot, unverified either way. factory_reset_device/factory_reset_config/shutdown_seconds are defined in admin.proto but not implemented in MeshMonitor at all currently, so moot for now.

Authored by NodeZero 0️⃣

[Yeraze]

Follow-up (credit to NullVoid again): does this same exposure already exist in a live automated path, not just a hypothetical future one?

Yes — Automatic Favorites Management (#2608) already has its own independent retry loop, separate from this issue's adminOperationService.

• Server: autoFavoriteManagementService.ts, a once-a-minute scheduler (CHECK_INTERVAL_MS, line 33), initialized at server.ts:367.
• It waits for a routing ACK per attempt via sendFavoriteNodeAwaitAck() (lines 387, 403) → favoritesService.ts:227 → AdminTransactionService.sendAdminCommandAwaitAck (adminTransactionService.ts:162) — the same low-level primitive the manual admin route uses.
• Unconfirmed assignments are re-sent on the next scheduled cycle (autoFavoriteManagementService.ts:396-407; doc comment lines 19-23 explicitly notes: "Favorite assignment over LoRa is fire-and-forget... The re-favorite pass exists to re-assert assignments").
• Each re-send goes through the same packet-ID-per-send path noted above (protobufService.ts:1917-1918) — no dedup, so a node with a working RX / broken TX path would receive (and process) a fresh set_favorite_node admin message every cycle until the ACK gets through. Functionally the same asymmetric-link exposure discussed for reboot, except this one is real and already shipped, not prospective.

Separately, there's also an older, simpler distance/0-hop-based auto-favorite (favoritesService.ts:234-310, checkAutoFavorite/autoFavoriteSweep) that's fully fire-and-forget with no ACK wait at all (favoritesService.ts:150).

For comparison: the Automation Engine's scheduled-reboot action (action.deviceReboot, #3995/#4126, actionExecutor.ts:434-465) has no ACK wait on either the local (meshtasticManager.ts:13356) or remote (remoteAdminService.ts:598-649) path — so it carries no retry-loop exposure at all, same as the manual reboot button.

Given setFavoriteNode/removeFavoriteNode are idempotent (re-applying the same favorite flag is harmless, unlike reboot), this isn't a safety bug — just flagging that the warning comment drafted above should also note that any automated caller reusing these ACK-awaited commands inherits the same no-dedup-on-resend behavior, so idempotency is a real requirement for anything added to ACK_AWAITED_COMMANDS, not just a theoretical concern.

Authored by NodeZero 0️⃣