Repository navigation
20261024 - Leave the stack running when a rollback has nothing to restore - #66
Merged
Merged
Conversation
…tore A rollback can arrive with nothing staged at all, most often when the Download phase is cut short by a power loss. The store then holds only current/, which is the live composition. Upstream's post-commit branch only asks whether new/ and previous/ are absent, and that is equally true when nothing was ever staged, so it renamed the live current/ to new/, stopped it, found no previous/ to restore, and still exited 0. The node was left running nothing at all while Mender recorded the rollback as a success. A node lost its stack exactly this way on 2026-10-10, after losing power a minute into a download. The guard added earlier for the same shape of failure during install cannot catch this one, because it tests for current/ and the rename has already taken current/ away by the time it runs. So this adds a guard above that branch instead: when new/, previous/ and cleanup/ are all absent and current/ is present, there is nothing to roll back to, so leave the running composition alone. cleanup/ is what separates this from a genuine rollback after a commit, which always has it, because commit_artifact puts the superseded composition there. Verified on a test node against the module it ships today, byte-identical to the one in this repo, by exercising rollback_artifact in all four store states it can be entered in, using a throwaway compose project and its own store. Before the fix the download-cut case ended with zero containers running; after it the stack is untouched, and the other three states give identical traces either way. README.md gains a docker-compose Update Module section covering the four fork behaviours and a table of the rollback states, since the fork was documented only in comments. The pointer comment in the mender role no longer says the fork fixes three things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An OTA rollback that arrives with nothing staged stopped the node's running
stack and left nothing in its place. The store holds only
current/in thatsituation, which is the live composition, and that is what the module stopped.
It then exited 0, so Mender recorded a successful rollback on a node that was no
longer running anything.
A node lost its stack exactly this way on 2026-10-10, after losing power about a
minute into a download.
Why
Upstream's "rolling back after a Commit" branch only asks whether
new/andprevious/are absent. That is equally true when nothing was ever staged, so aDownload-phase interruption takes the post-commit path: it renames the live
current/tonew/, stops it, finds noprevious/to restore, and reportssuccess.
The guard added earlier for the same shape of failure during install cannot
catch this one. It tests for
current/, and the rename above it has alreadytaken
current/away by the time it runs.Fix
A guard above that branch: when
new/,previous/andcleanup/are allabsent and
current/is present, there is nothing to roll back to, so leave therunning composition alone.
cleanup/is the discriminator.commit_artifactalways moves the supersededcomposition there, so a genuine post-commit rollback has it and a
nothing-staged rollback does not. The guard has to stay above the upstream
branch, because that branch's rename is what defeated the previous one.
Verification
Verified on a test node, against the module it ships today, which is
byte-identical to the one in this repo (
8abfe66d45970e8c). The real module wasexercised through its own
MENDER_DOCKER_COMPOSE_CONFIG_FILEoverride, withPERSISTENT_STOREpointed at a throwaway store holding its own compose project,so the node's real stack was never involved. The harness asserts the redirect is
in effect before it runs, and each run ends by confirming the real store and all
eight retina containers are untouched.
rollback_artifactwas exercised in all four store states it can be entered in:current/[cleanup], nothing running, exit 0[current], still runningcurrent/+cleanup/new/+current/new/+previous/The three states that already worked produce identical traces before and after,
so the guard changes nothing but the first row.
Not verified: a real interrupted Mender deployment. The failure was reproduced
by constructing the store states the module is entered in, not by cutting power
to a node mid-download.
Docs
owl-os had no doc covering the update module, which was described only in
comments.
README.mdgains adocker-compose Update Modulesection setting outthe four fork behaviours and a table of the rollback states and the correct
action in each. The pointer comment in the mender role no longer says the fork
fixes three things.
After this
This ships in
os-v0.19.0. The module that handles that rollout is the onealready installed on each node, so the fix starts protecting deployments from
the one after it.
🤖 Generated with Claude Code