Skip to content

Fix branch switch failure with symlink changes - #1077

Open
jojinkb wants to merge 1 commit into
fluxcd:mainfrom
jojinkb:fix-delete-nonexistent-file
Open

Fix branch switch failure with symlink changes#1077
jojinkb wants to merge 1 commit into
fluxcd:mainfrom
jojinkb:fix-delete-nonexistent-file

Conversation

@jojinkb

@jojinkb jojinkb commented Jul 25, 2026

Copy link
Copy Markdown

Fixes #1037

When an ImageUpdateAutomation uses a push.branch that has fallen behind the checkout ref and the changes between the two branches involve symlinks, SwitchBranch can fail with:

failed to checkout source: could not checkout to branch '<push-branch>':
remove <workdir>/<symlink-target>: no such file or directory

Once hit, every reconcile fails the same way until the push branch is manually rebased or deleted.

Root cause

go-git/go-billy#135: go-billy v5's osfs.BoundOS resolves a trailing symlink path component (BoundOS.absSecureJoin walks every component, including the leaf), so Remove/RemoveAll on a symlink path operate on the symlink target instead of the symlink itself.

During the branch switch, go-git removes worktree files that differ between the two branches, in lexical path order. Two failure modes follow:

  1. Hard failure (this issue): a path that is a symlink on the checkout ref but a regular file on the push branch is a Modify change, handled as Remove + re-create. Remove resolves the symlink to its target — which the same checkout has already deleted (e.g. deploy/_stacks/config.yaml sorts before deploy/<service>/config.yaml) — and fails with ENOENT. This reproduces the exact error in Controller errors when attempting to delete a non-existent file #1037, including the error naming the target path rather than the symlink being switched away from.
  2. Silent corruption: when the removal resolves to a target that still exists, the target is deleted in place of the symlink. The stale symlink (untracked) and/or the missing tracked target then show up in Status() and get committed to the push branch by CommitAndPush.

The go-billy fix exists only on the v6 (pre-release) line and won't be backported to v5 (see maintainer comments on go-git/go-billy#135), so waiting for a dependency bump would leave affected automations wedged indefinitely. source-controller hits the same root cause in fluxcd/source-controller#1921.

Fix

Wrap the worktree filesystem handed to the gogit client (worktreeFS in internal/source/worktree_fs.go): Remove and RemoveAll resolve intermediate path components securely like BoundOS does, but keep the final component unresolved so they operate on the path itself, never on a symlink target, and treat removal of an already-absent path as a no-op — matching git checkout semantics (and what go-billy v6 / os.Root do). Storage configuration moves from buildGitConfig to CheckoutSource, where the working directory is known.

The wrapper is entirely controller-side and can be dropped once go-git and go-billy are bumped to v6.

Test plan

  • New regression test TestSourceManager_CheckoutSource_symlinkSwitchBranch reproduces the scenario from the issue end-to-end (push branch behind the checkout ref, per-service config replaced by symlinks to a new _stacks file). On unpatched code it fails with the exact error from the issue (could not checkout to branch '...': remove .../deploy/_stacks/config.yaml: no such file or directory); with the fix the switch succeeds, the worktree matches the push branch state (symlink targets untouched, no stale symlinks) and is clean, so nothing bogus can be committed.
  • New unit tests TestWorktreeFS_* cover symlink-preserving removal, no-op removal of absent paths, root protection, and that relative paths cannot escape the worktree root.
  • go test -race ./internal/source/ passes
  • go test -race ./internal/controller/ (envtest) — tick after local verification

When switching to the push branch, go-git removes worktree files that
differ between the checkout ref and the push branch. The BoundOS billy
filesystem resolves a trailing symlink path component, so removing a
symlink path operates on its target instead of the symlink itself
(go-git/go-billy issue 135). When the target has already been deleted
by a preceding change of the same checkout, the branch switch fails
with "remove <target>: no such file or directory", permanently failing
the reconciliation until the push branch is manually rebased or
deleted. When the target still exists, it is deleted in place of the
symlink, corrupting the worktree with stale symlinks and missing files
that then get committed to the push branch.

Wrap the worktree filesystem handed to the gogit client so that Remove
and RemoveAll operate on the given path itself, never on the target it
may link to, and treat removal of an already-absent path as a no-op,
matching git checkout semantics. The upstream fix exists only on the
go-billy v6 pre-release line and won't be backported to v5; the
wrapper can be removed once go-git and go-billy are bumped to v6.

Assisted-by: Claude Code/claude-fable-5
Signed-off-by: Jojin <jojin.kb@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Controller errors when attempting to delete a non-existent file

1 participant