Skip to content

nan-vm-operator: failed/timed-out CreateInstance leaves ip_cidr pointing at a released IP (possible cross-member IP collision) #138

Closed

Description

@sre-helmcode

Found by the QA review of helmcode/nan-vm-operator#37 (pre-existing on main, not introduced by #37).

When CreateInstance fails in realFirstAction (dispatch.go, "ReleaseIP after CreateInstance failure"), the IP is released back to the pool but agents_runtime.ip_cidr still holds it. The next retry (hasDisk → stickyBoot → createOnHost, or on main the "" case → restartOrRecreate → createOnHost) recreates the VM with row.IPCIDR, an IP that may already be assigned to another agent. If the failure was a timeout and vmd actually finished the create, the VM runs on an IP the pool considers free.

Impact: possible IP collision between two members' microVMs on the same bridge (with nan-vmd network isolation the per-tap IP pin would drop one side's traffic; without it, ARP confusion / traffic to the wrong VM).

Acceptance:

  • On CreateInstance failure, restore the placeholder ip_cidr atomically with ReleaseIP, OR keep the IP allocated until a confirmed rollback (vmd reports the instance absent).
  • A timed-out create is reconciled against vmd (GetInstance) before the IP is released.
  • Tests for both failure modes.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions