Back to results
US 9,317,326 B2Granted
Consistent migration of a group of virtual machines using source and destination group messaging
AssigneeVMware, Inc.
InventorsArunachalam Ramanathan, Gabriel Tarasuk-Levin
Life of this patent
Estimating the term…
Abstract
A plurality of virtual machines (VMs) is migrated from a source group to a destination group in such as way as to achieve consistency and either availability or group preservation. Execution of VMs in the source group is selectively delayed during state migration so that memory transfer of all the VMs in the group will converge roughly at the same time. After VM state transfer to the destination group, execution switch-over is coordinated using different handshake and acknowledgement messages, passed either through a “leader” VM in each group, or directly between source-destination VM pairs.
Claims28
A method for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising:
- pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- stunning each source VM after completion of the pre-copying contents of the memory associated with it;
- checkpointing the stunned source VM and transferring its checkpoint to the destination platform;
- transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- starting execution of each destination VM for which the acknowledgement message has been received;
- and
- switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- selecting a source leader VM and a destination leader VM;
- when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs;
- when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM;
- and
- forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution.
10 more elements
A method for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising:
- pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- stunning each source VM after completion of the pre-copying contents of the memory associated with it;
- checkpointing the stunned source VM and transferring its checkpoint to the destination platform;
- transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- starting execution of each destination VM for which the acknowledgement message has been received;
- and
- switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- broadcasting from each destination VM a switch-over ready message to the other destination VMs;
- and
- upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution.
8 more elements
A virtualized computing system comprising:
- a processor;
- a system memory;
- a group of source virtual machines (VMs) on at least one source platform;
- a group of destination VMs on at least one destination platform;
- a central supervisory system provided for provisioning and managing the overall virtual infrastructure, including defining the groups of source and destination VMs and computing a migration specification for the source and destination VMs;
- and
- at least one system-level migration sub-system including computer-executable code for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs, for stunning each source VM upon completion of the pre-copying of its respective memory contents such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- for checkpointing and transferring to the destination platform a checkpoint for each stunned source VM;
- for transmitting from the at least one destination platform to the at least one source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- for starting execution of each destination VM for which the acknowledgement message has been received and for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- broadcasting from each destination VM a switch-over ready message to the other destination VMs;
- and
- upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution.
11 more elements
A virtualized computing system comprising:
- a processor;
- a system memory;
- a group of source virtual machines (VMs) on at least one source platform;
- a group of destination VMs on at least one destination platform;
- a central supervisory system provided for provisioning and managing the overall virtual infrastructure, including defining the groups of source and destination VMs and computing a migration specification for the source and destination VMs;
- and
- at least one system-level migration sub-system including computer-executable code for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs, for stunning each source VM upon completion of the pre-copying of its respective memory contents such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- for checkpointing and transferring to the destination platform a checkpoint for each stunned source VM;
- for transmitting from the at least one destination platform to the at least one source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- for starting execution of each destination VM for which the acknowledgement message has been received and for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- selecting a source leader VM and a destination leader VM;
- when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs;
- when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM;
- and
- forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution.
13 more elements
A non-transitory computer-readable storage medium having data stored therein representing software executable by a computer, the software including:
- instructions for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising:
- instructions for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- instructions for stunning each source VM after completion of the pre-copying contents of the memory associated with it;
- instructions for checkpointing the stunned source VM and transferring its checkpoint to the destination platform;
- instructions for transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- instructions for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- instructions for starting execution of each destination VM for which the acknowledgement message has been received;
- and
- instructions for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- selecting a source leader VM and a destination leader VM;
- when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs;
- when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM;
- and
- forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution.
11 more elements
A non-transitory computer-readable storage medium having data stored therein representing software executable by a computer, the software including:
- instructions for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising:
- instructions for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window;
- instructions for stunning each source VM after completion of the pre-copying contents of the memory associated with it;
- instructions for checkpointing the stunned source VM and transferring its checkpoint to the destination platform;
- instructions for transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform;
- instructions for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM;
- and
- instructions for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by:
- broadcasting from each destination VM a switch-over ready message to the other destination VMs;
- and
- upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution.
8 more elements
Dependencies are inferred from the claim text.