# US 2015/0149999 A1 · Virtual machine group migration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2015/0149999 A1 - Document ID: US20150149999A1 - Status: Published application - Kind: A1 - Application number: 14/092,619 - Filed: 2013-11-27 - Published: 2015-05-28 - Earliest claimed: 2013-11-27 - Assignees: VMware, Inc. - Applicants: VMware, Inc. - Inventors: Arunachalam RAMANATHAN; Gabriel TARASUK-LEVIN - Web page: https://patentgrep.com/patent/US20150149999A1 ## CPC - G06F 9/4856: {resumption being on a different machine, e.g. task migration, virtual machine migration (G06F9/5088 takes precedence)} - G06F 9/45533: {Hypervisors; Virtual machine monitors} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing} - G06F 2009/45583: {Memory management, e.g. access or allocation} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A plurality of virtual machines (VMs) is migrated from a source group to a destination group in such as way as to achieve consistency and either availability or group preservation. Execution of VMs in the source group is selectively delayed during state migration so that memory transfer of all the VMs in the group will converge roughly at the same time. After VM state transfer to the destination group, execution switch-over is coordinated using different handshake and acknowledgement messages, passed either through a “leader” VM in each group, or directly between source-destination VM pairs. ## Claims ### Claim 1 A method for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising: pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window; stunning each source VM after completion of the pre-copying contents of the memory associated with it; checkpointing the stunned source VM and transferring its checkpoint to the destination platform; transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform; receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM; and starting execution of each destination VM for which the acknowledgement message has been received. ### Claim 2 (depends on 1) The method of claim 1, comprising stunning the plurality of source VMs such that pre-copying for all the source VMs completes within the pre-copy convergence time window. ### Claim 3 (depends on 1) The method of claim 1, further comprising computing, for each source VM, a pre-copy convergence time. ### Claim 4 (depends on 3) The method of claim 3, comprising computing each pre-copy convergence time as a function of the respective VM's memory size, network transmit rate and dirty rate, where the dirty rate is a function of the rate at which the VM is currently changing the content of physical memory assigned to it. ### Claim 5 (depends on 4) The method of claim 4, further comprising transmitting the current pre-copy convergence time for each VM in the group to the other VMs in the group and adjusting the respective dirty rates and/or network transmit rates of the VMs in the group so that a memory transfer convergence time for all the VMs falls within a threshold period. ### Claim 6 (depends on 4) The method of claim 4, further comprising injecting stun times into at least one of the VMs in the group, whereby a pre-copy convergence time for that VM is shortened to fall within a threshold period. ### Claim 7 (depends on 4) The method of claim 4, further comprising, for at least one of the VMs in the group, installing memory traces, receiving pre-copy convergence times from other VMs in the group, and waiting in a trace phase before proceeding with memory pre-copy until a pre-copy convergence time for that VM falls within a threshold period determined as a function of the pre-copy convergence times. ### Claim 8 (depends on 1) The method of claim 1, further comprising starting execution of each destination VM only when execution of the respective source VM has been halted, whereby no VM is ever executing simultaneously on both the source and destination platforms. ### Claim 9 (depends on 1) The method of claim 1, further comprising switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: selecting a source leader VM and a destination leader VM; when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs; when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM; and forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution. ### Claim 10 (depends on 1) The method of claim 1, further comprising switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: broadcasting from each destination VM a switch-over ready message to the other destination VMs; and upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution. ### Claim 11 (depends on 1) The method of claim 1, further comprising pre-copying contents of disk storage associated with each source VM to disk storage associated with a corresponding respective one of the destination VMs. ### Claim 12 A virtualized computing system comprising: a group of source virtual machines (VMs) on at least one source platform; a group of destination VMs on at least one destination platform; a central supervisory system provided for provisioning and managing the overall virtual infrastructure, including defining the groups of source and destination VMs and computing a migration specification for the source and destination VMs; and at least one system-level migration sub-system including computer-executable code for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs, for stunning each source VM upon completion of the pre-copying of its respective memory contents such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window; for checkpointing and transferring to the destination platform a checkpoint for each stunned source VM; for transmitting from the at least one destination platform to the at least one source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform; for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM; and for starting execution of each destination VM for which the acknowledgement message has been received. ### Claim 13 (depends on 12) The system of claim 12, in which the migration sub-system is further provided for computing, for each source VM, a pre-copy convergence time (pre-copy convergence time). ### Claim 14 (depends on 13) The system of claim 13, further comprising: physical memory allocated to each source VM corresponding to virtual memory associated with that source VM; at least one network connecting the source and destination platforms, with a network transmit rate associated with each source VM; in which: each source VM has a current dirty rate that is a function of a rate at which the VM is currently changing the content of the physical memory allocated to it; and the migration sub-system is further provided for computing each pre-copy convergence time as a function of the respective VM's physical memory size, network transmit rate and dirty rate. ### Claim 15 (depends on 14) The system of claim 14, in which the migration sub-system is further provided for transmitting the current pre-copy convergence time for each VM in the group to the other VMs in the group and adjusting the respective dirty rates and/or network transmit rates of the VMs in the group so that a memory transfer convergence time for all the VMs falls within a threshold period. ### Claim 16 (depends on 14) The system of claim 14, in which the migration sub-system is further provided for injecting stun times into at least one of the VMs in the group, whereby a pre-copy convergence time for that VM is shortened to fall within a threshold period. ### Claim 17 (depends on 14) The system of claim 14, further comprising a respective system-level, monitoring component controlling each source VM, said monitoring component configured to install memory traces on the physical memory allocated to its source VM, to receive pre-copy convergence times from other VMs in the group, and to cause the respective source VM to wait in a trace phase before proceeding with memory pre-copy until a memory pre-copy convergence time for that VM falls within a threshold period determined as a function of the pre-copy convergence times. ### Claim 18 (depends on 12) The system of claim 12, in which, at all times, each VM executes exclusively on either the source platform or on the destination platform. ### Claim 19 (depends on 12) The system of claim 12, in which the migration sub-system is further configured for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: selecting a source leader VM and a destination leader VM; when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs; when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM; and forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution. ### Claim 20 (depends on 12) The system of claim 12, in which the migration sub-system is further configured for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: broadcasting from each destination VM a switch-over ready message to the other destination VMs; and upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution. ### Claim 21 A computer-readable storage medium having data stored therein representing software executable by a computer, the software including: instructions for migrating a group of virtual machines (VMs) from at least one source platform to at least one destination platform, comprising: instructions for pre-copying contents of memory associated with each source VM to memory associated with a corresponding respective one of the destination VMs such that copying the memories of all source VMs to be migrated is completed within a pre-copy convergence time window; instructions for stunning each source VM after completion of the pre-copying contents of the memory associated with it; instructions for checkpointing the stunned source VM and transferring its checkpoint to the destination platform; instructions for transmitting from the destination platform to the source platform a handshake message indicating readiness to begin execution of at least one of the destination VMs on the destination platform; instructions for receiving from the source platform an acknowledgement message indicating readiness to transfer execution from at least one source VM to the corresponding destination VM; and instructions for starting execution of each destination VM for which the acknowledgement message has been received. ### Claim 22 (depends on 21) The storage medium of claim 21, further comprising instructions for stunning the plurality of source VMs such that pre-copying for all the source VMs completes within the pre-copy convergence time window. ### Claim 23 (depends on 21) The storage medium of claim 21, further comprising instructions for computing, for each source VM, a pre-copy convergence time. ### Claim 24 (depends on 23) The storage medium of claim 23, further comprising instructions for computing each pre-copy convergence time as a function of the respective VM's memory size, network transmit rate and dirty rate, where the dirty rate is a function of the rate at which the VM is currently changing the content of physical memory assigned to it. ### Claim 25 (depends on 24) The storage medium of claim 24, further comprising instructions for transmitting the current pre-copy convergence time for each VM in the group to the other VMs in the group and adjusting the respective dirty rates and/or network transmit rates of the VMs in the group so that a memory transfer convergence time for all the VMs falls within a threshold period. ### Claim 26 (depends on 24) The storage medium of claim 24, further comprising instructions for injecting stun times into at least one of the VMs in the group, whereby a pre-copy convergence time for that VM is shortened to fall within a threshold period. ### Claim 27 (depends on 24) The storage medium of claim 24, further comprising, for at least one of the VMs in the group, instructions for installing memory traces, receiving pre-copy convergence times from other VMs in the group, and waiting in a trace phase before proceeding with memory pre-copy until a pre-copy convergence time for that VM falls within a threshold period determined as a function of the pre-copy convergence times. ### Claim 28 (depends on 21) The storage medium of claim 21, further comprising instructions for starting execution of each destination VM only when execution of the respective source VM has been halted, whereby no VM is ever executing simultaneously on both the source and destination platforms. ### Claim 29 (depends on 21) The storage medium of claim 21, further comprising instructions for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: selecting a source leader VM and a destination leader VM; when the destination leader VM has received destination VM resumption messages from all the other destination VMs, transmitting to the source leader VM a destination group resumption message, which the source leader VM forwards to the source VMs; when the source leader VM has received source VM acknowledgement messages from all source VMs, which each power off after sending the respective acknowledgement message, sending a source group acknowledgement message to the destination leader VM; and forwarding the source group acknowledgement message to each destination VM, whereupon the destination VMs resume execution. ### Claim 30 (depends on 21) The storage medium of claim 21, further comprising instructions for switching execution from the VMs running on the source platform to the corresponding VMs on the destination platform by: broadcasting from each destination VM a switch-over ready message to the other destination VMs; and upon receipt by any destination VM of switch-over ready messages from all the other destination VMs, sending a destination VM resumption message from that destination VM to a corresponding one of the source VMs, whereupon the corresponding source VM returns an acknowledgement message to the respective destination VM and powers off and the destination VM resumes execution. ### Claim 31 (depends on 21) The storage medium of claim 21, further comprising instructions for pre-copying contents of disk storage associated with each source VM to disk storage associated with a corresponding respective one of the destination VMs.