# US 10,503,543 B1 · Hosting virtual machines on a secondary storage system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,543 B1 - Document ID: US10503543B1 - Status: Granted - Kind: B1 - Application number: 16/389,201 - Filed: 2019-04-19 - Granted: 2019-12-10 - Earliest claimed: 2019-02-04 - Assignees: Cohesity, Inc. - Applicants: Cohesity, Inc. - Inventors: Anand Bhat; Anil Kumar Boggarapu; Arvind Jagannath - Web page: https://patentgrep.com/patent/US10503543B1 ## CPC - G06F 9/4856: {resumption being on a different machine, e.g. task migration, virtual machine migration (G06F9/5088 takes precedence)} - G06F 9/45533: {Hypervisors; Virtual machine monitors} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)} - G06F 9/5088: {involving task migration} - G06F 11/202: {where processing functionality is redundant (redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089)} - G06F 2009/45562: {Creating, deleting, cloning virtual machine instances} - G06F 2009/45579: {I/O management, e.g. providing access to device drivers or storage} - G06F 2009/45583: {Memory management, e.g. access or allocation} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract At least a portion of a virtual machine is hosted on at least one node of a first subset of a plurality of nodes of a secondary storage system. The virtual machine comprises a plurality of portions that can be distributed between the plurality of nodes and is configured into a first state of a plurality of states, such that, in the first state, the plurality of portions is distributed between a first subset of the plurality of nodes and each of the first subset of nodes stores a portion of the virtual machine in its corresponding storage device. A node from the second subset of the plurality of nodes to host the virtual machine in a second state of the plurality of states is selected based on at least one of storage, memory or processing resources of one or more nodes of a second subset of the plurality of nodes. ## Claims ### Claim 1 A method comprising: hosting at least a portion of a virtual machine on at least one node of a first subset of a plurality of nodes of a secondary storage system, wherein the secondary storage system stores data associated with one or more primary storages, and the virtual machine comprises a plurality of portions that are distributed between the plurality of nodes of the secondary storage system and the virtual machine is configured into a first state of a plurality of states, and wherein, in the first state of the plurality of states, the plurality of portions of the virtual machine are distributed between the first subset of the plurality of nodes of the secondary storage system and each of the plurality of nodes of the secondary storage system included in the first subset of the plurality of nodes of the secondary storage system is configured to store a portion of the virtual machine in a corresponding storage device of the corresponding node; selecting based on at least one of storage, memory, and processing resources of one or more nodes of a second subset of the plurality of nodes of the secondary storage system, a node from the second subset of the plurality of nodes of the secondary storage system to host the virtual machine in a second state of the plurality of states, wherein, in the second state of the plurality of states, the plurality of portions of the virtual machine are distributed between more than one node among the second subset of the plurality of nodes of the secondary storage system; moving to nodes of the second subset of the plurality of nodes of the secondary storage system the plurality of portions of the virtual machine hosted on the first subset of the plurality of nodes of the secondary storage system, wherein the selected node is included in the nodes of the second subset of the plurality of nodes of the secondary storage system and is configured to host at least one portion of the plurality of portions of the virtual machine; and running the virtual machine in the second state of the plurality of states on the selected node. ### Claim 2 (depends on 1) The method of claim 1, wherein the portion of the virtual machine hosted on the at least one node of the first subset of the plurality of nodes of the secondary storage system includes an executable portion and a data portion. ### Claim 3 (depends on 2) The method of claim 2, wherein moving to nodes of the second subset of the plurality of nodes of the secondary storage system the plurality of portions of the virtual machine hosted on the first subset of the plurality of nodes of the secondary storage system includes moving the executable portion to a memory of the selected node. ### Claim 4 (depends on 2) The method of claim 2, further comprising moving the data portion to one of the nodes of the second subset of the plurality of nodes of the secondary storage system. ### Claim 5 (depends on 4) The method of claim 4, wherein the data portion is moved to the selected node. ### Claim 6 (depends on 1) The method of claim 1, wherein the second subset of the plurality of nodes of the secondary storage system includes the at least one node of the first subset of the plurality of nodes of the secondary storage system. ### Claim 7 (depends on 1) The method of claim 1, wherein the second subset of the plurality of nodes of the secondary storage system includes none of the at least one node of the first subset of the plurality of nodes of the secondary storage system. ### Claim 8 (depends on 1) The method of claim 1, wherein the plurality of portions of the virtual machine are moved based on one or more trigger conditions. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more trigger conditions include at least one of an out-of-memory condition, a change in operational condition, a failure or predicted failure of a node, an insertion or predicted insertion of a new node into the secondary storage system, load balancing between the plurality of nodes of the secondary storage system, or a round robin assignment of executables between the plurality of nodes of the secondary storage system. ### Claim 10 (depends on 1) The method of claim 1, wherein the node from the second subset of the plurality of nodes of the secondary storage system is selected based on an affinity between the virtual machine and the selected node. ### Claim 11 (depends on 1) The method of claim 1, wherein the node from the second subset of the plurality of nodes of the secondary storage system is selected based on an affinity between an application hosted by the virtual machine and one or more applications hosted by one or more virtual machines hosted on the selected node. ### Claim 12 (depends on 1) The method of claim 1, further comprising: determining that the virtual machine is offline, wherein a memory of the at least one node of the first subset of the plurality of nodes of the secondary storage system stores one or more stale writes associated with the offline virtual machine; and cloning a volume associated with the offline virtual machine. ### Claim 13 (depends on 12) The method of claim 12, further comprising associating the cloned volume with the second subset of the plurality of nodes of the secondary storage system. ### Claim 14 (depends on 13) The method of claim 13, further comprising removing the volume associated with the offline virtual machine. ### Claim 15 (depends on 1) The method of claim 1, wherein the selected node is configured to: receive at least one change to the virtual machine as part of one or more test and development operations; and store new data resulting from the at least one change as a portion of the virtual machine in at least one node of the second subset of the plurality of nodes of the secondary storage system. ### Claim 16 (depends on 1) The method of claim 1, further comprising: determining that at least one of memory and processor of the selected node has met a related condition; and in response to determining that the at least one of the memory and the processor of the selected node has met the related condition, selecting the node from the second subset of the plurality of nodes of the secondary storage system to run an executable portion for the virtual machine in the second state of the plurality of states. ### Claim 17 (depends on 16) The method of claim 16, wherein the related condition is a decrease in an available memory or processing resources on the selected node. ### Claim 18 (depends on 1) The method of claim 1, further comprising: determining that the virtual machine has stopped running on the selected node; selecting another node from the second subset of the plurality of nodes of the secondary storage system to run the virtual machine based on at least one of storage, memory, and processing resources across other nodes of the second subset of the plurality of nodes of the secondary storage system; and restarting the virtual machine on the selected another node. ### Claim 19 A system comprising: a processor configured to: host at least a portion of a virtual machine on at least one node of a first subset of a plurality of nodes of a secondary storage system, wherein the secondary storage system stores data associated with one or more primary storages, and the virtual machine comprises a plurality of portions that are distributed between the plurality of nodes of the secondary storage system and the virtual machine is configured into a first state of a plurality of states, and wherein, in the first state of the plurality of states, the plurality of portions of the virtual machine are distributed between the first subset of the plurality of nodes of the secondary storage system and each of the plurality of nodes of the secondary storage system included in the first subset of the plurality of nodes of the secondary storage system is configured to store a portion of the virtual machine in a corresponding storage device of the corresponding node; select based on at least one of storage, memory, and processing resources of one or more nodes of a second subset of the plurality of nodes of the secondary storage system, a node from the second subset of the plurality of nodes of the secondary storage system to host the virtual machine in a second state of the plurality of states, wherein, in the second state of the plurality of states, the plurality of portions of the virtual machine are distributed between more than one node among the second subset of the plurality of nodes of the secondary storage system; move to nodes of the second subset of the plurality of nodes of the secondary storage system the plurality of portions of the virtual machine hosted on the first subset of the plurality of nodes of the secondary storage system, wherein the selected node is included in the nodes of the second subset of the plurality of nodes of the secondary storage system and is configured to host at least one portion of the plurality of portions of the virtual machine; and run the virtual machine in the second state of the plurality of states on the selected node; and a memory coupled to the processor and configured to provide the processor with instructions. ### Claim 20 A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions to: host at least a portion of a virtual machine on at least one node of a first subset of a plurality of nodes of a secondary storage system, wherein the secondary storage system stores data associated with one or more primary storages, and the virtual machine comprises a plurality of portions that are distributed between the plurality of nodes of the secondary storage system and the virtual machine is configured into a first state of a plurality of states, and wherein, in the first state of the plurality of states, the plurality of portions of the virtual machine are distributed between the first subset of the plurality of nodes of the secondary storage system and each of the plurality of nodes of the secondary storage system included in the first subset of the plurality of nodes of the secondary storage system is configured to store a portion of the virtual machine in a corresponding storage device of the corresponding node; select based on at least one of storage, memory, and processing resources of one or more nodes of a second subset of the plurality of nodes of the secondary storage system, a node from the second subset of the plurality of nodes of the secondary storage system to host the virtual machine in a second state of the plurality of states, wherein, in the second state of the plurality of states, the plurality of portions of the virtual machine are distributed between more than one node among the second subset of the plurality of nodes of the secondary storage system; move to nodes of the second subset of the plurality of nodes of the secondary storage system the plurality of portions of the virtual machine hosted on the first subset of the plurality of nodes of the secondary storage system, wherein the selected node is included in the nodes of the second subset of the plurality of nodes of the secondary storage system and is configured to host at least one portion of the plurality of portions of the virtual machine; and run the virtual machine in the second state of the plurality of states on the selected node.