Virtual machine live migration with direct-attached non-volatile memory express device

#Hypervisor-based virtualization technologies allocate portions of a computer system's physical resources (e.g., processor cores and/or time, physical memory regions, storage resources, etc.) into separate partitions, and execute software within each of those partitions. Hypervisor-based virtualization technologies therefore facilitate creation of virtual machine (VM) guests that each executes guest software, such as an operating system (OS) and other software executing therein. A computer system that hosts VMs is commonly called a VM host or a VM host node. While hypervisor-based virtualization technologies can take a variety forms, many use an architecture comprising a hypervisor that has direct access to hardware and that operates in a separate execution environment than all other software in the system, a host partition that executes a host OS and host virtualization stack, and one or more guest partitions corresponding to VM guests. The host virtualization stack within the host partition manages guest partitions, and thus the hypervisor grants the host partition a greater level of access to the hypervisor, and to hardware resources, than it does to guest partitions.

#VM host nodes are frequently arranged into clusters of related VM host nodes, such as failover clusters. In these cluster environments, a given VM guest can generally be hosted at any VM host node in the cluster, and can be migrated among VM host nodes in the cluster (e.g., due to a VM host node failure, due to VM host node maintenance or upgrades, due to load balancing). One form of VM guest migration is live migration, in which a host virtualization stack transfers VM guest state (e.g., CPU register state, memory state) from a source VM host node to a target VM host node, and then transfers execution of the VM guest to the target VM host node in a transparent manner (e.g., such that a VM guest OS and workload continues executing substantially uninterrupted).

#VM guest storage is often backed by a file (or group of files) referred to as a virtual hard drive image. In these environments, live migration includes transferring this virtual hard drive image from a source VM host node to a target VM host node, or making the virtual hard drive image available to each of the source VM host node and the target VM host node via shared storage.

#In order to describe the manner in which the advantages and features of the systems and methods described herein can be obtained, a more particular description of the embodiments briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the systems and methods described herein, and are not therefore to be considered to be limiting of their scope, certain systems and methods will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

#FIG. 1 illustrates an example computer architecture that facilitates VM live migration with a direct-attached NVMe device;

#FIG. 2 illustrates an example of logic of a virtual NVMe controller;

#Prior techniques to enable a non-volatile memory express (NVMe) controller to be directly assigned to a virtual machine (VM) guest presented significant challenges to live migration of that VM guest from one VM host node to another VM host node. This is because the VM guest had access to a physical NVMe controller's administrative queues. Even though commands written to those administrative queues may have been intercepted and filtered by a host virtualization stack (e.g., based on intercepting accesses to doorbell registers), those administrative queues still exposed physical properties of the NVMe controller to the VM guest. With this configuration, a VM guest could not be live migrated to another VM host node unless that target node had an identical NVMe controller (e.g., identical vendor identical model, and identical firmware), because the VM guest would be exposed to different properties of the new NVMe controller at the target VM host node.

#The embodiments described herein overcome this challenge by providing a virtual NVMe controller at the host virtualization stack, and by exposing that virtual NVMe controller to a VM guest rather than exposing the underlying physical NVMe controller. This virtual NVMe controller inspects administrative commands submitted by a VM guest, and either emulates execution of the command (e.g., for commands that obtain state information), or relies on hardware-accelerated execution of the command by the underlying physical NVMe controller (e.g., for commands that create data queues). By providing virtual NVMe controllers at the host virtualization stacks of VM host nodes, a VM guest sees consistent NVMe controller properties regardless of which of those VM host nodes the VM guest is operated on, enabling the VM guest to be live migrated between different VM host nodes.

#Additionally, prior techniques intercepted accesses by a VM guest to a guest memory page mapped to the NVMe doorbell registers. However, NVMe doorbell registers for both administrative queues and data queues reside on the same memory page, and interception of accesses to doorbell registers thus adversely affected the performance of data queue submissions. Embodiments described herein overcome this challenge by intercepting accesses by a VM guest to memory page(s) mapped to NVMe administrative queues, rather than intercepting accesses by the VM guest to a memory page mapped to NVMe doorbell registers. Embodiments described herein also map NVMe data queues into the VM guest's memory space. Mapping NVMe data queues into a VM guest's memory space, combined with intercepting accesses to NVMe administrative queues rather than access to NVMe doorbell registers, enables the VM guest unrestricted access to those NVMe data queues with native (or near-native) performance characteristics.