# US 10,768,965 B1 · Reducing copy operations for a virtual machine migration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,768,965 B1
- Document ID: US10768965B1
- Status: Granted
- Kind: B1
- Application number: 15/969,684
- Filed: 2018-05-02
- Granted: 2020-09-08
- Earliest claimed: 2018-05-02
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Adi Habusha; Ali Ghassan Saidi
- Web page: https://patentgrep.com/patent/US10768965B1

## CPC

- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 12/0292: {using tables or multilevel address translation means (G06F12/023 takes precedence; address translation in virtual memory systems G06F12/10)}
- G06F 12/1036: for multiple virtual address spaces, e.g. segmentation (G06F12/1045 takes precedence)
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45583: {Memory management, e.g. access or allocation}
- G06F 2212/1024: Latency reduction
- G06F 2212/1041: Resource optimization
- G06F 2212/152: Virtualized environment, e.g. logically partitioned system
- G06F 2212/50: Control mechanisms for virtual memory, cache or TLB
- G06F 2212/657: Virtual address space management
- G06F 2212/683: Invalidation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods are provided to reduce the number of redundant copy operations performed as part of a live migration of a virtual machine executing a guest. A hypervisor can queue the copy operations in a processing engine. While pre-copying for the live migration of the VM, the guest may continue to write to the pages. In one embodiment, the processing engine may clear a dirty page just before performing the copy operation of the modified page to a target device, thus extending the window of time to capture any future writes to that page.

## Claims

### Claim 1

A system comprising:
a host processor configured to execute a hypervisor, wherein the hypervisor is configured to:
provide information, associated with a page table entry (PTE) corresponding to a page of memory, to schedule a copy operation of the page for a live migration of a virtual machine (VM) running on the host processor, wherein the page is associated with a guest operating system (OS) executing within the VM, and wherein the PTE comprises a dirty page indicator to indicate that the page has been modified; and
a processing engine separate from the hypervisor executing on the host processor, the processing engine configured to:
receive the information associated with the PTE corresponding to the page, the information including an address of the PTE associated with the guest OS;
read the dirty page indicator in the PTE using the address to determine that the page has been modified;
schedule the page for the copy operation;
determine that the page has later been modified again;
execute an atomic operation to clear the dirty page indicator in the PTE after the later modification and before performing the copy operation; and
perform the copy operation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the address of the PTE includes a page descriptor address, and wherein the information associated with the PTE further includes a physical address associated with the guest, a virtual machine identifier (VIVID)) for the VM, and a page size.

### Claim 3 (depends on 2)

The system of claim 2, wherein the hypervisor is further configured to: invalidate an entry in a translation lookaside buffer (TLB) corresponding to the page using the VMID and the physical address associated with the guest prior to performing the copy operation.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processing engine is part of an input/output (I/O) adapter device coupled to the host processor via a host interface.

### Claim 5

A method, comprising:
receiving, by a processing engine separate from a hypervisor executing on a host processor, an address of a page table entry (PTE) in memory corresponding to a page associated with a guest, the address being received from the hypervisor to perform a copy operation subsequent to modification of the page by the guest, the hypervisor executing on the host processor in a host device, wherein the guest is executing within a virtual machine (VM) running on the host processor, the PTE comprising a dirty page indicator to indicate that the page has been modified;
determining that the page has been modified;
scheduling the page for the copy operation;
determining that the page has later been modified again;
executing, by the processing engine, an atomic operation to clear the dirty page indicator in the PTE after the later modification and before performing the copy operation; and
performing, by the processing engine, the copy operation.

### Claim 6 (depends on 5)

The method of claim 5 further comprising:
prior to performing the copy operation, invalidating a corresponding entry in a translation lookaside buffer (TLB) in the host processor using a virtual machine identifier (VMID) associated with the VM for a guest physical address corresponding to the page.

### Claim 7 (depends on 5)

The method of claim 5, wherein executing the atomic operation to clear the dirty page indicator comprises performing a read-modify-write of the dirty page indicator in the PTE in a single step.

### Claim 8 (depends on 5)

The method of claim 5, wherein a guest physical address for the modified page is logged in an entry in a page modification log (PML) subsequent to modification of the page by the guest.

### Claim 9 (depends on 5)

The method of claim 5, wherein the copy operation is performed as part of a live migration of the VM from the host device to a target device.

### Claim 10 (depends on 5)

The method of claim 5, wherein the copy operation comprises copying memory states associated with the VM from the host device to a target device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the processing engine includes a direct memory access (DMA) controller configured to perform the copy operation to the target device.

### Claim 12 (depends on 11)

The method of claim 11, wherein the DMA controller is configured to perform the copy operation asynchronous to scheduling of the copy operation by the hypervisor.

### Claim 13 (depends on 5)

The method of claim 5, wherein the dirty page indicator in the PTE was asserted by the host processor to indicate that the page has been modified.

### Claim 14 (depends on 5)

The method of claim 5, wherein the address of the PTE is a page descriptor address.

### Claim 15 (depends on 5)

The method of claim 5, wherein the processing engine includes a processor configured to execute instructions, and the processor does not execute the hypervisor.

### Claim 16

An apparatus comprising:
a memory configured to store information associated with a copy operation to be performed for a page, wherein the information is received from a hypervisor executing on a host processor, and wherein the page is associated with a guest executing within a virtual machine (VM) executing on the host processor;
a processor separate from the hypervisor executing on the host processor, the processor configured to:
determine that the page has been modified;
schedule the page for the copy operation;
determine that the page has later been modified again; and
clear a dirty page indicator in a page table entry (PTE) corresponding to the page using the information after the later modification and prior to performing the copy operation; and
a direct memory access (DMA) controller configured to perform the copy operation.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the information includes a guest physical address, a page descriptor address, a virtual machine identifier (VMID) for the VM, and a page size.

### Claim 18 (depends on 17)

The apparatus of claim 17, further comprising:
a translation lookaside buffer (TLB) manager configured to invalidate an entry in the TLB corresponding to the PTE using the VMID and the guest physical address.

### Claim 19 (depends on 17)

The apparatus of claim 17, wherein clearing the dirty page indicator in the PTE is performed by reading the PTE and clearing the dirty page indicator in the PTE using an atomic operation.

### Claim 20 (depends on 17)

The apparatus of claim 17, wherein the processor and the host processor are on a same die.