# US 10,503,658 B2 · Page migration with varying granularity > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,658 B2 - Document ID: US10503658B2 - Status: Granted - Kind: B2 - Application number: 15/499,313 - Filed: 2017-04-27 - Published: 2018-11-01 - Granted: 2019-12-10 - Earliest claimed: 2017-04-27 - Assignees: Advanced Micro Devices, Inc. - Applicants: Advanced Micro Devices, Inc. - Inventors: Arkaprava Basu; Jee Ho Ryoo - Web page: https://patentgrep.com/patent/US10503658B2 ## CPC - G06F 12/1009: using page tables, e.g. page table structures - G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F 11/34: Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} - G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)} - G06F 12/1027: using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB] - G06F 2212/1024: Latency reduction - G06F 2212/205: Hybrid memory, e.g. using both volatile and non-volatile memory - G06F 2212/502: using adaptive policy - G06F 2212/601: Reconfiguration of cache memory - G06F 2212/65: Details of virtual memory and virtual address translation - G06F 2212/657: Virtual address space management - G06F 2212/683: Invalidation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present disclosure is directed to techniques for migrating data between heterogeneous memories in a computing system. More specifically, the techniques involve migrating data between a memory having better access characteristics (e.g., lower latency but greater capacity) and a memory having worse access characteristics (e.g., higher latency but lower capacity). Migrations occur with a variable migration granularity. A migration granularity specifies a number of memory pages, having virtual addresses that are contiguous in virtual address space, that are migrated in a single migration operation. A history-based technique that adjusts migration granularity based on the history of memory utilization by an application is provided. A profiling-based technique that adjusts migration granularity based on a profiling operation is also provided. ## Claims ### Claim 1 A method for migrating data between a first memory and a second memory, the method comprising: dynamically adjusting a migration granularity for an application based on characteristics of the application; identifying subject data for migration; identifying a portion of data to migrate based on the subject data and on the migration granularity, the migration granularity specifying the portion of data to migrate as a number of memory pages forming a contiguous range of virtual addresses and including the subject data; performing, for a translation lookaside buffer, one translation lookaside buffer shootdown; and migrating, in a single migration operation, the portion of data to migrate, from a source memory, in which the memory pages of the portion of data reside, to a destination memory, wherein migrating the portion of data comprises a move-in operation, and dynamically adjusting the migration granularity comprises: first tracking of move-in operations for a virtual address range having a coarser granularity than the migration granularity, and second tracking of move-in operations for the application, responsive to determining, based on the first tracking and the second tracking, that a condition is satisfied, upgrading the migration granularity. ### Claim 2 (depends on 1) The method of claim 1, wherein determining, based on the first tracking and the second tracking, that a condition is satisfied comprises: determining that a tracked number of move-in operations for the virtual address range is above a first threshold; and determining that a tracked number for move-in operations for the application is above a second threshold. ### Claim 3 (depends on 2) The method of claim 2, wherein: dynamically adjusting the migration granularity is performed according to a history-based technique; migrating the portion of data comprises a move-in operation; and dynamically adjusting the migration granularity according to a history-based technique comprises, responsive to determining that the move-in operation is to be performed: incrementing a counter for a virtual address range having a coarser granularity than a migration granularity for an application from which the memory access originated, if the counter for the virtual address range is above a first threshold, then incrementing a counter for the application and if the counter for the virtual address range is not above the first threshold, then maintaining the value of the counter for the virtual address range, and if the counter for the application is above a second threshold, then upgrading the migration granularity for the application, and if the counter for the application is not above the second threshold, then maintaining the migration granularity for the application. ### Claim 4 (depends on 3) The method of claim 3, further comprising: performing a move-out operation to move data from the destination memory to the source memory in order to make room for the memory pages of the portion of data; and dynamically adjusting the migration granularity based on a history-based technique further comprises responsive to determining that the move-out operation is to be performed: recording a number of memory address ranges of finer-than-current migration granularity for the application for which no access has been made since data for those memory address ranges were moved in, if the number of memory address ranges of finer-than-current migration granularity is greater than a third threshold, then incrementing a second counter for the application and if the number is not greater than the third threshold, then maintaining the value of the second counter for the application, and if the second counter is greater than a fourth threshold, then downgrading the migration granularity for the application, and if the second counter for the application is not above the fourth threshold, then maintaining the migration granularity for the application. ### Claim 5 (depends on 1) The method of claim 1, wherein the migration granularity is specified on a per-application basis. ### Claim 6 (depends on 1) The method of claim 1, wherein dynamically adjusting the migration granularity is performed according to a profiling technique. ### Claim 7 (depends on 6) The method of claim 6, wherein dynamically adjusting the migration granularity according to the profiling technique comprises: during a profiling period, causing an application to run with different test migration granularities; recording profiling data for the application with the different test migration granularities; and selecting a migration granularity based on the profiling data for the different test migration granularities. ### Claim 8 (depends on 7) The method of claim 7, wherein: the profiling data comprises data indicating a number of translation lookaside buffer shootdowns that occur for the application run with the different test migration granularities; and selecting the migration granularity comprises selecting a migration granularity, of the test migration granularities, for which the fewest translation lookaside buffer shootdowns occurred. ### Claim 9 (depends on 1) The method of claim 1, wherein the destination memory has lower latency and lower capacity than the source memory. ### Claim 10 A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to migrate data between a first memory and a second memory by: dynamically adjusting a migration granularity for an application based on characteristics of the application; identifying subject data for migration; identifying a portion of data to migrate based on the subject data and on the migration granularity, the migration granularity specifying the portion of data to migrate as a number of memory pages forming a contiguous range of virtual addresses and including the subject data; performing, for a translation lookaside buffer, one translation lookaside buffer shootdown; and migrating, in a single migration operation, the portion of data to migrate, from a source memory, in which the memory pages of the portion of data reside, to a destination memory, wherein migrating the portion of data comprises a move-in operation, and dynamically adjusting the migration granularity comprises: first tracking of move-in operations for a virtual address range having a coarser granularity than the migration granularity, and second tracking of move-in operations for the application, responsive to determining, based on the first tracking and the second tracking, that a first condition is satisfied, upgrading the migration granularity. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the destination memory has lower latency and lower capacity than the source memory. ### Claim 12 A computing system, comprising: a source memory; a destination memory; and an operating system configured to: dynamically adjust a migration granularity for an application based on characteristics of the application; identify subject data for migration; identify a portion of data to migrate based on the subject data and on the migration granularity, the migration granularity specifying the portion of data to migrate as a number of memory pages forming a contiguous range of virtual addresses and including the subject data; perform, for a translation lookaside buffer, one translation lookaside buffer shootdown; and migrate, in a single migration operation, the portion of data to migrate, from the source memory, in which the memory pages of the portion of data reside, to the destination memory, wherein migrating the portion of data comprises a move-in operation, and dynamically adjusting the migration granularity comprises: first tracking of move-in operations for a virtual address range having a coarser granularity than the migration granularity, and second tracking of move-in operations for the application, responsive to determining, based on the first tracking and the second tracking, that a condition is satisfied, upgrading the migration granularity. ### Claim 13 (depends on 12) The computing system of claim 12, wherein determining, based on the first tracking and the second tracking, that a condition is satisfied comprises: determining that a tracked number of move-in operations for the virtual address range is above a first threshold; and determining that a tracked number for move-in operations for the application is above a second threshold. ### Claim 14 (depends on 13) The computing system of claim 13, wherein: the operating system is configured to dynamically adjust the migration granularity according to a history-based technique; migrating the portion of data comprises a move-in operation; and the operating system is configured to dynamically adjust the migration granularity according to a history-based technique by, responsive to determining that the move-in operation is to be performed: incrementing a counter for a virtual address range having a coarser granularity than a migration granularity for an application from which the memory access originated, if the counter for the virtual address range is above a first threshold, then incrementing a counter for the application and if the counter for the virtual address range is not above the first threshold, then maintaining the value of the counter for the virtual address range, and if the counter for the application is above a second threshold, then upgrading the migration granularity for the application, and if the counter for the application is not above the second threshold, then maintaining the migration granularity for the application. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the operating system is further configured to: perform a move-out operation to move data from the destination memory to the source memory in order to make room for the memory pages of the portion of data; and dynamically adjust the migration granularity based on a history-based technique by, responsive to determining that the move-out operation is to be performed: recording a number of memory address ranges of finer-than-current migration granularity for the application for which no access has been made since data for those memory address ranges were moved in, if the number of memory address ranges of finer-than-current migration granularity is greater than a third threshold, then incrementing a second counter for the application and if the number is not greater than the third threshold, then maintaining the value of the second counter for the application, and if the second counter is greater than a fourth threshold, then downgrading the migration granularity for the application, and if the second counter for the application is not above the fourth threshold, then maintaining the migration granularity for the application. ### Claim 16 (depends on 12) The computing system of claim 12, wherein the migration granularity is specified on a per-application basis. ### Claim 17 (depends on 12) The computing system of claim 12, wherein the operating system is configured to dynamically adjust the migration granularity according to a profiling technique. ### Claim 18 (depends on 17) The computing system of claim 17, wherein the operating system is configured to dynamically adjust the migration granularity according to the profiling technique by: during a profiling period, causing an application to run with different test migration granularities; recording profiling data for the application with the different test migration granularities; and selecting a migration granularity based on the profiling data for the different test migration granularities. ### Claim 19 (depends on 18) The computing system of claim 18, wherein: the profiling data comprises data indicating a number of translation lookaside buffer shootdowns that occur for the application run with the different test migration granularities; and the operating system is configured to select the migration granularity by selecting a migration granularity, of the test migration granularities, for which the fewest translation lookaside buffer shootdowns occurred. ### Claim 20 (depends on 12) The computing system of claim 12, wherein the destination memory has lower latency and lower capacity than the source memory.