# US 10,372,498 B2 · Dynamic virtual CPU core allocation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,498 B2
- Document ID: US10372498B2
- Status: Granted
- Kind: B2
- Application number: 15/276,252
- Filed: 2016-09-26
- Published: 2018-03-29
- Granted: 2019-08-06
- Earliest claimed: 2016-09-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Alexander W. Min
- Web page: https://patentgrep.com/patent/US10372498B2

## CPC

- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 1/3206: Monitoring of events, devices or parameters that trigger a change in power modality
- G06F 1/3243: {Power saving in microcontroller unit}
- G06F 1/3275: {Power saving in memory, e.g. RAM, cache}
- G06F 1/3287: by switching off individual functional units in the computer system
- G06F 1/329: by task scheduling
- G06F 9/45533: {Hypervisors; Virtual machine monitors}
- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06F 11/301: {where the computing system is a virtual computing platform, e.g. logically partitioned systems (virtual machines G06F9/45533; logical partitioning of resources G06F9/5077)}
- G06F 11/3037: {where the computing system component is a memory, e.g. virtual memory, cache (accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00)}
- G06F 11/3051: {Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs (verification or detection of system hardware configuration G06F11/2247)}
- G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)}
- G06F 12/0864: using pseudo-associative means, e.g. set-associative or hashing
- G06F 11/3024: {where the computing system component is a central processing unit [CPU]}
- G06F 11/3062: {where the monitored property is the power consumption (power management in a computing system G06F1/3203)}
- G06F 11/3096: {wherein the means or processing minimize the use of computing system or of computing system component resources, e.g. non-intrusive monitoring which minimizes the probe effect: sniffing, intercepting, indirectly deriving the monitored data from other directly available data}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45583: {Memory management, e.g. access or allocation}
- G06F 2009/45591: {Monitoring or debugging support}
- G06F 2212/6032: Way prediction in set-associative cache
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

The systems and methods described herein provide power management circuitry that factors one or more cache parameters (e.g., cache utilization) of an application or VM when determining pCPU-vCPU core remapping. By considering a more robust mix of both processor and cache memory related parameters, system performance and stability are increased by improving CPU and cache utilization and efficiency while reducing cache related issues such as collisions and/or pollution.

## Claims

### Claim 1

A performance monitoring system for virtual machines, comprising:
performance monitoring circuitry to determine performance data indicative of at least one parameter logically associated with each of a plurality of virtual machines;
resource monitoring circuitry to determine resource data indicative of:
at least one physical core parameter logically associated with each of a plurality of physical cores; and
at least one physical cache parameter logically associated with each of a plurality of physical caches, each of the physical caches communicably coupled to a respective one of the plurality of physical cores; and
power management circuitry to:
determine whether at least two of the plurality of virtual machines can be consolidated in a first physical core included in the plurality of physical cores based on the at least one physical core parameter received from the resource monitoring circuitry:
responsive to a successful determination that the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter, determine whether the at least two of the plurality of virtual machines can be consolidated in a first physical cache coupled to the first physical core based on the at least one physical cache parameter received from the resource monitoring circuitry, wherein the at least one physical cache parameter includes one or more of cache turnover or cache access time; and
consolidate the at least two virtual machines on the first physical core responsive to a successful determination that:
the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter; and
the at least two of the plurality of virtual machines can be consolidated in a first physical cache based on the at least one physical cache parameter.

### Claim 2 (depends on 1)

The performance monitoring system of claim 1, further comprising:
performance determination circuitry communicably coupled to the performance monitoring circuitry and to the power management circuitry, the performance determination circuitry to:
determine a value indicative of a performance parameter for each of the plurality of virtual machines based at least in part on the determined performance data; and
communicate the determined value indicative of the performance parameter for each of the plurality of virtual machines to the power management circuitry.

### Claim 3 (depends on 2)

The performance monitoring system of claim 2 wherein the performance determination circuitry to:
determine a value indicative of a performance parameter for each of the plurality of virtual machines based at least in part on virtual CPU utilization data logically associated with each of at least some of the plurality of virtual machines.

### Claim 4 (depends on 1)

The performance monitoring system of claim 1, further comprising:
resource allocation determination circuitry communicably coupled to the resource monitoring circuitry and to the power management circuitry, the resource allocation determination circuitry to:
determine a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource data; and
communicate the determined value indicative of the physical resource allocation for each of the plurality of virtual machines to the power management circuitry.

### Claim 5 (depends on 4)

The performance monitoring system of claim 4, further comprising at least one storage device communicably coupled to the resource allocation determination circuitry, the at least one storage device having data representative of one or more resource allocation policies stored thereon.

### Claim 6 (depends on 5)

The performance monitoring system of claim 5 wherein the resource allocation determination circuitry determines the value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the one or more resource allocation policies stored on the at least one storage device.

### Claim 7 (depends on 1)

The performance monitoring system of claim 1 wherein the performance monitoring circuitry determines performance data that includes virtual CPU utilization data logically associated with at least some of the plurality of virtual machines.

### Claim 8 (depends on 1)

The performance monitoring system of claim 1 wherein the resource monitoring circuitry generates resource data indicative of physical core utilization data logically associated with at least some of the plurality of physical cores and physical cache utilization data logically associated with at least some of the plurality of physical caches.

### Claim 9 (depends on 1)

The performance monitoring system of claim 1 wherein the resource monitoring circuitry generates resource data indicative of a memory utilization trend logically associated with at least some of the plurality of physical caches.

### Claim 10

A performance monitoring method to consolidate a plurality of virtual machines on a plurality of physical cores, each communicably coupled to a respective one of a corresponding plurality of physical caches, the method comprising:
determining, by performance monitoring circuitry, performance data indicative of at least one parameter logically associated with each of a plurality of virtual machines;
determining, by resource monitoring circuitry, resource data indicative of:
at least one physical core parameter logically associated with each of the plurality of physical cores; and
at least physical cache parameter logically associated with each of the plurality of physical caches, each of the physical caches communicably coupled to a respective one of the plurality of physical cores; and
determining, by power management circuitry, whether at least two of the plurality of virtual machines can be consolidated in a first physical core included in the plurality of physical cores based on the at least one physical core parameter received from the resource monitoring circuitry:
responsive to a successful determination that the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter, determining, by the power management circuitry, whether the at least two of the plurality of virtual machines can be consolidated in a first physical cache coupled to the first physical core based on the at least one physical cache parameter received from the resource monitoring circuitry, wherein the at least one physical cache parameter includes one or more of cache turnover or cache access time; and
consolidating, by the power management circuitry, the at least two virtual machines on the first physical core responsive to a successful determination that:
the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter; and
the at least two of the plurality of virtual machines can be consolidated in a first physical cache based on the at least one physical cache parameter.

### Claim 11 (depends on 10)

The performance monitoring method of claim 10 wherein determining performance data indicative of at least one parameter logically associated with each of a plurality of virtual machines comprises:
determining, by performance determination circuitry communicably coupled to the performance monitoring circuitry, a value indicative of a performance parameter for each of the plurality of virtual machines based at least in part on the determined performance data; and
communicating the determined value indicative of the performance parameter for each of the plurality of virtual machines to the power management circuitry.

### Claim 12 (depends on 11)

The performance monitoring method of claim 11 wherein determining a value indicative of a performance parameter for each of the plurality of virtual machines comprises:
determining, by the performance monitoring circuitry, virtual CPU utilization data logically associated with each of at least some of the plurality of virtual machines.

### Claim 13 (depends on 10)

The performance monitoring method of claim 10, further comprising:
determining, by resource allocation determination circuitry communicably coupled to the resource monitoring circuitry, a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource data; and
communicating the determined value indicative of the physical resource allocation for each of the plurality of virtual machines to the power management circuitry.

### Claim 14 (depends on 13)

The performance monitoring method of claim 13, further comprising:
retrieving, by the resource allocation determination circuitry from at least one communicably coupled storage device, data representative of one or more resource allocation policies.

### Claim 15 (depends on 14)

The performance monitoring method of claim 14 wherein determining a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource allocation data comprises:
determining, by the resource allocation determination circuitry, a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource allocation data and based at least in part on the one or more resource allocation policies.

### Claim 16 (depends on 10)

The performance monitoring method of claim 10 wherein determining resource data indicative of at least a physical cache parameter logically associated with each of a plurality of physical caches comprises:
determining, by resource allocation determination circuitry communicably coupled to the resource monitoring circuitry and to the power management circuitry, a value indicative of a physical cache utilization trend logically associated with at least some of the plurality of physical caches; and
communicating the determined value indicative of the physical cache utilization trend to the power management circuitry.

### Claim 17 (depends on 10)

The performance monitoring method of claim 10 wherein determining resource data indicative of at least at least one physical core parameter logically associated with each of a plurality of physical cores comprises:
determining, by resource allocation determination circuitry communicably coupled to the resource monitoring circuitry and to the power management circuitry, a value indicative of a physical core utilization logically associated with at least some of the plurality of physical cores; and
communicating the determined value indicative of the physical core utilization to the power management circuitry.

### Claim 18 (depends on 10)

The performance monitoring method of claim 10 wherein determining resource data indicative of at least at least one physical core parameter logically associated with each of a plurality of physical cores comprises:
determining, by resource allocation determination circuitry communicably coupled to the resource monitoring circuitry and to the power management circuitry, a value indicative of a physical core utilization trend logically associated with at least some of the plurality of physical cores; and
communicating the determined value indicative of the physical core utilization trend to the power management circuitry.

### Claim 19

A performance monitoring system to consolidate a plurality of virtual machines on a plurality of physical cores, each communicably coupled to a respective one of a corresponding plurality of physical caches, the system comprising:
a means for determining performance data indicative of at least one parameter logically associated with each of a plurality of virtual machines;
a means for determining resource data indicative of:
at least one physical core parameter logically associated with each of the plurality of physical cores; and
at least physical cache parameter logically associated with each of the plurality of physical caches, each of the physical caches communicably coupled to a respective one of the plurality of physical cores; and
a means for determining whether at least two of the plurality of virtual machines can be consolidated in a first physical core included in the plurality of physical cores based on the at least one physical core parameter received from the resource monitoring circuitry: a means for determining, by the power management circuitry, whether the at least two of the plurality of virtual machines can be consolidated in a first physical cache coupled to the first physical core based on the at least one physical cache parameter received from the resource monitoring circuitry responsive to a successful determination that the at least two virtual machines can be consolidated in the first physical core based on the at least one physical cache parameter, wherein the at least one physical cache parameter includes one or more of cache turnover or cache access time; and
a means for consolidating, by the power management circuitry, the at least two virtual machines on the first physical core responsive to a successful determination that:
the at least two virtual machines can be consolidated in the first physical core based on the at least one physical cache parameter; and
the at least two of the plurality of virtual machines can be consolidated in a first physical cache based on the at least one physical cache parameter.

### Claim 20 (depends on 19)

The performance monitoring system of claim 19, further comprising:
a means for determining, by resource allocation determination circuitry communicably coupled to the resource monitoring circuitry, a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource data; and
a means for communicating the determined value indicative of the physical resource allocation for each of the plurality of virtual machines to the power management circuitry.

### Claim 21 (depends on 20)

The performance monitoring system of claim 20, further comprising:
a means for retrieving, by the resource allocation determination circuitry from at least one communicably coupled storage device, data representative of one or more resource allocation policies.

### Claim 22

A storage device that includes one or more machine readable instruction sets, that when executed by circuitry, cause the circuitry to transform to a virtual machine manager, the virtual machine manager to:
cause performance monitoring circuitry to determine performance data indicative of at least one parameter logically associated with each of a plurality of virtual machines;
cause resource monitoring circuitry to determine resource data indicative of:
at least one physical core parameter logically associated with each of a plurality of physical cores; and
at least physical cache parameter logically associated with each of a plurality of physical caches, each of the physical caches communicably coupled to a respective one of the plurality of physical cores; and
cause power management circuitry to determine whether at least two of the plurality of virtual machines can be consolidated in a first physical core included in the plurality of physical cores based on the at least one physical core parameter received from the resource monitoring circuitry:
responsive to a successful determination that the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter, determine whether the at least two of the plurality of virtual machines can be consolidated in a first physical cache coupled to the first physical core based on the at least one physical cache parameter received from the resource monitoring circuitry, wherein the at least one physical cache parameter includes one or more of cache turnover or cache access time; and
consolidate the at least two virtual machines on the first physical core responsive to a successful determination that:
the at least two virtual machines can be consolidated in the first physical core based on the at least one physical core parameter; and
the at least two of the plurality of virtual machines can be consolidated in a first physical cache based on the at least one physical cache parameter.

### Claim 23 (depends on 22)

The storage device of claim 22, wherein the machine readable instructions cause the virtual machine manager to further:
cause resource allocation determination circuitry communicably coupled to the resource monitoring circuitry to determine a value indicative of a physical resource allocation for each of the plurality of virtual machines based at least in part on the determined resource data; and
cause the resource allocation determination circuitry to communicate to the power management circuitry the determined value indicative of the physical resource allocation for each of the plurality of virtual machines.

### Claim 24 (depends on 23)

The storage device of claim 23 wherein the machine readable instructions cause the virtual machine manager to further:
cause the resource allocation determination circuitry to retrieve from at least one communicably coupled storage device, data representative of one or more resource allocation policies.