# US 2021/0232399 A1 · Method, System, and Computer Program Product for Dynamically Assigning an Inference Request to a CPU or GPU

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232399 A1
- Document ID: US20210232399A1
- Status: Published application
- Kind: A1
- Application number: 16/750,100
- Filed: 2020-01-23
- Published: 2021-07-29
- Earliest claimed: 2020-01-23
- Assignees: Visa International Service Association
- Applicants: Visa International Service Association
- Inventors: Hao Yang; Biswajit Das; Yu Gu; Peter Walker; Igor Karpenko; Robert Brian Christensen
- Web page: https://patentgrep.com/patent/US20210232399A1

## CPC

- G06F 9/5005: {to service a request}
- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 9/5044: {considering hardware capabilities}
- G06F 9/5055: {considering software capabilities, i.e. software resources associated or available to the machine}
- G06F 9/5088: {involving task migration}
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution}
- G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)}
- G06N 3/045: Combinations of networks
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/025: Extracting rules from data
- G06N 5/04: Inference or reasoning models
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

A method for dynamically assigning an inference request is disclosed. A method for dynamically assigning an inference request may include determining at least one model to process an inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU), obtaining, with at least one processor, profile information of the at least one model, the profile information including measured characteristics of the at least one model, dynamically determining a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information, and routing, with at least one processor, the inference request to the selected computing platform. A system and computer program product are also disclosed.

## Claims

### Claim 1

A computer-implemented method, comprising:
determining, with at least one processor based on an inference request, at least one model to process the inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU);
obtaining, with at least one processor, profile information of the at least one model, the profile information including measured characteristics of the at least one model;
dynamically determining, with at least one processor, a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information; and
routing, with at least one processor, the inference request to the selected computing platform.

### Claim 2 (depends on 1)

The method of claim 1, further comprising determining the optimized objective by balancing throughput associated with processing the at least one model based on at least one of the following: a latency target, a power budget, a server node count, or any combination thereof.

### Claim 3 (depends on 1)

The method of claim 1, wherein the profile information is obtained based on the at least one model and comprises a runtime deployment model trained for at least one of the plurality of computing platforms.

### Claim 4 (depends on 1)

The method of claim 1, wherein the measured characteristics of the at least one model comprise at least one of the following: a throughput, a size, a power efficiency, an accuracy level, a model type, or any combination thereof.

### Claim 5 (depends on 1)

The method of claim 1, wherein the at least one GPU is configured in a blocking state until a job size is maximized to start inferences.

### Claim 6 (depends on 1)

The method of claim 1, wherein the status of the computing platform is determined based on at least one of the following: CPU utilization, GPU utilization, RAM utilization, inference throughput, number of jobs, latency data, historical completion data, CPU type, GPU type, a size of the at least one model, a memory footprint of the at least one model, or any combination thereof.

### Claim 7 (depends on 1)

The method of claim 1, further comprising at least one of:
sending the inference request to the at least one CPU if a throughput is less than a threshold;
sending the inference request to the at least one GPU if the at least one GPU is accumulating inference jobs and the inference request can fit into a batch based on the measured characteristics;
sending the inference request to the at least one CPU if the at least one GPU is unavailable to process the at least one model; or
creating a new job for the at least one GPU.

### Claim 8

A system, comprising:
at least one processor programmed or configured to:
determine, based on an inference request, at least one model to process the inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU);
obtain profile information of the at least one model, the profile information including measured characteristics of the at least one model;
dynamically determine a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information; and
route the inference request to the selected computing platform.

### Claim 9 (depends on 8)

The system of claim 8, wherein the at least one processor is further programmed or configured to determine the optimized objective by balancing throughput associated with processing the at least one model based on at least one of the following: a latency target, a power budget, a server node count, or any combination thereof.

### Claim 10 (depends on 8)

The system of claim 8, wherein the profile information is obtained based on the at least one model and comprises a runtime deployment model trained for at least one of the plurality of computing platforms.

### Claim 11 (depends on 8)

The system of claim 8, wherein the measured characteristics of the at least one model comprise at least one of the following: a throughput, a size, a power efficiency, an accuracy level, a model type, or any combination thereof.

### Claim 12 (depends on 8)

The system of claim 8, wherein the at least one GPU is configured in a blocking state until a job size is maximized to start inferences.

### Claim 13 (depends on 8)

The system of claim 8, wherein the status of the computing platform is determined based on at least one of the following: CPU utilization, GPU utilization, RAM utilization, inference throughput, number of jobs, latency data, historical completion data, CPU type, GPU type, a size of the at least one model, a memory footprint of the at least one model, or any combination thereof.

### Claim 14 (depends on 8)

The system of claim 8, wherein the at least one processor is further programmed or configured to:
send the inference request to the at least one CPU if a throughput is less than a threshold;
send the inference request to the at least one GPU if the at least one GPU is accumulating inference jobs and the inference request can fit into a batch based on the measured characteristics;
send the inference request to the at least one CPU if the at least one GPU is unavailable to process the at least one model; or
create a new job for the at least one GPU.

### Claim 15

A computer program product, comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
determine, based on an inference request, at least one model to process the inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU);
obtain profile information of the at least one model, the profile information including measured characteristics of the at least one model;
dynamically determine a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information; and
route the inference request to the selected computing platform.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the at least one non-transitory computer-readable medium includes further program instructions that, when executed by at least one processor, cause the at least one processor to determine the optimized objective by balancing throughput associated with processing the at least one model based on at least one of the following: a latency target, a power budget, a server node count, or any combination thereof.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the profile information is obtained based on the at least one model and comprises a runtime deployment model trained for at least one of the plurality of computing platforms.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the measured characteristics of the at least one model comprise at least one of the following: a throughput, a size, a power efficiency, an accuracy level, a model type, or any combination thereof.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the status of the computing platform is determined based on at least one of the following: CPU utilization, GPU utilization, RAM utilization, inference throughput, number of jobs, latency data, historical completion data, CPU type, GPU type, a size of the at least one model, a memory footprint of the at least one model, or any combination thereof.

### Claim 20 (depends on 15)

The computer program product of claim 15, wherein the at least one non-transitory computer-readable medium includes further program instructions that, when executed by at least one processor, cause the at least one processor to:
send the inference request to the at least one CPU if a throughput is less than a threshold;
send the inference request to the at least one GPU if the at least one GPU is accumulating inference jobs and the inference request can fit into a batch based on the measured characteristics;
send the inference request to the at least one CPU if the at least one GPU is unavailable to process the at least one model; or
create a new job for the at least one GPU.
