# US 2020/0004596 A1 · Attached accelerator based inference service

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0004596 A1
- Document ID: US20200004596A1
- Status: Published application
- Kind: A1
- Application number: 16/020,776
- Filed: 2018-06-27
- Published: 2020-01-02
- Earliest claimed: 2018-06-27
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Sudipta SENGUPTA; Poorna Chand Srinivas PERUMALLA; Dominic Rajeev DIVAKARUNI; Nafea BSHARA; Leo Parker DIRAC; Bratin SAHA; Matthew James WOOD; Andrea OLGIATI; Swaminathan SIVASUBRAMANIAN
- Web page: https://patentgrep.com/patent/US20200004596A1

## CPC

- G06F 9/5022: {Mechanisms to release resources}
- G06N 20/00: Machine learning
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 9/5044: {considering hardware capabilities}
- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06N 5/04: Inference or reasoning models
- G06F 9/5055: {considering software capabilities, i.e. software resources associated or available to the machine}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45583: {Memory management, e.g. access or allocation}
- G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances}
- G06F 2209/509: Offload
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Implementations detailed herein include description of a computer-implemented method. In an implementation, the method at least includes receiving an application instance configuration, an application of the application instance to utilize a portion of an attached accelerator during execution of a machine learning model and the application instance configuration including: an indication of the central processing unit (CPU) capability to be used, an arithmetic precision of the machine learning model to be used, an indication of the accelerator capability to be used, a storage location of the application, and an indication of an amount of random access memory to use.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving, in a multi-tenant web services provider, an application instance configuration, an application of the application instance to utilize a portion of an attached graphics processing unit (GPU) during execution of a machine learning model and the application instance configuration including:
an indication of the central processing unit (CPU) capability to be used,
an arithmetic precision of the machine learning model to be used,
an indication of the GPU capability to be used,
a storage location of the application, and
an indication of an amount of random access memory to use;
provisioning the application instance and the portion of the GPU attached to the application instance, wherein the application instance is implemented using a physical compute instance in a first instance location, wherein the portion of the GPU is implemented using a physical GPU in the second location, and wherein the physical GPU is accessible to the physical compute instance over a network;
attaching the portion of the GPU to the application instance;
loading the machine learning model onto the attached portion of the GPU; and
performing inference using the loaded machine learning model of the application using the portion of the GPU on the attached GPU.

### Claim 2 (depends on 1)

The method of claim 1, wherein the application instance and the portion of the attached GPU are within different virtual networks.

### Claim 3 (depends on 1)

The method of claim 1, wherein the machine learning model includes a description of a computation graph for inference and weights obtained from training.

### Claim 4 (depends on 1)

The method of claim 1, wherein the machine learning model is in TensorFlow, MXNet, or ONNX format.

### Claim 5

A computer-implemented method, comprising:
receiving, in a multi-tenant web services provider, an application instance configuration, an application of the application instance to utilize a portion of an attached accelerator during execution of a machine learning model and the application instance configuration including:
an indication of the central processing unit (CPU) capability to be used,
an arithmetic precision of the machine learning model to be used,
an indication of the accelerator capability to be used,
a storage location of the application, and
an indication of an amount of random access memory to use;
provisioning the application instance and the portion of the accelerator attached to the application instance, wherein the application instance is implemented using a physical compute instance in a first instance location, wherein the portion of the accelerator is implemented using a physical accelerator in the second location, and wherein the physical accelerator is accessible to the physical compute instance over a network;
attaching the portion of the accelerator to the application instance;
loading the machine learning model onto the attached portion of the accelerator; and
performing inference using the loaded machine learning model of the application using the portion of the accelerator on the attached accelerator.

### Claim 6 (depends on 5)

The method of claim 5, wherein the machine learning model includes a description of a computation graph for inference and weights obtained from training.

### Claim 7 (depends on 5)

The method of claim 5, wherein the machine learning model is in a TensorFlow, MXNet, or ONNX format.

### Claim 8 (depends on 5)

The method of claim 5, wherein the application instance and the portion of the attached accelerator are within different virtual networks.

### Claim 9 (depends on 5)

The method of claim 5, wherein the accelerator is one of a plurality of accelerators of an accelerator appliance.

### Claim 10 (depends on 9)

The method of claim 9, wherein the accelerator appliance includes accelerators of different capabilities.

### Claim 11 (depends on 10)

The method of claim 10, wherein a central processing unit of the accelerator appliance is shared proportional to capabilities of the plurality of accelerators.

### Claim 12 (depends on 5)

The method of claim 5, further comprising:
deattaching the portion of an attached accelerator; and
migrating the machine learning model to a different portion of the attached accelerator.

### Claim 13 (depends on 5)

The method of claim 5, further comprising:
deattaching the portion of an attached accelerator; and
migrating the machine learning model to a portion of a different accelerator.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
prior performing inference using the loaded machine learning model of the application using the portion of the accelerator on the attached accelerator, determining an inference engine to use based on the loaded machine learning model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the inference engine is compatible with the version number of the machine learning model format.

### Claim 16

A system comprising:
storage to store an application, the application including a machine learning model; and
an elastic inference service implemented by a second one or more electronic devices, the elastic inference service including an application instance and an accelerator appliance, the elastic inference service to:
receive a configuration for the application instance, the application of the application instance to utilize a portion of an attached accelerator of the accelerator appliance during execution of the machine learning model and the application instance configuration including:
an indication of the central processing unit (CPU) capability to be used,
an arithmetic precision of the machine learning model to be used,
an indication of the accelerator capability to be used,
a storage location of the application, and
an indication of an amount of random access memory to use;
provision the application instance and the portion of the accelerator attached to the application instance, wherein the application instance is implemented using a physical compute instance in a first instance location, wherein the portion of the accelerator is implemented using a physical accelerator in the second location, and wherein the physical accelerator is accessible to the physical compute instance over a network;
attach the portion of the accelerator to the application instance;
load the machine learning model onto the attached portion of the accelerator; and
perform inference using the loaded machine learning model of the application using the portion of the accelerator on the attached accelerator.

### Claim 17 (depends on 16)

The system of claim 16, wherein the elastic inference service is to deattach the portion of an attached accelerator and migrate the machine learning model to a different portion of the attached accelerator.

### Claim 18 (depends on 16)

The system of claim 16, wherein the elastic inference service is to deattach the portion of an attached accelerator and migrate the machine learning model to a different accelerator.

### Claim 19 (depends on 16)

The system of claim 16, wherein the machine learning model includes a description of a computation graph for inference and weights obtained from training.

### Claim 20 (depends on 16)

The system of claim 16, wherein the machine learning model is in a TensorFlow, MXNet, or ONNX format.
