# US 11,444,845 B1 · Processing requests using compressed and complete machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,444,845 B1
- Document ID: US11444845B1
- Status: Granted
- Kind: B1
- Application number: 16/293,494
- Filed: 2019-03-05
- Granted: 2022-09-13
- Earliest claimed: 2019-03-05
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: David Leen
- Web page: https://patentgrep.com/patent/US11444845B1

## CPC

- H04L 41/16: using machine learning or artificial intelligence
- H04L 67/1004: Server selection for load balancing
- G06N 20/00: Machine learning
- H04L 41/5009: Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
- H04L 43/16: Threshold monitoring
- H04L 67/1008: based on parameters of servers, e.g. available memory or workload (monitoring of computer activity G06F11/30)
- H04L 67/60: Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A machine learning-based service processes requests using compressed and complete models to provide faster response times for servicing requests to process data. Initially, a host processes data using a compressed model that is stored in the host's memory and then switches to a larger, more accurate complete model after it is loaded into the host's memory. A host of the machine-learning based service may receive one or more requests to process data. In response, the host uses a compressed version of a model to begin processing the data. The host starts loading the complete version of the model into the host's memory. When the complete version of the model is loaded into memory, the host switches to process a remaining portion of the data using the complete version of the model.

## Claims

### Claim 1

A system, comprising:
one or more hosts of a provider network respectively comprising processors and memory to implement at least a portion of a machine learning-based service, individual ones of the hosts comprising a plurality of compressed models stored in the memory, wherein each of the compressed models is a compressed version of a corresponding complete version of a model, and wherein at least one of the hosts is configured to:
receive a request to process data, wherein the request indicates the data to be processed;
in response to reception, by the host of the provider network, of the request to process the data, process, by the host of the provider network, an initial portion of the data using one of the compressed versions of models that corresponds to a complete version of a model to generate one or more results and begin loading, by the host of the provider network, of the complete version of the model into the memory of the host of the provider network;
return the one or more results in response to the generation of the one or more results;
complete the loading of the complete version of the model into the memory of the host;
in response to a determination, by the host, that the complete version of the model is loaded into the memory of the host, process a remaining portion of the data indicated by the same request to process the data using the complete version of the model instead of the compressed version of the model that was used to process the initial portion of the data indicated by the same request to process the data to generate one or more other results,
wherein the complete version of the model is larger than the compressed version and produces more accurate results than the compressed version; and
return the one or more other results in response to the generation of the one or more other results.

### Claim 2 (depends on 1)

The system as recited in claim 1, further comprising a load balancer configured to distribute requests to respective hosts of a plurality of the hosts respectively comprising the plurality of compressed models, wherein the load balancer is further configured to perform:
receive the request to process the data;
determine that there are not any hosts of the plurality of hosts available to process the data using the complete version of the model;
in response to the determination that there are not any hosts available to process the data using the complete version of the model, select the host from among the plurality of hosts according to a load balancing algorithm; and
send the request to the host.

### Claim 3 (depends on 2)

The system as recited in claim 2, wherein the host is further configured to perform one or more of:
indicate to the load balancer that the complete model began loading into the memory of the host, or
indicate to the load balancer that the complete model has completed loading into the memory of the host.

### Claim 4 (depends on 1)

The system as recited in claim 1, wherein the request indicates a location of the data to be processed or include the data to be processed, and wherein to begin the loading of the complete version of the model into the memory of the host, the host is configured to perform:
determine that an amount of data to be processed using the compressed version of the model exceeds a threshold amount; and
in response, begin to load the complete version of the model into the memory of the host.

### Claim 5 (depends on 1)

The system as recited in claim 1, wherein the compressed model comprises:
a quantized version of the complete model, a fewer number of layers than the complete model, or
a different type of model than the complete model.

### Claim 6

A method, comprising:
performing, by a host of a machine learning-based service of a provider network:
receiving a request to process data, wherein the request indicates the data to be processed;
in response to receiving the request to process the data, processing an initial portion of the data using a compressed version of a model that corresponds to a complete version of the model to generate one or more results and begin loading, by the host of the provider network, of the complete version of the model into the memory of the host of the provider network;
returning the one or more results;
complete the loading of the complete version of the model into a memory of the host;
in response to a determination, by the host, that the complete version of the model is loaded into the memory of the host, process a remaining portion of the data indicated by the same request to process the data using the complete version of the model instead of the compressed version of the model that was used to process the initial portion of the data indicated by the same request to process data to generate one or more other results; and
returning the one or more other results.

### Claim 7 (depends on 6)

The method as recited in claim 6, wherein the host is one of a plurality of hosts of the machine learning-based service, each host comprising one or more compressed models stored in the memory, wherein each of the compressed models is a compressed version of a corresponding complete version of a model, and further comprising:
performing, by a load balancer:
receiving the request to process the data;
determining that there are not any hosts of the plurality of hosts available to process the data using the complete version of the model;
in response to determining that there are not any hosts available to process the data using the complete version of the model, selecting the host from among the plurality of hosts according to a load balancing algorithm; and
sending the request to the host.

### Claim 8 (depends on 7)

The method as recited in claim 7, further comprising:
performing, by the host:
indicating to the load balancer that the complete model began loading into the memory of the host, or
indicating to the load balancer that the complete model has completed loading into the memory of the host.

### Claim 9 (depends on 6)

The method as recited in claim 6, further comprising:
performing, by the load balancer:
receiving from a client one or more other requests to process other data, wherein the one or more other requests indicate another model to process the other data;
sending a compressed version of the other model to the client;
determining that a complete version of the other model is available to process requests at the host or another host of the machine learning-based service;
notifying the client that the complete version of the other model is available to process requests at the machine learning-based service;
receiving from the client one or more additional requests to process additional data, wherein the one or more additional requests indicate the other model to process the additional data; and
sending the one or more additional requests to the host or the other host to process the additional data using the complete version of the other model.

### Claim 10 (depends on 9)

The method as recited in claim 9, wherein determining that a complete version of the other model is available to process requests at the host or another host comprises:
determining, by the load balancer, that:
the complete version of the other model has completed loading into memory of the host or the other host, or
an overall load at the host or the other host is below a threshold amount, wherein the host or the other host comprises the other model in memory.

### Claim 11 (depends on 6)

The method as recited in claim 6, further comprising:
performing, by the host:
determining that an amount of data to be processed using the compressed version of the model exceeds a threshold amount; and
in response, begin loading the complete version of the model into the memory of the host.

### Claim 12 (depends on 6)

The method as recited in claim 6, wherein the determining that the amount of data to be processed using the compressed version exceeds a threshold amount is based on one or more of:
a size of the data to be processed using the compressed version,
a number of received requests to be processed using the compressed version,
a rate of received requests to be processed using the compressed version, a change in the rate of received requests to be processed using the compressed version, or
a predicted amount of data to be processed using the compressed version.

### Claim 13 (depends on 6)

The method as recited in claim 6, wherein the compressed model is one of a plurality of compressed models stored in the memory of the host, wherein each of the compressed models is a compressed version of a corresponding complete version of a model, and wherein processing the portion of the data using the compressed version of the model comprises:
determining that the compressed version corresponds to the model by the request;
in response to determining that the compressed version corresponds to the model by the request, selecting the compressed version from among the plurality of compressed models to process the data; and
processing the portion of the data using the compressed version of the model.

### Claim 14 (depends on 6)

The method as recited in claim 6, wherein the request indicates the model.

### Claim 15

One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
perform, by a host of a machine learning-based service of a provider network:
receiving a request to process data, wherein the request indicates the data to be processed;
in response to receiving the request to process the data, process an initial portion of the data using a compressed version of a model that corresponds to a complete version of the model to generate one or more results and begin loading, by the host of the provider network, of the complete version of the model into the memory of the host of the provider network;
return the one or more results;
complete the loading of the complete version of the model into a memory of the host;
in response to a determination, by the host, that the complete version of the model is loaded into the memory of the host, process a remaining portion of the data indicated by the same request to process the data using the complete version of the model instead of the compressed version of the model that was used to process the initial portion of the data indicated by the same request to process data to generate one or more other results; and
return the one or more other results.

### Claim 16 (depends on 15)

The one or more storage media as recited in claim 15, wherein to load the complete version of the model into the memory of the host, the program instructions when executed on or across the one or more processors:
determine that an amount of data to be processed using the compressed version of the model exceeds a threshold amount; and
in response, begin to load the complete version of the model into the memory of the host.

### Claim 17 (depends on 16)

The one or more storage media as recited in claim 16, wherein the determination that the amount of data to be processed using the compressed version exceeds a threshold amount is based on one or more of:
a size of the data to be processed using the compressed version,
a number of received requests to be processed using the compressed version,
a rate of received requests to be processed using the compressed version, a change in the rate of received requests to be processed using the compressed version, or
a predicted amount of data to be processed using the compressed version.

### Claim 18 (depends on 15)

The one or more storage media as recited in claim 15, wherein the host is one of a plurality of hosts of the machine learning-based service, each of the hosts respectively comprising one or more compressed models, and wherein the program instructions when executed on or across the one or more processors:
indicate to one or more other hosts of the plurality of hosts that the complete model began loading into the memory of the host, or
indicate to the one or more other hosts of the plurality of hosts that the complete model has completed loading into the memory of the host.

### Claim 19 (depends on 15)

The one or more storage media as recited in claim 15, wherein the host is one of a plurality of hosts of the machine learning-based service, each of the hosts respectively comprising one or more compressed models, and wherein the program instructions when executed on or across the one or more processors:
indicate to the load balancer that the complete model began loading into the memory of the host, or
indicate to the load balancer that the complete model has completed loading into the memory of the host.

### Claim 20 (depends on 15)

The one or more storage media as recited in claim 15, wherein the compressed model comprises:
a quantized version of the complete model, a fewer number of layers than the complete model, or
a different type of model than the complete model, wherein the compressed model is smaller than the complete model.