# US 2024/0176674 A1 · Hypertuning a machine learning model microservices configuration to optimize latency

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0176674 A1
- Document ID: US20240176674A1
- Status: Published application
- Kind: A1
- Application number: 18/072,700
- Filed: 2022-11-30
- Published: 2024-05-30
- Earliest claimed: 2022-11-30
- Assignees: Maplebear, Inc.(dba Instacart)
- Applicants: Maplebear Inc. (dba Instacart)
- Inventors: Sahil Khanna; Reza Sadri; Jacob Jensen
- Web page: https://patentgrep.com/patent/US20240176674A1

## CPC

- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06F 11/3442: {for planning or managing the needed capacity}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An online system facilitates various functions using machine learning model microservices. A tuning mechanism tunes various configuration parameters for each microservice that control allocation of computing resources and other configurations of physical and/or virtual machines that implement the microservices. Tuning may be performed in part by executing tests under various configurations and evaluating an objective function associated with the different configurations. Furthermore, parameters of the objective function may be set based on a trained learning model that learns baseline parameters and weights of the objective function based on historical data.

## Claims

### Claim 1

A method comprising, at a computer system comprising a processor and a computer-readable medium:
accessing an objective function associated with tuning a configuration of a microservice that performs machine learning functions associated with an online system;
accessing a configuration search space that defines a range of possible configurations of the microservice;
for each of a set of test configurations in the configuration search space, configuring respective allocations of resources associated with executing the microservice;
executing the microservice on respective sample requests under the respective allocations of resources while measuring respective operating parameters associated with the objective function;
evaluating the objective function using the respective operating parameters to generate respective objective values for each of the set of test configurations;
selecting a desired configuration from the set of test configurations based on the respective objective values; and
configuring the microservice in accordance with the desired configuration.

### Claim 2 (depends on 1)

The method of claim 1, wherein the objective function seeks to minimize a combination of latency and cost associated with resource utilization.

### Claim 3 (depends on 2)

The method of claim 2, wherein the resource utilization includes at least processor utilization and memory utilization.

### Claim 4 (depends on 1)

The method of claim 1, wherein the microservice is deployed in a containerized computing environment in which a set of single-threaded or multi-threaded processes execute in containers utilizing dynamically allocated processing and memory resources.

### Claim 5 (depends on 1)

The method of claim 1, wherein the configuration search space includes a two-dimensional array of variables and values for the variables, wherein the variables include one or more of: allocated processors, allocated memory, number of containers, number of processes, number of threads, and machine learning model compression.

### Claim 6 (depends on 1)

The method of claim 1, wherein the objective function comprises respective baseline values and respective weights associated with each of the operating parameters, and wherein evaluating the objective function is based on the respective operating parameters, the respective baseline values, and the respective weights.

### Claim 7 (depends on 6)

The method of claim 6, wherein the respective weights and the respective baseline values are obtained from a trained model that optimizes the respective weights and the respective baseline values based on historical data.

### Claim 8 (depends on 1)

The method of claim 1, wherein selecting the desired configuration from the set of test configurations further comprises:
obtaining guardrail constraints associated with one or more of the operating parameters, the guardrail constraints derived from a graph of dependent microservices; and
selecting the desired configuration in which the one or more operating parameters are within the guardrail constraints.

### Claim 9

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps including:
accessing an objective function associated with tuning a configuration of a microservice that performs machine learning functions associated with an online system;
accessing a configuration search space that defines a range of possible configurations of the microservice;
for each of a set of test configurations in the configuration search space, configuring respective allocations of resources associated with executing the microservice;
executing the microservice on respective sample requests under the respective allocations of resources while measuring respective operating parameters associated with the objective function;
evaluating the objective function using the respective operating parameters to generate respective objective values for each of the set of test configurations;
selecting a desired configuration from the set of test configurations based on the respective objective values; and
configuring the microservice in accordance with the desired configuration.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the objective function seeks to minimize a combination of latency and cost associated with resource utilization.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the resource utilization includes at least processor utilization and memory utilization.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the microservice is deployed in a containerized computing environment in which a set of single-threaded or multi-threaded processes execute in containers utilizing dynamically allocated processing and memory resources.

### Claim 13 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the configuration search space includes a two-dimensional array of variables and values for the variables, wherein the variables include one or more of: allocated processors, allocated memory, number of containers, number of processes, number of threads, and machine learning model compression.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the objective function comprises respective baseline values and respective weights associated with each of the operating parameters, and wherein evaluating the objective function is based on the respective operating parameters, the respective baseline values, and the respective weights.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the respective weights and the respective baseline values are obtained from a trained model that optimizes the respective weights and the respective baseline values based on historical data.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein selecting the desired configuration from the set of test configurations further comprises:
obtaining guardrail constraints associated with one or more of the operating parameters, the guardrail constraints derived from a graph of dependent microservices; and
selecting the desired configuration in which the one or more operating parameters are within the guardrail constraints.

### Claim 17

A computer system comprising:
one or more processors; and
a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the computer system to perform steps including:
accessing an objective function associated with tuning a configuration of a microservice that performs machine learning functions associated with an online system;
accessing a configuration search space that defines a range of possible configurations of the microservice;
for each of a set of test configurations in the configuration search space, configuring respective allocations of resources associated with executing the microservice;
executing the microservice on respective sample requests under the respective allocations of resources while measuring respective operating parameters associated with the objective function;
evaluating the objective function using the respective operating parameters to generate respective objective values for each of the set of test configurations;
selecting a desired configuration from the set of test configurations based on the respective objective values; and
configuring the microservice in accordance with the desired configuration.

### Claim 18 (depends on 17)

The computer system of claim 17, wherein the objective function seeks to minimize a combination of latency and cost associated with resource utilization.

### Claim 19 (depends on 18)

The computer system of claim 18, wherein the resource utilization includes at least processor utilization and memory utilization.

### Claim 20 (depends on 17)

The computer system of claim 17, wherein the microservice is deployed in a containerized computing environment in which a set of single-threaded or multi-threaded processes execute in containers utilizing dynamically allocated processing and memory resources.
