# US 2025/0094237 A1 · Capacity-based load balancing in shared resource pool

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094237 A1
- Document ID: US20250094237A1
- Status: Published application
- Kind: A1
- Application number: 18/470,772
- Filed: 2023-09-20
- Published: 2025-03-20
- Earliest claimed: 2023-09-20
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Wenbin MENG; Hemant KUMAR; Rakesh KELKAR; Karthik RAMAN; Sanjay RAMANUJAN; Kevin Joseph RIEHM; Theodore Dragov TODOROV
- Web page: https://patentgrep.com/patent/US20250094237A1

## CPC

- G06F 9/505: {considering the load}
- G06F 9/5083: {Techniques for rebalancing the load in a distributed system}
- G06F 2209/5011: Pool
- G06F 2209/503: Resource availability
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system provides capacity-based load balancing across model endpoints of a cloud-based artificial intelligence (AI) model. The system includes a consumption determination engine executable to determine a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model. The system also includes a load balancer that determines a distribution of available resource capacity in a shared resource pool comprising compute resources at each of the multiple model endpoints. The load balancer allocates parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource pool.

## Claims

### Claim 1

A system for improved utilization of compute hardware distributed among a plurality of endpoints of a cloud-based service operating a trained machine learning model, the system comprising:
a consumption determination engine stored in memory and executable to determine a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model; and
a load balancer stored in the memory and executable to:
determine a distribution of available resource capacity in a shared resource pool comprising compute resources at the plurality of model endpoints executing instances of the trained machine learning model; and
allocate parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource pool.

### Claim 2 (depends on 1)

The system of claim 1, wherein the parallelizable tasks of the workload are allocated among the plurality of model endpoints according to a target allocation distribution that is based on a fractional distribution of the available resource capacity across the plurality of model endpoints within the shared resource pool.

### Claim 3 (depends on 2)

The system of claim 2, wherein the target allocation distribution is a distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints that is proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

### Claim 4 (depends on 1)

The system of claim 1, wherein the consumption determination engine is configured to:
determine resource consumption characteristics of the workload; and
determine the net resource consumption for each of the parallelizable tasks of the workload based on the resource consumption characteristics.

### Claim 5 (depends on 4)

The system of claim 4, wherein the trained machine learning model is a transformer model and the net resource consumption for each of the parallelizable tasks is determined based on an identity of the transformer model and a set of inputs to the workload.

### Claim 6 (depends on 4)

The system of claim 4, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on a size of data input to each of the parallelizable tasks and an estimated size of data output in response to processing of the data input.

### Claim 7 (depends on 1)

The system of claim 1, wherein the load balancer determines the distribution of available resource capacity by requesting, from an endpoint discovery mechanism, capacity measurements pertaining to availability of compute resources supporting execution of the model instances at the model endpoints, the endpoint discovery mechanism being configured to retrieve the capacity measurements from the model endpoints.

### Claim 8

A method for improved utilization of compute hardware distributed among multiple model endpoints of a cloud-based service operating a trained machine learning model, the method comprising:
determining a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model;
determining a distribution of available resource capacity in a shared resource pool comprising compute resources the a plurality of model endpoints executing instances of the trained machine learning model; and
allocating parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource pool.

### Claim 9 (depends on 8)

The method of claim 8, wherein the parallelizable tasks of the workload are allocated among the multiple model endpoints according to a target allocation distribution that is based on a fractional distribution of the available resource capacity across the multiple model endpoints within the shared resource pool.

### Claim 10 (depends on 9)

The method of claim 9, wherein the target allocation distribution is a distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints that is proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
determining resource consumption characteristics of the workload;
determining the net resource consumption for each of the parallelizable tasks of the workload based on the resource consumption characteristics; and
requesting capacity measurements pertaining to availability of compute resources supporting execution of the model instances at the model endpoints.

### Claim 12 (depends on 11)

The method of claim 11, wherein the trained machine learning model is a transformer model and the net resource consumption for each of the parallelizable tasks is determined based on an identity of the transformer model and a set of inputs to the workload.

### Claim 13 (depends on 11)

The method of claim 11, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on a size of data input to each of the parallelizable tasks and an estimated size of data output in response to processing of the data input.

### Claim 14 (depends on 8)

The method of claim 8, wherein the workload is a batch processing request and the method further comprising determining a net resource consumption associated with processing each of multiple different files.

### Claim 15

One or more tangible computer-readable storage media encoding processor-executable instructions for executing a computer process for improved utilization of compute hardware distributed among a plurality of model endpoints of a cloud-based service operating a trained machine learning model, the computer process comprising:
determining a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model;
determining a distribution of available resource capacity in a shared resource pool comprising compute resources at the plurality of model endpoints executing instances of the cloud-based AI mode; and
allocating parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource.

### Claim 16 (depends on 15)

The one or more tangible computer-readable storage media of claim 15, wherein the parallelizable tasks of the workload are allocated among the multiple model endpoints according to a target allocation distribution that is based on a fractional distribution of the available resource capacity across the multiple model endpoints within the shared resource pool.

### Claim 17 (depends on 16)

The one or more tangible computer-readable storage media of claim 16, wherein the target allocation distribution is a distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints that is proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

### Claim 18 (depends on 15)

The one or more tangible computer-readable storage media of claim 15, wherein the computer process further comprises:
determining resource consumption characteristics of the workload;
determining the net resource consumption for each of the parallelizable tasks of the workload based on the resource consumption characteristics; and
requesting capacity measurements pertaining to availability of compute resources supporting execution of the model instances at the model endpoints.

### Claim 19 (depends on 18)

The one or more tangible computer-readable storage media of claim 18, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on a size of data input to each of the parallelizable tasks and an estimated size of data output in response to processing of the data input.

### Claim 20 (depends on 18)

The one or more tangible computer-readable storage media of claim 18, wherein the trained machine learning model is a transformer model and the net resource consumption for each of the parallelizable tasks is determined based on an identity of the transformer model and a set of inputs to the workload.
