Capacity-based load balancing in shared resource pool

A method for improved utilization of compute hardware distributed among multiple model endpoints of a cloud-based service operating a trained machine learning model, the method comprising:

• determining a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model;
• determining a distribution of available resource capacity in a shared resource pool comprising compute resources the a plurality of model endpoints executing instances of the trained machine learning model;
• and
• allocating parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource pool.

One or more tangible computer-readable storage media encoding processor-executable instructions for executing a computer process for improved utilization of compute hardware distributed among a plurality of model endpoints of a cloud-based service operating a trained machine learning model, the computer process comprising:

• determining a net resource consumption for processing tasks in a workload generated by a client application for input to the trained machine learning model;
• determining a distribution of available resource capacity in a shared resource pool comprising compute resources at the plurality of model endpoints executing instances of the cloud-based AI mode;
• and
• allocating parallelizable tasks of the workload among the compute resources at the multiple model endpoints based on the net resource consumption of the tasks and on the distribution of available resource capacity in the shared resource.