Using container and model information to select containers for executing models

A non-transitory computer program product comprising computer-readable program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code including instructions to:

• identify a version of a machine-learning model associated with a request, in response to receiving the request from an application;
• identify model information associated with machine-learning models corresponding to a cluster of available serving containers associated with the version of the machine-learning model;
• select, based on the model information, an existing serving container from the cluster of available serving containers to respond to the request by executing the version of the machine-learning model based on respective resources available to load the version of the machine-learning model of one or more serving containers that have the version of the machine-learning model loaded and one or more serving containers that do not have the version of the machine-learning model loaded, wherein available serving containers are available to respond to the request;
• when the existing serving container that has the version of the machine-learning model loaded is selected, execute, in the existing serving container, the version of the machine-learning model that is loaded in the existing serving container on behalf of the request when not being used by another request, and when the version of the machine-learning model is being used by another request, instead of waiting for the another request to finish, load another copy of the version of the machine-learning model into the existing serving container such that multiple copies of the version of the machine-learning model are stored in the existing serving container, and execute, in the existing serving container, the another copy of the version of the machine-learning model on behalf of the request;
• when the existing serving container that does not have the version of the machine-learning model loaded is selected, load a copy of the version of the machine-learning model in the existing serving container;
• and
• respond to the request based on executing the machine-learning model on behalf of the request.

A computer-implemented method for using container and model information to select containers for executing models, the computer-implemented method comprising:

• identifying a version of a machine-learning model associated with a first request and a second request, in response to receiving the first request and the second request from an application;
• identifying model information associated with machine-learning models corresponding to a cluster of available serving containers associated with the version of the machine-learning model;
• selecting, based on the model information, an existing serving container from the cluster of available serving containers to respond to the request by executing the version of the machine- learning model based on respective resources available to load the version of the machine-learning model of one or more serving containers that have the version of the machine-learning model loaded and one or more serving containers that do not have the version of the machine-learning model loaded, wherein available serving containers are available to respond to the request;
• selecting, for the first request, the existing serving container that has the version of the machine-learning model loaded, and executing, in the existing serving container, the version of the machine-learning model that is loaded in the existing serving container on behalf of the first request when not being used by another request, and when the version of the machine-learning model is being used by another request, instead of waiting for the another request to finish, loading another copy of the version of the machine-learning model into the existing serving container such that multiple copies of the version of the machine-learning model are stored in the existing serving container, and executing, in the existing serving container, the another copy of the version of the machine-learning model on behalf of the request;
• selecting, for the second request, the existing serving container that does not have the version of the machine-learning model loaded, and loading a copy of the version of the machine-learning model in the existing serving container;
• and
• responding to the first request or the second request based on executing the machine-learning model on behalf of the first request or the second request.