Multi-instance gpu aware autoscaling in ai model service

A computer program product comprising:

• One or more computer readable storage media;
• and
• program instructions stored on the one or more storage media and configured to perform operations comprising:
• analyzing an inference request to determine a set of parameters of execution corresponding to the inference request;
• computing, for a Large Language Model (LLM), a first amount of a computing resource that will be needed to produce and store the LLM output, the LLM output comprising a set of intermediate results produced while processing the inference request by executing the LLM using a set of multi-instance Graphical Processing Units (GPUs) (MIGs), a MIG in the set of MIGs comprising a set of slices of a corresponding GPU (set of MIG slices);
• causing, by sending a set of instructions to a controller associated with the MIG, the controller to modify a second amount of the computing resource available to a MIG slice in the set of MIG slices;
• and
• scheduling the inference request to execute using the first amount of computing resource at the MIG slice.

A computer system comprising a processor and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the processor to cause the processor to perform operations comprising:

• analyzing an inference request to determine a set of parameters of execution corresponding to the inference request;
• computing, for a Large Language Model (LLM), a first amount of a computing resource that will be needed to produce and store the LLM output, the LLM output comprising a set of intermediate results produced while processing the inference request by executing the LLM using a set of multi-instance Graphical Processing Units (GPUs) (MIGs), a MIG in the set of MIGs comprising a set of slices of a corresponding GPU (set of MIG slices);
• causing, by sending a set of instructions to a controller associated with the MIG, the controller to modify a second amount of the computing resource available to a MIG slice in the set of MIG slices;
• and
• scheduling the inference request to execute using the first amount of computing resource at the MIG slice.