Method and system for deployment of large language models (llm) in cloud instances

A system, comprising:

• one or more hardware processors;
• a communication interface;
• and
• a memory storing a plurality of instructions, wherein the plurality of instructions cause the one or more hardware processors to:
• receive configuration of a target Large Language Model (LLM) to be deployed, a user workload, an application type, and a plurality of latency constraints, as an input;
• determine feasibility of each of a plurality of cloud instances comprising a plurality of hardware modules, with respect to accommodating the target LLM, by:
• receiving configuration data of each of the plurality of cloud instances;
• and
• performing a feasibility check for each of the plurality of cloud instances, comprises, determining, by comparing the configuration data of each of the plurality of cloud instances with the configuration of the target LLM, whether the target LLM is feasible with each of the plurality of cloud instances in terms of size of the target LLM and available storage space at each of the plurality of cloud instances;
• determine by a performance model, a latency value for a plurality of batch sizes, for a plurality of target LLM-accelerator pairs, in each of a plurality of cloud instances identified as feasible based on the feasibility check;
• and
• generate a recommendation of one of the plurality of cloud instances identified as feasible, based on the determined latency, a measured cost of deployment, the user workload, the application type, the plurality of latency constraints, and an evaluated performance, wherein the recommended cloud instance is used to host the target LLM to perform an application for which the application type is received.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving configuration of a target Large Language Model (LLM) to be deployed, a user workload, an application type, and a plurality of latency constraints, as an input;
• determining feasibility of each of a plurality of cloud instances comprising a plurality of hardware modules, with respect to accommodating the target LLM, comprising:
• receiving configuration data of each of the plurality of cloud instances;
• and
• performing a feasibility check for each of the plurality of cloud instances, comprises, determining, by comparing the configuration data of each of the plurality of cloud instances with the configuration of the target LLM, whether the target LLM is feasible with each of the plurality of cloud instances in terms of size of the target LLM and available storage space at each of the plurality of cloud instances;
• determining by a performance model, a latency value for a plurality of batch sizes, for a plurality of target LLM-accelerator pairs, in each of a plurality of cloud instances identified as feasible based on the feasibility check;
• and
• generating a recommendation of one of the plurality of cloud instances identified as feasible, based on the determined latency, a measured cost of deployment, the user workload, the application type, the plurality of latency constraints, and an evaluated performance, wherein the recommended cloud instance is used to host the target LLM to perform an application for which the application type is received.