AI enabler for application performance automation

A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

• provisioning a first node in a first cluster from a first availability zone (AZ);
• receiving, at the first node, a request to deploy a large language model (LLM);
• determining an amount of GPU resources that are to be used to operate the LLM;
• collecting data associated with a plurality of additional AZs different from the first AZ, the plurality of additional AZs including different types of compute resources, the compute resources including different types of CPUs and GPUs, wherein collected data associated with the plurality of additional AZs includes metrics indicating GPU availability in each respective AZ;
• analyzing data associated with the plurality of additional AZs that have GPU resources, the data including metrics about the GPU resources in each of the plurality of additional AZs;
• selecting a second AZ from the plurality of additional AZs based on the metrics about the GPU resources in the plurality of additional AZs;
• provisioning a second node including GPU resources from the second AZ as a node in a second cluster;
• registering the second node as a virtual node in the first cluster, wherein the first cluster schedules a workload associated with deployment of the LLM on the second node notwithstanding that the second node is in a second cluster from the second AZ;
• and
• deploying the LLM onto the virtual node, wherein the first cluster tracks status of the workload on the second node notwithstanding that the second node is in a second cluster from the second AZ.

A computing system, comprising:

• one or more processors;
• and
• a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
• provisioning a first node in a first cluster from a first availability zone (AZ);
• receiving, at the first node, a request to deploy a large language model (LLM);
• determining an amount of GPU resources that are to be used to operate the LLM;
• collecting data associated with a plurality of additional AZs different from the first AZ, the plurality of additional AZs including different types of compute resources, the compute resources including different types of CPUs and GPUs, wherein collected data associated with the plurality of additional AZs includes metrics indicating GPU availability in each respective AZ;
• analyzing data associated with the plurality of additional AZs that have GPU resources, the data including metrics about the GPU resources in each of the plurality of additional AZs;
• selecting a second AZ from the plurality of additional AZs based on the metrics about the GPU resources in the plurality of additional AZs;
• provisioning a second node including GPU resources from the second AZ as a node in a second cluster;
• registering the second node as a virtual node in the first cluster, wherein the first cluster schedules a workload associated with deployment of the LLM on the second node notwithstanding that the second node is in a second cluster from the second AZ;
• and
• deploying the LLM onto the virtual node, wherein the first cluster tracks status of the workload on the second node notwithstanding that the second node is in a second cluster from the second AZ.