Method and system for resource optimization to perform an operation

A system comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions that when executed by the one or more processors cause the one or more processors to perform a set of operations comprising:
• accessing a request for allocating graphical processing unit (GPU) resources for performing an operation, wherein the request includes metadata identifying a client identifier (ID) associated with a client, a throughput, and a latency of the operation;
• determining a predicted resource limit for performing the operation based on the metadata;
• obtaining at least one parameter of a plurality of GPU resources present in a plurality of nodes, wherein the at least one parameter comprises a status indicating whether a corresponding GPU resource is occupied for performing another operation;
• determining a GPU resource utilization value of each node of the plurality of nodes based on the status of each GPU resource, wherein the GPU resource utilization value indicates an amount of utilization of GPU resources of corresponding node;
• comparing the GPU resource utilization value of each node with a pre-defined resource utilization threshold value;
• in response to determining that the GPU resource utilization value is less than the pre-defined resource utilization threshold value, re-scheduling the plurality of GPU resources based on the predicted resource limit;
• and
• allocating a set of GPU resources from the plurality of re-scheduled GPU resources for performing the operation.

A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors that cause the one or more processors to perform a set of operations comprising:

• accessing a request for allocating graphical processing unit (GPU) resources for performing an operation, wherein the request includes metadata identifying a client identifier (ID) associated with a client, a throughput, and a latency of the operation;
• determining a predicted resource limit for performing the operation based on the metadata;
• obtaining at least one parameter of a plurality of GPU resources present in a plurality of nodes, wherein the at least one parameter comprises a status indicating whether a corresponding GPU resource is occupied for performing another operation;
• determining a GPU resource utilization value of each node of the plurality of nodes based on the status of each GPU resource, wherein the GPU resource utilization value indicates an amount of utilization of GPU resources of corresponding node;
• comparing the GPU resource utilization value of each node with a pre-defined resource utilization threshold value;
• in response to determining that the GPU resource utilization value is less than the pre-defined resource utilization threshold value, re-scheduling the plurality of GPU resources based on the predicted resource limit;
• and
• allocating a set of GPU resources from the plurality of re-scheduled GPU resources for performing the operation.