System, method, and medium for elastic allocation of resources for deep learning jobs

A system comprising:

• a cloud computing resource pool comprising a plurality of nodes;
• a resource allocation processor device;
• and
• a memory storing instructions that, when executed by the resource allocation processor device, cause the resource allocation processing unit to train a plurality of models by:
• obtaining a plurality of job profiles, each job profile comprising training information for a training job, wherein a training job comprises training one of the plurality of models;
• for each job profile, processing the respective training information to:
• generate one or more node count sequences, each node count sequence indicating, for each of a first plurality of time periods beginning with a first time period and ending with a final time period, a node count for the respective training job;
• and
• for each node count sequence, generate a respective estimated progress value of the respective training job at the end of the final time period;
• processing the estimated progress values corresponding to each of the one or more node count sequences of each of the plurality of training jobs to generate an estimated optimal allocation sequence comprising a respective selected node count sequence for each training job;
• and
• for each training job, over the first time period, using a number of the plurality of nodes indicated by the node count of the respective selected node count sequence for the first time period to train the respective model based on the training information for the respective model.

A non-transitory computer-readable medium storing instructions thereon to be executed by at least one processor in a cloud computing system, the instructions, when executed, causing the cloud computing system to perform a method comprising:

• obtaining a plurality of job profiles, each job profile comprising training information for a training job, wherein a training job comprises training one of the plurality of models;
• for each job profile, processing the respective training information to:
• generate one or more node count sequences, each node count sequence indicating, for each of a first plurality of time periods beginning with a first time period and ending with a final time period, a node count for the respective training job;
• and
• for each node count sequence, generate a respective estimated progress value of the respective training job at the end of the final time period;
• processing the estimated progress values corresponding to each of the one or more node count sequences of each of the plurality of training jobs to generate an estimated optimal allocation sequence comprising a respective selected node count sequence for each training job;
• and
• for each training job, over the first time period, using a number of the plurality of nodes indicated by the node count of the respective selected node count sequence for the first time period to train the respective model based on the training information for the respective model.