Training large dl models via serverless architecture using cloud storage services-based communication channel

A system for training large Deep Learning (DL) models, the system comprising:

• a memory storing instructions;
• one or more Input/Output (I/O) interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
• spawn a plurality of serverless instances defining a plurality of workers;
• and
• train a DL model via the plurality of workers, by partitioning a training data among the plurality of workers, wherein the partitioned training data split into a plurality of mini batches is utilized sequentially during iterative execution of a plurality of training instances, until the plurality of mini batches are exhausted, the iterative execution of the plurality of training instances comprising:
• initiating a current training instance from among the plurality of training instances by each of the plurality of workers using a current mini-batch among the plurality of mini batches and generating a plurality of trained local DL models;
• chunking, by each of the plurality of workers, each of the plurality of trained local DL models into a plurality of segments based on a mod& size of each of the plurality of trained local DL models and a maximum size of one data item allowed in a cloud storage services-based communication channel;
• writing, the plurality of segments corresponding to each of the plurality of trained local DL models to the cloud storage services-based communication channel in accordance with the maximum size of one data item allowed in the cloud storage services-based communication channel;
• aggregating, by one or more aggregators, corresponding segments of each of the plurality of segments of each of the plurality of trained local DL models to generate a plurality of aggregated segment level DL models;
• writing, by one or more aggregators, the plurality of aggregated segment level DL models into the cloud storage services-based communication channel;
• and
• concurrently reading and concatenating, the plurality of aggregated segment level DL models by each of the plurality of workers to obtain an aggregated model generated by the current training instance, wherein the aggregated model is used by each of the plurality of workers during the successive training instances using a successive mini batch.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• spawning a plurality of serverless instances defining a plurality of workers;
• and
• training a DL mod& by the plurality of workers executed by one or more hardware processors, by partitioning a training data among the plurality of workers, wherein the partitioned training data split into a plurality of mini batches is utilized sequentially during iterative execution of a plurality of training instances, until the plurality of mini batches are exhausted, the iterative execution of the plurality of training instances comprising:
• initiating a current training instance among the plurality of training instances by each of the plurality of workers using a current mini-batch among the plurality of mini batches and generating a plurality of trained local DL models;
• chunking by each of the plurality of workers, each of the plurality of trained local DL models into a plurality of segments based on a mod& size of each of the plurality of trained local DL models and a maximum size of one data item allowed in a cloud storage services-based communication channel;
• writing the plurality of segments corresponding to each of the plurality of trained local DL models to the cloud storage services-based communication channel in accordance with the maximum size of one data item allowed in the cloud storage services-based communication channel;
• aggregating by one or more aggregators, corresponding segments of each of the plurality of segments of each of the plurality of trained local DL models to generate a plurality of aggregated segment level DL models;
• writing the plurality of aggregated segment level DL models into the cloud storage services-based communication channel;
• and
• concurrently reading and concatenating the plurality of aggregated segment level DL models by each of the plurality of workers to obtain an aggregated model generated by the current training instance, wherein the aggregated mod& is used by each of the plurality of workers during the successive training instances using a successive mini batch.