Dynamic computation rates for distributed deep learning
AssigneeInternational Business Machines Corporation
InventorsWei Zhang, Xiaodong Cui, Abdullah Kayi, Alper Buyuktosunoglu
Life of this patent
Estimating the term…
Abstract
Embodiments of a method are disclosed. The method includes performing distributed deep learning training on multiple batches of training data using corresponding learners. Additionally, the method includes determining training times wherein the learners perform the distributed deep learning training on the batches of training data. The method also includes modifying a processing aspect of the straggler to reduce a future training time of the straggler for performing the distributed deep learning training on a new batch of training data in response to identifying a straggler of the learners by a centralized control.
Claims19
A computer-implemented method, comprising:
- determining a plurality of training times wherein a corresponding plurality of learners performs the distributed deep learning training on a corresponding plurality of batches of training data;
- and
- balancing a straggler training time and a remaining learners training time for performing the distributed deep learning training on a new plurality of batches of training data by:
- identifying a ping-pong straggler by determining that the ping-pong straggler changes from a straggler learner to a non-straggler learner in a first deep learning training, and changes from the non-straggler learner to the straggler learner in a second deep learning training;
- maintaining the ping-pong straggler as the straggler learner based on identifying the ping-pong straggler;
- modifying a frequency rate of a computational circuit of at least one of the corresponding plurality of learners in response to identifying a subsequent straggler during a subsequent distributed deep learning training using:
- the corresponding plurality of learners;
- and
- a centralized control;
- and
- performing distributed deep learning training on the new plurality of batches of training data using the corresponding plurality of learners and the modified frequency rate.
A computer program product comprising program instructions stored on a computer readable storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:
- determining a plurality of training times wherein the plurality of learners performs the distributed deep learning training on the plurality of batches of training data;
- and
- balancing a straggler training time and a remaining learners training time for performing the distributed deep learning training on a new plurality of batches of training data by:
- identifying a ping-pong straggler by determining that the ping-pong straggler changes from a non-straggler learner to a straggler learner in a first deep learning training, and changes from the straggler learner to the non-straggler learner in a second deep learning training;
- maintaining the ping-pong straggler as the non-straggler learner based on identifying the ping-pong straggler;
- modifying a frequency rate of a computational circuit of at least one of the corresponding plurality of learners in response to identifying a subsequent straggler during a subsequent distributed deep learning training using:
- the corresponding plurality of learners;
- and
- a centralized control;
- and
- performing distributed deep learning training on the new plurality of batches of training data using the corresponding plurality of learners and the modified frequency rate.
A system comprising:
- a computer processing circuit;
- and
- a computer-readable storage medium storing instructions, which, when executed by the computer processing circuit, are configured to cause the computer processing circuit to perform a method comprising:
- determining a plurality of training times wherein the plurality of learners performs the distributed deep learning training on the plurality of batches of training data;
- and
- balancing a straggler training time and a remaining learners training time for performing the distributed deep learning training on a new plurality of batches of training data by:
- identifying a ping-pong straggler by determining that the ping-pong straggler changes from a straggler learner to a non-straggler learner in a first deep learning training, and changes from the non-straggler learner to the straggler learner in a second deep learning training;
- maintaining the ping-pong straggler as the straggler learner based on identifying the ping-pong straggler;
- modifying a frequency rate of a computational circuit of at least one of the corresponding plurality of learners in response to identifying a subsequent straggler during a subsequent distributed deep learning training using:
- the corresponding plurality of learners;
- and
- a centralized control;
- and
- performing distributed deep learning training on the new plurality of batches of training data using the corresponding plurality of learners and the modified frequency rate.
Dependencies are inferred from the claim text.