Dynamic network bandwidth in distributed deep learning training
AssigneeInternational Business Machines Corporation
InventorsWei Zhang, Xiaodong Cui, Abdullah Kayi, Alper Buyuktosunoglu
Life of this patent
Estimating the term…
Abstract
Embodiments of a method are disclosed. The method includes performing distributed deep learning training on a batch of training data. The method also includes determining training times representing an amount of time between a beginning batch time and an end batch time. Further, the method includes modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to the centralized parameter server determining that the learner is the communication straggler.
Claims14
A computer-implemented method, comprising:
- performing distributed deep learning training on a batch of training data;
- determining a training time representing an amount of time between:
- a beginning batch time for a learner;
- and
- an end batch time for the learner;
- determining that the learner is a communication straggler by determining that the training time exceeds a predetermined threshold time;
- and
- modifying a communication aspect of the learner to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler.
A computer program product comprising program instructions stored on a computer readable storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:
- a learner performing distributed deep learning training on a batch of training data;
- the learner determining a plurality of training times representing an amount of time between:
- a beginning batch time;
- and
- an end batch time;
- the learner determining that the learner is a communication straggler by determining that a training time of the learner exceeds a predetermined threshold time;
- and
- the learner modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler.
A system comprising:
- a computer processing circuit;
- and
- a computer-readable storage medium storing instructions, which, when executed by the computer processing circuit, are configured to cause the computer processing circuit to perform a method comprising:
- performing distributed deep learning training on a batch of training data;
- determining a training time representing an amount of time between:
- a beginning batch time for a learner;
- and
- an end batch time for the learner;
- determining that the learner is a communication straggler by determining that the training time exceeds a predetermined threshold time;
- and
- modifying a communication aspect of the learner to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler.
Dependencies are inferred from the claim text.