Dynamic network bandwidth in distributed deep learning training
AssigneeInternational Business Machines Corporation
InventorsWei Zhang, Xiaodong Cui, Abdullah Kayi, Alper Buyuktosunoglu
Life of this patent
Estimating the term…
Abstract
Embodiments of a method are disclosed. The method includes performing distributed deep learning training on a batch of training data. The method also includes determining training times representing an amount of time between a beginning batch time and an end batch time. Further, the method includes modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to the centralized parameter server determining that the learner is the communication straggler.
Claims20
A computer-implemented method, comprising:
- performing distributed deep learning training on a batch of training data;
- determining a training time representing an amount of time between:
- a beginning batch time for a learner;
- and
- an end batch time for the learner;
- and
- modifying a communication aspect of the learner to reduce a future network communication time for a communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler.
A computer program product comprising program instructions stored on a computer readable storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:
- a learner performing distributed deep learning training on a batch of training data;
- the learner determining a plurality of training times representing an amount of time between:
- a beginning batch time;
- and
- an end batch time;
- and
- the learner modifying a communication aspect of a communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler.
A system comprising:
- a computer processing circuit;
- and
- a computer-readable storage medium storing instructions, which, when executed by the computer processing circuit, are configured to cause the computer processing circuit to perform a method comprising:
- identifying a communication straggler of a plurality of learners based on whether a network communication time of the communication straggler exceeds a threshold training time for the plurality of learners;
- and
- requesting the communication straggler to modify a communication aspect to reduce a future network communication time for the communication straggler to send a future result to a centralized parameter server.
Dependencies are inferred from the claim text.