Hierarchical decentralized distributed deep learning training
AssigneeInternational Business Machines Corporation
InventorsWei Zhang, Xiaodong Cui, Abdullah Kayi, Alper Buyuktosunoglu
Life of this patent
Estimating the term…
Abstract
Embodiments of a method are disclosed. The method includes performing a batch of decentralized deep learning training for a machine learning model in coordination with multiple local homogenous learners on a deep learning training compute node, and in coordination with multiple super learners on corresponding deep learning training compute nodes. The method also includes exchanging communications with the super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol. The communications are associated with the batch of deep learning training.
Claims20
A computer-implemented method, comprising:
- performing a batch of decentralized deep learning training for a machine learning model in coordination with a local plurality of homogenous learners on a deep learning training compute node, and in coordination with a super plurality of super learners on a corresponding plurality of deep learning training compute nodes;
- and
- exchanging communications with the super plurality of super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol, wherein the communications are associated with the batch of deep learning training.
A computer program product comprising program instructions stored on a computer readable storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:
- performing a batch of deep learning training for a machine learning model in coordination with a local plurality of homogenous learners on a deep learning training compute node, and in coordination with a super plurality of super learners on a corresponding plurality of deep learning training compute nodes;
- and
- exchanging communications with the super plurality of super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol, wherein the communications are associated with the batch of deep learning training.
A hierarchical system comprising:
- a super plurality of super learners at a parent level of the hierarchical system;
- and
- a local plurality of homogenous learners at a child level of the hierarchical system, wherein each of the super plurality of super learners is a parent to an associated subset of the local plurality of homogenous learners, and wherein respective super learners are associated with respective subsets of the local plurality of homogenous learners, and wherein each of the super plurality of learners comprises:
- a computer processing circuit;
- and
- a computer-readable storage medium storing instructions, which, when executed by the computer processing circuit, are configured to cause the computer processing circuit to perform a method comprising:
- performing a batch of deep learning training for a machine learning model in coordination with the associated subset of the local plurality of homogenous learners on a deep learning training compute node, and in coordination with the super plurality of super learners on a corresponding plurality of deep learning training compute nodes;
- and
- exchanging communications with the super plurality of super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol, wherein the communications are associated with the batch of deep learning training.
Dependencies are inferred from the claim text.