Systems and methods for exchange of data in distributed training of machine learning algorithms
AssigneeNeuralmagic Inc.
InventorsAlexander MATVEEV, Nir Shavit
Life of this patent
Estimating the term…
Abstract
Systems and methods may make exchanging data in a neural network (NN) during training more efficient. Exchanging weights among a number of processors training a NN across iterations may include sorting generated weights, compressing the sorted weights, and transmitting the compressed sorted weights. On each Kth iteration a sort order of the sorted weights may be created and transmitted. Exchanging weights among processors training a NN may include executing a forward pass to produce a set of loss values for processors, transmitting loss values to other processors, and at each of the processors, performing backpropagation on at least one layer of the NN using loss values received from other processors.
Claims20
A method of conducting training on a neural network (NN), the NN comprising neurons arranged into layers, the method comprising:
- at each of a plurality of processors, executing a forward pass on the NN to produce a set of loss values for the processor;
- at each of the plurality of the processors, transmitting the set of loss values to a set of processors of the plurality of processors;
- and
- at each of the plurality of the processors, performing backpropagation on at least one layer of the NN using loss values received from a set of processors of the plurality of processors, wherein the ratio of compute burden of the at least one layer on which backpropagation is performed to the compute burden for the other layers in the NN is less than the ratio of the number of weights for the at least one layer on which backpropagation is performed to the number of weights for the other layers in the NN.
A method of conducting training on a model of a neural network (NN), the NN arranged into layers, the method comprising, at each or a plurality of nodes:
- receiving training data to produce a set of local losses;
- sending the losses to other nodes of the plurality of nodes;
- for at least a first layer, performing backpropagation using local losses and losses received from other nodes of the plurality of nodes;
- and
- for at least a second layer, performing backpropagation using local losses and not using losses received from other nodes of the plurality of nodes, wherein the ratio of compute burden of the at least one layer on which backpropagation is performed to the compute burden for the other layers in the NN is less than the ratio of the number of weights for the at least one layer on which backpropagation is performed to the number of weights for the other layers in the NN.
A 5y stem conducting training on a neural network (NN), the NN comprising neurons arranged into layers, the system comprising:
- a plurality of nodes, each node comprising a memory and a processor configured to:
- execute a forward pass on the NN to produce a set of loss values for the node;
- transmit the set of loss values to a set of nodes of the plurality of nodes;
- and
- perform backpropagation on at least one layer of the NN using loss values received from a set of nodes of the plurality of nodes.
A method of conducting training of a neural network (NN), the NN arranged into layers and represented as a NN model, the method comprising:
- at each of a plurality of non-master nodes:
- receiving training data to produce a set of losses,
- sending the losses to a master node;
- at the master node:
- for a subset of he NN layers, performing backpropagation using the losses;
- updating the NN model;
- and
- transmitting the NN model to the non-master nodes.
Dependencies are inferred from the claim text.