Systems and methods for exchange of data in distributed training of machine learning algorithms
AssigneeNeuralmagic Inc.
InventorsAlexander MATVEEV, Nir SHAVIT
Life of this patent
Estimating the term…
Abstract
Systems and methods may make exchanging data in a neural network (NN) during training more efficient. Exchanging weights among a number of processors training a NN across iterations may include sorting generated weights, compressing the sorted weights, and transmitting the compressed sorted weights. On each Kth iteration a sort order of the sorted weights may be created and transmitted. Exchanging weights among processors training a NN may include executing a forward pass to produce a set of loss values for processors, transmitting loss values to other processors, and at each of the processors, performing backpropagation on at least one layer of the NN using loss values received from other processors.
Claims20
A method of exchanging weights among a plurality of processors conducting training on a neural network (NN), the NN comprising neurons, the method comprising:
- over a plurality of iterations:
- at least two processors of the plurality of processors:
- sorting weights generated at the processor to create sorted weights;
- compressing the sorted weights to produce compressed sorted weights;
- and
- transmitting the compressed sorted weights to least one processor of the plurality of processors;
- wherein, on each Kth iteration of the plurality of iterations, K being an integer greater than 1:
- a sort order of the sorted weights is created;
- and
- the sort order is transmitted to least one processor of the plurality of processors.
A method of exchanging parameters among a plurality of computing nodes, the method comprising:
- over a plurality of iterations including a periodically occurring ordering iteration:
- at a first computing node:
- if the iteration is an ordering iteration, arranging parameters by value and storing the order of the arranging;
- if the iteration is not an ordering iteration, arranging parameters by the order of the arranging;
- compressing the arranged parameters to produce compressed parameters;
- and
- sending the compressed parameters to a second computing node.
A system comprising a plurality of nodes conducting training on a neural network (NN), the NN comprising neurons, each node comprising:
- a memory;
- and
- a processor;
- the processor at least one node configured to:
- over a plurality of iterations:
- sort weights generated at the processor to create sorted weights;
- compress the sorted weights to produce compressed sorted weights;
- and
- transmit the compressed sorted weights to least one node;
- wherein, on each Kth iteration of the plurality of iterations, K being an integer greater than 1:
- a sort order of the sorted weights is created by the processor;
- and
- the sort order is transmitted to least one node.
Dependencies are inferred from the claim text.