Hardware-assisted gradient optimization using streamed gradients
AssigneeMicrosoft Technology Licensing, LLC
InventorsJinwen XI, Bharadwaj PUDIPEDDI, Marc TREMBLAY
Life of this patent
Estimating the term…
Loading patent…
AssigneeMicrosoft Technology Licensing, LLC
InventorsJinwen XI, Bharadwaj PUDIPEDDI, Marc TREMBLAY
Estimating the term…
Systems and methods related to hardware-assisted gradient optimization using streamed gradients are described. An example method in a system comprising a memory configured to store weights associated with a neural network model comprising L layers, where L is an integer greater than one, a gradient optimizer, and a plurality of workers is described. The method includes during a single burst cycle moving a first set of gradients, received from each of the plurality of workers, from at least one gradient buffer to the gradient optimizer and moving weights from at least one buffer, coupled to the memory, to the gradient optimizer. The method further includes during the single burst cycle writing back the new weights, calculated by the gradient optimizer, to the memory. The method further includes during the single burst cycle transmitting the new weights, from the gradient optimizer, to each of the plurality of workers.
1-20. (canceled)
A method comprising:
A system comprising:
A method for operating a system comprising a memory configured to store weights associated with a neural network model comprising L layers, wherein L is an integer greater than one, a gradient optimizer, and a plurality of workers, wherein each of the plurality of workers is configured to perform a forward pass and a backward pass on any one of the L layers associated with the neural network model, the method comprising:
Dependencies are inferred from the claim text.