Alternating gradient computation
AssigneeZoox, Inc.
InventorEthan Miller Pronovost
Life of this patent
Estimating the term…
Abstract
Techniques for training a machine learning model are described herein. For example, the techniques may include determining a gradient variable to store in memory to determine gradient values for two or more operations of a neural network. The gradient variable may represent an input or an output of an operation and can be accessed from the memory to determine multiple different gradient values during backward propagation. Such techniques may be used to train a neural network that can reduce a number of gradient variables stored in memory during forward propagation based on determining a gradient variable that is usable as an input to multiple gradient functions associated with a sequence of operations.
Claims20
A system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform actions comprising:
- determining that an output of a first operation is an input to a second operation associated with forward propagation of a neural network;
- determining a first gradient function associated with a backward propagation of the neural network, the first gradient function associated with the first operation and implemented as a function of the output of the first operation;
- determining a second gradient function associated with the backward propagation of the neural network, the second gradient function associated with the second operation and implemented as the function of the output of the first operation;
- determining a variable associated with the output of the first operation based at least in part on the second gradient function being implemented as the function of the output of the first operation;
- computing, based at least in part on the variable, a first gradient value associated with the first operation and a second gradient value associated with the second operation;
- training the neural network based at least in part on the first operation, the second operation, the first gradient function, the second gradient function, and the variable, the neural network, once trained, is configured to control operation of a vehicle;
- and
- reducing, based at least in part on training the neural network, a number of gradient values stored in a memory.
A method comprising:
- receiving a first operation associated with forward propagation of a neural network;
- determining, based at least in part on the first operation, a second operation associated with backward propagation of the neural network, the second operation comprising one of a first gradient value function or a second gradient value function, wherein:
- the first gradient value function and the second gradient value function yield a same gradient value result;
- the first gradient value function is based, at least in part, on an input to the first operation;
- and
- the second gradient value function is based, at least in part, on an output of the first operation;
- determining, based at least in part on the first gradient value function and the second gradient value function yielding the same gradient value result, a variable associated with the output of the first operation to compute a first gradient value associated with the first operation and a second gradient value associated with the second operation;
- training the neural network based at least in part on the first operation, the second operation, and the variable;
- and
- reducing, based at least in part on training the neural network, a number of gradient values stored in a memory.
One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform actions comprising:
- identifying a variable used in forward propagation of a neural network, the variable being an output from a first operation of the neural network and an input to a second operation of the neural network;
- providing the variable to a first gradient value function and a second gradient value function at a same time, wherein:
- the first gradient value function is associated with the first operation;
- and
- the second gradient value function is associated with the second operation;
- training, based on providing the variable to the first gradient value function and the second gradient value function, the neural network;
- and
- reducing, based at least in part on training the neural network, a number of gradient values stored in a memory.
Dependencies are inferred from the claim text.