Method and apparatus for fast adaptive neural network training
OwnerCynthia Katherine Nguyen, Darrion Vinh Nguyen +1inventors
InventorsCYNTHIA KATHERINE NGUYEN, DARRION VINH NGUYEN, ANTHONY JUSTIN NGUYEN
Life of this patent
Estimating the term…
Abstract
A method and apparatus for training neural networks and other machine learning models using higher-order derivative approximation. The method uses two or more gradients in a parameter update path to calculate an approximation of the directional derivative of the gradient in the opposite direction of the gradient. This approximation is then used to compute per-parameter adaptive learning rates for updating model parameters. The method can provide fast convergence for many artificial intelligence training applications while maintaining computational and memory efficiency comparable to those of known first-order methods.
Claims20
A computer-implemented method to train a neural network by iteratively updating parameters of the network to minimize a loss function, the method comprising:
- obtaining two or more gradients of the loss function at two or more parameter vectors wherein the two or more parameter vectors are in a parameter update path;
- using the two or more gradients and a distance(s) between the two or more parameter vectors to calculate a derivative of gradient in its opposite direction (DGOD) vector which approximates a directional derivative of a gradient at a current parameter vector of the network in the opposite direction of the gradient at the current parameter vector of the network;
- using the DGOD vector to calculate a per-parameter adaptive learning rate vector, wherein an adaptive learning rate of a parameter of the network is inversely proportional to an element of the DGOD vector associated with the parameter, except when the element of the DGOD vector associated with the parameter is smaller than a predetermined threshold in which case the adaptive learning rate of the parameter is a predetermined bounded value;
- and
- using the adaptive learning rate vector and either the gradient at the current parameter vector of the network or a gradient moving average to calculate an update to the current parameter vector of the network.
An apparatus to train a neural network by iteratively updating parameters of the network to reduce a loss function, the apparatus comprising a processing unit and a storage device storing a computer program that, when executed by the processing unit, causes the processing unit to perform operations comprising:
- obtaining two or more gradients of the loss function at two or more parameter vectors wherein the two or more parameter vectors are in a parameter update path;
- using the two or more gradients and a distance(s) between the two or more parameter vectors to calculate a derivative of gradient in its opposite direction (DGOD) vector which approximates a directional derivative of a gradient at a current parameter vector of the network in the opposite direction of the gradient at the current parameter vector of the network;
- using the DGOD vector to calculate a per-parameter adaptive learning rate vector, wherein an adaptive learning rate of a parameter of the network is inversely proportional to an element of the DGOD vector associated with the parameter, except when the element of the DGOD vector associated with the parameter is smaller than a predetermined threshold in which case the adaptive learning rate of the parameter is a predetermined bounded value;
- and
- using the adaptive learning rate vector and either the gradient at the current parameter vector of the network or a gradient moving average to calculate an update to the current parameter vector of the network.
Dependencies are inferred from the claim text.