Adaptive Optimization with Improved Convergence
AssigneeGoogle Inc.
InventorsSashank Jakkam Reddi, Sanjiv Kumar, Satyen Chandrakant Kale
Life of this patent
Estimating the term…
Abstract
Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.
Claims20
A computer-implemented method for optimizing machine-learned models that provides improved convergence properties, the method comprising:
- determining, by one or more computing devices, a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters;
- determining, by the one or more computing devices, a candidate learning rate control value based at least in part on the gradient of the loss function;
- comparing, by the one or more computing devices, the candidate learning rate control value to a maximum previously observed learning rate control value;
- when the candidate learning rate control value is greater than the maximum previously observed learning rate control value:
- setting a current learning rate control value equal to the candidate learning rate control value;
- and
- setting the maximum previously observed learning rate control value equal to the candidate learning rate control value;
- when the candidate learning rate control value is less than the maximum previously observed learning rate control value:
- setting the current learning rate control value equal to the maximum previously observed learning rate control value;
- determining, by the one or more computing devices, a current learning rate based at least in part on the current learning rate control value;
- and
- determining, by the one or more computing devices, an updated set of values for the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to the current learning rate.
A computing system, comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
- determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters;
- determining a candidate learning rate based at least in part on the gradient of the loss function;
- selecting a minimum of the candidate learning rate and a minimum previously observed learning rate to serve as a current learning rate;
- and
- updating at least one of the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to the current learning rate.
One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
- for each of a plurality of iterations:
- determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters;
- determining a candidate learning rate control value based at least in part on the gradient of the loss function;
- selecting a maximum of the candidate learning rate control value and a maximum previously observed learning rate control value as a current learning rate control value;
- and
- updating at least one of the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to a current learning rate that is a function of the current learning rate control value.
Dependencies are inferred from the claim text.