Controlled adaptive optimization
AssigneeGoogle Inc.
InventorsSashank Jakkam Reddi, Sanjiv Kumar, Manzil Zaheer, Satyen Chandrakant Kale
Life of this patent
Estimating the term…
Abstract
Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.
Claims20
A computing system for utilization of a machine-learned model that is optimized to perform a task, the computing system comprising:
- one or more processors;
- an optimized machine-learned model comprising a plurality of optimized parameters, wherein the plurality of optimized parameters have been optimized over a plurality of iterations based at least in part on a gradient of a loss function and an effective learning rate, wherein the effective learning rate is based at least in part on a current learning rate control value that equals a recent learning rate control value minus an update value, wherein a magnitude of the update value is a function of the gradient of the loss function, wherein a polarity of the update value is a function of both the gradient of the loss function and the recent learning rate control value, and wherein, for at least one of the plurality of iterations, the polarity of the update value is positive such that the current learning rate control value is less than the most recent learning rate control value, whereby the current effective learning rate is greater than a most recent effective learning rate;
- and
- one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
- obtaining input data for a task that the optimized machine-learned model is optimized to perform;
- and
- processing the input data using the optimized machine-learned model to obtain optimized output data associated with the task.
A process for the production of an optimized machine-learned model product trained using an adaptive learning algorithm with improved convergence properties and stored on one or more tangible, non-transitory computer readable media, wherein the process comprises the steps of:
- (a) for a plurality of iterations:
- determining a gradient of a loss function that evaluates a performance of a machine-learned model, wherein the machine-learned model comprises a plurality of parameters;
- determining a current learning rate control value based on the gradient of the loss function, wherein the current learning rate control value equals a most recent learning rate control value minus an update value, wherein a magnitude of the update value is a function of the gradient of the loss function but not the most recent learning rate control value, and wherein a polarity of the update value is a function of both the gradient of the loss function and the most recent learning rate control value;
- and
- determining an updated set of values for the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to the current effective learning rate;
- wherein, for at least one of the plurality of iterations, the polarity of the update value is positive such that the current learning rate control value is less than the most recent learning rate control value, whereby the current effective learning rate is greater than a most recent effective learning rate;
- and
- (b) storing the optimized machine-learned model product on the one or more tangible, non-transitory computer readable media, wherein the optimized machine-learned model product comprises an optimized version of the machine-learned model comprising a final set of values for the parameters.
A computer-implemented method for utilization of a machine-learned model that is optimized to perform a task, comprising:
- obtaining, by a computing system comprising one or more computing devices, input data for a task that an optimized machine-learned model is optimized to perform, wherein the optimized machine-learned model comprises a plurality of optimized parameters, wherein the plurality of optimized parameters have been optimized over a plurality of iterations based at least in part on a gradient of a loss function and an effective learning rate, wherein the effective learning rate is based at least in part on a current learning rate control value that equals a recent learning rate control value minus an update value, wherein the update value is equal to a square of the gradient of the loss function multiplied by a sign function applied to the recent learning rate control value minus the square of the gradient of the loss function and multiplied by a scaling coefficient that is equal to one minus an update scaling parameter;
- and
- processing, by the computing system, the input data using the optimized machine-learned model to obtain optimized output data associated with the task.
Dependencies are inferred from the claim text.