Controlled Adaptive Optimization

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising, for each of one or more iterations:
• determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters;
• determining a current learning rate control value based on the gradient of the loss function, wherein the current learning rate control value equals a most recent learning rate control value minus an update value, wherein a magnitude of the update value is equal to a square of the gradient of the loss function times a scaling coefficient, and wherein a polarity of the update value is a function of both the gradient of the loss function and the most recent learning rate control value;
• determining a current effective learning rate based at least in part on the current learning rate control value;
• and
• determining an updated set of values for the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to the current effective learning rate.

One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:

• for each of a plurality of iterations:
• determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters;
• determining a current learning rate control value based on the gradient of the loss function, wherein the current learning rate control value equals a most recent learning rate control value minus an update value, wherein the update value is equal to a square of the gradient of the loss function multiplied by a sign function applied to the most recent learning rate control value minus the square of the gradient of the loss function and multiplied by a scaling coefficient that is equal to one minus an update scaling parameter;
• determining, by the one or more computing devices, a current effective learning rate based at least in part on the current learning rate control value;
• and
• updating at least one of the plurality of parameters of the machine-learned model based at least in part on the gradient of the loss function and according to a current effective learning rate that is a function of the current learning rate control value.