Fast adaptive optimization

A computer system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
• training, by one or more computing devices, a machine-learned model by:
• determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters, wherein the gradient of the loss function comprises a plurality of coordinates that respectively correspond to the plurality of parameters;
• determining a respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function, wherein the respective adaptive per-coordinate threshold for each coordinate is based at least in part on a previous adaptive per-coordinate threshold for the coordinate and the gradient of the loss function;
• clipping a respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate;
• generating, by the one or more computing devices, a plurality of current update values respectively for the plurality of coordinates using, for each of the plurality of coordinates, the clipped respective current first moment for the coordinate;
• and
• updating, by the one or more computing devices, the plurality of parameters of the machine-learned model using the plurality of current update values.

One or more non-transitory computer-readable media that store a machine-learned model that has been trained through execution of operations, the operations comprising:

• training, by one or more computing devices, a machine-learned model by:
• for each of a plurality of iterations:
• determining a gradient of a loss function that evaluates a performance of the machine-learned model that comprises a plurality of parameters, wherein the gradient of the loss function comprises a plurality of coordinates that respectively correspond to the plurality of parameters;
• determining a respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function, wherein the respective adaptive per-coordinate threshold for each coordinate is based at least in part on a previous adaptive per-coordinate threshold for the coordinate and the gradient of the loss function;
• clipping a respective current first moment for each of the plurality of coordinates based at least in part on the respective adaptive per-coordinate threshold for the coordinate;
• generating, by the one or more computing devices, a plurality of current update values respectively for the plurality of coordinates using, for each of the plurality of coordinates, the clipped respective current first moment for the coordinate;
• and
• updating, by the one or more computing devices, the plurality of parameters of the machine-learned model using the plurality of current update values.