# US 12,001,509 B2 · Fast adaptive optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,509 B2
- Document ID: US12001509B2
- Status: Granted
- Kind: B2
- Application number: 16/821,509
- Filed: 2020-03-17
- Published: 2021-09-23
- Granted: 2024-06-04
- Earliest claimed: 2020-03-17
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Seungyeon Kim; Jingzhao Zhang; Andreas Veit; Sanjiv Kumar; Sashank Reddi; Praneeth Karimireddy
- Web page: https://patentgrep.com/patent/US12001509B2

## CPC

- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive per coordinate clipping threshold to clip a current first moment of the coordinate to obtain a current update value that enables faster convergence for the machine-learned model when the noise in the stochastic gradients is heavy tailed.

## Claims

### Claim 1

A computer-implemented method for training machine-learned models with improved convergence properties, the method comprising:
training, by one or more computing devices, a machine-learned model by:
determining, by the one or more computing devices, a gradient of a loss function that evaluates a performance of the machine-learned model, wherein the gradient of the loss function comprises a plurality of coordinates that respectively correspond to a plurality of parameters of the machine-learned model;
determining, by the one or more computing devices, a respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function, wherein the respective adaptive per-coordinate threshold for each coordinate is based at least in part on a previous adaptive per-coordinate threshold for the coordinate and the gradient of the loss function;
clipping, by the one or more computing devices, a respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate;
generating, by the one or more computing devices, a plurality of current update values respectively for the plurality of coordinates using, for each of the plurality of coordinates, the clipped respective current first moment for the coordinate; and
updating, by the one or more computing devices, the plurality of parameters of the machine-learned model using the plurality of current update values.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining, by the one or more computing devices, the respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function comprises determining the respective adaptive per-coordinate threshold based on a sum of an exponential average of past and current gradients and a previous adaptive per-coordinate threshold for the coordinate.

### Claim 3 (depends on 1)

The computer implemented method of claim 1, wherein clipping, by the one or more computing devices, the respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the respective adaptive per-coordinate threshold divided by a value indicative of the respective current first moment for the coordinate is less than 1, setting the current update value for the coordinate equal to the respective adaptive per-coordinate threshold for the coordinate times the respective current first moment for the coordinate divided by the value indicative of the respective current first moment for the coordinate.

### Claim 4 (depends on 1)

The computer implemented method of claim 1, wherein clipping, by the one or more computing devices, the respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the respective adaptive per-coordinate threshold divided by a value indicative of the current first moment for the coordinate is greater than 1, setting the current update value for the coordinate equal to the respective current first moment for the coordinate.

### Claim 5 (depends on 1)

The computer implemented method of claim 1, wherein the respective current first moment for each of the plurality of coordinates is based at least in part on a respective previous current moment for the coordinate and the gradient of the loss function.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein determining, by the one or more computing devices, the respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient descent of the loss function is based at least in part on an exponential moving average of past gradients for the coordinate.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising: performing, by the one or more computing devices, the method of claim 1 for each of a plurality of iterations.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the learning rate is constant for each iteration.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein clipping comprises placing a bound on an exponential moving average of past gradients with an initial value of an alpha moment as 1.

### Claim 10 (depends on 1)

The computer implemented method of claim 1, wherein a noise distribution for the gradient of the loss function is heavy-tailed.

### Claim 11 (depends on 1)

The computer implemented method of claim 1, wherein the machine-learned model is an attention model.

### Claim 12

A computer system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
training, by one or more computing devices, a machine-learned model by:
determining a gradient of a loss function that evaluates a performance of a machine-learned model that comprises a plurality of parameters, wherein the gradient of the loss function comprises a plurality of coordinates that respectively correspond to the plurality of parameters;
determining a respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function, wherein the respective adaptive per-coordinate threshold for each coordinate is based at least in part on a previous adaptive per-coordinate threshold for the coordinate and the gradient of the loss function;
clipping a respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate;
generating, by the one or more computing devices, a plurality of current update values respectively for the plurality of coordinates using, for each of the plurality of coordinates, the clipped respective current first moment for the coordinate; and
updating, by the one or more computing devices, the plurality of parameters of the machine-learned model using the plurality of current update values.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein determining the respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function comprises determining the respective adaptive per-coordinate threshold based on a sum of an exponential average of past and current gradients and a previous adaptive per-coordinate threshold.

### Claim 14 (depends on 12)

The computing system of claim 12, wherein clipping the respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the respective adaptive per-coordinate threshold divided by a value indicative of the respective current first moment for the coordinate is less than 1, setting the current update value for the coordinate equal to the respective adaptive per-coordinate threshold for the coordinate times the respective current first moment for the coordinate divided by the value indicative of the respective current first moment for the coordinate.

### Claim 15 (depends on 12)

The computing system of claim 12, wherein clipping the respective current first moment for each of the plurality of coordinates of the gradient of the loss function based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the respective adaptive per-coordinate threshold divided by a value indicative of the current first moment for the coordinate is greater than 1, setting the current update value for the coordinate equal to the respective current first moment for the coordinate.

### Claim 16 (depends on 12)

The computing system of claim 12, wherein the respective current first moment for each of the plurality of coordinates is based at least in part on a respective previous current moment for the coordinate and the gradient of the loss function.

### Claim 17

One or more non-transitory computer-readable media that store a machine-learned model that has been trained through execution of operations, the operations comprising:
training, by one or more computing devices, a machine-learned model by:
for each of a plurality of iterations:
determining a gradient of a loss function that evaluates a performance of the machine-learned model that comprises a plurality of parameters, wherein the gradient of the loss function comprises a plurality of coordinates that respectively correspond to the plurality of parameters;
determining a respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function, wherein the respective adaptive per-coordinate threshold for each coordinate is based at least in part on a previous adaptive per-coordinate threshold for the coordinate and the gradient of the loss function;
clipping a respective current first moment for each of the plurality of coordinates based at least in part on the respective adaptive per-coordinate threshold for the coordinate;
generating, by the one or more computing devices, a plurality of current update values respectively for the plurality of coordinates using, for each of the plurality of coordinates, the clipped respective current first moment for the coordinate; and
updating, by the one or more computing devices, the plurality of parameters of the machine-learned model using the plurality of current update values.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein determining the respective adaptive per-coordinate threshold for each of the plurality of coordinates of the gradient of the loss function comprises determining the respective adaptive per-coordinate threshold based on a sum of an exponential average of past and current gradients and a previous adaptive per-coordinate threshold.

### Claim 19 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein clipping the respective current first moment for each of the plurality of coordinates based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the adaptive per-coordinate threshold divided by a value indicative of the respective current first moment for the coordinate is less than 1, setting the current update value for the coordinate equal to the respective adaptive per-coordinate threshold for the coordinate times the respective current first moment for the coordinate divided by the value indicative of the respective current first moment for the coordinate.

### Claim 20 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein clipping the respective current first moment for each of the plurality of coordinates based at least in part on the respective adaptive per-coordinate threshold for the coordinate to obtain the current update value for the coordinate comprises, upon determining that the respective adaptive per-coordinate threshold divided by a value indicative of the current first moment for the coordinate is greater than 1, setting the adaptive per-coordinate threshold for said coordinate equal to the respective current first moment for the coordinate.