# US 10,769,528 B1 · Deep learning model training system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,528 B1
- Document ID: US10769528B1
- Status: Granted
- Kind: B1
- Application number: 16/590,544
- Filed: 2019-10-02
- Granted: 2020-09-08
- Earliest claimed: 2019-06-07
- Assignees: SAS Institute Inc.
- Applicants: SAS Institute Inc.
- Inventors: Ben-hao Wang; Joshua David Griffin; Seyedalireza Yektamaram; Yan Xu
- Web page: https://patentgrep.com/patent/US10769528B1

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer trains a neural network model. (B) A neural network is executed to compute a post-iteration gradient vector and a current iteration weight vector. (C) A search direction vector is computed using a Hessian approximation matrix and the post-iteration gradient vector. (D) A step size value is initialized. (E) An objective function value is computed that indicates an error measure of the executed neural network. (F) When the computed objective function value is greater than an upper bound value, the step size value is updated using a predefined backtracking factor value. The upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values. (G) (E) and (F) are repeated until the computed objective function value is not greater than the upper bound value. (H) An updated weight vector is computed to describe a trained neural network model.

## Claims

### Claim 1

A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:
(A) select a batch of observation vectors, wherein the batch of observation vectors includes a mini-batch size value number of observation vectors selected from a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a value for each variable of a plurality of variables;
(B) execute a neural network to compute a post-iteration gradient vector and a current iteration weight vector using the selected batch of observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
(C) compute a search direction vector using a Hessian approximation matrix and the post-iteration gradient vector;
(D) initialize a step size value with a predefined step size value;
(E) compute an objective function value that indicates an error measure of the executed neural network given the current iteration weight vector, the step size value, and the computed search direction vector;
(F) when the computed objective function value is greater than an upper bound value, update the step size value using a predefined backtracking factor value, wherein the upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values;
(G) repeat (E) and (F) until the computed objective function value is not greater than the upper bound value;
(H) compute an updated weight vector as a function of the current iteration weight vector, the updated step size value, and the computed search direction vector;
(I) repeat (A) to (H) until a convergence parameter value indicates training of the neural network is complete, wherein the current iteration weight vector for a next iteration is the computed updated weight vector; and
output the computed updated weight vector to describe a trained neural network model.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the upper bound value is computed using
U
k
+
1
=
0
.
1
⁢
(
1
t
U
⁢
∑
i
=
1
t
U
⁢
U
k
-
i
)
+
0
.
9
⁢
U
0
,
where Uk+1 is the upper bound value, tU is the predefined upper bound updating interval value, Uk−i is a previous upper bound value for iteration k−i, k is an index to a current iteration, and U0 is a predefined initial upper bound value.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the step size value is updated using αk=ταk, where αk is the step size value, and τ is the predefined backtracking factor value.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the predefined backtracking factor value is between zero and one.

### Claim 5 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the predefined backtracking factor value is τ=ln (2).

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the predefined step size value is one.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein, after (H) and before (I), the computer-readable instructions further cause the computing device to update the upper bound value when mod(k, tU)=0, where mod indicates the modulus function, and tU is the predefined upper bound updating interval value.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the convergence parameter value is a number of iterations of (I).

### Claim 9 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the Hessian approximation matrix is computed using gk−gk−1, where gk is the post-iteration gradient vector of a current iteration of (B), and gk−1 is the post-iteration gradient vector of a previous iteration of (B).

### Claim 10 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein, after (I), the computer-readable instructions further cause the computing device to:
read a new observation vector from a dataset;
input the read new observation vector to the trained neural network model to predict a characteristic value of the read new observation vector; and
output the predicted characteristic value.

### Claim 11 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein, before (A), the computer-readable instructions further cause the computing device to:
(J) select a second batch of observation vectors that includes the mini-batch size value number of observation vectors selected from the plurality of observation vectors;
(K) execute the neural network to compute an initial post-iteration gradient vector and the current iteration weight vector using the selected second batch of observation vectors and an initial weight vector;
(L) compute the search direction vector using an initial Hessian approximation matrix and the initial post-iteration gradient vector;
(M) initialize the step size value to the predefined step size value;
(N) compute the objective function value given the current iteration weight vector computed in (K), the step size value, and the search direction vector computed in (L);
(O) when the computed objective function value is greater than the upper bound value, update the step size value using the predefined backtracking factor value;
repeat (N) and (O) until the computed objective function value is not greater than the upper bound value; and
compute the updated weight vector as the function of the current iteration weight vector computed in (K), the updated step size value computed in (O), and the search direction vector computed in (L), wherein the current iteration weight vector in (B) is the computed updated weight vector.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein, after (A) and before (B), the computer-readable instructions further cause the computing device to:
determine when it is time to compute a pre-iteration gradient vector based on a predefined gradient updating interval value; and
when it is time to compute the pre-iteration gradient vector, execute the neural network to compute the pre-iteration gradient vector using the selected batch of observation vectors and the current iteration weight vector,
wherein the Hessian approximation matrix in (C) is computed using gk−gk−1 when it is not time to compute the pre-iteration gradient vector, where gk is the post-iteration gradient vector of a current iteration of (B), and gk−1 is the post-iteration gradient vector of a previous iteration of (B),
wherein the Hessian approximation matrix in (C) is computed using gk−ĝk when it is time to compute the pre-iteration gradient vector, where ĝk is the pre-iteration gradient vector,
wherein the search direction vector in (C) is computed using gk when it is not time to compute the pre-iteration gradient vector,
wherein the search direction vector in (C) is computed using ĝk when it is time to compute the pre-iteration gradient vector.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein, computing the Hessian approximation matrix in (C) when it is time to compute the pre-iteration gradient vector comprises:
computing sk+1=αk pk, where αk is the step size value, and pk is the search direction vector computed in (L); and
computing yk+1=gk−ĝk; and
computing the Hessian approximation matrix for a next iteration of (A) based on sk+1 and yk+1.

### Claim 14 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein, computing the Hessian approximation matrix in (C) when it is not time to compute the pre-iteration gradient vector comprises:
computing Hk+1=Hk, where Hk+1 is the Hessian approximation matrix for a next iteration of (A), and Hk is a current Hessian approximation matrix.

### Claim 15

A computing device comprising:
a processor; and
a computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
(A) select a batch of observation vectors, wherein the batch of observation vectors includes a mini-batch size value number of observation vectors selected from a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a value for each variable of a plurality of variables;
(B) execute a neural network to compute a post-iteration gradient vector and a current iteration weight vector using the selected batch of observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
(C) compute a search direction vector using a Hessian approximation matrix and the post-iteration gradient vector;
(D) initialize a step size value with a predefined step size value;
(E) compute an objective function value that indicates an error measure of the executed neural network given the current iteration weight vector, the step size value, and the computed search direction vector;
(F) when the computed objective function value is greater than an upper bound value, update the step size value using a predefined backtracking factor value, wherein the upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values;
(G) repeat (E) and (F) until the computed objective function value is not greater than the upper bound value;
(H) compute an updated weight vector as a function of the current iteration weight vector, the updated step size value, and the computed search direction vector;
(I) repeat (A) to (H) until a convergence parameter value indicates training of the neural network is complete, wherein the current iteration weight vector for a next iteration is the computed updated weight vector; and
output the computed updated weight vector to describe a trained neural network model.

### Claim 16 (depends on 15)

The computing device of claim 15, wherein the upper bound value is computed using
U
k
+
1
=
0
.
1
⁢
(
1
t
U
⁢
∑
i
=
1
t
U
⁢
U
k
-
i
)
+
0
.
9
⁢
U
0
,
where Uk+1 is the upper bound value, tU is the predefined upper bound updating interval value, Uk−i is a previous upper bound value for iteration k−i, k is an index to a current iteration, and U0 is a predefined initial upper bound value.

### Claim 17

A method of training a neural network model, the method comprising:
(A) selecting, by a computing device, a batch of observation vectors, wherein the batch of observation vectors includes a mini-batch size value number of observation vectors selected from a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a value for each variable of a plurality of variables;
(B) executing, by the computing device, a neural network to compute a post-iteration gradient vector and a current iteration weight vector using the selected batch of observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
(C) computing, by the computing device, a search direction vector using a Hessian approximation matrix and the post-iteration gradient vector;
(D) initializing, by the computing device, a step size value with a predefined step size value;
(E) computing, by the computing device, an objective function value that indicates an error measure of the executed neural network given the current iteration weight vector, the step size value, and the computed search direction vector;
(F) when the computed objective function value is greater than an upper bound value, updating, by the computing device, the step size value using a predefined backtracking factor value, wherein the upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values;
(G) repeating, by the computing device, (E) and (F) until the computed objective function value is not greater than the upper bound value;
(H) computing, by the computing device, an updated weight vector as a function of the current iteration weight vector, the updated step size value, and the computed search direction vector;
(I) repeating, by the computing device, (A) to (H) until a convergence parameter value indicates training of the neural network is complete, wherein the current iteration weight vector for a next iteration is the computed updated weight vector; and
outputting, by the computing device, the computed updated weight vector to describe a trained neural network model.

### Claim 18 (depends on 17)

The method of claim 17, wherein the upper bound value is computed using
U
k
+
1
=
0
.
1
⁢
(
1
t
U
⁢
∑
i
=
1
t
U
⁢
U
k
-
i
)
+
0
.
9
⁢
U
0
,
where Uk+1 is the upper bound value, tU is the predefined upper bound updating interval value, Uk−i is a previous upper bound value for iteration k−i, k is an index to a current iteration, and U0 is a predefined initial upper bound value.

### Claim 19 (depends on 17)

The method of claim 17, wherein the step size value is updated using αk=ταk, where αk is the step size value, and τ is the predefined backtracking factor value.

### Claim 20 (depends on 19)

The method of claim 19, wherein the predefined backtracking factor value is between zero and one.

### Claim 21 (depends on 19)

The method of claim 19, wherein the predefined backtracking factor value is τ=ln (2).

### Claim 22 (depends on 17)

The method of claim 17, wherein the predefined step size value is one.

### Claim 23 (depends on 17)

The method of claim 17, wherein, after (H) and before (I), further comprising updating, by the computing device, the upper bound value when mod(k, tU)=0, where mod indicates the modulus function, and tU is the predefined upper bound updating interval value.

### Claim 24 (depends on 17)

The method of claim 17, wherein the convergence parameter value is a number of iterations of (I).

### Claim 25 (depends on 17)

The method of claim 17, wherein the Hessian approximation matrix is computed using gk−gk−1, where gk is the post-iteration gradient vector of a current iteration of (B), and gk−1 is the post-iteration gradient vector of a previous iteration of (B).

### Claim 26 (depends on 17)

The method of claim 17, wherein, after (I), further comprising:
reading, by the computing device, a new observation vector from a dataset;
inputting, by the computing device, the read new observation vector to the trained neural network model to predict a characteristic value of the read new observation vector; and
outputting, by the computing device, the predicted characteristic value.

### Claim 27 (depends on 17)

The method of claim 17, wherein, before (A), further comprising:
(J) selecting, by the computing device, a second batch of observation vectors that includes the mini-batch size value number of observation vectors selected from the plurality of observation vectors;
(K) executing, by the computing device, the neural network to compute an initial post-iteration gradient vector and the current iteration weight vector using the selected second batch of observation vectors and an initial weight vector;
(L) computing, by the computing device, the search direction vector using an initial Hessian approximation matrix and the initial post-iteration gradient vector;
(M) initializing, by the computing device, the step size value to the predefined step size value;
(N) computing, by the computing device, the objective function value given the current iteration weight vector computed in (K), the step size value, and the search direction vector computed in (L);
(O) when the computed objective function value is greater than the upper bound value, updating, by the computing device, the step size value using the predefined backtracking factor value;
repeating, by the computing device, (N) and (O) until the computed objective function value is not greater than the upper bound value; and
computing, by the computing device, the updated weight vector as the function of the current iteration weight vector computed in (K), the updated step size value computed in (O), and the search direction vector computed in (L), wherein the current iteration weight vector in (B) is the computed updated weight vector.

### Claim 28 (depends on 27)

The method of claim 27, wherein, after (A) and before (B), further comprising:
determining, by the computing device, when it is time to compute a pre-iteration gradient vector based on a predefined gradient updating interval value; and
when it is time to compute the pre-iteration gradient vector, executing, by the computing device, the neural network to compute the pre-iteration gradient vector using the selected batch of observation vectors and the current iteration weight vector,
wherein the Hessian approximation matrix in (C) is computed using gk−gk−1 when it is not time to compute the pre-iteration gradient vector, where gk is the post-iteration gradient vector of a current iteration of (B), and gk−1 is the post-iteration gradient vector of a previous iteration of (B),
wherein the Hessian approximation matrix in (C) is computed using gk−ĝk when it is time to compute the pre-iteration gradient vector, where ĝk is the pre-iteration gradient vector,
wherein the search direction vector in (C) is computed using gk when it is not time to compute the pre-iteration gradient vector,
wherein the search direction vector in (C) is computed using ĝk when it is time to compute the pre-iteration gradient vector.

### Claim 29 (depends on 28)

The method of claim 28, wherein, computing the Hessian approximation matrix in (C) when it is time to compute the pre-iteration gradient vector comprises:
computing sk+1=αk pk, where αk is the step size value, and pk is the search direction vector computed in (L); and
computing yk+1=gk−ĝk; and
computing the Hessian approximation matrix for a next iteration of (A) based on sk+1 and yk+1.

### Claim 30 (depends on 28)

The method of claim 28, wherein, computing the Hessian approximation matrix in (C) when it is not time to compute the pre-iteration gradient vector comprises:
computing Hk+1=Hk, where Hk+1 is the Hessian approximation matrix for a next iteration of (A), and Hk is a current Hessian approximation matrix.