Deep learning model training system

A computing device comprising:

• a processor;
• and
• a computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
• (A) select a batch of observation vectors, wherein the batch of observation vectors includes a mini-batch size value number of observation vectors selected from a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a value for each variable of a plurality of variables;
• (B) execute a neural network to compute a post-iteration gradient vector and a current iteration weight vector using the selected batch of observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
• (C) compute a search direction vector using a Hessian approximation matrix and the post-iteration gradient vector;
• (D) initialize a step size value with a predefined step size value;
• (E) compute an objective function value that indicates an error measure of the executed neural network given the current iteration weight vector, the step size value, and the computed search direction vector;
• (F) when the computed objective function value is greater than an upper bound value, update the step size value using a predefined backtracking factor value, wherein the upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values;
• (G) repeat (E) and (F) until the computed objective function value is not greater than the upper bound value;
• (H) compute an updated weight vector as a function of the current iteration weight vector, the updated step size value, and the computed search direction vector;
• (I) repeat (A) to (H) until a convergence parameter value indicates training of the neural network is complete, wherein the current iteration weight vector for a next iteration is the computed updated weight vector;
• and
• output the computed updated weight vector to describe a trained neural network model.

A method of training a neural network model, the method comprising:

• (A) selecting, by a computing device, a batch of observation vectors, wherein the batch of observation vectors includes a mini-batch size value number of observation vectors selected from a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a value for each variable of a plurality of variables;
• (B) executing, by the computing device, a neural network to compute a post-iteration gradient vector and a current iteration weight vector using the selected batch of observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
• (C) computing, by the computing device, a search direction vector using a Hessian approximation matrix and the post-iteration gradient vector;
• (D) initializing, by the computing device, a step size value with a predefined step size value;
• (E) computing, by the computing device, an objective function value that indicates an error measure of the executed neural network given the current iteration weight vector, the step size value, and the computed search direction vector;
• (F) when the computed objective function value is greater than an upper bound value, updating, by the computing device, the step size value using a predefined backtracking factor value, wherein the upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values;
• (G) repeating, by the computing device, (E) and (F) until the computed objective function value is not greater than the upper bound value;
• (H) computing, by the computing device, an updated weight vector as a function of the current iteration weight vector, the updated step size value, and the computed search direction vector;
• (I) repeating, by the computing device, (A) to (H) until a convergence parameter value indicates training of the neural network is complete, wherein the current iteration weight vector for a next iteration is the computed updated weight vector;
• and
• outputting, by the computing device, the computed updated weight vector to describe a trained neural network model.