Deep learning model training system

A computing device comprising:

• a processor;
• and
• a computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
• (A) randomly select observation vectors from a plurality of observation vectors, wherein a number of observation vectors selected is a mini-batch size value;
• (B) execute a forward and backward propagation of a neural network to compute a gradient vector and a weight vector using the selected observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
• (C) compute a search direction vector using the gradient vector;
• (D) compute a step size value using a line search with an objective function value that indicates an error measure of the executed neural network given the weight vector, the step size value, and the computed search direction vector;
• (E) compute an updated weight vector as a function of the weight vector, the computed step size value, and the computed search direction vector;
• (F) based on a predefined progress check frequency value,
• randomly select second observation vectors from the plurality of observation vectors;
• compute a progress check objective function value given the weight vector, the step size value, the computed search direction vector, and the randomly selected second observation vectors;
• and
• based on an accuracy test performed using the objective function value and the computed progress check objective function value, update the mini-batch size value;
• (G) repeat (A) to (F) until a convergence parameter value indicates training of the neural network is complete, wherein the weight vector for a next iteration is the computed updated weight vector;
• and
• output the computed updated weight vector to describe a trained neural network model.

A method of training a neural network model, the method comprising:

• (A) randomly selecting, by a computing device, observation vectors from a plurality of observation vectors, wherein a number of observation vectors selected is a mini-batch size value;
• (B) executing, by the computing device, a forward and backward propagation of a neural network to compute a gradient vector and a weight vector using the selected observation vectors, wherein the neural network includes a layer type for each layer of a plurality of neural network layers;
• (C) computing, by the computing device, a search direction vector using the gradient vector;
• (D) computing, by the computing device, a step size value using a line search with an objective function value that indicates an error measure of the executed neural network given the weight vector, the step size value, and the computed search direction vector;
• (E) computing, by the computing device, an updated weight vector as a function of the weight vector, the computed step size value, and the computed search direction vector;
• (F) based on a predefined progress check frequency value,
• randomly selecting, by the computing device, second observation vectors from the plurality of observation vectors;
• computing, by the computing device, a progress check objective function value given the weight vector, the step size value, the computed search direction vector, and the randomly selected second observation vectors;
• and
• based on an accuracy test performed using the objective function value and the computed progress check objective function value, updating, by the computing device, the mini-batch size value;
• (G) repeating, by the computing device, (A) to (F) until a convergence parameter value indicates training of the neural network is complete, wherein the weight vector for a next iteration is the computed updated weight vector;
• and
• outputting, by the computing device, the computed updated weight vector to describe a trained neural network model.