Adjusting precision and topology parameters for neural network training based on a performance metric

A method of operating a computing system implementing a neural network, the method comprising:

• with the computing system, dynamically updating a parameter of the neural network during training of the neural network, the training of the neural network comprising:
• for each of multiple iterations in an epoch of training of the neural network, in a given iteration of the multiple iterations:
• for a first set of one or more training values, training at least one layer of the neural network by forward propagating and backward propagating activation or gradient values, respectively, for a number of training epochs using a first parameter value for a parameter of the neural network;
• determining a performance metric for the neural network during the training of the neural network, the performance metric being selected from accuracy of the neural network, entropy of the neural network, mean square error of the neural network, perplexity of the neural network, or gradient signal to noise ratio of the neural network;
• for the given iteration, using the performance metric, dynamically adjusting the parameter of the neural network during the training of the neural network to provide an adjusted parameter of the neural network and updating the neural network using the adjusted parameter to produce an updated neural network;
• and
• with the updated neural network and during the training of the neural network, performing additional training for at least one layer of the neural network by forward propagating and backward propagating activation or gradient values, respectively, for at least one additional training epoch using a second set of one or more training values in another iteration of the multiple iterations.

One or more non-transitory computer-readable storage devices or media comprising:

• computer-executable instructions that, when executed by a computing system comprising at least one processor and at least one memory coupled to the at least one processor, cause the computer system to implement a first layer of the neural network using first node weights and/or first activation values expressed in a first floating-point format;
• computer-executable instructions that, when executed by the computing system, cause the computer system to, during training of the neural network, for a first set of one or more training values, forward propagate values from the first layer of the neural network to a second layer of the neural network using a first value for a parameter of the neural network, wherein the parameter is a precision parameter or a topology parameter, and to perform backward propagation for at least one layer of the neural network using the first parameter value;
• computer-executable instructions that, when executed by the computing system, cause the computer system to, for the given iteration, a performance metric for the neural network, the performance metric being selected from accuracy of the neural network, entropy of the neural network, mean square error of the neural network, perplexity of the neural network, or gradient signal to noise ratio of the neural network, determine a training performance metric for the neural network;
• computer-executable instructions that, when executed by the computing system, cause the computer system to dynamically adjust the parameter for the neural network based on the performance metric determined for the given iteration to provide an adjusted parameter, the adjusted parameter being selected to improve the performance metric in a neural network having the adjusted parameter;
• and
• computer-executable instructions that, when executed by the computing system, cause the computer system to modify the neural network during the training of the neural network based on the adjusted precision parameter;
• computer-executable instructions that, when executed by the computing system, cause the computer system to, for a second set of one or more training values in another iteration of the training, perform additional forward and backward propagation operations for at least one layer of the neural network using the adjusted parameter.