Training neural networks using normalized target outputs

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network on training data to generate normalized outputs that are mappable to un-normalized outputs in accordance with a set of normalization parameters, wherein the training data comprises a sequence of training items and, for each training item in the sequence, a respective target output associated with the training item, wherein the target output is an output expected to be generated by the neural network for the training item, and wherein the operations comprise, for each training item in the sequence:

• obtaining a target output for the training item from the training data;
• updating current values of the normalization parameters to account for magnitudes of target outputs for training inputs changing during training by updating the current values so that the normalized target outputs for the training items up to and including the training item in the sequence of training data have a specified distribution, wherein the normalization parameters are different from the main parameters of the neural network, wherein the normalization parameters includes a set of parameters that are used to normalize target outputs;
• determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values;
• processing the training item using the neural network to generate a normalized output of the neural network for the training item in accordance with current values of the main parameters of the neural network;
• determining an error for the training item using an objective function that takes as input (i) the normalized output generated for the training item by the neural network and (ii) the normalized target output generated by normalizing the target output associated with the training item in the training data;
• and
• using the error to adjust the current values of the main parameters of the neural network.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network having main parameters on training data to generate normalized outputs that are mappable to un-normalized outputs in accordance with a set of normalization parameters, wherein the training data comprises a sequence of training items and, for each training item in the sequence, a respective target output associated with the training item, wherein the target output is an output expected to be generated by the neural network for the training item, the operations comprising, for each training item in the sequence:

• obtaining a target output for the training item from the training data;
• updating current values of the normalization parameters to account for magnitudes of target outputs for training inputs changing during training by updating the current values so that the normalized target outputs for the training items up to and including the training item in the sequence of training data have a specified distribution wherein the normalization parameters are different from the main parameters of the neural network, wherein the normalization parameters includes a set of parameters that are used to normalize target outputs;
• determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values;
• processing the training item using the neural network to generate a normalized output of the neural network for the training item in accordance with current values of main parameters of the neural network;
• determining an error for the training item using an objective function that takes as input (i) the normalized output generated for the training item by the neural network and (ii) the normalized target output generated by normalizing the target output associated with the training item in the training data;
• and
• using the error to adjust the current values of the main parameters of the neural network.