Whitened neural network layers

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a neural network system that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input, the sequence of neural network layers comprising:

• a whitened neural network layer, wherein the neural network layer is configured to perform operations comprising:
• receiving an input activation generated by a layer before the whitened neural network layer in the sequence;
• processing the received activation in accordance with a set of whitening parameters to generate a whitened activation;
• processing the whitened activation in accordance with a set of layer parameters to generate an output activation, wherein:
• the set of whitening parameters and the set of layer parameters have been determined during training of the neural network layers to adjust the whitening parameters and the layer parameters,
• the training comprises repeatedly (i) adjusting the values of the whitening parameters based at least in part on an eigen decomposition of a matrix of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence and (ii) in response, adjusting the values of the set of layer parameters so that output activations generated by the whitened neural network layer remain unchanged;
• and
• providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence.

A method of training a neural network that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input,

• wherein the sequence of neural network layers comprises a whitened neural network layer,
• wherein the neural network layer is configured to receive an input activation generated by a layer before the whitened neural network layer in the sequence, process the received activation in accordance with a set of whitening parameters to generate a whitened activation, process the whitened activation in accordance with a set of layer parameters to generate an output activation, and provide the output activation as input to a neural network layer after the whitened neural network layer in the sequence, and wherein the method comprises:
• training the neural network on training data to adjust values of the set of layer parameters of the whitened neural network layer using a gradient descent technique;
• and
• during the training, (i) adjusting the values of the whitening parameters using an eigen decomposition of a matrix of an estimate of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence and (ii) in response to adjusting the values of the whitening parameters, further adjusting the values of the set of layer parameters so that output activations generated by the whitened neural network layer remain unchanged.

A system for training a neural network that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input,

• wherein the sequence of neural network layers comprises a whitened neural network layer,
• wherein the neural network layer is configured to receive an input activation generated by a layer before the whitened neural network layer in the sequence, process the received activation in accordance with a set of whitening parameters to generate a whitened activation, process the whitened activation in accordance with a set of layer parameters to generate an output activation, and provide the output activation as input to a neural network layer after the whitened neural network layer in the sequence, and wherein the system comprises one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
• training the neural network on training data to adjust values of the set of layer parameters of the whitened neural network layer using a gradient descent technique;
• and
• during the training, (i) adjusting the values of the whitening parameters using an eigen decomposition of a matrix of an estimate of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence and (ii) in response to adjusting the values of the whitening parameters, further adjusting the values of the set of layer parameters so that output activations generated by the whitened neural network layer remain unchanged.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input,

• wherein the sequence of neural network layers comprises a whitened neural network layer,
• wherein the neural network layer is configured to receive an input activation generated by a layer before the whitened neural network layer in the sequence, process the received activation in accordance with a set of whitening parameters to generate a whitened activation, process the whitened activation in accordance with a set of layer parameters to generate an output activation, and provide the output activation as input to a neural network layer after the whitened neural network layer in the sequence, and wherein the operations comprise:
• training the neural network on training data to adjust values of the set of layer parameters of the whitened neural network layer using a gradient descent technique;
• and
• during the training, (i) adjusting the values of the whitening parameters using an eigen decomposition of a matrix of an estimate of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence and (ii) in response to adjusting the values of the whitening parameters, further adjusting the values of the set of layer parameters so that output activations generated by the whitened neural network layer remain unchanged.