Batch normalization layers

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving a neural network input;
• and
• generating a neural network output for the neural network input by processing the neural network input using a neural network system, wherein:
• the neural network system includes a sequence of neural network layers, wherein the sequence includes a third layer between a first neural network layer and a second neural network layer in the sequence of neural network layers of the neural network system,
• the neural network system generates the neural network output from the neural network input by processing the neural network input through each of the layers in the sequence, and
• the third layer is configured to:
• receive a first layer output generated for the neural network input by the first neural network layer, the first layer output comprising a plurality of components;
• normalize each component of the first layer output to generate a normalized output comprising a plurality of normalized components;
• generate a transformed version of the normalized output by transforming each normalized component of the normalized output in accordance with a set of normalization parameters for the third layer;
• and
• provide the transformed version of the normalized output as input to the second neural network layer.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving a neural network input;
• and
• generating a neural network output for the neural network input by processing the neural network input using a neural network system, wherein:
• the neural network system includes a sequence of neural network layers, wherein the sequence includes a third layer between a first neural network layer and a second neural network layer in the sequence of neural network layers of the neural network system,
• the neural network system generates the neural network output from the neural network input by processing the neural network input through each of the layers in the sequence, and
• the third layer is configured to:
• receive a first layer output generated for the neural network input by the first neural network layer, the first layer output comprising a plurality of components;
• normalize each component of the first layer output to generate a normalized output comprising a plurality of normalized components;
• generate a transformed version of the normalized output by transforming each normalized component of the normalized output in accordance with a set of normalization parameters for the third layer;
• and
• provide the transformed version of the normalized output as input to the second neural network layer.