Training neural networks by resetting dormant neurons

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network having network parameters and configured to process a network input to generate a network output for the network input, the neural network comprising:
• a plurality of neural network layers, the plurality of neural network layers comprising a first neural network layer having a plurality of neurons, and the network parameters comprising:
• for each of the neurons, a respective set of incoming weights associated with the neuron and a respective set of outgoing weights associated with the neuron, the operations comprising:
• at each of a plurality of training steps:
• obtaining a set of training data for the training step;
• training the neural network on the training data to update the network parameters;
• determining whether a resetting criterion is satisfied at the training step;
• in response to determining that the criterion is satisfied:
• determining, for each of the plurality of neurons, an expected absolute value of an activation generated by the neuron during processing of a given network input;
• determining, for each neuron and based on the expected absolute value for the neuron, whether to classify the neuron as a dormant neuron;
• and
• for any neuron that is classified as a dormant neuron, modifying the incoming weights associated with the dormant neuron and the outgoing weights associated with the dormant neuron.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network having network parameters and configured to process a network input to generate a network output for the network input, the neural network comprising:

• a plurality of neural network layers, the plurality of neural network layers comprising a first neural network layer having a plurality of neurons, and the network parameters comprising:
• for each of the neurons, a respective set of incoming weights associated with the neuron and a respective set of outgoing weights associated with the neuron, the operations comprising:
• at each of a plurality of training steps:
• obtaining a set of training data for the training step;
• training the neural network on the training data to update the network parameters;
• determining whether a resetting criterion is satisfied at the training step;
• in response to determining that the criterion is satisfied:
• determining, for each of the plurality of neurons, an expected absolute value of an activation generated by the neuron during processing of a given network input;
• determining, for each neuron and based on the expected absolute value for the neuron, whether to classify the neuron as a dormant neuron;
• and
• for any neuron that is classified as a dormant neuron, modifying the incoming weights associated with the dormant neuron and the outgoing weights associated with the dormant neuron.