Training neural networks using posterior sharpening

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters to determine trained values of the network parameters,

• wherein the neural network is configured to receive a network input and to process the network input in accordance with the network parameters to generate a network output,
• wherein the operations comprise:
• maintaining data specifying, for each of the network parameters, current values of a respective set of distribution parameters that define a posterior distribution over possible values for the network parameter;
• determining a respective temporary parameter value for each of the network parameters from the current values of the respective set of distribution parameters for the network parameter;
• obtaining a mini-batch comprising a plurality of training network inputs and, for each training network input, a respective training network output;
• determining a respective temporary gradient value for each of the network parameters by determining a gradient of an objective function for the mini-batch and with respect to the respective temporary parameter values;
• determining a respective current training value for each of the network parameters from the respective temporary gradient value for the network parameter;
• updating the current values of the respective sets of distribution parameters for the network parameters in accordance with the respective current training values for the network parameters;
• and
• determining the trained values of the network parameters based on the updated current values of the respective sets of distribution parameters.

A non-transitory computer program product storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters to determine trained values of the network parameters,

• wherein the neural network is configured to receive a network input and to process the network input in accordance with the network parameters to generate a network output,
• wherein the operations comprise:
• maintaining data specifying, for each of the network parameters, current values of a respective set of distribution parameters that define a posterior distribution over possible values for the network parameter;
• determining a respective temporary parameter value for each of the network parameters from the current values of the respective set of distribution parameters for the network parameter;
• obtaining a mini-batch comprising a plurality of training network inputs and, for each training network input, a respective training network output;
• determining a respective temporary gradient value for each of the network parameters by determining a gradient of an objective function for the mini-batch and with respect to the respective temporary parameter values;
• determining a respective current training value for each of the network parameters from the respective temporary gradient value for the network parameter;
• updating the current values of the respective sets of distribution parameters for the network parameters in accordance with the respective current training values for the network parameters;
• and
• determining the trained values of the network parameters based on the updated current values of the respective sets of distribution parameters.