Training neural networks using a variational information bottleneck

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• receiving training data, the training data comprising a plurality of training inputs and, for each training input, a respective target output;
• training a neural network on the training data, wherein the neural network is configured to:
• receive a network input,
• generate a latent representation of the network input, comprising:
• processing the network input through one or more initial neural network layers of the neural network to generate an intermediate output that defines a distribution over latent representations;
• and
• sampling the latent representation of the network input from the distribution defined by the intermediate output, and
• process the latent representation through one or more additional neural network layers of the neural network to generate a network output from the network input,
• wherein training the neural network on the training data comprises training the neural network on the training data to have improved generalization to inputs outside the training data by, for each training input, performing an iteration of stochastic gradient descent on a lower bound of a variational information bottleneck objective to determine an update to current values of parameters of the neural network, wherein the lower bound is represented as an objective function to be minimized that satisfies, for a given training input xn:
• 1/N([−log(q(yn|ƒ(xn,∈))]+βKL[p(Z|xn),r(Z)]),
• where N is the total number of training inputs in the set of training data, q(yn|ƒ(xn,∈)) is a score assigned to the target output yn for the training input xn by the network output for the training input xn, ∈ is noise sampled from a noise distribution, ƒ(xn,∈) is the latent representation generated from the sampled noise and the intermediate output for the training input xn, β is a fixed constant value, KL is the Kullback-Leibler divergence, p(Z|xn) is the probability distribution over latent representations Z defined by the intermediate output, and r(Z) is a variational approximation of a marginal distribution of the latent representations Z;
• and
• providing data specifying the trained neural network for use in processing new network inputs.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving training data, the training data comprising a plurality of training inputs and, for each training input, a respective target output;
• training a neural network on the training data, wherein the neural network is configured to:
• receive a network input,
• generate a latent representation of the network input, comprising:
• processing the network input through one or more initial neural network layers of the neural network to generate an intermediate output that defines a distribution over latent representations;
• and
• sampling the latent representation of the network input from the distribution defined by the intermediate output, and
• process the latent representation through one or more additional neural network layers of the neural network to generate a network output from the network input,
• wherein training the neural network on the training data comprises training the neural network on the training data to have improved generalization to inputs outside the training data by, for each training input, performing an iteration of stochastic gradient descent on a lower bound of a variational information bottleneck objective to determine an update to current values of parameters of the neural network, wherein the lower bound is represented as an objective function to be minimized that satisfies, for a given training input xn:
• 1/N([−log(q(yn|ƒ(xn,∈))]+βKL[p(Z|xn),r(Z)]),
• where N is the total number of training inputs in the set of training data, q(yn|ƒ(xn,∈)) is a score assigned to the target output yn for the training input xn by the network output for the training input xn, ∈ is noise sampled from a noise distribution, ƒ(xn,∈) is the latent representation generated from the sampled noise and the intermediate output for the training input xn, β a is a fixed constant value, KL is the Kullback-Leibler divergence, p(Z|xn) is the probability distribution over latent representations Z defined by the intermediate output, and r(Z) is a variational approximation of a marginal distribution of the latent representations Z;
• and
• providing data specifying the trained neural network for use in processing new network inputs.