Training neural networks using learned optimizers

One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network configured to perform a machine learning task by processing a network input in accordance with at least a set of parameter tensors each including a plurality of respective parameters to generate a network output for the machine learning task, the operations comprising:

• performing, using a plurality of training examples, a training step to obtain respective gradients of a loss function for the machine learning task with respect to each of the parameters in the parameter tensors;
• obtaining a validation loss that measures a performance of the neural network on the machine learning task for a plurality of validation examples that are different from the plurality of training examples;
• generating an optimizer input from at least the respective gradients and the validation loss;
• processing the optimizer input using an optimizer neural network to generate an output defining a respective update for each of the parameters in the parameter tensors;
• and
• for each of the parameters in the parameter tensors, applying the respective update to a current value of the parameter to generate an updated value for the parameter.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network configured to perform a machine learning task by processing a network input in accordance with at least a set of parameter tensors each including a plurality of respective parameters to generate a network output for the machine learning task, the operations comprising:

• performing, using a plurality of training examples, a training step to obtain respective gradients of a loss function for the machine learning task with respect to each of the parameters in the parameter tensors;
• obtaining a validation loss that measures a performance of the neural network on the machine learning task for a plurality of validation examples that are different from the plurality of training examples;
• generating an optimizer input from at least the respective gradients and the validation loss;
• processing the optimizer input using an optimizer neural network to generate an output defining a respective update for each of the parameters in the parameter tensors;
• and
• for each of the parameters in the parameter tensors, applying the respective update to a current value of the parameter to generate an updated value for the parameter.