Computationally efficient neural network architecture search

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations comprising:

• receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples;
• and
• searching for an optimized neural network architecture for performing the machine learning task, comprising:
• maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) a measure of fitness representing the performance of the candidate architecture;
• maintaining threshold data comprising a set of threshold fitness values, wherein each threshold fitness value is associated with a corresponding checkpoint during the training of the neural network, each checkpoint being a different specific point over a course of the training of the neural network;
• and
• repeatedly performing the following operations:
• selecting one or more candidate architectures from the population data;
• generating a new architecture from the one or more selected candidate architectures;
• for the new architecture:
• training a neural network having the new architecture at least until reaching a first checkpoint of the plurality of checkpoints and until termination criteria for the training are satisfied, wherein the termination criteria specify that at each checkpoint, the training is terminated at the checkpoint and will not be resumed at any other checkpoints that are after the checkpoint whenever a measure of fitness of the neural network having the new architecture as of the checkpoint does not satisfy the threshold fitness value corresponding to the checkpoint;
• and
• determining a final measure of fitness of the neural network having the new architecture after the training;
• and
• adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations comprising:

• receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples;
• and
• searching for an optimized neural network architecture for performing the machine learning task, comprising:
• maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) a measure of fitness representing the performance of the candidate architecture;
• maintaining threshold data comprising a set of threshold fitness values, wherein each threshold fitness value is associated with a corresponding checkpoint during the training of the neural network, each checkpoint being a different specific points over a course of the training of the neural network;
• and
• repeatedly performing the following operations:
• selecting one or more candidate architectures from the population data;
• generating a new architecture from the one or more selected candidate architectures;
• for the new architecture:
• training a neural network having the new architecture at least until reaching a first checkpoint of the plurality of checkpoints and until termination criteria for the training are satisfied, wherein the termination criteria specify that at each checkpoint, the training is terminated at the checkpoint and will not be resumed at any other checkpoints that are after the checkpoint whenever a measure of fitness of the neural network having the new architecture as of the checkpoint does not satisfy the threshold fitness value corresponding to the checkpoint;
• and
• determining a final measure of fitness of the neural network having the new architecture after the training;
• and
• adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.