Regularized neural network architecture search

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples;
• and
• determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
• maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) data specifying how recently a neural network having the candidate architecture has been trained while determining the optimized neural network architecture, and
• repeatedly performing the following operations using each of a plurality of worker computing units each operating asynchronously from each other worker computing unit:
• selecting, by the worker computing unit, a plurality of candidate architectures from the population,
• training, for each selected candidate architecture and by the worker computing unit, a new neural network having the candidate architecture on a training subset of the training data to determine trained values of parameters of the new neural network;
• determining, for each selected candidate architecture and by the worker computing unit, a measure of fitness by evaluating a performance of the trained new neural network on a validation subset of the training data,
• generating, by the worker computing unit, a new candidate architecture based on the selected candidate architecture having the best measure of fitness, wherein generating the new candidate architecture comprises:
• randomly selecting a mutation from a set of mutations, the set of mutations comprising a hidden state mutation that:
• randomly selects one of a plurality of cells of the selected candidate architecture, the selected cell being associated with one or more of pairwise combinations, each of the one or more of pairwise combinations comprising two network operations,
• selects one of the one or more of pairwise combinations uniformly at random,
• selects one of two network operations of the selected pairwise combination uniformly at random, and
• replaces a hidden state associated with the selected network operation with another existing hidden state from within the selected cell, and
• applying the randomly selected mutation to the selected candidate architecture,
• adding the new candidate architecture to the population, and
• removing from the population the candidate architecture that was trained least recently.

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples;
• and
• determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
• maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) data specifying how recently a neural network having the candidate architecture has been trained while determining the optimized neural network architecture, and
• repeatedly performing the following operations using each of a plurality of worker computing units each operating asynchronously from each other worker computing unit:
• selecting, by the worker computing unit, a plurality of candidate architectures from the population,
• training, for each selected candidate architecture and by the worker computing unit, a new neural network having the candidate architecture on a training subset of the training data to determine trained values of parameters of the new neural network;
• determining, for each selected candidate architecture and by the worker computing unit, a measure of fitness by evaluating a performance of the trained new neural network on a validation subset of the training data,
• generating, by the worker computing unit, a new candidate architecture based on the selected candidate architecture having the best measure of fitness, wherein generating the new candidate architecture comprises:
• randomly selecting a mutation from a set of mutations, the set of mutations comprising a hidden state mutation that:
• randomly selects one of a plurality of cells of the selected candidate architecture, the selected cell being associated with one or more of pairwise combinations, each of the one or more of pairwise combinations comprising two network operations,
• selects one of one or more of pairwise combinations uniformly at random,
• selects one of two network operations of the selected pairwise combination uniformly at random, and
• replaces a hidden state associated with the selected network operation with another existing hidden state from within the selected cell, and
• applying the randomly selected mutation to the selected candidate architecture,
• adding the new candidate architecture to the population, and
• removing from the population the candidate architecture that was trained least recently.