Neural architecture search

A method of determining an architecture for a neural network for performing a particular neural network task, the method comprising:

• generating, in accordance with current values of a plurality of controller parameters, a batch of output sequences, each output sequence in the batch specifying a respective subset of a plurality of components of a large neural network that should be active during the processing of one or more inputs by the large neural network, wherein the large neural network has a plurality of large network parameters;
• for each output sequence in the batch:
• determining a performance metric of the large neural network on the particular neural network task (i) in accordance with current values of the large network parameters and (ii) with only the components specified by the output sequence active;
• and
• using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for determining an architecture for a neural network for performing a particular neural network task, the operations comprising:

• generating, in accordance with current values of a plurality of controller parameters, a batch of output sequences, each output sequence in the batch specifying a respective subset of a plurality of components of a large neural network that should be active during the processing of one or more inputs by the large neural network, wherein the large neural network has a plurality of large network parameters;
• for each output sequence in the batch:
• determining a performance metric of the large neural network on the particular neural network task (i) in accordance with current values of the large network parameters and (ii) with only the components specified by the output sequence active;
• and
• using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for determining an architecture for a neural network for performing a particular neural network task, the operations comprising:

• generating, in accordance with current values of a plurality of controller parameters, a batch of output sequences, each output sequence in the batch specifying a respective subset of a plurality of components of a large neural network that should be active during the processing of one or more inputs by the large neural network, wherein the large neural network has a plurality of large network parameters;
• for each output sequence in the batch:
• determining a performance metric of the large neural network on the particular neural network task (i) in accordance with current values of the large network parameters and (ii) with only the components specified by the output sequence active;
• and
• using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters.