Neural architecture search through a graph search space
AssigneeGoogle Inc.
InventorsNeil Matthew Tinmouth Houlsby, Quentin Lascombes de Laroussilhe, Stanislaw Kamil Jastrzebski, Andrea Gesmundo
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes receiving training data for training a task neural network to perform a particular machine learning task; and selecting, from a space of possible architectures, an architecture for the task neural network, wherein the space of possible architectures is represented as a graph of nodes connected by edges, each node in the graph representing a decision point in selecting the architecture and each edge in the graph representing an action.
Claims21
A method performed by one or more computers, the method comprising:
- receiving training data for training a task neural network to perform a particular machine learning task;
- and
- selecting, from a space of possible architectures, an architecture for the task neural network, wherein the space of possible architectures is represented as a graph of nodes connected by edges, each node in the graph representing a decision point in selecting the architecture and each edge in the graph representing an action, the selecting comprising repeatedly performing the following:
- selecting, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, one or more paths through the graph, each path comprising a plurality of nodes that are each connected by an edge to at least one other node in the path, and each path defining a candidate architecture for the task neural network;
- for each selected path:
- generating an instance of the task neural network having the candidate architecture defined by the path;
- training the instance to perform the particular machine learning task;
- and
- determining, for the trained instance, a performance measure on the particular task;
- and
- training the controller neural network using the performance measures for the candidate architectures to determine an update to the current values of the controller parameters that improves the performance measures for architectures defined by paths generated by the task neural network.
(canceled)
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- receiving training data for training a task neural network to perform a particular machine learning task;
- and
- selecting, from a space of possible architectures, an architecture for the task neural network, wherein the space of possible architectures is represented as a graph of nodes connected by edges, each node in the graph representing a decision point in selecting the architecture and each edge in the graph representing an action, the selecting comprising repeatedly performing the following:
- selecting, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, one or more paths through the graph, each path comprising a plurality of nodes that are each connected by an edge to at least one other node in the path, and each path defining a candidate architecture for the task neural network;
- for each selected path:
- generating an instance of the task neural network having the candidate architecture defined by the path;
- training the instance to perform the particular machine learning task;
- and
- determining, for the trained instance, a performance measure on the particular task;
- and
- training the controller neural network using the performance measures for the candidate architectures to determine an update to the current values of the controller parameters that improves the performance measures for architectures defined by paths generated by the task neural network.
A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- receiving training data for training a task neural network to perform a particular machine learning task;
- and
- selecting, from a space of possible architectures, an architecture for the task neural network, wherein the space of possible architectures is represented as a graph of nodes connected by edges, each node in the graph representing a decision point in selecting the architecture and each edge in the graph representing an action, the selecting comprising repeatedly performing the following:
- selecting, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, one or more paths through the graph, each path comprising a plurality of nodes that are each connected by an edge to at least one other node in the path, and each path defining a candidate architecture for the task neural network;
- for each selected path:
- generating an instance of the task neural network having the candidate architecture defined by the path;
- training the instance to perform the particular machine learning task;
- and
- determining, for the trained instance, a performance measure on the particular task;
- and
- training the controller neural network using the performance measures for the candidate architectures to determine an update to the current values of the controller parameters that improves the performance measures for architectures defined by paths generated by the task neural network.
Dependencies are inferred from the claim text.