Hardware-optimized neural architecture search
AssigneeGoogle Inc.
InventorsSheng Li, Norman Paul Jouppi, Quoc V. Le, Mingxing Tan, Ruoming Pang, Liqun Cheng, Andrew Li
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining an architecture for a task neural network that is configured to perform a particular machine learning task on a target set of hardware resources. When deployed on a target set of hardware, such as a collection of datacenter accelerators, the task neural network may be capable of performing the particular machine learning task with enhanced accuracy and speed.
Claims20
A method of determining an architecture for a task neural network that is configured to perform a particular machine learning task when deployed on a target set of hardware resources, wherein the method comprises:
- receiving training data for performing the particular machine learning task;
- performing, using the training data, a search through a space of candidate neural network architectures to identify one or more candidate neural network architectures, wherein performing the search comprises repeatedly performing the following:
- selecting a candidate neural network architecture from the space, wherein selecting the candidate neural network architecture comprises selecting, for each of one or more components of the candidate neural network architecture, an operation to be performed by the component from a set of operations comprising (1) a space-to-depth convolution operation that increases a depth of an input tensor while decreasing a spatial extent of the input tensor and (2) one or more other types of convolution operations;
- and
- determining a measure of performance of the selected candidate neural network architecture based on (i) a performance of selected candidate neural network architecture on the particular machine learning task and (ii) a performance of the selected candidate neural network architecture when deployed on the target set of hardware resources;
- generating an architecture for the task neural network using the identified candidate neural network architectures.
One or more non-transitory storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for determining an architecture for a task neural network that is configured to perform a particular machine learning task when deployed on a target set of hardware resources, wherein the operations comprise:
- receiving training data for performing the particular machine learning task;
- performing, using the training data, a search through a space of candidate neural network architectures to identify one or more candidate neural network architectures, wherein performing the search comprises repeatedly performing the following:
- selecting a candidate neural network architecture from the space, wherein selecting the candidate neural network architecture comprises selecting, for each of one or more components of the candidate neural network architecture, an operation to be performed by the component from a set of operations comprising (1) a space-to-depth convolution operation that increases a depth of an input tensor while decreasing a spatial extent of the input tensor and (2) one or more other types of convolution operations;
- and
- determining a measure of performance of the selected candidate neural network architecture based on (i) a performance of selected candidate neural network architecture on the particular machine learning task and (ii) a performance of the selected candidate neural network architecture when deployed on the target set of hardware resources;
- generating an architecture for the task neural network using the identified candidate neural network architectures.
A method comprising:
- receiving a network input;
- and
- processing the network input using a task neural network to generate an output for the network input for a particular machine learning task, wherein processing the network input using the task neural network comprises:
- processing the network input using an initial convolutional neural network sub-network comprising one or more convolutional layers and configured to generate an initial feature representation of the network input that has a first spatial extent and a first number of depth channels;
- performing a space-to-depth convolutional operation on the initial feature representation of the network input to generate a second feature representation that has:
- (i) second spatial extent that is smaller than the first spatial extent and (ii) a second number of depth channels that is larger than the first number;
- and
- processing the second feature representation using one or more additional sub-networks to generate the output for the network input.
Dependencies are inferred from the claim text.