# US 2020/0401899 A1 · Computationally efficient neural network architecture search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401899 A1
- Document ID: US20200401899A1
- Status: Published application
- Kind: A1
- Application number: 16/447,866
- Filed: 2019-06-20
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: David Martin Dohan; David Richard So; Chen Liang; Quoc V. Le
- Web page: https://patentgrep.com/patent/US20200401899A1

## CPC

- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for receiving training data for training a neural network to perform a machine learning task and for searching for, using the training data, an optimized neural network architecture for performing the machine learning task is described. Searching for the optimized neural network architecture includes: maintaining population data; maintaining threshold data; and repeatedly performing the following operations: selecting one or more candidate architectures from the population data; generating a new architecture from the one or more selected candidate architectures; for the new architecture: training a neural network having the new architecture until termination criteria for the training are satisfied; and determining a final measure of fitness of the neural network having the new architecture after the training; and adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

## Claims

### Claim 1

A method comprising:
receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples; and
searching for an optimized neural network architecture for performing the machine learning task, comprising:
maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) a measure of fitness representing the performance of the candidate architecture;
maintaining threshold data comprising a set of threshold fitness values, wherein each threshold fitness value is associated with a corresponding checkpoint during the training of the neural network, each checkpoint being a different specific point over a course of the training of the neural network; and
repeatedly performing the following operations:
selecting one or more candidate architectures from the population data;
generating a new architecture from the one or more selected candidate architectures;
for the new architecture:
training a neural network having the new architecture at least until reaching a first checkpoint of the plurality of checkpoints and until termination criteria for the training are satisfied, wherein the termination criteria specify that at each checkpoint, the training is terminated at the checkpoint and will not be resumed at any other checkpoints that are after the checkpoint whenever a measure of fitness of the neural network having the new architecture as of the checkpoint does not satisfy the threshold fitness value corresponding to the checkpoint; and
determining a final measure of fitness of the neural network having the new architecture after the training; and
adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the operations further comprise:
determining whether to add a new threshold fitness value to the set of threshold fitness values; and
in response to determining to add a new threshold fitness value:
determining a mean of the measures of fitness of at least some of the candidate architectures in the population data;
adding the mean of the measures of fitness as a new threshold fitness value to the set of threshold fitness values; and
associating the new threshold fitness value with a corresponding checkpoint.

### Claim 3 (depends on 1)

The method of claim 1, wherein the operations further comprise:
removing one or more candidate architectures from the population data based on their respective measures of fitness.

### Claim 4 (depends on 1)

The method of claim 1, wherein the operations further comprise:
initializing the population with a plurality of default candidate architectures.

### Claim 5 (depends on 1)

The method of claim 1, wherein the data defining the candidate architecture comprises:
data specifying an architecture for one or more components that are each repeated multiple times to generate the candidate architecture.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating a new architecture from the one or more selected candidate architectures comprises:
modifying the architecture for at least one of the components in one of the selected candidate architectures.

### Claim 7 (depends on 6)

The method of claim 6, wherein modifying the architecture for at least one of the components in one of the selected candidate architectures comprises:
randomly selecting a mutation from a set of mutations; and
applying the randomly selected mutation to the architecture for the component.

### Claim 8 (depends on 6)

The method of claim 6, wherein modifying the architecture for at least one of the components in one of the selected candidate architectures further comprises:
selecting a component in a first one of the selected candidate architectures; and
modifying a corresponding component in a second one of the selected candidate architectures to have the same architecture as the selected component in the first one of the selected candidate architectures.

### Claim 9 (depends on 1)

The method of claim 1, wherein selecting one or more candidate architectures from the population data comprises:
randomly selecting a plurality of candidate architectures from the population data; and
selecting, from the plurality of randomly selected candidate architectures, one or more candidate architectures based on their respective measures of fitness.

### Claim 10 (depends on 1)

The method of claim 1, wherein searching for an optimized neural network architecture for performing the machine learning task further comprises:
after repeatedly performing the following operations, selecting the candidate architecture in the population having the best measure of fitness as the optimized architecture.

### Claim 11 (depends on 1)

The method of claim 1, wherein the machine learning task is one or more of the following: image classification, object detection, machine translation or speech recognition.

### Claim 12

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations comprising:
receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples; and
searching for an optimized neural network architecture for performing the machine learning task, comprising:
maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) a measure of fitness representing the performance of the candidate architecture;
maintaining threshold data comprising a set of threshold fitness values, wherein each threshold fitness value is associated with a corresponding checkpoint during the training of the neural network, each checkpoint being a different specific point over a course of the training of the neural network; and
repeatedly performing the following operations:
selecting one or more candidate architectures from the population data;
generating a new architecture from the one or more selected candidate architectures;
for the new architecture:
training a neural network having the new architecture at least until reaching a first checkpoint of the plurality of checkpoints and until termination criteria for the training are satisfied, wherein the termination criteria specify that at each checkpoint, the training is terminated at the checkpoint and will not be resumed at any other checkpoints that are after the checkpoint whenever a measure of fitness of the neural network having the new architecture as of the checkpoint does not satisfy the threshold fitness value corresponding to the checkpoint; and
determining a final measure of fitness of the neural network having the new architecture after the training; and
adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

### Claim 13 (depends on 12)

The system of claim 12, wherein the search operations further comprise:
determining whether to add a new threshold fitness value to the set of threshold fitness values; and
in response to determining to add a new threshold fitness value:
determining a mean of the measures of fitness of at least some of the candidate architectures in the population data;
adding the mean of the measures of fitness as a new threshold fitness value to the set of threshold fitness values; and
associating the new threshold fitness value with a corresponding checkpoint.

### Claim 14 (depends on 12)

The system of claim 12, wherein the search operations further comprise:
removing one or more candidate architectures from the population data based on their respective measures of fitness.

### Claim 15 (depends on 12)

The system of claim 12, wherein the search operations further comprise:
initializing the population with a plurality of default candidate architectures.

### Claim 16 (depends on 12)

The system of claim 12, wherein the data defining the candidate architecture comprises:
data specifying an architecture for one or more components that are each repeated multiple times to generate the candidate architecture.

### Claim 17 (depends on 16)

The system of claim 16, wherein generating a new architecture from the one or more selected candidate architectures comprises:
modifying the architecture for at least one of the components in one of the selected candidate architectures.

### Claim 18 (depends on 17)

The system of claim 17, wherein modifying the architecture for at least one of the components in one of the selected candidate architectures comprises:
randomly selecting a mutation from a set of mutations; and
applying the randomly selected mutation to the architecture for the component.

### Claim 19 (depends on 17)

The system of claim 17, wherein modifying the architecture for at least one of the components in one of the selected candidate architectures further comprises:
selecting a component in a first one of the selected candidate architectures; and
modifying a corresponding component in a second one of the selected candidate architectures to have the same architecture as the selected component in the first one of the selected candidate architectures.

### Claim 20 (depends on 12)

The system of claim 12, wherein selecting one or more candidate architectures from the population data comprises:
randomly selecting a plurality of candidate architectures from the population data; and
selecting, from the plurality of randomly selected candidate architectures, one or more candidate architectures based on their respective measures of fitness.

### Claim 21

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations comprising:
receiving training data for training a neural network to perform a machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples; and
searching for an optimized neural network architecture for performing the machine learning task, comprising:
maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) a measure of fitness representing the performance of the candidate architecture;
maintaining threshold data comprising a set of threshold fitness values, wherein each threshold fitness value is associated with a corresponding checkpoint during the training of the neural network, each checkpoint being a different specific points over a course of the training of the neural network; and
repeatedly performing the following operations:
selecting one or more candidate architectures from the population data;
generating a new architecture from the one or more selected candidate architectures;
for the new architecture:
training a neural network having the new architecture at least until reaching a first checkpoint of the plurality of checkpoints and until termination criteria for the training are satisfied, wherein the termination criteria specify that at each checkpoint, the training is terminated at the checkpoint and will not be resumed at any other checkpoints that are after the checkpoint whenever a measure of fitness of the neural network having the new architecture as of the checkpoint does not satisfy the threshold fitness value corresponding to the checkpoint; and
determining a final measure of fitness of the neural network having the new architecture after the training; and
adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

### Claim 22 (depends on 1)

The method of claim 1, wherein:
each checkpoint specifies a specific number of training iterations and is reached when the specific number of training iterations have been completed since a preceding checkpoint or since a beginning of the training of the neural network having the new architecture.

### Claim 23 (depends on 1)

The method of claim 1, wherein:
each checkpoint specifies a specific quantity of elapsed time and is reached when the specified quantity of time has elapsed since a preceding checkpoint or since the beginning of the training of the neural network having the new architecture.